Research / Benchmarking · Configured local deployment

Strata on a 16 GiB RTX 4080, through a 256K window

Published 9 November 2026

How does a memory-constrained local configuration behave as input grows, and what changes when the prefix is reused?

131.039 sCold prefill at 261,112 input tokens

96 msPrefix-cached prefill at the same length

160 / 160Raw exact JSON in a separate stronger-prompt pass

How the system works

The standalone research article How Strata uses GPU, RAM and SSD explains expert routing, weight placement and KV cache, with a sourced mechanism comparison to vLLM.

The setup

One configured RTX 4080 (16 GiB, 200 W cap), i9-13900KF and approximately 64 GB RAM. The system ran NVIDIA driver 615.71.09 and Linux 7.0.0-34-generic. The model was two GGUF shards of Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS; weights are not redistributed here. CPU/RAM and NVMe offload with expert caching allow the model to run beyond the GPU’s 16 GiB of VRAM.

Strata 0.1.54-sessionmem was a local CUDA 13 / sm89 build based on upstream revision 870aedd9, with no-MTP save/restore and model-file-cache-advice changes. This measures the whole configuration, not stock Strata. Runtime and source disclosure.

Method

The context task placed three authoritative key/value records among synthetic filler; exact retrieval required copying each value as a string into KEY_0, KEY_1 and KEY_2. This is a narrow retrieval test, not a test of broad 256K reasoning. V7 used ten prompts repeated across five consecutive sessions: 50 observations per context and cache condition, not 50 independent prompts. Eight input lengths ran from 2,048 to 261,112 tokens. The frontier budget was 261,112 input + 1,024 reserved output + 8 internal tokens = 262,144.

Across 6 h 41 min 22 s, 1,375 calls completed: 850 measurements, 85 warmups and 440 resets, with no unresolved calls or inference retry. Cold means zero KV reuse, not a global OS cache flush; cached calls reused all but seven input tokens. The sessions do not establish across-day reliability.

Results

Cold and cached prefill medians across eight input lengths, with observed ranges. — V7 native prefill: medians and observed minimum–maximum ranges, 50 observations per condition. Cold is in seconds; cached is in milliseconds.

Cold and cached decode medians and observed ranges across eight input lengths. — V7 decode rates for short context-test answers of 72–83 tokens, 50 observations per condition. Ranges are observed minima and maxima, not confidence intervals.

Exact V7 timing medians

TTFT is dispatch to the first generated response delta; native prefill is the engine’s prompt-read phase. Each cell summarises 50 observations.

Input tokens	Cold prefill s	Cached prefill ms	Cold TTFT s	Cached TTFT ms	Cold decode tok/s	Cached decode tok/s
2,048	3.731	91.0	3.751	111.0	43.4	44.5
8,192	7.329	96.0	7.363	118.9	42.0	44.0
24,576	12.677	91.0	12.718	118.5	43.8	45.1
32,768	19.608	95.0	19.655	125.6	43.7	45.0
65,536	33.561	93.5	33.623	132.2	45.2	46.8
131,072	65.948	94.5	66.054	155.0	45.1	45.8
196,608	95.784	96.5	95.930	169.3	44.1	44.7
261,112	131.039	96.0	131.247	176.7	44.3	46.3

A separate sustained test used 50 input tokens and exactly 1,024 output tokens in each of 50 samples: median 41.614 tok/s, observed range 35.297–44.747. It was not a long-output test at 256K.

When exact JSON mattered

In V7, 0/800 context responses passed the original strict JSON-only check because they included Markdown fences. After posthoc fence-only removal, 800/800 matched the three expected values. The original score remains unchanged.

A separate 2K prompt intervention compared original, instruction-at-beginning and instruction-at-end arms. Exact values passed 0/10, 9/10 and 10/10; bare JSON syntax passed 0/10, 10/10 and 10/10 respectively.

A subsequent stronger-prompt run tested ten prompts per context/cache condition in one pass. All 160/160 raw responses passed exact JSON and complete-stop checks across the eight lengths in 1 h 03 min 51 s. This is not a new n=50 study.

Ten of ten cold and ten of ten cached responses passed exact JSON at each of eight input lengths. — Stronger-prompt follow-up: 10 cold and 10 cached observations per input length.

The tested directive was:

Return exactly one JSON object with exactly these three case-sensitive keys: KEY_0, KEY_1, KEY_2. For each key, copy the corresponding AUTHORITATIVE record's value exactly as a JSON string. AUTHORITATIVE is a source label and must not appear in the output key names. Do not add other fields, Markdown fences, headings or commentary.

Interpretation

These results describe inference and exact retrieval on the tested fixtures. They do not establish arbitrary-schema reliability, a quantization cause, or continuous autonomous agent and tool-use reliability. There was no matched stock-engine or alternative-precision baseline.

The current stock boot blacklists gpu_unlock and the module is absent; the current silicon rate is unknown. An experimental unlock succeeded on a different historical boot, but no practical speed benefit has been established.

Reproduce and inspect

The reproduction archive replays V7 timings and scoring, the 2K prompt intervention and the stronger-prompt follow-up offline. Exact generation replication additionally requires the recorded model, tokenizer, patched engine and operating profile. See the disclosure before interpreting a rerun.

Benchmark repository →Reproduction archive ↓V7 results CSV ↓Environment JSON ↓Source disclosure →Licence review →

The pinned engine source is MIT licensed. The quantization card’s Apache tag conflicts with its body’s inheritance of the base Qwen Community 1.0 licence. This report makes no blanket Apache claim for the weights or commercial-service clearance.

References and software credits

These sources credit the software, model and methods used. The measurements here are our own configured-deployment observations; citations do not imply upstream endorsement or transfer upstream benchmark scores to this setup.

Niko1221 and the Strata contributors. Strata, revision 870aedd9e1ebe1c91f94da93b59af17729f6c5c0. We measured a local 0.1.54-sessionmem build with the two disclosed patches; the upstream MIT notice is retained in the reproduction package.
ISTA-DASLab. Qwen3.8-Flash-Next-GSQ-RCO-GGUF, revision ed59f92082b1e93c0e96d60a8b11aab089b52f09, IQ2_XS, two shards. Base model: Qwen, Qwen3.8-Flash-Next. The quantizer does not pin a base revision. Model licence inheritance and metadata disagreement are documented separately.
Dadgarnia et al. (2026). GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling. arXiv:2604.18556v2, preprint.
Helcig and Alistarh (2026). Model Compression with Exact Budget Constraints via Riemannian Manifolds. arXiv:2605.00649v3, preprint.
NVIDIA. CUDA Toolkit 13.0 release notes. This documents the build-toolchain family, not a performance comparison. The observed driver version is recorded in the environment disclosure.
Hunter (2007). Matplotlib: A 2D Graphics Environment. Figures used Matplotlib 3.11.2; official citation guidance. Font notices remain in the package.
gpu-unlock project, background reference only. Our historical experimental fork/revision is disclosed in the source note. The present study does not attribute a speedup to an unlock.

Licences

Ryan approved MIT for original code and CC BY 4.0 for the authored report, original fixtures, measurements and figures. Third-party licences and generated-output rights remain separate.

Credits

Ryan Duffy led research design, configuration, acceptance and operational verification. Codex provided agent-assisted implementation, testing and analysis.