Holo3-35B-A3B · The KV-cache optimum — bounded screenshots, cache hit every step SYSTEMS · GUIDE

Cross-reference: the architecture facts, the eviction/no-eviction mechanics, and the hybrid-model constraints referenced below are established in the sibling page Holo3 on OSWorld — Acceleration. Read it first if you have not.

1. The goal, and the optimum defined

The goal is a Holo3-35B-A3B GUI agent on OSWorld-Verified, served on a single Intel NUC (96 GB) via llama.cpp CPU, that runs at eviction-level quality with no eviction-level cost. We have two real arms today, and each pays for what the other gets for free:

• Evict (re-prefill the last 3 screenshots + the growing text history every step) keeps context bounded — so quality, decode speed, and memory are all good — but pays a full re-prefill every step (131.5 s/step prefill).
• No-evict (append everything, reuse the byte-stable prefix as a real cache hit) prefills only the genuinely new content per step (62.9 s) — but lets context grow unbounded, so decode degrades, memory blows up into an OOM tail, and accuracy drops 6.4 pp.

The theoretical optimum is: "evict, with the re-prefill cost magically removed." Concretely — keep evict's bounded 3-screenshot context (so quality, decode rate, memory, trajectory, and score are byte-identical to evict), but pretend every step is a perfect cache hit so that only the genuinely new content (the one new screenshot + the text delta) is ever prefilled (so prefill equals no-evict's only-new cost).

It is essential to be clear: this is not a system we built and not a run we executed. It is a number COMPUTED from the two arms we already measured (theoretical_optimum.py over the live no-evict run). It is the tightest upper bound a perfect KV cache could reach on a bounded screenshot window, and nothing more.

2. Deriving the numbers

The optimum is an honest cross-arm splice. Each column is sourced from exactly one measured arm, with a stated justification — it is not a cherry-pick of each arm's best cell.

Prefill ← no-evict (62.9 s, 2353 tok/step). No-evict physically measured the cost of prefilling only the new content per step: its byte-stable prefix is a genuine cache hit, so its reported per-step prompt_tokens (~2353) are the non-cached tokens actually processed, not the full context. (This is verified: no-evict's cached_tokens grow 0 → 44K while the new prompt_tokens it must prefill stays flat at ~2160–2225 regardless of context depth. If 2353 were the total context it would exceed evict's 5143 on multi-step tasks — it is smaller, proving it is post-cache.) In a bounded 3-window served by a perfect cache, the genuinely-new content per step is the same quantity — add one screenshot and a text delta, drop the oldest screenshot — so no-evict's measured only-new prefill is the optimum's prefill. This is why the prefill-equivalence is real, not assumed.

Decode, score, steps, memory ← evict (19.5 s decode, 0.858, 9.4 steps, bounded). The optimum keeps evict's bounded context and feeds the model byte-identical inputs — the only thing the fiction changes is prefill timing, never the KV contents decode sees. Therefore:

• The model emits the exact same output tokens as evict ⇒ identical score (0.858) and identical step count (9.4). These are copied from evict by construction, not independently measured — presenting them as a third system's observed outcomes would mislead.
• Decode runs over evict's small KV at evict's fast rate (19.5 s, ~15 t/s), not no-evict's degrading 23.0 s.

Why decode is legitimately evict's, and why it must NOT be no-evict's. Decode rate is governed by total KV residency (context length), not by how the KV got there. A cache hit and a fresh recompute produce numerically identical KV tensors at identical positions; decode-time attention is a fixed memory-bandwidth sweep over those tensors, so there is no mechanism by which provenance changes per-token cost. Empirically, no-evict's per-step decode falls monotonically 17.5 → 4.0 t/s as reconstructed context grows 5K → 44K (n=631 step-profiles) — purely a context-size effect. Cross-check: evict's measured mean decode is 14.96 t/s and no-evict's decode in the matched 5–10K-token bucket is 14.9 t/s — equal context ⇒ equal decode across arms, to within noise. The optimum holds evict's bounded context, so it inherits evict's ~15 t/s by construction. (Note the hybrid architecture is why decode is still context-dependent at all: the 30 Gated-DeltaNet layers are O(1) per token regardless of arm, but the 10 attention layers impose a ctx-linear term — a linear fit gives decode_time/token ≈ 31.3 ms + 4.6e-6 s × ctx. The O(1) recurrent majority does not make decode context-free.) Predicted-token counts are comparable across arms (~268 no-evict vs ~291 evict), so evict's faster decode is a genuine small-KV effect, not an artifact of generating fewer tokens.

Wall ← computed by swapping only the prefill component.

opt_wall = evict_wall − (evict_prefill − optimum_prefill) × steps
         = 1818.9 − (131.5 − 62.9) × 9.4
         ≈ 858.1 s

Only the prefill component is replaced; VM/action time and decode are held fixed at evict's values. This is safe because model time is 94.9 % of wall and non-model time is only 5.1 % — a small, prefill-independent constant (verified: 0/78 negative residuals). The wall identity swaps only prefill; the implied VM/overhead residual (~5 % of wall) is nonnegative on all 78 tasks and held fixed, so there is no hidden second saving.

Energy ≈ wall (9.4 Wh). Energy is scaled by the wall ratio under near-constant package power (47.2 W ± 9.5 W). Because the removed seconds are compute-bound prefill (above-average power), constant-power scaling under-credits the saving — so 9.4 Wh is, if anything, slightly conservative (a hair too high), which is the safe direction for an upper bound. The per-task energy ratio equals the per-task wall ratio by construction; the aggregate gap (energy 1.93× vs wall 2.12×) is a mean-of-ratios-vs-ratio-of-means weighting artifact, not an independent energy effect.

Why the optimum's wall (858 s) can EXCEED no-evict's (784 s) yet still dominate it. The optimum keeps evict's 9.4-step trajectory at full quality, while no-evict reaches the finish in only 8.1 steps — but at −6.4 pp accuracy and with an unbounded-memory OOM tail. The optimum does not double-dip on no-evict's cheaper, lower-quality trajectory. Per step, the optimum (82.4 s) actually beats no-evict (85.9 s): it pairs no-evict's cheap prefill with evict's non-degrading decode (19.5 s vs no-evict's 23.0 s). So within the optimum's contract — byte-identical outputs to evict, bounded window, VM fixed — it is the same speed class as no-evict with none of the accuracy loss or memory blowup.

What the fiction assumes away. On this hybrid model the 2 retained screenshots' attention KV depends on the evicted screenshot (cross-attention), and their absolute positions shift when the oldest is dropped — so a real bounded cache cannot reuse them for free. The 30 Gated-DeltaNet recurrent layers carry an O(1) running state that cannot be mid-window spliced at all. The bound assumes every step is a free cache hit anyway. This is exactly what makes 858 s an upper bound (unreachable while bounded) and not a buildable number.

3. The coefficient table

Matched clean subset, n = 78, COMPUTED upper bound. The subset excludes 8 runaway tasks (> 20 steps / > ~60K-token context — the memory-blowup tail); all three arms are over the same tasks. evict and no-evict are measured; the theoretical optimum is computed from those two arms (it is not a run).

Notes that belong with the table:

• The optimum keeps evict's 9.4-step trajectory, not no-evict's 8.1 steps. It borrows evict's verbatim output trajectory; that is what makes score, steps, and decode-seconds all inherit exactly.
• The "100 % cache hit" cell is an assumed fiction, distinct from no-evict's measured ~64 % real prefix hit.
• The gain is length-dependent: per-task prefill gain rises from 1.21× on ≤3-step tasks to 2.61× on 11–20-step tasks. The single optimum column is a mean over a length mix — short tasks are near-parity (on 1 of 78 — a 2-step Chrome task — no-evict prefill 61.4 s marginally exceeds evict's 61.3 s, so the per-task optimum is a hair worse; benign averaging noise on ultra-short tasks), and long tasks dominate the win.
• On this clean subset, no-evict's context stays modest (max ~3K tokens even at 10 steps). The dramatic unbounded-growth / OOM tail lives in the 8 excluded tasks — so the bound's memory/decode-degradation advantage is real but modest on-subset, and the catastrophic divergence is argued off-subset.

4. Why the pure bound is unreachable while bounded — and how to physically approach it

Why exact is impossible on this hybrid model. Of Holo3's 40 layers, only 10 are full-attention (position-indexed KV); 30 are Gated-DeltaNet recurrent layers carrying an O(1) running state, not per-token KV (see the sibling page). Splicing a token out of the middle of the sequence — which is what a bounded sliding window requires — is inert and unsound on the recurrent half: seq_rm + seq_add + K-shift / --cache-reuse would corrupt 30/40 layers. So the pure bound (prefill only the ~1 new screenshot, ~2.4K tok) cannot be realized exactly while the window is bounded. The bound is the target; it is not the build.

No-evict is the arm that realizes the prefill bound (2353 tok/step, only-new) — but only by leaving context unbounded, which is what costs it decode degradation, the OOM tail, and −6.4 pp. It is the measured proof that only-new prefill is achievable; it is not a stable system.

The closest physically-realizable approach — terminal-slot + state checkpoint/restore. This is exact for all 40 layers and bounded, at the price of re-prefilling the current ≤3-image window (~6K tok/step) instead of the bound's ~1 image (~2.4K). Lay the prompt as:

[preamble] [append-only text history] [ ≤3 screenshots at the END ]

Checkpoint the full state (attention KV + recurrent state) at the start of the image region. llama.cpp already ships the primitives: --ctx-checkpoints, and llama_state_seq_get_data_ext / llama_state_seq_set_data_ext with LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY.

Per-step algorithm:

1. Restore the checkpoint (state up through the preamble + text history; image region empty).
2. Prefill only the current ≤3-image window (~6K tok constant; the text history is free — it is inside the checkpoint).
3. Decode the action over the small, bounded KV.
4. Append the new action text to the history.
5. Advance the checkpoint to absorb that new text, so the image region's start stays at the tail.

Because the checkpoint sits before the image region, this pays more than the bound (3 images/step ≈ 6K tok vs the bound's ~1 image ≈ 2.4K) — but it is exact, bounded, and constant-cost. On long tasks it crushes evict outright (constant ~6K vs evict's growing 5K → 27K), so 858 s should be read as the target, with terminal-slot as the realizable system that approaches it from above.

5. Correctness note

• Terminal-slot is exact for all 40 layers. It never mid-splices the recurrent state; it restores a real checkpoint and re-prefills a real suffix, so attention and Gated-DeltaNet layers are both numerically correct.
• The bound itself is a fiction. 858 s / 9.4 Wh is a computed upper bound, not a run and not today-achievable. Keep this disclaimer adjacent to the headline numbers, and keep the bound (~2.4K tok/step, ~1 image) visibly distinct from the buildable terminal-slot approximation (~6K tok/step, 3 images) so 858 s is never misread as shippable today.
• Could anything beat it? Only by changing the contract — accepting no-evict's 8.1-step trajectory lowers wall but is a different, lower-quality trajectory (−6.4 pp), outside this bound's "byte-identical outputs to evict" definition. Within the contract you cannot prefill fewer than the genuinely-new tokens, so the bound is a legitimate floor on prefill work and thus a legitimate ceiling on speed/efficiency.
• If you wanted to approach the ~2.4K bound rather than the ~6K terminal-slot cost, the knob is selective recompute of the image region à la CacheBlend / EPIC — recompute only the KV entries whose cross-attention dependence on the dropped screenshot is significant, reusing the rest. That is an approximate middle-splice and trades exactness for getting nearer the bound; it is the research direction, not the shipping path.

Read first: this section assumes the architecture facts and the three-arm A/B numbers established on the sibling page, /demos/holo3-osworld-acceleration.html — the 10-attention / 30-Gated-DeltaNet hybrid split, why --cache-reuse is inert and unsound here, and the evict / no-evict / theoretical-optimum table. All numbers below are the matched clean subset, n = 78 (exclude rule: > 20 steps ≈ > 60 K-token context = the 8-task memory-blowup tail), recomputed live by theoretical_optimum.py over the current no-evict run.

The asterisk on everything in Clusters A–D: almost every result below was developed on standard transformers with per-token KV on all layers. On those models the "relocate / evict a KV segment in the middle and re-rotate positions" family is sound. On Holo3 it is sound only for the 10 attention layers — the 30 recurrent layers have no per-token KV to splice, so the exact analogue of any "KV segment" trick is a state checkpoint, not a KV edit.

1 · Related work, mapped to the problem (5 corpus clusters)

For each cluster: the most useful works, the concrete building block, and scope — does it apply to the bound (the computed 858 s / 9.4 Wh upper bound), to the buildable terminal-slot approach (the realizable ~6 k-tok/step approximation), or only to the 10 attention layers?

Cluster A — Modular / position-independent KV reuse (segment / slot caching)

This is the conceptual ancestor of "a screenshot is a reusable slot."

• Prompt Cache (Gim et al., MLSys 2024) — the segment/slot ancestor. It defines reusable prompt modules with an explicit schema that assigns each module a position range in the new prompt, so a cached segment can be slotted at a different offset than where it was computed (up to 8× lower TTFT on GPU, 60× on CPU, no quality loss). The screenshot-as-reusable-segment analogy is exact at the abstraction level: each screenshot is a modular KV slot with an assigned [p0,p1) range tracked from the mtmd injection. Its limit here is architectural — Prompt Cache assumes the segment is per-token KV re-slottable by re-assigning positions; on our 30 recurrent layers there is no per-token KV to re-slot. The faithful realization of a "reusable screenshot slot" is therefore a state checkpoint of the prefix that ends where the screenshots begin — not a relocatable KV chunk. This is exactly why our slots must be terminal. → informs the buildable terminal-slot layout.
• Block-Attention (Ma et al., ICLR 2025) — the de-rotate → zero → re-rotate-to-target RoPE math for shifting a cached block to its true global position (98.7 % TTFT reduction, ≤1 % RAG gap after a block fine-tune). Gives the exact angle arithmetic — but 1-D-RoPE text only and needs a fine-tune we cannot apply to a fixed Q4_K_M model. → 10 attention layers only.
• ChunkAttention (Ye et al., ACL Findings 2024) — chunked KV in a prefix tree + two-phase (shared-prefix / fresh-suffix) kernel. This is the compute shape of "restore prefix + prefill only the new screenshots." → buildable terminal-slot (kernel pattern).

Verdict (A): the slot abstraction is right; the implementation (relocatable KV chunk + RoPE re-rotation) is sound only for the 10 attention layers. The 30 recurrent layers force the exact analogue to be a state checkpoint, which is what makes terminal-slot checkpoint-restore — not a KV-pointer swap — correct.

Cluster B — Streaming / sliding-window KV + attention sinks + position renumbering

Legitimizes "renumber surviving positions inside the cache" — the path the sibling page ruled out here.

• StreamingLLM (Xiao et al., ICLR 2024) — keep ~4 attention-sink tokens + a rolling window and assign positions within the cache, not from source text (22.2× over sliding-window recompute, stable to 4 M tokens). This is exactly the seq_rm+seq_add+K-shift triad we cannot use on the recurrent half.
• LM-Infinite (Han et al., NAACL 2024) / Self-Extend (Jin et al., ICML 2024) — the distance-limit principle and the proof that position ids can be rewritten inference-only.

Verdict (B): every primitive edits per-token KV positions → 10 attention layers only, and on Holo3 the full triad silently corrupts the 30 recurrent ones. What this cluster does give the buildable scheme are two disciplines: (1) pin the sink — keep the preamble prefix stable (it is the checkpoint root); (2) distance-limit — because the ≤K screenshots sit at the end and text history is append-only, retained positions never leave the trained range, which is why the terminal layout is benign.

Cluster C — Importance-based KV eviction / compression (which tokens to keep)

Mostly dissolved by terminal slots — with the ≤3 most-recent screenshots always re-prefilled at the end, we never choose which old screenshot KV to drop.

• GUI-KV (Huang et al., Salesforce, 2025) — screenshot-trajectory-native (spatial L2-saliency + cross-frame temporal redundancy); reports +4.1 pp step accuracy at −38.9 % FLOPs on a 5-screenshot context. Direct empirical support for our load-bearing claim that bounded screenshot context ≠ worse — the same claim that lets the optimum copy evict's score (0.858) at a bounded window. STaR-KV (Han et al., 2026) and VL-Cache (Tu et al., ICLR 2025) extend this (modality-aware, layer-adaptive budgets).
• Quest (Tang et al., ICML 2024) — the conceptual pivot for the cache-HIT constraint: it never evicts, keeps full KV resident, and gates which pages enter attention, so the prefix stays bit-identical. The "keep it resident, don't truncate" worldview behind the bound.

Verdict (C): these choose what to drop; terminal-slot needs no victim selection. They become relevant only for K > 3 (GUI-KV/STaR-KV redundancy scoring to thin extra screenshots) or to push the bound on the attention layers only (VL-Cache budgets, Quest page-gating) — both safe to layer on top of checkpoint-restore, neither replaces it. → mostly bound rationale + optional attention-layer add-ons.

Cluster D — Serving systems: paged / prefix / block KV + reuse infra

The production substrate that would hit the same hybrid constraint.

• PagedAttention / vLLM (Kwon et al., SOSP 2023) — block-table indirection (the structure that makes "free one screenshot's blocks" expressible). RadixAttention / SGLang (Zheng et al., NeurIPS 2024) — radix-tree prefix index + LRU.
• LMCache (Liu et al., 2025) — the decisive independent fact: production context truncation drops the prefix-cache hit ratio from ~85 % to ~45 % on real traces — production-scale corroboration of our measured 64 % → 0 % collapse (evict vs no-evict), and the motivation for not truncating the prompt.
• CachedAttention / AttentionStore (Gao et al., USENIX ATC 2024) — the closest serving analogue of our mechanism: a positional-encoding-decoupled KV save/restore that survives a window slide and re-embeds positions on load (87 % TTFT reduction) — the session-granularity ancestor of checkpoint-restore.

Verdict (D): all treat KV as a first-class object — the worldview in which "checkpoint this prefix, restore it next step" is a scheduling decision. But the same hybrid constraint applies: every one is prefix/block/text oriented; none models M-RoPE image positions or a Gated-DeltaNet recurrent state, and vLLM/SGLang currently lack mtmd + Gated-DeltaNet checkpoint serialization — which llama.cpp already ships via llama_state_seq_get/set_data_ext(LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY). → infra for the buildable path, but only llama.cpp is recurrent-state-ready today.

Cluster E — Multimodal / VLM KV management + M-RoPE position correctness

The model-spec and risk cluster.

• Qwen3-VL Technical Report (2025) / Qwen2-VL (Wang et al., 2024) — define M-RoPE (temporal/height/width); this is Holo3's position spec. The llama.cpp artifacts PR #13870 (M-RoPE shift as NEOX whole-vector, θ=−1) and issue #17930 (Qwen2.5-VL M-RoPE sequence-position error) confirm that position-relocating image spans is a fragile workaround still failing upstream — the open risk we avoid entirely by never relocating image positions.
• VLCache (Qin et al., Qwen Team / Alibaba, 2025) — the multimodal realization on the same family as Holo3 (Qwen3-VL-30B-A3B): store image KV pre-RoPE, hash-key by pixel content, re-apply position at the destination, recompute only 2–5 % of vision tokens (1.2–16× TTFT, in SGLang). The most transplantable template for the ~2.4 k bound — but it presumes per-token KV on every layer (no defined meaning on the recurrent half) and lives above libllama. → template for approaching the bound on the 10 attention layers, not a bounded recurrent solution.

Verdict (E): the headline gift is negative and clarifying — the M-RoPE-shift correctness risk that dominates Clusters A/B is designed out by the terminal-slot layout, because we never move an image token's position; we restore an exact prefix and re-encode the screenshots fresh.

2 · Practical paths to APPROACH the bound, ranked (easiest / most reliable first)

3 · Concrete experiment plan

Instrument, per step, on all arms (extend theoretical_optimum.py's step-profile logging): - per-step prefill tokens (split: cached-hit vs newly-processed — the post-cache "genuinely new" quantity), - KV cache-hit ratio (and label fictions vs measurements: optimum = assumed 100 %, no-evict = measured ~64 %, evict = 0 %), - decode t/s (the load-bearing variable — verify it tracks total KV residency, not provenance), - score (per-task, OSWorld-Verified), - peak KV memory.

A/B design — same n=78 clean subset, same tasks, exclude > 20 steps:

Success criteria for the terminal-slot build: 1. Match evict's score (no −6.4 pp) — token-for-token output parity is the goal; first run the exactness sanity check (assert restore-then-prefill yields identical logits to a from-scratch full prefill on a fixed prompt — numerical proof the checkpoint is exact on this hybrid model). 2. Hold per-step prefill ≈ constant while bounded (~6 k tok/step regardless of task length), so wall/energy approach the 858 s / 9.4 Wh bound as task length grows (the gain is length-dependent: ~1.21× on ≤3-step tasks rising to ~2.61× on 11–20-step tasks). 3. Bounded peak KV memory (no growth, no OOM tail).

Failure signal: terminal-layout accuracy drift — if clustering screenshots at the tail and keeping older observations only as text moves agent behavior, score drops below evict. The fix is a layout tweak (what text survives, guided by GUI-KV/STaR-KV redundancy and FastV's "visual info migrates to text in early layers", ECCV 2024), not a correctness fix — the mechanism stays exact.

4 · Verdict

The theoretical optimum is a COMPUTED bound, not a system we run: it splices prefill from no-evict (the genuinely-new 2353 tok/step, trajectory-independent) onto score / steps / decode / memory from evict (copied by construction under the no-input-change argument — it keeps evict's 9.4-step trajectory, not no-evict's cheaper 8.1, so it does not cherry-pick), yielding 2.12× wall (1818.9 → 858.1 s), 1.93× energy (18.3 → 9.4 Wh), and zero quality loss vs evict (0.858) while bounded — dominating no-evict because it pairs cheap prefill with evict's non-degrading decode (82.4 s/step vs no-evict's 85.9 s). It is a fiction because on this hybrid model the retained screenshots' attention KV depends on the evicted one and their positions shift, while the 30 Gated-DeltaNet layers carry an O(1) state that cannot be mid-window spliced — so a real bounded cache cannot reuse them for free. The buildable terminal-slot + checkpoint-restore scheme is the closest physically-realizable approach: exact for all 40 layers, bounded, full-quality, but paying ~6 k tok/step (3 images) rather than the bound's ~2.4 k (≈1 image) because the checkpoint sits before the image region — so it lands between no-evict and the bound on short tasks and crushes evict on long ones. The single highest-leverage first step is path 2(a): build terminal-slot + --ctx-checkpoints and run the exactness sanity check — it needs no model change, the llama.cpp primitives (--ctx-checkpoints, llama_state_seq_*_data_ext(PARTIAL_ONLY)) already ship, and a passing logit-parity assertion immediately converts the computed bound from an argument into a realizable, bounded, full-quality acceleration. See /demos/holo3-osworld-acceleration.html §2b for the mechanism's architectural derivation.