Holo3 · where prefill time actually goes

The measured answer — ms per token, by operation, as context grows

Flat bands (FFN/MoE, projections, Gated-DeltaNet recurrence, norms) are per-token constant — they cost the same whether the new chunk sits at depth 0 or 40k. Only the attention core rises, because each new token's flash-attention sweeps the growing KV of the 10 full-attention layers — while the 30 Gated-DeltaNet layers carry an O(1) state and stay flat regardless of depth. The dashed line marks where the rising attention band overtakes FFN/MoE.

The crossover, made literal — each band as its own line

Same data, un-stacked. The attention-core line climbs from 0.11 ms/tok at depth 0 to 12.66 ms/tok at 40k and crosses the flat FFN/MoE line (4.26 ms/tok) at L ≈ 13.6k tokens. Below the crossover the model is FFN/MoE-bound; above it, attention-bound — the regime almost every multi-step OSWorld task lives in once a few screenshots accumulate.

Per-depth table (subsampled)

How this was measured (and what each number is)

• Instrument. A standalone program links llama.cpp's build-cpu libraries and sets the public llama_context_params.cb_eval callback. In TIMED mode the callback returns true for every node, which makes the scheduler compute each ggml node alone and synchronise after it (ggml-backend.cpp compute-splits) — so observe−ask is that node's wall time. In FUSED mode it returns false, nodes fuse and run at native speed; we wrap llama_decode with a monotonic clock for the true marginal prefill time of a 512-token chunk at depth L.
• Bands. Each node is mapped to a band by its weight name (blk.<L>.attn_q/…), its ggml op (FLASH_ATTN_EXT→attention core, GATED_DELTA_NET/SSM_CONV→GDN, MUL_MAT_ID→MoE experts), and the layer's recurrent/attention type. Coverage of timed wall by the five bands is 100% — nothing falls through.
• ms/token. per-band = (band's TIMED share) × (FUSED chunk ms) ÷ 512. Anchoring shares to the fused wall removes the per-node isolation overhead; TIMED-total and FUSED-total agree to ~1.5% here (the heavy ops — MoE GEMMs, flash-attn, delta-net — don't fuse with neighbours, so their absolute time is the same in both modes).
• Honesty. Attention is a single fused FLASH_ATTN_EXT op per attention layer, so the "attention core" band already includes its softmax and the value sweep — it is not decomposed further. Quality is irrelevant to op timing (these matmul/softmax/scan kernels are data-independent), so the probe uses filler tokens. Measured CPU-only, -t 8 / -tb 16, matching the production server; clean room (server stopped), 3 cycles, median per depth. The 4 slow LP-E cores are inside the 16-thread batch set, identical for both modes, so they add wall jitter but cannot bias the shares.
• Independently verified. Three adversarial reviewers re-derived these numbers from the raw per-node dump: timing fidelity (per-node-isolated sum vs fused wall agree to 1.4%, no depth drift), band coverage (100%, with a cross-band audit that no heavy op is mis-binned), and physics (absolute scale 122.04 tok/s at depth 0 vs the server's measured ~118 tok/s; attention linear in L at R²≈0.9997). One real bug was caught and fixed — the 30 Gated-DeltaNet input GEMMs were briefly counted under projections — which is why GDN, not projections, is the second flat band.

model Holo3-35B-A3B.i1-Q4_K_M.gguf · n_ctx 41984 · ubatch 512 · 80 depths to 40k · cycles 3