Compile Ahead, Not Up Front

The result is the end of a progression. OpenVINO begins with static 8K allocation. A pyramid adds smaller graphs but still reserves the maximum KV. Copy-and-grow fixes average KV reservation but keeps every graph’s compile cost up front. One-ahead JIT finally overlaps those future compiles with useful NPU decoding.

Baseline 0Direct 8K graph

Baseline 1Shared-max pyramid

Improvement 1Copy-and-grow KV

BottleneckAll compiles up front

Improvement 2One-ahead JIT

Baseline · OpenVINO implementation

Two ways to run an 8K request

Long-context generation is dynamic: each token makes the KV cache a little larger. The NPU compiler, however, wants concrete tensor shapes. OpenVINO therefore has to choose what to compile and reserve before inference begins.

Design	Compiled graphs	KV allocation	Tradeoff
B0 · Direct	one 8K graph	8K from startup	simple, but small contexts run through a padded 8K shape
B1 · Upstream pyramid	1K / 2K / 4K / 8K	shared 8K maximum	better-sized graphs for short contexts, but every graph and the maximum KV are prepared up front

B1 solves the graph-shape problem: the runtime can select the smallest compiled graph that fits the current context. It does not solve proportional memory growth. Both baselines reserve an 8K KV allocation for the entire request.

Improvement 1 · Memory grows with context

Keep the pyramid, copy-and-grow the KV

Our first change keeps B1’s 1K/2K/4K/8K graph pyramid but replaces the shared-max KV with capacity-sized allocations. The request starts with 1K. When it crosses a boundary, the runtime allocates the next buffer, copies live KV state forward, binds the next graph, and continues.

Hand-drawn context-window ladder comparing preallocation and incremental KV growth — **Sketch 1 · Copy-and-grow.** Select the smallest graph that fits, allocate KV for that capacity, and move the live cache when the request crosses a boundary. Drawing by Vikranth Srivatsa.

Qwen2.5-7B prototype: crossing all three boundaries

The first full prototype successfully grew from 1K to 8K. Copy cost increased with the amount of live state, but remained small when amortized across the 8,064 decoded tokens.

Boundary	New KV bytes	Migration time	Top-1	Logit cosine
1K → 2K	+56 MiB	277.7 ms	7 / 7	0.9999944
2K → 4K	+112 MiB	317.0 ms	7 / 7	0.9999960
4K → 8K	+224 MiB	449.8 ms	7 / 7	0.9999976

31.9%lower average reserved KV: 319.7 MiB vs 469.7 MiB for B0

1.045 stotal migration time across 392 MiB of KV growth

0.1295 msmigration cost amortized per decoded token

Steady-state throughput remained close to the direct 8K baseline rather than collapsing under the extra handoffs:

Context	Copy-and-grow	Direct 8K	Δ vs direct
0–1K	0.32259 tok/s	0.32066 tok/s	+0.599%
1K–2K	0.32217 tok/s	0.32063 tok/s	+0.479%
2K–4K	0.32203 tok/s	0.32033 tok/s	+0.529%
4K–8K	0.32103 tok/s	0.32041 tok/s	+0.190%

This was the initial Qwen2.5-7B feasibility run. It establishes mechanism and boundary quality; the replicated N=5 experiment below evaluates the later JIT scheduling change.

What remained after Improvement 1

The KV is smaller, but startup is still eager

Three remaining bottlenecks

Compiled-capacity footprint. The runtime still materializes every 1K/2K/4K/8K artifact before decoding, so unified memory must hold work for capacities the request may never reach.
Transient peak memory. Copying needs the old and new KV buffers to coexist; the final boundary can temporarily approach two allocations.
Foreground compilation. Even though only the 1K graph is needed at token one, eager copy-and-grow waits for all four shapes before starting useful NPU work.

A second bottleneck was inside the compiler

Tracing the OpenVINO NPU compiler exposed 33.33 million repeated dependency-vector sorts in the scheduling path; 94.2% became cache hits under memoization. The patch reduced full compile time by 3.34–3.68% in 12/12 production A/B trials.

That optimization makes each specialization cheaper. The runtime change below attacks the orthogonal question: where should the remaining compile time occur?

Improvement 2 · JIT scheduling

Compile the next graph while the NPU decodes

One-ahead JIT changes only the schedule. Compile 1K before decode, then launch 2K on the CPU while 1K runs on the NPU. After switching, compile 4K while 2K runs, then repeat for 8K. The tokens remaining before each boundary become a natural latency-hiding window.

The switch is stall-free when the next artifact beats its boundary:T_compile(next) < T_decode(tokens remaining)

01Boot the 1K artifactPay only for the first usable specialization.

02Decode on the NPUThe active artifact keeps producing tokens.

03Compile 2K on the CPUPrepare exactly one successor in the background.

04Copy state and switchConsume the ready artifact at 1K; repeat for 4K and 8K.

One-ahead is the balance point. Compiling every future size in parallel recreates the up-front capacity problem. Waiting until the boundary creates a stall. Staying one step ahead bounds speculative work while giving compilation the longest useful runway.

Hand-drawn one-ahead JIT timeline with LLM inference and background compilation — **Sketch 2 · Overlap the timelines.** Red is useful LLM inference; blue is the next compiled shape. The growth rate of compilation has to fit inside the decode time available before each boundary. Drawing by Vikranth Srivatsa.

Replicated JIT result on NUC16

We tested Improvement 2 on NUC16 with Intel NPU 5 and OpenVINO 2026.3. The workload used Qwen2.5-0.5B FP16, a 128-token prompt, and 8,064 teacher-forced decoded tokens across the 1K → 2K → 4K → 8K ladder.

I1 control = precompile all four copy-and-grow variants before decode
I2 treatment = compile 1K first, then compile 2K / 4K / 8K one step ahead
Design = N=5 paired, Williams-order counterbalanced runs

I1 and I2 use the same graphs and the same copy-and-grow KV path. Only compile scheduling changes, isolating how much foreground latency can be hidden behind decoding.

Endpoint	I1 eager	I2 one-ahead	Paired effect
Foreground compile wall	84.29 s	65.33 s	−18.96 s · 22.49%
Protocol critical path	232.43 s	212.98 s	−19.46 s · 8.37%
Decode inference	143.32 s	142.94 s	−0.38 s · 0.26%
Boundary wait	—	0.00 s	15 / 15 ready

What moved off the path: I2 performed 20.65 seconds of successor compilation during useful decoding. All 15 successor deadlines were met with zero wait, and all 20 boundary comparisons preserved the eager control’s top-1 output.

Paired-t 95% CI: compile savings 17.95–19.96 s; critical-path savings 18.71–20.21 s; decode change −1.05–0.29 s. With five pairs, the minimum exact two-sided sign-flip p-value is 0.0625, so we emphasize effect size, interval, replication, and boundary evidence—not a p<.05 claim.

Correctness is a transition property

A model that survives startup can still fail when live state crosses a compiled-shape boundary. Device checks confirmed every successor stayed on the NPU. Boundary logits matched the eager control in all 20 comparisons, with minimum cosine similarity reported as 1.0.

Do not combine the two memory claims. Improvement 1’s initial 7B experiment reduced average reserved KV relative to direct 8K. In the later I2-vs-I1 JIT campaign, scheduling alone did not improve memory: time-weighted NPU allocation was 8.2% higher and peak process RSS about 5.6% higher for I2. The replicated result is a latency win, not yet a memory win.

Next bottleneck · Remove the copy

Virtual remapping and broader evaluation

The next systems step is to decouple logical KV addresses from physical storage. A larger specialization could inherit existing pages instead of recopying all prior state, reducing the transient peak at each boundary.

Hand-drawn future design for virtual allocation and fast switching between context capacities — **Sketch 3 · Beyond copying.** A virtual allocation table could turn context growth into a mapping operation and reduce switch cost independently of batch size. This is a design direction, not a measured result in the campaign above. Drawing by Vikranth Srivatsa.

What remains to validate

Batching: the current end-to-end JIT campaign uses batch size 1.
Compile decomposition: separately profile and schedule prefill and decode compilation.
Generality: extend the replicated JIT study across model sizes, context ladders, and precisions.
Memory lifetime: replace full KV copies with page-aware remapping and measure both average and peak memory.

The idea in one sentence

Start with the smallest useful NPU program, grow its live KV only when needed, and treat each future specialization as a CPU deadline that can be hidden behind the current NPU decode window.

Compile ahead,
not up front.