NPUW Pyramid JIT

1 · Why the NPU, when the same chip has an iGPU

The honest answer is that this is a Pareto choice, not a dominance one. Measured on this machine, Qwen2.5-7B INT8 decode at a right-sized 1024 window:

The NPU costs 38% less energy and takes 1.8× longer. Prefill placement between the two is neutral. So the NPU is the right target whenever energy or thermal headroom is the binding constraint rather than latency — sustained background work, battery, or a device already using the iGPU for something else.

That last case is the non-energy argument, and it is the one shipping systems act on. The iGPU is also the display, compositor, video and game engine; the NPU is otherwise idle. Agent.xpu measures pure-iGPU prefill pinned at 100% utilization and reports a 32.5–37.1% reduction in iGPU utilization by moving work to the NPU. Microsoft gates Windows Studio Effects on an NPU with no iGPU fallback, on the stated rationale of “battery-friendly AI effects that reduce the burden on the device CPU and GPU.” That rationale is stated, not quantified — no vendor page in the surveyed set gives watts, joules or frame rates.

2 · Why the window is fixed at compile time

This is the deployment model of every edge NPU stack, not an OpenVINO quirk.

3 · Why no single window is the right one

Across workloads, the right answer differs by task class. In Microsoft’s Azure production trace the conversation service has a median context of 1,020 tokens with 86% of requests under 2K, while the coding service sits at 1,469 with only 62.5% under 2K. Same service family, different correct window.

Within a single on-device agent session it is worse, because the requirement is not drawn once — it grows. From our own OSWorld run on this machine (Holo3, 93 episodes, history eviction disabled so the true demand is visible):

Provision for the observed maximum and the median episode uses 11.6% of it. Growth within one episode is 5.8× at the median and 49× at the worst.

But the rate is predictable even though the endpoint is not. Each step appends a median of 2,227 tokens — one screenshot plus one action — and not a single step in the run shrank the context. So the crossings are near-deterministic: 8K is first exceeded at step 4, 16K at step 7–8, 32K at step 14–15, while session length itself runs from 8 steps at the median to 100 at the tail. That gap is the whole opportunity: you cannot know where a session ends, but you always know when it is about to outgrow the artifact it is running on — several steps ahead of the fact.

4 · What an oversized window costs

The decode-rate row is the one that surprises: on a static-shape NPU, decode cost tracks the compiled window rather than the tokens actually generated, so an oversized artifact is slower per token for the entire session. Our own arms show the same sign — the pyramid arms decode at 0.752–0.754 tok/s in the first 1K band against 0.740 for the model compiled at the full 16,384 context.

5 · Why existing mechanisms do not cover this

No published mechanism copies live KV state from one compiled static artifact into a differently-compiled larger one on the same device.

Full 1K→8K comparison · N=5 complete

Affinity answer and targeted validation

Yes, sub-stage affinity exists, but it is a property of a real graph in a specific stage, precision and compile path—not a permanent label attached to an operator name. The fresh Qwen2.5-3B validation below runs a full decoder block and two ablated versions at sequence 512, then prices attention and FFN by differencing those realistic graphs. GPU and NPU, FP16 and INT8, prefill and decode are all swept; the NPU's plain and NPUW paths are both tried, and the fastest valid one is retained.

Ratios are NPU ÷ iGPU; below 1 favors NPU. Qwen2.5-3B, sequence 512, three reps-slope windows per graph, idle-subtracted PSys, OpenVINO 2026.0. The plain-NPU no_attention prefill ablation fell onto the known isolated Parameter→MatMul slow path (0.888 s FP16, 0.453 s INT8), which would make the marginal negative. It is rejected; NPUW supplies the valid prefill-attention difference. That path sensitivity is part of the affinity result, not noise.

Method basis: OpenVINO HETERO affinity and query_model; NPU static-shape LLM execution; precision control; performance-counter interpretation; and Agent.xpu.

0 · The energy split is validated, not assumed

Panther Lake gives the NPU no RAPL domain of its own, so NPU energy has to be recovered as a residual. Before anything rests on that, each half of the attribution is tested by driving one engine at a time and watching every domain together with two independent hardware counters.

Convention used throughout: iGPU = uncore, NPU = package-0 − core − uncore − dram, psys = whole platform, all net of a measured idle baseline — so every number is the marginal cost of the work, not the machine's standing draw.

1 · An isolated kernel does not measure this NPU

The natural way to build an operator roofline is to compile one operator on its own and time it. Done that way, this NPU's FP16 compute roof comes out at 0.074 TFLOP/s — dead constant across seven GEMM shapes, which looks convincingly like a real ceiling. It is not. The same MatMuls, inside a decoder block on the same device and the same compile path, run at up to 9.00 TFLOP/s.

The block is the number that is right: its output matches a CPU run of the identical graph to a correlation of 0.999999 (max absolute difference 0.0059, which is FP16 rounding), so the arithmetic demonstrably happened. An ablation ladder locates the trigger — under NPUW, an FP16 MatMul that reads a graph Parameter directly runs at 0.074 TFLOP/s, while inserting any operation between the Parameter and the MatMul yields 1.22 TFLOP/s. That also explains why INT8 looked healthier than FP16 throughout: quantisation inserts FakeQuantize nodes, which are themselves intervening operations.

2 · Measured rooflines

3 · The efficiency claim, in the units the paper states it in

Agent.xpu's central claim is that “static-shaped NPU kernels achieve higher energy efficiency (TFLOPS/W) while delivering throughput comparable to iGPUs”. It never plots that axis. Here it is, together with the same comparison as a latency/energy trade.

4 · NPU vs iGPU at three granularities

An operator-level placement policy is only sound if operator-level numbers predict what a whole model does. Measuring the same comparison at three granularities tests exactly that — and the isolated-kernel row is kept in the table as the control that shows what happens when the measurement is built the usual way.

5 · Whole models, every size and precision

6 · Precision does not mean the same thing on the two engines

Agent.xpu profiles at W8A16 and treats 8-bit weights as the NPU's natural format, which is what an INT8 MAC array should want. On this stack it is the wrong choice: the NPU's best whole-model configuration is FP16, where it matches the iGPU's prefill throughput and undercuts its decode energy, while the INT8 decode path runs 11× slower than FP16 on the same weights. The microbench shows the same shape — an INT8 GEMV compiled through NPUW streams at 1.69 GB/s against 65.8 GB/s for the plain path — so the cost looks like per-token weight handling in the quantised decode path rather than anything about the arithmetic.

7 · Verdicts

Setup

Core Ultra 7 356H (Panther Lake): Xe3 iGPU, NPU 5, 96 GB DDR5-5600, Ubuntu 24.04, kernel 6.17.0, OpenVINO 2026.0. Energy from the powercap RAPL interface (package/core/uncore/dram/psys) with a measured idle baseline subtracted. Operator and block cells use a reps-slope fit so per-window fixed cost cancels; whole-model cells use direct windowing because the calls are long enough not to need it. Five small dense decoders — Qwen2.5 0.5B/1.5B/3B and Llama-3.2 1B/3B — each exported at FP16 and INT8 with on-disk element types verified, since optimum-intel will silently INT8-quantise an FP16 request. On the NPU both compile paths (plain and NPUW) are swept wherever the NPU appears, because neither dominates the other. Agent.xpu's own testbed is a Core Ultra 5 125H (Meteor Lake), so absolute ceilings differ; the operator definitions, byte accounting and W8A16 convention are the paper's.

1 · As released, it generates garbage

Fluent-looking text with no relation to the prompt, on NPU, iGPU and hetero alike. Bisected by a ladder that ruled out one layer at a time, so no step rested on a guess: the repo's own torch model matches Hugging Face greedy decode exactly; its W8 RTN quantisation gives coherent output at every group size; and the serialised .model file round-trips perfectly — all 1,794,773,700 bytes consumed, reproducing the in-memory quantised model. Everything was clean until the OpenVINO graphs.

An attractive early hypothesis — that the exporter ignores Llama-3.2's rope_scaling (rope_type: llama3, factor: 32) — was refuted by the first rung: that scaling is a no-op at short context and the unmodified model already matched the reference. It is a real latent bug for long contexts, but it was not this one.

The three defects

1. Prefill buffer index mismatch. The prefill buffer is chosen from the job's priority (reactive → 1), but swap-in was called with the default index 0. The prompt was embedded into buffer 0 while prefill ran out of buffer 1 — zero-allocated and never written. Every layer saw an all-zero input, RMSNorm computed rsqrt(0), and the logits came out NaN.
2. Two graph inputs declared in the wrong order. The post-attention graph declares {weighted_V, prompt_embedding} while all seven binding sites pass {prompt_embedding, weighted_V}, silently exchanging the residual with the attention output: no NaN, plausible magnitudes, wrong tokens.
3. The KV cache had no layer dimension. One {ctx_len, kv_dim} buffer that every layer read and wrote, so each layer saw the previous layer's keys. The first generated token (produced by prefill) was correct and every later one was not.

With these fixed, Llama-3.2-1B W8A16 on the NPU reproduces the Hugging Face reference generation token for token, and the prefill logits match a torch reference of the same file (argmax 2127, max 24.04 vs 23.93) within FP16 tolerance.

2 · The benchmark matrix, with energy

Llama-3.2-3B W8A16, the paper's own model, replaying agent workloads through the exp-* harness, every arm inside a RAPL window.

3 · Do the paper's results survive?

Caveats

Built against OpenVINO 2025.2, the version the repo targets; on 2026.0 the GPU plugin dies in clFinish with CL_OUT_OF_RESOURCES. Independently, iGPU prefill is numerically wrong on this Xe3 iGPU — the identical graphs are correct on the NPU and CPU plugins, in both shape modes and on both OpenVINO versions — so arms whose prefill runs on the iGPU execute the right amount of arithmetic (their timing and energy are comparable) but their generated text is wrong. The paper's own traces are not distributed; workloads are the synthetic profile it describes plus a real 1139-call on-device agent trace recorded on this machine, with prompts capped at the engine's 4096-token context.

Qwen2.5-3B path audit and timing semantics

The model artifact is internally consistent; the reversal came from the benchmark route. The old INT8 cell was named gpu_gpu but recorded compile_target=NPU, NPUW_DEVICES=GPU and execution_devices=NPU. The corrected cell compiles AUTO:GPU and records GPU.0.

1K–8K window sweep · matched FP16/INT8 scale

Every model is now shown as an INT8/FP16 pair with the same y-axis limits inside that pair, so the precision difference is directly visible. Each line covers the measured 1K, 2K, 4K and 8K context buckets; solid routes are whole-device measurements and dashed routes are the zero-boundary-overhead phase oracles.

All-window metric tables

Exact median values for every model, route and context bucket. TTFT is the post-compile prefill latency; decode is steady output throughput; energy is idle-subtracted whole-system inference energy.

8K cross-model latency, throughput and energy digest

These grouped bars compare all six models directly at the largest tested window. Whole-device routes are solid; overhead-free split oracles are hatched. Post-compile TTFT follows the prefill device, decode throughput follows the decode device, and total energy adds the independently measured phase components.

Additional four-case diagnosticsroute choice · first decode · TPOT · phase energy

Runtime qualification

The experiment uses OpenVINO NPUW with NPUW_DEVICES=NPU,GPU and tests automatic assignment plus 0:GPU,last:NPU, last:GPU, 0:NPU,last:GPU and 0:GPU maps. Changing the submodel boundary did not repair decode. Phase isolation then located the fault: combined prefill followed by NPU-only decode is token-correct, but a 10 ms GPU-frequency sampler records 0 MHz for every sample, so that arm silently executes on NPU only. Enabling combined execution for decode activates the iGPU and corrupts the state/logit path.

Below-stage affinity

The fine-grained study explains why a stage-level route is not enough. At sequence 512, in-situ decode attention and FFN usually favor NPU energy and often latency, whereas FP16 prefill attention strongly favors iGPU. But whole-model INT8 NPU decode reverses those kernel wins because composition, precision handling, state and runtime overhead dominate. The result is an affinity hierarchy, not a fixed device label: operator inside graph → decoder block → stage → whole stateful model.

Open the Fine-grained affinity tab for the experiment, fresh Qwen2.5-3B validation, rooflines and all operator/block/model tables.

Placement validation appendix

The compact qualification result stays above. Open a precision below for every diagnostic matrix and rejected-path measurement.

FP16 placement diagnosticsGPU→GPU · GPU→NPU · NPU+iGPU

FP16 expansion · GPU→GPU and GPU→NPU

FP16 repeats the six-model, four-window sweep with the same 16-token guard band. Stock AUTO again resolves to GPU.0. Explicit iGPU→iGPU matches the baseline at prefill, first decode and final decode in all 24 cells and all three repeats. iGPU→NPU matches at 1K for all six models, but every 2K–8K group diverges: 18 of 18 long-context groups fail. The failure is stateful and repeat-dependent, not a small FP16 rounding difference.

FP16 route	measured cells	valid cells	qualification boundary
OpenVINO AUTO → actual iGPU	24	24	valid 1K–8K
iGPU → iGPU	24	24	valid 1K–8K
iGPU → NPU	24	6	valid at 1K; wrong tokens at 2K–8K
NPU + iGPU subgraphs	6	0	wrong decode at the 1K capability gate

At the largest requested point, Qwen2.5-7B FP16 at 8K, iGPU→iGPU is effectively identical to the stock path: 13.53 s TTFT, 196.93 ms/token, 16.53 s end to end and 616.6 J. The measured iGPU→NPU diagnostic costs 63.90 s TTFT, 67.45 s end to end and 1,959.6 J—and is token-invalid. The 7B NPU+iGPU 1K gate is worse: 475.1 s compile, 118.05 s inference and 3,060.4 J, ending on token 0 instead of 42.

FP16 throughput and decode latency

FP16 end-to-end time and energy

INT8 placement diagnosticsqualified routes · failures · energy

INT8 qualified placement matrix and 8K overview

INT8 throughput and latency across 1K–8K

INT8 end-to-end time and energy