wuklab-NUC16Intel Xe3 iGPU + NPU 5OpenVINO int8

Does NPU/CPU-assisted prefill make Holo1.5-7B faster and cheaper on the NUC 16?

A one-shot microbenchmark + OSWorld pilot for Holo1.5-7B (Qwen2.5-VL-7B) on a fanless Intel Core Ultra mini-PC, measuring where each pipeline stage should run — and whether the memory system, not compute, is the real bottleneck.

One representative OSWorld step: a 262144px screenshot → 1248 patches / 312 vision tokens, prompt length S=347 tokens. Every number below is measured on the real int8 OpenVINO IR with RAPL energy (reps-slope, net-of-idle); config totals compose the measured stages plus a unified-memory cross-device transfer tax (37.837 pJ/byte, measured).

Key findings

The NPU does batch prefill after all. An earlier version of this study reported it couldn't — that was a compile-config timeout artifact (PERF_COUNT/f16 hints balloon the T=S compile past 20 min). The bare driver compiles it in ~5 min and runs fine, confirmed on Qwen2.5-1.5B / 7B controls and the Holo graph (785 ms, ~2.5 pJ/FLOP — on par with the iGPU).
So prefill placement is nearly irrelevant. H0 (prefill-NPU / decode-iGPU) ≈ G0.1 whole-iGPU — prefill is only a few % of a step; the 64 decode tokens dominate. Offloading prefill was the wrong lever.
The real choice is the decode device. Running the LLM decode on the NPU (G0.2) is 2.1× slower than the iGPU — the NPU streams weights at ~half the iGPU's DRAM bandwidth. In exchange it draws far less power: the right-sized control (Qwen2.5-7B, same LLM) shows NPU decode at ~40% lower energy per token. It is a latency↔energy Pareto, not a winner.
Memory is the decode bottleneck. Prefill is compute-bound (2.7 pJ/FLOP on the iGPU); decode is memory-bound (221.6 pJ/FLOP, ~82× worse) at 57.6 GB/s = 64% of DDR5-5600 peak — no accelerator fixes this; the lever is fewer weight bytes / token.
Success is not the discriminator. On the one-per-domain OSWorld subset the runnable placements score G0.3 0/10, G0.1 0/10 — so NPU placement does not reduce success (there is none to reduce). The deployment ceiling here is grounding + agent-loop robustness and the decode memory wall, not where prefill runs.

The five placements

Holo1.5's LLM is a plain transformer (KV cache only, 3-section M-RoPE — no Gated-DeltaNet), so it runs coherently on the NPU directly and prefill/decode can be split across devices with a zero-copy KV handover.

configuration	latency s/step	energy J/step	prefill	transfer ms	transfer J	missing
G0.1 whole-iGPU	8.535	223.72	batched	0.0	0.0	—
G0.2 vision-iGPU + LLM-NPU	18.272	223.89	batched	0.11	0.0004	—
G0.3 whole-NPU	18.395	223.28	batched	0.0	0.0	—
H0 prefill-NPU / decode+vision-iGPU	8.935	226.94	batched	0.56	0.0019	—
H1 op-split prefill	8.538	223.73	op-split	3.1	0.0105	—

End-to-end cost of one OSWorld step under each placement (measured stages + unified-memory transfer tax).

On latency the split is clean: G0.1 / H0 / H1 (iGPU decode) ≈ 8.5–9 s, the NPU-decode configs (G0.2 / G0.3) ≈ 18 s. The energy column looks nearly flat here for a methodological reason worth stating: the NPU needs a static KV buffer, and this run sizes it to the deployed max context (L=3072) while the iGPU decodes at the true context (~347) — so the NPU decode pays for 9× more attention than it uses, which cancels its per-token energy edge. The right-sized control below recovers the NPU's ~40% decode-energy advantage; the practical lesson is to size the NPU KV buffer to the real context.

Where each stage belongs

Per-stage latency and energy by device on the real int8 IR.

Vision and prefill are compute-heavy (the accelerators help); decode is a per-token full-weight read (the accelerator barely matters — the memory system does).

The memory wall

Decode achieved DRAM bandwidth vs KV context — decode saturates the memory system.

Roofline: prefill is compute-bound, decode memory-bound.

Each decode token streams the whole ~7 GB int8 weight set from DRAM, so decode runs near the DRAM roofline regardless of device. Prefill amortizes that same weight read across all S context tokens in one batched pass — which is why prefill is compute-bound (~2 pJ/FLOP on the iGPU) and decode is memory-bound (~100–170 pJ/FLOP). The intuition motivating this study was that moving the compute-bound prefill onto the NPU's matrix engine would help. It doesn't — not because the NPU can't do it (it can), but because prefill is a small slice of a step, so where it runs barely moves the total.

The decode device is the real choice — a latency↔energy Pareto. The NPU streams weights at only ~27 GB/s (≈30% of DDR5 peak) versus the iGPU's ~58 GB/s, so NPU decode is ~2× slower — but it draws far less power, so per token it uses less energy than the iGPU. There is no dominant decode device: the iGPU is the latency choice, the NPU is the energy choice. Prefill, by contrast, is compute-bound and both accelerators handle it about equally (the NPU compiles the batched prefill in ~5 min and runs it at ~2.6 pJ/FLOP, on par with the iGPU) — and since prefill is a small slice of a step, moving it around barely changes the total.

How to actually optimize the memory side. Because decode is bounded by weight bytes moved per token, the levers are all on the bytes, not device placement: coarser weight quantization (INT4/mixed), weight streaming/residency so the read is not re-paid every token, or amortizing one weight pass over more tokens (speculative / multi-token decode). Relocating decode only trades latency for energy along the Pareto — it does not break the DRAM roofline.

H1 — fine-grained operator split

Prefill compute composition by operator type; H1 assigns QKV/O + FFN to the NPU.

H1 would split each layer — QKV/O projections + FFN on the NPU, attention/norm/RoPE + decode on the iGPU. But the motivation for a fine-grained split evaporates once you know the whole prefill already compiles and runs on the NPU (~2.6 pJ/FLOP, on par with the iGPU): there is no compute-bound operator that needs rescuing onto a different engine. A per-op split would only add a per-layer NPU↔iGPU activation-shuffle tax (priced in Phase 1) for no compute win, so H1 lands at ≈ G0.1 / whole-NPU prefill. The interesting split is not within prefill but between prefill and decode — which is exactly the H0 / G0.2 latency↔energy trade above.

Control & correction — is the NPU behavior Holo-specific?

Control: plain dense LLMs (Qwen2.5-1.5B and -7B, the latter the identical LLM arch as Holo1.5). The NPU batches prefill for both; decode is the same latency↔energy trade.

To be sure the NPU-prefill finding wasn't an artifact of the Holo1.5 VL export (custom fork, 3-section M-RoPE), I exported two plain dense text LLMs with stock tooling — Qwen2.5-1.5B and Qwen2.5-7B, the latter the identical LLM architecture as Holo1.5 (28L / 3584 / 28 heads / 4 KV / FFN 18944) but text-only with standard RoPE — and profiled them the same way. Both compile and run batched prefill on the NPU (1.5B in 64 s, 7B in 284 s, NPU util 0.89), and both show the same shape: prefill NPU ≈ iGPU, decode a latency↔energy trade (NPU slower, ~40% lower energy). This is what corrected the earlier "NPU can't batch prefill" conclusion: it was the profiler's PERF_COUNT/f16 compile hints ballooning the large-T compile past its timeout — not a hardware limit. The bare driver config compiles the very same Holo graph in ~5 min.

Phase 2 — OSWorld pilot (success preservation)

One task per domain (round-robin, 8 steps max) on the runnable placements. The question was whether NPU-assisted prefill reduces success. It does not — because success is 0/10 either way on this hard subset. Inspecting trajectories, Holo1.5-7B forms coherent intents and roughly-correct grounding, but a single-image history (max_images=1) makes it repeat a click when a slightly-off target leaves the screen unchanged, and it occasionally emits out-of-range coordinates. The whole-NPU arm additionally can't reach the resolution its grounding needs (NPU vision compiler ceiling ~262144 px). Neither limit is about prefill placement — so the offload question is moot for success, and the latency/energy verdict above stands on its own.

config	solved	success	note
G0.3 whole-NPU	0/10	0.0	0 solved — NPU vision compiler caps resolution at ~262144px, so grounding lands off-screen (clicks miss). Prior full-NPU run.
G0.1 whole-iGPU	0/10	0.0	0 solved. Trajectories show coherent intent + roughly-correct grounding, but single-image history (max_images=1) causes repeated-click loops when a slightly-off click doesn't change the screen, plus occasional out-of-range coords. Success here is gated by grounding/agent-loop robustness, NOT prefill placement.

Domains covered: chrome, gimp, libreoffice_calc, libreoffice_impress, libreoffice_writer, multi_apps, os, thunderbird, vlc, vs_code.

Per-stage measurements

stage	dev	ctx	lat ms	psys J	dram J	dram frac	eff GB/s	pJ/FLOP
vision_patch	CPU		27.92	1.2085	0.0072	0.006	0.2
vision_patch	iGPU		1.79	0.0445	0.0014	0.031	3.3
vision_merger	CPU		2286.81	94.0501	0.492	0.005
vision_merger	iGPU		268.4	8.0517	0.3326	0.041	0.1
prefill	CPU		4723.48	191.468	1.5765	0.008	1.5	38.8
prefill	iGPU		386.44	13.1086	0.5899	0.045	18.4	2.7
decode	CPU	347	127.72	4.0054	0.2805	0.07	55.6	280.5
decode	iGPU	347	123.1	3.1643	0.2685	0.085	57.6	221.6
vision_patch	NPU		52.08	0.7826	0.1176	0.15	0.1
vision_merger	NPU		341.28	6.7037	0.3822	0.057	0.1
prefill	NPU		785.83	16.3308	0.9132	0.056	9.0	3.3
decode	NPU	3072	269.0	3.1165	0.3466	0.111	27.0	202.7

Memory-wall verdict (per device)

{
 "CPU": {
  "prefill_eff_bw_gbs": 1.5,
  "prefill_pJ_flop": 38.8,
  "decode_eff_bw_gbs": 55.6,
  "decode_bw_util_%": 62.0,
  "decode_dram_frac": 0.07,
  "decode_pJ_flop": 280.5
 },
 "iGPU": {
  "prefill_eff_bw_gbs": 18.4,
  "prefill_pJ_flop": 2.7,
  "decode_eff_bw_gbs": 57.6,
  "decode_bw_util_%": 64.3,
  "decode_dram_frac": 0.085,
  "decode_pJ_flop": 221.6
 },
 "NPU": {
  "prefill_eff_bw_gbs": 9.0,
  "prefill_pJ_flop": 3.3,
  "decode_eff_bw_gbs": 27.0,
  "decode_bw_util_%": 30.1,
  "decode_dram_frac": 0.111,
  "decode_pJ_flop": 202.7
 }
}

Method & reproducibility

Model Hcompany/Holo1.5-7B → OpenVINO int8 stateless IR (162/162 LLM layers int8_sym). Energy via powercap RAPL (no root), reps-slope net-of-idle marginal per-infer energy. Serial, quiet machine. Code: experiments/holo15/ in the edge-ai repo (microbench.py · analyze.py · serve.py). Run folder is write-once.