NPU Compilation Optimization

What dominates Intel NPU-5 compile time, why, and what to fix — 120 cold configs over 5 architecture families (Gemma-2, TinyLlama, Llama-3.2, Phi-3, Qwen2), sizes 0.5–7.6 B, prompt window 128 → 32768, three weight precisions (int4 / int8 / fp16), 10 cold repeats each (OpenVINO 2026.0).

The dominant pass is not fixed by architecture — it's set by how big the compile is. Every family walks the same path: tiling-search → memory-scheduling → serialization. And each pass's cost tracks the IR/schedule size it processes — which is the handle to fix it.

31%

peak FeasibleAllocation share (7B, mid-window)

architecture families — same pattern

120

cold-compile configurations

≤0.99×

int4 compile time vs int8 — cheapest at every size

regimes, gated by compile size

1 · The regime map — which stage dominates each compile

fig_regime_map — Every (model, window) cell, positioned by size × window, colored by its dominant stage. The regimes tile the plane: tiling-search bottom-left (cheap), memory-scheduling mid, serialization at the long-window top — independent of family.

2 · The handoff is universal across architectures

fig_universal — Dominant-stage share vs cold compile time, all families overlaid. The three trend lines cross in the same order regardless of architecture — the pattern is a property of the compiler, not the model.

3 · Why those passes are costly — cost = f(IR/schedule size)

fig_why_costly — The instrumentation the compiler will emit (`enable-schedule-trace` → #tasks, `write-strategy-to-json` → #tiles) shows each dominant pass's time scales with the amount of IR it processes. FeasibleAllocation time ∝ (#scheduled tasks)^0.69

4 · Same fingerprint, different architectures

fig_family_fingerprint — Stage breakdown at a fixed window across families — the shape barely moves from Llama to Qwen to Gemma to Mistral. Compile cost is structural.

5 · Precision: a two-way tug-of-war, not "fewer bits = faster"

Compile cost is a tug-of-war between two effects. The dequantization / weight-table machinery that int8 and int4 bolt on penalises the quantized formats — so for small-to-mid models fp16 compiles fastest (≈0.58× int8) despite its larger weights. But fp16 also carries 2× the weight bytes, and by 7 B that weight-DMA volume takes over: fp16 flips to ≈1.88× int8 (up to 2.4× at mid windows). int4 wins both ways — it has the dequant machinery but half the bytes — staying ≤0.99× int8 at every size. The mechanism shows in the pipeline: the quantized formats (int4, int8) land their heaviest work in the serialization back-end (the weight tables they serialize), while fp16 stays tiling-search-dominated.

fig_precision — Left — cold compile time vs model size at W=2048: fp16 (red) sits lowest through the mid-sizes but spikes above int8 (gold) at 7 B. Right — compile time ÷ int8, median over windows: **int4 (green) stays ≤ 1 at every size**, while **fp16 (red) crosses from below 1 to above 1** around 7 B — the dequant-machinery vs weight-volume crossover.

fig_precision_scaling — Memory-scheduling (FeasibleAllocation) share vs window, one line per precision. The three curves keep the same shape — precision shifts the *level* of back-end work, not the size×window regime structure.

6 · Convolutional graphs — same structure, different passes

The pattern generalises across modality, with a twist. Across 31 CNN configs (7 families — ResNet-18→152 swept 128→640 px, plus EfficientNet, MobileNetV3, DenseNet, RegNet, Inception, ConvNeXt), FeasibleAllocation never dominates a single compile (0/31). Conv graphs are tiling + vertical-fusion bound: MultiClusterStrategy (tiling) leads at low resolution and hands off to MergeVfSubgraphs (vertical fusion) as resolution grows. Same workload-gated shape as the LLMs — attention graphs escalate into scheduling/serialization; conv graphs escalate into tiling/fusion.

fig_cnn_regime — Every (size, resolution) CNN compile, coloured by its heaviest pass. A clean horizontal handoff: tiling (blue) owns 128–448 px; vertical-fusion (purple) takes over at 640 px — resolution-gated, largely size-independent.

fig_cnn_resolution — ResNet depth spine. Left — compile time is gated by input resolution (the plain-static workload axis). Right — the handoff: tiling rises then yields to vertical-fusion at 640 px, while FeasibleAllocation stays flat at ~5% — CNNs never enter the memory-scheduling regime that dominates 7B LLMs.

fig_cnn_fingerprint — Unlike the LLMs (whose fingerprint barely moved across architectures), the CNN op-mix **does** move the breakdown — DenseNet's dense concats push tiling past 80%. fp16 ≈ 0.86× int8 for CNNs — weights are small, so the precision effect is muted.

7 · The fixable pattern → what to optimize

Because cost is driven by IR/schedule size in a known regime, the levers are direct:

regime (when)	dominant pass	root cause	fix
every compile	—	recompiling from scratch	Warm blob cache — a recompile is 20–30× a blob-load. Biggest, universal win.
small / short (≤3B)	MultiClusterStrategy	serial tiling search (user/wall≈1.0)	Cache / reuse `strategy_out.json` across shapes; lower search effort.
7B, mid window	FeasibleAllocation	#tasks explodes → greedy simulate-and-spill re-run 2–3×	Fewer/larger tiles (cut #tasks); bound the window; ship precompiled ELF.
extreme window (≥16K)	ELF serialization	artifact size grows linearly with W	Precompiled-blob shipping; the materialization is mechanical, don't repeat it.

Method: Intel Core Ultra 7 356H, NPU 5 (intel_vpu), OpenVINO 2026.0, driver-compiler 1.33. Weights swept at int4 (symmetric, group-128), int8 and fp16 — each export's on-disk element types verified (fp16 defeats the optimum silent-int8 trap; int4 uses --sym to dodge a group-quant location-verifier compiler bug that fails asymmetric int4). Fully-cold compiles (NPU_BYPASS_UMD_CACHING, isolated subprocess per repeat), NPUW_LLM window sweep. Per-pass time = the VPUX compiler's MLIR PassTiming (wall, inclusive of analyses, pipeline groups excluded); mean over 10 repeats (4 at W≥8192). Why-costly metrics from the release compiler's own instrumentation via NPU_COMPILATION_MODE_PARAMS (enable-schedule-trace, write-strategy-to-json) + built-in dump-statistics-of-ie-ops. Sparse-MoE (Granite) and hybrid-SSM (Qwen3.5) do not compile on NPU 5 (dynamic shapes). CNNs exported via optimum --library timm, reshaped to static NCHW; input resolution is the plain-static workload axis. vgg16 excluded — its 102M-param fc1 fails SplitNCEOpsOntoWorkloads on NPU 5. Ungated model mirrors; no HF token.