wuklab-NUC16RAPL · powercapOpenVINO 2026.0

Operator-level inference energy on the NUC 16

A controlled microbenchmark that measures the energy of each transformer operator in isolation — swept over sequence length, precision, device (CPU / iGPU / NPU), and core type — to find which operators waste energy and where they should run.

Where the OSWorld study measured whole-task energy, this one turns the knobs one at a time. Every synthetic op uses Holo3-35B-A3B's real shapes (hidden 2048, GQA 16:2, head_dim 256, 256-expert MoE, expert-FFN 512); it is an fp16/int8 stand-in for the deployed Q4_K, run entirely through OpenVINO so the three devices are directly comparable. Energy is measured with Intel RAPL via a reps-slope amortization that cancels fixed overhead.

247
clean measurements
7 fallback-flagged · 0 errors
13.3
pJ/FLOP · iGPU fp16
vs 92.8 on CPU — 7.0× worse (compute ops)
1.8× less energy
CPU int8 vs fp16
AVX-VNNI; iGPU int8 is 1.6× more energy
42×
decode vs prefill energy
per FLOP, CPU gemm — memory-bound tax

1 · The energy roofline: memory-bound ops are the hogs

Plotting every operator by arithmetic intensity vs energy-per-FLOP reproduces a roofline in the energy domain. Compute-bound prefill GEMMs sit near the floor; memory-bound decode (GEMV, M=1) and elementwise norms cost 100–1000× more energy per FLOP because the cores burn power stalled on DRAM, not computing.

Energy roofline — arithmetic intensity vs pJ/FLOP, per device and regime. Filled = prefill, hollow = decode.
Energy roofline — arithmetic intensity vs pJ/FLOP, per device and regime. Filled = prefill, hollow = decode.
Decode is 42× the prefill energy per FLOP on the CPU. A single-token GEMV re-streams the weights with almost no arithmetic to amortize the movement — the operator-level face of the "data movement bills to stalled cores" finding.

2 · Operator → device affinity

At fp16, the iGPU is the most energy-efficient device for the compute-bound projections and FFNs, the CPU the worst. But memory-bound norms don't just narrow the gap — they invert it: on RMSNorm the iGPU (~2080 pJ/FLOP) is actually worse than the CPU (~1430), so memory-bound work has no home on the accelerators. This is the per-operator placement map.

Energy per FLOP by operator family and device (fp16, prefill).
Energy per FLOP by operator family and device (fp16, prefill).

3 · Precision: int8 is a CPU win and an iGPU loss

Quantizing to int8 (NNCF FakeQuantize, verified to actually execute an int8 kernel) cuts CPU energy sharply via AVX-VNNI — but on the Xe3 iGPU it increases energy: the fp16 matrix engine beats the int8 DP4a path for these shapes.

int8 vs fp16 energy per FLOP, per device (compute ops, prefill). Green = win, red = loss.
int8 vs fp16 energy per FLOP, per device (compute ops, prefill). Green = win, red = loss.
devicefp16int8int8/fp16
CPU91.450.00.55×
GPU11.318.31.62×
NPU18.214.80.82×
CPU: int8 ≈ 1.8× less energy (AVX-VNNI).
iGPU: int8 costs 1.6× more energy — a genuine, surprising loss; the Xe3 fp16 path with matrix cores outruns its int8 DP4a path. Report with the caveat that this may narrow as the int8 GPU kernels mature.

The aggregate hides a sign flip. On the CPU the int8 win is matmul-specific: gemm and FFN fall to ≈0.31× (a ~3.3× win), but attention int8 slightly regresses (≈1.09×) — its softmax has no int8 kernel to accelerate. On the iGPU the loss holds for matmul and attention, yet conv int8 wins (≈0.73×). The per-device ratios above use a matched gemm/ffn/attention/conv basket (moe_routing's int8 falls back — router = softmax + top-k, no int8 kernel); the per-family accelerator points each rest on a single measurement, so treat the iGPU/NPU precision detail as indicative rather than definitive.

4 · Heterogeneous cores: E-cores beat P-cores on energy

Pinning the same op to Cougar Cove P-cores vs Darkmont E-cores (via taskset) shows the E-cores are more energy-efficient per FLOP — lower voltage/frequency wins on energy even at equal or better aggregate throughput. Actionable: memory-bound work belongs on E-cores.

Energy per FLOP by operator on P-cores vs E-cores vs all-16 (fp16, prefill).
Energy per FLOP by operator on P-cores vs E-cores vs all-16 (fp16, prefill).
Fewer cores, less energy: all-16 is worse than the 8 E-cores alone (≈379.0 vs ≈208.4 pJ/FLOP). Beyond a point, extra cores add static power and contention faster than useful throughput — the energy-minimal placement for these ops is the E-cluster, not the whole chip.
E/P energy ratio ≈ 0.72×. LP-E cores could not be isolated: OpenVINO 2026.0's CPU plugin SIGFPEs when confined to an LP-E-only cpuset — a real limitation of heterogeneous-core support, noted rather than papered over.

5 · Decode energy grows with KV-cache depth

Sweeping the already-filled KV length at decode isolates the KV-streaming cost: attention energy per decode step rises with context depth as more cache is streamed from DRAM each token.

Attention energy per decode step vs KV-cache depth, per device.
Attention energy per decode step vs KV-cache depth, per device.

6 · The energy-optimal thread count

More CPU threads is not always lower energy: the fixed uncore/platform tax is amortized as threads increase, but past a point contention and frequency-throttling raise energy again.

Energy per FLOP vs CPU thread count, per operator (fp16).
Energy per FLOP vs CPU thread count, per operator (fp16).

7 · Energy vs sequence length

Per-operator energy shifts with prompt length: on the iGPU, small prefills underutilize the device (high energy/FLOP) and efficiency improves as sequences lengthen; attention's O(n²) work bends differently from the linear projections.

Per-operator energy vs sequence length, CPU and iGPU (fp16 prefill).
Per-operator energy vs sequence length, CPU and iGPU (fp16 prefill).

8 · Where the watts go

The RAPL decomposition shows the mechanism: on the CPU the energy is in core (compute + stall); on the iGPU it moves to uncore; the dram device draws little in every case — the movement cost is billed to the engines that wait on it, not the DIMMs.

Busy power by RAPL domain for an FFN, per device.
Busy power by RAPL domain for an FFN, per device.

Method & caveats

Instrument. Intel RAPL powercap sysfs (psys/package/core/uncore/dram) — the only non-root energy source on this box. Each cell is measured at reps {n, 2n, 3n} and a least-squares slope gives the marginal per-op energy, cancelling compile/allocation/idle overhead; idle power is bracketed and subtracted. Steady state is checked from a 20 Hz power trace.

int8 verification. Every int8 accelerator cell is checked to have actually executed an int8 kernel (exec_type on CPU/iGPU; npu_util on the NPU) — silent fp16/CPU fallbacks are flagged and excluded from the clean aggregates (7 flagged).

Caveats. (1) This is an fp16/int8 OpenVINO proxy for the deployed Q4_K — absolute numbers are a stand-in, the cross-device/precision comparisons are the result. (2) RAPL dram on this client SoC is partial/modeled — a lower bound. (3) The CPU governor is powersave (no root to pin it); warmup + a steady-state gate compensate. (4) LP-E cores excluded (OV SIGFPE). (5) iGPU int8 loss may reflect kernel immaturity.

idle: psys 9.9 W · core 0.7 W · dram 0.6 W  |  254 cells · 26.1 min · profile full

wuklab-NUC16 · Intel Core Ultra 7 356H (Panther Lake) · Xe3 iGPU · NPU 5 · generated from operator_energy_findings.json. Companion to the OSWorld inference-energy profile.