Holo3 · OSWorld inference energy profile (task → step → byte)

We ran the full OSWorld test set (all 10 desktop-app categories) through Holo3-35B-A3B on a fanless Intel Core Ultra 7 356H mini-PC and metered energy with Intel RAPL at three granularities — per task, per agent step, and per computation vs. data movement — then re-ran the workload on the iGPU and NPU. The starting hypothesis: moving data between DRAM and the processors dominates the energy.

The verdict is a "yes, but". The DRAM chips are nearly free — the dram RAPL domain stays at ~1.4% of platform energy (and that is a lower bound — see caveats). Yet data movement is the bottleneck: memory-bound decode is 4.6× more energy-costly per token than prefill, because each token streams the whole active weight set from DRAM. Its cost hides as active-core power — cores held at 27 W (~24 W above idle) starved for data — not as DRAM power. The real energy hogs are the repeated 8k-token prefill (~63%) and a fixed 27% platform tax. Offloading the compute-bound prefill of the full Holo3-35B to the iGPU cuts its energy 4.1× and frees the cores (58→1.6 W), but — on this unified-memory SoC — memory-bound decode is actually ~1.9× worse on the iGPU and belongs on the CPU.

Total metered

9.8 kWh

141 tasks · 2,136 steps · 132 h of pure inference

Per task

70 Wh

~56 min of compute · one desktop task

CPU cores

67%

of platform energy

DRAM devices

1.4%

the "movement" domain — tiny

Platform tax

27%

VRM · rails · I/O — always on

iGPU prefill

4.1×

less energy — full Holo3-35B
frees cores 58 W→1.6 W

RAPL psys/package/core/dramllama.cpp Q4_K_M OpenVINO 2026.0 iGPU+NPUno root · powercap sysfs run 2026-07-05

How the energy was measured

The only reliable energy instrument on this box (no root ⇒ turbostat, perf, intel_gpu_top are all blocked) is the powercap RAPL sysfs — which is the gold standard anyway and, crucially, exposes a separate dram domain. We read five domains: psys (whole platform), package-0 (CPU chip), core, uncore (iGPU/fabric), and dram (the DIMMs). Energy is Δenergy_uj across each work window; power traces are sampled at 10–20 Hz with a background thread that also logs iGPU frequency and NPU busy-time. Idle baselines (psys 19.5 W) are subtracted so the fixed platform draw is separated from the work.

Data sources: (A) a complete 141-task OSWorld run with per-step, per-domain energy already on disk; (M2) STREAM + variable-arithmetic-intensity microkernels in C (-O3 -march=native, OpenMP); (M3/M4) identical matmul + memory workloads on CPU/iGPU/NPU via OpenVINO, plus a real transformer (Qwen2.5-1.5B) on CPU vs. iGPU via llama.cpp Vulkan; (M5) the full Holo3-35B loaded onto the iGPU (all 21 GB, -ngl 99, after freeing ~54 GB) and metered on CPU vs. iGPU across seven OSWorld task sizes. Every offload number below is a direct measurement, not a projection.

1 · Per task — where the Joules go

Energy breakdown by hardware domain and by inference phase — Platform (psys) energy split two ways. **Left:** by hardware domain — CPU cores dominate (67%), the DRAM devices are a 1.4% sliver, and a 27% "platform tax" (VRM/rails/I/O) is spent just keeping the board powered. **Right:** by inference phase — the giant repeated prefill is ~63%, decode only ~10%, fixed overhead ~27% (prefill/fixed split is predictor-dependent; see caveats).

Across 141 completed tasks the agent spent 132 hours in the model for 135 hours of wall time — 98% of the run is inference, not the desktop environment. That is the macro-inefficiency: an average desktop task costs 70 Wh and ~56 minutes of compute. The counter-intuitive first result is that the dram domain — the literal DRAM chips your data moves to and from — draws under 1 W even at full tilt and is only 1.4% of the energy. If "data movement" meant "DRAM-device power," the hypothesis would be dead on arrival. It isn't — see §3.

2 · Per OSWorld category

Energy per agent step by OSWorld category — Energy per agent step by category. The spread (11.3–19.1 kJ/step) tracks prompt size × step count, not the app itself: multi_apps, calc and vs_code push the largest accessibility trees. The DRAM-device share is a near-constant ~1.3–1.4% *everywhere* — a fixed, tiny fraction independent of task.

category	tasks	steps	Wh	kJ/step	DRAM%
multi_apps	12	258	1370	19.1	1.3
libreoffice_calc	13	386	2011	18.8	1.3
vs_code	12	183	942	18.5	1.3
chrome	32	535	2593	17.4	1.4
vlc	12	175	702	14.4	1.4
libreoffice_writer	11	147	588	14.4	1.4
thunderbird	12	132	517	14.1	1.4
libreoffice_impress	13	150	579	13.9	1.4
os	11	81	263	11.7	1.4
gimp	13	89	278	11.3	1.4

Step energy is integrated from the consistent per-step profiles (core ≤ package ≤ psys). The as-shipped per-task summary field under-counts psys on ~10% of tasks and was not used.

3 · Per computation vs. data movement — testing the hypothesis

Two inference phases sit on opposite sides of the roofline: prefill reads the 8k-token prompt and is compute-bound (weights reused across many tokens); decode generates one token at a time and is memory-bound — every token streams the model's active weights from DRAM once.

Per-token energy: prefill vs decode across domains — A single **decode** token costs **~4.6×** more platform energy than a **prefill** token (5.9 J vs 1.3 J), and **~11.8×** more in the DRAM domain — because each decode token streams the full active weight set from DRAM once, while a prefill token amortizes that same read across the whole prompt. From a per-step regression over 2,136 real steps (R²=0.96); ratios are predictor-dependent (~4.2–5.3× psys, ~11–13× dram).

Core vs DRAM power for STREAM vs GEMM — The mechanism. Streaming from DRAM (STREAM triad, 56 GB/s, negligible arithmetic) holds the cores at **27 W** — **~24 W above idle** — starved for data. The DRAM devices themselves draw only **2.8 W**. The energy cost of data movement is real, but RAPL bills it to `core`, not `dram`.

Energy per FLOP vs arithmetic intensity — Energy per FLOP vs. arithmetic intensity (one variable-AI kernel). Streaming from DRAM at low intensity costs ~**5,000 pJ/FLOP** — roughly **48×** the true BLAS-GEMM compute floor (105 pJ/FLOP). Holo3 decode (AI ≈ 3.6) actually sits *near the efficient knee*: its per-token premium comes from the bytes it moves per token (left), not a low-intensity FLOP penalty.

Reframing the hypothesis. You were right that data movement is the bottleneck — decode is memory-bound and 4.6× more energy-costly per token. You were off about where the meter sees it: the DRAM chips sip <1 W; the cost is paid as high-power cores held active but starved on the DDR5 bus. A byte moved to/from DRAM registers 39 pJ in the dram domain but 768 pJ at the platform (~20×) — because slow movement keeps the whole ~20 W platform awake longer. Caveat: on this client SoC the RAPL dram domain is a partial/modeled estimate of DRAM access energy (it excludes refresh, I/O, termination and the on-die memory-controller/PHY, which fold into package), so the 1.4% and 39 pJ/byte are lower bounds on true memory-system energy — which strengthens, not weakens, the point that movement cost is hidden outside the dram meter.

So why isn't decode the top energy line? Because the OSWorld agent re-sends a ~8,000-token prompt (three screenshots + accessibility tree) every step but generates only ~284 tokens — a 28.5× prefill:decode token ratio. Prefill is cheap per token yet enormous in aggregate (~63% of energy); decode is expensive per token but small in volume (~10%).

4 · Per processor — CPU vs iGPU vs NPU

The decisive test: run the full Holo3-35B on the iGPU. We freed ~54 GB, loaded all 21 GB of the model onto the Xe3 via llama.cpp Vulkan (-ngl 99), and metered prefill + decode energy across seven OSWorld task sizes on CPU vs iGPU.

Full Holo3-35B prefill and decode energy, CPU vs iGPU, across OSWorld task sizes — Full Holo3-35B-A3B, CPU vs iGPU, seven OSWorld task sizes. **Prefill:** the iGPU uses **4.1× less energy** (≈170 vs ≈700 mJ/tok) and frees the cores (**58 → 1.6 W**) — though the MoE-on-Vulkan path caps throughput, so it is not faster (92 vs 107 tps). **Decode:** the iGPU is **~1.9× worse** on energy and (the robust signal) **2.5× slower** (8 vs 20 tps) — memory-bound work stays on the CPU.

Why 4.1×, and not more? Two supporting probes bracket the ceiling:

Compute energy per FLOP and freed core power across CPU/iGPU/NPU — Ideal ceiling — identical 137-GFLOP matmul. The **iGPU** and **NPU** do it at **~35× less energy per FLOP** (2.6–2.8 vs 93) and ~25× faster; core power collapses 41 W → 3–6 W. This is fp16/int8 vs fp32, the engines' native precisions.

Real transformer prefill/decode energy CPU vs iGPU — Small dense model — Qwen2.5-1.5B, llama.cpp Vulkan. The iGPU wins **prefill** **8.6×** (36 vs 311 mJ/tok) and runs it *faster* — because a 1.5B dense model actually saturates the iGPU. Decode still ties/loses.

The iGPU's raw compute is ~35× more efficient per FLOP; a 1.5B dense model captures 8.6× of that. But Holo3-35B is a Mixture-of-Experts, and the Vulkan MoE prefill tops out near CPU speed — so the realized energy win on the actual deployed model is 4.1×, earned entirely by freeing the cores, not by going faster. The win grows as the backend matures.

The unified-memory catch. CPU, iGPU and NPU share one DDR5 pool, so throwing more compute at a memory-bound op cannot make DRAM faster. On the real 35B, iGPU decode is ~1.9× worse on energy and 2.5× slower — it reaches only ~40% of the DRAM roofline (immature MoE-Vulkan decode) on top of the shared-memory ceiling. The takeaway is unambiguous: offload prefill to the iGPU (4.1× energy, cores freed); keep decode on the CPU.

engine	GFLOP/s	pJ/FLOP (compute)	vs CPU energy	core W during work	real-model prefill mJ/tok
CPU · 16C AVX2	474	93.2	1×	40.5	311
iGPU · Xe3	11,951	2.84	33×	5.9	36
NPU · AI Boost	11,408	2.61	36×	2.9	—

The synthetic 33×/36× compares iGPU fp16 / NPU int8 against CPU fp32 (each engine's native precision) — a device+precision figure, the upper bound on the device-alone gain. The real deployed model realizes less: full Holo3-35B (MoE) prefill on the iGPU is 4.1× (≈700 → ≈170 mJ/tok); a 1.5B dense model gets 8.6×. NPU was not run on the GGUF models (llama.cpp targets CPU/Vulkan). The "real-model prefill" column below is the 1.5B point.

5 · What's most inefficient — and what to fix

Ranked by measured leverage against the 9.8 kWh total:

#	lever	attacks	share	mechanism / measured basis
1	Shrink the re-sent prompt (downscale/crop screenshots, prune the a11y tree) + cache the stable head	repeated prefill	≤63%	8,078 prompt tokens re-encoded every step. The screenshot + a11y tree is fresh each step (the screen changes after each action), so prefix caching only removes the stable head (system + task) — a fraction of the 63%; the bigger lever is making the per-step prompt smaller.
2	Offload prefill to iGPU / NPU	prefill compute	63%	Prefill is compute-bound ⇒ 4.1× less energy on the full Holo3-35B (8.6× on a 1.5B dense model; 33× synthetic ceiling), and frees the cores (58→1.6 W). Keep decode on the CPU.
3	Race-to-idle	platform tax	27%	A fixed ~20 W (VRM/rails/I/O) is billed for every second the box is awake. Levers 1–2 shorten wall time ⇒ shrink the fixed-power × time integral directly.
4	Cut bytes/token (lower-bit quant, fewer active experts, spec-decode)	decode	10%	Decode is memory-bound; offload barely helps (shared DRAM). The lever that works is moving fewer bytes per token — decode pays a 4.6×/token premium (11.8× in the DRAM domain) precisely because it re-streams the active weights each token.

Bottom line. The lever is the prefill, which is ~63% of the energy: shrink the 8k-token prompt that is re-sent every step, and push the unavoidable prefill compute onto the iGPU (4.1× less energy on the full 35B, cores freed). Race-to-idle then reclaims part of the 27% platform tax for free. The DRAM chips were never the problem — the traffic pattern and the always-on platform are.

Appendix · instruments, operating points & caveats

Busy vs idle power

busy: psys 76.6 W · pkg 55.0 · core 52.1 · dram 0.93
idle: psys 19.5 W · pkg 9.1 · core 7.1 · dram 0.55
203k / 7k samples @ 2 s

Microbench roofline

DRAM roof 56 GB/s · compute roof 555 GFLOP/s (AVX2 GEMM)
memory byte: 39 pJ (DIMM) / 768 pJ (platform)
compute FLOP: 105 pJ (platform)

Workload shape

8,078 prefill + 284 decode tokens / step
prefill:decode 28.5×
17.3M prefill vs 0.6M decode tokens total

Honesty notes. (1) uncore reads ~0 during CPU inference — on Panther Lake the memory-controller/PHY energy folds into package, so the on-chip data-movement cost sits inside the core/package figures, not separately metered; and the dram domain is a partial/modeled client-SoC estimate, so the 1.4% share is a lower bound. (2) The phase split is an OLS regression on 2,136 real steps (R²=0.96), with prefill work defined as max(0, prompt_tokens − cached_tokens) (tokens actually re-encoded). It is predictor-dependent: prefill lands at 56–74% and fixed at 16–34% depending on whether cached tokens are counted; decode ~10% is robust across definitions, and the intercept independently matches idle-power × inference-time. (3) Per-domain watts are raw (busy); pJ/byte and pJ/FLOP are net of the idle baseline. (4) The accelerator 33×/36× is fp16/int8 vs fp32 (device+precision); the real-model figures are 4.1× (full Holo3-35B MoE) and 8.6× (1.5B dense). (5) The full 35B iGPU run is text-only (no vision projector — mmproj is not yet Vulkan-compatible, so the vision-encode energy is characterized on CPU only, via full361); prefill prompts are token-matched to real OSWorld sizes (this is the robust, headline comparison). Decode is a weaker probe: greedy generation hit EOS after only ~17 tokens on the short prompt, so the decode energy ratio (~1.9×) carries real uncertainty — the firm signal that decode belongs on the CPU is the clean 2.5× throughput deficit (server-side timing), and the absolute decode energy is a lower bound vs a full 8k KV cache. (6) Absolute totals use the consistent per-step integration; the shipped per-task energy_j field under-counts psys and was discarded.