Holo3-35B is a qwen3_5_moe vision-language model — an unusual mix that today's accelerator toolchains
don't fully support:
There are two ways to reach the accelerators, and each hits a different wall:
| Device | Path | What blocks the 35B |
|---|---|---|
| iGPU | OpenVINO | The exporter exists and the ViT runs on the iGPU, but the 35B LLM won't export: the converter's tracer rejects its data-dependent branching, and even past that the graph build needs ≥94 GB RAM. |
| llama.cpp (Vulkan) | The LLM offloads fine, but the vision part crashes the GPU — I reproduced it exactly this
round: vk::DeviceLostError inside clip_image_batch_encode, and it's size-triggered
(a small image encodes fine; a real screenshot loses the device). | |
| NPU | OpenVINO only | The NPU wants fixed shapes and no dynamic branching — but the MoE routing and the SSM scan are both dynamic, and it tops out around 8B while the LLM is 35B. |
In plain terms: the CPU works because llama.cpp implements this whole architecture. On the accelerators the vision half is fine; the 35B language half is the stubborn part — the MoE and the sheer size are what push the export past the memory wall.
Grouped by device, then by runtime. The 35B is the hard case; the dense 4B/9B are where it works.
▸ On the iGPU (Xe3)
| Attempt | Result |
|---|---|
| OpenVINO | |
| 35B · Vision (ViT) | ✓ exports + runs on the iGPU, bit-exact vs reference (cosine 1.0) — fastest device for it. |
| 35B · LLM | ✗ no standalone path — the whole-model export fails first (next row). |
| 35B · whole | ✗ two walls: the converter's tracer rejects the data-dependent branches, and the graph build needs ≥94 GB RAM (OOM at 94 GB; disk-offload + 71 GB swap didn't help — the wall is tracing, not RAM). |
| llama.cpp (Vulkan) | |
| 35B · Vision (ViT) | ✗ vk::DeviceLostError inside clip_image_batch_encode — a full-size screenshot's ViT encode hangs the GPU (a small image is fine). Unresolved Intel-Vulkan bug. |
| 35B · LLM | ✓ works and is correct, ~1.6× faster prefill than CPU. (The June garbage-output bug did not return.) |
| 35B · whole | ✗ the LLM offloads fine, but the vision crash blocks the whole pipeline on the iGPU. |
| Hybrid | |
| LLM via llama.cpp + Vision via OpenVINO | ◑ designed & located the splice — feed OpenVINO's ViT embeddings into llama.cpp at clip_image_batch_encode, reusing its M-RoPE / decode. Didn't need to ship it — the dense whole-model export (below) is cleaner. |
| Other models | |
4B / 9B · whole → OpenVINO (dense qwen3_5) |
✓ the win. No MoE, so the export traces and the graph fits (9B: 44 GB peak, 18.9 GB IR; 4B: 9.7 GB). Both compile and run the whole model on the iGPU and ground the screen correctly (4B ~257 / 7.2 tok/s, 9B ~200 / 4). |
▸ On the NPU (NPU 5 · Intel AI Boost)
| Attempt | Result |
|---|---|
35B (qwen3_5_moe) | |
| 35B · Vision / LLM / whole | ✗ out of scope for the NPU: it caps ~8B and needs fixed shapes + no dynamic branching, but the MoE routing and the SSM scan are both dynamic. (A tiny static-dense proxy did compile — proof it's possible — but the full 35B doesn't fit.) |
4B / 9B (qwen3_5 dense) — the real problem: make the LLM static | |
| Vision tower + token embeddings | ✓ compile and run on the NPU with fixed shapes — the ViT matches CPU to cosine 0.999995 (~22 ms). |
| LLM — as exported (stateful) | ✗ compile fails at the graph level, not on any op: all 7,450 ops are individually supported, but the
mutable KV + Gated-DeltaNet SSM/conv state (ReadValue/Assign) won't assemble into the NPU's
static dataflow. |
| LLM — re-exported stateless (the fix — partial) | ◑ stateful=False lifts the KV and the SSM/conv state out as explicit static
inputs, and the LLM decode runs on the NPU (static T=1 loop): 4B "the capital of France is Paris" at
6.4 step/s, 9B at 3.9. But only decode. A batched prefill graph (T>1) will not compile on the NPU —
the multi-token Gated-DeltaNet scan is rejected, exactly like the stateful graph (tried chunk sizes 64–512, contexts
512–2048; all fail). So a real prompt is prefilled either token-by-token on the NPU (~0.18 s/token → ~205 s for
a 1.1k-token OSWorld prompt) or on the iGPU — hence partial, not a full-NPU win.
And there's a second wall: even token-by-token, the NPU output goes incoherent past ~250 tokens and
reverts to echoing the prompt. It is not precision (the iGPU runs the same f16 and stays correct to
1.5k) and not a bug in my loop — I root-caused it to a specific NPU op-lowering fault in the Gated-DeltaNet
layers (sweep + op-level diff below) — then worked around it with an NPUW hybrid that keeps ~98% of the
model on the NPU and offloads only those layers (below). |
| Image→text — full VLM step (vision iGPU → LLM NPU) | ✓ a real screenshot → the LLM decodes the answer on the NPU: 4B "Go to DataAccess (unsafe)", 9B "the app is requesting access to sensitive Google-account info" — correct grounding. (The vision runs on the iGPU; the NPU ViT is static-shape and would need manual orchestration to join the loop.) |
| NPU decode — why it first failed | ✗ The first NPU subset (10 tasks) scored 0/10: at OSWorld prompt lengths the decode went
incoherent — every step echoed the tool schema / prior screenshot instead of an action, so all tasks fell back to
WAIT. An iGPU-vs-NPU sweep (same loop, same IR, only the device changes) localized it:
prompt tokens → does it still answer "capital of France?"
12 · 250 · 488 · 998 · 1508 iGPU ✓ ✓ ✓ ✓ ✓ ·
NPU ~ ✗ ✗ ✗ ✗ (NPU echoes from ~250 tokens on; iGPU is correct to 1.5k)
Chased to the bottom, it is an Intel NPU-compiler miscompute — not precision (CPU forced to f16 is
coherent; both devices default to f16), not the weights (bf16 and true fp16 fail identically), and not any single
op. Per-layer diff: the first Gated-DeltaNet layer is bit-perfect on the NPU (conv & SSM cos 1.0) and divergence
appears at layer 1 and compounds through depth — while every distinctive op (delta-rule SSM cell, causal conv1d,
gated RMSNorm) is NPU-exact in isolation (cos 1.0). Removing the once-only scan Loop and the
shape-derived NonZero gather changed nothing; no compiler toggle helps (optimization-level=0 ⇒
the bug is in mandatory DPU lowering, not fusion); it survives the latest driver (1.32.1 → 1.33.0). So the
NPU's lowering of the deep repeated 32-layer GDN graph is wrong — a fileable compiler defect (matches open
openvino#35222; Intel lists Qwen3-Next/3.5
GDN as CPU+GPU-only, NPU unvalidated). |
| NPU decode — the fix: NPUW hybrid | ◑ Coherence solved on the NPU. Since the miscompute is confined to the GDN recurrence, I use
OpenVINO's NPUW online partitioner to keep the model on the NPU while offloading just those ops to CPU-f16
(proven coherent): NPU_USE_NPUW=YES · NPUW_ONLINE_AVOID=Op:GroupConvolution/NPU,Op:Loop/NPU,Op:Scatter…/NPU.
The dumped partition plan: NPU 52 groups / 2613 layers (all MatMuls, the 8 full-attention layers, FFN, norms,
LM-head) vs CPU 50 groups / 51 layers — exactly the 24 GroupConvolution + 24 Loop + 3
Scatter GDN primitives. ~98% of ops stay on the NPU; the decode is now coherent to 1113+
tokens where it used to echo. Cost: the offloaded recurrence forces ~48 NPU↔CPU handoffs/token → ~1.7 tok/s
(vs iGPU 7.2), and the NPUW runtime GP-faults after ~1–2.5k inferences — worked around with a supervisor that
auto-restarts (~24 s, cached) and an agent that retries the in-flight step. |
| End-to-end on the NPU — image→action | ✓ Through the hybrid decoder (vision on the iGPU), a real Google-auth screenshot at a 1113-token prompt →
a coherent grounded action: it read the page, found the Cancel/Continue buttons, and emitted
click("Continue", 750,820). In a live OSWorld episode it produced 8/8 parsed, coherent click/drag
actions and — surviving a mid-run GP-fault via the auto-restart — ran the full 8-step episode to completion. This is a
real change from the old gibberish: the model now takes sensible actions on the NPU. |
| OSWorld subset on the NPU (hybrid, coherent) | ◑ Re-ran the one-per-domain subset on the coherent hybrid → 0/10 solved, but for a different, honest reason than before. Every step is now coherent (27/32 parsed; a mix of click/drag/answer), and every task scored happens to be one the iGPU 4B solved at full resolution — so the ceiling is grounding precision, not coherence: the NPU's ~1.7 tok/s (~13 min/step) forces tiny 400×400 screenshots, and the clicks land slightly off. The iGPU runs the same model at 1280×720 and 7.2 tok/s → 7/10. So: coherence + stability are solved on the NPU; task accuracy is throttled by the resolution the NPU's speed demands. The right next step for a fair NPU accuracy number is a standard-transformer VLM with no GDN penalty — Holo1.5-7B (Qwen2.5-VL), in progress. |
Per stage, where it runs — with throughput where a device runs it. The fastest in each row is green.
| Stage | 35B (MoE) | 9B (dense) | 4B (dense) |
|---|---|---|---|
| Vision encode (ViT) 1024 patches | iGPU + NPU ✓ | iGPU 212 ms · NPU 261 ms | iGPU 176 ms · NPU 175 ms (≈ tie) |
| LLM prefill | iGPU · 124 tok/s (llama.cpp) | iGPU · 200 tok/s (OV) | iGPU · 257 tok/s (OV) |
| LLM decode | iGPU · ~9 tok/s (llama.cpp, MoE) | iGPU · 4 tok/s (OV) | iGPU · 7.2 tok/s (OV) |
| Whole model, one runtime on the iGPU | ✗ | ✓ OpenVINO | ✓ OpenVINO |
| LLM on the NPU (stateless, generates) | ✗ | ✓ 3.9 step/s | ✓ 6.4 step/s |
| OSWorld, whole loop, local | CPU only | iGPU · 5/10 · 37 s/step | iGPU · 7/10 · 28 s/step |
| OSWorld on the NPU (LLM decode) | — | ✗ not run (export OOM) | ✗ 0/10 (incoherent ≥~250 tok) |
Where the NUC's own accelerators land next to the three regimes from the KV-cache study — the CPU baseline, the CPU measured optimum (fake-prefix patch), and the datacenter 2×A6000 — shown at two NUC model sizes (9B and 4B). One caveat up front: the two NUC columns are different, smaller models — the dense Holo-3.1-9B / -4B, not the 35B in the other three — so this is not a like-for-like race. It answers a different question: what can the NUC run on its own silicon, whole-model, no cloud?
| evict baseline | measured optimum | 2×A6000 (GPU ref) | NUC · 9B | NUC · 4B | |
|---|---|---|---|---|---|
| model | Holo3-35B | Holo3-35B | Holo3-35B | Holo-3.1-9B dense | Holo-3.1-4B dense |
| device · runtime | NUC CPU 16C · llama.cpp | NUC CPU · llama.cpp +patch | 2× RTX A6000 · cloud | Xe3 iGPU + NPU · OpenVINO | Xe3 iGPU + NPU · OpenVINO |
| weights | Q4 GGUF | Q4 GGUF | — | 16-bit OV (bf16) | 16-bit OV (bf16) |
| screenshot context | last 3 | last 3 | last 3 | last 1 (bounded) | last 1 (bounded) |
| prefill · tok/step | 5,148 | 2,058 | ~5.1K | ~0.9–2.2K | ~0.9–2.2K |
| prefill · tok/s | 38.7 | 36.9 | ~1.1–2.9K | ~200 | ~257 |
| decode · tok/s | 15.0 | 15.0 | ~150 | ~4 | ~7.2 |
| step latency | 151 s | 75 s | ~3.5 s | ~37 s | ~28 s |
| OSWorld score | 0.858 n=78 | 0.858 n=78 | 0.804 n=361 | 0.50 n=10 † | 0.70 n=10 ‡ |
| footprint | bounded | bounded | datacenter | ~18 GB unified | ~10 GB unified |
Baseline / optimum / A6000 columns from the KV-cache study (Holo3-35B). NUC columns measured this session: 9B prefill ~200 / decode ~4 tok/s; 4B prefill ~257 / decode ~7.2 tok/s. † 9B score = the one-per-domain OSWorld subset (10 tasks, one per domain). ‡ 4B score = the same one-per-domain subset (n=10): the 4B solved 7/10, beating the 9B's 5/10 (it added gimp + vs_code). Both are snapshots, not comparable to the 35B's n=78 / n=361 sets. Takeaway: decode is the weak point (Gated-DeltaNet under-optimized on the OV GPU plugin), but the smaller 4B is ~1.8× faster to decode (7.2 vs 4 tok/s → 28 s/step vs 37) and more accurate on this subset (7/10 vs 5/10) — the better speed/quality point for the NUC — and prefill (~257 tok/s) beats the 35B CPU baseline's 38.7. Either runs the whole agentic loop (LLM + vision) on a mini-PC's own iGPU/NPU, which neither the CPU baseline (vision stuck) nor a cloud GPU (not local) does.
qwen3_5 with
Gated-DeltaNet — the exact thing the NPU miscompiles, which forces the slow CPU-offload hybrid and the tiny 400 px
screenshots that cap accuracy. Holo1.5-7B is a plain full-attention transformer (Qwen2.5-VL, no GDN) and a
first-class OpenVINO arch, so it should run directly and coherently on the NPU — no hybrid, no recurrence ping-pong,
so it can afford full-resolution screenshots. In progress: export stateless → NPU coherence check → OSWorld
one-per-domain on both iGPU and NPU for an apples-to-apples device comparison.optimization-level=0
doesn't help ⇒ mandatory lowering; survives driver 1.32.1 → 1.33.0). A minimal repro is assembled (isolated ops pass,
full graph fails; matches open openvino#35222). Meanwhile the NPUW hybrid runs it coherently on the NPU
(98% of ops on-device, GDN recurrence on CPU) — the practical workaround until Intel fixes lowering. A second bug also
surfaced: the NPUW runtime GP-faults after ~1–2.5k inferences (worked around with an auto-restart supervisor). Also still
open: the T>1 prefill graph won't compile on the NPU, so prefill stays token-by-token.grid_thw to run the
whole VLM (vision + LLM) on the NPU.Measured on wuklab-NUC16 · Intel Core Ultra 7 356H (Panther Lake) · Xe3 iGPU · NPU 5 (Intel AI Boost) · 96 GB.
Models Hcompany/Holo3-35B-A3B (qwen3_5_moe) and Hcompany/Holo-3.1-9B/-4B
(qwen3_5 dense) · llama.cpp Vulkan · OpenVINO 2026.3 (rkazants fork) · NPU driver 1.33.0. OSWorld via the docker
provider (KVM) on the NUC. Runs under /data/runs/2026-07-{07..13}_wuklab-NUC16_holo3*, including the first NPU
subset (0/10, incoherent), the NPUW-hybrid coherent subset (0/10, resolution-limited but 27/32 steps coherent), and the
iGPU-vs-NPU coherence sweep.