Holo3-35B-A3B · OSWorld-Verified · NUC16 — profiling corpus SCHEMA

What this is

Every step of the full 361-task OSWorld-Verified run of Holo3-35B-A3B on the Intel Panther Lake NUC (CPU-only llama.cpp) is instrumented end to end. The run reproduces the official leaderboard config (100-step budget, screenshot-only, single rollout) — but alongside the score it emits a step-indexed, replayable corpus of state, action, and measured cost.

Two granularities. The atomic record is one step (one model call); the unit a router actually reasons over is one whole task — a trajectory of steps from the first screenshot to answer, plus episode-level aggregates. The research target is an upper-layer model router that sends each step to an appropriately-sized model. The intended heuristic was large model for early planning, small model for the routine tail — and because the corpus is step-indexed, that hypothesis is directly testable. We test it below, and the data refutes the simple version.

Measured so far: 933 steps across 63 completed tasks (median 8, mean 14.8, max 100 steps/task). Projected to the full 361-task set: roughly 8,000–15,000 rows.

Captured from four time-aligned sources

One logical row = one agent step (one model call). It is assembled by joining these on (domain, task_id, step) and the 2 Hz telemetry on the ts window.

Plus, on disk per step: the full-resolution screenshot PNG; per task: a screen recording (mp4) and the OSWorld config snapshot.

One row per agent step — fields grouped by role

Example values are real, sampled from chrome/06fe7178 step 1 (task: "Can you make my computer bring back the last tab I shut down?").

Zooming out — one task = one episode

The row above is one step. A task is the whole sequence of those rows for one instruction, wrapped with episode-level aggregates — and it is the unit a router policy acts on. Below: the episode schema (real values from chrome/030eeff7) and that task's complete 14-step trajectory.

One complete trajectory — all 14 steps of chrome/030eeff7

Every per-step quantity a step-level router consumes, for one full episode — grouped output · KV state · cost coefficients · label. ctx = full context (prefilled + reused); reuse% = KV-hit rate; split = prefilldecode share of the step; switch s = full re-prefill penalty if the router swapped models at that step (= ctx ÷ pf t/s); score = task-level label (the only outcome signal — per-step quality is the missing counterfactual). reasoning bar = <think> chars vs the task max 1047. Scroll within the box; goal: enable Chrome's "Do Not Track".

Per-step time decomposed as prefill = prompt_tokens ÷ pf t/s and decode = predicted_tokens ÷ dec t/s; the remainder (HTTP + sampling) is <0.6 s every step. Task totals: prefill 2354 s (89%), decode 291 s (11%), other 6 s; 38.2 Wh.

Corpus at a glance — 63 tasks, 933 steps so far

Solve rate collapses as tasks get longer — and long tasks are also where cost concentrates (latency rises with trajectory length). The long tail is the expensive, low-yield regime any efficiency layer most wants to act on.

Per task type — is prefill always the bottleneck? (as of 63 tasks / 933 steps)

All 10 OSWorld application domains, across every completed task. prefill% / decode% = share of compute time (prefill_s = prompt_tokens ÷ pf t/s, decode_s = predicted ÷ dec t/s); split shows it as prefilldecode. solve% is strict (score = 1.0). Sorted by prefill share.

One complete trajectory for every task type

A full per-step trajectory — like the chrome flagship above — for each of OSWorld's 10 domains (one representative completed task each, all fact-checked against the run). Click to expand. Columns are the router-relevant cost coefficients; reuse% = KV-hit rate, split = prefilldecode share of the step.

Same shape in every domain: a cold, cheap first step, then a one-time prefill jump when the 3-screenshot window fills and KV reuse collapses to ~950 tokens — after which prefill dominates every step regardless of the action. os is shown as a score-0 near-miss; the cost profile is identical to the successes.

Attacking the prefill wall — KV-cache management & stage→device affinity

The diagnosis is the same everywhere above: ~89% of inference time is prefill, and the KV prefix dies at the 3-screenshot eviction (reuse 70% → 10%), so the agent re-prefills ~7–8k tokens every step. A model router only moves the 11% decode term — the dominant 89% needs a different attack. Two levers, meant to stack:

1 · KV-cache-stable context. Keep the prompt prefix invariant so screenshot eviction stops invalidating it — evict within a stable window, pin images at fixed offsets, or carry KV across steps — restoring the 70% reuse we briefly see at step 3 instead of the frozen 947. Corroboration from the companion OpenClaw prefix-cache experiment: auto-compaction that breaks the prefix cache produced a ≈24× prefill blow-up between two consecutive turns — the same failure mode, measured on a different stack. Estimated 2–3× faster steps here if the prefix is preserved. [measured diagnosis + measured corroboration]

2 · Stage→device affinity. Inference stages have different bottlenecks, so place each on the silicon that fits: vision-encode + prefill are compute-bound and parallel → an iGPU/NPU; decode is memory-bandwidth-bound and sequential → the CPU. Holo3 is prefill-dominated, so moving prefill onto a compute engine attacks the 89% directly. On this box that path is currently blocked (the Xe3 vision encoder crashes — see the live-tracker root-cause); the durable fix needs OpenVINO MoE-VLM support or higher-bandwidth edge silicon. [proposed; device bring-up measured]

3 · Compose with the router. Because prefill dominates and switching models invalidates the (weight-specific) KV cache, the layers are complementary, not competing: cache-stable context + stage affinity cut the 89% prefill; a difficulty-gated, switch-aware model router trims the 11% decode and skips steps that don't need the big model. Stack them — don't pick one.

Does effort front-load? — testing the routing hypothesis on real data

The motivation assumed early steps are heavy (planning) and the tail is routine. With a step-indexed corpus that is directly checkable. An adversarial pass over the 36 successful episodes computed each effort proxy by within-task position (terciles), per task, then averaged across tasks. The output-side proxies (reasoning chars, generated tokens) are used because — unlike latency — they do not inflate mechanically as the prompt grows.

Method: within each successful task, step i is normalized to (i−1)/(N−1) and binned early/mid/late by tercile, plus the literal first/last step; per-task means are averaged equally across the 36 tasks (a per-task trend, not a global step pool). n is small (median successful task ≈ 6 steps; 11 tasks have ≥9 steps), so treat magnitudes as directional.

Host telemetry — 2 Hz, joined on ts

The iGPU and NPU read idle by design — the device bring-up (on the live tracker) shows why this MoE-VLM serves only on CPU on this silicon.

What this trains — and what it doesn't (yet)

Toward the step-level model router — now motivated by per-step difficulty rather than the refuted early/late split — the corpus is necessary, not yet sufficient.

✓ Fully covered — the state & the cost. Every feature a router conditions on at decision time (step index, domain, context size, cached-token ratio, the screenshot, the message history, the tool needed) and the full price term (per-step latency and energy) is measured. Most routing work hand-waves the cost side; here it's instrumented.

Two honest caveats baked into the schema: (1) steps_profile.tool is always null (a known bug) — join the per-step tool from model_calls.jsonl/traj.jsonl instead; nothing is lost. (2) reasoning_len is the char count of the <think> block — the reasoning text is discarded, which is the single most valuable missing field for a planning-vs-routine classifier and is worth capturing in a re-run.