SLM on Xbox Series S at 71 tok/s

Update, July 2026. Three of the six open problems below are now closed, including the GPU question, and two of the answers falsify numbers in this report.

Update, August 2026. A later measurement falsifies the July conclusion in turn: disk is no longer the binding constraint. Current performance figures live in the project's generated benchmark summary, not here.

The original text is unchanged as a snapshot. Both sets of corrections are in the update sections near the end.

Why this matters

An Xbox Series S is a Zen 2 machine: eight cores at 3.6 GHz with AVX2, 10 GB of unified GDDR6, an RDNA 2 GPU, and a one-time ~$19 Dev Mode unlock that lets you sideload your own code. On paper it is a capable, cheap, and largely unexplored substrate for local inference. So I tried to run a small language model on one, with no cloud in the path.

The short version: it works. The console runs SmolLM2-360M-Instruct (INT4) at about 71 tokens per second, fully on-device, through ONNX Runtime GenAI. The longer version is more useful, because the interesting part of this port is not the demo. It is the set of constraints I hit, and which of them I have not solved. This article leads with the open problems on purpose. I would rather publish the boundary of what I know than a screenshot that hides it.

I maintain the project, xllama, in the open. Every number below comes from its benchmark logs and constraint notes, not from memory.

What actually runs today

The working configuration is deliberately boring:

Model: SmolLM2-360M-Instruct, INT4, 403 MB on disk, context window 2048, ChatML prompt template.
Runtime: ONNX Runtime GenAI on the CPU execution provider (Zen 2).
Package: a self-contained UWP MSIX with the model bundled inside it, sideloaded via the Xbox Device Portal.

The decode loop is the standard ORT GenAI token pump, with an abort flag so the gamepad B button can cancel a generation mid-stream:

while (!OgaGenerator_IsDone(gen.get())) {
    if (params.abort_flag && params.abort_flag->load()) break;
    oga_check(OgaGenerator_GenerateNextToken(gen.get()), "GenerateNextToken");

    const int32_t* next = nullptr;
    size_t n = 0;
    oga_check(OgaGenerator_GetNextTokens(gen.get(), &next, &n), "GetNextTokens");
    for (size_t i = 0; i < n; ++i) {
        const char* piece = nullptr;
        oga_check(OgaTokenizerStreamDecode(stream.get(), next[i], &piece), "decode");
        if (piece && *piece && params.on_token) params.on_token(piece);
    }
}

Measured on the console (ORT GenAI 0.13.2, INT4, n_ctx 2048):

| Threads | Decode tok/s | Peak working set | | ------- | ------------ | ---------------- | | auto | 66.9 | 704 MB | | 4 | 71.4 | 771 MB | | 6 | 68.0 | 772 MB | | 8 | 28.2 | 771 MB |

That is the baseline. Everything below is what stands between this and a result I would actually call finished.

The open problems

1. I cannot prove the GPU is doing anything

This is the one that bothers me most. The whole reason to pick a console over a Raspberry Pi is the RDNA 2 GPU, and I cannot yet confirm I am using it.

SmolLM2-360M loads under the DirectML execution provider without crashing, once I disable the CPU memory arena and memory pattern planner:

"session_options": {
  "provider_options": [
    { "dml": { "enable_cpu_mem_arena": "0", "enable_mem_pattern": "0" } }
  ]
}

It then produces output at about 71.7 tok/s, which is suspiciously close to the CPU baseline. That number is the problem, not the reassurance. ORT can silently fall back to CPU for operators the GPU path does not support, and a result indistinguishable from the CPU number is exactly what a silent fallback looks like. To tell the two apart I need a D3D profiler (PIX) or GPU hardware counters, and I have no profiling instrumentation on the console yet. Until I do, the honest finding is narrow: the 360M model fits the GPU memory pool, but whether it executes on the GPU is unconfirmed.

2. The GPU memory pool is small, and that caps everything

Larger models do not get this far. When OgaCreateModel initializes the DirectML provider for a model whose weights exceed the available GPU pool, the allocator returns null and the next use of that pointer faults:

OgaCreateModel failed: SEH 0xC0000005 (STATUS_ACCESS_VIOLATION)

By watching where that boundary falls, the usable GPU-accessible pool for a UWP app on the Series S looks like roughly 768 MB. Phi-3.5-mini INT4 (~2.2 GB) reliably OOMs; the 360M model (403 MB) does not. I want to be careful here: that 768 MB figure is inference from observed out-of-memory behavior in my own tests, not a documented Xbox platform specification. I do not treat it as an authoritative claim about the console's internal memory layout. But as an engineering ceiling it is consistent, and it means any model near or above 1 GB is off the table for the GPU path regardless of whether problem 1 is ever solved.

3. The disk budget is tighter than the RAM budget

Before a model can OOM, it has to fit on disk. A freshly activated Dev Mode partition gives roughly 2.2 to 2.5 GB of free space, and deployment briefly needs about twice the package size because the MSIX is staged before it installs. In practice that means an on-disk model budget under 600 MB if I want to bundle it in the package, and a deploy that fails with 0x80070070 (disk full) if I push past it. The 403 MB model fits with room to spare. A 1.4 GB model does not, even though the console has 10 GB of RAM. Disk, not memory, is the first wall.

4. The in-app download path is written but unproven

The way out of the disk budget is to stop bundling the model and fetch it at first launch. I implemented that: a ModelDownloader that streams from a Hugging Face endpoint in chunks via HttpClient, with a resolution chain of LocalState, then the installed package, then a download fallback. The code exists and compiles. It has never actually run on the console, because the build that ships always finds the bundled model first and never reaches the fallback. So whether plain HTTPS to Hugging Face works from inside the Xbox AppContainer is still an open question. To test it I have to ship a build with no bundled model on purpose, which I have not done.

5. More threads make it slower

The thread table above hides a sharp cliff. Four threads is optimal at 71.4 tok/s. Eight threads drops to 28.2 tok/s, a regression of roughly 60%. The cores are not the constraint; memory bandwidth is. INT4 decoding on Zen 2 saturates the available bandwidth well before it saturates the eight cores, and adding threads past that point just adds contention. The fix is not clever code, it is a pinned intra_op_num_threads=4. But it is a reminder that the usual "use all the cores" instinct is actively wrong on this hardware.

6. It only runs in Dev Mode

Everything here depends on the ~$19 Dev Mode unlock. There is no path to a retail console. That is fine for a research baseline and a reproducible build, and it is a hard limit on calling this something a normal user could install. I am not going to pretend otherwise.

The scars already paid

Two problems are solved, but only after they cost me real time, so they are worth recording.

The first was a crash inside OgaCreateModel on a model that loaded fine on Linux. ORT 1.24.4 calls std::filesystem::weakly_canonical() to validate the path of an external .onnx.data file, and on Windows that walks the path from the drive root upward. One of the intermediate segments is the Xbox AppContainer's user-manager directory, which the sandbox cannot read, so the walk hits ACCESS_DENIED and throws. The fix is to merge the external data into a single self-contained model.onnx at build time, so the validation path is never taken. A small Python script in CI does the merge.

The second was the XAML compiler crashing (WMC9999) during the build of a C++/WinRT project on a current Windows SDK. Rather than fight the markup compiler, I build the entire UI programmatically in C++ with Windows.UI.Xaml.Controls. No .xaml files, no metadata provider, no compiler pass to crash.

What it would take to close each

To stop hand-waving about the GPU, I need on-device D3D profiling so I can confirm kernel execution and measure GPU tok/s against the CPU baseline for the same model and quantization. To make the GPU path worth confirming, I want a sub-400 MB INT4 candidate such as Qwen2.5-0.5B that comfortably fits the pool. To retire the disk budget as the binding constraint, I need to validate that Hugging Face download from the AppContainer actually works, then drop the bundled model from the package. None of these are research questions. They are instrumentation and legwork, which is usually where these projects actually live.

Update — July 2026

Two weeks after publishing, three of the six open problems are closed. Two of the answers falsify numbers above, so they belong here rather than in a changelog nobody reads.

Problem 1 is solved, and the suspicion was justified. The ~71.7 tok/s DirectML result was a silent CPU fallback. The GPU path was never initializing: ONNX Runtime GenAI's device factory collided with the D3D12 device the XAML compositor creates when the window activates (error 887A0036). Running the benchmark in a headless, D3D12-clean process fixed that, and the underlying fix was contributed upstream and validated on the console (onnxruntime-genai#2280). With the GPU actually executing, the verdict turns out to be per-workload, not binary. For chat decode the CPU still wins at this model scale: 68 tok/s on CPU INT4 against 46.8 on GPU fp16, and only 8.8 on GPU INT4, because DirectML implements the low-bit matmul as dequantize-to-fp16 plus a full GEMM, so INT4 moves more bandwidth than fp16, not less. But prefill inverts at scale: at roughly 1,000 prompt tokens the GPU does 354 tok/s against 198 on CPU, cutting time to first token from 5.3 s to 3.0 s. And on batch compute the GPU is not subtle: SD-Turbo generates a 512×512 image in about 5.6 s, 11.1× faster than the CPU on the same diffusion workload. The app now routes per conversation — CPU for decode-heavy chat, GPU for long-prompt prefill and image generation.

Problem 2's number was wrong. With the GPU path actually initializing, the memory budget could be measured instead of inferred from crash boundaries: the reported budget is 3,801 MB, not roughly 768 MB. I flagged that figure as an inference from observed OOM behavior rather than a documented specification, and the hedge earned its keep — the inference did not survive contact with a working initialization path. The corrected conclusion is the one problem 3 already pointed to: disk, not GPU memory, is the binding constraint.

Problem 4 shipped. The MSIX no longer bundles a model at all — it is about 19 MB — and on first launch the app downloads SmolLM2-360M (~417 MB) from a versioned GitHub Release catalogue, with a progress bar. HTTPS from inside the Xbox AppContainer works. The open question closed the boring way, by shipping the build that forces the code path.

The candidates moved too. These updated numbers are from ORT GenAI 0.14.1, where the same CPU INT4 configuration measures 66.3 tok/s (the tables above were measured on 0.13.2). Qwen2.5-0.5B, the candidate I named, turned out to weigh ~822 MB in its INT4 ONNX build — the 151k-token vocabulary embedding dominates — and is off the table. Instead, v1.1.0 compiles both ONNX Runtime GenAI and llama.cpp into one binary and dispatches per model, so the catalogue now carries GGUF builds of current-generation small models (Qwen3.5-0.8B at 508 MB, LFM2.5-350M at 219 MB) that the ORT model builder cannot produce. Their on-console benchmarks are pending, so the default model has not changed.

Problems 5 and 6 stand: four threads is still the optimum, and it still takes Dev Mode.

Update — August 2026

One conclusion in this report is now false, and it is the one the July update endorsed.

Disk is no longer the binding constraint. Problem 3 called storage the first wall, and the July update promoted that to the corrected conclusion. Both were right about a freshly activated Dev Mode partition, which is what I had measured. Both stopped being right on 8 July 2026, when I raised the Dev Mode allocation to 90 GB through Dev Home under Manage Dev Storage. The 2.2 to 2.5 GB figure now describes only the default state of a clean activation. What bounds model sizing is the GPU budget and RAM, which is what problem 2 was about all along. One disk limit survives: a reported 2 GB per-file ceiling in Dev Mode, which matters for merged ONNX files above that size.

The decode numbers here are a snapshot, and better ones exist. Every figure above is correct for the configuration it names, which is why the text is unchanged. But the tables were measured on ORT GenAI 0.13.2 at four threads, and the shipping configuration has moved since: six threads, and a repack path that was compiled but never enabled turned out to be costing a large share of prompt throughput. Current figures live in the project's generated benchmark summary, built from committed CSVs, where each number carries the command that reproduces it and CI fails when a published figure drifts from its evidence. Read them there rather than trusting a date-stamped paragraph in a blog post.

The project outgrew this report. In June it was one 360M model through ONNX Runtime GenAI. It now carries a catalogue from 270M to 3B across both runtimes, generates images on the console, and ships as a versioned package with a ten-gate console validation suite. The fastest catalogue model, LFM2.5-350M, decodes at 94.9 tokens per second on the CPU, median of three runs in that same summary. That is a different article rather than a correction to this one.

FAQ

How fast is it?

About 71 tokens per second of decode for SmolLM2-360M INT4 on the CPU execution provider at four threads, with a peak working set around 771 MB (measured on ORT GenAI 0.13.2; the same configuration measures 66.3 tok/s on 0.14.1). Going to eight threads drops it to about 28 tok/s because memory bandwidth, not compute, is the bottleneck.

Does the language model run on the Xbox GPU?

Yes — and the number that looked like a silent CPU fallback was one. With the initialization conflict fixed (see the July 2026 update), the measured verdict is per-workload: the CPU wins chat decode (68 vs 46.8 tok/s), the GPU wins long-prompt prefill (354 vs 198 tok/s at ~1k tokens) and image generation (11.1× on SD-Turbo). The app routes each conversation to the processor that wins its workload.

Can a normal user install this on their Xbox?

No. It requires Xbox Dev Mode, a one-time paid unlock, and there is no path to a retail console. This is a reproducible research baseline, not a consumer application.

Why such a small model?

Disk, at the time. A freshly activated Dev Mode partition has only a few gigabytes free and deployment needs roughly twice the package size during install, so a ~400 MB INT4 model fit with room to spare while a multi-gigabyte one failed the disk check before it ever loaded. The GPU memory pool turned out to be far larger than my early estimate (a measured 3,801 MB, not ~768 MB). That disk ceiling was lifted in July 2026 by raising the Dev Mode storage allocation, and model sizing is now bound by the GPU budget and RAM. See the August 2026 update.

The full build, benchmark logs, and constraint notes are in the xllama repository. "Xbox" is a Microsoft trademark; this is an independent research project and is not affiliated with Microsoft. If you have run inference on console hardware and have profiling data I do not, I would like to compare notes.

Running a Small Language Model on an Xbox Series S