Case study · slotstream · Efficient AI
A 104 GB model on a 48 GB Mac
slotstream keeps the shared part of a mixture-of-experts model in memory and reads each routed expert from SSD only when the model asks for it. One Swift binary. No swap storm.
103.8 GB
Model weights
~12 tok/s
Warm decode
32 GB
Peak memory
112
Acceptance checks
why it exists
Qwen3.8-Flash-Next has 125 billion parameters and 103.8 GB of 4-bit weights. Loading it normally on a 48 GB Mac pushed the machine into 48 GB of swap without producing a token.
Most of those weights are routed experts. Only ten of 512 experts in each layer are active for a token. Loading all of them is the wrong shape of work.
the design
The 3.8 GB dense trunk stays resident. Routed experts are read withpread into a fixed pool of cache slots shared across all 48 layers. The 32 GB n-gram table follows the same rule: fetch the row that is needed, not the whole file.
Memory is planned before generation starts. The server refuses unsupported API semantics instead of silently ignoring them, and greedy output is checked against a version-matched Python reference.
what is measured
- Machine
- M5 Pro, 48 GB unified memory
- Warm decode
- Approximately 12 tokens/second
- Peak memory
- 32 GB with the automatic plan
- Checks
- 81 repository checks and 31 installed-binary checks
Every public number has a measurement behind it. Failed and superseded experiments stay in the repository too. See MEASUREMENTS.md.
the launch
I posted slotstream to Hacker News on September 1, 2026. It reached #1 on Show HN and #8 on the main front page. The discussion later reached 225 points and 106 comments.
The ranking was fun. The useful part was the review. Readers asked for same-machine comparisons, SSD wear and energy measurements, longer-context behavior, thermals, and a clearer account of when 12 tokens per second is useful. Those are better questions than a star count, and they now shape the next measurements.
what it does not prove
The performance numbers come from one M5 Pro with 48 GB of memory. Smaller-memory rows are estimates from that measured curve, not runs on separate Macs. macOS 14 and 15 have installer coverage, not full runtime measurements.
A good Hacker News launch is evidence that the problem resonated. It is not evidence of broad adoption, and it does not replace the measurement record.
I bring the same habit to production-agent work: name the failure, change the system, and measure whether it got better.