I was serving DeepSeek V4 Flash locally and recomputing prompts took
suspiciously long. The cause turned out to be a single startup flag:
--mixed-prefill-quantum, whose default is 128. Raising it to 2048 makes prefill
2.2x faster. I confirmed this with a synthetic probe and again with real
coding agents.
There is a catch. The setting only applies while several requests overlap. This document covers that condition, the size of the effect, the cause, how to pick a value, and what you give up. Everything was measured on a single Apple M3 Ultra (512 GB) running antirez/ds4's ds4-server.
The short version
--mixed-prefill-quantum 2048 to your launch command. Prefill gets 2.2x faster and
the same workload finishes roughly 1.75x sooner.In the source, the value is used conditionally.
bool generation_active = s->active_generations > 0; /* server-wide counter */ return generation_active ? s->mixed_prefill_quantum : 2048;
At the moment prefill runs, at least one request must be emitting tokens for your value to be used. Otherwise the hardcoded 2048 applies. So if you send one request and wait for its answer, prefill and decode never overlap on the timeline and the setting is ignored.
Table 1. Same 5,964-token span, with and without decode load.
| Condition | quantum 128 | quantum 2048 | Verdict |
|---|---|---|---|
| No decode load | 279.3 tok/s | 279.4 / 279.5 / 279.7 | No difference |
| 3 other sessions decoding | 220.3 | 508.9 | 2.31x |
| One decode nearly finished | 230.3 | 520.6 | 2.26x |
The first row settles it. Four measurements agree to the first decimal. The third row shows where the threshold sits: a single decode that logged only 19 lines while the probe ran, almost finished, produces the same result as three busy sessions. The threshold is one session, not three, and how active that one session is does not matter.
I sampled the server metrics every 15 seconds and counted, out of the time prefill was running, how much of it had another request decoding.
Table 2. Overlap by usage pattern.
| Usage | Overlap | Sample |
|---|---|---|
| One agent, one request at a time | 1.4% | 151 min, 218 buckets |
| Three agents at once | 20 to 37% | 4 runs |
It turns on when you open two or more terminals, or fan out subagents in parallel. The question is not how many people are using the server but whether concurrent requests are in different phases at the same instant.
Figure 1. Everything else held fixed, only quantum varied. By 2048 it already reaches 91% of the ceiling.
The dashed line is the 559 tok/s ceiling measured with no yielding at all. Log-scaled x axis. Three sessions decoding, 20k-token cold prefill, spread under 0.2% across three repeats.
Table 3. Full synthetic sweep.
| quantum | Prefill | Of ceiling |
|---|---|---|
| 128 (default) | 220.3 tok/s | 39% |
| 512 | 332.9 | 60% |
| 1024 | 426.0 | 76% |
| 2048 (recommended) | 508.2 | 91% |
| 4096 | 542.3 | 97% |
| 8192 and up | 552 to 559 | 99 to 100% |
The numbers above come from prompts I generated myself, with dummy sessions faking the decode load. I checked whether the result survives real work. Three coding agents with file-read and bash tools each took a different analysis task over a real Rust codebase (68 files, 28k lines). Context grew on its own to 178k tokens and tool calls ranged from 44 to 82.
Figure 2. Prefill speed during real agent work. Reversing the order does not move the two groups into each other.
The denominator is the time prefill was actually running, taken as the union of request intervals. Since the amount of work differs per run, this is what I compare instead of total elapsed time.
Table 4. Four real agent runs.
| Run | Requests | New prefill | Compute time | Prefill speed | Tool calls |
|---|---|---|---|---|---|
| Run 1, q2048 | 47 | 139,599 | 485 s | 287.7 | 44 |
| Run 2, q2048 | 48 | 190,262 | 722 s | 263.5 | 45 |
| Run 1, q128 | 55 | 214,270 | 1,735 s | 123.5 | 52 |
| Run 2, q128 | 87 | 252,263 | 2,025 s | 124.6 | 82 |
Pooled, that is 273.2 against 124.1, a factor of 2.20, which lands where the synthetic probe put it (2.31).
The denominator is the union of request intervals. With three agents, prefills sometimes run concurrently, and simply summing each request's duration counts that stretch twice. The q128 side overlaps more (up to 1.14x double counting), so naive summing makes the slower side look slower than it is.
The amount of work also differs per run. Tool calls span 44 to 82 and new prefill spans 140k to 252k tokens. That is the agent taking a different path each time, not something quantum caused, which is why total elapsed time is not comparable directly.
Measuring the two phases separately makes it clear what this setting touches and what it leaves alone. Each rate is divided by the time that phase was actually running.
Table 5. Speed and wall-clock occupancy by phase.
| Prefill speed | Prefill occupancy | Decode speed | Decode occupancy | |
|---|---|---|---|---|
| quantum 2048 | 273.2 t/s | 52% | 29.2 t/s | 59% |
| quantum 128 | 124.1 | 90% | 21.6 | 48% |
| Ratio | 2.20x | 1.35x |
The 1.35x on decode is not this setting making decode faster. The longer prefill runs, the more often decode lands on top of it, and when they run together they share the GPU, so decode slows down. Overlap was 20 to 27% under q2048 and 32 to 37% under q128. It is a side effect of slow prefill, not a separate benefit.
The causation runs one way.
Prefill is 2.2x slower → the same amount of computation takes 2.2x longer → that time eats the wall clock (occupancy 52% to 90%) → decode gets fewer turns and slows down where they overlap → the job takes longer.
Prefill is the only starting point; every other number is downstream of it.
The numbers above aggregate per request. Here is the same work sampled every 15 seconds on the server side.
Figure 3. Four runs of the same task, taking anywhere from 17 to 44 minutes.
For each run, prefill is on top and decode below. The x axis is shared across all four. Grey shading marks stretches where both were running.
Three things stand out.
Prefill being 2.20x faster does not make the job finish 2.20x sooner. The wall clock also contains decode time, and that part is out of this setting's reach.
Table 6. Converted to identical work (165k new prefill, 20k decode) using the rates measured in each condition.
| Setting | Prefill | Decode | Total |
|---|---|---|---|
| quantum 2048 | 604 s | 685 s | 1,289 s |
| quantum 128 | 1,330 s | 926 s | 2,256 s |
| Ratio | 2.20x | 1.35x | 1.75x |
This assumes the two phases run back to back. In practice they overlap 20 to 37% of the time, so real durations are shorter than this. No pair of runs matched both prefill and decode workload at once, so I could not narrow it further.
What this setting is worth scales with how much of your work is prefill. For a coding agent that keeps pushing tool output back in, where reading vastly outweighs writing, it is large. For short prompts with long answers, it is close to nothing.
Figure 4. Every individual request from the real runs. The x axis is the context position where that request started.
Dot size is the number of new tokens; the thick lines are per-bucket medians. The two groups stay separated across the whole range, and both slope down to the right.
Table 7. Ratio by context position.
| Context | q2048 | q128 | Ratio |
|---|---|---|---|
| 0 to 30k | 296 tok/s | 138 | 2.14x |
| 30k to 60k | 282 | 119 | 2.37x |
| 60k to 100k | 254 | 120 | 2.12x |
| Beyond 100k | 225 | 100 | 2.25x |
Absolute speed falls as context grows, but the ratio barely moves, staying between 2.1 and 2.4.
This differs when requests do not overlap. In the synthetic setup, where a single probe computes while the others only decode, the ratio shrank from 2.31x at 20k context to 1.66x at 177k. With real agents, prefills also collide with each other, and that contention appears to punish the smaller quantum enough to offset what long context would otherwise take away. I did not isolate that effect on its own.
The cause is that this model is a mixture of experts. First I ruled out the obvious suspicion, that decode is simply stealing prefill's share. Dividing each phase by its standalone maximum gives the time each one occupies.
Table 8. What prefill should reach once decode's time is subtracted.
| quantum | Decode occupancy | Expected | Measured | Gap |
|---|---|---|---|---|
| 2048 | 9.3% | 507 | 509 | None |
| 128 | 15.2% | 474 | 220 | Less than half |
At 2048, prediction and measurement agree. That is textbook time slicing. At 128, subtracting every second decode used still leaves 474 expected against 220 measured. Decode did not take it. Prefill fails to reach its own speed during its own turn.
DeepSeek V4 Flash routes each token to 6 of 256 experts. The expected number of distinct
experts a chunk touches is 256 × (1 - (1 - 6/256)^Q).
Figure 5. Per-token cost of pure prefill with the decode step subtracted. It flattens at 48 tokens per expert.
Tokens per expert = Q × 6 / 256. Log-scaled x axis.
Table 9. How each chunk size uses the experts.
| quantum | Distinct experts | Tokens per expert | Cost per token | vs 2048 |
|---|---|---|---|---|
| 128 | 243.7 / 256 (95%) | 3.0 | 3.22 ms | 1.71x |
| 512 | 256 | 12 | 2.67 | 1.42x |
| 1024 | 256 | 24 | 2.18 | 1.16x |
| 2048 | 256 | 48 | 1.88 | 1.00 |
| 4096 and up | 256 | 96 and up | 1.79 to 1.80 | 0.95 to 0.96 |
This is the heart of it. Even at quantum 128 the server already reads 95% of the experts. It moves nearly as many weights as 2048 does, then performs a matrix multiply of three rows against each of them. At 2048 the same weights are read and 48 rows are computed. Pulling in all the weights and then barely computing leaves arithmetic intensity on the floor and the compute units idle. 128 is not faster because it reads less; it is slower because it reads the same and does less with it, which is where the 1.7x per-token cost comes from.
The cost curve flattening exactly at 48 tokens per expert fits the same reading. Past that point the per-expert matrix multiply is large enough that efficiency saturates. That happens to land on quantum 2048.
Hardware was ruled out. I wrote Metal kernels to measure it directly: sequential read hit 690 GB/s (84% of the published 819), reading 300 random 13.3 MB chunks to mimic scattered expert access gave 671 GB/s, and adding MXFP4 unpacking with scale multiply and accumulate still gave 672 GB/s. Neither bandwidth nor dequantization is the bottleneck.
2048 is a good value not because it is a power of two but because it falls out of the model's shape. Efficiency is set by tokens per expert, so pick a target and work backwards.
tokens per expert = quantum × n_expert_used / n_expert
backwards: quantum = target × n_expert / n_expert_used
= target × 256 / 6 (DeepSeek V4 Flash)
The same formula carries to other MoE models. For one with 128 experts and 8 per token,
48 tokens per expert works out to 48 × 128 / 8 = 768.
There are two axes to weigh. Going up pushes per-token cost toward its floor of 1.79 ms, but 2048 is already within 5% of that floor, so there is little left to gain. Going down shortens the gap between tokens for other sessions while prefill runs: one token every 4 seconds at 2048, every 2.4 seconds at 1024.
Table 10. Candidates.
| quantum | Tokens per expert | Cost per token | Gap between tokens | |
|---|---|---|---|---|
| 128 (default) | 3 | 3.22 ms | 0.58 s | Too small |
| 1024 | 24 | 2.18 | 2.40 s | When a human watches the stream |
| 2048 | 48 | 1.88 | 4.02 s | Recommended |
| 4096 and up | 96 and up | 1.79 | 7.6 s and up | Decode stops entirely |
Restrict candidates to powers of two. Using whatever the formula returns (36 tokens gives 1536, say) costs you, because it collides with the engine's graph boundary. Use the formula to decide which power of two sits nearest your target, then round to it. The reason is in the next section.
There is one cost. While prefill runs, decode in other sessions stalls more decisively, because a larger quantum yields fewer times.
Figure 6. Raising quantum lifts prefill and lowers decode while it runs.
The grey band is where a human perceives a stall. Normal decode is 11 to 12 tok/s per session, so this whole range reads as stalled at any setting.
Whether that is a real cost is worth working through. Against a normal 11 to 12 tok/s, both 1.77 and 1.08 look stopped to a person. And the stall shrinks from 112 seconds to 49. Over the same 112 seconds of wall clock, the three neighbouring sessions produce roughly 2,300 tokens instead of roughly 600. What decides the outcome is not how much trickles out during the stall but how quickly the stall ends.
Concurrent requests were not starved. With three other sessions decoding, I fired three 20k cold prefills at once and all three finished within 0.5 seconds of each other at either setting (q128: 447.5 / 447.7 / 448.0 s; q2048: 152.8 / 153.0 / 153.2 s). The round robin over prefill slots keeps them fair regardless of quantum.
The engine cuts GPU launches at prefill_cap boundaries, 4096 by default.
const uint32_t mod = pos0 % g->prefill_cap;
if (mod != 0) {
const uint32_t to_boundary = g->prefill_cap - mod;
if (to_boundary < local_cap) local_cap = to_boundary;
}
If quantum does not divide 4096 evenly, every boundary splits the request in two. That is why an in-between value like 1536 loses: it sits 7 to 8% below the log interpolation between 1024 (427) and 2048 (509).
There is a condition attached, though. A power of two escapes fragmentation only when the prompt is computed from zero. Real requests continue from the end of previous context, and that position is arbitrary. Of 23 real requests on this server, only 2 (9%) started at a multiple of 4096.
start is a multiple of 4096 2048, 2048, 2048, 2048, ... start % 4096 = 1539 2048, 509, 1539, 2048, 509, 1539, ... (never self-aligns)
I measured what that costs. Narrowing the boundary with --prefill-chunk changes
only the number of GPU launches and holds everything else fixed.
Table 11. Varying only launch count, over an 8,478-token span at 82k context.
| Boundary | GPU launches | Mean chunk | Speed | Delta |
|---|---|---|---|---|
| 4096 (default) | 5 | 1,695 | 367.0 t/s | baseline |
| 3072 | 7 | 1,211 | 354.5 | -3.4% |
| 2560 | 8 | 1,059 | 348.3 | -5.1% |
Even at 1.6x the launches, the loss is 5%. Fragmentation is real but small. You can remove it
by raising --prefill-chunk, but context buffers grow by
26 GiB at 8192 and 74 GiB at 16384. That is not a price worth paying for 5%.
And raising the boundary does not enlarge the chunks anyway: the server hands over at most one
quantum at a time, so the boundary only moves where the cuts land.
Add one line to your launch script.
exec /opt/ds4/ds4-server \ ... --batched-session 6 \ --mixed-prefill-quantum 2048
Confirm it in the server's startup log. If the same flag appears twice the last one wins, so if the value seems to have no effect, check that first.
ds4-server: batched mode enabled resident_sessions=6 prefill_quantum=2048 mixed_prefill_quantum=2048
To revert, delete the line and restart. The change leaves no state behind.
I also checked this against public discussion. I found no writing on this particular flag and
no stated rationale for the 128 default, but the surrounding facts line up. A public benchmark
of the same model on the same M3 Ultra 512 GB reports 560 tok/s at 2k input, essentially
identical to the 559 ceiling here. The llama.cpp community has converged on
ubatch 2048 as the fastest setting for prefill. And the Sarathi paper reports that
on a dense model, chunk 128 makes prefill more than 2x slower while 256 and 512 stay within 20%
and 10%. Different model, different GPU, and it collapses in the same place by a similar margin.
There is a camp on the other side. vLLM's chunked prefill deliberately splits finely to keep generating sessions alive, and 128 is a value from that camp. The catch is that the chunk sizes that literature actually uses are nearer 8K, not 128.
Table 12. Status of each claim.
| Claim | Basis | Status |
|---|---|---|
| The 128 default reaches 39% of the ceiling | 8 values, 3 repeats, spread under 0.2% | Confirmed |
| The ceiling is about 559 tok/s | Measured with no yielding; matches a public benchmark at 560 | Confirmed |
| Without decode, the setting does nothing | Same span 4 times: 279.3 vs 279.4 / 279.5 / 279.7 | Confirmed |
| A single decode is enough to activate it | 230 vs 521 with one nearly finished decode | Confirmed |
| 2.20x on real agent work | 2 repeats with order reversed, union-of-intervals basis | Confirmed |
| End to end about 1.75x | Converted to equal work, assuming the phases run back to back | Estimate |
| The cause is MoE tokens per expert | Bandwidth measurements rule out alternatives; saturates at 48 tokens | Confirmed |
| Fragmentation costs about 5% | 348.3 vs 367.0 at 1.6x the launches | Confirmed |
| 2048 is the right operating point | 91% of prefill against the decode cost | Judgement |
| Overlap is what keeps the ratio up at long context | Consistent with the data; no isolating experiment | Hypothesis |
Synthetic probe: with three sessions continuously writing, one new 20k-token prompt is fired. A unique string at the front keeps the disk KV cache from attaching. The server was restarted for each condition (7 to 9 seconds, since the model stays in page cache and is not re-read) with only the flag changed.
Real agents: three coding agents with file-read and bash tools, started 15 seconds apart. Starting them simultaneously makes them all prefill and then all decode, which never produces the situation being measured. I wrote segment markers directly into the server log and counted the requests between them.
Verification: for each condition, the value actually applied is read back from the startup log and the condition is marked invalid if it differs from what was requested. Without that check, one run was measured with the setting silently ignored, because the launch script specified the same flag twice.
Equal work: the synthetic runs pinned assistant turns to a fixed string and
forced output length with max_tokens, matching the workload character for character,
and each condition used an empty disk KV directory so it could not inherit the previous
condition's checkpoints. That is not possible with real agents, hence the comparison by effective
throughput.