One default was halving prefill in ds4-server

I was serving DeepSeek V4 Flash locally and recomputing prompts took suspiciously long. The cause turned out to be a single startup flag: --mixed-prefill-quantum, whose default is 128. Raising it to 2048 makes prefill 2.2x faster. I confirmed this with a synthetic probe and again with real coding agents.

There is a catch. The setting only applies while several requests overlap. This document covers that condition, the size of the effect, the cause, how to pick a value, and what you give up. Everything was measured on a single Apple M3 Ultra (512 GB) running antirez/ds4's ds4-server.

The short version

1. When does this setting apply?

In the source, the value is used conditionally.

bool generation_active = s->active_generations > 0;   /* server-wide counter */
return generation_active ? s->mixed_prefill_quantum : 2048;

At the moment prefill runs, at least one request must be emitting tokens for your value to be used. Otherwise the hardcoded 2048 applies. So if you send one request and wait for its answer, prefill and decode never overlap on the timeline and the setting is ignored.

Table 1. Same 5,964-token span, with and without decode load.

Conditionquantum 128quantum 2048Verdict
No decode load279.3 tok/s279.4 / 279.5 / 279.7No difference
3 other sessions decoding220.3508.92.31x
One decode nearly finished230.3520.62.26x

The first row settles it. Four measurements agree to the first decimal. The third row shows where the threshold sits: a single decode that logged only 19 lines while the probe ran, almost finished, produces the same result as three busy sessions. The threshold is one session, not three, and how active that one session is does not matter.

How often do they actually overlap?

I sampled the server metrics every 15 seconds and counted, out of the time prefill was running, how much of it had another request decoding.

Table 2. Overlap by usage pattern.

UsageOverlapSample
One agent, one request at a time1.4%151 min, 218 buckets
Three agents at once20 to 37%4 runs

It turns on when you open two or more terminals, or fan out subagents in parallel. The question is not how many people are using the server but whether concurrent requests are in different phases at the same instant.

2. How much faster?

Figure 1. Everything else held fixed, only quantum varied. By 2048 it already reaches 91% of the ceiling.

The dashed line is the 559 tok/s ceiling measured with no yielding at all. Log-scaled x axis. Three sessions decoding, 20k-token cold prefill, spread under 0.2% across three repeats.

Table 3. Full synthetic sweep.

quantumPrefillOf ceiling
128 (default)220.3 tok/s39%
512332.960%
1024426.076%
2048 (recommended)508.291%
4096542.397%
8192 and up552 to 55999 to 100%

Measuring again with real agents

The numbers above come from prompts I generated myself, with dummy sessions faking the decode load. I checked whether the result survives real work. Three coding agents with file-read and bash tools each took a different analysis task over a real Rust codebase (68 files, 28k lines). Context grew on its own to 178k tokens and tool calls ranged from 44 to 82.

Figure 2. Prefill speed during real agent work. Reversing the order does not move the two groups into each other.

The denominator is the time prefill was actually running, taken as the union of request intervals. Since the amount of work differs per run, this is what I compare instead of total elapsed time.

Table 4. Four real agent runs.

RunRequestsNew prefillCompute timePrefill speedTool calls
Run 1, q204847139,599485 s287.744
Run 2, q204848190,262722 s263.545
Run 1, q12855214,2701,735 s123.552
Run 2, q12887252,2632,025 s124.682

Pooled, that is 273.2 against 124.1, a factor of 2.20, which lands where the synthetic probe put it (2.31).

The denominator is the union of request intervals. With three agents, prefills sometimes run concurrently, and simply summing each request's duration counts that stretch twice. The q128 side overlaps more (up to 1.14x double counting), so naive summing makes the slower side look slower than it is.

The amount of work also differs per run. Tool calls span 44 to 82 and new prefill spans 140k to 252k tokens. That is the agent taking a different path each time, not something quantum caused, which is why total elapsed time is not comparable directly.

Only prefill gets slower

Measuring the two phases separately makes it clear what this setting touches and what it leaves alone. Each rate is divided by the time that phase was actually running.

Table 5. Speed and wall-clock occupancy by phase.

Prefill speedPrefill occupancyDecode speedDecode occupancy
quantum 2048273.2 t/s52%29.2 t/s59%
quantum 128124.190%21.648%
Ratio2.20x1.35x

The 1.35x on decode is not this setting making decode faster. The longer prefill runs, the more often decode lands on top of it, and when they run together they share the GPU, so decode slows down. Overlap was 20 to 27% under q2048 and 32 to 37% under q128. It is a side effect of slow prefill, not a separate benefit.

The causation runs one way.

Prefill is 2.2x slower → the same amount of computation takes 2.2x longer → that time eats the wall clock (occupancy 52% to 90%) → decode gets fewer turns and slows down where they overlap → the job takes longer.

Prefill is the only starting point; every other number is downstream of it.

The same runs, seen from server metrics

The numbers above aggregate per request. Here is the same work sampled every 15 seconds on the server side.

Figure 3. Four runs of the same task, taking anywhere from 17 to 44 minutes.

For each run, prefill is on top and decode below. The x axis is shared across all four. Grey shading marks stretches where both were running.

Three things stand out.

The wall clock does not shrink by the same factor

Prefill being 2.20x faster does not make the job finish 2.20x sooner. The wall clock also contains decode time, and that part is out of this setting's reach.

Table 6. Converted to identical work (165k new prefill, 20k decode) using the rates measured in each condition.

SettingPrefillDecodeTotal
quantum 2048604 s685 s1,289 s
quantum 1281,330 s926 s2,256 s
Ratio2.20x1.35x1.75x

This assumes the two phases run back to back. In practice they overlap 20 to 37% of the time, so real durations are shorter than this. No pair of runs matched both prefill and decode workload at once, so I could not narrow it further.

What this setting is worth scales with how much of your work is prefill. For a coding agent that keeps pushing tool output back in, where reading vastly outweighs writing, it is large. For short prompts with long answers, it is close to nothing.

As context grows

Figure 4. Every individual request from the real runs. The x axis is the context position where that request started.

Dot size is the number of new tokens; the thick lines are per-bucket medians. The two groups stay separated across the whole range, and both slope down to the right.

Table 7. Ratio by context position.

Contextq2048q128Ratio
0 to 30k296 tok/s1382.14x
30k to 60k2821192.37x
60k to 100k2541202.12x
Beyond 100k2251002.25x

Absolute speed falls as context grows, but the ratio barely moves, staying between 2.1 and 2.4.

This differs when requests do not overlap. In the synthetic setup, where a single probe computes while the others only decode, the ratio shrank from 2.31x at 20k context to 1.66x at 177k. With real agents, prefills also collide with each other, and that contention appears to punish the smaller quantum enough to offset what long context would otherwise take away. I did not isolate that effect on its own.

3. Why does it happen?

The cause is that this model is a mixture of experts. First I ruled out the obvious suspicion, that decode is simply stealing prefill's share. Dividing each phase by its standalone maximum gives the time each one occupies.

Table 8. What prefill should reach once decode's time is subtracted.

quantumDecode occupancyExpectedMeasuredGap
20489.3%507509None
12815.2%474220Less than half

At 2048, prediction and measurement agree. That is textbook time slicing. At 128, subtracting every second decode used still leaves 474 expected against 220 measured. Decode did not take it. Prefill fails to reach its own speed during its own turn.

Tokens per expert

DeepSeek V4 Flash routes each token to 6 of 256 experts. The expected number of distinct experts a chunk touches is 256 × (1 - (1 - 6/256)^Q).

Figure 5. Per-token cost of pure prefill with the decode step subtracted. It flattens at 48 tokens per expert.

Tokens per expert = Q × 6 / 256. Log-scaled x axis.

Table 9. How each chunk size uses the experts.

quantumDistinct expertsTokens per expertCost per tokenvs 2048
128243.7 / 256 (95%)3.03.22 ms1.71x
512256122.671.42x
1024256242.181.16x
2048256481.881.00
4096 and up25696 and up1.79 to 1.800.95 to 0.96

This is the heart of it. Even at quantum 128 the server already reads 95% of the experts. It moves nearly as many weights as 2048 does, then performs a matrix multiply of three rows against each of them. At 2048 the same weights are read and 48 rows are computed. Pulling in all the weights and then barely computing leaves arithmetic intensity on the floor and the compute units idle. 128 is not faster because it reads less; it is slower because it reads the same and does less with it, which is where the 1.7x per-token cost comes from.

The cost curve flattening exactly at 48 tokens per expert fits the same reading. Past that point the per-expert matrix multiply is large enough that efficiency saturates. That happens to land on quantum 2048.

Hardware was ruled out. I wrote Metal kernels to measure it directly: sequential read hit 690 GB/s (84% of the published 819), reading 300 random 13.3 MB chunks to mimic scattered expert access gave 671 GB/s, and adding MXFP4 unpacking with scale multiply and accumulate still gave 672 GB/s. Neither bandwidth nor dequantization is the bottleneck.

4. Which value should you use?

2048 is a good value not because it is a power of two but because it falls out of the model's shape. Efficiency is set by tokens per expert, so pick a target and work backwards.

tokens per expert = quantum × n_expert_used / n_expert
backwards:  quantum = target × n_expert / n_expert_used
                    = target × 256 / 6        (DeepSeek V4 Flash)

The same formula carries to other MoE models. For one with 128 experts and 8 per token, 48 tokens per expert works out to 48 × 128 / 8 = 768.

There are two axes to weigh. Going up pushes per-token cost toward its floor of 1.79 ms, but 2048 is already within 5% of that floor, so there is little left to gain. Going down shortens the gap between tokens for other sessions while prefill runs: one token every 4 seconds at 2048, every 2.4 seconds at 1024.

Table 10. Candidates.

quantumTokens per expertCost per tokenGap between tokens
128 (default)33.22 ms0.58 sToo small
1024242.182.40 sWhen a human watches the stream
2048481.884.02 sRecommended
4096 and up96 and up1.797.6 s and upDecode stops entirely

Restrict candidates to powers of two. Using whatever the formula returns (36 tokens gives 1536, say) costs you, because it collides with the engine's graph boundary. Use the formula to decide which power of two sits nearest your target, then round to it. The reason is in the next section.

5. What do you give up?

There is one cost. While prefill runs, decode in other sessions stalls more decisively, because a larger quantum yields fewer times.

Figure 6. Raising quantum lifts prefill and lowers decode while it runs.

The grey band is where a human perceives a stall. Normal decode is 11 to 12 tok/s per session, so this whole range reads as stalled at any setting.

Whether that is a real cost is worth working through. Against a normal 11 to 12 tok/s, both 1.77 and 1.08 look stopped to a person. And the stall shrinks from 112 seconds to 49. Over the same 112 seconds of wall clock, the three neighbouring sessions produce roughly 2,300 tokens instead of roughly 600. What decides the outcome is not how much trickles out during the stall but how quickly the stall ends.

Concurrent requests were not starved. With three other sessions decoding, I fired three 20k cold prefills at once and all three finished within 0.5 seconds of each other at either setting (q128: 447.5 / 447.7 / 448.0 s; q2048: 152.8 / 153.0 / 153.2 s). The round robin over prefill slots keeps them fair regardless of quantum.

Fragments at the 4096 boundary

The engine cuts GPU launches at prefill_cap boundaries, 4096 by default.

const uint32_t mod = pos0 % g->prefill_cap;
if (mod != 0) {
    const uint32_t to_boundary = g->prefill_cap - mod;
    if (to_boundary < local_cap) local_cap = to_boundary;
}

If quantum does not divide 4096 evenly, every boundary splits the request in two. That is why an in-between value like 1536 loses: it sits 7 to 8% below the log interpolation between 1024 (427) and 2048 (509).

There is a condition attached, though. A power of two escapes fragmentation only when the prompt is computed from zero. Real requests continue from the end of previous context, and that position is arbitrary. Of 23 real requests on this server, only 2 (9%) started at a multiple of 4096.

start is a multiple of 4096   2048, 2048, 2048, 2048, ...
start % 4096 = 1539           2048,  509, 1539, 2048, 509, 1539, ...   (never self-aligns)

I measured what that costs. Narrowing the boundary with --prefill-chunk changes only the number of GPU launches and holds everything else fixed.

Table 11. Varying only launch count, over an 8,478-token span at 82k context.

BoundaryGPU launchesMean chunkSpeedDelta
4096 (default)51,695367.0 t/sbaseline
307271,211354.5-3.4%
256081,059348.3-5.1%

Even at 1.6x the launches, the loss is 5%. Fragmentation is real but small. You can remove it by raising --prefill-chunk, but context buffers grow by 26 GiB at 8192 and 74 GiB at 16384. That is not a price worth paying for 5%. And raising the boundary does not enlarge the chunks anyway: the server hands over at most one quantum at a time, so the boundary only moves where the cuts land.

6. Applying it

Add one line to your launch script.

exec /opt/ds4/ds4-server \
  ...
  --batched-session 6 \
  --mixed-prefill-quantum 2048

Confirm it in the server's startup log. If the same flag appears twice the last one wins, so if the value seems to have no effect, check that first.

ds4-server: batched mode enabled resident_sessions=6 prefill_quantum=2048 mixed_prefill_quantum=2048

To revert, delete the line and restart. The change leaves no state behind.

Before you apply it

7. Evidence

I also checked this against public discussion. I found no writing on this particular flag and no stated rationale for the 128 default, but the surrounding facts line up. A public benchmark of the same model on the same M3 Ultra 512 GB reports 560 tok/s at 2k input, essentially identical to the 559 ceiling here. The llama.cpp community has converged on ubatch 2048 as the fastest setting for prefill. And the Sarathi paper reports that on a dense model, chunk 128 makes prefill more than 2x slower while 256 and 512 stay within 20% and 10%. Different model, different GPU, and it collapses in the same place by a similar margin.

There is a camp on the other side. vLLM's chunked prefill deliberately splits finely to keep generating sessions alive, and 128 is a value from that camp. The catch is that the chunk sizes that literature actually uses are nearer 8K, not 128.

Table 12. Status of each claim.

ClaimBasisStatus
The 128 default reaches 39% of the ceiling8 values, 3 repeats, spread under 0.2%Confirmed
The ceiling is about 559 tok/sMeasured with no yielding; matches a public benchmark at 560Confirmed
Without decode, the setting does nothingSame span 4 times: 279.3 vs 279.4 / 279.5 / 279.7Confirmed
A single decode is enough to activate it230 vs 521 with one nearly finished decodeConfirmed
2.20x on real agent work2 repeats with order reversed, union-of-intervals basisConfirmed
End to end about 1.75xConverted to equal work, assuming the phases run back to backEstimate
The cause is MoE tokens per expertBandwidth measurements rule out alternatives; saturates at 48 tokensConfirmed
Fragmentation costs about 5%348.3 vs 367.0 at 1.6x the launchesConfirmed
2048 is the right operating point91% of prefill against the decode costJudgement
Overlap is what keeps the ratio up at long contextConsistent with the data; no isolating experimentHypothesis
Appendix: how it was measured

Synthetic probe: with three sessions continuously writing, one new 20k-token prompt is fired. A unique string at the front keeps the disk KV cache from attaching. The server was restarted for each condition (7 to 9 seconds, since the model stays in page cache and is not re-read) with only the flag changed.

Real agents: three coding agents with file-read and bash tools, started 15 seconds apart. Starting them simultaneously makes them all prefill and then all decode, which never produces the situation being measured. I wrote segment markers directly into the server log and counted the requests between them.

Verification: for each condition, the value actually applied is read back from the startup log and the condition is marked invalid if it differs from what was requested. Without that check, one run was measured with the setting silently ignored, because the launch script specified the same flag twice.

Equal work: the synthetic runs pinned assistant turns to a fixed string and forced output length with max_tokens, matching the workload character for character, and each condition used an empty disk KV directory so it could not inherit the previous condition's checkpoints. That is not possible with real agents, hence the comparison by effective throughput.


Measured on a single Apple M3 Ultra 512 GB running antirez/ds4’s ds4-server with DeepSeek V4 Flash MXFP4. A Korean version is at moe-prefill-quantum.