After the original article, two kinds of responses came in: the temperature reproduced, but some readers measured a throughput hit from the fix (same TP=2, DeepSeek V4 Flash setup), and our own follow-up run showed single-stream decode dipping too. So we ran the whole thing again — this time with the spin as the only variable, after finding and removing two confounders in the original setup: a clock-cap systemd unit and an intake fan that was still spinning at low speed.
The temperature reproduced — faster than before. Fanless and with CPU clock caps lifted, stock (1-second spin) hit 85°C 90 seconds into load, 92°C at 3.5 minutes, peaking at 95°C at minute 25. The heat source was, again, not the GPU (69°C) but the performance-core cluster where the spin threads live. Under the identical load, the 2ms patch ran at 89.6 → 80.5°C average (−9.1°C), peak 95 → 85°C.
The concurrent-load throughput loss did not reproduce. Over 30 minutes of sustained load, patched actually came out ahead (85.8 vs 92.1 tok/s, within restart variance), and single-stream probes taken during load were equal. We checked actual core clocks, so this is not "stock was throttled into parity" either. The small single-stream loss (−2~6%) at low concurrency remains observable in separate measurements.
Both nodes' thermal zones were recorded every 15 seconds without interruption across the whole experiment (2h 26m): idle baseline → 30 min stock load → cooldown → restart → 30 min patched load → cooldown. Only busy_loop_s differs between the two phases; load, measurement, and conditions are identical.
Fig. 1 · Full-experiment temperatures (7 zones on the head node + worker TSOC). The bold line is head TSOC (which tracks the hottest spot); grays are the other zones. Dashed horizontal rules, top to bottom: forced shutdown (104.8°C), original article's peak (96°C), user-reported shutdown band (87°C). Only during the stock phase does TSOC spike while the GPU and other clusters sit near 70°C. Hover for per-zone values.
| Threshold | stock (1s spin) | patched (2ms) |
|---|---|---|
| 85°C reached | 90 s into load | 12.7 min (grazed once) |
| 90°C reached | 2.3 min | never |
| 92°C reached | 3.5 min | never |
| 95°C reached (peak) | 25.3 min | never (peak 85) |
Measured against the forced-shutdown trip point (104.8°C, read directly from trip_point_0 on the device), stock's peak left 9.8°C of margin, and it entered the 87°C user-reported-shutdown band two minutes into load. Stop the load and it falls back to the 50s within three minutes — the heat source is not a thermally massive board but a CPU that switches off instantly.
| Head node · under load | stock avg | max | patched avg | max | avg delta |
|---|---|---|---|---|---|
| TSOC · SoC package | 89.6 | 95 | 80.5 | 85 | −9.1 |
| TS1P · cluster-1 P-cores | 89.6 | 95 | 80.5 | 85 | −9.1 |
| TUNC · uncore/memory | 73.7 | 77 | 71.2 | 75 | −2.5 |
| TGPU · GPU | 69.3 | 71 | 68.8 | 72 | −0.5 |
| TS0P · cluster-0 P-cores | 69.0 | 75 | 67.2 | 71 | −1.8 |
The worker node measured 77.2 / 77.6°C average for stock/patched — no difference, consistent with the original observation that the worker's dominant wait is NCCL/CUDA collectives, not SpinCondition.
The load is closed-loop. Four client threads each keep exactly one request in flight and fire the next one the moment the previous response finishes — zero think time — so the server always has exactly four concurrent requests. Each request is a chat completion with a ~1.4k-token prompt and max_tokens=400, ignore_eos=true, forcing exactly 400 generated tokens every time regardless of content. Every prompt carries a unique tag to defeat the prefix cache, so each request pays a fresh prefill. Holding this state for 30 minutes is one phase.
| Measured under load | stock | patched |
|---|---|---|
| Head vLLM CPU total (instantaneous, 6s delta) | 396–400% | 211–214% |
| Worker vLLM CPU total | 208% | 206% |
| Requests completed in 30 min (fixed concurrency 4 · 400 generated tokens each) | 386 · 85.8 tok/s | 409 · 92.1 tok/s |
| Single-stream probe during load (median) | 18.1 tok/s | 17.8 tok/s |
The spin's share is exactly head's 400 − 211 ≈ 1.9 cores. The remaining 211% is not spin: it's NCCL/CUDA wait threads (proven by the worker staying flat across stock↔patched) plus real per-step work (scheduler, sampling, detokenizing four streams, draft verification), and it drops to 3% at idle. The original article's "89% after patching" was averaged over real-world load with idle gaps between turns; this run is gapless saturated load — different measurement conditions, same picture.
A fair suspicion, so we checked actual clocks. Under patched load (83°C) the performance cores read 3.9 / 3.978 GHz — full rated speed. We did not sample clocks during the stock phase, but stock's under-load single-stream probes matched patched (18.1 vs 17.8) — impossible if stock had been throttled down. In this temperature band, the two configurations genuinely perform the same under concurrent load, with no throttling involved.
Conditions and commands, so another GB10 cluster can run this as-is.
ghcr.io/anemll/dspark-vllm-gx10:0.1.1 (based on vLLM 0.25.2.dev0+g752a3a504), DeepSeek-V4-Flash-0731 original FP8, TP=2 mp backend, MTP drafter 5 tokensbusy_loop_s: float = 1 ↔ 0.002 in shm_broadcast.py — callers don't pass the argument, so changing the default is sufficient# 1) Toggle the spin (inside the container, before server start)
sed -i 's/busy_loop_s: float = [0-9.]*/busy_loop_s: float = 0.002/' \
/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py
# 2) Identify and record thermal zones (host, every 15s — names are in device/path)
for z in /sys/class/thermal/thermal_zone*; do
echo "$(cat $z/device/path): $(($(cat $z/temp)/1000))°C" # \_TZ_.TSOC, TS0P, TS1P, TGPU, TUNC ...
done
# 3) Load: 4-slot rolling generation (~1.4k-token prompt + 400 tokens ignore_eos per slot,
# repeat for 30 min). Put a unique tag in every prompt to defeat the prefix cache.
# 4) vLLM CPU total (instantaneous): read utime+stime from /proc/<pid>/stat twice, 6s apart,
# for every process whose cmdline contains "vllm"
# 5) Protocol: 10 min idle → phase A 30 min → cool down below 60°C → restart → phase B 30 min
# Swap or repeat phase order to check restart variance (±13–15%)The forced-shutdown threshold can be read from /sys/class/thermal/thermal_zone0/trip_point_0_temp (104800 = 104.8°C on this device). A fanless stock phase enters the 87°C band within two minutes — if you can't watch the temperature, keep the load window short.
Load shape: four slots each looping "~1.4k-token prompt + 400-token ignore_eos generation" back-to-back (unique tags defeat the prefix cache), reasoning effort low. Single-stream probes: three 256-token generations injected at minute 18 while the load keeps running. Temperatures: continuous 15s recording on both nodes, aggregated over the load windows (marker-delimited). CPU: sum of utime+stime deltas (6s) across all processes with "vllm" in cmdline. P-core clocks from cpufreq/policy5·policy15 scaling_cur_freq.
Raw data (full two-node zone records, per-request throughput, event logs) is preserved locally.