DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime.
https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning
Published on blog.n.ichol.ai
Where the last edition left off
Last time I grafted a CoLaR head (Compressed Latent Reasoning) onto DeepSeek Flash v4. The work had all the makings of a demo. A head that lets the model think in latent space. A learned stop head that decides when it has thought enough. A riddle that showed why plain autoregressive generation regurgitates cached answers instead of reasoning.
There was one honest problem though: the head was an adapter. Something bolted on the side. To serve it you had to assemble the base model, the head, the stop criterion, and a custom runtime by hand, then hope the pieces fit. The weights lived in one place. The inference machinery in another. "Here's how you run it" was a jump-through-hoops story.
This edition closes that gap. The work is now a complete, self-contained model. Every weight needed to serve it ships in one repository. The backbone is quantized down to NVFP4 so it fits on real silicon. And the latent loop is driven by a proper, benchmarked serving runtime.
Model: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning
The big change: this is not an adapter anymore
The whole point of this edition is packaging. The old release was a head you had to attach. The new one is a model.
Every weight needed to serve it now sits in one HuggingFace repo:
The DeepSeek-V4-Flash-0731 backbone, quantized to NVFP4 (group size 16) on the routed MoE experts. Attention, shared experts, LM head and draft block stay at higher precision. Roughly 79 GiB of weights per GPU at TP=2 (158–164 GiB total).
The DSpark draft block (3 layers), preserved from the source, so speculative decoding ships in the box.
The trained latent reasoning head. 35.7M params, loaded from a single latent_reasoning_head.safetensors (~152 MB).
Because the weights are complete, the model card can finally report a real benchmark instead of "more benchmarks soon."
Actually writing down the numbers this time
The previous post ended with a half-promise: "full benchmarking to come if I find time." I found time.
BBH (BIG-Bench Hard), cot_zeroshot, 27 subtasks: aggregate 0.880 ± 0.008. Measured with lm-evaluation-harness 0.4.12 against an OpenAI-compatible endpoint. Thinking enabled. 50 items per subtask, 1350 items total.
| Subtask |
Score |
Subtask |
Score |
| tracking_shuffled_objects_three_objects |
1.00 |
date_understanding |
0.92 |
| tracking_shuffled_objects_five_objects |
1.00 |
sports_understanding |
0.88 |
| tracking_shuffled_objects_seven_objects |
1.00 |
logical_deduction_five_objects |
0.88 |
| penguins_in_a_table |
1.00 |
web_of_lies |
0.86 |
| formal_fallacies |
1.00 |
snarks |
0.84 |
| boolean_expressions |
1.00 |
ruin_names |
0.84 |
| word_sorting |
0.98 |
movie_recommendation |
0.84 |
| temporal_sequences |
0.98 |
salient_translation_error_detection |
0.76 |
| object_counting |
0.98 |
geometric_shapes |
0.74 |
| navigate |
0.98 |
causal_judgement |
0.66 |
| logical_deduction_three_objects |
0.98 |
disambiguation_qa |
0.58 |
| reasoning_about_colored_objects |
0.96 |
dyck_languages |
0.26 |
| hyperbaton |
0.96 |
|
|
| multistep_arithmetic_two |
0.94 |
|
|
| logical_deduction_seven_objects |
0.94 |
|
|
The pattern is exactly what you would hope for from a latent reasoning model. It is strongest where reasoning means multi-step state tracking. tracking_shuffled_objects, boolean_expressions, formal_fallacies, penguins_in_a_table all hit 1.00. It is weakest on the mechanical, syntax-heavy jobs. dyck_languages (bracket matching) sits at 0.26, the clear outlier. That is a genuine weakness, not a measurement artifact.
Two honest notes on reading the table:
Read flexible-extract, not strict-match. BBH's strict-match regexes for the literal phrase The answer is X. This model does not emit that phrase, because it reasons in latent space. Its near-zero strict-match score is an answer-formatting artifact, not a reasoning failure.
Per-subtask values carry about ±0.05–0.07 at 50 items each. The aggregate of 0.880 is the reliable number.
Why the head looks different now
The architecture picks up where the original CoLaR idea left off, but it has a proper home in the model now. A small head reads the backbone's layer-35 hidden state, projects it into a 1024-d latent, and decodes it back into the residual stream. One latent step stands in for several reasoning tokens (a recorded compression_factor of 6). A learned stop head self-terminates the loop at a variable, content-dependent depth.
layer 35 hidden (4096-d)
|
v LayerNorm
+--------- ReasoningCompressionHead ----------+
| Linear 4096 -> 2048 . SiLU |
| Linear 2048 -> 2048 . SiLU |
| Linear 2048 -> 2048 -> [mu, log_sigma] |
| |
| stop_head: |
| Linear 4096 -> 1024 . SiLU |
| Linear 1024 -> 1 | -> end-of-reasoning
+---------------------------------------------+
| mu (1024-d latent)
v LayerNorm
+-------------- LatentDecoder ----------------+
| Linear 1024 -> 2048 . SiLU |
| Linear 2048 -> 2048 . SiLU |
| Linear 2048 -> 4096 |
+---------------------------------------------+
|
v written back into the residual stream
DeepSeek-V4-Flash-0731 backbone (frozen, NVFP4)
| Config |
Value |
| hidden_size |
4096 |
| latent_dim |
1024 |
| mlp_dim |
2048 |
| source_layer / target_layer |
35 / 42 |
| activation |
SiLU |
| learned stop head |
yes |
| head + decoder params |
35.7M (float32) |
| backbone layers |
43 |
One detail worth mentioning: the head is a variational compression. It predicts [mu, log_sigma] and clamps log_sigma. The decoder redistributes the latent back into the 4096-d stream. The whole thing is a single flat tensor dict distinguished by key prefix. The geometry lives in the checkpoint's own metadata, so the serving runtime needs zero configuration; it reads the head's shape straight from the file. target_proj, a frozen projection that defined the regression target during training, is included for completeness but is not used at inference.
The serving runtime is now a real thing
The old runtime was a set of env vars and headers bolted onto a fork, with the model weights kept separate. This time the runtime is released in its own right, split cleanly into two pieces you install once:
| Repository |
What it is |
nickmitchko/ds4-reasoning-addon |
The latent-reasoning addon: the closed-loop driver, serve script, pinned requirements and GPU-sizing guide. Start here. |
nickmitchko/vllm-ds4-sm120 |
The DS4 vLLM fork it runs on (branch ds4-sm120-preview-dev). Required. Upstream vLLM cannot serve this model. |
Installing the addon does nothing by itself. The plugin registers but stays dormant until VLLM_DS4_REASONING_CKPT is set. That is a deliberate safety choice; it is safe to leave installed. The supported entrypoint is a single script that sets all the measured-good defaults for you:
# 1. the engine (a full build takes a while)
git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120
git checkout ds4-sm120-preview-dev
export CUDA_HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH
export TORCH_CUDA_ARCH_LIST="12.0"
pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130
pip install -e . --no-build-isolation
# 2. the addon + its pinned deps
git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon
pip install -e . --no-deps
pip install -r release/requirements-serve.txt \
--extra-index-url https://flashinfer.ai/whl/cu130/torch2.11
# 3. serve: no arguments needed, the head is resolved from the model repo
release/serve_ds4_reasoning.sh
The serve script defaults to the bundled model head (latent_reasoning_head.safetensors, ~152 MiB, cached after the first run), so the recommended configuration is already applied. If you trained your own head, point HEAD_BUNDLE at it and you are done.
Why a fork at all?
This is not a cargo-cult fork. There is a real technical reason stock vLLM can't serve this model.
DeepSeek-V4 routes MoE experts by a hash keyed on input_ids. vLLM's native prompt_embeds path nulls input_ids when you supply embeddings, which would crash the engine at startup with DeepSeek V4 hash MoE routing requires input_ids. So the addon cannot use the standard embeddings path.
Instead, injection overwrites the embed_tokens output at the target positions with the decoded latent, while token ids keep flowing normally so hash-MoE routing still works. Latent steps carry a reserved pad token id purely for accounting. A latent step occupies a real KV position, so it must advance num_tokens in lockstep with num_computed_tokens. That id never reaches the client, and its embedding is never read (the row is marked is_token_ids=False, so the injected latent survives). Because this runs at the execute_model seam, it stays on the cudagraph fast path. No enforce_eager, and it batches across concurrent requests.
Speculative drafting is suppressed during the latent phase (VLLM_DS4_SUPPRESS_LATENT_DRAFTS, on by default). Leave it on. The rider injects one embedding at the last query row, so a draft slot would steal it and the real position would get the pad token's embedding instead of the latent.
The knobs, revised
The env-var / header split survives, but it is cleaner now, with a couple of new dials the model-card work made necessary.
Server-wide environment variables (defaults = measured-good)
| Variable |
Default |
What it does |
MAX_LATENT |
256 |
Safety cap on latent steps (bounds a stop-head misfire). Matches the head's K=256 training. |
MIN_LATENT |
4 |
Floor on latent steps, so answers never no-think. |
USE_STOP |
1 |
Learned stop; 0 uses a fixed-N cap. |
STOP_THRESHOLD |
0.5 |
Stop-head threshold. 0.5 is correct. Don't "fix" warmup by lowering it. |
MIN_OUTPUT_TOKENS |
4096 |
Floor on the answer's token budget. |
RIDER |
1 |
0 serves the bare backbone (A/B baseline against latent reasoning). |
DEBUG |
0 |
1 prints per-request latent stats. |
MAX_MODEL_LEN |
262144 |
Context window. |
MAX_NUM_SEQS |
2 |
Concurrent sequences. Trades against context. |
GPU_UTIL |
0.95 |
Narrow viable band. 0.97 OOMs, and lower can fail the KV check. |
TP |
2 |
Tensor-parallel size. |
Same as before, plus one new one. These win over env defaults and work on both /v1/chat/completions and /v1/messages (useful for clients like Claude Code that set headers but not body fields).
| Header |
Effect |
x-ds4-thinking |
Enable thinking without a body field. |
x-ds4-max-latent |
Per-request latent cap. |
x-ds4-min-latent |
Per-request latent floor. |
x-ds4-use-stop |
Toggle the learned stop. |
x-ds4-stop-threshold |
Per-request stop threshold. |
x-ds4-min-output-tokens |
Per-request answer-budget floor. |
Two knobs worth understanding
MAX_LATENT is a safety cap, not a reasoning-depth dial. The learned stop normally ends the phase. The cap only bounds a misfire. Where you set it matters: pushing far past the K=256 the head saw in training drifts into garbage rather than thinking harder. The bundle's compression_factor is also inert at serve time. Nothing in the rider reads it. It describes what the head learned, not a budget the loop enforces.
MIN_OUTPUT_TOKENS exists because the latent phase and the answer share one budget. Each latent step bills one reserved accounting token. A client sending a modest max_tokens can spend the whole budget thinking, then get an empty answer back with stop_reason=length. The floor only ever raises a client's max_tokens. The learned stop still ends generation early once the answer is done, so a generous floor does not force verbosity.
The canonical client
Two non-obvious requirements for any client that talks to the server. Both were learned the hard way.
1. You must ask for thinking. The reasoning phase is gated on chat_template_kwargs={"thinking": true} (or the header x-ds4-thinking: 1). Without it the output is garbage. That is not hyperbole.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="dummy")
resp = client.chat.completions.create(
model="nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning",
messages=[{"role": "user", "content": "Write a Python LRU cache."}],
extra_body={"chat_template_kwargs": {"thinking": True}}, # REQUIRED
max_tokens=4096,
temperature=0.6,
)
print(resp.choices[0].message.content)
2. Give the answer real token headroom. Reasoning and answer share one budget. A tight max_tokens can be consumed entirely by the latent phase. When in doubt, use the MIN_OUTPUT_TOKENS floor (default 4096) rather than hand-tuning per request.
Two things that look like bugs and are not
Serving this model surfaced two behaviors that will (and did) get reported as bugs. They are not.
The first request or two after startup can be degenerate repetition. Low draft acceptance, "be be be" output, even with the stop head firing normally. It settles by itself and then stays correct. Send one throwaway request after startup and treat a single bad early answer as unwarmed, not broken. And do not lower STOP_THRESHOLD or force SPEC_TOKENS=1 in response. Neither is the cause.
An armed rider and a silently-dormant one produce identical server logs. If VLLM_DS4_REASONING_CKPT is not set, or the request does not ask for thinking, the plugin quietly does nothing and you have no idea. VLLM_DS4_REASONING_DEBUG=1 is the only way to confirm injection is actually happening. It prints steps=N stop_step=... max_p=... end=stop|cap per completion, and says so explicitly when nothing was injected.
What it takes to run it
The backbone is NVFP4, so this is Blackwell-class hardware (sm120) or better. There is no way around it. The sm120 sparse-MLA kernel path is specific to this fork, and Hopper and earlier are untested. Sizing is entirely about fitting the ~158–164 GiB of weights plus the fp8 KV cache.
| VRAM (total) |
Verdict |
| ≥ 192 GiB (2× 96, or 4× 48+) |
Recommended. Long context (256k) with room for the KV cache. |
| 160–192 GiB |
Workable. Weights fit; drop MAX_MODEL_LEN to 32k–64k so KV fits. |
| < 160 GiB |
Not enough for NVFP4 weights at any context. |
The verified configuration: 2× RTX PRO 6000 Blackwell Max-Q (96 GiB each), TP=2, context 262,144 (443,012 fp8 KV tokens), gpu_memory_utilization 0.95, throughput 11.1 ms/token with the head plus DSpark spec decode (1.40×), and 89–91% draft acceptance.
A few hard-won notes:
max_num_seqs trades against context. At 256k the fp8 KV cache only holds a couple of full-length sequences. Raising concurrency starves KV and either OOMs at startup or silently truncates usable context. Raise it only alongside a lower MAX_MODEL_LEN.
If the engine fails its KV-cache check at startup, lower MAX_MODEL_LEN first, then MAX_NUM_SEQS. gpu_memory_utilization has a narrow viable band in both directions.
MoE backend: leave it alone. On sm120 with this model's swiglu_limit=10.0, FLASHINFER_CUTLASS (the auto choice) is the only working option. marlin crashes with an illegal address as soon as a latent is injected; trtllm, b12x and both cutedsl variants refuse to start. There is nothing to tune.
Loading is what kills machines, not serving. Reading a ~164 GiB checkpoint fills the page cache. On a 124 GB host, systemd-oomd kills the process with no traceback once user-slice pressure holds above 50%. That reads like a model bug; it is not (check /var/log/syslog). DSpark makes it worse by re-reading all 48 shards for the drafter. Under ~256 GB of RAM, cap the page cache your server can accumulate rather than trusting it to behave.
The closed latent loop, one more time
With all the packaging in place, here is what actually happens when a request hits the server. The backbone prefills your prompt once, then takes autoregressive latent decode steps. Each step's input embedding is decoder(head(previous layer-35 hidden)), fed back from the model's own hidden state via a per-request anchor store. This continues until the learned stop head crosses its threshold, emits response, and the answer decodes as ordinary tokens.
h_src (layer 35) --layer_norm--> head -> mu (latent, 1024-d)
mu --layer_norm--> decoder -> hidden vector (4096-d)
This runs batched across concurrent requests and on the cudagraph fast path (no enforce_eager, no --max-num-seqs 1), and it streams. One latent step stands in for roughly six reasoning tokens. The model thinks compressed, then speaks.
Limitations and Future Work
The honest caveats from the last edition carry forward, plus a few that only became visible once there was a real model to poke at.
Blackwell or bust. NVFP4 needs sm120 native kernels, and the sparse-MLA path is fork-specific. If you do not have a Blackwell card, you get the backbone, not latent reasoning.
The surfaced trace is not the computation. Reasoning happens in latent space, so the text you see is not a faithful token-level record of the thinking that produced the answer. The latent space remains opaque.
Evaluation is BBH-only at 50 items/subtask. No multi-task or long-context suite reported. The aggregate is solid; the coverage is still narrow.
dyck_languages at 0.26 is a real weak spot, not a formatting artifact. Syntax-heavy, mechanical tasks are precisely what compressed-latent reasoning struggles with.
Future work includes:
A broader benchmark suite: multi-task and long-context, so the BBH 0.880 is not one number carrying the whole story.
Making the latent reasoning interpretable, so we can see what the model thinks, not just that it thinks.
Better handling of the mechanical-syntax tasks (dyck_languages) that are the current weak point.
Pushing the DSpark interplay further. The closed latent loop and speculative decoding together already give 1.40× at 89–91% draft acceptance. There is more to squeeze there.
The previous edition proved the idea: a model can think in compressed latent space and learn when to stop. This edition proves it can be shipped. The riddle was a microcosm of the general failure mode. Models recite memorized answers with confidence even when the premises changed. A CoLaR head is a step toward making a model check itself before it speaks. But an idea you cannot serve is just a paper. An NVFP4-quantized, DSpark-drafted, benchmarked, self-contained model with a one-command serving runtime is the same idea with the receipts attached.
The next time you see an LLM confidently wrong, remember: it just needs a chance to think twice. And now that thinking can run in production.
This blog was authored with the help of nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning
Implementation: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning · ds4-reasoning-addon · vllm-ds4-sm120Blog: blog.n.ichol.ai