# Packaging Latent Reasoning as a Real Model

**DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime.**

https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning

*Published on* [*blog.n.ichol.ai*](https://blog.n.ichol.ai/)

* * *

## Where the last edition left off

Last time I grafted a **CoLaR head** (Compressed Latent Reasoning) onto DeepSeek Flash v4. The work had all the makings of a demo. A head that lets the model think in latent space. A learned stop head that decides when it has thought enough. A riddle that showed why plain autoregressive generation regurgitates cached answers instead of reasoning.

There was one honest problem though: the head was an *adapter*. Something bolted on the side. To serve it you had to assemble the base model, the head, the stop criterion, and a custom runtime by hand, then hope the pieces fit. The weights lived in one place. The inference machinery in another. "Here's how you run it" was a jump-through-hoops story.

This edition closes that gap. The work is now a **complete, self-contained model**. Every weight needed to serve it ships in one repository. The backbone is quantized down to NVFP4 so it fits on real silicon. And the latent loop is driven by a proper, benchmarked serving runtime.

Model: [**nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning**](https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning)

* * *

## The big change: this is not an adapter anymore

The whole point of this edition is packaging. The old release was a head you had to attach. The new one is a model.

Every weight needed to serve it now sits in one HuggingFace repo:

*   **The DeepSeek-V4-Flash-0731 backbone**, quantized to **NVFP4** (group size 16) on the routed MoE experts. Attention, shared experts, LM head and draft block stay at higher precision. Roughly **79 GiB of weights per GPU at TP=2** (158–164 GiB total).
    
*   **The DSpark draft block** (3 layers), preserved from the source, so speculative decoding ships in the box.
    
*   **The trained latent reasoning head**. 35.7M params, loaded from a single `latent_reasoning_head.safetensors` (~152 MB).
    

Because the weights are complete, the model card can finally report a **real benchmark** instead of "more benchmarks soon."

* * *

## Actually writing down the numbers this time

The previous post ended with a half-promise: *"full benchmarking to come if I find time."* I found time.

**BBH (BIG-Bench Hard),** `cot_zeroshot`**, 27 subtasks: aggregate 0.880 ± 0.008.** Measured with lm-evaluation-harness 0.4.12 against an OpenAI-compatible endpoint. Thinking enabled. 50 items per subtask, 1350 items total.

| Subtask | Score | Subtask | Score |
| --- | --- | --- | --- |
| tracking\_shuffled\_objects\_three\_objects | 1.00 | date\_understanding | 0.92 |
| tracking\_shuffled\_objects\_five\_objects | 1.00 | sports\_understanding | 0.88 |
| tracking\_shuffled\_objects\_seven\_objects | 1.00 | logical\_deduction\_five\_objects | 0.88 |
| penguins\_in\_a\_table | 1.00 | web\_of\_lies | 0.86 |
| formal\_fallacies | 1.00 | snarks | 0.84 |
| boolean\_expressions | 1.00 | ruin\_names | 0.84 |
| word\_sorting | 0.98 | movie\_recommendation | 0.84 |
| temporal\_sequences | 0.98 | salient\_translation\_error\_detection | 0.76 |
| object\_counting | 0.98 | geometric\_shapes | 0.74 |
| navigate | 0.98 | causal\_judgement | 0.66 |
| logical\_deduction\_three\_objects | 0.98 | disambiguation\_qa | 0.58 |
| reasoning\_about\_colored\_objects | 0.96 | dyck\_languages | 0.26 |
| hyperbaton | 0.96 |  |  |
| multistep\_arithmetic\_two | 0.94 |  |  |
| logical\_deduction\_seven\_objects | 0.94 |  |  |

The pattern is exactly what you would hope for from a latent reasoning model. It is strongest where reasoning means multi-step state tracking. `tracking_shuffled_objects`, `boolean_expressions`, `formal_fallacies`, `penguins_in_a_table` all hit 1.00. It is weakest on the mechanical, syntax-heavy jobs. `dyck_languages` (bracket matching) sits at 0.26, the clear outlier. That is a genuine weakness, not a measurement artifact.

Two honest notes on reading the table:

1.  **Read** `flexible-extract`**, not** `strict-match`**.** BBH's `strict-match` regexes for the literal phrase `The answer is X`. This model does not emit that phrase, because it reasons in latent space. Its near-zero `strict-match` score is an answer-formatting artifact, not a reasoning failure.
    
2.  **Per-subtask values carry about ±0.05–0.07** at 50 items each. The aggregate of 0.880 is the reliable number.
    

* * *

## Why the head looks different now

The architecture picks up where the original CoLaR idea left off, but it has a proper home in the model now. A small head reads the backbone's **layer-35** hidden state, projects it into a **1024-d latent**, and decodes it back into the residual stream. One latent step stands in for several reasoning tokens (a recorded `compression_factor` of 6). A learned **stop head** self-terminates the loop at a variable, content-dependent depth.

```plaintext
                    layer 35 hidden (4096-d)
                             |
                             v  LayerNorm
        +--------- ReasoningCompressionHead ----------+
        |  Linear 4096 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048  ->  [mu, log_sigma]   |
        |                                             |
        |  stop_head:                                 |
        |     Linear 4096 -> 1024 . SiLU              |
        |     Linear 1024 -> 1                        |  -> end-of-reasoning
        +---------------------------------------------+
                             | mu (1024-d latent)
                             v  LayerNorm
        +-------------- LatentDecoder ----------------+
        |  Linear 1024 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 4096                        |
        +---------------------------------------------+
                             |
                             v  written back into the residual stream
              DeepSeek-V4-Flash-0731 backbone (frozen, NVFP4)
```

| Config | Value |
| --- | --- |
| hidden\_size | 4096 |
| latent\_dim | 1024 |
| mlp\_dim | 2048 |
| source\_layer / target\_layer | 35 / 42 |
| activation | SiLU |
| learned stop head | yes |
| head + decoder params | 35.7M (`float32`) |
| backbone layers | 43 |

One detail worth mentioning: the head is a **variational** compression. It predicts `[mu, log_sigma]` and clamps `log_sigma`. The decoder redistributes the latent back into the 4096-d stream. The whole thing is a single flat tensor dict distinguished by key prefix. The geometry lives in the checkpoint's own metadata, so the serving runtime needs **zero configuration**; it reads the head's shape straight from the file. `target_proj`, a frozen projection that defined the regression target during training, is included for completeness but is **not used at inference**.

* * *

## The serving runtime is now a real thing

The old runtime was a set of env vars and headers bolted onto a fork, with the model weights kept separate. This time the runtime is released in its own right, split cleanly into two pieces you install once:

| Repository | What it is |
| --- | --- |
| [`nickmitchko/ds4-reasoning-addon`](https://github.com/nickmitchko/ds4-reasoning-addon) | The latent-reasoning addon: the closed-loop driver, serve script, pinned requirements and GPU-sizing guide. **Start here.** |
| [`nickmitchko/vllm-ds4-sm120`](https://github.com/nickmitchko/vllm-ds4-sm120) | The DS4 vLLM fork it runs on (branch `ds4-sm120-preview-dev`). Required. Upstream vLLM cannot serve this model. |

Installing the addon **does nothing by itself**. The plugin registers but stays dormant until `VLLM_DS4_REASONING_CKPT` is set. That is a deliberate safety choice; it is safe to leave installed. The supported entrypoint is a single script that sets all the measured-good defaults for you:

```bash
# 1. the engine (a full build takes a while)
git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120
git checkout ds4-sm120-preview-dev
export CUDA_HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH
export TORCH_CUDA_ARCH_LIST="12.0"
pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130
pip install -e . --no-build-isolation

# 2. the addon + its pinned deps
git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon
pip install -e . --no-deps
pip install -r release/requirements-serve.txt \
    --extra-index-url https://flashinfer.ai/whl/cu130/torch2.11

# 3. serve: no arguments needed, the head is resolved from the model repo
release/serve_ds4_reasoning.sh
```

The serve script defaults to the bundled model head (`latent_reasoning_head.safetensors`, ~152 MiB, cached after the first run), so the recommended configuration is already applied. If you trained your own head, point `HEAD_BUNDLE` at it and you are done.

### Why a fork at all?

This is not a cargo-cult fork. There is a real technical reason stock vLLM can't serve this model.

DeepSeek-V4 routes MoE experts by a **hash keyed on** `input_ids`. vLLM's native `prompt_embeds` path **nulls** `input_ids` when you supply embeddings, which would crash the engine at startup with `DeepSeek V4 hash MoE routing requires input_ids`. So the addon cannot use the standard embeddings path.

Instead, injection **overwrites the** `embed_tokens` **output** at the target positions with the decoded latent, while token ids keep flowing normally so hash-MoE routing still works. Latent steps carry a reserved pad token id purely for *accounting*. A latent step occupies a real KV position, so it must advance `num_tokens` in lockstep with `num_computed_tokens`. That id never reaches the client, and its embedding is never read (the row is marked `is_token_ids=False`, so the injected latent survives). Because this runs at the `execute_model` seam, it stays on the **cudagraph fast path**. No `enforce_eager`, and it batches across concurrent requests.

Speculative drafting is suppressed *during* the latent phase (`VLLM_DS4_SUPPRESS_LATENT_DRAFTS`, on by default). Leave it on. The rider injects one embedding at the **last** query row, so a draft slot would steal it and the real position would get the pad token's embedding instead of the latent.

* * *

## The knobs, revised

The env-var / header split survives, but it is cleaner now, with a couple of new dials the model-card work made necessary.

### Server-wide environment variables (defaults = measured-good)

| Variable | Default | What it does |
| --- | --- | --- |
| `MAX_LATENT` | 256 | **Safety cap** on latent steps (bounds a stop-head misfire). Matches the head's `K=256` training. |
| `MIN_LATENT` | 4 | Floor on latent steps, so answers never no-think. |
| `USE_STOP` | 1 | Learned stop; `0` uses a fixed-N cap. |
| `STOP_THRESHOLD` | 0.5 | Stop-head threshold. `0.5` is correct. Don't "fix" warmup by lowering it. |
| `MIN_OUTPUT_TOKENS` | 4096 | Floor on the answer's token budget. |
| `RIDER` | 1 | `0` serves the bare backbone (A/B baseline against latent reasoning). |
| `DEBUG` | 0 | `1` prints per-request latent stats. |
| `MAX_MODEL_LEN` | 262144 | Context window. |
| `MAX_NUM_SEQS` | 2 | Concurrent sequences. **Trades against context.** |
| `GPU_UTIL` | 0.95 | Narrow viable band. 0.97 OOMs, and lower can fail the KV check. |
| `TP` | 2 | Tensor-parallel size. |

### Per-request HTTP headers

Same as before, plus one new one. These win over env defaults and work on both `/v1/chat/completions` and `/v1/messages` (useful for clients like Claude Code that set headers but not body fields).

| Header | Effect |
| --- | --- |
| `x-ds4-thinking` | Enable thinking without a body field. |
| `x-ds4-max-latent` | Per-request latent cap. |
| `x-ds4-min-latent` | Per-request latent floor. |
| `x-ds4-use-stop` | Toggle the learned stop. |
| `x-ds4-stop-threshold` | Per-request stop threshold. |
| `x-ds4-min-output-tokens` | Per-request answer-budget floor. |

### Two knobs worth understanding

`MAX_LATENT` **is a safety cap, not a reasoning-depth dial.** The learned stop normally ends the phase. The cap only bounds a misfire. Where you set it matters: pushing far past the `K=256` the head saw in training drifts into garbage rather than thinking harder. The bundle's `compression_factor` is also **inert at serve time**. Nothing in the rider reads it. It describes what the head *learned*, not a budget the loop enforces.

`MIN_OUTPUT_TOKENS` **exists because the latent phase and the answer share one budget.** Each latent step bills one reserved accounting token. A client sending a modest `max_tokens` can spend the whole budget thinking, then get an empty answer back with `stop_reason=length`. The floor only ever *raises* a client's `max_tokens`. The learned stop still ends generation early once the answer is done, so a generous floor does not force verbosity.

* * *

## The canonical client

Two non-obvious requirements for any client that talks to the server. Both were learned the hard way.

**1\. You must ask for thinking.** The reasoning phase is gated on `chat_template_kwargs={"thinking": true}` (or the header `x-ds4-thinking: 1`). Without it the output is garbage. That is not hyperbole.

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="dummy")

resp = client.chat.completions.create(
    model="nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning",
    messages=[{"role": "user", "content": "Write a Python LRU cache."}],
    extra_body={"chat_template_kwargs": {"thinking": True}},   # REQUIRED
    max_tokens=4096,
    temperature=0.6,
)
print(resp.choices[0].message.content)
```

**2\. Give the answer real token headroom.** Reasoning and answer share one budget. A tight `max_tokens` can be consumed entirely by the latent phase. When in doubt, use the `MIN_OUTPUT_TOKENS` floor (default 4096) rather than hand-tuning per request.

* * *

## Two things that look like bugs and are not

Serving this model surfaced two behaviors that will (and did) get reported as bugs. They are not.

**The first request or two after startup can be degenerate repetition.** Low draft acceptance, "be be be" output, even with the stop head firing normally. It settles by itself and then stays correct. Send one throwaway request after startup and treat a single bad early answer as *unwarmed*, not broken. And do **not** lower `STOP_THRESHOLD` or force `SPEC_TOKENS=1` in response. Neither is the cause.

**An armed rider and a silently-dormant one produce identical server logs.** If `VLLM_DS4_REASONING_CKPT` is not set, or the request does not ask for thinking, the plugin quietly does nothing and you have no idea. `VLLM_DS4_REASONING_DEBUG=1` is the only way to confirm injection is actually happening. It prints `steps=N stop_step=... max_p=... end=stop|cap` per completion, and says so explicitly when nothing was injected.

* * *

## What it takes to run it

The backbone is NVFP4, so this is **Blackwell-class hardware (sm120) or better**. There is no way around it. The sm120 sparse-MLA kernel path is specific to this fork, and Hopper and earlier are untested. Sizing is entirely about fitting the ~158–164 GiB of weights plus the fp8 KV cache.

| VRAM (total) | Verdict |
| --- | --- |
| **≥ 192 GiB** (2× 96, or 4× 48+) | **Recommended.** Long context (256k) with room for the KV cache. |
| 160–192 GiB | Workable. Weights fit; drop `MAX_MODEL_LEN` to 32k–64k so KV fits. |
| < 160 GiB | Not enough for NVFP4 weights at any context. |

The verified configuration: **2× RTX PRO 6000 Blackwell Max-Q (96 GiB each), TP=2, context 262,144 (443,012 fp8 KV tokens),** `gpu_memory_utilization` **0.95, throughput 11.1 ms/token with the head plus DSpark spec decode (1.40×), and 89–91% draft acceptance.**

A few hard-won notes:

*   `max_num_seqs` **trades against context.** At 256k the fp8 KV cache only holds a couple of full-length sequences. Raising concurrency starves KV and either OOMs at startup or silently truncates usable context. Raise it only alongside a lower `MAX_MODEL_LEN`.
    
*   **If the engine fails its KV-cache check at startup**, lower `MAX_MODEL_LEN` first, then `MAX_NUM_SEQS`. `gpu_memory_utilization` has a narrow viable band in *both* directions.
    
*   **MoE backend: leave it alone.** On sm120 with this model's `swiglu_limit=10.0`, `FLASHINFER_CUTLASS` (the `auto` choice) is the only working option. `marlin` crashes with an illegal address as soon as a latent is injected; `trtllm`, `b12x` and both `cutedsl` variants refuse to start. There is nothing to tune.
    
*   **Loading is what kills machines, not serving.** Reading a ~164 GiB checkpoint fills the page cache. On a 124 GB host, `systemd-oomd` kills the process **with no traceback** once user-slice pressure holds above 50%. That reads like a model bug; it is not (check `/var/log/syslog`). DSpark makes it worse by re-reading all 48 shards for the drafter. Under ~256 GB of RAM, cap the page cache your server can accumulate rather than trusting it to behave.
    

* * *

## The closed latent loop, one more time

With all the packaging in place, here is what actually happens when a request hits the server. The backbone prefills your prompt **once**, then takes autoregressive **latent** decode steps. Each step's input embedding is `decoder(head(previous layer-35 hidden))`, fed back from the model's own hidden state via a per-request anchor store. This continues until the learned stop head crosses its threshold, emits `response`, and the answer decodes as ordinary tokens.

```plaintext
h_src (layer 35)  --layer_norm-->  head     -> mu   (latent, 1024-d)
mu                --layer_norm-->  decoder  -> hidden vector (4096-d)
```

This runs **batched across concurrent requests and on the cudagraph fast path** (no `enforce_eager`, no `--max-num-seqs 1`), and it streams. One latent step stands in for roughly six reasoning tokens. The model thinks compressed, then speaks.

* * *

## Limitations and Future Work

The honest caveats from the last edition carry forward, plus a few that only became visible once there was a real model to poke at.

*   **Blackwell or bust.** NVFP4 needs sm120 native kernels, and the sparse-MLA path is fork-specific. If you do not have a Blackwell card, you get the backbone, not latent reasoning.
    
*   **The surfaced trace is not the computation.** Reasoning happens in latent space, so the text you see is not a faithful token-level record of the thinking that produced the answer. The latent space remains opaque.
    
*   **Evaluation is BBH-only** at 50 items/subtask. No multi-task or long-context suite reported. The aggregate is solid; the coverage is still narrow.
    
*   `dyck_languages` **at 0.26 is a real weak spot**, not a formatting artifact. Syntax-heavy, mechanical tasks are precisely what compressed-latent reasoning struggles with.
    

Future work includes:

*   A broader benchmark suite: multi-task and long-context, so the BBH 0.880 is not one number carrying the whole story.
    
*   Making the latent reasoning interpretable, so we can see *what* the model thinks, not just *that* it thinks.
    
*   Better handling of the mechanical-syntax tasks (`dyck_languages`) that are the current weak point.
    
*   Pushing the DSpark interplay further. The closed latent loop and speculative decoding together already give 1.40× at 89–91% draft acceptance. There is more to squeeze there.
    

* * *

The previous edition proved the *idea*: a model can think in compressed latent space and learn when to stop. This edition proves it can be *shipped*. The riddle was a microcosm of the general failure mode. Models recite memorized answers with confidence even when the premises changed. A CoLaR head is a step toward making a model check itself before it speaks. But an idea you cannot serve is just a paper. An NVFP4-quantized, DSpark-drafted, benchmarked, self-contained model with a one-command serving runtime is the same idea with the receipts attached.

The next time you see an LLM confidently wrong, remember: it just needs a chance to think twice. And now that thinking can run in production.

* * *

*This blog was authored with the help of* [*nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning*](https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning)

*Implementation:* [*nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning*](https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning) · [*ds4-reasoning-addon*](https://github.com/nickmitchko/ds4-reasoning-addon) · [*vllm-ds4-sm120*](https://github.com/nickmitchko/vllm-ds4-sm120)*Blog:* [*blog.n.ichol.ai*](https://blog.n.ichol.ai/)