Compressing Reasoning Without Breaking It

Reasoning models spend most of their tokens talking to themselves. They restate the problem, narrate their transitions, announce what they are about to do next, and only occasionally do the arithmetic that decides the answer. It seems wasteful to pay for thousands of tokens of narration to deliver a few hundred tokens of computation. If you could remove the chatter and keep the math, you would get the same answers faster.

That is the intuition. Whether it survives training is less obvious. The question is: can a model be trained on its own compressed reasoning, and if so, which spans can you actually compress? To find out, we ran the obvious experiment: take a small thinking model, collect its own verified reasoning, compress those traces under different policies, fine-tune on each variant, and benchmark. We tracked accuracy, token count and — for reasons that will be made clear later — whether the model can still stop.

Everything below is based on Qwen3.5-4B distilled on its own traces, plus a cross-family replication and scale-up on Gemma-4-12b: roughly fifteen LoRA arms and five falsified hypotheses, and three of them our own favorite explanations.

We find that:

• Reasoning traces are not uniformly compressible. Compress everything, and greedy decoding collapses into loops (GSM8K .03, looping on 93% of problems), while the same checkpoint scores .90 at temperature 1.0 on the loop-mined GSM8K subset. Compress only the narration and keep the computation, and accuracy goes up at 2–3× fewer tokens than uncompressed SFT, our control.
• What you must keep is the scratch work. Spans that do things (calculation, verification) serve a dual purpose: they hold intermediate results the model reads back, and they keep greedy decoding on track toward termination. The narration between them is cheap to remove.
• The system prompt is a behavioral dial that training re-wires. To the original model, “please reason step by step” means think harder. To a model trained on compression, it may mean be concise. And an explicit “compress your reasoning” instruction, which helps the original model, can hurt the model trained on compression.
• Compression can be a switch. Training with an identity prompt that describes the desired style binds the behavior to it: prompt on, concise reasoning; prompt off, normal-length reasoning.
• The costs are purely behavioral. Every failure we observed (looping, rumination, length inflation on code) was a change in how the model behaves. What it knows, as far as we could measure, stays intact.

Why self-distillation

An earlier project of ours showed cross-model compressed distillation degrading math performance and increasing looping. That setup confounded three things at once: the traces came from a different model (teacher mismatch), in that model’s own stylistic voice (style mismatch), and via a different data pipeline entirely (data mismatch). Self-distillation removes all three: the model generates its own traces, verifies them, compresses them, and retrains on them. Nothing changes between arms except the compression policy, so any change in behavior is attributable to it.

Related work approaches the same goal from other directions. Chain-of-Draft and token-budget prompting compress at inference time, without training. LiquidAI’s Antidoom gives a taxonomy of loop failures and a token-level repair (FTPO), which we reimplement below. The distillation literature at large, meanwhile, almost never asks which spans of a trace carry the teaching signal — that gap is what this study covers.

Setup

Data. We start from a pool of 37,460 prompts across math, code, general, and creative domains, decontaminated against our evals with 8-gram and containment matching. For the verifiable domains we sample four reasoning traces per prompt at temperature 0.6 (math answers are checked in a symbolic engine, code runs against sandboxed tests) and one trace elsewhere. The selected trace is the first that verifies and does not loop. For this study we scale the pool down to a “mini” build: 1,328 raw rows filtered to 648 selected (191 math, 219 general, 137 creative, 101 code). Individual arms train on 322–648 of these rows, depending on how many conversions each variant’s linters reject.

Compression variants. Each variant applies a different policy to the same source rows:

• Flat. The whole reasoning trace is rewritten in a very terse “caveman” style: drop articles, filler, and transitions; keep numbers, negations, and code. Three intensities (light, heavy, adaptive) all realize a compression ratio around 0.35 with the 4B self-converter.
• Section-aware. Each trace is segmented into functional spans: compute, verify, plan, transition, restate, narration. The compute and verify spans stay unchanged; the rest is compressed or deleted, except the final wind-down span. Realized ratio: 0.67.

We lint every conversion, checking that numbers, negations, and code blocks survive exactly, and every row records which converter and segmenter produced it.

Training and evaluation. We train every arm with Unsloth QLoRA at rank 64 for two epochs, under identical settings; the only thing that varies across model families is the reasoning-data channel, since different families use different chat and reasoning tokens. We evaluate on GSM8K, MATH-500, HumanEval, MBPP, MMLU-Pro, and a held-out set of creative prompts judged pairwise. We also measure loop rate at temperatures 0, 0.6, and 1.0, and accuracy under token budgets via prefix analysis.^{1 1}Cells evaluated without a system prompt are scored leniently (strict boxed-answer match OR last-number match). Without the prompt, models answer in prose, and a strict scorer undercounts by up to 60 points. ¹Cells evaluated without a system prompt are scored leniently (strict boxed-answer match OR last-number match). Without the prompt, models answer in prose, and a strict scorer undercounts by up to 60 points.

Compression is uneven across a trace

The headline numbers are easiest to read on the accuracy–token plane, one point per training arm:

Section-aware compression outperforms uncompressed SFT on the paired test: +.15 accuracy (p=.008) at 1.7× fewer reasoning tokens.^{2 2}P-values are exact McNemar tests: two arms answer the same questions, and the test asks whether the questions where exactly one arm is correct split evenly. The intervals in Figure 3 come from a different procedure, a percentile bootstrap over all paired outcomes, so the two can disagree at the margin; on HumanEval they do (p=.073 with an interval just clearing zero), and the conservative exact test is the one to trust. Everything is uncorrected for the grid's many comparisons: treat p=.04 as suggestive, while .008, .0008, and .005 survive any reasonable correction. ²P-values are exact McNemar tests: two arms answer the same questions, and the test asks whether the questions where exactly one arm is correct split evenly. The intervals in Figure 3 come from a different procedure, a percentile bootstrap over all paired outcomes, so the two can disagree at the margin; on HumanEval they do (p=.073 with an interval just clearing zero), and the conservative exact test is the one to trust. Everything is uncorrected for the grid's many comparisons: treat p=.04 as suggestive, while .008, .0008, and .005 survive any reasonable correction.. The same data trained without a system prompt and evaluated with one (an asymmetry we return to below) does better still: +.20 (p=.0008) at 2.9× fewer tokens. Under a hard budget the gap widens further. Given 1,024 thinking tokens, the section-aware arms solve 40–69% of GSM8K, depending on conditioning; uncompressed SFT solves 8%.

Flat compression fails at termination, not recall

Calling the flat arms “worse” is an understatement. They are broken in one specific way, and the diagnosis is what’s most interesting.

At greedy decoding, the flat-compressed model loops on 93% of GSM8K problems, often right after reaching the correct answer. We tried two standard repairs, keeping the wind-down text verbatim and matching the training system prompt at eval; neither changed anything. We localized the failure by sweeping the sampling temperature on a GSM8K subset mined from observed loop failures, henceforth the loops suite:

The checkpoint that scores .02 at greedy decoding scores .90 at temperature 1.0 — the same weights, the same questions. Nothing was forgotten; what the model failed to learn is a termination behavior that survives deterministic decoding.

Section-aware compression differs in exactly one respect: it preserves the spans where the model did the actual work of computing and verifying, in the model’s own words. Forcing the model to rephrase everything in a new register is too far out-of-distribution for reliable greedy termination.

Two measurements taken before any training point the same way. Section-aware conversions pass our content linters at 85%, versus 50% for flat. And they are answer-preserving: conditioning the original model on the prompt plus its own compressed reasoning recovers the verified answer in 114 of 114 checkable rows. Both trained arms recover under sampling; only section-aware is robust to deterministic decoding.

We will call this the scratch-work hypothesis: spans that compute or verify are working memory the model actually reads back, and they double as termination anchors; the narration between them is packaging. The rest of the study tests it.

Gains vary by task domain

Where does the +.15 come from? The plot below shows paired per-question deltas against uncompressed SFT, with bootstrap confidence intervals:

The pattern is what the scratch-work hypothesis predicts. Grade-school math traces are mostly narration, so deleting narration is nearly pure profit. Hard math is truncation-bound for every arm, so nothing moves. Code traces are nearly all computation, with no narration to delete, and compression training actively teaches code verbosity, a cost we decompose below.

What the system prompt means depends on training

This became the deepest thread of the study. We varied one factor at a time: whether the innocuous math prompt (“Please reason step by step, and put your final answer within \boxed{}.”) is present during training, present during evaluation, both, or neither, all on the same section-aware data.

Three findings, in increasing order of strangeness.

Evaluation-time conditioning dominates. With the math prompt at eval, compression-trained checkpoints score around .80 at roughly 2k tokens; without it, around .63 at 3.4k, regardless of training conditioning. On uncompressed checkpoints the same swap does nothing. The prompt works as a key: it turns on a behavior the training installed.

Training with the prompt creates a dependency on it. The easy suite hides this; MATH-500 exposes it. The prompt-trained model collapses without its prompt (.53 to .35, p=.005), while the bare-trained model (trained without the prompt) does not care (.57 either way; only token counts grow). Training bare leaves the prompt as a pure efficiency trigger with no downside. That asymmetry gives the deployment recipe in one line — train bare, serve with the prompt: .82 at 1,524 tokens, the best cell in the study.

The trigger and the instruction come apart. We also evaluated an explicit compression prompt — an identity prompt describing terse, fragmentary reasoning. On the original model it works exactly as instruction-following should: .72 at 3,823 tokens, up from .575 at 4,413. Prompt-only compression is real. On compression-trained models, however, the same prompt is poison: .45–.55, a 35% loop rate, and runs that reach the correct answer (often literally emitting the boxed answer inside the think block) and then ruminate until the budget dies. Asking a compression-trained model for the very style it was trained on is out-of-distribution conditioning, and what breaks is, once again, termination.

Compression as a switch

If the trigger is trained conditioning, you should be able to choose the trigger. To test this, we trained the same section-aware data with the identity prompt, hoping that the style would bind to it. It does, cleanly: prompt on, .80 at 1,945 tokens with a .09 loop rate and the best Qwen MATH-500 cell (.59); prompt off, the model decompresses to about 4.1k tokens at ungated accuracy (.59–.64). Statistically, the gated arm ties the always-on champion (−.02 GSM8K, +.02 MATH-500, both not significant). The price of the switch is the ~800-token system prompt it takes to flip it.

What didn’t work

We also wanted to shoot down a few hypotheses that we found plausible ourselves. The negative results are just as interesting as the headline:

1. “Keep the wind-down text verbatim and the loops will stop.” No: the flat collapse persists with the exact wind-down present. The stop signal is understood through familiarity within the distribution, and can’t be replaced by a magic suffix.
2. “Exempting code rows from compression avoids the code tax.” The tax was doubled, with HumanEval at .33 and MBPP at .27, with 66-71% of the runs being truncated. Having long code rows next to short math rows teaches the model that code is naturally longer: 9.4k tokens per code problem, compared to 4.5k for the uncompressed SFT and 821 for the original model. Contrast between domains amplifies per-domain length.
3. “A third epoch will train the style in more deeply.” Training on our data for a third epoch resulted in a model that was both longer and worse.
4. “A stronger teacher gives more headroom.” We had step-3.7-flash do both the segmentation and the conversion of the traces in place of Qwen. The data came out cleaner (97% lint pass, versus 85% for the self-built variant) but the model came out worse: .59 on GSM8K, below the uncompressed SFT control at .62 and far below the self-built arm at .77, with twice the self-built arm’s loop rate (.25 versus .11), at a ratio of 0.81. There is no compression signal to learn from when the data is barely compressed.
5. “Then the foreign voice must be the problem.” No again. A hybrid arm (step-3.7 segments, self-voice conversion) landed at ratio 0.874 and scored .58, indistinguishable from the full step-3.7 arm. The failure traces to over-preservation by the stronger segmenter; the narration’s voice is exonerated. Real compression is what matters (or so we concluded here; the Gemma build below gave us more data to work with).^{3 3}One caveat: realized ratio is confounded across these arms (0.67 / 0.81 / 0.87). Isolating voice at matched ratio would need a deliberately down-sampled build; we didn't run it. ³One caveat: realized ratio is confounded across these arms (0.67 / 0.81 / 0.87). Isolating voice at matched ratio would need a deliberately down-sampled build; we didn't run it.

The code regression is a token-budget effect

At an 8k thinking budget, the compression arms lose about .13–.15 on the code suites. Raising the budget to 12k recovers some of it (+.04 section-aware, +.10 mixed), but a gap of roughly .12 persists. The mechanism is visible in the token counts: compression training shortens math thinking and lengthens code thinking. The failures at 8k are healthy reasoning hitting the ceiling mid-thought: plain budget exhaustion, with loop rates staying low and the knowledge intact.

We did not find a fix that lives inside a single training mix; falsification #2 above is what happened when we tried the obvious one. The honest mitigations are per-domain adapters, budget-aware training objectives, or accepting the trade where math throughput dominates. (The Gemma replication below adds a twist: the sign of the code cost turns out to follow the training data.)

Where loop repair helps and where it doesn’t

We reimplemented FTPO, LiquidAI’s token-level preference recipe from Antidoom, on 463 loop points mined from our own eval records. It substantially repairs sampled decoding on the flat arm: at temperature 0.6, accuracy goes from .20 to .54 and loops drop from .74 to .22. It cannot touch the greedy attractor: loops fall only from .93 to .81, and full-suite greedy accuracy stays at zero. At temperature 1.0 there was little left to repair — loops were already at .04 — and accuracy slipped from .90 to .80. Token-level preference works where loops are shallow decoding accidents (e.g. “Wait, le me check…”). A distribution-deep attractor needs a different fix.

Creative writing survives, mostly

Pairwise-judged on 50 held-out prompts, in both orders: uncompressed SFT improves creative writing over the original model (20 wins, 8 losses, 22 ties); section-aware compression keeps most of that gain (17W/12L/18T against the original) at a modest cost against uncompressed SFT (12W/19L/17T).^{4 4}Wins are LLM-judged, so read them as retention checks rather than quality claims. Our compression policy targeted the think block, not the prose after it; we expected to preserve writing ability, not improve it. ⁴Wins are LLM-judged, so read them as retention checks rather than quality claims. Our compression policy targeted the think block, not the prose after it; we expected to preserve writing ability, not improve it. Compression costs something for prose, but the price is small.

Does it transfer? Gemma-4-12b

Everything above is one 4B model trained on its own traces. To test transfer, we ran the full pipeline end-to-end on gemma-4-12b-it: 1,620 self-generated traces, 824 selected, section-aware self-voice compression, trained bare, evaluated with the math prompt. Three times the parameters, a different family, a different tokenizer. Two things make this more than a box-check.

The build itself was a natural experiment. Gemma’s self-segmentation preserved far more than Qwen’s (realized ratio 0.806 versus 0.67) which puts it squarely inside the ratio band where the Qwen teacher probes trained worse-than-SFT models. We registered the prediction before training: if ratio rules, the Gemma arm fails; if distributional self-consistency rules, it works.

Familiarity won, decisively (all cells n=100, greedy, math prompt):

The HumanEval cell deserves a double-take: the Qwen code cost inverts. On Qwen, compression training lengthened code thinking and cost about .13. On Gemma it shortens code thinking and nearly doubles accuracy. Both follow the same rule: models reproduce the per-domain trace lengths they were trained on. The sign flips with the data — Gemma’s self-segmentation compressed code hardest (ratio 0.757, its most-compressed domain), while Qwen’s code spans resisted compression. What looked like an intrinsic cost of the code domain is a property of the data recipe.

The second thing is the temperature sweep, because base Gemma is natively a temperature-1.0 model, and greedy overthink-and-truncate is its out-of-the-box pathology:

The dividend holds at every temperature: best operating point versus best operating point is +.16 at 1.8× fewer tokens. Notably, Gemma keeps its upward temperature slope after compression training, where Qwen’s flipped downward. Compression lifts and tightens the whole curve while the family’s sampling character stays put.

The uncompressed SFT control sharpens that contrast into a small finding of its own: plain SFT anchors the curve. It lifts greedy hard (+.18 at t=0, .72 vs .54) but flattens the family’s temperature response and surrenders the original model’s t=1.0 edge (.76 vs .80). The compression arm does neither: it keeps the upward slope, ties the control at greedy, and pulls away as temperature rises (.96 vs .76 at t=1.0). The one place plain SFT pulls ahead is MATH-500, where the control posts the best cell of the three Gemma arms (.60, versus .49 section-aware and .51 original): hard math rewards the extra tokens uncompressed SFT leaves in, and compression’s wash there (−.02, as on Qwen) is the cost of spending fewer. Both models are truncation-bound on hard math at an 8k budget regardless.

A methodological note this section owes to live review: a greedy-versus-greedy comparison against a base model tuned for temperature 1.0 partly measures the base’s greedy handicap; the temperature sweep and the uncompressed SFT control column are what make the claim fair.

So the recipe transfers across family and size, and the mechanism conclusion gets its confirmation from an experiment we could not have designed on purpose: ratio 0.81 with fully self-consistent voice and structure wins decisively, while the same ratio with foreign segment structure lost. Real compression matters, but distributional self-consistency is the binding constraint.

Takeaways

1. Compressibility follows computational function. Narration is compressible at a profit; externalized computation is scratch memory the model actually reads, and verification spans double as termination anchors. The recipe lives in the span map; an average compression ratio hides it.
2. Termination is a first-class failure axis. Loop rate at greedy decoding exposed every pathology in this study before accuracy did. Evaluate at temperature 0 explicitly; sampled evals hide attractors.
3. Conditioning is a lever. Report the full train-prompt × eval-prompt grid. Train bare unless you want a gate; if you want a gate, train it in deliberately. Never assume a style instruction will be followed by a model that was trained on the style.
4. Self-consistency is what matters most; ratio isn’t the dial we thought. We spent the Qwen teacher probes blaming under-compression for the worse-than-SFT arms; Gemma trained at the same ratio, with self-built structure, and won decisively. What sank the probes was foreign segment structure. Compress in the model’s own voice and its own segmentation, and how much narration disappears matters less than whose distribution the result lives in.
5. Costs are behavioral; audit them behaviorally. Length inflation, truncation, rumination: all of them invisible to an accuracy number at a generous budget, and all of them found here by measuring behavior directly.
6. Small-model self-distillation is a real deployment path. +.20 GSM8K at 2.9× fewer tokens over uncompressed SFT, from 552 training rows and about 1.5 GPU-hours, with a switchable variant that costs nothing but a system prompt.

Limitations and open questions

• Scale. Every trained arm is 322–648 rows on one 4B model (plus the 12B replication). That is what made a fifteen-arm falsification grid affordable, but it means the absolute numbers are not the recipe’s ceiling, the code findings rest on 101 code rows, and the data-dose curve (going from the 342-row v1 build to the 648-row v2 build — 293 versus 552 section-aware training rows — changed the terse operating point substantially) is the key unexplored axis. A ~1,500-row arm is queued; a 25–30k-row full recipe is pre-staged and waiting on compute. If you have GPU time to spare and want to see it run at scale, get in touch.
• Voice at matched ratio remains open. The teacher probes confound compression ratio with converter identity; settling it needs a deliberately down-sampled build.
• Creative and chat domains were judged pairwise on 50 prompts, with no compression policy tuned for them.

Artifacts

• Datasets and models: the verified self-distillation corpus, every compression variant with converter/segmenter provenance and span labels on every row, and LoRA adapters for every arm named in this post, released on Hugging Face in two collections: caveman-reasoning-compression and reasoning-compression-ablation-arms.
• Code: marcodsn/reasoning-compression has the three-stage pipeline (harvest → compress → evaluate), one-command reproduction, all manifests, and the unedited experiment journal this post is distilled from.

@misc{desantis2026reasoningcompression,
  author = {Marco De Santis},
  title = {Compressing Reasoning Without Breaking It},
  year = {2026},
  month = jul,
  url = {https://marcodsn.me/blog/reasoning-compression}
}