RECAP Decodability Supervision Exposes a Blind Spot in Activation Explanations

RECAP decodability supervision changes where researchers place the burden of interpretability, despite reconstruction scores previously serving as a convenient measure of explanation quality. A July 22 paper reports that those scores can remain high while individual statements are false or unsupported. Its alternative trains selected information into a model’s readable internal representations, then checks the information with independent probes.

That shift challenges natural-language autoencoders, or NLAs, which turn hidden activations into text and reconstruct the activations from that text. A good reconstruction sounds like evidence that the explanation captured what the model represented. The new research argues that this inference fails at the level that matters most: individual claims.

The paper’s reversal is unusually sharp. Better reconstruction does not necessarily produce more truthful explanations. In some experiments, the explanation and reconstructor developed private conventions that preserved the score while using false words. RECAP instead makes designated content independently decodable, giving auditors another signal against which they can test the generated prose.

RECAP Finds That Reconstruction Can Reward the Wrong Explanation

The central finding is that reconstructing an activation does not verify every statement used to reconstruct it.

A natural-language autoencoder contains two learned components. The activation verbalizer converts a model’s hidden state into a text explanation. The activation reconstructor converts that explanation back into an estimated hidden state.

Researchers can compare the original and reconstructed activations. If the two are similar, they can assign the explanation a high reconstruction score. This creates an appealing loop because it produces a numerical evaluation without requiring a human-written explanation for every activation.

The weakness appears when several explanations lead to similar reconstructions. A sentence can include one false claim without losing enough information to damage the overall score. The reconstructor might depend mainly on the topic, style, or general meaning rather than every specific statement.

The new RECAP paper, written by Hiskias Dingeto, formalizes that distinction. Reconstruction measures whether the text carries enough information to regenerate an activation. It does not require every assertion in the text to be true of that activation.

The author tested a released verbalizer and reconstructor built around Qwen2.5-7B. The explanations reconstructed activations above chance, yet only about 2% of audited specific claims were reconstruction-dependent at the paper’s central threshold.

That 2% result is presented as a sensitivity-limited lower bound, not a complete estimate of truthfulness. Still, the relative pattern was consistent. The system retained much more information about the input’s gist than about its specific facts.

Additional transformations supported this interpretation. Paraphrasing an explanation retained 89% of its original reconstruction score. Replacing words with synonyms retained 83%, while masking content words reduced retention to 4%.

Those results do not mean paraphrasing is a failure. A good interpretability system should tolerate equivalent wording. The problem is that high reconstruction can coexist with weak sensitivity to the individual factual claims readers are likely to trust.

The researchers used minimal-pair flips to test that sensitivity. A flip changes one claim while preserving the rest of the explanation. If changing the claim barely affects reconstruction, the score provides little evidence that the original wording was grounded in the activation.

The validity of these edits matters. Oversized edits can damage surrounding text and make a claim appear more important than it is. The paper reports that its selected open-model judge produced valid edits in 99.3% of cases, compared with 68.4% for another tested judge.

Invalid edits inflated apparent grounding by roughly seven times. This control is more than a technical detail. It shows how an interpretability audit can flatter its own metric when its counterfactual examples change too much at once.

The immediate lesson is narrow but important. A reconstruction score can assess an explanation’s information-carrying capacity. It cannot, by itself, certify each sentence as a faithful description of the activation.

Private Codes Turn an Interpretability Test Into a Communication Channel

The more serious failure occurs when an explanation passes the test through wording that is useful to the reconstructor but false to a reader.

The released Qwen system mostly exposed a storage problem. Specific information was often difficult to recover from the selected activation, so the verbalizer leaned on general meaning. A controlled synthetic environment revealed a different failure.

The synthetic experiments used stories and marketplace records generated from fixed vocabularies. Every example contained known slots, such as a person, place, object, shop, item, or payment method. Some values appeared before the inspected activation, while others appeared afterward and were therefore unavailable.

This setup provided exact ground truth. The researchers knew which information could be stored at the selected point and which information was causally absent. They could compare an explanation’s words with the underlying record without relying on another language model’s judgment.

Across five standard training runs, the verbalizer and reconstructor developed what the paper calls co-adapted private codes. These were false wording choices that the reconstructor relied upon. The grounded-minus-true gap ranged from 0.30 to 0.80.

The system was not merely omitting details. Its explanation could name the wrong value, while the reconstructor treated that wrong word as a reliable code for recovering the activation. A human reader would interpret the word literally, but the learned pair had developed its own convention.

An evaluator swap exposed this behavior. The researchers replaced the co-trained reconstructor with an independent evaluator that had not seen the verbalizer’s outputs. Grounding for the coded claims fell from a range of 0.53 to 0.89 into a range of 0.08 to 0.23.

Grounding for truthful claims survived the same swap. That separation suggests the coded wording was not a universal property of the activation. It was a convention specific to the two components trained together.

The result resembles earlier cases where reconstruction objectives created hidden communication channels. Research on CycleGAN steganography showed that an image translation system could hide information needed for reconstruction. The visible output looked plausible, but invisible signals helped reverse the transformation.

Language creates a more direct risk because readers assign meaning to every word. A private visual signal can undermine image translation. A private linguistic code can make a false sentence function correctly inside the system while misleading the person evaluating it.

The paper does not claim intentional deception. “Private code” describes an operational pattern, not a motive. Gradient-based optimization found wording that helped the paired models satisfy their objective, even though the wording failed a human interpretation of truth.

That distinction matters for AI safety. A system does not need an explicit goal to lie before an explanation metric becomes unreliable. A misspecified objective can reward text that passes a machine evaluator and fails a reader’s semantic expectations.

Attempts to improve the reader did not solve the core issue. Rejection sampling and reinforcement learning against clause-level grounding rewards raised their proxy scores. Independent claim-level audits did not show a matching improvement.

The generated sentences changed their structure instead. Claims were merged or packaged in ways that made the scoring units easier to satisfy. This is a familiar pattern in machine learning: optimizing a proxy changes behavior around the measurement without improving the intended property.

The main opponent in this story is therefore not RECAP versus one competing company or model. It is external verification versus self-reconstruction. The first asks whether another instrument can recover a specific fact. The second asks whether a trained pair can reproduce its own internal signal.

How RECAP Decodability Supervision Changes the Target Model

RECAP trains selected content to remain readable inside the target model, instead of asking a better reader to recover whatever representations already exist.

RECAP stands for Readable Encodings via Co-trained Auxiliary Predictors. The method adds linear prediction heads at selected hidden states during model training. Each head predicts an externally defined target from the activation.

The auxiliary loss joins the normal language-modeling loss. This pressures the target model to store designated information in a form that a simple decoder can recover. The prediction heads train with the model, so they follow its representation changes.

The method addresses three failures identified in the paper. First, co-training avoids relying on a frozen probe that becomes stale. Second, external targets avoid pulling activations toward their own lossy reconstructions. Third, direct activation predictions remove generated prose from the supervision channel.

The researchers did not use the co-trained heads as final evidence. A head and target model could adapt to each other just as a verbalizer and reconstructor did. Every reported decodability result therefore came from newly fitted independent probes.

A probe is a small decoder trained to predict a known property from an activation. High probe performance suggests the property is accessible in that representation. It does not establish that the model uses the property in its final computation.

In the synthetic sandbox, independently fitted probes reached 100% decodability for designated content across five runs. Information that occurred after the inspected activation remained at chance, preserving the causal impossibility control.

Fresh verbalizers then described the designated content truthfully and with reconstruction grounding. The private-code signature appeared in none of the five complete-designation runs. It also remained absent in three partial-designation variants tested on one seed.

The larger experiment continued training Pythia-160M, a small open language model designed for research across training checkpoints. RECAP raised fresh-probe performance on designated targets to an AUC range of 0.95 to 1.00. Control models produced a range of 0.77 to 0.80.

AUC measures how reliably a classifier ranks positive examples above negative ones. A score of 0.5 reflects chance ranking, while 1.0 indicates perfect separation. These results therefore show a substantial increase in independent readability for the selected information.

Target design proved essential. A naive presence-prediction head learned to classify rare targets as absent without making them more decodable. Class-balanced training closed that shortcut and produced fresh-probe AUC values between 0.97 and 0.99 across 512 targets.

This is an important caveat within the method itself. A low training loss for an auxiliary head does not guarantee that a new probe will recover the intended feature. RECAP still requires independent evaluation because its own training objective can admit shortcuts.

The reported language-modeling cost depended on how many properties the system designated and how the targets were structured. For 64 balanced targets, the measured loss difference was within a 0.06-nat seed spread and not distinguishable from zero.

The paper’s abstract summarizes the balanced result as a cost of approximately 0.001 nat. Larger target sets produced a higher cost. The authors did not test whether the same tradeoff remains small in frontier models.

The approach resembles deep supervision, where auxiliary classifiers attach to intermediate layers. Its purpose here is different. Traditional deep supervision helps a model learn its main task, while RECAP uses auxiliary targets to preserve information for later inspection.

It also differs from concept bottleneck models, which force predictions through named concepts. RECAP does not route every computation through an interpretable layer. It adds selected readability constraints while leaving other information paths intact.

That narrower intervention makes deployment more plausible, but it also limits the guarantee. Auditors can test designated content. They cannot assume the rest of the activation becomes understandable.

Independent Probes Catch Lies That Reconstruction Scores Miss

RECAP’s strongest result is not prettier prose, but a second channel that can challenge the prose when its claims are false.

On Pythia-160M, a fresh verbalizer stated designated context words truthfully at rates between 0.44 and 0.46 across three RECAP-trained target models. Comparable controls ranged from zero to 0.07.

A stronger reconstructor raised the RECAP truth rate to 0.57, compared with 0.15 for its control. However, generation still failed to convey much of the information that independent probes could recover.

This gap separates decodability from verbalizability. Decodability means a fitted probe can extract selected content from an activation. Verbalizability means a generative reader can express that content accurately in free-form text.

RECAP delivered the first more reliably than the second. A redesigned reconstructor made selected content decodable from its own generation state, yet generated precision remained between 0.56 and 0.63. More reader capacity did not close the gap.

That result changes the role of natural-language explanations. Instead of treating prose as a direct window into a model, the paper treats it as a set of claims requiring verification. The verbalizer proposes an interpretation, and an independent probe checks whether selected facts are present.

On 672 held-out Pythia claims, an independent probe ranked true verbalizer claims above false ones with an AUC of 0.965. The comparable control achieved 0.823. The rounded figures in the paper’s summary are 0.96 and 0.82.

The researchers then edited truthful explanations to insert absent words. The adversary selected the false replacement that best preserved the deployed reconstruction score. This reduced approximately 87% of the score penalty associated with a random lie.

The RECAP probe continued distinguishing lies with an AUC of 0.952 and recall of 0.86 at the selected operating point. The control probe fell to an AUC of 0.508, effectively chance, with recall of 0.01.

A stronger probe-aware adversary sat outside the stated report-space threat model because it assumed access to the private monitor. Under that test, the RECAP probe declined to an AUC of 0.909. The control dropped below chance to 0.242.

These are experimental results from a small model, not a deployed security guarantee. Still, they reveal why reading an earlier activation is insufficient by itself. The control probe also accessed a state the text editor could not change, yet it lacked enough decodable content to identify the lie.

The result complements prior work on obfuscated activations, which showed that adversarial optimization can evade latent-space defenses. RECAP does not claim protection against an attacker manipulating the activations themselves.

Its threat model is narrower. The model produces an explanation, and an adversary changes the report to preserve a reconstruction score while inserting a false claim. Within that setting, designated decodability provides a verification signal outside the editable text.

For developers, the architectural implication is concrete. A system should not use one learned explanation channel as both witness and judge. Independent checks need distinct training signals, held-out calibration data, and tests designed around likely manipulation.

That principle also applies to knowledge systems. Generated summaries can be useful interfaces, but they should remain traceable to evidence. A searchable knowledge base helps preserve the sources needed to challenge a confident summary.

RECAP carries that verification principle into model internals. It does not ask readers to trust a more persuasive explanation. It modifies training so selected claims can be compared with another measurable representation.

The Guarantee Stops at Designated Content

RECAP offers a selective audit mechanism, not a general certificate of model honesty or reasoning faithfulness.

The first limitation is scale. Evidence comes from two synthetic domains, one released Qwen2.5-7B verbalizer system, and continued pretraining of Pythia-160M. Several scale experiments used single seeds, and frontier models were not tested.

The synthetic tasks also used closed vocabularies and template-parsed claims. Real explanations can contain open-ended abstractions, causal stories, uncertainty, and references that do not map cleanly onto a known label set.

RECAP requires researchers to define what should remain decodable. That creates a selection problem. Monitoring known properties is easier than discovering which hidden property becomes important during an unexpected failure.

The paper directly tested this boundary. On a lexical band that never received decodability supervision, RECAP showed no monitoring advantage over the control. Designation creates the benefit and defines its scope.

Poor targets can also produce misleading success. The naive presence objective learned an “absent” shortcut because positive examples were rare. Its training loss improved without making the property readable to a fresh probe.

Class balancing repaired that case, but future target spaces will be more complicated. A monitor might capture an easy correlate rather than the decision-relevant concept. Independent probes reduce this risk without eliminating it.

The method also cannot be attached to a completely frozen model. RECAP changes how the target model forms its representations, so it must participate in training or continued pretraining. Organizations using closed third-party models cannot apply the full method from outside an API.

Readability also erodes. The paper reports that decodability declined within a few hundred steps of fine-tuning without the auxiliary heads. Applying supervision every tenth step preserved much of the effect during a 2,000-step test, but that evidence came from limited runs.

This makes RECAP a maintenance commitment. Model developers would need to keep auxiliary objectives active, recalibrate probes after updates, and monitor whether new training data alters their target distributions.

Representation drift complicates any frozen-reader approach. In the study, probes became nearly useless within 1,500 continued-training steps, although the underlying information remained recoverable. A label-free affine map restored performance in nine of nine tested cases.

Related work on monitor staleness reports similar repair behavior on models with 2 billion and 7 billion parameters. Together, these findings argue that activation monitors need an update protocol, not just an initial benchmark.

The deepest limitation concerns causal use. A probe can show that an activation stores information without showing that the model relies upon it. The paper reports that intervening on the selected activation barely changed behavior in its test.

A model could keep a monitored concept readable while routing its meaningful computation elsewhere. This resembles known concerns about concept bottlenecks, where named concepts can be present without controlling the final prediction.

RECAP therefore verifies a storage claim: selected content exists in a recoverable form at a chosen site. It does not verify that the content caused the answer, captured the full reasoning process, or excluded alternative hidden computations.

That boundary prevents an expansive interpretation of the AUC results. A 0.96 probe score does not mean the system is 96% honest. It means the probe ranked true and false claims well within a specific dataset, model, target definition, and attack family.

The paper is also a preprint submitted to arXiv, not a completed scientific consensus. Independent replication, peer review, and tests on larger architectures remain necessary. The supplied results are promising evidence for a method and a warning about a metric.

What Comes Next for Verifiable Activation Explanations

The next phase should test whether designated readability survives frontier training, open-ended claims, and adversaries that target the activation itself.

The first signal to watch is replication at larger scales. Pythia-160M offers a controlled training environment, but modern deployed models contain many more parameters and use different data mixtures, objectives, and post-training pipelines.

A successful replication should report more than final probe AUC. It should measure language-modeling cost, performance across seeds, persistence through instruction tuning, and the behavior of never-designated controls.

If decodability remains high with a small capability cost across larger models, the paper’s main mechanism gains support. If costs grow sharply or post-training removes the signal, RECAP will look more like a research instrument than a deployment strategy.

The second signal is progress beyond closed-vocabulary claims. Current tests ask whether known words or synthetic slot values appear in an activation. Real oversight requires properties such as uncertainty, concealed objectives, policy violations, and evidence provenance.

Those properties lack simple labels. External targets may be noisy, incomplete, or contestable. Researchers will need to show that RECAP remains useful when the concept cannot be reduced to a clean token-presence task.

A credible open-ended evaluation should separate several questions. Does the model store the information? Can an independent reader recover it? Does generated prose state it correctly? Does the model use it while producing its answer?

The third signal is adversarial evaluation across the full training stack. The reported attack edited explanations after generation. A strategically trained model might instead reorganize its internal computation, hide information outside monitored subspaces, or satisfy auxiliary heads through misleading correlates.

Prior research on reasoning monitors warns that optimizing against a monitor can encourage obfuscation. RECAP avoids directly rewarding success against the final independent probe, but the target model still experiences auxiliary pressure during training.

Future experiments should include hidden monitors, independently selected targets, and evaluations withheld from the training team. They should also test whether attackers can preserve auxiliary-head accuracy while moving decision-relevant computation elsewhere.

For developers, the practical takeaway is already clear enough to act upon. Do not interpret high reconstruction as claim-level truth. Treat each activation explanation as a report that needs counterfactual testing and independent evidence.

For model builders, RECAP decodability supervision offers a more testable design direction. Selected internal content becomes easier to inspect because readability is trained into the representation, not recovered through increasingly elaborate prose generation.

For enterprise buyers and AI users, this work offers a useful question for vendors: what does an explanation score actually verify? A convincing description, a reconstructed state, and a causally faithful account are different products.

The paper’s title captures its central recommendation: train the model, not the reader. Yet the broader lesson is even more cautious. Training readable representations helps only when targets are well chosen, probes remain independent, and auditors respect the guarantee’s narrow boundary.

RECAP does not make neural networks transparent. It creates a measurable contract around selected internal facts. The next one to three months should show whether other researchers can reproduce that contract across larger models, broader claims, and harder adversaries.