SALT Retrieves More Memory, but Smaller Models Lose Accuracy

A horizon machinelearning post has exposed a sharp retrieval conflict: SALT searches a full memory trie efficiently, yet smaller models hallucinate when it returns too much.

The developer says SALT stores every input in a trie, a tree-shaped structure that shares common prefixes to reduce repeated storage. It then uses theme dominance and CELF selection under a 20 percent retrieval budget. Those implementation claims come from the developer’s retrieval discussion, not an independently reviewed evaluation.

The system reportedly works with a chatbot, but agents are now joining the architecture. That change raises the stakes. Multiple modules can retrieve overlapping memories, repeat marginally relevant facts, and consume the generator’s limited attention before it takes any action.

The central problem is not whether SALT can find related sentences. It apparently finds too many. The harder question is whether a retrieval policy can preserve every necessary dependency while excluding plausible but distracting material.

That places two objectives in direct conflict. Theme coverage rewards a set for representing more of the active topic. Evidence precision rewards only material that changes the correct answer or action. For smaller models, the second objective can matter more than the first.

The SALT Proposal Turns Memory Recall Into a Selection Problem

SALT’s reported bottleneck begins after storage succeeds, because efficient access does not guarantee useful context.

According to the post, all inputs enter a trie in DRAM. DRAM is the system’s fast working memory, while a trie organizes sequences through shared prefixes. This design can make repeated text patterns compact and quickly addressable.

The developer then retrieves sentences through a keyword and theme-dominance system. A CELF procedure selects material within a budget set at 20 percent. CELF, or Cost-Effective Lazy Forward selection, accelerates greedy optimization by avoiding unnecessary recalculation of every candidate’s marginal value.

The attractive property is diminishing returns. A sentence covering a new theme can initially offer substantial value. Another sentence covering the same theme should contribute less once the first sentence has entered the selected set.

That logic fits retrieval when diversity and coverage matter. It discourages a result set filled with near-duplicates, while still allowing several parts of a topic to appear. It also makes a large memory collection manageable without scoring every possible subset.

However, a fixed percentage does not represent a fixed information need. Twenty percent of a short conversation can produce a compact prompt. Twenty percent of a large, persistent agent memory can produce far more material than a small model can reliably use.

The budget also grows with the candidate pool unless another cap intervenes. As more modules write memories, theme-consistent candidates can multiply. The selector can remain computationally efficient while its output becomes cognitively expensive for the language model.

This distinction matters because retrieval has at least three separate stages. The system must generate candidates, rank or select them, and then package them for a model. Speed in the first two stages cannot establish accuracy in the third.

The public material does not yet document SALT’s theme representation, sentence boundaries, deduplication rules, or evaluation set. It also does not establish whether the 20 percent budget measures sentences, tokens, stored nodes, or another unit.

Those details change the diagnosis. A sentence budget can hide wide differences in token length. A token budget can still admit repeated propositions. A node budget in a trie may not correspond cleanly to readable evidence.

The reported repository location, SALT source code, was not consistently accessible during research. Therefore, architectural details beyond the post should be treated as provisional until code and reproducible tests are available.

What has changed is still clear. SALT is moving from a chatbot setting toward an agent setting, where memory retrieval influences actions across multiple modules. That transition turns excessive recall from a conversational nuisance into a system-level reliability problem.

Why Horizon MachineLearning Attention Focused on Too Much Context

The horizon machinelearning discussion matters because adding relevant-looking context can reduce accuracy, even when every retrieved sentence shares the query’s theme.

Language models do not treat all supplied information as equally useful. A context window sets the maximum input size, but capacity does not guarantee dependable use. Position, repetition, ambiguity, and task complexity all affect what the model actually follows.

The classic long-context study tested multi-document question answering and key-value retrieval. Researchers changed where relevant evidence appeared while keeping the desired answer unchanged. Performance often followed a U-shaped curve, favoring information near the beginning or end.

In one reported setup, GPT-3.5-Turbo performed worse than its 56.1 percent closed-book baseline when the relevant document sat poorly within longer contexts. The researchers also found diminishing returns from retrieving additional documents.

Moving from 20 documents to 50 improved results only marginally in their open-domain question-answering case study. The added retrieval recall did not translate into comparable answer gains. The generator could not exploit all the extra material effectively.

Newer evidence sharpens that warning. A 2025 context-length study found that longer inputs can hurt performance even when retrieval itself is perfect. That result separates two failure sources that teams often combine.

The first is retrieval error, where the system chooses missing, misleading, or incomplete evidence. The second is utilization error, where the model receives adequate evidence but fails to reason over it reliably. Reducing the first does not automatically solve the second.

This distinction explains why theme coverage can look good in a retrieval dashboard while answers deteriorate. A sentence can belong to the correct theme without helping resolve the current request. It can also introduce an outdated value, an exception, or a nearby concept.

Consider an agent preparing a software deployment. Memories about deployment policy, prior incidents, testing, permissions, and customer impact all match the broad theme. Yet only the current environment, approved version, active incident state, and required checks might govern today’s action.

A coverage objective can reward the historical incident because it adds thematic breadth. The generator may then blend old constraints with current ones. A smaller model has less spare capacity for distinguishing chronology, authority, and conditional applicability.

Theme membership is therefore a weak proxy for causal usefulness. The best retrieved evidence is not merely related to the question. It must materially support, constrain, contradict, or disambiguate the answer.

This pressure grows in multi-turn systems. Microsoft researchers reported an average 39 percent performance drop across six generation tasks when tested models handled conversations over multiple turns. Their conversation benchmark found that leading open and closed models performed worse than in equivalent single-turn settings.

Agents add another layer. Each module can create summaries, plans, tool results, observations, and status messages. A shared memory system then faces several versions of the same fact, each written for a different local purpose.

Compression helps storage, but it does not guarantee decision relevance. A compressed distractor remains a distractor. Several compressed summaries can also obscure which original source was authoritative.

For builders of a searchable knowledge base, this is the practical lesson. Retrieval quality must be evaluated at the final answer or action, not only at the index or ranking layer.

Coverage and Precision Pull Sentence Retrieval in Opposite Directions

SALT’s main design conflict is coverage versus precision, not tries versus vector databases or CELF versus another optimizer.

Coverage asks whether the selected set represents enough distinct aspects of a topic. Precision asks whether each selected item deserves scarce prompt space for this specific decision. Both are useful, but they reward different behavior.

A pure relevance ranker often returns redundant sentences. The highest-scoring items may restate the same prominent concept with minor wording changes. Submodular selection can improve diversity by discounting candidates that add little beyond items already selected.

SALT’s reported CELF use appears aimed at that problem. If the underlying objective is submodular, lazy greedy selection can efficiently approximate a high-value set. Yet an optimizer can only pursue the values encoded in its objective.

If theme coverage assigns value to every new subtheme, the system will seek breadth. It does not know that one subtheme is merely background while another contains the decisive constraint. It also cannot infer this from computational efficiency alone.

The retrieval unit compounds the issue. Sentences are easy to score and rearrange, but facts do not always respect sentence boundaries. A qualifying sentence can depend on a definition, timestamp, speaker, or exception located nearby.

Retrieving only the apparent answer sentence can strip away necessary provenance. Retrieving its entire thematic neighborhood can restore provenance but add noise. The system needs an evidence unit that preserves dependencies without importing a whole topic cluster.

One option is claim-centered retrieval. The system would represent each memory as a claim plus metadata, including source, time, scope, confidence, and links to required qualifiers. Selection would operate over these evidence bundles instead of isolated sentences.

Another option is question-conditioned marginal gain. A candidate’s value would depend on whether it improves an answer, resolves ambiguity, supplies a missing step, or contradicts the current draft. General theme novelty would become a supporting signal rather than the primary goal.

Neither approach eliminates tradeoffs. Claim extraction can introduce errors during ingestion. Question-conditioned scoring can add latency and depend on another model that carries its own biases.

Still, both approaches expose the actual optimization target. The retrieval layer should maximize expected task utility under a token and latency budget. It should not maximize memory coverage and assume that the generator will discard the excess.

The fixed 20 percent policy deserves particular scrutiny. Percentages are convenient for storage sampling, but prompt capacity depends on absolute tokens. Model reliability also changes with query complexity, evidence structure, and the generator being used.

A better budget would adapt to evidence need. A direct lookup might require one supported claim. A comparison might require several alternatives. A multi-step agent plan might require a dependency chain plus explicit contradictions.

That suggests a staged retrieval process. The first pass should retrieve a small, high-precision core. A second pass should expand only when the answer lacks support, contains uncertainty, or requires another reasoning hop.

The expansion decision needs measurable criteria. A model can identify unsupported claims, but self-reported confidence alone is unreliable. More dependable signals include missing citations, unresolved entities, contradictory timestamps, and failed answerability checks.

The system should also separate stable memory from episodic memory. Stable memory contains durable preferences, policies, and verified facts. Episodic memory records events, transient observations, and prior steps whose relevance decays.

Without that separation, a theme selector can mix permanent rules with temporary state. An agent might follow an old workaround after the underlying incident ends. Temporal metadata should therefore influence selection before text reaches the model.

Authority matters as much as recency. A user instruction should outrank an agent-generated summary of that instruction. A verified tool result should outrank a speculative plan. Theme similarity alone cannot express those priorities.

The best sentence retrieval method will likely combine several signals. These include lexical matching, semantic similarity, dependency coverage, time, authority, contradiction, and estimated task utility. CELF can still perform final set selection if the objective incorporates these distinctions.

This does not make the trie irrelevant. The storage structure determines lookup speed, memory overhead, update behavior, and available relationships. It simply means storage efficiency and answer reliability belong to different evaluation layers.

Smaller Models Expose the Retrieval Failure First

Smaller models are not merely weaker generators here; they act as stress tests for whether the retrieval layer has separated evidence from thematic noise.

A large model can sometimes recover from a cluttered prompt through stronger instruction following and better contextual discrimination. That tolerance can hide weaknesses in the retriever. The same context can overwhelm a smaller model immediately.

The developer’s observation about hallucination therefore deserves careful interpretation. Excess retrieval may correlate with unsupported answers, but the post does not establish causation. Other sources include weak prompts, missing evidence, contradictory memories, decoding settings, or model-specific limitations.

The term hallucination can also collapse several failures. A model might invent a fact, merge two memories, follow outdated instructions, or choose the wrong retrieved alternative. Each failure requires a different measurement and potentially a different fix.

SALT needs an error taxonomy before changing its selector. Every failed answer should identify whether the required evidence was absent, present but ignored, contradicted, incomplete, or overwhelmed by distractors.

The evaluation should compare at least four retrieval conditions. One should provide no external memory. Another should provide only an oracle evidence set selected by a human. A third should use SALT’s current output, and a fourth should use an aggressively pruned output.

This comparison separates retrieval from generation. If the small model fails with the oracle set, changing CELF weights will not solve the core problem. If it succeeds with oracle evidence but fails with SALT output, precision becomes the leading target.

The benchmark should preserve realistic task categories. Direct factual questions test exact recall. Multi-hop questions test dependency completeness. Agent tasks test whether retrieved memory leads to the correct tool choice, parameters, and stopping condition.

Each category needs negative cases. The corpus should contain plausible but irrelevant sentences from the same theme, outdated versions of true facts, explicit contradictions, and duplicated paraphrases. Easy random distractors will overstate system quality.

The evaluation must also vary evidence position. The context-limit results for Gemini 2.5 Flash show that newer models can handle simple needle retrieval across long contexts far better than earlier systems. That finding is an important counterweight to broad claims about universal context failure.

However, simple fact retrieval is not the same as reasoning across competing memories. A model can locate one planted fact while still struggling with several related claims, exceptions, and temporal changes. SALT’s agent use case belongs closer to the second category.

Testing should therefore cross model size with context composition. The same retrieved set should go to a smaller local model, a stronger model, and an oracle-style evaluator. Differences will show whether improvements come from cleaner memory or greater generator capacity.

Precision should be reported at several levels. Sentence precision counts how many retrieved sentences prove useful. Claim precision counts supported propositions. Action precision measures whether an agent chooses the correct operation and parameters.

Recall must remain visible. Pruning everything except one obvious sentence can raise precision while destroying multi-hop completeness. A safe selector should identify the smallest sufficient evidence set, not simply the smallest set.

“Sufficient” means the set supports the correct result and retains necessary qualifiers. It should also include decisive contradictions when the memory contains conflicting claims. Otherwise, a compact prompt can become confidently wrong.

Ablation testing can reveal which SALT components help. Researchers should disable theme coverage, change the percentage budget, cap absolute tokens, remove duplicates, add recency, and add authority weighting separately.

These experiments should use identical stored memories and queries. Changing the corpus between runs would make comparisons difficult. Repeating trials also matters when generation uses sampling.

Latency and memory use should remain secondary metrics, not disappear. A reranker that improves accuracy but adds unacceptable delay can undermine an interactive agent. The goal is a measured operating frontier across accuracy, tokens, latency, and DRAM.

The developer should also log what each retrieved sentence contributed. A concise reason code can identify lexical match, new claim, contradiction, temporal dependency, or source authority. Those logs make excessive retrieval diagnosable without asking the generator to explain itself.

One important risk remains unverified. There is no public benchmark establishing SALT’s current chatbot accuracy, memory compression, or agent performance. The architecture should be discussed as an early project report, not a validated advance.

Three Signals Will Show Whether SALT Can Scale to Agents

SALT’s next milestone should be a reproducible precision result, not a larger memory store or a more generous context budget.

The first signal is an oracle-gap benchmark. The developer should compare current retrieval against a minimal human-selected evidence set across direct, multi-hop, and agent-action tasks. Results should be broken down by model size.

If SALT approaches oracle performance while using fewer tokens, the evidence will support its theme-selection strategy. If the gap expands for smaller models, the current objective is selecting breadth that the generator cannot use.

The second signal is adaptive budgeting. A fixed 20 percent policy should compete against absolute token caps and staged retrieval. The comparison should measure answer accuracy, action success, retrieved claims, latency, and unsupported assertions.

An adaptive policy wins only if it preserves complete evidence chains. Lower token counts alone would weaken the system if they remove exceptions or dependencies. The strongest result would improve precision without reducing task-level recall.

The third signal is module-aware provenance during agent trials. Each memory should identify its originating module, timestamp, authority, and source material. Tests should include conflicting and outdated memories from different agents.

If provenance-aware selection reduces wrong actions, multi-agent memory needs more than theme dominance. It needs explicit rules for authority, freshness, and contradiction. If provenance has little effect, the main issue likely sits elsewhere in ranking or generation.

These signals should appear within an open evaluation package. The package needs fixed queries, labeled evidence, retrieval logs, generated outputs, model settings, and scoring rules. Without those artifacts, outside contributors cannot isolate why a proposed change works.

Community suggestions can still be useful before a full benchmark exists. The project could test maximal marginal relevance, cross-encoder reranking, claim clustering, and query-focused compression. However, no method should replace the current selector based on anecdotes alone.

The strongest near-term design is likely conservative. Retrieve a compact evidence core, preserve attached qualifiers, and expand only after detecting a specific information gap. Route selected evidence according to each module’s task instead of broadcasting one broad context.

Agent systems also need memory hygiene. They should merge duplicates, expire temporary state, preserve original sources, and distinguish observations from conclusions. Otherwise, retrieval quality will degrade as agents repeatedly store derivatives of earlier outputs.

That pattern resembles lossy copying. A tool produces a fact, one agent summarizes it, another agent summarizes that summary, and the memory system stores every version. Theme coverage may reward these items as consistent evidence despite their common origin.

Provenance-aware deduplication can prevent apparent consensus from repeated copies. The selector should group derived memories under their original evidence. It can then avoid spending prompt tokens on several paraphrases of one fact.

Teams building a personal knowledge system face a related challenge. Capturing everything is useful only when recall preserves relevance, source boundaries, and time. Agent memory magnifies the cost of getting those distinctions wrong.

For the horizon machinelearning debate, that is the decisive point. SALT does not need to prove that a trie can store extensive conversational memory. It needs to show that its selector can recover the smallest sufficient evidence set as memory and module counts grow.

The next useful contribution is therefore measurable: publish a failure set, label the required evidence, and compare retrieval policies under identical model conditions. Which policy keeps small models accurate without hiding necessary facts?