Meituan LongCat Releases LoHoSearch, and Search Agents Hit a New Difficulty Wall

Meituan LongCat has released LoHoSearch, and its results challenge the idea that leading search agents are approaching mastery. The Meituan LongCat LoHoSearch benchmark contains 544 questions, yet the best tested model answered only 34.74% correctly.

That result looks especially stark beside BrowseComp, the benchmark that helped define difficult web research during the previous year. Leading systems now reportedly exceed 90% on BrowseComp, limiting its ability to distinguish between them.

LoHoSearch attempts to reset that measuring scale. Instead of asking people to invent harder questions, its creators generated complex problems from a Wikipedia knowledge graph containing 7.62 million entities.

The benchmark matters because search agents increasingly support research, analysis, and knowledge work. A system that retrieves one plausible page can appear capable. A system that must follow dozens of uncertain leads exposes very different weaknesses.

Meituan publicized the project through its LongCat account after releasing the paper and dataset. The underlying LoHoSearch paper lists six Meituan researchers and carries a June 2026 revision date.

The headline score is not an independent verdict on every model or commercial research product. It comes from one benchmark, one tool setup, and automated grading. Still, the size of the performance collapse deserves attention.

The Meituan LongCat LoHoSearch Benchmark Changes the Test

LoHoSearch changes benchmark difficulty by controlling the search space, not merely adding more reasoning steps.

The benchmark includes 544 human-reviewed English questions across 11 domains. Those domains include music, sports, film and television, geography, and other general knowledge categories.

Its evaluation set contains 282 tree-structured questions and 262 graph-structured questions. A tree question follows a hierarchy of clues. A graph question adds cross-connections and cycles that resist clean decomposition.

This distinction matters because many difficult-looking questions still have a narrow route to the answer. An agent finds one recognizable entity, follows a few links, and verifies the result.

LoHoSearch tries to remove that shortcut. Its questions hide low-popularity entities behind several indirect relationships, while numerous candidates satisfy each individual clue.

The benchmark begins with the full English Wikipedia dump. Each Wikipedia page becomes an entity, while links between pages become directed relationships.

Wikidata classifications supply entity types. The resulting knowledge graph contains approximately 7.62 million entities and 265 million directed edges.

The researchers use this global graph to find relationships with large candidate pools. Human writers rarely know how many entities share a relationship across Wikipedia, but a graph can calculate that scale.

For tree-structured questions, the pipeline selects a hidden answer connected to several intermediate entities. Each intermediate entity connects to additional leaf entities or properties.

Every major relationship must remain necessary. Removing one should leave multiple possible answers, while combining all relationships should identify a single answer inside the graph.

Graph-structured questions are harder to separate into smaller tasks. They can contain cycles and cross-constraints among as many as ten entities.

These structures averaged ten nodes and 12.5 edges. Tree structures averaged 7.9 nodes and 6.9 edges.

A language model then converts the sampled structure into a natural-language question. Entity names stay hidden, and relationship descriptions are deliberately obscured.

The pipeline tests whether individual descriptions can be found directly through search. It also checks whether the descriptions become too revealing when combined.

DeepSeek-V3.2 handled the language-model stages, according to the paper. Those stages included relationship extraction, question generation, and automated validation.

Questions then passed coverage and answer-satisfaction checks. Multiple search agents looked for alternative valid answers, while professional annotators reviewed the surviving items.

The publicly available LoHoSearch dataset includes the 544-item benchmark and 2,000 automatically generated training questions. Only the benchmark questions received the stated human verification.

This automated construction method is the real event. A low score alone can be produced by obscure trivia, ambiguous wording, or broken evaluation.

LoHoSearch instead proposes a repeatable difficulty generator. Its parameters can increase candidate volume, relationship complexity, and dependence between clues.

That makes the benchmark more than another leaderboard. It presents a method for producing new tests when existing ones stop separating leading systems.

BrowseComp Success No Longer Settles the Search-Agent Question

The main conflict is between performance on a familiar benchmark and reliability during much longer, less structured searches.

OpenAI introduced BrowseComp as a set of difficult questions whose answers could be checked concisely. Its expert-written prompts were designed to resist ordinary search and require persistent browsing.

The original BrowseComp research reported 1,266 questions. OpenAI Deep Research scored 51.5%, while human participants scored 33.3% under the study’s conditions.

That result made BrowseComp useful because it presented substantial headroom. It rewarded agents that could reformulate queries, inspect pages, connect clues, and continue after failed searches.

According to Meituan’s researchers, performance on BrowseComp rose from roughly 30% to above 90% within about ten months. That rapid rise created a benchmark saturation problem.

Saturation does not mean every web research task has been solved. It means the benchmark provides less information when several advanced systems cluster near its ceiling.

LoHoSearch applies pressure to model developers making broad claims about deep research. It asks whether progress reflects general search ability or adaptation to recognizable benchmark patterns.

The comparison using DeepSeek-V4-Flash illustrates the gap. That model scored 58.84% on BrowseComp under Meituan’s experiment, but only 10.02% on LoHoSearch.

Correct LoHoSearch trajectories also required much more tool activity. Mean tool calls rose from 35 on BrowseComp to 61 on LoHoSearch, a 74% relative increase.

The median changed even more sharply, rising from 26 tool calls to 59. A few unusually long runs therefore do not explain the entire difference.

Long searches create compounding problems. An agent must remember which candidates failed, preserve evidence, track unsatisfied conditions, and avoid returning to exhausted paths.

Each search result can also introduce new entities. The agent must decide whether a new lead narrows the answer or merely expands the investigation.

That process resembles real research more closely than retrieving a single known fact. Analysts often start with incomplete descriptions, conflicting terminology, and evidence spread across unrelated sources.

However, LoHoSearch remains a synthetic construction derived from Wikipedia relationships. It does not reproduce every difficulty found in business research, legal review, scientific analysis, or current-event verification.

Its value lies in isolating a particular capability. It tests whether an agent can control a wide, interdependent search without losing the original constraints.

This is why the benchmark pressures agent architectures as much as base models. A stronger model cannot help if its surrounding system forgets evidence or stops searching too early.

Developers may need explicit candidate tracking, condition-level verification, and durable research memory. They may also need better ways to divide a graph-shaped question among parallel workers.

For knowledge workers, the lesson is practical. A polished answer and several citations do not show that an agent exhausted plausible alternatives.

Users evaluating research systems should inspect the search process. They should ask whether the system preserved failed paths, checked every condition, and separated confidence from fluency.

A searchable knowledge base can preserve source material, but retrieval alone does not solve the coordination problem. The agent must still connect and verify the evidence correctly.

The Meituan LongCat LoHoSearch benchmark therefore reframes the race. The critical metric is no longer whether an agent can browse, but whether it can sustain disciplined browsing across dozens of uncertain steps.

The Real Failure Appears in Context Management

LoHoSearch suggests that longer context windows do not automatically produce coherent long-horizon research.

Every evaluated model received the same two basic tools. The search tool submitted keyword queries, while the browse tool retrieved content from selected URLs.

The researchers also used the BrowseComp system prompt. Models operated with a 200,000-token context allocation, including 184,000 input tokens and 16,000 output tokens.

That is enough space for a substantial research trail. Yet available context and usable memory are not the same thing.

An agent can fill its window with duplicated results, abandoned hypotheses, and irrelevant page text. More tokens then create a larger organization problem.

The paper tested several context-management strategies using DeepSeek-V4-Flash within a ReAct framework. ReAct alternates reasoning steps with tool actions, allowing a model to search and update its plan.

The first strategy summarized the trajectory after token usage crossed 80% of the context window. It then restarted the search using that compressed account.

A second strategy discarded previous tool calls and restarted from the original question. Both approaches could also use a verification module before submitting an answer.

The baseline LoHoSearch score was 10.02%. Summarization raised it to 11.31%, while discarding the history raised it to 12.41%.

Adding verification helped more. Summarization with verification reached 15.35%, while discard-all with verification reached 16.82%.

That best result represents a 6.8 percentage-point gain over the baseline. The same family of interventions added 14.03 points on BrowseComp in the paper’s comparison.

The difference reveals a basic limitation. Compressing a trajectory can remove the evidence needed to reject candidates later.

Discarding it creates the opposite problem. The agent gets a clean context, but it can repeat searches and rediscover the same dead ends.

A verification module catches some premature conclusions. It does not necessarily tell the agent where an incomplete investigation went wrong.

This helps explain why context management is becoming a central agent design problem. Long-horizon search requires structured state, not just a longer transcript.

A useful state representation might record candidate entities, supporting evidence, rejected relationships, open conditions, and source quality separately. Plain conversation history mixes them together.

Graph-structured questions make this weakness more visible. DeepSeek-V4-Flash scored 8.01% on those items, compared with 11.89% on tree-structured questions.

Trees can be divided into branches. An agent can solve each branch and combine the results.

Graphs contain dependencies across branches. Evidence found for one entity can change how several other clues should be interpreted.

That requires revisiting earlier conclusions without erasing them. It also requires knowing which claims depend on which sources.

These are familiar problems in serious knowledge work. A researcher may collect the right documents yet fail to maintain the relationships among their contents.

Personal information systems face a related challenge. Capturing material is only the first step, while knowledge blending connects evidence from separate sources without flattening their context.

For agent developers, LoHoSearch points toward several architectural priorities. Search history should become a queryable workspace rather than an ever-growing prompt.

Verification should operate at the condition level. The agent should show which requirements remain open before selecting an answer.

Parallel search also needs coordination. Multiple trajectories can cover more ground, but only if the system can compare their evidence and detect incompatible conclusions.

The paper’s repeated-sampling experiment supports that view. Sixteen independent DeepSeek-V4-Flash responses produced a theoretical pass rate that rose from 9.3% to 38.3%.

Yet selecting a usable answer remained difficult. The best confidence-based selection method reached 24.6%, below the available pass-rate ceiling.

Generating a correct answer somewhere in the batch is therefore insufficient. The system must identify that answer reliably among plausible alternatives.

What the 34.74% Result Does Not Prove

LoHoSearch exposes a large capability gap, but its leaderboard should not be treated as a universal ranking of AI systems.

The researchers evaluated 11 models from several major families. GPT-5.5 led with 34.74%, followed by DeepSeek-V4-Pro at 15.99%.

Claude Opus 4.6 scored 15.62%, while Kimi-K2.6 reached 15.53%. Gemini 3.1 Pro scored 13.32%, and GLM-5.1 scored 12.77%.

Claude Opus 4.7 reached 10.29%, while DeepSeek-V4-Flash scored 10.02%. LongCat-Flash-Thinking-2601 recorded 9.74%.

MiniMax-M2.7 and MiniMax-M2.5 scored 2.48% and 2.29%, respectively. These figures come from the benchmark authors’ experiment and have not been independently replicated here.

The unusually wide gap between first and second place deserves investigation. It can reflect model capability, but tool behavior and provider implementation can also influence results.

Meituan notes that DeepSeek-V4-Pro and Kimi-K2.6 encountered service instability or safety refusals. Those issues may have depressed their scores.

Models also used their default thinking settings. A uniform prompt and context limit improve comparability, but they do not guarantee optimal configuration for every system.

The benchmark uses automated judging. Each response received one assessment using a BrowseComp prompt with GPT-4.1 and another using a SimpleQA prompt with Qwen2.5-32B.

The final score averages those two judgments. This reduces dependence on one judge, but it does not eliminate grading noise.

Some answers can be semantically correct yet phrased differently from the reference. Others may name a plausible entity without satisfying every hidden condition.

The paper also reports high confidence-calibration errors. However, some models failed to follow the requested confidence format, which introduced additional noise.

Question validity presents another limitation. Human annotators definitively confirmed a unique answer for 70.8% of the benchmark.

For the remaining 29.2%, annotators found no alternative after searching but could not conclusively rule one out. The knowledge graph guarantees uniqueness only within its representation.

Wikipedia can omit links, contain stale statements, or describe relationships inconsistently. A valid alternative might exist outside the graph even when no reviewer discovers it.

The construction pipeline also filters difficulty using DeepSeek-V3.2. Questions that model answered correctly across several attempts were removed.

That choice may create model-family bias. The remaining questions could emphasize failure patterns specific to DeepSeek-V3.2 or closely related search behavior.

Static benchmarks face contamination and temporal drift as well. Once questions circulate, model developers can encounter them during evaluation, training, or prompt optimization.

The dataset includes a canary warning intended to discourage accidental training use. Such markers help detect exposure, but they cannot prevent every form of contamination.

There is also a broader question about ecological validity. LoHoSearch rewards finding one entity that satisfies an intricate collection of clues.

Professional research often values a different output. Users may need a qualified synthesis, several defensible options, or an explanation of uncertainty.

An agent that struggles with LoHoSearch can still perform useful document summarization. Conversely, a benchmark leader might still fail on live websites, private data, or time-sensitive claims.

The benchmark should therefore supplement other evaluations. It should not replace tests involving source quality, factual consistency, task completion, latency, and human usefulness.

Its strongest contribution is diagnostic. It creates controlled pressure on candidate elimination, structural reasoning, and long-session memory.

The 34.74% result is best read as evidence that these abilities remain uneven. It is not proof that one model dominates every research setting.

Three Signals Will Show Whether LoHoSearch Matters

LoHoSearch will matter if independent replications, stronger agent memory, and refreshed benchmark versions preserve its diagnostic value.

The first signal is independent reproduction. Research groups and model providers need to rerun the benchmark with documented tools, prompts, access dates, and inference settings.

A stable ranking across replications would strengthen Meituan’s central claim. Large changes would show that the benchmark measures harness configuration alongside underlying model ability.

Replication should also examine the automated judges. Human review of a statistically meaningful result sample would reveal whether grading errors affect the leaderboard materially.

The second signal is architectural progress in context management. The paper’s best intervention improved DeepSeek-V4-Flash by only 6.8 percentage points.

A system that maintains explicit candidates and condition-level evidence should outperform transcript summarization. That result would support the benchmark’s diagnosis rather than merely raising one score.

Researchers should report tool calls, repeated searches, abandoned paths, and verification failures. Accuracy alone cannot show why one architecture handled the long search better.

The third signal is benchmark renewal. Meituan says the generation pipeline can adjust search-space size and structural complexity, which should support refreshed versions.

A regenerated set would test whether improvements transfer to unseen structures. It would also reduce the risk that public questions become familiar to evaluated systems.

Multilingual variants would provide another important test. The current release covers English, although its creators describe the construction process as language-agnostic.

A translated benchmark alone would not be enough. New versions should rebuild relationships from language-specific sources and account for different search environments.

The open MIT-licensed release gives researchers room to inspect the data and build alternative evaluation harnesses. That openness also accelerates exposure, making refreshes more urgent.

Developers should watch whether performance gains come from better models, more sampling, or improved coordination. Those routes have different implications for cost, latency, and reliability.

Repeated sampling already shows hidden headroom. However, the gap between pass rates and answer selection proves that spending more computation does not guarantee a dependable final result.

Enterprise buyers should ask vendors for evaluations on tasks resembling their own workflows. A generic benchmark score cannot replace testing against internal documents, terminology, and verification requirements.

Knowledge workers should treat agent output as a research trail, not just a response. The most useful systems will make unresolved conditions and discarded alternatives visible.

Meituan LongCat has not shown that LoHoSearch is the final search-agent benchmark. It has shown why a high score on yesterday’s benchmark cannot settle today’s reliability question.

The next few months should reveal whether other teams reproduce the 34.74% ceiling, whether structured memory breaks through it, and whether regenerated questions remain difficult.

Those results will determine whether LoHoSearch becomes a durable standard or a temporary stress test. Until then, its warning is clear: fluent browsing is not the same as controlled research.

When evaluating a search agent, ask it to preserve evidence across a long investigation. Check whether every condition receives support and whether rejected candidates stay rejected.

Then ask the system to explain what remains uncertain. That behavior matters more than a confident final sentence, especially when decisions depend on obscure or scattered evidence.

The Meituan LongCat LoHoSearch benchmark gives developers a demanding place to test those habits. Its lasting value will depend on whether the wider research community can verify, extend, and repeatedly defeat it.