LangSmith AI Search Finds Agent Failures That Traditional Trace Filters Miss

LangSmith released AI Search in public beta, giving teams a new way to find agent failures that ordinary filters miss. The feature accepts natural-language descriptions of behavior, then searches recorded agent interactions for matching traces.

A developer can ask for cases where a user became frustrated, an agent claimed success without completing its task, or personal information appeared in a response. Those questions depend on meaning and sequence, not a known tag or exact phrase.

That distinction creates the real tension behind LangSmith AI Search. Agent observability has depended largely on structured filters, full-text queries, and evaluators configured before teams know which failures matter. LangSmith is betting that a model can investigate unfamiliar behavior when the question appears.

The approach also introduces a less comfortable question. If another model decides which production conversations match a subjective description, teams must understand what that model reads, excludes, and misclassifies.

LangSmith AI Search Adds Behavioral Queries to Trace Exploration

The release turns trace investigation from a database query problem into an interpretation problem.

LangChain announced the public beta on October 13, 2026. It is available through LangSmith Cloud in Google Cloud's US, European, and Asia-Pacific regions. An organization administrator must enable it before other members can use it.

LangSmith already lets teams filter traces through fields such as status, latency, tags, metadata, costs, and feedback. Its full-text search can also match words found inside trace inputs, outputs, metadata, and errors.

Those tools work when an investigator knows what field or phrase identifies the target. A status filter can find errors, while a text query can locate conversations containing "refund." Neither can reliably decide whether a user silently gave up after receiving an unhelpful answer.

LangSmith AI Search accepts a behavioral description instead. The company's behavioral trace search examples include users showing frustration, agents exposing internal implementation details, and agents failing to complete assigned work.

The feature works across threads, traces, and individual runs. A trace records one request and its nested operations, while a thread connects multiple traces from the same conversation.

Teams can combine the natural-language query with existing structured filters. They might first limit results to production traces with errors, then ask which conversations show users becoming confused during onboarding.

That combination matters. It lets conventional filters reduce the search area before a model examines the remaining conversations. It also preserves deterministic controls for attributes that should not require interpretation.

Results arrive with confidence scores. Opening a match takes the reviewer into a trajectory view, where LangSmith highlights available messages that influenced the match.

A trajectory is a normalized sequence of human messages, agent responses, tool calls, and tool results. It presents the session once instead of repeating the conversation history included in successive model calls.

This is not merely a friendlier syntax layer over existing filters. LangSmith says the decision model reads each candidate trajectory and evaluates whether the requested behavior occurred.

The difference becomes clear with a request such as "the agent said it completed the task but did not." Both a successful and unsuccessful trace can contain nearly identical topics, tools, and vocabulary.

An embedding, which represents text as a numerical vector, can place those conversations close together because they discuss the same task. Behavioral classification must instead compare the claim with the actions recorded across the trace.

The company has therefore moved an LLM-as-a-judge pattern directly into interactive trace discovery. Engineers no longer need to define every evaluator before they start investigating.

That shift makes exploratory analysis faster. It does not make every result authoritative.

Why Agent Trace Search Has Become a Data Problem

Long agent sessions contain valuable evidence, but their scale and repetition make manual review increasingly impractical.

Traditional application monitoring focuses on predictable signals. Engineers search for error codes, slow requests, failed database calls, or unusual resource consumption. Those signals usually fit structured fields.

AI agents produce a different failure surface. An agent can return a technically successful response while misunderstanding the request, calling an inappropriate tool, or overstating what it accomplished.

The resulting telemetry is also unusually dense. One agent task can contain user messages, model outputs, retrieved documents, shell commands, tool responses, retries, and handoffs to other agents.

LangChain examined 150 randomly selected conversations from Open SWE, its open-source coding agent. According to the company, several exceeded one million tokens.

The median conversation contained about 72,000 tokens. Only around 2,000 came from user messages and final agent replies, while approximately 85 percent consisted of tool output.

Those figures come from LangChain's sample, not an independent industry benchmark. They still illustrate why searching every raw event creates both context and inference costs.

Agents often send the full conversation history with each new model call. A naïve trace representation therefore stores the same earlier messages repeatedly.

LangSmith addresses that repetition through trajectories. Its trace data layer converts supported trace formats into a common representation as data arrives.

The formats include traces produced through LangChain, LangGraph, OpenAI and Anthropic SDKs, the Vercel AI SDK, and OpenTelemetry. Normalization gives the search system a consistent sequence to inspect.

This also expands the competitive stakes. Observability vendors increasingly offer trace-level evaluation, not merely storage and filtering.

Datadog, for example, documents trace-level evaluations that place every span from a trace into an LLM judge prompt. Those evaluations can classify goal completion, tool order, or conversation quality.

Arize Phoenix similarly combines tracing, annotations, evaluations, datasets, and experiments in an open-source tracing workflow. Its evaluators can score production traces and convert selected runs into datasets for later testing.

The distinction is mainly about timing and workflow. Evaluators usually represent criteria that a team defines, tests, and applies repeatedly. LangSmith AI Search targets questions that emerge during an investigation.

Consider a support agent that receives a sudden rise in cancellation requests. A structured dashboard might show stable latency and few explicit errors. Full-text search might identify the word "cancel," but not the reason behind it.

A product manager could instead search for users who tried to cancel after receiving contradictory answers. That description combines intent, sequence, and perceived quality.

If the matches are useful, the team can review them, add examples to a dataset, and build a durable evaluator. Search becomes the exploratory front end for a more formal quality process.

That workflow pressures observability platforms to connect discovery with evaluation. Storing detailed traces is no longer enough when customers cannot retrieve the behavior hidden inside them.

It also changes which teams can use trace data. Product managers, safety reviewers, and support leaders can describe a problem without first learning a platform-specific query language.

The technical query layer remains necessary. Natural language simply moves the first investigative step closer to the people who recognize the business problem.

How LangSmith AI Search Reads a Trace Without Reading Everything

LangSmith reduces the search cost by prioritizing conversational evidence, but that compression determines which failures remain visible.

When a user submits an AI Search description, existing filters first narrow the candidate set. SmithDB then prepares trajectories while a decision model checks them in parallel.

Matches stream into the interface as the system identifies them. Users do not need to wait until the entire candidate set has been processed.

The central mechanism is an 8,000-token context budget for each trajectory. LangSmith fills that budget in a fixed priority order rather than sending every recorded token.

User messages and the agent's direct replies receive the highest priority. Intermediate model messages, tool calls, tool results, and system messages fill the remaining space.

If the next message does not fit, LangSmith trims it. Later content falls outside the model's view once the budget is full.

This choice makes many conversational searches practical. Frustration, refusal, confusion, cancellation intent, and unsupported promises often appear in the visible dialogue.

It also creates a predictable blind spot. Some failures exist only inside tool output or intermediate execution steps.

An agent might encounter an authorization error, ignore it, and still tell the user that a task succeeded. The visible reply contains the false claim, but the decisive evidence sits inside a tool result.

The search model can miss that evidence when earlier, higher-priority content consumes the token budget. LangChain acknowledges this limitation and says it is exploring query-dependent context selection.

Such selection would change the evidence order according to the question. A query about ignored tool errors should prioritize tool responses, while a query about user sentiment should emphasize the conversation.

The current release instead uses one general ordering. That is understandable for a beta, but teams should treat the resulting coverage as query-dependent.

The mechanism also explains why LangSmith did not rely only on vector search. Vector retrieval works well when the goal is finding material about a topic.

A conversation about billing will likely sit near other billing conversations in an embedding space. However, successful and failed billing interactions can remain semantically similar.

Behavioral queries need a judgment about what occurred. A decision model evaluates the trajectory against one narrow proposition, such as whether the agent completed the requested task.

That model-based approach improves expressiveness, but it replaces one retrieval uncertainty with another. The system now depends on a model's interpretation of ambiguous language.

"User frustration" can include direct complaints, repeated questions, sarcasm, abrupt departures, or polite rejection. Different reviewers might disagree about borderline cases.

Confidence scores help rank results, but they do not establish ground truth. A high model score still reflects the behavior definition, available context, and judge model.

The strongest operational pattern therefore includes human review. Teams can inspect highlighted evidence, correct weak matches, and refine the description before treating the result set as representative.

For recurring risks, they can convert reviewed cases into an evaluation dataset. LangSmith's existing online evaluators can then apply a defined criterion to future traces.

This separates exploration from monitoring. AI Search helps teams discover a pattern, while an evaluator measures that pattern consistently after its definition stabilizes.

That separation is especially important for security and compliance. A natural-language search for personal information can surface suspicious traces, but it should not replace deterministic scanners or formal controls.

Model judgment can widen an investigation. It should not become the sole enforcement layer for high-consequence decisions.

The Main Competition Is Predefined Evaluation

LangSmith is challenging the assumption that teams must know every important failure before production traffic reveals it.

Before behavioral search, teams generally had three ways to locate problematic traces. They could search text, filter known attributes, or run an evaluator against selected traffic.

Full-text search provides speed and reproducibility. It is useful when the relevant evidence includes a distinctive phrase, identifier, error message, or product name.

Structured filtering is even more precise. Teams can isolate traces with failed status codes, long latency, expensive model calls, specific tools, or low feedback scores.

Predefined evaluators handle meaning. An LLM judge can decide whether an answer was helpful, grounded, safe, or complete, then attach the result to each trace.

The evaluator route is effective when a team already understands the failure category. It becomes less convenient during an unexpected incident.

Someone must define the rubric, select the required trace fields, choose sampling rules, test the evaluator, and apply it to past or future traffic. That preparation creates durable measurement, but it slows open-ended investigation.

LangSmith AI Search compresses that work into a description entered at query time. It offers a faster route from suspicion to candidate evidence.

The feature does not eliminate evaluators. It creates a discovery stage before them.

A team might notice an increase in support escalations without knowing why. An analyst could search for users who repeated a request after the agent claimed resolution.

Reviewers might discover that the agent regularly interpreted account closure as subscription cancellation. That pattern can then become a named evaluator with a tested rubric.

This sequence is the most consequential part of the launch. It connects qualitative exploration, dataset construction, evaluation, and production monitoring.

It also creates pressure beyond LangSmith's direct competitors. Internal analytics teams often export traces into warehouses, notebooks, or custom search systems because product questions exceed observability dashboards.

Behavioral trace search could keep more of that analysis inside the observability platform. Product and engineering teams would work from the same underlying execution record.

The approach resembles the broader shift from keyword search to model-assisted retrieval in enterprise knowledge systems. Both attempt to find meaning that users did not encode as metadata.

The difference is that agent traces contain actions, not only documents. A useful result depends on reconstructing what happened across messages and tools.

For engineering teams, that makes trace organization part of the product's quality infrastructure. The same principle applies when teams build a searchable knowledge base from fragmented technical records.

The advantage will belong to systems that preserve evidence while reducing investigative friction. Natural-language access helps, but reviewability matters just as much.

LangSmith highlights matched messages inside the trajectory view. That design gives reviewers a path from the model's classification back to the recorded conversation.

However, the documentation notes that not every match can be highlighted. Repeated text or messages absent from the trajectory view can prevent evidence localization.

A result can therefore carry a confidence score without displaying every reason behind it. Teams should distinguish between model confidence and auditable evidence.

Competitors can respond by making their own evaluator interfaces more conversational. They can also expose ad hoc judges, saved behavioral queries, or automatic dataset generation.

The product contest is not simply who offers natural-language search first. It is who turns uncertain discoveries into repeatable tests with the fewest gaps.

Context, Privacy, and Reproducibility Define the Beta's Limits

The feature's usefulness depends on whether teams understand the data it excludes and the governance obligations it adds.

LangSmith's filter documentation identifies several product limitations. AI Search descriptions cannot currently be saved as views or preserved in copied page links.

Searches also do not appear in recent-filter history. The statistics panel becomes unavailable while an AI Search query is active.

Those restrictions affect reproducibility. An engineer investigating an incident cannot simply send a saved behavioral query to another reviewer and expect an identical workflow.

Teams can record the wording elsewhere, but that adds manual process. It also matters because small wording changes can influence model classifications.

The system applies rate limits at the workspace level. LangChain does not publish the threshold in the referenced documentation, so teams cannot yet infer suitability for frequent automated investigation.

More important limits come from the trajectory itself. AI Search reads the main agent conversation, but not subagent, middleware, or compaction message content.

A multi-agent workflow can therefore hide decisive behavior outside the searchable representation. The parent agent might report success while a subagent records the actual failure.

The system also cannot infer run status, tags, metadata, or trace structure from the natural-language description. Users must apply conventional filters for those properties.

That division should appear in team playbooks. Behavioral search handles meaning inside the trajectory, while structured filters handle known operational attributes.

Privacy presents a separate constraint. Enabling AI Search permits trace data to be transferred to and processed in the United States through LangChain-managed inference and subprocessors.

The organization administrator enables the feature for every member. No separate model-provider key is required.

LangChain explicitly says customers should not use AI Search for protected health information. The feature is not covered by any Business Associate Agreement an organization has signed with LangChain.

That warning deserves attention because the launch highlights searches for leaked personally identifiable information. Searching for sensitive data requires sending the relevant trace content through the classification system.

Organizations must distinguish general PII review from regulated health-data processing. They also need to confirm retention, access, residency, and subprocessor requirements before enabling the feature.

A broader observability risk remains regardless of vendor. Rich traces can capture prompts, model responses, customer records, retrieved documents, and tool results.

More complete telemetry improves debugging while increasing the consequences of excessive collection. Teams should minimize sensitive fields before they reach any observability platform.

They should also test false positives and false negatives against reviewed samples. A useful beta benchmark would include known behavioral incidents, difficult negatives, and examples spanning different agent frameworks.

The public material does not report precision, recall, latency distributions, or independent comparisons with embeddings and predefined evaluators. It describes the architecture and an internal trace-length sample.

That leaves buyers without a common measure of search quality. A feature that returns plausible matches can still overlook rare failures that matter most.

The right interpretation is therefore narrow. LangSmith has introduced a promising investigation interface, not a verified replacement for evaluation, security scanning, or human review.

Its value will depend on how quickly users can validate a result, reproduce the query, and convert the discovery into a durable quality control.

What to Watch After the LangSmith AI Search Beta

Three signals will show whether behavioral trace search becomes core infrastructure or remains a useful investigative shortcut.

The first signal is adaptive context selection. LangChain says it is exploring methods that choose evidence according to the query and indicate when the model needs a second pass.

That work directly addresses the 8,000-token limit. If tool-focused searches begin prioritizing tool results, the system should detect more execution failures hidden from conversational evidence.

Without adaptive selection, users will need separate approaches for different trace types. Conversational searches may work well while operational searches produce inconsistent coverage.

The second signal is reproducibility. Saved searches, shareable queries, stable result sets, and documented rate limits would make AI Search easier to integrate into incident response.

Grouping similar matches would also help teams move beyond individual traces. LangChain says it wants to summarize common patterns instead of returning only a flat list.

That capability could turn an ad hoc search into a product insight workflow. Teams could identify recurring failure clusters, inspect representative traces, and create targeted evaluators.

The third signal is competitive response. Datadog, Arize, and other observability providers already combine tracing with model-based evaluation.

If they add interactive behavioral discovery, natural-language trace search will become an expected interface. Competition would then shift toward evidence quality, governance, and integration with datasets.

If competitors emphasize evaluated precision and repeatable monitoring instead, LangSmith will need to show that exploration produces measurable improvements after deployment.

Users should also watch for published search-quality benchmarks. Precision and recall across frustration, incomplete tasks, unsafe disclosure, and ignored tool errors would provide a clearer basis for comparison.

For now, teams evaluating LangSmith AI Search should begin with a reviewed set of known traces. They can test several descriptions, compare returned matches, and document which evidence the model misses.

They should then use the feature for discovery, not final adjudication. Confirm high-impact findings manually and promote stable behavioral definitions into evaluators or deterministic controls.

The launch matters because it changes when teams can ask questions about agent behavior. They no longer need to predict every failure before they collect production traces.

That is a meaningful step for agent observability. The harder test is whether LangSmith AI Search can make those discoveries repeatable, auditable, and safe across real production workloads.