LangSmith adds AI Search for debugging agent traces

LangChain added AI Search to LangSmith on 13 October, a way to query agent traces by describing behaviour in plain language instead of matching keywords. Org admins turn it on in settings, and it then appears directly on the tracing page.

You type something like “agent said it finished but didn’t” or “user seemed frustrated or pushed back” and LangSmith returns the matching traces, stacked alongside the filters you already use, so you can scope a search to errored runs only. Under the hood, a decision model reads the content of each trace rather than relying on text similarity, and a component called SmithDB normalises traces from different agent frameworks into a common format of messages, tool calls and results before it searches them. Each trace gets a fixed 8,000-token budget, with user messages and agent replies read first.

Why it matters: this replaces manually scrolling through transcripts or pre-building an evaluator for every failure mode you can think of. Catching a silent failure or a PII leak becomes a search query instead of a review pass.

The caveat: LangChain says the hardest conversations to classify are the subtle ones, like telling user frustration apart from user satisfaction, so treat results as a lead to verify rather than a final answer.