When we build traditional software, we usually know what testing means.

A function receives an input, returns an output, and we can write something like:

assert add(2, 3) == 5

An AI agent changes the problem.

An agent can decide which tool to call, in what order, how many times to retry, what information to keep, and how to compose the final response. Two valid runs may follow different paths.

That creates an uncomfortable question:

How do we know whether the agent actually did its job well?

That is where an agent evaluator comes in.

An agent evaluator is a component that observes an agent run and turns it into an evaluable signal: a boolean, category, score, probability, or a collection of metrics.

It is not the agent solving the task.

It is the system that judges the run.

The simplest definition

Think of an agent like this:

user
 │
 ▼
agent
 │
 ├── decides
 ├── calls tools
 ├── observes results
 ├── decides again
 └── answers
       │
       ▼
    evaluator
       │
       ├── pass / fail
       ├── score
       ├── category
       └── probability

The evaluator may inspect only the final answer, or it may receive much more context:

  • the original request;
  • the final answer;
  • tool calls;
  • tool outputs;
  • the full sequence of steps;
  • an expected answer;
  • a rubric;
  • previous human labels.

We usually call that execution history a trace.

Evaluation turns a complex trace into signals that can be stored, charted, used in CI, or monitored in production.

What questions can it answer?

A useful evaluator does not need to answer a vague question such as “is this agent good?”

It is better to break that down into focused criteria.

For example:

Did it answer what the user asked?
Is the answer supported by the tool results?
Did it search when search was necessary?
Did it invent facts that are absent from the evidence?
Did it choose the right tool?
How useful is the answer?
Should this run be allowed to continue?

Each question can become a separate metric.

That matters because an agent can pass some dimensions and fail others.

For example:

relevance        0.96
groundedness     0.91
tool_selection   PASS
completeness     0.72
policy_gate      PASS

Reducing all of that to a single “8/10” would throw away useful information.

First type: code-based evaluators

The simplest form of evaluation is still deterministic code.

assert "weather" in called_tools
assert final_answer != ""
assert latency_ms < 3000

This kind of evaluator has major advantages:

  • cheap;
  • fast;
  • reproducible;
  • easy to run thousands of times;
  • excellent for invariants.

The difficulty appears when we want to evaluate meaning.

We can check whether an agent called the weather tool.

It is much harder to encode manually:

Did the agent use that tool result correctly to answer the user’s question?

Open-ended tasks may have many correct answers and many valid trajectories.

Trying to enumerate all of them in code quickly becomes impractical.

Second type: LLM-as-a-judge

A common solution in modern agent systems is to use another LLM as the judge.

The flow looks roughly like this:

question + trace + evidence + rubric
                 │
                 ▼
                LLM
                 │
                 ▼
        explanation + score

An LLM can read unstructured text and evaluate properties that are difficult to express with rigid rules.

For example:

Rate from 1 to 5 how well the answer actually solved the user’s request.

That works surprisingly well for many tasks.

But it introduces three problems.

1. Variance

The judge is itself a generative model.

The same trace can receive slightly different scores across runs.

2. Latency

To reach a decision, the model generates tokens.

Even if we only need a number, we are using a system designed to generate language.

3. Cost

If we want to evaluate thousands or millions of production traces, every extra call matters.

That is why online evaluation often forces teams to decide what fraction of runs they can afford to judge.

Jev introduces a third shape

LangChain’s article Jev-as-a-Judge for Agent Evals explores another option.

Jev, from TypeSafe AI, is not designed to generate text.

It receives a state plus typed questions and returns decisions and probabilities directly.

In other words:

trace / state
     │
     ▼
    Jev
     │
     ├── Choice
     ├── Score
     └── Noul

Those three question types cover a large fraction of common evaluation tasks.

Choice

Selects one option and returns probabilities and confidence.

Which outcome best describes this run?

- searched_appropriately
- searched_unnecessarily
- failed_to_search

Score

Evaluates against an ordered rubric.

How useful was the answer?

1  useless
2  poor
3  acceptable
4  good
5  excellent

Noul

Returns the probability that a yes/no statement is true.

Is the final answer grounded in the evidence?

0.94

This removes a common LLM-as-a-judge step: generating language only to convert it back into a structured decision.

We previously looked at what the technical community is saying about Jev. The interesting point here is not Jev as an isolated product, but how naturally its output shape fits the agent-evaluation problem.

What LangChain did in the experiment

LangChain built a small weather agent with Deep Agents and stored fixed runs in a LangSmith dataset.

That is crucial.

If every judge observed a different agent run, it would be difficult to know whether a scoring difference came from the evaluator or from the agent itself.

So they compared the judges on exactly the same traces.

The dataset contained five weather requests. For each example, they stored the response and full execution and measured two signals:

  • quality, a continuous score;
  • does_pass, a binary decision.

A human reviewer labelled the same runs to create a reference oracle.

Then each evaluator was repeated 100 times using Jev, GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6.

According to LangChain’s published results, Jev matched the human oracle on all 500 binary decisions in the experiment. Terra reached 99.8%, Luna 96.4%, and Claude 80.0%.

For continuous scoring, Jev had an observed mean per-case variance of 0.0000149. LangChain reported 433× higher variance for Luna, 913× higher for Terra, and 92× higher for Claude.

They also measured cost and latency. Jev averaged 0.44 seconds and about $0.00035 per call in that test.

The numbers are striking, but they need the right interpretation.

LangChain explicitly describes the experiment as small and narrow. Five traces from a weather agent do not establish that Jev will be the best evaluator across every domain.

And low variance does not automatically mean high accuracy.

A judge can be extremely consistent and consistently wrong.

That caveat may be the most important part of the experiment.

LangSmith is not the evaluator

It is useful to separate two components.

LangSmith can store traces, datasets, runs, experiments, and metrics.

The evaluator produces the judgment.

Visualized:

agent
  │
  ▼
trace ───────────────► LangSmith
                          │
                          ▼
                     evaluator
                   /      |      \
                code     LLM     Jev
                   \      |      /
                          ▼
                   metrics / scores
                          │
                          ▼
                    dashboards / CI

LangSmith provides evaluation and observability infrastructure.

The evaluator is a function — or a model — inside that system.

Why this looks like testing

This leads to a powerful idea.

An evaluator gives us something close to a semantic test.

In a traditional pipeline we can check:

Did the process finish?
Was there an exception?
Does the JSON match the schema?

In an agentic pipeline we can also ask:

Did the output preserve the important facts?
Does the title actually represent the content?
Is the answer grounded in the sources?
Was the selected tool appropriate?
Is this run good enough to publish?

Conventional code remains the first line of defense for invariants.

The evaluator adds a layer for properties that depend on meaning.

Example: evaluating a content pipeline

Imagine a pipeline:

sources
   ↓
extraction
   ↓
script
   ↓
title + description
   ↓
publication

Before publishing, we could run several evaluators.

grounded_in_sources     0.97
covers_main_claims      0.92
title_matches_script    0.95
description_is_faithful 0.98
publishable             0.96

Then code applies the policy:

if grounded_in_sources < 0.90:
    send_to_review()

if title_matches_script < 0.85:
    regenerate_title()

if publishable > 0.95:
    publish()

AI performs the semantic judgment.

Software retains control of the policy.

That separation matters.

Offline evals and online evals

Evaluators usually appear in two different places.

Offline

Before deploying a new version of the agent.

prompt change
    ↓
trace dataset
    ↓
evaluators
    ↓
compare with baseline
    ↓
CI pass / fail

This helps detect regressions.

A change can pass every unit test and still reduce the semantic quality of the agent’s answers.

Online

Against real production runs.

production
    ↓
traces
    ↓
evaluators
    ↓
time series
    ↓
alerts

Now we can detect degradations that never appeared in the test dataset.

For example:

groundedness fell from 0.94 to 0.81 after the latest deploy.

That turns agent quality into an operationally observable signal.

An evaluator should not govern alone

The temptation is to treat the evaluator’s score as absolute truth.

That would be a mistake.

A serious system combines several layers:

deterministic invariants
        +
semantic evaluators
        +
labelled datasets
        +
human review
        +
monitoring

The evaluator itself must also be evaluated.

Teams need to periodically check whether its judgments remain aligned with the human decisions they actually care about.

Especially when those judgments gate costly, destructive, or irreversible actions.

The architectural idea

The most interesting part of the agent-evaluator concept is not adding “one more model” to the stack.

It is separating two responsibilities:

AGENT
What should I do?

EVALUATOR
How well did I do it?

That separation enables feedback loops.

agent
  ↓
run
  ↓
evaluator
  ↓
metric
  ↓
regression / alert / learning
  ↺

Once an agent stops being a demo and starts operating continuously, this layer becomes increasingly important.

Executing the task is not enough.

We need a systematic way to know whether the executions remain good.

That is, in essence, what an agent evaluator does.

Sources