Agent Evals Need Decisions, Not Essays: Building with Jev

I came across LangChain's experiment with Jev as an agent evaluator while thinking about a familiar problem: we often ask a writing model to produce a critique when the application ultimately needs a small decision. The reported accuracy, variance, latency, and cost were interesting. The architecture behind them was more interesting.

What is decision-first agent evaluation?

Decision-first agent evaluation starts with the action software must take, such as pass, review, or fail, then works backward to the smallest semantic judgments needed to justify it. Exact checks stay in ordinary code. Jev handles bounded questions that require language understanding. Code combines those typed signals into explicit policy.

Key takeaways

• Use deterministic code before paying for semantic judgment.
• Keep grounding, tool use, and usefulness as separate signals.
• Let Jev return typed probabilities and choices, not a prose verdict.
• Route uncertainty to review instead of forcing every trace into pass or fail.

This pattern fits a broader way of designing reliable AI systems: define the user-facing decision, make tradeoffs visible, and preserve a recovery path. If the earlier question is whether AI belongs in the workflow at all, I use the framework in AI-first does not mean AI everywhere.

I wanted to see whether that separation felt useful in code, so I built a small runnable Jev evaluator. It reads saved agent traces, performs deterministic checks, asks Jev three semantic questions, and combines the answers into a verdict. The build is the center of this article, not a new model leaderboard.

Start with the decision software needs. Ask a model only for the judgment code cannot make exactly.

What I built and what I did not

The sample uses three frozen weather traces. Each contains a user request, a final answer, retrieved evidence, and the recorded tool call. One answer agrees with the evidence, one contradicts it, and one gives a grounded but incomplete answer against a mixed forecast.

The project does not run a weather agent, call a weather service, or recreate LangChain's five-case experiment. Freezing the traces removes variation from the agent and weather service, but it does not make Jev deterministic. That lets the example focus on one question: how should typed semantic judgments become application policy? It is an architecture sample, not an independent reproduction of LangChain's benchmark.

01 / FREEZELoad a saved request, answer, evidence, and tool calls

02 / CHECKReject missing required data in ordinary code

03 / JUDGEAsk Jev about grounding, tool use, and usefulness

04 / ROUTEApply explicit pass, review, and fail thresholds

The repository keeps those responsibilities visible. fixtures/ holds the frozen traces. src/parse.ts validates incoming JSON. src/questions.ts defines the three Jev questions. src/evaluate.ts runs exact checks before the API call and preserves Jev's typed signals, distributions, and token usage. src/policy.ts contains the routing thresholds. The offline tests exercise parsing, ordering, mapping, and policy without spending an API call.

A review verdict is deliberately only a handoff boundary. The sample returns the decision and reasons; it does not contain a human-review interface, call a reasoning model, open a ticket, or decide who reviews the trace. A production application must connect that boundary to the escalation path appropriate for its risk.

The three judgments

Agent evaluation usually starts at two extremes. Deterministic code can verify exact conditions such as whether required fields exist or whether a known event occurred. An LLM judge can interpret an entire trace and write an open-ended critique. The gap between them is a semantic question with a bounded answer.

Jev is TypeSafe AI's first “System One” model. It accepts text or structured text state and returns typed answers and probabilities instead of generated prose. I used one of each primitive in the same request:

These dimensions stay separate on purpose. A trace can use the correct tool and still contradict the evidence. It can be grounded but incomplete. Keeping the raw signals lets policy treat a grounding failure differently from a mildly weak answer instead of hiding both inside one “quality” number.

The TypeSafe documentation makes an important distinction: Noul's value is already the probability of yes, while Choice and Score also return confidence derived from how concentrated their probability distributions are. Confidence is useful for routing, but it is not proof that the judgment is correct and it is not permission to act.

The code path, in the order that matters

Exact validation must happen before Jev. If the request is empty, the answer is missing, or no evidence was supplied, code already knows the trace cannot be evaluated as designed. Paying for a semantic judgment would add cost without adding information.

async function evaluateRun(run) {
  const exactFailures = validateRun(run);
  if (exactFailures.length > 0) {
    return {
      decision: "fail",
      reasons: exactFailures
    };
  }

  const response = await client.systemOne({
    state: run,
    questions: evaluationQuestions
  });

  const signals = {
    groundedProbability: response.answers.grounded.noul,
    toolUse: response.answers.toolUse.choice,
    toolUseConfidence: response.answers.toolUse.confidence,
    usefulnessScore: response.answers.usefulness.score,
    usefulnessConfidence: response.answers.usefulness.confidence
  };

  return applyPolicy(run, signals);
}

This is a condensed view of the evaluator's control flow, not its literal return shape. The full implementation also returns the source, model version, token usage, probability distributions, signals, and verdict. The questions are declared once and run together over the same state. The policy then fails clear semantic violations, passes clear successes, and returns review when a signal does not justify either automatic outcome.

That separation is the practical value of the pattern. A question can change without rewriting the CLI. A threshold can change without inventing a new prompt. Policy tests can run with fixed signals and no network. The model supplies bounded interpretation; conventional software remains responsible for workflow.

How the policy behaves

The repository's offline tests exercise the policy with fixed signals. This matters because the verdict logic is deterministic even though live Jev signals can vary across runs and model versions. The examples below describe the thresholds in code. They are not recorded Jev benchmark results.

The three fixtures are written to explore these paths. One answer agrees with its evidence, one is grounded but incomplete, and one contradicts its evidence. Their names do not predetermine a live Jev result. The policy applies the thresholds above to whatever signals the API returns.

This makes the architecture concrete without treating a one-off API response as benchmark evidence. A weather lookup can be sensible while the final answer still contradicts the forecast. Independent signals preserve both facts, and a short policy decides which one controls the outcome.

The useful output is not an eloquent critique. It is a set of signals that code can test, log, and route.

What I took from LangChain's benchmark

LangChain's benchmark supplied the prompt for this build. It froze five weather-agent traces and asked each judge for a binary pass-or-fail decision and a continuous quality score one hundred times per trace. One human reviewer supplied the reference labels. The comparison included Jev, GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6.

ACCURACYJev matched the five human pass-fail labels in all 500 repeated decisions

VARIANCEThe LLM judges' observed mean per-case quality variance was 92–913× Jev's

LATENCY0.44 seconds per Jev evaluation on average

COST$0.00035 per Jev call in that experiment

The scoreboard should be read cautiously. Five traces cover one agent, one domain, and one rubric. The binary reference came from one human reviewer. Provider and gateway defaults were used for the LLM judges, and the hosted Jev service version was not recorded in the experiment metadata. LangChain presents the result as promising but early.

Consistency is not correctness. A judge can repeat the same wrong answer with low variance; repeatability does not make it correct. In this narrow dataset, Jev matched the reference labels while remaining stable and fast. The Luna comparison is another useful check: Jev's reported call cost was only slightly lower there, so latency and variance were the more interesting differences.

My sample cannot validate those claims. LangChain's benchmark produced 500 repeated decisions per judge: five traces evaluated 100 times by each of four judges. The sample does not attempt that comparison. What it can show is the software shape that the benchmark suggested: frozen traces, atomic questions, typed signals, explicit thresholds, and offline policy tests.

The evaluator stack I would ship

I would not replace every evaluator with Jev. I would give each layer the kind of uncertainty it can handle well.

This ordering keeps an expensive or open-ended evaluator out of the default path. A reasoning model can investigate a novel failure, and a person can decide a consequential case, but neither integration is implemented by the sample repository. Its contract ends at a verdict and reasons.

The thresholds are deliberately teaching defaults. A low-risk assistant may accept more cases automatically. A financial or medical workflow should have a narrower automatic path and stronger independent checks. The values must be calibrated on representative traces, labeled outcomes, and the actual cost of each mistake.

Where decision-first evaluation can fail

1. The questions reward the wrong behavior

If usefulness ignores unsupported claims, an agent may learn to sound complete rather than remain grounded. Keep independently important dimensions separate and preserve their raw outputs.

2. The state omits necessary evidence

A model cannot judge evidence it never receives. Log the exact state sent to the evaluator, distinguish missing context from model error, and avoid treating absence as a confident negative.

3. Thresholds drift across tasks and versions

The repository's cutoffs are teaching defaults, not calibrated operating thresholds. A cutoff that seems reasonable in a small weather example may fail on support tickets, code changes, or a new Jev version. Record the concrete model version, watch distributions by task, and recalibrate when the data or evaluator changes.

4. Review becomes an unlabeled queue

A handoff is useful only if something owns it. Define where reviewed traces go, what context follows them, how outcomes are recorded, and whether corrections become new evaluation data.

How I would take the sample further

1. Add representative traces. Include ordinary successes, known failures, ambiguous cases, and adversarial inputs from the real application.
2. Collect independent labels. Resolve reviewer disagreement before treating a label as truth. One reviewer is a starting point, not an oracle.
3. Measure correctness and repeatability separately. Stable wrong answers and noisy right averages create different operational risks.
4. Run in shadow mode. Compare decisions with real outcomes before changing user-visible behavior.
5. Connect the review boundary. Route uncertainty to a named owner or deeper evaluator and capture the eventual outcome.
6. Monitor the evaluator. Track model version, latency, cost, distributions, overrides, and performance by task type.

The durable asset is not one model call. It is the trace set, question design, policy, tests, and feedback loop around it. That is why I like the decision-first framing: it makes the model one typed component in a system instead of letting a generated critique become the system.

The broader lesson

Generative judges made semantic evaluation accessible, but not every judgment needs generation. When the output is bounded, a decision model can return the small signals an application needs while code keeps control of consequence and recovery.

LangChain's result does not prove that Jev should replace every LLM judge, and my three fixtures do not reproduce its benchmark. Together, they make a useful engineering question harder to ignore: before asking a model to explain whether an agent succeeded, can we define the decision and the evidence precisely enough to judge it directly?

Source note: Benchmark numbers come from LangChain's published article and experiment repository. The Jev description follows TypeSafe's System One documentation; primitive outputs and confidence behavior follow its primitives and confidence documentation. I did not independently reproduce LangChain's comparison.

Better agents need more than models that act intelligently. They need feedback loops whose decisions are small enough to inspect, test, and improve.