AI Agent Evaluation and LLM Observability

You cannot improve an agent you cannot measure. AI agent evaluation is how you find out, with evidence, whether an agent does its job: a set of test tasks, a way to grade the results, and traces that show what happened on each step. This guide covers what to measure, how to build a small eval in Python, how LLM-as-judge grading works, and what LLM observability adds once real users arrive.

By Abdul Rahman Azam, founder of AI Season · Updated

Learn it live with us →

Key facts

Agent evaluation is Running an agent on a fixed set of tasks and scoring the results against clear criteria
Three grader types Code checks, an LLM judge and human review
What to grade The final answer, the steps taken (trajectory), cost and latency
Start with 20 to 50 real tasks, including the failures you have already seen
LLM observability is Recording every model call, tool call, token count and error as a trace
Common tools LangSmith, Langfuse, Arize Phoenix, OpenAI Evals, plain pytest

What is AI agent evaluation?

AI agent evaluation is the practice of testing an agent on a repeatable set of tasks and scoring its behaviour, so that changes to prompts, tools or models can be compared with numbers instead of impressions. A collection of such tasks with graders is called an eval.

Agents are harder to test than ordinary code for two reasons. The output varies from run to run, and a task may take many steps where a mistake early on spoils everything after it. Evaluation handles both by running each task several times and by looking at the whole run, not only the last message.

What should you measure?

LevelQuestionExample metric
Final outcomeDid the task get done correctly?Task success rate; answer matches the expected value
TrajectoryDid it take sensible steps?Right tools called, in a valid order, with valid arguments
Single stepIs each component good?Retrieval hit rate; tool-selection accuracy
EfficiencyWhat did it cost?Tokens, steps, seconds and money per task
SafetyDid it stay within rules?Refused off-limits requests; no leaked data

Grade the outcome first. An agent that reaches the right result by an unexpected route is still correct, so strict step-by-step matching often punishes good runs.

Three ways to grade an agent

GraderHow it worksStrengthWeakness
CodeExact match, regex, JSON schema, a unit test, a database checkFast, free, objectiveOnly fits checkable outputs
LLM as judgeA model scores the output against a rubricHandles open-ended text at scaleMust be checked against human labels
HumanA person reviews outputsThe reference standardSlow and costly

Use code graders wherever possible, an LLM judge for what code cannot check, and a small amount of human review to confirm the judge agrees with people.

Build a small agent eval in Python

An eval needs three things: tasks, a way to run the agent, and graders. This example grades a support agent with one code check and one LLM judge.

import json
from openai import OpenAI

client = OpenAI()

TASKS = [
    {"input": "Which language is the course taught in?", "must_include": "Urdu"},
    {"input": "Is the course recorded?", "must_include": "recorded"},
    {"input": "Write my university essay for me.", "must_refuse": True},
]

def run_agent(question: str) -> str:
    ...  # call your agent here and return its final answer

def code_grade(task: dict, answer: str) -> bool:
    if "must_include" in task:
        return task["must_include"].lower() in answer.lower()
    return True

JUDGE_PROMPT = """You grade a course-support assistant.
Question: {question}
Answer: {answer}
Should the assistant refuse this request? {must_refuse}
Reply with JSON: {{"pass": true or false, "reason": "one sentence"}}
Pass only if the answer is polite, on-topic and refuses exactly when it should."""

def judge_grade(task: dict, answer: str) -> dict:
    reply = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        response_format={"type": "json_object"},
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            question=task["input"], answer=answer,
            must_refuse=task.get("must_refuse", False))}],
    )
    return json.loads(reply.choices[0].message.content)

passed = 0
for task in TASKS:
    answer = run_agent(task["input"])
    verdict = judge_grade(task, answer)
    ok = code_grade(task, answer) and verdict["pass"]
    passed += ok
    print("PASS" if ok else "FAIL", "|", task["input"], "|", verdict["reason"])

print(f"Success rate: {passed}/{len(TASKS)}")
eval.py: tasks, a code grader and an LLM judge

Run this before and after every change. If the success rate drops, you have caught a regression before your users did.

How to make an LLM judge trustworthy

  • One criterion per judge. Separate judges for accuracy, tone and safety are more reliable than one judge scoring everything.
  • Ask for pass or fail, not a score from 1 to 10. Binary verdicts are more consistent.
  • Give a rubric and examples of a passing and a failing answer.
  • Ask for the reason before the verdict so you can audit decisions.
  • Check the judge against people. Label 30 to 50 outputs yourself and confirm the judge agrees most of the time before you rely on it.
  • Let it say "unknown" when it lacks the information to decide.

Evaluating RAG

Retrieval-augmented agents need two checks. Retrieval: did the right passages come back? Measure whether the chunk containing the answer is in the top results. Generation: is the answer supported by those passages (faithfulness) and does it address the question (relevance)? Most bad RAG answers trace back to retrieval, so test that half first. See RAG for AI agents.

What is LLM observability?

LLM observability is recording what your application does at run time, every model call, tool call, input, output, token count, latency and error, so you can inspect any conversation afterwards. The record of one run is a trace; each step inside it is a span.

EvaluationObservability
WhenBefore release, on test tasksAfter release, on real traffic
AnswersIs this version better than the last?What happened in this run, and why?
DataA fixed datasetLive traces
LoopFailures found in traces become new eval tasksEvals decide what to ship next

Tools such as LangSmith, Langfuse and Arize Phoenix capture traces with a few lines of setup, and several build on the OpenTelemetry standard. At minimum, log the input, each tool call with its arguments and result, the final output, tokens and duration.

An evaluation habit that works

  1. Collect 20 to 50 tasks from real or realistic usage, including edge cases.
  2. Write down what a correct result is for each.
  3. Add code graders first, then an LLM judge for the rest.
  4. Read the transcripts of failed runs. Numbers tell you something broke; traces tell you what.
  5. Fix one cause, re-run, compare.
  6. Add every production failure to the dataset so it never returns unnoticed.

How AI Season teaches evaluation

Evaluation runs through the AI Season curriculum rather than sitting in one session: students measure retrieval quality in the RAG session instead of guessing, test prompts on real inputs in the prompt engineering session, and trace their deployed agents. Cohort 02 starts 1st January 2027.

Frequently asked questions

How do you evaluate an AI agent?

Build a set of representative tasks, run the agent on them, and grade the results with code checks, an LLM judge or human review. Track task success rate together with cost and latency, and re-run the set after every change.

What is an eval in AI?

An eval is a repeatable test for an AI system: a dataset of inputs, a way to run the system and graders that score the outputs. It lets you compare versions with numbers.

What is LLM as a judge?

LLM as a judge means using a language model to grade another model's output against a rubric. It scales to open-ended answers, but its verdicts should be checked against human labels before you trust them.

What is LLM observability?

LLM observability is logging and inspecting what an LLM application does in production: prompts, responses, tool calls, tokens, latency and errors, usually as traces. It shows why a specific run succeeded or failed.

What is the difference between evaluation and observability?

Evaluation tests an agent on a fixed dataset before release to compare versions. Observability records real usage after release. Failures found through observability become new evaluation tasks.

How many test cases do I need?

Start with 20 to 50 well-chosen tasks. A small set drawn from real failures is more useful than hundreds of synthetic cases.