Agent observability glossary

AI agent

An AI agent is a system that uses a large language model (LLM) to pursue a goal, instead of generating a single response.

Along the way, it can reason, call tools, retrieve information, remember context, and make decisions across multiple steps.

Agent run

Also called: session

An agent run is one complete execution of an agent, from the moment it receives an input until it produces its final output.

If a customer asks for a refund, everything the agent does to handle that request — from retrieving account details to calling APIs and generating a response — belongs to the same run.

Trace

A trace is the complete, ordered record of an agent run.

It captures every step the agent took, in the specific order it happened, so you can understand both the final answer and how the agent arrived there. That usually includes LLM calls, tool calls, retrievals, memory operations, and other system events — all nested to show what triggered what.

Span

Also called: observation

A span is a single operation within a trace — one LLM call, one tool execution, or one retrieval.

Together, spans form the sequence of events that makes up a trace, with each span recording its inputs, outputs, timing, and other metadata.

Instrumentation

Instrumentation is the code that records what an agent is doing while it runs.

Instead of only producing a final response, the agent also emits information about each step it takes: LLM calls, tool calls, retrievals, and other events. That information becomes the traces and spans you use to debug and understand the system.

Instrumentation is usually added through an SDK or an OpenTelemetry integration.

OpenTelemetry GenAI conventions

The OpenTelemetry GenAI conventions are an emerging standard for describing LLM and agent activity as telemetry.

The goal is portability. If different tools record traces in the same way, teams can move observability data between platforms without changing how their applications are instrumented. The specification is still evolving, and some conventions continue to change as the ecosystem matures.

Tool call

A tool call is the moment an agent asks another system to do work — searching the web, querying a database, sending an email, calling an API, or running custom code.

In a trace, each tool call is typically recorded as its own span, along with the inputs, outputs, execution time, and result status (whether it succeeded or not).

Nested spans

Nested spans preserve the parent-child relationships between operations in an agent run, so you can see what happened as well as what caused each step to happen.

Agent runs are rarely a flat sequence of events. An LLM may call a tool, which starts another workflow, which calls another model. Without that structure, complex traces quickly become difficult to follow.

Metadata

Also called: attributes

Metadata is the additional context attached to a trace or span that describes the circumstances around it — for example the model version, prompt version, user, environment, latency, cost, or experiment.

Metadata makes it possible to filter, compare, group, and investigate runs without reading every trace individually.

Offline evaluation

Offline evaluation is running an agent against a fixed, curated set of test cases before it reaches production.

Because every version is evaluated against the same dataset, teams can compare changes over time and catch regressions before they affect actual users.

Online evaluation

Online evaluation is scoring agent runs on live production traffic, as they happen, instead of waiting for users to report problems.

This makes it easier to detect regressions, unexpected behavior, or changes in performance in real time.

TL;DR: Offline evaluation tells you how an agent performs on known examples; online evaluation tells you how it behaves in the real world.

LLM-as-a-judge

LLM-as-a-judge is using one LLM to evaluate the output of another.

This is useful when quality depends on judgment rather than exact correctness — when it's too subjective for a fixed rule. For example, whether an answer is helpful, follows a policy, or explains something clearly.

Because the evaluation itself is model-based, teams often combine LLM judges with deterministic checks and human review.

Trajectory evaluation

Trajectory evaluation looks beyond the final output and evaluates the entire sequence of decisions an agent made along the way.

That can include the reasoning process, the tools it chose, the order it used them, and how efficiently it reached its goal.

An agent can produce the right answer for the wrong reasons. Trajectory evaluation helps uncover those cases.

Tool-calling evaluation

Tool-calling evaluation checks whether an agent interacted with external tools correctly.

Depending on the workflow, that might mean checking whether it selected the right tool, supplied the correct arguments, called tools in the right order, or avoided unnecessary calls.

For many production agents, the quality of tool use matters just as much as the quality of the final response.

Golden dataset

A golden dataset is a collection of representative test cases with expected outcomes, used as a fixed benchmark for evaluating new versions of an agent.

It makes it easier to measure improvements, detect regressions, and compare different prompts, models, or workflows under consistent conditions.

Scorer

A scorer is a function or judge that evaluates one aspect of an agent run.

• Some scorers are deterministic, checking conditions with exact rules.
• Others use LLMs to assess qualities that require judgment, such as helpfulness or policy compliance.
• Most evaluation systems combine multiple scorers, with each one measuring a different dimension of quality.

Task completion

Task completion is a measure of whether the agent actually achieved the user's goal, rather than producing a plausible-sounding output.

An answer can be well written and still fail to complete the task the user asked for, which is why, for many production agents, task completion is the metric that matters most.

Plan adherence

Plan adherence measures how closely an agent followed its own plan as it completed a task.

Some agents create a plan before taking action. Large deviations can sometimes indicate that the agent became confused, encountered unexpected information, or made unnecessary decisions along the way.

Faithfulness

Faithfulness is a measure of whether an agent's output is supported by the information it retrieved or was given.

A faithful answer stays grounded in its available evidence instead of introducing unsupported claims or details.

Faithfulness is especially important in retrieval-augmented generation (RAG) systems, where the goal is to answer using retrieved documents rather than the model's own knowledge.

Hallucination

A hallucination is information an agent presents as true even though it is not supported by its inputs, retrieved context, or tool outputs.

Hallucinations are often written confidently, which makes them difficult to detect without checking the underlying evidence.

Not every incorrect answer is a hallucination, though. The defining characteristic is that the agent invents information instead of relying on available facts.

Latency

Latency is the time an agent takes to complete a full run.

It can be measured across the entire workflow or broken down into individual spans, making it easier to identify whether delays come from model inference, retrieval, external APIs, or other parts of the system.

Cost

Cost is the token usage and API spend of an agent run.

Teams typically track both, for the full run as well as for individual spans. Breaking cost down this way makes it easier to identify workflows, prompts, or tools that are disproportionately expensive.

Drift

Drift is the gradual change in an agent's behavior or output quality over time.

The workflow may not have changed, but the agent's outputs become different because something around it has — a new model version, updated retrieval data, changes in user behavior, or shifting production inputs. Because drift happens gradually, it often goes unnoticed until quality has already declined.

Production tracing

Production tracing is recording traces from live user traffic.

Development and test environments can uncover many issues, but they rarely capture the variety of situations that appear in production. Tracing live runs helps teams understand how agents behave under real workloads and investigate failures after they happen.

Guardrails

Guardrails are automated checks that help prevent an agent from producing unsafe, off-policy, or low-quality outputs.

Some guardrails run before the model generates a response. Others evaluate the output before it reaches the user. Depending on the application, they may enforce safety policies, validate structured outputs, detect sensitive information, or block responses that fail quality standards.

Guardrails reduce risk, but they do not replace evaluation or human oversight.

Regression detection

Regression detection is identifying a drop in quality across different agent versions.

Teams often compare a new version against a previous one by running the same evaluation suite before deployment. If the agent performs worse on important tasks, the change can be investigated before it reaches production.

The goal is to prevent improvements in one area from introducing problems somewhere else.

Drift detection

Drift detection is monitoring a deployed agent for changes in behavior or quality relative to an established baseline.

Unlike regression detection, which compares versions before release, drift detection monitors production systems over time. It helps teams spot when changes in models, data, or user behavior cause an agent to drift away from its expected performance.

Alerting

Alerting is notifying a team when a monitored signal crosses a set threshold.

Alerts can be triggered by rising latency or error rates, or by quality signals such as failing evaluations, increasing hallucination rates, or declining task completion.

The goal is to surface meaningful problems early, before they become widespread.

Human-in-the-loop evaluation

Human-in-the-loop evaluation brings people into the evaluation process alongside automated checks.

Some aspects of quality can be measured with rules or language-model judges. Others require human judgment — whether a response is genuinely helpful, follows company policy, or handles an unusual situation appropriately is often something only a person can decide. Human review is especially important for high-stakes, ambiguous, or customer-facing workflows.

Human review queue

A human review queue is a collection of agent runs routed to people for inspection.

Teams typically prioritize runs that failed an evaluation, triggered a guardrail, or crossed a predefined risk threshold, so reviewers spend their time on the cases that matter most.

Annotation

Annotation is the process of adding human labels to agent runs.

Those labels might identify whether a response was correct or incorrect, rate its quality, explain what went wrong, or suggest a better outcome. Over time, annotations become valuable training data for evaluation datasets, LLM judges, and future improvements.

Escalation policy

An escalation policy is the set of rules that define when an agent should hand a task to a person instead of continuing on its own.

The decision might depend on confidence, risk, failed evaluations, policy requirements, or the type of request. A well-designed escalation policy ensures people spend their time where human judgment adds the most value.

Feedback loop

Also called: eval flywheel

A feedback loop is the cycle of turning production failures into future improvements.

When an agent makes a mistake, that run becomes a new evaluation case. Future versions of the agent are then tested against it, making it less likely that the same failure will happen again. Over time, the evaluation suite grows from actual user behavior rather than hypothetical examples, helping reliability improve with every iteration.