AI Engineering

AgentCore Evaluations Makes Agent Quality a Trace Contract

AWS's August 26 AgentCore Evaluations update shows how OpenTelemetry and OpenInference can make agent evaluation portable across frameworks.

The hard part of evaluating production agents is not inventing another score. It is getting enough comparable execution data from agents that were built with different frameworks, deployed by different teams, and instrumented by different libraries.

AWS published a technical post on August 26, 2026 describing how Amazon Bedrock AgentCore Evaluations now reads agent traces across several frameworks through OpenTelemetry and OpenInference conventions. That sounds like an observability implementation detail. It is more important than that. It is a useful signal that agent quality is becoming a telemetry contract, not a framework feature.

What Happened

On August 26, AWS published "Evaluate any agent framework with Amazon Bedrock AgentCore Evaluations." The post explains how AgentCore Evaluations reconstructs agent sessions from OpenTelemetry spans and event records, then applies the same evaluators across agents built with different SDKs.

The supported framework list in the AgentCore docs is concrete. AWS documents support for Python versions of Strands Agents, LangGraph, OpenAI Agents, LlamaIndex, Google ADK, and the Claude Agent SDK, using either framework-native telemetry or instrumentation packages from the OpenTelemetry and OpenInference ecosystems. The same page lists the scope names and recommended versions that AgentCore Evaluations recognizes, such as opentelemetry.instrumentation.openai_agents, openinference.instrumentation.openai_agents, opentelemetry.instrumentation.llamaindex, openinference.instrumentation.google_adk, and openinference.instrumentation.claude_agent_sdk.

The generic support page is the more interesting part. AgentCore Evaluations does not only hard-code a few framework integrations. It can also read spans from instrumentation scopes under opentelemetry.instrumentation.* or openinference.instrumentation.*, as long as those spans use the expected semantic attributes. Dedicated framework handling wins when available; generic handling is a fallback for convention-compliant frameworks that are not explicitly named.

The Evaluate API reference confirms the underlying contract: the synchronous Evaluate API accepts agent traces in OpenTelemetry format, takes an evaluator ID, optionally accepts reference inputs such as expected responses or expected tool trajectories, and returns scoring results with labels, values, explanations, and token usage metadata.

This is not the first release of AgentCore Evaluations. AWS announced general availability in March 2026. The August 26 post matters because it explains the portability boundary. If your agent emits the right trace shape, the evaluation pipeline can operate above the framework layer.

Why It Matters

Agent frameworks are fragmenting faster than evaluation practices. That is not necessarily bad. LangGraph, LlamaIndex, OpenAI Agents, Google ADK, Claude Agent SDK, Strands Agents, CrewAI, custom orchestration code, and serverless harnesses make different tradeoffs around control flow, tool calling, memory, retrieval, and deployment.

The problem appears after the prototype. Platform teams want consistent release gates. Security teams want evidence for what tools an agent called and why. Product teams want to compare prompt changes. SRE teams want to know whether production quality is degrading after a model swap or tool schema change. None of those teams should have to rebuild the evaluation pipeline every time an application team picks a different agent framework.

Traditional unit tests do not solve this by themselves. They are still necessary, especially for tools and deterministic business logic. But an agent run is not a function call with one stable path. A single user turn can include model calls, tool calls, retrieval, reranking, memory lookup, guardrail checks, prompt rendering, and final response generation. The relevant question is often not only "did the output equal X?" It is also "did the agent call the correct tool, with the expected arguments, in a sensible order, while staying inside the policy boundary?"

That is why trace-level evaluation is a better primitive than framework-level evaluation. A trace captures what happened. The framework is just one way to produce it.

For experienced engineers, the main consequence is architectural. If agent evaluation depends on a vendor-specific callback surface, it becomes another migration tax. If it depends on a documented telemetry convention, it can become part of the platform substrate. You can standardize dashboards, CI quality gates, regression datasets, and production sampling without forcing every team onto the same agent SDK.

There is still lock-in at the service layer if you choose AgentCore Evaluations. The evaluator execution, storage integration, IAM boundary, and CloudWatch path are AWS products. But the instrumentation boundary is less proprietary than many agent-platform APIs. That distinction matters. It lets teams make a pragmatic AWS decision while still investing in trace semantics that are useful outside AWS.

How It Works

AgentCore Evaluations works by reconstructing agent behavior from sessions, traces, and spans.

A session represents a conversation or workflow execution. Within the session, a trace typically maps to one user turn. Within each trace, spans represent units of work: invoking the agent, calling a model, executing a tool, retrieving documents, running a guardrail, reading memory, rendering a prompt, or doing orchestration work.

AWS says the evaluation service primarily needs three span roles.

The invoke-agent span anchors the user turn. It carries the top-level input and final response. Inference spans carry the model message history and model output. Execute-tool spans carry the tool name, arguments, and result. Retrieval, reranking, memory, prompt, and guardrail spans can provide useful context, but the evaluator does not require every possible span type to score the run.

The classification mechanism depends on the telemetry convention.

In the OpenTelemetry GenAI conventions, agent and tool activity is identified with attributes such as gen_ai.operation.name. Relevant operation names include invoke_agent, chat, execute_tool, retrieval, plan, and memory-related operations. The OpenTelemetry GenAI agent-span document is still marked as development, so teams should treat the schema as evolving rather than permanently frozen.

In OpenInference, span classification uses openinference.span.kind. The convention includes values such as LLM, TOOL, AGENT, CHAIN, RETRIEVER, RERANKER, EMBEDDING, GUARDRAIL, EVALUATOR, and PROMPT. AgentCore's generic support page says it reads OpenInference message and tool fields from attributes such as indexed llm.input_messages and llm.output_messages, plus tool input and output fields.

The bridge between those worlds is the instrumentation scope. OpenTelemetry records a scope name on spans and event records. AgentCore uses that scope name to decide whether to apply dedicated parsing for a known framework or generic parsing for convention-compliant instrumentation. A random custom scope name will not get the same treatment just because the payload looks similar. The prefix is part of the contract.

The operational path is AWS-native. On AgentCore Runtime, AWS Distro for OpenTelemetry exports spans and event records to CloudWatch. On-demand evaluation can collect a specific session and score it through the AgentCore CLI, SDK, or the Evaluate API. Online evaluation samples live traffic and writes results to a CloudWatch log group under /aws/bedrock-agentcore/evaluations/results/<online-evaluation-config-id>, with scores also emitted as CloudWatch metrics.

The ground-truth docs show why this is more than sentiment scoring. Builtin.Correctness can use an expected response at the trace level. Builtin.GoalSuccessRate can use session-level assertions. Trajectory evaluators can compare expected tool sequences exactly, in order, or in any order. That gives teams a path from "the answer sounds good" toward regression checks that understand tool behavior.

Example

Imagine a platform team supporting three internal agents:

A LangGraph incident triage agent that reads alerts, searches runbooks, and opens tickets.
A LlamaIndex support agent that answers questions over product documentation.
An OpenAI Agents based engineering assistant that calls internal service-catalog tools.

Forcing all three onto one framework just to run evals would be a bad platform decision. The better contract is: every agent must emit evaluation-compatible traces, and every release candidate must run the same evaluation suite before promotion.

For an on-demand gate, the pipeline shape can stay simple:

from bedrock_agentcore.evaluation import EvaluationClient, ReferenceInputs

client = EvaluationClient(region_name="us-east-1")

results = client.run(
    evaluator_ids=[
        "Builtin.Correctness",
        "Builtin.GoalSuccessRate",
        "Builtin.TrajectoryInOrderMatch",
    ],
    agent_id=agent_id,
    session_id=session_id,
    reference_inputs=ReferenceInputs(
        expected_response={
            trace_id: "The deployment is blocked because the canary error rate is above policy."
        },
        assertions=[
            "The agent identifies the failing service before suggesting remediation.",
            "The agent does not open a ticket until it has checked the active incident state.",
        ],
        expected_trajectory=["get_alert", "search_runbook", "get_incident_state"],
    ),
)

failed = [r for r in results if r["value"] < 0.8]
if failed:
    raise SystemExit("Agent evaluation gate failed")

The exact SDK surface may differ by version, and production code needs stronger result handling than a threshold check. The design point is the same: the evaluator consumes traces, not framework internals.

The contract should be written down in the platform docs:

Agent evaluation telemetry requirements:

1. Emit a stable session identifier for each conversation or task.
2. Preserve one trace per user turn or workflow step.
3. Emit a top-level invoke-agent span where the framework supports it.
4. Emit model-call spans with message input and output content.
5. Emit tool-call spans with tool name, arguments, result, and correlation IDs.
6. Use supported OpenTelemetry or OpenInference instrumentation scope names.
7. Flush spans and event records before the runtime returns or suspends.

The last item is easy to miss. AWS's post calls out a practical failure mode: batch processors can still be holding spans or log event records when a serverless-style runtime suspends after the handler returns. If the process does not force a telemetry flush, evaluation fails later with missing context. That is not an evaluation problem. It is a delivery guarantee problem.

This is where agent engineering starts to look like normal distributed systems work. Telemetry is not "best effort debug output" if downstream release gates and quality monitors depend on it. It is production data.

My Take

The useful shift here is that AWS is treating agent evaluation as a consumer of standard traces. That is the right direction.

I would not read this as "agent quality is solved." LLM-as-judge evaluators can be useful, but they are not a substitute for deterministic tests, domain-specific assertions, security checks, or human review on high-impact workflows. A green evaluation score means the system passed a particular measurement. It does not mean the agent is safe, correct, or production-ready in the abstract.

I also would not ignore the privacy implications. Evaluation-compatible telemetry often contains user prompts, model responses, tool arguments, retrieved context, and tool results. That can include sensitive data. If those records go to CloudWatch, your access controls, retention policy, encryption choices, redaction strategy, and incident procedures need to match the sensitivity of the underlying agent workload. Treat prompt and tool telemetry as application data, not harmless logs.

The Python-only note in the supported-framework docs also matters. Many production agent stacks are TypeScript-heavy, especially teams building around web apps, Workers, edge runtimes, and Node-based tool servers. The August 26 architecture is still useful for those teams, but the current documented support matrix should shape rollout plans. Do not promise cross-language coverage that the docs do not claim.

The bigger lesson is portable instrumentation discipline. Agent teams should stop treating traces as an afterthought added by the observability team after the agent works. The trace schema is part of the agent interface. It determines whether you can compare versions, reproduce failures, build regression datasets, monitor production behavior, and move between frameworks without losing your evaluation investment.

For platform teams, I would make this a paved-road requirement. Pick the supported instrumentation packages. Pin versions. Add a small conformance test that invokes the agent, waits for telemetry ingestion, and verifies that the session includes invoke, inference, and tool spans with the fields your evaluators need. Fail the build if the trace is empty or malformed, even before judging response quality.

That may feel bureaucratic during prototyping. It pays for itself the first time a model upgrade changes tool-call behavior, a framework upgrade renames a span attribute, or an agent starts returning good-looking answers after skipping the retrieval step that was supposed to ground them.

Conclusion

AWS's August 26 AgentCore Evaluations post is worth paying attention to because it puts the evaluation boundary in the trace layer. Frameworks can differ. Models can differ. Orchestrators can differ. The evaluation system needs a stable record of what the agent actually did.

That is a better primitive for production engineering than SDK-specific callbacks or screenshot-driven demos. It lets teams build common quality gates across heterogeneous agent stacks, provided they do the less glamorous work: consistent instrumentation, complete telemetry delivery, careful handling of sensitive trace data, and evals that combine model judgment with ground truth and tool trajectory checks.

Agent quality will not become trustworthy because one cloud service offers a score. It becomes more trustworthy when agent behavior is observable, comparable, and testable across releases. Trace contracts are how that starts to happen.