What is AWS UAEF, and how does it evaluate agents across frameworks?

AWS UAEF, introduced November 11, 2026, scores agents from any framework on 36 metrics and flags regressions against a baseline run.

한국어로 읽기

Key takeaways

  • AWS introduced UAEF, an open-source agent evaluation framework, on November 11, 2026.
  • Adapters turn LangGraph, LangChain, Strands and Bedrock traces into one AgentTrace format.
  • UAEF scores agents on 36 built-in metrics in 7 dimensions, each from 0 to 1.
  • Runs are compared to a baseline; the default regression threshold is 5%.
On this page
  1. What problem does UAEF solve?
  2. How does UAEF work?
  3. What does UAEF measure?
  4. How do you run a first evaluation?
  5. How does UAEF catch regressions between agent versions?
  6. What does this mean for teams running AI agents?
  7. What are the limits and open questions?

UAEF (Universal Agent Evaluation Framework) is an open-source tool from AWS that scores AI agents the same way no matter which framework built them. AWS introduced it on its Machine Learning blog on November 11, 2026, and it is licensed under Apache 2.0. Adapters turn LangGraph, LangChain, Strands, Bedrock and other traces into one format, 36 built-in metrics score it, and stored runs can be compared against a baseline to catch regressions [1].

What problem does UAEF solve?

UAEF exists because agent frameworks record execution in different shapes. AWS's announcement notes that LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore each log runs differently, so teams often rebuild their evaluation tooling when they switch frameworks [1]. The fix is to normalize first and score second, so one set of metrics and one history of results covers every agent a team runs.

AWS says 12 organizations in telecom, healthcare and financial services have used UAEF to evaluate agents and compare versions before release [1]. The code, the deployable service and runnable notebooks live in the awslabs/uaef repository [2].

How does UAEF work?

UAEF runs as a pipeline: adapters produce a canonical trace, a PII screen runs before storage, an evaluation engine scores the trace, and a store keeps results for comparison. Only the canonical AgentTrace object gets scored, never the raw framework output [1].

Pipeline diagram: agent traces pass through adapters into one AgentTrace, then PII screening, an evaluation engine with 36 metrics, storage in DynamoDB and S3, a baseline comparison and a regression report.
How UAEF moves a trace from any framework to a regression report (sources [1] and [2]).

The inputs are the agent under test and a dataset. Ground truth is optional, but tool-calling metrics need the expected tool calls, including the tools, their order and their arguments [1]. UAEF can invoke an agent itself from a runtime ARN or an HTTP endpoint, or ingest traces you already have, including traces stored in Langfuse.

The repository ships seven adapters: LangGraphAdapter, LangChainAdapter, BedrockAgentAdapter, LangfuseAdapter, StrandsAdapter, AgentCoreAdapter and GenericJSONAdapter, the last of which maps any framework through a custom schema [2]. The evaluation engine has four modes: SingleAgent, MultiTurn, MultiAgent and Offline [3].

Before anything is stored, a PII screen looks for items such as e-mail addresses, phone numbers and credit card numbers, and stored copies can be redacted [1]. Amazon DynamoDB keeps per-experiment metadata (evaluation count, per-metric averages, overall average), and Amazon S3 keeps the full JSON results, including per-metric scores and judge reasoning.

What does UAEF measure?

UAEF ships 36 built-in metrics in 7 dimensions, and every metric returns a score from 0 to 1, where 1 is best [1]. The names below come from the project README [2].

DimensionMetrics
Tool calling (4)tool_selection_accuracy, tool_sequence_correctness, parameter_quality, mcp_compliance
Response quality (4)answer_relevance, completeness, hallucination_score, accuracy
Responsible AI (4)safety_score, bias_score, prompt_injection_detection, toxicity_score
Performance (4)latency_score, token_efficiency, cost_efficiency, throughput
Multi-turn (12)context_retention, coherence, conversation_completeness, turn_efficiency, role_adherence, holistic_llm_judge, user_satisfaction, sentiment, agent_tone, naturalness, instruction_compliance, optimum_turns
Multi-agent (4)agent_utilization, delegation_quality, workflow_completion, coordination_efficiency
Reasoning (4)chain_of_thought_coherence, logical_consistency, reasoning_step_correctness, fallacy_detection

The metrics come in two families. Deterministic metrics give the same result on every run, and LLM-as-a-judge metrics prompt a judge model on Amazon Bedrock [1]. Beyond the 36, the docs list 2 opt-in use-case metrics and 15 more through the RAGAS and DeepEval integrations, and you can register custom metrics [3].

The judge is built to resist the agent it grades. UAEF treats agent output as untrusted input: the rubric is sent separately from the content, the content is escaped, and a judge response must cite a verbatim span of the scored content or the score is rejected [1].

How do you run a first evaluation?

The fastest path is the Python library with deterministic metrics, which needs no AWS credentials [4]. The project requires Python 3.10 or later and the uv package manager. You clone the repository and run uv sync, and AWS credentials only come in for Bedrock-based metrics and for DynamoDB and S3 persistence [2].

This example, shortened from the quickstart, scores one tool call against the expected one [4]:

Python
from datetime import datetime
from uuid import uuid4

from uaef.api import evaluate
from uaef.models import AgentTrace, Message, MessageRole, ToolCall, GroundTruth

call = ToolCall(
    name="get_weather",
    arguments={"city": "Paris", "units": "celsius"},
    result={"temperature": 22, "condition": "sunny"},
    timestamp=datetime.now(),
)

trace = AgentTrace(
    trace_id=uuid4(),
    framework="langgraph",
    messages=[
        Message(role=MessageRole.USER, content="What's the weather in Paris?", timestamp=datetime.now()),
        Message(role=MessageRole.ASSISTANT, content="The weather in Paris is sunny, 22 C.", timestamp=datetime.now()),
    ],
    tool_calls=[call],
    input_tokens=150,
    output_tokens=75,
    latency=1.5,
)

ground_truth = GroundTruth(
    expected_output="The weather in Paris is sunny, 22 C.",
    expected_tool_calls=[ToolCall(name="get_weather", arguments={"city": "Paris", "units": "celsius"}, timestamp=datetime.now())],
)

result = evaluate(
    trace=trace,
    ground_truth=ground_truth,
    metrics=["tool_selection_accuracy", "mcp_compliance", "latency_score"],
)
print(result.overall_score, result.passed)

To score output from a framework instead of a hand-built trace, the quickstart uses get_adapter("langgraph") and calls transform_to_canonical on the framework's output [4]. Adding answer_relevance or safety_score to the metric list turns on LLM-judged scoring, and that needs Bedrock credentials.

How does UAEF catch regressions between agent versions?

UAEF catches regressions by grouping results into experiments and runs, marking one run as the baseline, and comparing per-metric averages. Each run records the framework, model, tools, system prompt and parameters, and can also carry a configuration snapshot and a Git commit [1]. The default regression threshold is 5%, and regressions are grouped by dimension.

SeverityDrop against the baseline
Minor5% to 10%
Moderate10% to 20%
Severemore than 20%

In code, the experiment guide uses ExperimentManager to create experiments and runs, set_baseline to mark the reference run, ComparisonEngine to compare two runs, and RegressionDetector to grade regressions [5]. If you get too many false positives, the guide suggests raising the thresholds, for example to 10, 20 and 30.

Python
comparison_engine = ComparisonEngine(
    regression_threshold=5.0,
    improvement_threshold=5.0,
)
report = comparison_engine.compare_runs(baseline, current)

On reporting, UAEF produces a results view, a statistical report computed without an LLM, and optional LLM-generated insights on Amazon Bedrock [1]. The web interface's Compare tab uses the same engine as the Python API.

What does this mean for teams running AI agents?

For teams, UAEF offers one scorecard for agents that different people built on different stacks. A support agent on LangGraph and a research agent on Strands can be scored on the same tool-calling and response-quality metrics, and every prompt or model change becomes a run you can compare against the last good one.

The multi-agent dimension stands out for teams that hand work between agents. Metrics such as delegation_quality and coordination_efficiency [2] score the handoffs between agents, not only the final answer.

There are two ways to adopt it. Developers can call the Python library from notebooks, scripts and CI/CD pipelines, and teams that prefer a browser can deploy an AWS service with a low-code web interface for uploading datasets, picking metrics and reviewing scores; both use the same engine and stores [1].

Deployment targetWhat it runs
Local demoReact and FastAPI on localhost
AWS with_uiCognito, API Gateway, Worker Lambda, Step Functions, DynamoDB, S3 and a CloudFront-hosted UI
AWS with_eksThe same backend, with the HTTP API served by FastAPI on EKS and your own UI

The two AWS modes deploy with one script driven by a config.yaml, and the cors_allowed_origins setting is required with no default [2].

What are the limits and open questions?

UAEF's main limits are its AWS dependencies, its security boundary and its early version. Judge-based metrics and persistence depend on AWS services: LLM-judged scores run on Amazon Bedrock, and saved experiments go to DynamoDB and S3 [1]. Only the deterministic metrics run without AWS credentials [4].

Within one deployment, jobs and experiments are scoped to the caller's Cognito sub, which the README calls defense in depth rather than a tenant boundary [2]. AWS also says judge scores are an evaluation signal, not a safety decision, and should not gate a security control; runtime enforcement belongs in Amazon Bedrock Guardrails [1].

The project is young. The repository distributes a 0.2.0 wheel and, as of November 13, 2026, lists no tagged releases [2]. Neither the post nor the docs give limits on dataset size, publish benchmark results for the judge metrics against human ratings, or say how adapters will keep up as framework trace formats change.

Frequently asked questions

What is UAEF?

UAEF (Universal Agent Evaluation Framework) is an open-source, Apache 2.0 tool from AWS that evaluates AI agents regardless of the framework they were built with. It converts traces into one AgentTrace format and scores them with 36 built-in metrics.

Which agent frameworks does UAEF support?

The repository ships adapters for LangGraph, LangChain, Bedrock Agents, Langfuse, Strands and AgentCore, plus a GenericJSONAdapter that maps any other framework through a custom schema. UAEF can also invoke an agent through an HTTP endpoint or a runtime ARN.

Do you need an AWS account to use UAEF?

Not for deterministic metrics such as tool_selection_accuracy, which run without AWS credentials. LLM-judged metrics use a judge model on Amazon Bedrock, and saving experiments uses DynamoDB and S3, so those need AWS credentials.

How does UAEF detect regressions between agent versions?

You record each agent version as a run in an experiment and mark one run as the baseline. UAEF compares per-metric averages and treats a 5% drop as a regression by default, graded minor (5 to 10%), moderate (10 to 20%) or severe (over 20%).

Can UAEF scores be used as a security gate?

No. AWS says judge scores are an evaluation signal, not a safety decision, and should not gate a security control. It points to Amazon Bedrock Guardrails for runtime enforcement.

Sources

  1. UAEF: A framework-agnostic system for closed-loop evaluation of AI agents — AWS Machine Learning Blog, accessed
  2. awslabs/uaef — AWS Labs (GitHub), accessed
  3. UAEF documentation — AWS Labs, accessed
  4. Getting Started with UAEF — AWS Labs, accessed
  5. Experiment Workflow Guide — AWS Labs, accessed