What is AWS UAEF, and how does it evaluate agents across frameworks?
AWS UAEF, introduced November 11, 2026, scores agents from any framework on 36 metrics and flags regressions against a baseline run.
Key takeaways
- AWS introduced UAEF, an open-source agent evaluation framework, on November 11, 2026.
- Adapters turn LangGraph, LangChain, Strands and Bedrock traces into one AgentTrace format.
- UAEF scores agents on 36 built-in metrics in 7 dimensions, each from 0 to 1.
- Runs are compared to a baseline; the default regression threshold is 5%.
On this page
UAEF (Universal Agent Evaluation Framework) is an open-source tool from AWS that scores AI agents the same way no matter which framework built them. AWS introduced it on its Machine Learning blog on November 11, 2026, and it is licensed under Apache 2.0. Adapters turn LangGraph, LangChain, Strands, Bedrock and other traces into one format, 36 built-in metrics score it, and stored runs can be compared against a baseline to catch regressions [1].
What problem does UAEF solve?
UAEF exists because agent frameworks record execution in different shapes. AWS's announcement notes that LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore each log runs differently, so teams often rebuild their evaluation tooling when they switch frameworks [1]. The fix is to normalize first and score second, so one set of metrics and one history of results covers every agent a team runs.
AWS says 12 organizations in telecom, healthcare and financial services have used UAEF to evaluate agents and compare versions before release [1]. The code, the deployable service and runnable notebooks live in the awslabs/uaef repository [2].
How does UAEF work?
UAEF runs as a pipeline: adapters produce a canonical trace, a PII screen runs before storage, an evaluation engine scores the trace, and a store keeps results for comparison. Only the canonical AgentTrace object gets scored, never the raw framework output [1].
The inputs are the agent under test and a dataset. Ground truth is optional, but tool-calling metrics need the expected tool calls, including the tools, their order and their arguments [1]. UAEF can invoke an agent itself from a runtime ARN or an HTTP endpoint, or ingest traces you already have, including traces stored in Langfuse.
The repository ships seven adapters: LangGraphAdapter, LangChainAdapter, BedrockAgentAdapter, LangfuseAdapter, StrandsAdapter, AgentCoreAdapter and GenericJSONAdapter, the last of which maps any framework through a custom schema [2]. The evaluation engine has four modes: SingleAgent, MultiTurn, MultiAgent and Offline [3].
Before anything is stored, a PII screen looks for items such as e-mail addresses, phone numbers and credit card numbers, and stored copies can be redacted [1]. Amazon DynamoDB keeps per-experiment metadata (evaluation count, per-metric averages, overall average), and Amazon S3 keeps the full JSON results, including per-metric scores and judge reasoning.
What does UAEF measure?
UAEF ships 36 built-in metrics in 7 dimensions, and every metric returns a score from 0 to 1, where 1 is best [1]. The names below come from the project README [2].
| Dimension | Metrics |
|---|---|
| Tool calling (4) | tool_selection_accuracy, tool_sequence_correctness, parameter_quality, mcp_compliance |
| Response quality (4) | answer_relevance, completeness, hallucination_score, accuracy |
| Responsible AI (4) | safety_score, bias_score, prompt_injection_detection, toxicity_score |
| Performance (4) | latency_score, token_efficiency, cost_efficiency, throughput |
| Multi-turn (12) | context_retention, coherence, conversation_completeness, turn_efficiency, role_adherence, holistic_llm_judge, user_satisfaction, sentiment, agent_tone, naturalness, instruction_compliance, optimum_turns |
| Multi-agent (4) | agent_utilization, delegation_quality, workflow_completion, coordination_efficiency |
| Reasoning (4) | chain_of_thought_coherence, logical_consistency, reasoning_step_correctness, fallacy_detection |
The metrics come in two families. Deterministic metrics give the same result on every run, and LLM-as-a-judge metrics prompt a judge model on Amazon Bedrock [1]. Beyond the 36, the docs list 2 opt-in use-case metrics and 15 more through the RAGAS and DeepEval integrations, and you can register custom metrics [3].
The judge is built to resist the agent it grades. UAEF treats agent output as untrusted input: the rubric is sent separately from the content, the content is escaped, and a judge response must cite a verbatim span of the scored content or the score is rejected [1].
How do you run a first evaluation?
The fastest path is the Python library with deterministic metrics, which needs no AWS credentials [4]. The project requires Python 3.10 or later and the uv package manager. You clone the repository and run uv sync, and AWS credentials only come in for Bedrock-based metrics and for DynamoDB and S3 persistence [2].
This example, shortened from the quickstart, scores one tool call against the expected one [4]:
from datetime import datetime
from uuid import uuid4
from uaef.api import evaluate
from uaef.models import AgentTrace, Message, MessageRole, ToolCall, GroundTruth
call = ToolCall(
name="get_weather",
arguments={"city": "Paris", "units": "celsius"},
result={"temperature": 22, "condition": "sunny"},
timestamp=datetime.now(),
)
trace = AgentTrace(
trace_id=uuid4(),
framework="langgraph",
messages=[
Message(role=MessageRole.USER, content="What's the weather in Paris?", timestamp=datetime.now()),
Message(role=MessageRole.ASSISTANT, content="The weather in Paris is sunny, 22 C.", timestamp=datetime.now()),
],
tool_calls=[call],
input_tokens=150,
output_tokens=75,
latency=1.5,
)
ground_truth = GroundTruth(
expected_output="The weather in Paris is sunny, 22 C.",
expected_tool_calls=[ToolCall(name="get_weather", arguments={"city": "Paris", "units": "celsius"}, timestamp=datetime.now())],
)
result = evaluate(
trace=trace,
ground_truth=ground_truth,
metrics=["tool_selection_accuracy", "mcp_compliance", "latency_score"],
)
print(result.overall_score, result.passed)To score output from a framework instead of a hand-built trace, the quickstart uses get_adapter("langgraph") and calls transform_to_canonical on the framework's output [4]. Adding answer_relevance or safety_score to the metric list turns on LLM-judged scoring, and that needs Bedrock credentials.
How does UAEF catch regressions between agent versions?
UAEF catches regressions by grouping results into experiments and runs, marking one run as the baseline, and comparing per-metric averages. Each run records the framework, model, tools, system prompt and parameters, and can also carry a configuration snapshot and a Git commit [1]. The default regression threshold is 5%, and regressions are grouped by dimension.
| Severity | Drop against the baseline |
|---|---|
| Minor | 5% to 10% |
| Moderate | 10% to 20% |
| Severe | more than 20% |
In code, the experiment guide uses ExperimentManager to create experiments and runs, set_baseline to mark the reference run, ComparisonEngine to compare two runs, and RegressionDetector to grade regressions [5]. If you get too many false positives, the guide suggests raising the thresholds, for example to 10, 20 and 30.
comparison_engine = ComparisonEngine(
regression_threshold=5.0,
improvement_threshold=5.0,
)
report = comparison_engine.compare_runs(baseline, current)On reporting, UAEF produces a results view, a statistical report computed without an LLM, and optional LLM-generated insights on Amazon Bedrock [1]. The web interface's Compare tab uses the same engine as the Python API.
What does this mean for teams running AI agents?
For teams, UAEF offers one scorecard for agents that different people built on different stacks. A support agent on LangGraph and a research agent on Strands can be scored on the same tool-calling and response-quality metrics, and every prompt or model change becomes a run you can compare against the last good one.
The multi-agent dimension stands out for teams that hand work between agents. Metrics such as delegation_quality and coordination_efficiency [2] score the handoffs between agents, not only the final answer.
There are two ways to adopt it. Developers can call the Python library from notebooks, scripts and CI/CD pipelines, and teams that prefer a browser can deploy an AWS service with a low-code web interface for uploading datasets, picking metrics and reviewing scores; both use the same engine and stores [1].
| Deployment target | What it runs |
|---|---|
| Local demo | React and FastAPI on localhost |
AWS with_ui | Cognito, API Gateway, Worker Lambda, Step Functions, DynamoDB, S3 and a CloudFront-hosted UI |
AWS with_eks | The same backend, with the HTTP API served by FastAPI on EKS and your own UI |
The two AWS modes deploy with one script driven by a config.yaml, and the cors_allowed_origins setting is required with no default [2].
What are the limits and open questions?
UAEF's main limits are its AWS dependencies, its security boundary and its early version. Judge-based metrics and persistence depend on AWS services: LLM-judged scores run on Amazon Bedrock, and saved experiments go to DynamoDB and S3 [1]. Only the deterministic metrics run without AWS credentials [4].
Within one deployment, jobs and experiments are scoped to the caller's Cognito sub, which the README calls defense in depth rather than a tenant boundary [2]. AWS also says judge scores are an evaluation signal, not a safety decision, and should not gate a security control; runtime enforcement belongs in Amazon Bedrock Guardrails [1].
The project is young. The repository distributes a 0.2.0 wheel and, as of November 13, 2026, lists no tagged releases [2]. Neither the post nor the docs give limits on dataset size, publish benchmark results for the judge metrics against human ratings, or say how adapters will keep up as framework trace formats change.
Frequently asked questions
What is UAEF?
UAEF (Universal Agent Evaluation Framework) is an open-source, Apache 2.0 tool from AWS that evaluates AI agents regardless of the framework they were built with. It converts traces into one AgentTrace format and scores them with 36 built-in metrics.
Which agent frameworks does UAEF support?
The repository ships adapters for LangGraph, LangChain, Bedrock Agents, Langfuse, Strands and AgentCore, plus a GenericJSONAdapter that maps any other framework through a custom schema. UAEF can also invoke an agent through an HTTP endpoint or a runtime ARN.
Do you need an AWS account to use UAEF?
Not for deterministic metrics such as tool_selection_accuracy, which run without AWS credentials. LLM-judged metrics use a judge model on Amazon Bedrock, and saving experiments uses DynamoDB and S3, so those need AWS credentials.
How does UAEF detect regressions between agent versions?
You record each agent version as a run in an experiment and mark one run as the baseline. UAEF compares per-metric averages and treats a 5% drop as a regression by default, graded minor (5 to 10%), moderate (10 to 20%) or severe (over 20%).
Can UAEF scores be used as a security gate?
No. AWS says judge scores are an evaluation signal, not a safety decision, and should not gate a security control. It points to Amazon Bedrock Guardrails for runtime enforcement.