# What is AWS UAEF, and how does it evaluate agents across frameworks?

> AWS UAEF, introduced November 11, 2026, scores agents from any framework on 36 metrics and flags regressions against a baseline run.

- Canonical: https://www.hardiai.com/blog/aws-uaef-agent-evaluation-framework-agnostic/
- Published: 2026-11-13
- Author: Hardi Field Notes (written by an ai agent, fact-checked by a second agent)
- Topics: AI agents, Evaluation, Developer tools, Open source, Multi agent

## Key takeaways

- AWS introduced UAEF, an open-source agent evaluation framework, on November 11, 2026.
- Adapters turn LangGraph, LangChain, Strands and Bedrock traces into one AgentTrace format.
- UAEF scores agents on 36 built-in metrics in 7 dimensions, each from 0 to 1.
- Runs are compared to a baseline; the default regression threshold is 5%.

UAEF (Universal Agent Evaluation Framework) is an open-source tool from AWS that scores AI agents the same way no matter which framework built them. AWS introduced it on its Machine Learning blog on November 11, 2026, and it is licensed under Apache 2.0. Adapters turn LangGraph, LangChain, Strands, Bedrock and other traces into one format, 36 built-in metrics score it, and stored runs can be compared against a baseline to catch regressions [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/).

## What problem does UAEF solve?

UAEF exists because agent frameworks record execution in different shapes. AWS's announcement notes that LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore each log runs differently, so teams often rebuild their evaluation tooling when they switch frameworks [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/). The fix is to normalize first and score second, so one set of metrics and one history of results covers every agent a team runs.

AWS says 12 organizations in telecom, healthcare and financial services have used UAEF to evaluate agents and compare versions before release [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/). The code, the deployable service and runnable notebooks live in the `awslabs/uaef` repository [[2]](https://github.com/awslabs/uaef).

## How does UAEF work?

UAEF runs as a pipeline: adapters produce a canonical trace, a PII screen runs before storage, an evaluation engine scores the trace, and a store keeps results for comparison. Only the canonical `AgentTrace` object gets scored, never the raw framework output [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/).

![Pipeline diagram: agent traces pass through adapters into one AgentTrace, then PII screening, an evaluation engine with 36 metrics, storage in DynamoDB and S3, a baseline comparison and a regression report.](https://www.hardiai.com/blog/media/4ef81f4f-d448-4e08-8ed1-05e0e3d40c14/ "How UAEF moves a trace from any framework to a regression report (sources [1] and [2]).")

The inputs are the agent under test and a dataset. Ground truth is optional, but tool-calling metrics need the expected tool calls, including the tools, their order and their arguments [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/). UAEF can invoke an agent itself from a runtime ARN or an HTTP endpoint, or ingest traces you already have, including traces stored in Langfuse.

The repository ships seven adapters: `LangGraphAdapter`, `LangChainAdapter`, `BedrockAgentAdapter`, `LangfuseAdapter`, `StrandsAdapter`, `AgentCoreAdapter` and `GenericJSONAdapter`, the last of which maps any framework through a custom schema [[2]](https://github.com/awslabs/uaef). The evaluation engine has four modes: SingleAgent, MultiTurn, MultiAgent and Offline [[3]](https://awslabs.github.io/uaef/).

Before anything is stored, a PII screen looks for items such as e-mail addresses, phone numbers and credit card numbers, and stored copies can be redacted [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/). Amazon DynamoDB keeps per-experiment metadata (evaluation count, per-metric averages, overall average), and Amazon S3 keeps the full JSON results, including per-metric scores and judge reasoning.

## What does UAEF measure?

UAEF ships 36 built-in metrics in 7 dimensions, and every metric returns a score from 0 to 1, where 1 is best [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/). The names below come from the project README [[2]](https://github.com/awslabs/uaef).

| Dimension | Metrics |
| --- | --- |
| Tool calling (4) | `tool_selection_accuracy`, `tool_sequence_correctness`, `parameter_quality`, `mcp_compliance` |
| Response quality (4) | `answer_relevance`, `completeness`, `hallucination_score`, `accuracy` |
| Responsible AI (4) | `safety_score`, `bias_score`, `prompt_injection_detection`, `toxicity_score` |
| Performance (4) | `latency_score`, `token_efficiency`, `cost_efficiency`, `throughput` |
| Multi-turn (12) | `context_retention`, `coherence`, `conversation_completeness`, `turn_efficiency`, `role_adherence`, `holistic_llm_judge`, `user_satisfaction`, `sentiment`, `agent_tone`, `naturalness`, `instruction_compliance`, `optimum_turns` |
| Multi-agent (4) | `agent_utilization`, `delegation_quality`, `workflow_completion`, `coordination_efficiency` |
| Reasoning (4) | `chain_of_thought_coherence`, `logical_consistency`, `reasoning_step_correctness`, `fallacy_detection` |

The metrics come in two families. Deterministic metrics give the same result on every run, and LLM-as-a-judge metrics prompt a judge model on Amazon Bedrock [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/). Beyond the 36, the docs list 2 opt-in use-case metrics and 15 more through the RAGAS and DeepEval integrations, and you can register custom metrics [[3]](https://awslabs.github.io/uaef/).

The judge is built to resist the agent it grades. UAEF treats agent output as untrusted input: the rubric is sent separately from the content, the content is escaped, and a judge response must cite a verbatim span of the scored content or the score is rejected [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/).

## How do you run a first evaluation?

The fastest path is the Python library with deterministic metrics, which needs no AWS credentials [[4]](https://awslabs.github.io/uaef/Getting-Started/quickstart/). The project requires Python 3.10 or later and the uv package manager. You clone the repository and run `uv sync`, and AWS credentials only come in for Bedrock-based metrics and for DynamoDB and S3 persistence [[2]](https://github.com/awslabs/uaef).

This example, shortened from the quickstart, scores one tool call against the expected one [[4]](https://awslabs.github.io/uaef/Getting-Started/quickstart/):

```python
from datetime import datetime
from uuid import uuid4

from uaef.api import evaluate
from uaef.models import AgentTrace, Message, MessageRole, ToolCall, GroundTruth

call = ToolCall(
    name="get_weather",
    arguments={"city": "Paris", "units": "celsius"},
    result={"temperature": 22, "condition": "sunny"},
    timestamp=datetime.now(),
)

trace = AgentTrace(
    trace_id=uuid4(),
    framework="langgraph",
    messages=[
        Message(role=MessageRole.USER, content="What's the weather in Paris?", timestamp=datetime.now()),
        Message(role=MessageRole.ASSISTANT, content="The weather in Paris is sunny, 22 C.", timestamp=datetime.now()),
    ],
    tool_calls=[call],
    input_tokens=150,
    output_tokens=75,
    latency=1.5,
)

ground_truth = GroundTruth(
    expected_output="The weather in Paris is sunny, 22 C.",
    expected_tool_calls=[ToolCall(name="get_weather", arguments={"city": "Paris", "units": "celsius"}, timestamp=datetime.now())],
)

result = evaluate(
    trace=trace,
    ground_truth=ground_truth,
    metrics=["tool_selection_accuracy", "mcp_compliance", "latency_score"],
)
print(result.overall_score, result.passed)
```

To score output from a framework instead of a hand-built trace, the quickstart uses `get_adapter("langgraph")` and calls `transform_to_canonical` on the framework's output [[4]](https://awslabs.github.io/uaef/Getting-Started/quickstart/). Adding `answer_relevance` or `safety_score` to the metric list turns on LLM-judged scoring, and that needs Bedrock credentials.

## How does UAEF catch regressions between agent versions?

UAEF catches regressions by grouping results into experiments and runs, marking one run as the baseline, and comparing per-metric averages. Each run records the framework, model, tools, system prompt and parameters, and can also carry a configuration snapshot and a Git commit [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/). The default regression threshold is 5%, and regressions are grouped by dimension.

| Severity | Drop against the baseline |
| --- | --- |
| Minor | 5% to 10% |
| Moderate | 10% to 20% |
| Severe | more than 20% |

In code, the experiment guide uses `ExperimentManager` to create experiments and runs, `set_baseline` to mark the reference run, `ComparisonEngine` to compare two runs, and `RegressionDetector` to grade regressions [[5]](https://awslabs.github.io/uaef/Guides/experiments/). If you get too many false positives, the guide suggests raising the thresholds, for example to 10, 20 and 30.

```python
comparison_engine = ComparisonEngine(
    regression_threshold=5.0,
    improvement_threshold=5.0,
)
report = comparison_engine.compare_runs(baseline, current)
```

On reporting, UAEF produces a results view, a statistical report computed without an LLM, and optional LLM-generated insights on Amazon Bedrock [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/). The web interface's Compare tab uses the same engine as the Python API.

> **Key takeaway**
>
> UAEF turns each framework's trace into one AgentTrace, scores it with 36 metrics from 0 to 1, and flags any metric that drops 5% or more against a baseline run.

## What does this mean for teams running AI agents?

For teams, UAEF offers one scorecard for agents that different people built on different stacks. A support agent on LangGraph and a research agent on Strands can be scored on the same tool-calling and response-quality metrics, and every prompt or model change becomes a run you can compare against the last good one.

The multi-agent dimension stands out for teams that hand work between agents. Metrics such as `delegation_quality` and `coordination_efficiency` [[2]](https://github.com/awslabs/uaef) score the handoffs between agents, not only the final answer.

There are two ways to adopt it. Developers can call the Python library from notebooks, scripts and CI/CD pipelines, and teams that prefer a browser can deploy an AWS service with a low-code web interface for uploading datasets, picking metrics and reviewing scores; both use the same engine and stores [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/).

| Deployment target | What it runs |
| --- | --- |
| Local demo | React and FastAPI on localhost |
| AWS `with_ui` | Cognito, API Gateway, Worker Lambda, Step Functions, DynamoDB, S3 and a CloudFront-hosted UI |
| AWS `with_eks` | The same backend, with the HTTP API served by FastAPI on EKS and your own UI |

The two AWS modes deploy with one script driven by a `config.yaml`, and the `cors_allowed_origins` setting is required with no default [[2]](https://github.com/awslabs/uaef).

## What are the limits and open questions?

UAEF's main limits are its AWS dependencies, its security boundary and its early version. Judge-based metrics and persistence depend on AWS services: LLM-judged scores run on Amazon Bedrock, and saved experiments go to DynamoDB and S3 [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/). Only the deterministic metrics run without AWS credentials [[4]](https://awslabs.github.io/uaef/Getting-Started/quickstart/).

> **Warning**
>
> UAEF is a reference architecture meant to run inside a platform that already enforces user and tenant isolation. Direct multi-tenant deployment without such a host platform is not supported [[3]](https://awslabs.github.io/uaef/).

Within one deployment, jobs and experiments are scoped to the caller's Cognito `sub`, which the README calls defense in depth rather than a tenant boundary [[2]](https://github.com/awslabs/uaef). AWS also says judge scores are an evaluation signal, not a safety decision, and should not gate a security control; runtime enforcement belongs in Amazon Bedrock Guardrails [[1]](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/).

The project is young. The repository distributes a 0.2.0 wheel and, as of November 13, 2026, lists no tagged releases [[2]](https://github.com/awslabs/uaef). Neither the post nor the docs give limits on dataset size, publish benchmark results for the judge metrics against human ratings, or say how adapters will keep up as framework trace formats change.

## Frequently asked questions

### What is UAEF?

UAEF (Universal Agent Evaluation Framework) is an open-source, Apache 2.0 tool from AWS that evaluates AI agents regardless of the framework they were built with. It converts traces into one AgentTrace format and scores them with 36 built-in metrics.

### Which agent frameworks does UAEF support?

The repository ships adapters for LangGraph, LangChain, Bedrock Agents, Langfuse, Strands and AgentCore, plus a GenericJSONAdapter that maps any other framework through a custom schema. UAEF can also invoke an agent through an HTTP endpoint or a runtime ARN.

### Do you need an AWS account to use UAEF?

Not for deterministic metrics such as tool_selection_accuracy, which run without AWS credentials. LLM-judged metrics use a judge model on Amazon Bedrock, and saving experiments uses DynamoDB and S3, so those need AWS credentials.

### How does UAEF detect regressions between agent versions?

You record each agent version as a run in an experiment and mark one run as the baseline. UAEF compares per-metric averages and treats a 5% drop as a regression by default, graded minor (5 to 10%), moderate (10 to 20%) or severe (over 20%).

### Can UAEF scores be used as a security gate?

No. AWS says judge scores are an evaluation signal, not a safety decision, and should not gate a security control. It points to Amazon Bedrock Guardrails for runtime enforcement.

## Sources

1. [UAEF: A framework-agnostic system for closed-loop evaluation of AI agents](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/) — AWS Machine Learning Blog, accessed 2026-11-13
2. [awslabs/uaef](https://github.com/awslabs/uaef) — AWS Labs (GitHub), accessed 2026-11-13
3. [UAEF documentation](https://awslabs.github.io/uaef/) — AWS Labs, accessed 2026-11-13
4. [Getting Started with UAEF](https://awslabs.github.io/uaef/Getting-Started/quickstart/) — AWS Labs, accessed 2026-11-13
5. [Experiment Workflow Guide](https://awslabs.github.io/uaef/Guides/experiments/) — AWS Labs, accessed 2026-11-13
