10 Best AI Agent Evaluation Tools in 2026, Open Source and Paid

Last updated October 12, 2026. Prices, plan limits, licenses, ownership, and eval features were checked against each vendor’s own pricing pages, documentation, and GitHub repositories on that date.

An agent can return the right answer after calling the wrong tool, skipping a policy check, or looping six times, and a score on the final reply will never show it.

The best AI agent evaluation tools in 2026 are Braintrust for eval-gated releases, LangSmith for LangGraph agents, Langfuse or Arize Phoenix for free self-hosting, and DeepEval for evals written as tests. This guide covers the evals layer of our AgentOps tools map, and every pick can score tool calls and trajectories as well as final answers.

We compared 10 AI agent evaluation tools on agent-specific metrics, online and offline scoring, CI gates, self-hosting, licensing, and price. No vendor paid for placement or reviewed this page, per our editorial standards and disclosure policy.

The tool matters less than whether it scores all three levels of an agent run: the final response, the trajectory, and each tool call. Most teams also need one place where offline test suites and live production scores meet, which is where the platforms on this list differ most.

The 10 Best AI Agent Evaluation Tools at a Glance

The 10 best AI agent evaluation tools split into hosted platforms (Braintrust, LangSmith, Maxim AI), open-source platforms you can self-host (Langfuse, Arize Phoenix, Opik, MLflow), and code-first frameworks (DeepEval, promptfoo, Inspect). The table compares them on the facts that usually decide a shortlist.

Tool	Type	License and self-hosting	Agent-specific evals	Online scoring	CI gate	Entry price	Best for
Braintrust	Hosted platform	Proprietary; self-hosted data plane on Enterprise	Custom scorers on trace spans, Loop agent	Yes, async on live traces	GitHub Action with PR comment	Free; Pro $249/month	Eval-gated release workflows
LangSmith	Hosted platform	Proprietary; self-hosted on Enterprise	Trajectory match and judge (agentevals), multi-turn	Yes, runs and threads	pytest, Vitest, Jest	Free 1 seat; Plus $39/seat	LangChain and LangGraph agents
Langfuse	Open-source platform	MIT core; free self-host	LLM judges on observations or experiments	Yes, observation-level judges	experiment-action GitHub Action	Free; Core $29/month	Open-source teams that want one platform
Arize Phoenix	Open-source platform	Elastic License 2.0; free self-host	Tool Selection, Tool Invocation, Tool Response Handling	Yes, on traces	SDK experiments in your pipeline	Free; AX Pro $50/month	OpenTelemetry-first, self-hosted evals
DeepEval and Confident AI	Framework plus platform	Apache 2.0 framework; Confident AI on-prem on Enterprise	Task Completion, Step Efficiency, Tool Correctness, Argument Correctness	Confident AI Starter and up	deepeval test run (pytest)	Free; Starter $200/month	Python teams that want evals as unit tests
Opik (Comet)	Open-source platform	Apache 2.0; free self-host	Trajectory Accuracy, Agent Tool Correctness	Yes, online evaluation rules	SDK test suites in your pipeline	Free; Pro Cloud $19/month	Budget teams that want Apache 2.0
Maxim AI	Hosted platform	Proprietary; in-VPC on Enterprise	Multi-turn simulation, agent trajectory, step completion	Professional plan and up	CI/CD integrations	Free 3 seats; $29/seat	Chat and voice agents before launch
promptfoo	CLI framework	MIT; runs locally	Trajectory tool-sequence, tool-args-match, goal-success	Enterprise monitoring only	GitHub Action, before vs after	Free; Enterprise custom	Config-driven evals and security tests in CI
Inspect (UK AISI)	Python framework	MIT; runs locally	Sandboxed agent tasks, custom scorers, 200+ ready evals	None	CLI in your pipeline	Free	Sandboxed, benchmark-style agent evals
MLflow	Open-source platform	Apache 2.0; free self-host, managed on Databricks	ToolCallCorrectness, ToolCallEfficiency judges	Yes, production monitoring	Regression testing docs	Free; Databricks usage	Teams already on MLflow or Databricks

Verdict: Braintrust is the most complete hosted choice, and Langfuse is the safest open-source default. The code-first frameworks cost nothing to license and pair well with either, so many teams run DeepEval or promptfoo in CI and a platform for production scoring.

What Is AI Agent Evaluation, and Why Is It Harder Than LLM Evaluation?

AI agent evaluation is the practice of scoring how an agent completes a task, including its plan, tool calls, and intermediate steps, as well as its final reply. It is harder than LLM evaluation because a correct answer can hide a wrong path, and because agents take actions that a text score never sees.

The three levels every agent eval should score

• Final response: did the agent complete the task? Metrics include task completion, goal success, and correctness against expected facts.
• Trajectory: did it take a sound, efficient path? A trajectory is the ordered list of plans, model calls, tool calls, retries, and handoffs in one run, scored with sequence matching or an LLM judge.
• Single step: was each tool call right? Step-level metrics check tool selection, argument correctness, and how the agent handled each tool’s result.

A trajectory eval catches the refund agent that issues a refund without checking the policy, even when the customer gets the right answer. Google’s Gen AI evaluation service, for example, scores trajectories with exact-match, in-order, any-order, precision, and recall metrics, per its agent evaluation docs.

Offline evals, online evals, and CI gates

Offline evals run your agent against a fixed dataset of test cases before release, so results are reproducible and can block a merge. Online evals score a sample of live production traces after release, usually with LLM-as-a-judge, because live traffic has no expected answer.

The strongest setups connect the two. A failing production trace becomes a new test case, the suite runs on every pull request, and a score threshold fails the build. Braintrust’s evaluation docs describe this loop from playground to experiment to CI to online scoring.

LLM judges versus code checks

Code checks, such as exact tool-name matches or argument validation, are cheap, deterministic, and best for tool-call correctness. LLM judges handle open-ended questions like plan quality or helpfulness, but they cost tokens and drift, so teams calibrate them against human labels before trusting them in a CI gate.

What Changed in AI Agent Evaluation Tools in 2026

Six ownership and product changes in 2026 reshaped the AI agent evaluation market. OpenAI is shutting its hosted Evals platform, promptfoo agreed to join OpenAI, Cisco bought Galileo, Patronus AI pivoted to training environments, Humanloop is sunsetting its platform, and W&B Weave has a CoreWeave successor. Many ranking pages still recommend the affected tools.

• OpenAI Evals platform: existing evals become read-only on October 31, 2026, and the dashboard and API shut down on November 30, 2026, per OpenAI’s deprecations page. OpenAI’s migration guide recommends promptfoo and can export supported evals as promptfoo configs.
• promptfoo and OpenAI: promptfoo agreed to join OpenAI on March 9, 2026, with closing subject to customary conditions. promptfoo says it will stay open source and keep supporting other model providers.
• Galileo and Cisco: Cisco completed its acquisition of Galileo on May 22, 2026, and Galileo’s pricing page now redirects to Splunk. Our Braintrust vs Maxim vs Galileo comparison covers what changed for Galileo buyers.
• Patronus AI: its pricing page now describes training data, tasks, and simulated environments for frontier AI labs, after a $50 million Series B built around digital world models.
• Humanloop: the team joined Anthropic and the Humanloop platform is being sunset, per its homepage.
• W&B Weave: the Weave docs now call CoreWeave Agent Lens the successor to Weave.

Langfuse also changed hands, joining ClickHouse in January 2026 with its MIT license and self-hosting unchanged, per the Langfuse announcement. We kept it on the list because its product and license terms did not change.

How We Chose These AI Agent Evaluation Tools

We started from 19 tools that appear in current rankings for AI agent evaluation tools and LLM evaluation tools, and in AI answers to those queries. We kept the 10 that are actively maintained, score agent tool calls or trajectories, and publish pricing or a free license, then reviewed each vendor’s pages on October 12, 2026.

1. Agent depth: built-in metrics or assertions for trajectories and tool calls, beyond final-answer scoring.
2. Offline and online coverage: test suites before release and scoring on live traces after it.
3. CI integration: a documented way to fail a build on a score regression.
4. Hosting and license: whether you can self-host for free, and under which license.
5. Pricing clarity: public plans, what is metered, and the first paid tier.
6. Ownership and momentum: recent releases, acquisitions, and shutdowns.

We ran the code example below ourselves with agentevals 0.0.9 on October 12, 2026, but did not benchmark judge accuracy across vendors, so treat accuracy claims as vendor claims. GitHub star counts come from the GitHub API on the same date. For tracing-first tools, see our 12 best AI agent observability tools.

The 10 Best AI Agent Evaluation Tools, Reviewed

The list starts with the platforms that cover the most of the eval loop, from datasets and judges to CI gates and production scoring, then moves to specialists. Braintrust, LangSmith, Langfuse, Phoenix, and Opik are full platforms, DeepEval, promptfoo, and Inspect are frameworks you run in code, and Maxim and MLflow each lead on one job.

1. Braintrust: best for teams whose eval scores gate releases

Verdict: Braintrust is the most complete hosted AI agent evaluation tool, and it ties datasets, scorers, CI, and production scoring into one loop.

Braintrust is a proprietary AI evaluation and observability platform built around datasets, experiments, scorers, and playgrounds, for product and engineering teams that ship on eval results. Its autoevals scorer library is MIT licensed and had 1,055 GitHub stars on October 12, 2026.

Braintrust’s agent evaluation guide recommends scoring both the whole run and each step, including tool choice and argument building. Self-hosting is Enterprise only, with the data plane in your cloud, per the self-hosting docs.

Best for: product and engineering teams that block releases on eval regressions and want reviewers in the same tool.

Key eval features

• Experiments on versioned datasets, compared side by side with regressions flagged per metric.
• Scorers from autoevals, LLM-as-a-judge, or custom code, plus classifiers that label outputs.
• Remote evals and sandboxes that connect custom agent code to the browser playground.
• Loop, a built-in agent that runs evals, generates test cases, and iterates on prompts.

Agent evals, online scoring, and CI

Agent runs are logged as traces, so scorers can read intermediate spans and grade tool calls as well as the final output. Online scoring runs asynchronously on production traces with no added latency, which lets the same scorers run before and after release.

For CI, the eval-action GitHub Action runs Braintrust evals on each pull request in Node, Python, or Go and posts a score summary as a PR comment.

Pricing

• Starter is free with $10 in model credits, 1 GB of processed data ($4 per extra GB), 10,000 scores ($2.50 per extra 1,000), and 14-day retention.
• Pro costs $249 per month with $100 in credits, 5 GB ($3 per extra GB), 50,000 scores ($1.50 per extra 1,000), and 30-day retention.
• Enterprise is custom priced and adds on-prem or hosted deployment.

Prices come from Braintrust pricing. Every plan includes unlimited users, and qualifying startups get Pro free for 6 to 12 months.

Pros and cons

Pros

• Unlimited users on every plan, so reviewers and product managers cost nothing extra.
• The PR comment workflow is the most polished CI gate on this list.
• One scorer definition runs offline and online.

Cons

• No open-source server, so there is no free self-hosted fallback.
• Starter allows one human review score per project.
• Billing on gigabytes plus scores takes modeling to forecast.

For a head-to-head with two close rivals, see our Langfuse vs LangSmith vs Braintrust comparison.

2. LangSmith: best for LangChain and LangGraph agents

Verdict: LangSmith is the natural AI agent evaluation tool for LangGraph teams, and it has the clearest trajectory-matching toolkit of the hosted platforms.

LangSmith is LangChain’s platform for tracing, evaluating, and deploying agents, used most by teams that build on LangChain and LangGraph. It supports offline evaluation on curated datasets and online evaluation of live runs and threads, per its evaluation docs.

LangChain’s agentevals package is MIT licensed, had 745 GitHub stars on October 12, 2026, and works with or without a LangSmith account.

Best for: teams that build agents on LangChain or LangGraph and want evals, tracing, and deployment in one account.

Key eval features

• Datasets from curated cases, production traces, or synthetic generation, with experiments that compare versions.
• Evaluators written as code rules, LLM-as-a-judge prompts, pairwise comparisons, or human review in annotation queues.
• Multi-turn online evaluators that score whole conversation threads.
• Code evaluators, composite evaluators, and Tuned Evaluators in beta, billed per evaluation run.

Agent evals, online scoring, and CI

agentevals compares an agent’s trajectory to a reference in strict, unordered, subset, or superset mode, and can match tool arguments exactly or with overrides. That turns a required set of tool calls into a deterministic check that needs no LLM judge.

For CI, LangSmith’s pytest plugin and Vitest and Jest integrations define datasets and evals as test cases. Online evaluators upgrade the scored trace to extended retention, which raises trace cost, per the online evaluator docs.

Pricing

• Developer is free for 1 seat with 5,000 base traces a month.
• Plus costs $39 per seat per month with 10,000 base traces, then 0.005 LangChain Standard Units per base trace, where 1 unit costs $1.
• Upgrading a trace to 180-day extended retention costs another 0.0025 units, and Tuned Evaluators cost 0.015 units per run.
• Enterprise is custom priced and adds self-hosted and hybrid deployment.

Prices come from LangSmith pricing, including the tooltip rates on the plan table. LangChain’s pricing FAQ counts an evaluator execution as a trace.

Pros and cons

Pros

• Trajectory match modes cover most agent sequence checks without judge tokens.
• Native tracing for LangGraph, Studio, and Deployment.
• Mature pytest and Vitest workflows for evals in code.

Cons

• Every invited user is a paid seat on Plus.
• Online evals push traces into the pricier extended tier.
• Self-hosting requires an Enterprise contract.

If seat pricing or Enterprise-only self-hosting rules LangSmith out, our LangSmith alternatives guide compares eight replacements. Teams still picking a runtime can start with our AI agent frameworks comparison.

3. Langfuse: best open-source agent eval platform

Verdict: Langfuse is the safest open-source AI agent evaluation platform, with free self-hosting, unlimited users from $29, and a ready-made CI gate.

Langfuse is an open-source LLM engineering platform for tracing, evals, and prompt management, used by teams that want evaluation data on infrastructure they control. The core is MIT licensed, and the repository had 35,657 GitHub stars on October 12, 2026, the most of any eval platform here.

Langfuse runs evals both online on live traces and offline on datasets, per its evaluation overview. Its v4 release claims up to 165 times faster performance.

Best for: teams that want one open-source platform for tracing, datasets, judges, and CI gates.

Key eval features

• Datasets and experiments via SDK, UI, or OpenTelemetry, with side-by-side comparison.
• LLM-as-a-judge evaluators and deterministic code evaluators.
• Judge calibration against human labels, plus annotation queues and user feedback scores.
• Score analytics and custom dashboards to track quality over time.

Agent evals, online scoring, and CI

LLM judges can target observations, such as a single tool call, or whole experiments, per the LLM-as-a-judge docs. Observation-level judges do not load sibling or child observations, so whole-trajectory judging means targeting the root observation or running an experiment.

For CI, the langfuse/experiment-action runs an experiment on each pull request, comments the result, and fails the job when your script raises RegressionError, per the CI/CD guide. It works with Cloud and self-hosted Langfuse.

Pricing

• Self-hosted open source is free under MIT, with no usage caps.
• Hobby is free with 50,000 units a month, 2 users, and 30 days of data access.
• Core costs $29 per month for 100,000 units, then $8 per 100,000, with unlimited users and 90 days of data access.
• Pro costs $199 per month with 3 years of data access, and Enterprise costs $2,499 per month.

Prices come from Langfuse pricing. A unit is any trace, observation, or score, so judge scores count toward usage.

Pros and cons

Pros

• Free self-hosting with every core eval feature.
• A packaged GitHub Action for CI regression gates.
• Unlimited users from the $29 Core plan.

Cons

• No built-in trajectory-match metric; sequence checks are custom code or judges.
• Span-heavy agents raise unit counts on Cloud.
• Running ClickHouse in production takes real operations work.

4. Arize Phoenix: best for ready-made tool-call evaluators you can self-host

Verdict: Arize Phoenix gives you the most complete set of pre-built tool-call evaluators of any free, self-hosted AI agent evaluation tool.

Arize Phoenix is an AI observability and evaluation platform from Arize AI, built on OpenTelemetry and OpenInference for teams that want evals next to their traces. It is free to self-host with no feature gates, per the self-hosting docs, and the repository had 11,793 GitHub stars on October 12, 2026.

Phoenix is source-available under the Elastic License 2.0, which allows internal use but restricts offering it as a managed service. Arize sells a separate managed platform, Arize AX.

Best for: OpenTelemetry-first teams that want strong step-level agent evals on their own infrastructure.

Key eval features

• Pre-built LLM evaluators for faithfulness, hallucination, completeness, correctness, retrieval relevance, refusal, user friction, and PII.
• Tool Selection, Tool Invocation, and Tool Response Handling evaluators for agents.
• Client-side evals in Python and TypeScript, plus server-side evaluators configured in the UI.
• Upload of ATIF agent trajectories as Phoenix traces, added in April 2026.

Agent evals, online scoring, and CI

Phoenix says its LLM eval templates are tested against golden datasets and reach an F1 score of 85% or higher, per its pre-built metrics docs. The three tool evaluators split a tool call into choice, arguments, and use of the result, which maps cleanly onto step-level agent scoring.

Evaluations run on production traces, experiment results, or any dataset, per the Phoenix evaluation docs. For CI, run Phoenix experiments from the SDK in your pipeline, since there is no dedicated Phoenix CI action.

Pricing

• Self-hosted Phoenix is free, with no license fees, usage limits, or feature gates.
• Arize AX Free includes 25,000 spans, 1 GB, 15-day retention, and unlimited users and evals.
• Arize AX Pro costs $50 per month for 50,000 spans, 10 GB, and 30-day retention.
• AX Enterprise is custom priced and adds self-hosted deployment.

Prices come from Arize pricing, reviewed October 12, 2026.

Pros and cons

Pros

• Ready-made tool-call evaluators save weeks of judge prompt writing.
• OpenTelemetry-native, so instrumentation stays portable.
• Unlimited users and evals on every AX plan.

Cons

• The Elastic License 2.0 is not an OSI-approved open-source license.
• AX Pro caps at 50,000 spans a month.
• No packaged CI gate.

5. DeepEval and Confident AI: best for agent evals written as unit tests

Verdict: DeepEval is the most popular open-source framework for writing agent evals like pytest tests, with the broadest set of named agent metrics on this list.

DeepEval is an Apache 2.0 LLM evaluation framework from Confident AI, used by Python teams that want evals in their test suite. It offers 50+ ready-to-use metrics, per its metrics docs, and the repository had 18,748 GitHub stars on October 12, 2026.

Confident AI, from the same team, is the hosted platform for test reports, tracing, online evals, and annotation. A TypeScript version of DeepEval is also open source.

Best for: Python teams that want agent evals as unit tests, with an optional hosted dashboard.

Key eval features

• Trajectory metrics: Task Completion, Step Efficiency, Plan Adherence, and Plan Quality.
• Component metrics: Tool Correctness and Argument Correctness on individual tool calls.
• G-Eval and DAG custom metrics written in plain language, plus RAG, safety, and multi-turn metrics.
• A conversation simulator and synthetic golden generation for test data.

Agent evals, online scoring, and CI

DeepEval scores the full trace of an agent run as one trajectory, then scores single spans such as a tool call or sub-agent, per its trajectory evaluation docs. Tool Correctness can require matching tool names, input parameters, or outputs.

For CI, deepeval test run wraps pytest and adds more than 8 optional LLM-testing flags, per the CI/CD docs. Online evals on live traffic need a Confident AI Starter plan or higher.

Pricing

• DeepEval is free and open source.
• Confident AI Free covers 2 seats, 1 project, 5 test runs a week, and 1 GB-month of trace spans.
• Starter costs $200 per month with unlimited seats, 5 projects, online evals, and 5 GB-months, then $1 per GB-month.
• Team costs $2,000 per month with SOC 2 and SSO, and Enterprise adds on-prem deployment.

Prices come from Confident AI pricing, reviewed October 12, 2026.

Pros and cons

Pros

• The deepest library of named agent metrics, free under Apache 2.0.
• Evals live in your repo and run with your existing tests.
• Unlimited seats from the Starter plan.

Cons

• The jump from free to $200 a month is steep for small teams.
• Most metrics use an LLM judge, so CI runs cost tokens.
• SOC 2 and SSO start at the $2,000 Team plan.

6. Opik by Comet: best Apache 2.0 platform on a budget

Verdict: Opik is the cheapest full AI agent evaluation platform on Cloud, and its open-source build has the full feature set under Apache 2.0.

Opik is an open-source LLM observability and evaluation platform built by Comet for tracing, testing, and monitoring agents. The Opik repository is Apache 2.0 licensed and had 22,518 GitHub stars on October 12, 2026.

Comet describes the open-source build as the same codebase as its hosted versions. Self-hosting runs on Docker or a Kubernetes Helm chart, per the self-hosting docs.

Best for: cost-conscious teams that want a full open-source platform without license caveats.

Key eval features

• Test Suites with natural-language assertions checked automatically by an LLM judge.
• 30+ pre-built metrics for hallucination, relevance, coherence, and more.
• Annotation queues for subject matter experts and experiment comparison.
• Ollie, a coding assistant that analyzes traces and writes fixes into your codebase.

Agent evals, online scoring, and CI

Opik’s agent metrics include Agent Task Completion, Agent Tool Correctness, and Trajectory Accuracy, which scores how closely a run follows expected steps, per its metrics overview. Its evaluation loop turns real production failures into test cases, per the Opik evaluation docs.

Production scoring runs through online evaluation rules. Opik’s docs do not list a dedicated CI action, so teams run test suites from the SDK inside their own pipeline.

Pricing

• Open source is free, with the full observability and testing feature set.
• Free Cloud includes up to 10 team members, 25,000 spans a month, and 60-day retention.
• Pro Cloud costs $19 per month for up to 50 members and 100,000 spans, then $5 per 100,000 spans.
• Extending retention to 400 days costs $29 per 100,000 spans, and Enterprise is custom priced.

Prices come from Comet pricing, reviewed October 12, 2026. Opik’s pricing page does not meter scores separately.

Pros and cons

Pros

• Apache 2.0 with the same codebase as Cloud.
• The lowest paid Cloud price on this list.
• Named trajectory and tool metrics built in.

Cons

• Cloud plans cap members at 10 on Free and 50 on Pro.
• No packaged CI gate.
• Trajectory Accuracy needs expected steps written for each case.

7. Maxim AI: best for simulating multi-turn agents before launch

Verdict: Maxim AI is the strongest hosted choice when your agent holds multi-turn chat or voice conversations and must face simulated users before release.

Maxim AI is a hosted simulation, evaluation, and observability platform for AI agents, aimed at teams shipping customer-facing chat and voice agents. It is proprietary, with in-VPC deployment on Enterprise.

Maxim also builds the open-source Bifrost LLM gateway, which now leads its homepage, so judge the eval product on its own roadmap. For a head-to-head on simulation, CI gates, and cost, see Braintrust vs Maxim vs Galileo.

Best for: teams shipping customer support, sales, or voice agents that must pass simulated conversations before release.

Key eval features

• AI-powered simulations of multi-turn conversations across scenarios and user personas.
• Pre-built and custom evaluators, including task success, agent trajectory, and step completion.
• Human evaluation pipelines alongside automated evals.
• Datasets curated from synthetic and production data, with comparison dashboards.

Agent evals, online scoring, and CI

A simulation run scores the whole conversation, the trajectory, and each step, with judge reasoning shown for every verdict. Teams can also load a partial production trace and evaluate only the next step.

Online evals on production logs start on the Professional plan, and Maxim’s FAQ describes automated eval pipelines that plug into CI/CD workflows.

Pricing

• Developer is free for up to 3 seats, 10,000 logs a month, and 3-day retention.
• Professional costs $29 per seat per month with 100,000 logs, 7-day retention, simulation runs, and online evals.
• Business costs $49 per seat per month with 500,000 logs, 30-day retention, and RBAC.
• Log overages cost $1 per 10,000, and Enterprise adds in-VPC deployment.

Prices come from Maxim AI pricing, reviewed October 12, 2026.

Pros and cons

Pros

• Simulation is a first-class workflow.
• Voice agents appear in the evaluation feature set.
• Clear verdicts with judge reasoning per evaluator.

Cons

• Per-seat pricing adds up for large review teams.
• Log retention is short below the Business plan.
• No self-hosting below Enterprise.

8. promptfoo: best for config-driven agent evals in CI

Verdict: promptfoo is the fastest way to put agent evals into CI from a single config file, and it is the migration path OpenAI recommends for its Evals platform.

promptfoo is an MIT-licensed CLI and library for evaluating and red teaming LLM applications and agents. The repository had 25,902 GitHub stars on October 12, 2026, the most of any framework here.

OpenAI agreed to acquire promptfoo in March 2026, and promptfoo says it will remain open source and keep supporting other model providers, per its announcement.

Best for: developers who want evals and security tests in CI without adopting a platform.

Key eval features

• Deterministic and model-graded assertions defined in YAML, run across prompts and models.
• Trajectory assertions for traced agents: tool-used, tool-args-match, tool-sequence, step-count, and goal-success.
• A local web viewer for side-by-side results.
• Red teaming for prompt injection, data leaks, and insecure tool use, with 10,000 free probes a month.

Agent evals, online scoring, and CI

promptfoo can assert that a traced agent called specific tools, in a specific order, with the expected arguments, per its assertion reference. The goal-success assertion adds an LLM judge for whether the run achieved its goal.

For CI, the promptfoo GitHub Action runs a before and after comparison on each pull request that edits a prompt and posts the pass and fail counts as a comment. Production monitoring requires the Enterprise plan.

Pricing

• Community is free and open source, with all evaluation features and red teaming up to 10,000 probes a month.
• Enterprise is custom priced and adds team sharing, continuous monitoring, and a managed cloud.
• On-Premise is custom priced with full data isolation.

Prices come from promptfoo pricing, reviewed October 12, 2026.

Pros and cons

Pros

• Free, local, and fast to adopt from one config file.
• Agent trajectory assertions with no platform required.
• Exports from OpenAI Evals land here.

Cons

• No production online scoring without Enterprise.
• Its roadmap will follow OpenAI once the deal closes.
• YAML configs get long for complex agents.

9. Inspect: best for sandboxed, benchmark-style agent evals

Verdict: Inspect is the strongest free framework when agents must run real tools in a sandbox and you want reproducible, benchmark-style scores.

Inspect is an open-source framework for large language model evaluations developed by the UK AI Security Institute and Meridian Labs. It is MIT licensed, and the repository had 2,984 GitHub stars on October 12, 2026.

It ships more than 200 pre-built evaluations through inspect_evals, plus the Inspect View log viewer and a VS Code extension.

Best for: safety, research, and platform teams that need sandboxed, repeatable agent tasks and benchmark scores.

Key eval features

• Composable datasets, solvers, agents, tools, and scorers.
• Built-in ReAct and Deep Agent solvers, multi-agent primitives, and an Agent Bridge for external frameworks, per the agents docs.
• Runs external agents such as Claude Code, Codex CLI, and Gemini CLI.
• Run limits, parallelism, early stopping, and checkpointing for long eval sets.

Agent evals, online scoring, and CI

Inspect runs untrusted model code in Docker, Kubernetes, Modal, Proxmox, or Vagrant sandboxes, per its sandboxing docs. That matters after Anthropic pulled internal evals off the live internet when agents took real actions, as we covered in our report on Anthropic offline evals.

Inspect is offline only, so pair it with a platform for production scoring. In CI, run eval sets from the CLI in your own pipeline, and read results in Inspect View.

Pricing

• Inspect is free and open source under MIT.
• There is no hosted plan; you pay only for model calls and sandbox compute.

Pricing reflects the open-source license on GitHub, reviewed October 12, 2026.

Pros and cons

Pros

• Real sandboxes for agents that run code and browse.
• 200+ ready-made evals and benchmarks.
• Broad model provider support.

Cons

• No online evaluation or hosted dashboard.
• Built for researchers, so product teams face a learning curve.
• Smaller community than DeepEval or promptfoo.

10. MLflow: best for teams already on MLflow or Databricks

Verdict: MLflow is the natural AI agent evaluation tool for data teams that already track models in MLflow, since agent evals land next to their experiments.

MLflow is an Apache 2.0 open-source AI engineering platform for agents, LLMs, and ML models, and Databricks offers a managed version. The MLflow repository had 28,346 GitHub stars on October 12, 2026.

Its evaluation and monitoring docs cover datasets, human feedback, LLM judges, regression testing in CI/CD, and production monitoring.

Best for: ML and data teams on Databricks, and teams that want one open-source system for models and agents.

Key eval features

• Built-in judges for correctness, relevance, completeness, guidelines, safety, and RAG groundedness.
• ToolCallCorrectness and ToolCallEfficiency judges for agent tool calls.
• A Judge Builder UI for custom LLM judges, from MLflow 3.9.0.
• Third-party judges from DeepEval, RAGAS, Phoenix, TruLens, and Guardrails AI.

Agent evals, online scoring, and CI

MLflow can run DeepEval, RAGAS, and Phoenix judges inside its own evaluation runs, so teams can mix libraries without a second platform. Its tool call judges require traces and are marked experimental, per the built-in judges docs.

For CI, call mlflow.genai.evaluate() from a test job and compare scores against a baseline run.

Pricing

• Open-source MLflow is free, and your eval data stays on your own infrastructure.
• Managed MLflow runs on Databricks, which bills by platform consumption.
• There is no separate per-trace or per-seat MLflow license fee.

We did not model Databricks consumption costs, which vary by workspace and cloud contract.

Pros and cons

Pros

• Free, Apache 2.0, and backed by a very large community.
• Agent evals and classic ML experiments share one tool.
• Plugs in other libraries’ judges.

Cons

• Tool call judges are still experimental.
• Self-hosting means running the tracking server, database, and artifact store.
• Managed MLflow ties you to Databricks pricing.

Eval Coverage Compared: Final Response, Trajectory, Tool Calls, and CI

All 10 tools score final responses, but they differ sharply on trajectories, tool calls, simulation, and CI. LangSmith, DeepEval, Opik, Maxim, and promptfoo ship named trajectory checks, Phoenix and MLflow ship tool-call judges, and Braintrust and Langfuse rely on custom scorers. The matrix shows what is built in and what you write yourself.

Tool	Trajectory scoring	Tool-call scoring	Multi-turn or simulation	Online production scoring	CI gate
Braintrust	Custom scorers on trace spans	Custom or autoevals scorers	Remote evals and sandboxes	Built in	GitHub Action, PR comment
LangSmith	Built in (agentevals match and judge)	Built in (tool args match)	Multi-turn thread evaluators	Built in	pytest, Vitest, Jest
Langfuse	LLM judge on root observation or experiment	Observation-level judges	Session tracking	Built in	GitHub Action, RegressionError
Arize Phoenix	Custom evaluators; ATIF trajectory import	Built in (3 tool evaluators)	Custom evaluators	Built in	SDK in your pipeline
DeepEval	Built in (4 trajectory metrics)	Built in (Tool and Argument Correctness)	Conversation simulator	Confident AI Starter and up	deepeval test run
Opik	Built in (Trajectory Accuracy)	Built in (Agent Tool Correctness)	Conversation metrics	Built in	SDK in your pipeline
Maxim AI	Built in (agent trajectory)	Step completion evaluator	Built in simulation	Professional and up	CI/CD pipelines
promptfoo	Built in (tool-sequence, step-count)	Built in (tool-used, tool-args-match)	Multi-turn test cases	Enterprise only	GitHub Action, before vs after
Inspect	Custom scorers on message history	Custom scorers	Multi-agent tasks in sandboxes	None	CLI in your pipeline
MLflow	ToolCallEfficiency judge	ToolCallCorrectness judge	Multi-turn judges	Built in	Regression testing docs

Built-in trajectory metrics save the most time on agents with fixed procedures, such as refunds, claims, or account changes. Open-ended research agents need LLM judges either way, so judge tooling and calibration features matter more there.

What AI Agent Evaluation Tools Cost for the Same Workload

On one workload of 120,000 traces and 50,000 scores a month for 5 users, list-price platform bills ranged from about $74 on Opik Pro Cloud to about $770 on LangSmith Plus. The open-source frameworks cost nothing to license, and judge model tokens often cost more than any platform fee.

Our scenario is a team of 5 running 100,000 production agent traces a month at 10 spans each, scoring a 10% sample with 3 LLM judges (30,000 scores). A 500-case regression suite runs 40 times a month in CI (20,000 runs and 20,000 scores). We assume about 10 KB per trace, which is our estimate and not a vendor figure.

Tool and plan	Platform or seats	Usage	Scores	Estimated monthly total
LangSmith Plus	$195 (5 x $39)	110,000 extra base traces = $550	10,000 extended upgrades = $25	About $770
Braintrust Pro	$249	1.2 GB, inside 5 GB = $0	50,000, inside 50,000 = $0	About $249
Confident AI Starter	$200	1.2 GB-months, inside 5 = $0	Not metered	About $200
Maxim AI Professional	$145 (5 x $29)	100,000 production logs, inside allowance = $0	Not metered	About $145
Langfuse Core	$29	1,320,000 traces and observations	50,000 scores, counted as units	About $131
Opik Pro Cloud	$19	1,100,000 extra spans = $55	Not metered	About $74
Arize Phoenix or AX	$0 self-hosted; AX Pro $50	1.2 million spans exceeds the 50,000 AX Pro cap	Unlimited evals	Self-host infrastructure, or an AX Enterprise quote
DeepEval, promptfoo, Inspect	$0	No license metering	No license metering	$0 license, offline only
MLflow	$0 self-hosted	No license metering	No license metering	Self-host infrastructure, or Databricks consumption

Judge tokens are often the bigger bill. If each of the 50,000 judge calls reads 3,000 tokens and writes 300, that is 150 million input and 15 million output tokens. At the AgentCore built-in evaluator rates of $0.0024 and $0.012 per 1,000 tokens, that comes to about $540 a month, more than every platform fee above except LangSmith’s.

Three caveats shape the table. LangSmith counts an evaluator execution as a trace, so judge runs can add to its total, which makes $770 a floor. Braintrust Starter could run the same workload for about $101 with 14-day retention, and Maxim’s Professional plan keeps logs for only 7 days.

How to Run Agent Evals in CI: A Working Example

The simplest CI gate for an agent is a deterministic trajectory check. Run the agent on a known task, compare its tool calls to a golden list, and fail the build if a required call is missing. It needs no LLM judge, so it is fast, free, and gives the same result on every run.

The test below uses LangChain’s open-source agentevals package with pytest, and it works with or without a LangSmith account. We ran it on October 12, 2026: the correct run passed, and a run that skipped the policy check failed.

# test_refund_agent.py  (run with: pytest)
import json
from agentevals.trajectory.match import create_trajectory_match_evaluator
from my_agent import run_agent  # returns OpenAI-style messages

def call(name, **args):
    return {"function": {"name": name, "arguments": json.dumps(args)}}

# Golden trajectory: the tools a correct refund must call
REFERENCE = [
    {"role": "user", "content": "Refund order 1042"},
    {"role": "assistant", "content": "", "tool_calls": [call("lookup_order", order_id="1042")]},
    {"role": "assistant", "content": "", "tool_calls": [call("check_refund_policy", days_since_delivery=3)]},
    {"role": "assistant", "content": "", "tool_calls": [call("issue_refund", order_id="1042")]},
]

# "superset": extra tool calls pass, a missing required call fails
evaluator = create_trajectory_match_evaluator(trajectory_match_mode="superset")

def test_refund_agent_calls_required_tools():
    result = evaluator(outputs=run_agent("Refund order 1042"), reference_outputs=REFERENCE)
    assert result["score"], result

Superset mode ignores the order of calls, so use strict mode with full reference messages when order matters. Layer an LLM judge on top for open-ended quality, and run it on a sample rather than every commit to control token cost.

AI Agent Evaluation Tools We Left Off, and Why

We left out nine tools that still appear in many AI agent evaluation rankings, because of shutdowns, ownership changes, product pivots, or a narrower scope. Two of them are cloud-native services that make sense for teams already committed to that cloud, and we note where each one fits.

• OpenAI Evals platform: it goes read-only on October 31, 2026 and shuts down on November 30, 2026, per OpenAI’s deprecations page.
• Galileo: Cisco closed its acquisition on May 22, 2026, and standalone pricing now redirects to Splunk, per Cisco.
• Patronus AI: it now sells training data and simulated environments to frontier labs, per its pricing page.
• Humanloop: the team joined Anthropic and the platform is being sunset, per its homepage.
• W&B Weave: CoreWeave Agent Lens is now the named successor, per the Weave docs.
• Ragas: a strong RAG metrics library with Tool Call Accuracy and Agent Goal Accuracy metrics, per its docs, but its repository last received a commit on February 24, 2026, per the GitHub API.
• Amazon Bedrock AgentCore Evaluations: 13 built-in evaluators on OpenTelemetry traces from Strands, LangGraph, and other frameworks, per AWS docs. Strong for AWS-native agents, less portable elsewhere.
• Google Gen AI evaluation service: trajectory exact-match, in-order, any-order, precision, and recall metrics, per its agent evaluation docs. Best for teams building on Google Cloud.
• Datadog, Laminar, and other tracing-first tools: these lead on tracing more than evals, and our agent observability tools guide covers them.

Which AI Agent Evaluation Tool Should You Choose?

Choose by your hardest constraint first. Release gating points to Braintrust, LangGraph points to LangSmith, data residency points to Langfuse or Phoenix, and evals as tests point to DeepEval or promptfoo. Most mature teams pair one framework in CI with one platform for production scoring.

• Braintrust: best overall when eval scores decide releases and many reviewers need access.
• LangSmith: best for LangChain and LangGraph agents and deterministic trajectory matching.
• Langfuse: best open-source platform with a packaged CI gate.
• Arize Phoenix: best ready-made tool-call evaluators on your own infrastructure.
• DeepEval and Confident AI: best for agent evals written as pytest tests.
• Opik: best Apache 2.0 platform and the cheapest Cloud plan.
• Maxim AI: best for simulating multi-turn chat and voice agents.
• promptfoo: best for config-driven evals and security tests in CI.
• Inspect: best for sandboxed, benchmark-style agent tasks.
• MLflow: best for teams already on MLflow or Databricks.

Frequently Asked Questions About AI Agent Evaluation Tools

The Bottom Line: Score the Path as Well as the Answer

The best AI agent evaluation tool scores trajectories and tool calls in CI and on live traffic with the same definitions. Braintrust does that most completely as a hosted platform, Langfuse and Phoenix do it on your own servers, and DeepEval, promptfoo, and Inspect make it free in code. Start with deterministic tool-call checks, then add calibrated judges.

We recheck these prices quarterly, and vendors can send corrections through our contact page. For the tracing side of the same stack, read our guide to the best AI agent observability tools.

Related guides: the AgentOps tools map, Langfuse vs LangSmith vs Braintrust, Braintrust vs Maxim vs Galileo, LangSmith alternatives, AI agent frameworks, AgentOps: observability, evals, and monitoring, agent evals and testing, and agent observability.