Evaluations

Evaluations give you automated quality assurance for your agents. After every interaction completes, an AI evaluator analyzes the full conversation transcript and produces a structured report: a grade, per-criterion pass/fail results with rationales, sentiment analysis, applied tags, and extracted data points. Think of it as having a QA analyst reviewing every single conversation your agent has, 24/7, without you lifting a finger.

Where to Find Evaluations

Open your agent, click Performance in the sidebar, then open the Evaluations tab. Performance is the home for everything about how the agent is doing — Insights, Evaluations, and Tasks. Agent Performance is the tour of that screen; this page is the deep dive on evaluations themselves.

Performance page on the Evaluations tab showing grading, criteria, impact report, and reflections

The Building Blocks

Before jumping into setup, here’s the mental model. Evaluations have four building blocks:

Criteria

Quality rules the evaluator checks pass/fail on every conversation. Like a QA checklist.

Data Points

Structured values extracted from conversations. Like columns in a spreadsheet.

Sentiment

Emotional tone of the interaction. A customer satisfaction thermometer.

Criteria answer: “Did the agent do what it was supposed to do?” Each one is a yes/no check. Tags answer: “What kind of conversation was this?” Use them to filter and find patterns. Data Points answer: “What specific facts or values came up?” They pull structured data out of unstructured conversation. Sentiment answers: “How did the user feel?” Sentiment is optional reporting data and does not affect the grade.

How Evaluations Work

Interaction Completes

Your agent finishes a conversation (reaches the “completed” state). Incognito chats and internal system interactions are never evaluated.

Transcript is Built

The evaluator constructs a role-tagged transcript of the entire conversation, including user messages, agent responses, tool calls, and results. Long transcripts are automatically trimmed.

AI Evaluator Runs

A single structured-output LLM call analyzes the transcript against your configured criteria, tags, data points, and sentiment settings.

Grade is Computed

The overall grade is determined deterministically based on which criteria failed, the action set on each failed criterion, and call outcome. Sentiment does not affect the grade.

Results Stored & Alerts Fired

Results are persisted. If the grade is Notify, the notification channels you enabled fire immediately. Configure Delivery and To on the Performance evaluation card, including email recipients and selected Slack channels. Recipients must have permission to manage the agent.

When does the evaluation fire?

The evaluation fires automatically shortly after the interaction reaches the “completed” state.

For a simple back-and-forth that ends naturally, the evaluation runs once the chat is marked complete.
The evaluation covers the entire transcript up to that point.
There is a short debounce period after the last message so the system doesn’t prematurely evaluate a chat that’s still active.

What happens if I keep chatting after the evaluation ran?

If you continue a conversation after an evaluation has already run:

The new messages extend the transcript, and the interaction re-enters an active state.
Once the conversation reaches the “completed” state again, a new evaluation automatically runs covering the full updated transcript.
The new evaluation replaces the previous result. You don’t need to manually re-run it.
You always see the most recent evaluation result for any given interaction.

Do I have to manually re-run it, or is it automatic?

Fully automatic. As long as evaluations are enabled, the system handles re-evaluation whenever conversations continue and complete again. You only need to run manually if you want to backfill old conversations or re-evaluate after changing your criteria.

Setting Up Evaluations

Go to Performance → Evaluations. You’ll see the settings panel:

Evaluation settings panel showing the enable toggle, model selector, sentiment analysis, and auto-tags options

Setting	What It Does
Enable evaluations	Toggle to start automatically grading interactions
Default analysis model	Which LLM runs the evaluation. “Smartest” = most accurate but costs more per token
Sentiment analysis	Captures overall sentiment (Positive / Neutral / Negative). Reporting only; it does not affect the grade.
Suggest tags automatically	Lets the evaluator propose new tags beyond your predefined vocabulary

For sentiment, you can provide custom guidance (for example, “Consider the customer’s final message tone, not their initial frustration”). Sentiment remains reporting data and does not change the grade.

Configuring Criteria, Tags & Data Points

Criteria
Tags
Data Points

Criteria are the quality rules your evaluator checks every interaction against. Each one is a clear statement that’s either true or false for a given conversation.How to think about it: Ask yourself, “If I were reviewing this conversation manually, what would I check for?” Each answer becomes a criterion.

Adding a Criterion

Click + Add criterion and fill in:

Add criterion form showing Name, Evaluation prompt, Type, and the Action dropdown with Flag and Notify options

Field	Description
Name	A short label (e.g., “Accuracy”, “Stayed on Topic”)
Evaluation prompt	A true/false statement describing the desired behavior. Be specific.
Type	Categorizes the criterion: Prohibited action, Prohibited words, Voice & tone, or Other
Action	What happens when the criterion fails: Flag (flags the chat for review without notifying you) or Notify (notifies you the moment the evaluator spots one)

Types & Actions

Type	When to Use
Prohibited action	Agent must NOT do something (e.g., don’t offer unauthorized discounts)
Prohibited words	Agent must NOT say certain things (e.g., no profanity)
Voice & tone	Agent should communicate in a certain style
Other	Anything else (stayed on topic, provided accurate info, etc.)

Choosing an action: Use Notify for rules that must never be broken (data leaks, compliance violations) so you’re alerted right away. Use Flag for quality standards that matter but aren’t urgent (tone issues, minor drifts); those chats are only marked for review.

”Notify” alerts the people responsible for the agent, not the person chatting with it. Configure Delivery and To for email recipients and Slack destinations. Recipients must have permission to manage the agent. End users never see evaluation results.

Example criteria for common use cases

Use Case	Criterion	Prompt	Type	Action
Support	Stayed on Topic	”The agent stayed focused on resolving the customer’s issue and did not go off on tangents.”	Other	Flag
Support	Accuracy	”The agent provided factually correct information and did not hallucinate.”	Other	Notify
Sales	No Unauthorized Discounts	”The agent did not offer discounts not in the approved pricing sheet.”	Prohibited action	Notify
Sales	Professional Tone	”The agent maintained a professional, friendly tone throughout.”	Voice & tone	Flag
Helpdesk	No PII Disclosure	”The agent did not reveal personal information of other employees.”	Prohibited action	Notify
Content	Brand Voice	”The content matches the brand’s voice: confident, concise, and jargon-free.”	Voice & tone	Flag

Write evaluation prompts as true/false statements. The evaluator returns “success” if it holds, “failure” if it doesn’t, and “unknown” if there’s not enough info to judge.

Pausing a Criterion

Every criterion is either Active or Paused. Use the criterion’s action menu to switch:

Action	What happens
Pause	Skips this criterion when grading chats. It’s left out of the evaluation entirely, so new interactions are never graded against it.
Resume	Grades chats against this criterion again.

Pausing keeps the criterion’s name, prompt, type, and action intact, so you can bring it back later without rewriting it. It’s the middle ground between deleting a rule you might need again and leaving a noisy one switched on.

Pausing only affects future interactions. Grades already recorded against the criterion stay as they are.

Limit: 30 criteria per agent. Paused criteria still count toward that limit, since they’re kept on the agent rather than removed.

Tags are labels for categorization. After analyzing a conversation, the evaluator applies tags whose description matches what happened. You can then filter your interactions list by tag.How to think about it: Tags are like labels in Gmail. They don’t pass/fail anything. They just categorize. Ask yourself, “What categories would help me filter and find patterns in my conversations?”

Tags section showing AI_SLOP and OFF_COURSE tags

Adding a Tag

Click + Add tag and provide:

Field	Description
Name	Automatically uppercased to `UPPER_SNAKE_CASE`. Max 100 characters.
Description	Tells the evaluator when to apply this tag. Be specific. Max 500 characters.

How Tags Work

You define a vocabulary of tags with descriptions
The evaluator applies matching tags after analyzing the conversation
Tag names are normalized (e.g., “off course” → OFF_COURSE)
You can manually add/remove tags on any evaluated interaction
Use them to filter the interactions list

Example tags for common use cases

Tag	Description	Use Case
`ESCALATION_NEEDED`	”Customer asked to speak with a human or the issue is too complex for the agent.”	Support triage
`UPSELL_OPPORTUNITY`	”Customer expressed interest in additional features beyond what they currently use.”	Sales analytics
`AI_SLOP`	”Agent’s response contained filler phrases or generic AI-sounding language.”	Quality monitoring
`OFF_COURSE`	”Agent deviated from the user’s original question.”	Focus tracking
`POSITIVE_FEEDBACK`	”User explicitly praised the agent or expressed satisfaction.”	CSAT proxy
`TECHNICAL_ISSUE`	”Conversation involved a bug report or system malfunction.”	Issue categorization

Enable “Suggest tags automatically” in Settings and the evaluator will also generate new tags for patterns it notices beyond your predefined vocabulary.

Limit: 50 tags per agent.

Data collection lets you extract structured values from every interaction. Define what to extract, and the evaluator pulls it out automatically.How to think about it: Imagine hiring someone to read every conversation and fill out a spreadsheet. Each column is a data point. You define the column name, value type, and extraction instructions.

Data collection section showing a Confidence Score data point

Adding a Data Point

Click + Add data point and configure:

Field	Description
Name	What you’re extracting (e.g., “Confidence Score”)
Type	Text, Boolean, Integer, or Number
Description	Extraction instructions. Be precise about what to look for and valid values.

Data Point Types

Type	Returns	Best For
Text	String	Categories, summaries, reasons
Boolean	Yes / No	Binary checks (was something done?)
Integer	Whole number	Counts, quantities
Number	Decimal	Scores, ratings, percentages

Example data points for common use cases

Name	Type	Description	Use Case
Confidence Score	Number	”Rate 1-10 how confident the agent appeared based on hedging language vs. direct statements.”	Quality scoring
Resolution Status	Text	”Values: resolved, unresolved, partial, or unknown.”	Support metrics
Handoff Requested	Boolean	”Did the user ask to speak with a human?”	Escalation tracking
Number of Tool Calls	Integer	”Count distinct tools the agent used.”	Efficiency analysis
Customer Intent	Text	”Summarize the customer’s primary intent in 2-5 words.”	Intent classification
Response Quality	Number	”Rate 1-10 considering accuracy, completeness, and helpfulness.”	Performance benchmarking

Data points that return null mean the evaluator couldn’t find the information in the transcript. This is expected for data points that don’t apply to every conversation.

Limit: 40 data points per agent.

Understanding Results

Once an interaction is evaluated, you can see the full results in the interaction detail view’s Overview tab.

Evaluation result showing summary, grade, outcome, sentiment, criteria results, tags, and collected data

Grades

Grade	API Value	Meaning	What Triggers It
Pass	`pass`	Met all criteria	No failures, and the call wasn’t a failure
Flag	`needs_review`	Flagged for review	A criterion set to Flag failed, OR call outcome “failure”
Notify	`needs_attention`	Immediate attention	A criterion set to Notify failed (fires your enabled notification channels)

When using the Evaluations API, use the API values (pass, needs_review, needs_attention) for the grade query parameter, not the UI labels.

Grade computation logic (deterministic)

The grade is computed deterministically (not by the LLM) after results come back:

If any criterion set to Notify failed → Notify
If any other criterion failed → Flag
If overall call outcome was “failure” → Flag
Otherwise → Pass

What’s Shown in Results

Summary: One or two sentence narrative of what happened
Grade: Pass / Flag / Notify badge
Outcome: Successful / Failed / Unknown
Sentiment: Positive / Neutral / Negative (if enabled)
Tags: Applied tags from your vocabulary + auto-generated
Criteria: Per-criterion pass/fail with the evaluator’s rationale
Collected data: Extracted values for each data point

Interactions List View

The interactions list includes an Evaluation column showing the grade and criteria pass rate at a glance (e.g., “Pass 3/3” or “Notify 1/3”). Lifecycle states also appear: Queued, Evaluating, or Failed.

Running Evaluations Manually

Evaluations run automatically, but you can also trigger them manually from the task list.

Go to Tasks

Open Performance → Tasks on your agent.

Find the task

Locate the task you want to evaluate. The Evaluation column shows the current state.

Open the Actions Menu

Click the three-dot menu (⋮) on the right side of the interaction row.

Click Run Evaluation

Select “Run evaluation” from the dropdown. Results will appear shortly.

Interactions list showing the three-dot menu with Run evaluation option

You can also select multiple interactions for bulk evaluation. When to use manual runs:

Backfilling existing conversations after enabling evaluations
Re-evaluating after changing your criteria/tags/data points
Retrying a failed evaluation
Spot-checking specific conversations on demand

Manual evaluations use the current configuration. If you’ve changed your setup, a re-run reflects the new rules.

You can also queue evaluations from the API, the Python SDK (client.agents.run_evaluations), or the CLI (gumloop agents eval-run).

Letting the agent manage its own evaluations

Turn on the Manage Evaluations ability in the agent’s configuration and anyone who can edit the agent can ask it, in chat, to review its results, add or change criteria, or re-run evaluations on past chats. Reads happen immediately. Every change and every run shows an approval card listing exactly what will change before it happens. The ability is available on Pro and Enterprise plans, on custom agents.

Agent Abilities list showing the Manage Evaluations ability turned on

What you can ask for	What happens
”What’s failing?”	Reads the rubric, a failure-rate rollup per criterion, and individual results. Both default to the last 30 days and can look back up to a year. No approval.
”Add criteria for X” / “Rewrite this one” / “Drop these two”	Proposes the exact criteria and waits for your approval before saving. The 30-criteria limit still applies.
”Change the settings”	Patches only the settings you name — on/off, frequency, analysis model, language, sentiment, notifications, automatic tags — and leaves the rest alone. Approval required.
”Re-run these chats”	Queues up to 200 finished chats, using the current configuration. Costs credits per chat, and skips chats that are already being evaluated. Approval required.

The agent only sees the chats you can see. If your role cannot view other people’s interactions, it analyzes your own chats and says so. It also cannot evaluate the chat it is having with you — that chat is still running.

Notifications

Notifications fire only when a chat receives a Notify grade, i.e. a criterion whose action is Notify failed. Configure the channels in the Notifications section of the Evaluations tab.

Channel	Who receives it
Slack	The Slack destinations you select under Delivery and To, including selected channels. Recipients must have permission to manage the agent.
Email	The email recipients you list. Recipients must have permission to manage the agent. Up to 50 recipients.

Notifications section with Slack notifications and Email notifications toggles and a recipient input

”You” in this UI means the agent’s owner and the recipients you configure, never the end user chatting with the agent.

Flag-grade chats are marked in the interactions list but never send a notification. Review them periodically by filtering for the Flag grade.

Credits and Costs

Evaluations are billed as AI credits under the “AI Utilities” category. Each evaluation is a single LLM call.

What determines the cost?

Three factors:

Transcript length: Longer conversations use more input tokens. A 5-message chat costs significantly less than a 50-message conversation.
Analysis model: The model you select determines the per-token rate. “Smartest” costs more than faster alternatives.
Schema complexity: More criteria, tags, and data points = more output tokens to generate.

The evaluator checks that the user has sufficient credits before running. If credits are insufficient, the evaluation is skipped silently.

Where to See Credit Usage

View all evaluation credit usage on your Usage & Limits page. Filter by “AI Utilities” to see individual evaluation runs and their credit amounts.

Credit Usage Logs page filtered to AI Utilities showing Interaction Evaluation entries

Limits

Resource	Maximum
Criteria per agent	30
Tags per agent	50
Data points per agent	40
Tag name length	100 characters
Tag description length	500 characters
Email notification recipients	50

Exporting Evaluation Data

Every evaluation result, including grades, criteria outcomes, extracted data points, tags, and sentiment, is available through the Evaluations API. You can use this to export evaluation data to any external system (spreadsheets, databases, BI tools, etc.). On the Enterprise plan there is a second route that needs no code: Interaction Evaluations is a data type in Data Export and Data Drains, so you can download every agent’s evaluations across the organization as a CSV, or keep them flowing continuously into S3, Datadog, or your own endpoint. Those rows cover the headline result — grade, call outcome, sentiment, summary, evaluation model, and credit cost — one row per rubric per chat. Use the API when you need per-criterion detail, extracted data points, or applied tags.

API

Use the List evaluations endpoint to pull all results for an agent:

curl 'https://api.gumloop.com/api/v1/agents/AGENT_ID/evaluations?page_size=100' \
  -H 'Authorization: Bearer YOUR_API_KEY'

Each result includes:

grade (pass / needs_review = Flag / needs_attention = Notify)
criteria_results with per-criterion pass/fail and rationale
data_results with extracted values for each configured data point
applied_tags
sentiment
summary

Paginate through all results using the next_cursor field. Narrow the list with grade, status, session_id, created_after, and created_before (ISO 8601), for example ?grade=needs_attention&created_after=2026-01-01T00:00:00Z. For a single evaluation, use the Retrieve evaluation endpoint. For aggregate metrics over time, use the Get metrics endpoint.

Call the evaluations API on a schedule and push results to Google Sheets, a database, or a webhook for automated reporting.

Python

import csv
import requests

API_KEY = "YOUR_API_KEY"
AGENT_ID = "YOUR_AGENT_ID"

# Paginate through all evaluations
cursor = None
all_evaluations = []
while True:
    params = {"page_size": 100}
    if cursor:
        params["cursor"] = cursor
    response = requests.get(
        f"https://api.gumloop.com/api/v1/agents/{AGENT_ID}/evaluations",
        headers={"Authorization": f"Bearer {API_KEY}"},
        params=params,
    )
    response.raise_for_status()
    data = response.json()
    all_evaluations.extend(data["evaluations"])
    cursor = data.get("next_cursor")
    if not cursor:
        break

# Export data points to CSV
with open("eval_export.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["interaction_id", "grade", "sentiment", "summary"])
    for ev in all_evaluations:
        writer.writerow([ev["interaction_id"], ev["grade"], ev["sentiment"], ev["summary"]])

FAQ

Which interactions get evaluated?

All completed interactions are evaluated, except:

Incognito chats: Never evaluated (privacy guarantee)
Internal interactions: Agent-to-agent feedback loops like reflections are excluded
Non-terminal states: Only interactions that reach “completed” are evaluated

Can I evaluate interactions from before I enabled evaluations?

Yes. Go to Performance → Tasks, click the three-dot menu on any interaction, and select “Run evaluation.” You can also select multiple for bulk evaluation. They’ll be graded against your current criteria.

What happens if I keep chatting after an evaluation already ran?

The interaction re-enters an active state. Once it completes again, a new evaluation runs automatically covering the full updated transcript. The new result replaces the previous one.

Does the evaluation fire only once per chat?

By default, yes. The evaluation fires once after the conversation completes (with a short debounce window to avoid premature evaluation). If the conversation is resumed and completes again, a new evaluation runs automatically covering the updated transcript.

What happens if the evaluation fails?

It’s marked as “Failed” rather than showing a misleading pass. Doesn’t affect metrics. You can manually re-run it from Performance → Tasks, or it will be re-evaluated if the conversation continues.

How is the analysis model different from my agent's model?

It’s a separate LLM call dedicated to evaluation. Doesn’t affect your agent’s behavior. You can choose a different model. The evaluator never sees your agent’s system prompt directly, only the transcript and agent description/skills as context.

Does the evaluator see my agent's system prompt?

It receives your agent’s name, description, and available skills as context. It does NOT receive the full system prompt. Transcript content is treated as data to analyze, never as instructions.

Can I set up evaluations at the organization level?

Yes, on the Enterprise plan. Organization admins with the Manage evaluations permission create evaluations under Organization → Evaluations and apply them to the whole organization, specific teams, people, or individual agents. They are also available through the API, SDK and CLI.

What happens if I delete a criterion or tag?

Past evaluation results retain their original data. Deleting only affects future evaluations.

Can I edit tags on an evaluation result?

Yes. Manually add or remove tags using the tag editor in the evaluation detail view.

Why does my evaluation show 'Inconclusive'?

All criteria returned “unknown” (evaluator couldn’t determine pass/fail). Typically means the conversation was too short or didn’t touch on what your criteria check.

How much do evaluations cost in credits?

Each evaluation is billed based on tokens used. Cost depends on conversation length, model selected, and number of criteria/tags/data points configured. Track usage on your Usage & Limits page under “AI Utilities.”

Can I export evaluation data?

Yes, in two ways. The Evaluations API gives you full programmatic access to all evaluation results, including grades, criteria outcomes, extracted data points, tags, and sentiment. On Enterprise, organization admins can also export or continuously drain evaluations without writing code by picking the Interaction Evaluations type in Data Export. See Exporting Evaluation Data above.

Get Started

Core Concepts

Enterprise Features

Connectors