Evaluations is a Pro feature. If you’re on the free tier, you’ll see an upgrade prompt when you try to enable it.
Where to Find Evaluations
Open your agent and click Evaluations in the left-hand sidebar.
The Building Blocks
Before jumping into setup, here’s the mental model. Evaluations have four building blocks:Criteria
Quality rules the evaluator checks pass/fail on every conversation. Like a QA checklist.
Tags
Labels applied for categorization and filtering. Like folders in your inbox.
Data Points
Structured values extracted from conversations. Like columns in a spreadsheet.
Sentiment
Emotional tone of the interaction. A customer satisfaction thermometer.
How Evaluations Work
Interaction Completes
Your agent finishes a conversation (reaches the “completed” state). Incognito chats and internal system interactions are never evaluated.
Transcript is Built
The evaluator constructs a role-tagged transcript of the entire conversation, including user messages, agent responses, tool calls, and results. Long transcripts are automatically trimmed.
AI Evaluator Runs
A single structured-output LLM call analyzes the transcript against your configured criteria, tags, data points, and sentiment settings.
Grade is Computed
The overall grade is determined deterministically based on criterion failures, priority levels, call outcome, sentiment, and action failures.
When does the evaluation fire?
When does the evaluation fire?
The evaluation fires automatically shortly after the interaction reaches the “completed” state.
- For a simple back-and-forth that ends naturally, the evaluation runs once the chat is marked complete.
- The evaluation covers the entire transcript up to that point.
- There is a short debounce period after the last message so the system doesn’t prematurely evaluate a chat that’s still active.
What happens if I keep chatting after the evaluation ran?
What happens if I keep chatting after the evaluation ran?
If you continue a conversation after an evaluation has already run:
- The new messages extend the transcript, and the interaction re-enters an active state.
- Once the conversation reaches the “completed” state again, a new evaluation automatically runs covering the full updated transcript.
- The new evaluation replaces the previous result. You don’t need to manually re-run it.
- You always see the most recent evaluation result for any given interaction.
Do I have to manually re-run it, or is it automatic?
Do I have to manually re-run it, or is it automatic?
Fully automatic. As long as evaluations are enabled, the system handles re-evaluation whenever conversations continue and complete again. You only need to run manually if you want to backfill old conversations or re-evaluate after changing your criteria.
Setting Up Evaluations
Navigate to your agent’s Evaluations tab in the sidebar. You’ll see the settings panel:
| Setting | What It Does |
|---|---|
| Enable evaluations | Toggle to start automatically grading interactions |
| Default analysis model | Which LLM runs the evaluation. “Smartest” = most accurate but costs more per token |
| Sentiment analysis | Captures overall sentiment (Positive / Neutral / Negative). Optionally affects grade. |
| Suggest tags automatically | Lets the evaluator propose new tags beyond your predefined vocabulary |
Configuring Criteria, Tags & Data Points
- Criteria
- Data Points
Criteria are the quality rules your evaluator checks every interaction against. Each one is a clear statement that’s either true or false for a given conversation.How to think about it: Ask yourself, “If I were reviewing this conversation manually, what would I check for?” Each answer becomes a criterion.
Choosing priority: Use “Critical” for rules that must never be broken (data leaks, compliance violations). Use “Warning” for quality standards that matter but aren’t urgent (tone issues, minor drifts).
Limit: 30 criteria per agent.
Adding a Criterion
Click + Add criterion and fill in:| Field | Description |
|---|---|
| Name | A short label (e.g., “Accuracy”, “Stayed on Topic”) |
| Evaluation prompt | A true/false statement describing the desired behavior. Be specific. |
| Type | Categorizes the criterion: Prohibited action, Prohibited words, Voice & tone, or Other |
| Priority | Warning (downgrades to Warning) or Critical (downgrades to Critical + Slack alert) |
Types & Priority
| Type | When to Use |
|---|---|
| Prohibited action | Agent must NOT do something (e.g., don’t offer unauthorized discounts) |
| Prohibited words | Agent must NOT say certain things (e.g., no profanity) |
| Voice & tone | Agent should communicate in a certain style |
| Other | Anything else (stayed on topic, provided accurate info, etc.) |
Example criteria for common use cases
Example criteria for common use cases
| Use Case | Criterion | Prompt | Type | Priority |
|---|---|---|---|---|
| Support | Stayed on Topic | ”The agent stayed focused on resolving the customer’s issue and did not go off on tangents.” | Other | Warning |
| Support | Accuracy | ”The agent provided factually correct information and did not hallucinate.” | Other | Critical |
| Sales | No Unauthorized Discounts | ”The agent did not offer discounts not in the approved pricing sheet.” | Prohibited action | Critical |
| Sales | Professional Tone | ”The agent maintained a professional, friendly tone throughout.” | Voice & tone | Warning |
| Helpdesk | No PII Disclosure | ”The agent did not reveal personal information of other employees.” | Prohibited action | Critical |
| Content | Brand Voice | ”The content matches the brand’s voice: confident, concise, and jargon-free.” | Voice & tone | Warning |
Understanding Results
Once an interaction is evaluated, you can see the full results in the interaction detail view’s Overview tab.
Grades
| Grade | Meaning | What Triggers It |
|---|---|---|
| Pass | Met all criteria | No failures, call wasn’t a failure, sentiment not negative |
| Warning | Needs review | Warning-priority criterion failed, OR call outcome “failure”, OR negative sentiment affects grade |
| Critical | Immediate attention | Critical-priority criterion failed, OR tool/action failures during interaction |
Grade computation logic (deterministic)
Grade computation logic (deterministic)
The grade is computed deterministically (not by the LLM) after results come back:
- If there were action failures (tool errors) → Critical
- If any Critical-priority criterion failed → Critical
- If any Warning-priority criterion failed → Warning
- If overall call outcome was “failure” → Warning
- If sentiment is negative AND affects grade → Warning
- Otherwise → Pass
What’s Shown in Results
- Summary: One or two sentence narrative of what happened
- Grade: Pass / Warning / Critical badge
- Outcome: Successful / Failed / Unknown
- Sentiment: Positive / Neutral / Negative (if enabled)
- Tags: Applied tags from your vocabulary + auto-generated
- Criteria: Per-criterion pass/fail with the evaluator’s rationale
- Collected data: Extracted values for each data point
Interactions List View
The interactions list includes an Evaluation column showing the grade and criteria pass rate at a glance (e.g., “Pass 3/3” or “Critical 1/3”). Lifecycle states also appear: Queued, Evaluating, or Failed.Running Evaluations Manually
Evaluations run automatically, but you can also trigger them manually from the Chats page.Find the Interaction
Locate the chat you want to evaluate. The Evaluation column shows the current state.
- Backfilling existing conversations after enabling evaluations
- Re-evaluating after changing your criteria/tags/data points
- Retrying a failed evaluation
- Spot-checking specific conversations on demand
Manual evaluations use the current configuration. If you’ve changed your setup, a re-run reflects the new rules.
Alerts
When an interaction receives a Critical grade, the agent owner gets an immediate Slack DM with the agent’s name, which criteria failed, and a direct link to the interaction. No additional setup required, as long as Slack is connected.Credits and Costs
Evaluations are billed as AI credits under the “AI Utilities” category. Each evaluation is a single LLM call.What determines the cost?
What determines the cost?
Three factors:
- Transcript length: Longer conversations use more input tokens. A 5-message chat costs significantly less than a 50-message conversation.
- Analysis model: The model you select determines the per-token rate. “Smartest” costs more than faster alternatives.
- Schema complexity: More criteria, tags, and data points = more output tokens to generate.
Where to See Credit Usage
View all evaluation credit usage on your Usage & Limits page. Filter by “AI Utilities” to see individual evaluation runs and their credit amounts.
Limits
| Resource | Maximum |
|---|---|
| Criteria per agent | 30 |
| Tags per agent | 50 |
| Data points per agent | 40 |
| Tag name length | 100 characters |
| Tag description length | 500 characters |
FAQ
Which interactions get evaluated?
Which interactions get evaluated?
All completed interactions are evaluated, except:
- Incognito chats: Never evaluated (privacy guarantee)
- Internal interactions: Agent-to-agent feedback loops like reflections are excluded
- Non-terminal states: Only interactions that reach “completed” are evaluated
Can I evaluate interactions from before I enabled evaluations?
Can I evaluate interactions from before I enabled evaluations?
Yes. Go to your agent’s Chats page, click the three-dot menu on any interaction, and select “Run evaluation.” You can also select multiple for bulk evaluation. They’ll be graded against your current criteria.
What happens if I keep chatting after an evaluation already ran?
What happens if I keep chatting after an evaluation already ran?
The interaction re-enters an active state. Once it completes again, a new evaluation runs automatically covering the full updated transcript. The new result replaces the previous one.
Does the evaluation fire only once per chat?
Does the evaluation fire only once per chat?
By default, yes. The evaluation fires once after the conversation completes (with a short debounce window to avoid premature evaluation). If the conversation is resumed and completes again, a new evaluation runs automatically covering the updated transcript.
What happens if the evaluation fails?
What happens if the evaluation fails?
It’s marked as “Failed” rather than showing a misleading pass. Doesn’t affect metrics. You can manually re-run it from the Chats page, or it will be re-evaluated if the conversation continues.
How is the analysis model different from my agent's model?
How is the analysis model different from my agent's model?
It’s a separate LLM call dedicated to evaluation. Doesn’t affect your agent’s behavior. You can choose a different model. The evaluator never sees your agent’s system prompt directly, only the transcript and agent description/skills as context.
Does the evaluator see my agent's system prompt?
Does the evaluator see my agent's system prompt?
It receives your agent’s name, description, and available skills as context. It does NOT receive the full system prompt. Transcript content is treated as data to analyze, never as instructions.
What counts as an 'action failure'?
What counts as an 'action failure'?
Tool/integration errors during the interaction (e.g., an API call that returned an error, a tool that timed out). These automatically cause a “Critical” grade.
Can I set up evaluations at the organization level?
Can I set up evaluations at the organization level?
Currently configured per agent. Organization-level configs are planned for a future release.
What happens if I delete a criterion or tag?
What happens if I delete a criterion or tag?
Past evaluation results retain their original data. Deleting only affects future evaluations.
Can I edit tags on an evaluation result?
Can I edit tags on an evaluation result?
Why does my evaluation show 'Inconclusive'?
Why does my evaluation show 'Inconclusive'?
All criteria returned “unknown” (evaluator couldn’t determine pass/fail). Typically means the conversation was too short or didn’t touch on what your criteria check.
How much do evaluations cost in credits?
How much do evaluations cost in credits?
Each evaluation is billed based on tokens used. Cost depends on conversation length, model selected, and number of criteria/tags/data points configured. Track usage on your Usage & Limits page under “AI Utilities.”