Evaluation tools

Use evaluation tools to configure grading, evaluate completed agent sessions, and review results and metrics.

Agent evaluations

Tool	Description	Required inputs
`get_agent_evaluation_config`	Fetch an agent’s evaluation configuration	`agent_id`
`update_agent_evaluation_config`	Partially update an agent’s evaluation configuration	`agent_id`
`list_agent_evaluations`	List an agent’s evaluation results, newest first	`agent_id`
`get_agent_evaluation`	Fetch one agent evaluation result	`agent_id`, `evaluation_id`
`get_agent_evaluation_metrics`	Get evaluation grade and tag counts over a trailing window	`agent_id`
`run_agent_evaluations`	Queue evaluations of finished sessions with the agent’s configuration	`agent_id`, `session_ids`

Tool	Description	Required inputs
`list_organization_evaluations`	List an organization’s evaluations and result rollups	`organization_id`
`create_organization_evaluation`	Create an organization evaluation	`organization_id`, `name`
`get_organization_evaluation`	Fetch one organization evaluation	`evaluation_id`
`update_organization_evaluation`	Partially update an organization evaluation	`evaluation_id`
`delete_organization_evaluation`	Delete an organization evaluation	`evaluation_id`
`set_organization_evaluation_targets`	Replace the targets an organization evaluation grades	`evaluation_id`, `targets`
`run_organization_evaluation`	Queue an organization evaluation against existing sessions	`evaluation_id`, `session_ids`
`list_organization_evaluation_results`	List an evaluation’s results, newest first	`evaluation_id`
`get_organization_evaluation_result`	Fetch one organization evaluation result	`evaluation_id`, `result_id`
`get_organization_evaluation_metrics`	Get evaluation grade counts over a trailing window	`evaluation_id`

Both run_* tools accept dry_run to report the cost and skipped sessions without queuing evaluations.

Tool	Description	Required inputs
`get_evaluation_options`	Get allowed values and limits for evaluation configuration fields	None

Show me the evaluation configuration for my Research Assistant agent.

Run a dry run of evaluations for my Research Assistant agent's latest sessions and estimate the cost.

List our organization evaluations and summarize their latest results.