Evaluations - Gumloop

gumloop evaluations manages organization evaluations: rubrics an organization admin defines once and applies to teams, people, or individual agents. Every command accepts --json to print the raw response payload.

These commands require the Enterprise plan and the Manage evaluations organization permission. Commands that need an organization default to the one you belong to; pass --organization org_abc to be explicit.

List evaluations

gumloop evaluations list

Returns a tab-separated table with ID, NAME, ENABLED, AGENTS (how many agents the targets currently cover), GRADED, and SUCCESS.

Flag	Description
`--organization`	Organization id. Defaults to yours.
`--limit`	Maximum number of evaluations to return.
`--cursor`	Pagination cursor from a previous list call.

Get an evaluation

gumloop evaluations get eval_abc

Prints the name, whether it is enabled, its targets, how many agents it covers, how many criteria it has, and its graded count and success rate so far.

Create an evaluation

gumloop evaluations create --name "Support tone"
gumloop evaluations create --name "Refund policy" --config-file rubric.json

rubric.json holds the rubric fields — criteria, tags, data_points, model_name, frequency, language, session_types:

{
  "criteria": [
    {
      "name": "Greets the customer",
      "prompt": "Did the agent greet the customer by name?",
      "priority": "needs_review"
    }
  ],
  "tags": [{ "name": "refund request", "description": "The customer asked for a refund." }]
}

Flag	Description
`--name`	Evaluation name, unique within the organization.
`--description`	Optional description.
`--organization`	Organization id. Defaults to yours.
`--config-json`	Inline JSON rubric.
`--config-file`	Path to a JSON file containing the rubric. Mutually exclusive with `--config-json`.

A new evaluation starts disabled and grades nothing until you give it targets and turn it on:

gumloop evaluations targets eval_abc --team-ids team_1
gumloop evaluations update eval_abc --enable

Unknown values — a priority that isn’t needs_review or needs_attention, a frequency that isn’t debounced, per_turn, or manual, a criterion missing name or prompt — are rejected with the offending field path rather than silently changed. gumloop evaluations options lists every allowed value.

Update an evaluation

gumloop evaluations update eval_abc --enable
gumloop evaluations update eval_abc --disable
gumloop evaluations update eval_abc --name "Support tone v2" --config-file rubric.json

Only the flags you pass change. A rubric list you send (criteria, tags, data_points) replaces that list wholesale, so send the full list you want to keep. Pass --description "" to clear the description. Turning an evaluation on requires at least one criterion, tag, or data point and at least one covered agent.

Choose which agents it grades

gumloop evaluations targets eval_abc --whole-organization
gumloop evaluations targets eval_abc --team-ids team_1,team_2 --agent-ids agent_9
gumloop evaluations targets eval_abc --user-ids user_5

Each call replaces the whole target set. Targets expand to agents live: a team target grades every agent that team owns, including ones created later; --user-ids grades members’ personal agents. The output lists the saved targets and how many agents they currently cover. Removing the last target pauses an enabled evaluation.

Flag	Description
`--whole-organization`	Grade every agent in the organization.
`--team-ids`	Comma-separated team ids.
`--user-ids`	Comma-separated member user ids.
`--agent-ids`	Comma-separated agent ids.

Run it on past sessions

gumloop evaluations run eval_abc session_1 session_2
gumloop evaluations run eval_abc session_1 --dry-run

Queues up to 200 sessions for grading and prints one line per session: queued ones with their result id, skipped ones with the reason (ineligible when the session’s agent isn’t covered, in_flight when it is already being graded). Each queued session costs one credit. --dry-run prints the same breakdown without queuing or charging. Grading is asynchronous — check back with gumloop evaluations results.

See results

gumloop evaluations results eval_abc
gumloop evaluations results eval_abc --grade needs_attention
gumloop evaluations results eval_abc --agent agent_9 --since 2026-09-01T00:00:00Z --json

Returns a tab-separated table with ID, SESSION, AGENT, STATUS, GRADE, and CREATED. Failed results show their error code in the grade column.

Flag	Description
`--agent`	Only results for this agent.
`--session`	Only results for this session.
`--grade`	`pass`, `needs_review`, or `needs_attention`.
`--status`	`queued`, `in_progress`, `completed`, or `failed`.
`--since` / `--until`	Created-at bounds, RFC 3339 with an offset (for example `2026-09-01T00:00:00Z`).
`--limit`	Maximum number of results to return.
`--cursor`	Pagination cursor from a previous call.

Metrics

gumloop evaluations metrics eval_abc --days 7

Prints how many sessions received each grade in the window (default 30 days).

Delete an evaluation

gumloop evaluations delete eval_abc

The evaluation stops running and disappears from lists. Results it already produced stay attached to their sessions.

​List evaluations

​Get an evaluation

​Create an evaluation

​Update an evaluation

​Choose which agents it grades

​Run it on past sessions

​See results

​Metrics

​Delete an evaluation

List evaluations

Get an evaluation

Create an evaluation

Update an evaluation

Choose which agents it grades

Run it on past sessions

See results

Metrics

Delete an evaluation