> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gumloop.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Create organization evaluations, choose which agents they grade, and run them over past sessions.

`gumloop evaluations` manages [organization evaluations](/core-concepts/evaluations): rubrics an organization admin defines once and applies to teams, people, or individual agents. Every command accepts `--json` to print the raw response payload.

<Note>These commands require the Enterprise plan and the **Manage evaluations** organization permission. Commands that need an organization default to the one you belong to; pass `--organization org_abc` to be explicit.</Note>

## List evaluations

```bash theme={"dark"}
gumloop evaluations list
```

Returns a tab-separated table with `ID`, `NAME`, `ENABLED`, `AGENTS` (how many agents the targets currently cover), `GRADED`, and `SUCCESS`.

| Flag             | Description                                  |
| ---------------- | -------------------------------------------- |
| `--organization` | Organization id. Defaults to yours.          |
| `--limit`        | Maximum number of evaluations to return.     |
| `--cursor`       | Pagination cursor from a previous list call. |

## Get an evaluation

```bash theme={"dark"}
gumloop evaluations get eval_abc
```

Prints the name, whether it is enabled, its targets, how many agents it covers, how many criteria it has, and its graded count and success rate so far.

## Create an evaluation

```bash theme={"dark"}
gumloop evaluations create --name "Support tone"
gumloop evaluations create --name "Refund policy" --config-file rubric.json
```

`rubric.json` holds the rubric fields — `criteria`, `tags`, `data_points`, `model_name`, `frequency`, `language`, `session_types`:

```json theme={"dark"}
{
  "criteria": [
    {
      "name": "Greets the customer",
      "prompt": "Did the agent greet the customer by name?",
      "priority": "needs_review"
    }
  ],
  "tags": [{ "name": "refund request", "description": "The customer asked for a refund." }]
}
```

| Flag             | Description                                                                         |
| ---------------- | ----------------------------------------------------------------------------------- |
| `--name`         | Evaluation name, unique within the organization.                                    |
| `--description`  | Optional description.                                                               |
| `--organization` | Organization id. Defaults to yours.                                                 |
| `--config-json`  | Inline JSON rubric.                                                                 |
| `--config-file`  | Path to a JSON file containing the rubric. Mutually exclusive with `--config-json`. |

A new evaluation starts disabled and grades nothing until you give it targets and turn it on:

```bash theme={"dark"}
gumloop evaluations targets eval_abc --team-ids team_1
gumloop evaluations update eval_abc --enable
```

Unknown values — a `priority` that isn't `needs_review` or `needs_attention`, a `frequency` that isn't `debounced`, `per_turn`, or `manual`, a criterion missing `name` or `prompt` — are rejected with the offending field path rather than silently changed. `gumloop evaluations options` lists every allowed value.

## Update an evaluation

```bash theme={"dark"}
gumloop evaluations update eval_abc --enable
gumloop evaluations update eval_abc --disable
gumloop evaluations update eval_abc --name "Support tone v2" --config-file rubric.json
```

Only the flags you pass change. A rubric list you send (`criteria`, `tags`, `data_points`) replaces that list wholesale, so send the full list you want to keep. Pass `--description ""` to clear the description.

Turning an evaluation on requires at least one criterion, tag, or data point and at least one covered agent.

## Choose which agents it grades

```bash theme={"dark"}
gumloop evaluations targets eval_abc --whole-organization
gumloop evaluations targets eval_abc --team-ids team_1,team_2 --agent-ids agent_9
gumloop evaluations targets eval_abc --user-ids user_5
```

Each call replaces the whole target set. Targets expand to agents live: a team target grades every agent that team owns, including ones created later; `--user-ids` grades members' personal agents. The output lists the saved targets and how many agents they currently cover. Removing the last target pauses an enabled evaluation.

| Flag                   | Description                            |
| ---------------------- | -------------------------------------- |
| `--whole-organization` | Grade every agent in the organization. |
| `--team-ids`           | Comma-separated team ids.              |
| `--user-ids`           | Comma-separated member user ids.       |
| `--agent-ids`          | Comma-separated agent ids.             |

## Run it on past sessions

```bash theme={"dark"}
gumloop evaluations run eval_abc session_1 session_2
gumloop evaluations run eval_abc session_1 --dry-run
```

Queues up to 200 sessions for grading and prints one line per session: queued ones with their result id, skipped ones with the reason (`ineligible` when the session's agent isn't covered, `in_flight` when it is already being graded). Each queued session costs one credit. `--dry-run` prints the same breakdown without queuing or charging.

Grading is asynchronous — check back with `gumloop evaluations results`.

## See results

```bash theme={"dark"}
gumloop evaluations results eval_abc
gumloop evaluations results eval_abc --grade needs_attention
gumloop evaluations results eval_abc --agent agent_9 --since 2026-09-01T00:00:00Z --json
```

Returns a tab-separated table with `ID`, `SESSION`, `AGENT`, `STATUS`, `GRADE`, and `CREATED`. Failed results show their error code in the grade column.

| Flag                  | Description                                                                      |
| --------------------- | -------------------------------------------------------------------------------- |
| `--agent`             | Only results for this agent.                                                     |
| `--session`           | Only results for this session.                                                   |
| `--grade`             | `pass`, `needs_review`, or `needs_attention`.                                    |
| `--status`            | `queued`, `in_progress`, `completed`, or `failed`.                               |
| `--since` / `--until` | Created-at bounds, RFC 3339 with an offset (for example `2026-09-01T00:00:00Z`). |
| `--limit`             | Maximum number of results to return.                                             |
| `--cursor`            | Pagination cursor from a previous call.                                          |

## Metrics

```bash theme={"dark"}
gumloop evaluations metrics eval_abc --days 7
```

Prints how many sessions received each grade in the window (default 30 days).

## Delete an evaluation

```bash theme={"dark"}
gumloop evaluations delete eval_abc
```

The evaluation stops running and disappears from lists. Results it already produced stay attached to their sessions.