> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sudoiq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals, tests, and feedback

> Test examples, execution feedback, and improving agent workflows

**Agents** improve when you close the loop: define expectations (**tests / evals**), run workflows in production or staging, collect **human or automated feedback**, and use that signal to change prompts, tools, graph structure, or routing.

## Agent test examples

Store **input/output pairs** (and optional **quality** grades) on an agent so you can regression-check behavior as you edit the graph.

In Python, use **`client.tests.create_tests`** with **`TestExampleItem`** rows (input/output types and content, graded quality, and a short reason):

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
from agentsapi import AgentServiceAPIClient
from agentsapi.models.agent_tests import TestExampleItem

async def seed_tests():
    client = AgentServiceAPIClient()
    client.set_api_key("sk_...")

    examples = [
        TestExampleItem(
            input_type="text",
            input_content="Summarize: Q3 revenue was up 12%.",
            input_description="",
            output_type="text",
            output_content="Q3 revenue increased about 12% year over year.",
            output_description="",
            graded_quality="high",
            quality_reason="Accurate, concise summary aligned with input.",
        ),
    ]
    await client.tests.create_tests("YOUR_AGENT_UUID", examples)

# asyncio.run(seed_tests())
```

Use **`client.tests.get_tests(agent_id)`** (and related helpers) to read back what is stored. Exact fields match your product’s test UX and any CI you run against the API.

## Automated graders

<Note>
  **Default path:** On **Sudoiq**, platform **analyst agents** normally **design and attach graders for you** (rubric text, versions, proposed graders for comparison, promotion in the product UI). You do **not** need to call the API to get a useful eval loop.

  **This section is reference** for engineers: exact **SDK** calls, the **standard grader JSON** shape, and optional **post-hoc** or **pipeline-integrated** grading when you build custom automation or debug outside the guided flow.
</Note>

A **grader** is a dedicated **grader-type workflow** linked to an agent. You give it a natural-language **rubric**; the service builds a small graph that scores a **completed run** (original prompt plus node output) and returns structured JSON.

You can **iterate on graders** over time—adjust the rubric, create **proposed** graders for side-by-side comparison before promoting one, and treat grading like any other workflow you refine. For **post-hoc** scoring you call the client after the fact (see below). For **tighter integration**, graph nodes can reference grader workflows as **blocking** or **non-blocking exit validation** so evaluation runs in the execution pipeline (for example validating a specific node before the run continues or finishes).

### Create a grader

Use **`AgentServiceAPIClient.graders.create_grader`**: pass **`rubric`** (task-specific criteria), **`rubric_name`**, and optionally **`proposed=True`** when the agent already has a primary grader and you want a **non-primary** grader for A/B comparison. The response includes **`grader_agent_id`** (the new grader workflow), **`agent_id`**, **`version`**, and **`message`**.

The platform **appends a fixed “standard rubric” block** to your text so model outputs stay in the JSON shape below—you only maintain the custom part of the rubric.

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from agentsapi import AgentServiceAPIClient

client = AgentServiceAPIClient()
client.set_api_key("sk_...")
resp = await client.graders.create_grader(
    agent_id="YOUR_AGENT_UUID",
    rubric="The answer must cite the quarter and percentage change. Penalize hallucinated figures.",
    rubric_name="Q&A accuracy grader",
    proposed=False,  # True: add a proposed grader if a primary already exists
)
# resp.grader_agent_id — grader workflow id
```

### Standard grader output format

The grader model is instructed to return **JSON** with exactly these top-level fields:

| Field                | Type    | Meaning                                                                                                                                                                                                        |
| -------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pass`               | boolean | Whether the graded run satisfied the task to your rubric’s satisfaction.                                                                                                                                       |
| `quality_score`      | string  | One of **`very-low`**, **`low`**, **`medium`**, **`high`**, **`very-high`**.                                                                                                                                   |
| `quality_attributes` | array   | Each element is an **object with a single key**: the same tier string as `quality_score`, and the **value** is a **reason** string. Multiple entries are allowed, including several objects with the same key. |

Example (matches the platform’s reference shape in **`GraderOutputExample`**):

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "pass": true,
  "quality_score": "high",
  "quality_attributes": [
    { "high": "The agent interpreted the task and produced accurate results." },
    { "high": "Output structure matched expectations." },
    { "medium": "Slightly verbose but complete." }
  ]
}
```

For parsing in Python, **`agentsapi.models.grader.GraderOutputResult`** validates this shape. To inspect the **`example_output`** shape before you create a grader, use **`await client.graders.get_agent_rubric(agent_id)`** — the response includes it even when **`has_grader`** is false.

### Run a grader on a finished run

After a run completes, **`await client.graders.execute_grader(agent_id=..., task_id=..., node_id=..., tenant_id=...)`** starts the grader workflow against that run (optional **`node_id`** grades a specific node’s output, for example from **non-blocking exit validation**). The call returns an **`ExecuteAgentResponse`** with the grader’s **`task_id`** when a grader run was started, or **`None`** if no compatible grader was found.

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from agentsapi import ExecuteAgentResponse

# After `client = AgentServiceAPIClient()` and `set_api_key`:
started: ExecuteAgentResponse | None = await client.graders.execute_grader(
    agent_id="YOUR_AGENT_UUID",
    task_id=12345,
    # node_id="some_node",  # optional
    # tenant_id="...",      # optional, for tenant-scoped runs
)
```

Use **`await client.graders.get_grader_runs_for_agent(agent_id)`** to list grader runs and read stored **`grader_result`** payloads.

## Execution feedback

After a **run** completes (or while reviewing it), record structured notes or simple pass/fail for **optimization** and analytics.

The **`agentsapi`** namespaces **`client.execution_feedback`** and **`client.agent_feedback`** (aliases) expose methods such as:

* **`list_feedback`** — page through feedback for an agent’s runs (cursor parameters vary by release; see your installed client).
* **`save_feedback`** — attach detailed **`ExecutionFeedbackData`** for a **`task_id`** (admin vs user **source**).
* **`save_simple_feedback` / `get_simple_feedback`** — binary **passed** / **failed** for a run, with optional **source**.

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# After you know task_id from a run:
await client.agent_feedback.save_simple_feedback(
    task_id=12345,
    passed=True,
    source="user",
)
```

On **Sudoiq**, **analyst agents** continuously review the same signals—**stored tests**, **grader results**, **human or simple feedback**, and run telemetry—as they arrive. They use that picture to **recommend changes** (prompts, graph structure, tools) and to **try improvements in the product** (for example proposing rubrics, running comparisons, or staging tweaks) so you are not only collecting data but driving an eval loop inside the platform.

## How this improves workflows

1. **Tests** define “what good looks like” on fixed examples—use them before/after edits to catch regressions.
2. **Live runs** surface real user or operational inputs.
3. **Feedback** (simple or rich) marks which runs succeeded or failed expectations.
4. **Graders** score outputs against your rubric so you can trend quality and compare workflow versions.

Together, these feeds are what **analyst agents** and your own processes use to steer continuous improvement (see above).

This page documents the main **Python client** entry points for that data: **`client.tests`**, **`client.graders`**, and **`client.agent_feedback`** on **`AgentServiceAPIClient`** (see [Python client](/clients/python)). Your workspace may also offer UI or batch eval APIs on top of the same signals.

## Related

* [Python client](/clients/python)
* [Output validators](/validation/output-validators)
* [CLI](/clients/cli) — `agentservice demo` for a sample tool/validator/handler loop