> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veadk.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation loop

Evaluation closes the "develop → deploy → evaluate → tune" loop: use a fixed dataset and one or more evaluators to score a deployed runtime, then iterate on the quantified results. This page walks through the full flow from initializing a project to viewing evaluation results using a question-answering agent as the example. For details on each command, see [View evaluation backend](https://docs.veadk.xyz/productions/agentkit-cli/preview/en/commands/eval/backend.md), [Manage datasets](https://docs.veadk.xyz/productions/agentkit-cli/preview/en/commands/eval/dataset.md), [Manage evaluators](https://docs.veadk.xyz/productions/agentkit-cli/preview/en/commands/eval/evaluator.md), [Manage evaluation targets](https://docs.veadk.xyz/productions/agentkit-cli/preview/en/commands/eval/target.md), [Run an experiment](https://docs.veadk.xyz/productions/agentkit-cli/preview/en/commands/eval/run.md), and [View experiments and results](https://docs.veadk.xyz/productions/agentkit-cli/preview/en/commands/eval/experiment.md).

> [!NOTE]
>
> Before you start, configure Volcengine AK/SK credentials or obtain valid short-lived STS credentials through `agentkit login`. Evaluation is currently in Preview. The CLI first resolves whether the current account uses the Coze or TEA evaluation backend; run `agentkit eval backend` to inspect the resolved backend.

> [!WARNING]
>
> Uploading cases stores their inputs and reference answers on the evaluation platform. A submitted experiment calls the target agent and judging model and may incur charges. Use data approved for evaluation. `--dry-run` still contacts the backend to resolve resources, but does not submit an experiment or automatically publish a dataset version

## End-to-end flow

**1. Initialize and deploy an agent**

Follow [High-code agents](https://docs.veadk.xyz/productions/agentkit-cli/preview/en/workflows/high-code) to initialize a project, configure its model, and deploy it as `qa-agent`, or use an existing callable Runtime. Verify the target and evaluation backend

```bash lines theme={null}
agentkit runtime show qa-agent
agentkit invoke run "hello" --runtime-id "<qa-agent runtime ID>"
agentkit eval backend
```

Supply the actual Runtime ID. The remaining examples show the TEA backend. Coze and TEA have different templates and version requirements; use the results returned for your account

**2. Create a dataset and add cases**

The dataset is the data source for an experiment. Create it with the `input` and `reference_output` fields, then add cases one at a time or in bulk:

```bash lines theme={null}
agentkit eval dataset create --name qa-set --schema "input,reference_output"

# Add one at a time
agentkit eval dataset add qa-set \
  --field "input=What is the capital of France?" \
  --field "reference_output=Paris"

# Or add in bulk from a JSON file
agentkit eval dataset add qa-set --file cases.json
```

`cases.json` is an array of `{field: value}` objects:

```json cases.json lines theme={null}
[
  { "input": "What is the capital of France?", "reference_output": "Paris" },
  { "input": "What is the chemical formula of water?", "reference_output": "H2O" }
]
```

After adding, run `agentkit eval dataset show qa-set` to verify the cases.

**3. Create an evaluator**

An evaluator is the judging model that assigns scores. Derive one from a built-in template and point it at your own Ark model endpoint:

```bash lines theme={null}
# Browse built-in templates
agentkit eval evaluator template list

# Derive from the "relevance" template and swap in your own endpoint
agentkit eval evaluator create \
  --name relevance \
  --from-template "<template name from list>" \
  --model ep-xxxxxxxx
```

The TEA backend requires an evaluator version. Replace the template placeholder with a name returned by the list command, then submit a version after creating or updating the draft. Coze users skip this step and omit `--evaluator-version 0.0.1` from subsequent commands:

```bash lines theme={null}
agentkit eval evaluator version submit 0.0.1 --evaluator relevance
```

**4. Run an experiment**

Submit the dataset, evaluator, and target runtime as one experiment. On the first run, add `--dry-run` to confirm the resolved dataset version, evaluator version, target, and field mapping are correct:

```bash lines theme={null}
# Check first
agentkit eval run --dataset qa-set --evaluator relevance --evaluator-version 0.0.1 --target qa-agent --dry-run

# Submit
agentkit eval run --dataset qa-set --evaluator relevance --evaluator-version 0.0.1 --target qa-agent
# → returns an experiment id, e.g. 75901...
```

> [!NOTE]
>
> On Coze you may omit `--evaluator-version`; on TEA you must pass one `--evaluator-version` for each `--evaluator`. To confirm available target versions, first run `agentkit eval target version-list --source-target-id <source-target-id>`.

**5. View results**

Use the returned experiment id to view the aggregate scores and per-case results:

```bash lines theme={null}
# Status, weighted overall score, and per-evaluator averages
agentkit eval experiment show 75901xxxxxxxxxxxxx

# Per-case results: each case, the target output, and each evaluator's score and rationale
agentkit eval experiment results 75901xxxxxxxxxxxxx
```

**6. Iterate**

Improve the agent's prompt, tools, or model based on the results, redeploy, then run the experiment again with the same dataset and evaluator. Compare the aggregate scores and per-case results of the two runs to judge whether the change is an improvement or a regression:

```bash lines theme={null}
agentkit launch
agentkit eval run --dataset qa-set --evaluator relevance --evaluator-version 0.0.1 --target qa-agent
```

## Wire it into CI

The evaluation commands need no browser login and all support `--json`, so they can be orchestrated directly in a CI pipeline: the `experimentId` returned by `eval run` feeds into `eval experiment show` or `results`.

```bash lines theme={null}
# A minimal loop you can drop into CI
set -euo pipefail
experiment_id=$(agentkit eval run --dataset qa-set --evaluator relevance --evaluator-version 0.0.1 --target qa-agent --json | jq -er .experimentId)
agentkit eval experiment show "$experiment_id" --json
```

The CI example submits an experiment and reads its status once; it does not wait for all cases or enforce a quality threshold. To gate a release, poll until a terminal state and decide the pipeline outcome from actual scores and failed cases. Receiving an experiment ID does not mean the evaluation passed

## Practical guidance

* **Fix the dataset**: use the same dataset across iterations so results can be compared side by side. Stabilize it before changing the agent.
* **Keep only stable inputs and reference answers**: a dataset typically needs only `input` and `reference_output`. The model's actual output is produced by the evaluated target during the experiment and should not be written into the dataset.
* **Prefer deriving evaluators from built-in templates**: `evaluator template list` provides mature scoring rubrics; clone one and swap in your own judge model rather than writing one from scratch.
* **Use a strong judging model**: point `--model` at a capable Ark endpoint; weaker models produce noisier scores.
* **Weight multiple evaluators**: a single `eval run` can take several `--evaluator` flags to score across dimensions such as relevance and completeness.
* **Confirm the target runtime is online**: `--target` must point to a deployed, responsive runtime, or cases will fail. When cases fail without an experiment-level error, the target runtime usually did not respond correctly — it is not an evaluation-config problem.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.