Evaluation loop

Evaluation closes the “develop → deploy → evaluate → tune” loop: use a fixed dataset and one or more evaluators to score a deployed runtime, then iterate on the quantified results. This page walks through the full flow from initializing a project to viewing evaluation results using a question-answering agent as the example. For details on each command, see View evaluation backend, Manage datasets, Manage evaluators, Manage evaluation targets, Run an experiment, and View experiments and results.

Before you start, configure Volcengine AK/SK credentials or obtain valid short-lived STS credentials through agentkit login. Evaluation is currently in Preview. The CLI first resolves whether the current account uses the Coze or TEA evaluation backend; run agentkit eval backend to inspect the resolved backend.

Uploading cases stores their inputs and reference answers on the evaluation platform. A submitted experiment calls the target agent and judging model and may incur charges. Use data approved for evaluation. --dry-run still contacts the backend to resolve resources, but does not submit an experiment or automatically publish a dataset version

End-to-end flow

Initialize and deploy an agent

Follow High-code agents to initialize a project, configure its model, and deploy it as qa-agent, or use an existing callable Runtime. Verify the target and evaluation backend

agentkit runtime show qa-agent
agentkit invoke run "hello" --runtime-id "<qa-agent runtime ID>"
agentkit eval backend

Supply the actual Runtime ID. The remaining examples show the TEA backend. Coze and TEA have different templates and version requirements; use the results returned for your account

Create a dataset and add cases

The dataset is the data source for an experiment. Create it with the input and reference_output fields, then add cases one at a time or in bulk:

agentkit eval dataset create --name qa-set --schema "input,reference_output"

# Add one at a time
agentkit eval dataset add qa-set \
  --field "input=What is the capital of France?" \
  --field "reference_output=Paris"

# Or add in bulk from a JSON file
agentkit eval dataset add qa-set --file cases.json

cases.json is an array of {field: value} objects:

cases.json

[
  { "input": "What is the capital of France?", "reference_output": "Paris" },
  { "input": "What is the chemical formula of water?", "reference_output": "H2O" }
]

After adding, run agentkit eval dataset show qa-set to verify the cases.

Create an evaluator

An evaluator is the judging model that assigns scores. Derive one from a built-in template and point it at your own Ark model endpoint:

# Browse built-in templates
agentkit eval evaluator template list

# Derive from the "relevance" template and swap in your own endpoint
agentkit eval evaluator create \
  --name relevance \
  --from-template "<template name from list>" \
  --model ep-xxxxxxxx

The TEA backend requires an evaluator version. Replace the template placeholder with a name returned by the list command, then submit a version after creating or updating the draft. Coze users skip this step and omit --evaluator-version 0.0.1 from subsequent commands:

agentkit eval evaluator version submit 0.0.1 --evaluator relevance

Run an experiment

Submit the dataset, evaluator, and target runtime as one experiment. On the first run, add --dry-run to confirm the resolved dataset version, evaluator version, target, and field mapping are correct:

# Check first
agentkit eval run --dataset qa-set --evaluator relevance --evaluator-version 0.0.1 --target qa-agent --dry-run

# Submit
agentkit eval run --dataset qa-set --evaluator relevance --evaluator-version 0.0.1 --target qa-agent
# → returns an experiment id, e.g. 75901...

On Coze you may omit --evaluator-version; on TEA you must pass one --evaluator-version for each --evaluator. To confirm available target versions, first run agentkit eval target version-list --source-target-id <source-target-id>.

View results

Use the returned experiment id to view the aggregate scores and per-case results:

# Status, weighted overall score, and per-evaluator averages
agentkit eval experiment show 75901xxxxxxxxxxxxx

# Per-case results: each case, the target output, and each evaluator's score and rationale
agentkit eval experiment results 75901xxxxxxxxxxxxx

Iterate

Improve the agent’s prompt, tools, or model based on the results, redeploy, then run the experiment again with the same dataset and evaluator. Compare the aggregate scores and per-case results of the two runs to judge whether the change is an improvement or a regression:

agentkit launch
agentkit eval run --dataset qa-set --evaluator relevance --evaluator-version 0.0.1 --target qa-agent

Wire it into CI

The evaluation commands need no browser login and all support --json, so they can be orchestrated directly in a CI pipeline: the experimentId returned by eval run feeds into eval experiment show or results.

# A minimal loop you can drop into CI
set -euo pipefail
experiment_id=$(agentkit eval run --dataset qa-set --evaluator relevance --evaluator-version 0.0.1 --target qa-agent --json | jq -er .experimentId)
agentkit eval experiment show "$experiment_id" --json

The CI example submits an experiment and reads its status once; it does not wait for all cases or enforce a quality threshold. To gate a release, poll until a terminal state and decide the pipeline outcome from actual scores and failed cases. Receiving an experiment ID does not mean the evaluation passed

Practical guidance

Fix the dataset: use the same dataset across iterations so results can be compared side by side. Stabilize it before changing the agent.
Keep only stable inputs and reference answers: a dataset typically needs only input and reference_output. The model’s actual output is produced by the evaluated target during the experiment and should not be written into the dataset.
Prefer deriving evaluators from built-in templates: evaluator template list provides mature scoring rubrics; clone one and swap in your own judge model rather than writing one from scratch.
Use a strong judging model: point --model at a capable Ark endpoint; weaker models produce noisier scores.
Weight multiple evaluators: a single eval run can take several --evaluator flags to score across dimensions such as relevance and completeness.
Confirm the target runtime is online: --target must point to a deployed, responsive runtime, or cases will fail. When cases fail without an experiment-level error, the target runtime usually did not respond correctly — it is not an evaluation-config problem.

开始使用

AI-Friendly

命令参考

End-to-end flow

Wire it into CI

Practical guidance

​End-to-end flow

​Wire it into CI

​Practical guidance

End-to-end flow

Wire it into CI

Practical guidance