Manage evaluators

The eval evaluator command group manages evaluators: browse built-in templates, list and show existing evaluators, create evaluators, update TEA drafts, submit TEA versions, and delete evaluators. An evaluator is typically a rubric prompt backed by a judge model; it receives input fields and produces a score or distribution.

--project applies only to the Coze evaluation backend; the TEA backend ignores it. TEA version commands are available only on the TEA evaluation backend.

evaluator template list

List built-in evaluator templates. evaluator templates is a compatibility entry point equivalent to evaluator template list.

Flag / Argument	Description	Default
`--type <type>`	TEA built-in template type; use `prompt` or a numeric type.	`prompt`
`--locale <locale>`	TEA template locale; `cn` or `zh-CN` appends the Chinese locale.	`zh-CN`
`-p, --project <name>`	Coze project name; ignored on TEA.	`default`
`--json`	Output raw JSON.	`false`

agentkit eval evaluator template list
agentkit eval evaluator templates --type prompt

evaluator template show

Show a built-in evaluator template, including its prompt and input fields.

Flag / Argument	Description	Default
`<key>`	TEA template key, or Coze template ID / name.	Required
`--type <type>`	TEA built-in template type; use `prompt` or a numeric type.	`prompt`
`--language <type>`	TEA code template `language_type`.	—
`--locale <locale>`	TEA template locale; `cn` or `zh-CN` appends the Chinese locale.	`zh-CN`
`-p, --project <name>`	Coze project name; ignored on TEA.	`default`
`--json`	Output raw JSON.	`false`

agentkit eval evaluator template show relevance

evaluator list

List evaluators in the current project or workspace.

Flag / Argument	Description	Default
`-p, --project <name>`	Coze project name; ignored on TEA.	`default`
`--json`	Output raw JSON.	`false`

agentkit eval evaluator list

evaluator show

Show an evaluator’s details: rubric prompt, model configuration, and input fields. On TEA, you can select a specific evaluator version.

Flag / Argument	Description	Default
`<id\|name>`	Evaluator ID or exact name.	Required
`--evaluator-version <id\|name>`	TEA evaluator version ID or version string, such as `0.0.1`.	—
`-p, --project <name>`	Coze project name; ignored on TEA.	`default`
`--json`	Output raw JSON.	`false`

agentkit eval evaluator show relevance

agentkit eval evaluator show relevance --evaluator-version 0.0.1

evaluator create

Create an evaluator. You can clone rubric and input schema from a built-in template, or define the evaluator with a custom prompt file.

Flag / Argument	Description	Default
`--name <name>`	Evaluator name.	Required
`--from-template <key\|id\|name>`	Clone the rubric and schema from a built-in template. TEA uses the template key; Coze uses the template ID or name.	—
`--model <name\|id>`	Judge model. TEA accepts `Doubao 2.0 Lite`, `Doubao 2.0 Pro`, `Doubao 2.0 Mini`, or model IDs `1`, `2`, `3`; Coze uses an Ark endpoint ID.	—
`--prompt-file <path>`	Custom rubric text file.	—
`--description <text>`	Evaluator description.	—
`--input-schemas <keys>`	Comma-separated input field keys; used only on TEA.	—
`-p, --project <name>`	Coze project name; ignored on TEA.	`default`
`--json`	Output raw JSON.	`false`

On Coze, provide at least one of --from-template or --prompt-file. On TEA, you can create from a template or from a custom prompt and input fields.

agentkit eval evaluator create \
  --name relevance \
  --from-template relevance \
  --model "Doubao 2.0 Pro"

Create rubric.txt before running the custom evaluator example:

rubric.txt

Question: {{input}}
Reference answer: {{reference_output}}
Actual answer: {{output}}
Check whether the actual answer correctly answers the question and matches the reference
Score 1 for a correct and complete answer, otherwise 0, and briefly explain the score

agentkit eval evaluator create \
  --name custom-score \
  --prompt-file ./rubric.txt \
  --input-schemas input,output,reference_output \
  --model 2

Reference input fields in the prompt with placeholders like , , and ; these field names are the evaluator’s input schema and are mapped automatically during eval run.

evaluator update-draft

Update a TEA evaluator draft. When --evaluator-content-file is passed, that JSON file is used as the complete evaluator content and overrides --prompt-file, --model, and --input-schemas-json.

evaluator update-draft is supported only on the TEA evaluation backend.

Flag / Argument	Description	Default
`<id>`	Evaluator ID.	Required
`--prompt-file <path>`	Custom rubric text file.	—
`--model <name\|id>`	Judge model: `Doubao 2.0 Lite`, `Doubao 2.0 Pro`, `Doubao 2.0 Mini`, or model IDs `1`, `2`, `3`.	—
`--input-schemas-json <json>`	JSON array of input field schemas.	—
`--evaluator-content-file <path>`	Complete `evaluator_content` JSON file.	—
`--evaluator-type <type>`	Evaluator type; unchanged when omitted.	Unchanged
`--json`	Output raw JSON.	`false`

agentkit eval evaluator update-draft <evaluator-id> \
  --prompt-file ./rubric.txt \
  --input-schemas-json '[{"key":"input"},{"key":"output"},{"key":"reference_output"}]'

evaluator version list

List TEA evaluator versions.

Flag / Argument	Description	Default
`--evaluator <id\|name>`	Evaluator ID or exact name.	Required
`-r, --region <region>`	Volcengine region.	Environment
`--json`	Output raw JSON.	`false`

agentkit eval evaluator version list --evaluator relevance

evaluator version submit

Submit a TEA evaluator draft as a version.

Flag / Argument	Description	Default
`<version>`	Version string, such as `0.0.2`.	Required
`--evaluator <id\|name>`	Evaluator ID or exact name.	Required
`--description <text>`	Version description.	—
`--desc <text>`	Alias for `--description`.	—
`-r, --region <region>`	Volcengine region.	Environment
`--json`	Output raw JSON.	`false`

agentkit eval evaluator version submit 0.0.2 --evaluator relevance --description "Update rubric"

evaluator delete

Delete an evaluator. (Alias: rm)

This command deletes the evaluator. Check the target first; the CLI prompts for confirmation unless you pass --yes.

Flag / Argument	Description	Default
`<id\|name>`	Evaluator ID or exact name.	Required
`-p, --project <name>`	Coze project name; ignored on TEA.	`default`
`-y, --yes`	Skip the confirmation prompt.	`false`

agentkit eval evaluator delete relevance -y

After creation, use evaluator show to inspect the model, rubric, and input fields. Submit a TEA draft before using an explicit version in an experiment. Submit a new version after further draft edits; editing a draft does not update an existing version

agentkit eval evaluator show relevance
agentkit eval evaluator version submit 0.0.1 --evaluator relevance
agentkit eval evaluator version list --evaluator relevance

Template keys and models must be available on the selected backend. Coze model arguments use an Ark endpoint ID available to your account, not a TEA numeric model ID

Getting Started

AI-Friendly

Command reference

evaluator template list

evaluator template show

evaluator list

evaluator show

evaluator create

evaluator update-draft

evaluator version list

evaluator version submit

evaluator delete

​evaluator template list

​evaluator template show

​evaluator list

​evaluator show

​evaluator create

​evaluator update-draft

​evaluator version list

​evaluator version submit

​evaluator delete

evaluator template list

evaluator template show

evaluator list

evaluator show

evaluator create

evaluator update-draft

evaluator version list

evaluator version submit

evaluator delete