> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veadk.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Run an evaluation

> Select cases with evalCaseIds; omission or an empty array falls back to evalIds. All cases are evaluated when neither selects cases. evalMetrics is required; the response includes per-case status and metric results

Exposed by the verified SDK/ADK versions for trusted development and debugging. Paths depend on the installed ADK version and should not be used as public application entry points

The server must expose development evaluation endpoints and have evaluation dependencies installed. Prepare a set and cases, then select `evalMetrics`; query [List evaluation metrics](/productions/api-reference/preview/en/harness-runtime/list-eval-metrics) for available metrics

Use `evalCaseIds` to select cases. When omitted or empty, it falls back to the deprecated `evalIds`; if neither selects cases, the entire set is evaluated

<Warning>
  Evaluation runs the agent and selected metrics again. It can call models and tools, incur charges, and perform external actions. Check samples, tool permissions, and metric settings before running it
</Warning>

HTTP `200` means evaluation results were returned, not that all cases passed. Inspect `finalEvalStatus` and metric results for each item in `runEvalResults`


## OpenAPI

````yaml productions/api-reference/openapi/en/harness-runtime.json POST /dev/apps/{app_name}/eval-sets/{eval_set_id}/run
openapi: 3.1.0
info:
  title: Harness Runtime
  version: agentkit-cli 0.54.0 / SDK 0.8.0 / ADK 2.2.0
  description: >-
    Harness Runtime deployed by AgentKit CLI 0.54.0, verified with VeADK 1.0.8,
    AgentKit SDK 0.8.0, and Google ADK 2.2.0; scheduled tasks require separate
    enablement
servers:
  - url: http://localhost:8000
    description: Local server
security:
  - {}
  - RuntimeBearer: []
tags:
  - name: Service and applications
  - name: Sessions
  - name: Agent execution
  - name: Artifacts
  - name: Memory
  - name: Evaluation · development server
  - name: Debugging · development server
  - name: Compatibility endpoints
  - name: Harness cronjobs
paths:
  /dev/apps/{app_name}/eval-sets/{eval_set_id}/run:
    post:
      tags:
        - Evaluation · development server
      summary: Run an evaluation
      description: >-
        Select cases with evalCaseIds; omission or an empty array falls back to
        evalIds. All cases are evaluated when neither selects cases. evalMetrics
        is required; the response includes per-case status and metric results


        Exposed by the verified SDK/ADK versions for trusted development and
        debugging. Paths depend on the installed ADK version and should not be
        used as public application entry points
      operationId: harnessRuntimeRunEvaluation
      parameters:
        - name: app_name
          in: path
          required: true
          schema:
            type: string
            title: App Name
          description: Application name; harness_agent for the CLI deployment
          example: harness_agent
        - name: eval_set_id
          in: path
          required: true
          schema:
            type: string
            title: Eval Set Id
          description: Evaluation set ID
          example: travel-eval
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/RunEvalRequest'
            example:
              evalCaseIds:
                - case-001
              evalMetrics:
                - metricName: tool_trajectory_avg_score
                  threshold: 1
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/RunEvalResponse'
        '400':
          description: Invalid request or a required service/dependency is unavailable
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ApiError'
        '411':
          description: 'Shared OAuth only: a valid Content-Length is required'
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
              example:
                error: CONTENT_LENGTH_REQUIRED
        '413':
          description: 'Shared OAuth only: request body exceeds 128 KiB'
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
              example:
                error: REQUEST_TOO_LARGE
        '422':
          description: Request validation failed
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
components:
  schemas:
    RunEvalRequest:
      properties:
        evalIds:
          items:
            type: string
          type: array
          title: Evalids
          deprecated: true
          description: Deprecated; use evalCaseIds
        evalCaseIds:
          items:
            type: string
          type: array
          title: Evalcaseids
          description: >-
            Case IDs to evaluate; omission or an empty array falls back to
            evalIds. Evaluate all cases when neither selects cases
        evalMetrics:
          items:
            $ref: '#/components/schemas/EvalMetric'
          type: array
          title: Evalmetrics
          description: Metrics to apply in this evaluation
      type: object
      required:
        - evalMetrics
      title: RunEvalRequest
    RunEvalResponse:
      properties:
        runEvalResults:
          items:
            $ref: '#/components/schemas/RunEvalResult'
          type: array
          title: Runevalresults
          description: Per-case evaluation results
      type: object
      required:
        - runEvalResults
      title: RunEvalResponse
    ApiError:
      type: object
      properties:
        detail:
          type: string
          description: Error details
      required:
        - detail
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          type: array
          title: Detail
          description: Error details
      type: object
      title: HTTPValidationError
    EvalMetric:
      properties:
        metricName:
          type: string
          title: Metricname
          description: Evaluation metric name
        threshold:
          anyOf:
            - type: number
            - type: 'null'
          title: Threshold
          description: >-
            Threshold field scheduled for deprecation; prefer criterion. The
            metric defines its interpretation
        criterion:
          anyOf:
            - $ref: '#/components/schemas/BaseCriterion'
            - type: 'null'
          description: Evaluation criterion used by the metric
        customFunctionPath:
          anyOf:
            - type: string
            - type: 'null'
          title: Customfunctionpath
          description: Import path of a custom evaluation function
      additionalProperties: false
      type: object
      required:
        - metricName
      title: EvalMetric
    RunEvalResult:
      properties:
        evalSetFile:
          type: string
          title: Evalsetfile
        evalSetId:
          type: string
          title: Evalsetid
          description: Evaluation set ID
        evalId:
          type: string
          title: Evalid
          description: Evaluation case ID
        finalEvalStatus:
          $ref: '#/components/schemas/EvalStatus'
          description: Final evaluation status
        evalMetricResults:
          items:
            prefixItems:
              - $ref: '#/components/schemas/EvalMetric'
              - $ref: '#/components/schemas/EvalMetricResult'
            type: array
            maxItems: 2
            minItems: 2
          type: array
          title: Evalmetricresults
          default: []
          deprecated: true
        overallEvalMetricResults:
          items:
            $ref: '#/components/schemas/EvalMetricResult'
          type: array
          title: Overallevalmetricresults
          description: Case-level metric results
        evalMetricResultPerInvocation:
          items:
            $ref: '#/components/schemas/EvalMetricResultPerInvocation'
          type: array
          title: Evalmetricresultperinvocation
          description: Per-invocation metric results
        userId:
          type: string
          title: Userid
          description: User ID
        sessionId:
          type: string
          title: Sessionid
          description: Session ID
      type: object
      required:
        - evalSetFile
        - evalSetId
        - evalId
        - finalEvalStatus
        - overallEvalMetricResults
        - evalMetricResultPerInvocation
        - userId
        - sessionId
      title: RunEvalResult
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
              - type: string
              - type: integer
          type: array
          title: Location
          description: Location of the invalid field
        msg:
          type: string
          title: Message
          description: Validation error message
        type:
          type: string
          title: Error Type
        input:
          title: Input
        ctx:
          type: object
          title: Context
      type: object
      required:
        - loc
        - msg
        - type
      title: ValidationError
    BaseCriterion:
      properties:
        threshold:
          type: number
          title: Threshold
        includeIntermediateResponsesInFinal:
          type: boolean
          title: Includeintermediateresponsesinfinal
          default: false
      additionalProperties: true
      type: object
      required:
        - threshold
      title: BaseCriterion
    EvalStatus:
      type: integer
      enum:
        - 1
        - 2
        - 3
      title: EvalStatus
    EvalMetricResult:
      properties:
        metricName:
          type: string
          title: Metricname
          description: Evaluation metric name
        threshold:
          anyOf:
            - type: number
            - type: 'null'
          title: Threshold
        criterion:
          anyOf:
            - $ref: '#/components/schemas/BaseCriterion'
            - type: 'null'
          description: Evaluation criterion used by the metric
        customFunctionPath:
          anyOf:
            - type: string
            - type: 'null'
          title: Customfunctionpath
          description: Import path of a custom evaluation function
        score:
          anyOf:
            - type: number
            - type: 'null'
          title: Score
        evalStatus:
          $ref: '#/components/schemas/EvalStatus'
        details:
          $ref: '#/components/schemas/EvalMetricResultDetails'
      additionalProperties: false
      type: object
      required:
        - metricName
        - evalStatus
      title: EvalMetricResult
    EvalMetricResultPerInvocation:
      properties:
        actualInvocation:
          $ref: '#/components/schemas/Invocation-Output'
        expectedInvocation:
          anyOf:
            - $ref: '#/components/schemas/Invocation-Output'
            - type: 'null'
        evalMetricResults:
          items:
            $ref: '#/components/schemas/EvalMetricResult'
          type: array
          title: Evalmetricresults
          default: []
      additionalProperties: false
      type: object
      required:
        - actualInvocation
      title: EvalMetricResultPerInvocation
    EvalMetricResultDetails:
      properties:
        rubricScores:
          anyOf:
            - items:
                $ref: '#/components/schemas/RubricScore'
              type: array
            - type: 'null'
          title: Rubricscores
      additionalProperties: false
      type: object
      title: EvalMetricResultDetails
    Invocation-Output:
      properties:
        invocationId:
          type: string
          title: Invocationid
          default: ''
          description: Invocation ID
        userContent:
          $ref: '#/components/schemas/Content-Output'
        finalResponse:
          anyOf:
            - $ref: '#/components/schemas/Content-Output'
            - type: 'null'
        intermediateData:
          anyOf:
            - $ref: '#/components/schemas/IntermediateData-Output'
            - $ref: '#/components/schemas/InvocationEvents-Output'
            - type: 'null'
          title: Intermediatedata
        creationTimestamp:
          type: number
          title: Creationtimestamp
          default: 0
          description: Creation time as a Unix timestamp in seconds
        rubrics:
          anyOf:
            - items:
                $ref: '#/components/schemas/Rubric'
              type: array
            - type: 'null'
          title: Rubrics
        appDetails:
          anyOf:
            - $ref: '#/components/schemas/AppDetails'
            - type: 'null'
      additionalProperties: false
      type: object
      required:
        - userContent
      title: Invocation
    RubricScore:
      properties:
        rubricId:
          type: string
          title: Rubricid
        rationale:
          anyOf:
            - type: string
            - type: 'null'
          title: Rationale
        score:
          anyOf:
            - type: number
            - type: 'null'
          title: Score
      additionalProperties: false
      type: object
      required:
        - rubricId
      title: RubricScore
    Content-Output:
      properties:
        parts:
          anyOf:
            - items:
                $ref: '#/components/schemas/Part-Output'
              type: array
            - type: 'null'
          title: Parts
          description: Ordered message parts
        role:
          anyOf:
            - type: string
            - type: 'null'
          title: Role
          description: Message role, such as user or model
      additionalProperties: false
      type: object
      title: Content
    IntermediateData-Output:
      properties:
        toolUses:
          items:
            $ref: '#/components/schemas/FunctionCall'
          type: array
          title: Tooluses
          default: []
        toolResponses:
          items:
            $ref: '#/components/schemas/FunctionResponse-Output'
          type: array
          title: Toolresponses
          default: []
        intermediateResponses:
          items:
            prefixItems:
              - type: string
              - items:
                  $ref: '#/components/schemas/Part-Output'
                type: array
            type: array
            maxItems: 2
            minItems: 2
          type: array
          title: Intermediateresponses
          default: []
      additionalProperties: false
      type: object
      title: IntermediateData
    InvocationEvents-Output:
      properties:
        invocationEvents:
          items:
            $ref: '#/components/schemas/InvocationEvent-Output'
          type: array
          title: Invocationevents
      additionalProperties: false
      type: object
      title: InvocationEvents
    Rubric:
      properties:
        rubricId:
          type: string
          title: Rubricid
        rubricContent:
          $ref: '#/components/schemas/RubricContent'
        description:
          anyOf:
            - type: string
            - type: 'null'
          title: Description
          description: Description
        type:
          anyOf:
            - type: string
            - type: 'null'
          title: Type
      additionalProperties: false
      type: object
      required:
        - rubricId
        - rubricContent
      title: Rubric
    AppDetails:
      properties:
        agentDetails:
          additionalProperties:
            $ref: '#/components/schemas/AgentDetails'
          type: object
          title: Agentdetails
      additionalProperties: false
      type: object
      title: AppDetails
    Part-Output:
      properties:
        mediaResolution:
          anyOf:
            - $ref: '#/components/schemas/PartMediaResolution'
            - type: 'null'
        codeExecutionResult:
          anyOf:
            - $ref: '#/components/schemas/CodeExecutionResult'
            - type: 'null'
          description: Code execution result
        executableCode:
          anyOf:
            - $ref: '#/components/schemas/ExecutableCode'
            - type: 'null'
          description: Executable code
        fileData:
          anyOf:
            - $ref: '#/components/schemas/FileData'
            - type: 'null'
          description: File content referenced by URI
        functionCall:
          anyOf:
            - $ref: '#/components/schemas/FunctionCall'
            - type: 'null'
          description: Function call requested by the model
        functionResponse:
          anyOf:
            - $ref: '#/components/schemas/FunctionResponse-Output'
            - type: 'null'
          description: Function execution result
        inlineData:
          anyOf:
            - $ref: '#/components/schemas/Blob'
            - type: 'null'
          description: Inline binary content with Base64-encoded data
        text:
          anyOf:
            - type: string
            - type: 'null'
          title: Text
          description: Text content
        thought:
          anyOf:
            - type: boolean
            - type: 'null'
          title: Thought
          description: Whether this is thought content
        thoughtSignature:
          anyOf:
            - type: string
              contentEncoding: base64
              contentMediaType: application/octet-stream
            - type: 'null'
          title: Thoughtsignature
          description: Thought signature for subsequent requests
        videoMetadata:
          anyOf:
            - $ref: '#/components/schemas/VideoMetadata'
            - type: 'null'
          description: Video metadata
        toolCall:
          anyOf:
            - $ref: '#/components/schemas/ToolCall'
            - type: 'null'
        toolResponse:
          anyOf:
            - $ref: '#/components/schemas/ToolResponse'
            - type: 'null'
        partMetadata:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Partmetadata
      additionalProperties: false
      type: object
      title: Part
    FunctionCall:
      properties:
        id:
          anyOf:
            - type: string
            - type: 'null'
          title: Id
          description: Identifier
        args:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Args
          description: Function arguments
        name:
          anyOf:
            - type: string
            - type: 'null'
          title: Name
          description: Name
        partialArgs:
          anyOf:
            - items:
                $ref: '#/components/schemas/PartialArg'
              type: array
            - type: 'null'
          title: Partialargs
        willContinue:
          anyOf:
            - type: boolean
            - type: 'null'
          title: Willcontinue
      additionalProperties: false
      type: object
      title: FunctionCall
    FunctionResponse-Output:
      properties:
        willContinue:
          anyOf:
            - type: boolean
            - type: 'null'
          title: Willcontinue
        scheduling:
          anyOf:
            - $ref: '#/components/schemas/FunctionResponseScheduling'
            - type: 'null'
        parts:
          anyOf:
            - items:
                $ref: '#/components/schemas/FunctionResponsePart'
              type: array
            - type: 'null'
          title: Parts
          description: Ordered message parts
        id:
          anyOf:
            - type: string
            - type: 'null'
          title: Id
          description: Identifier
        name:
          anyOf:
            - type: string
            - type: 'null'
          title: Name
          description: Name
        response:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Response
          description: Response content
      additionalProperties: false
      type: object
      title: FunctionResponse
    InvocationEvent-Output:
      properties:
        author:
          type: string
          title: Author
          description: Event author, usually user or an agent name
        content:
          anyOf:
            - $ref: '#/components/schemas/Content-Output'
            - type: 'null'
          description: Message content
      additionalProperties: false
      type: object
      required:
        - author
        - content
      title: InvocationEvent
    RubricContent:
      properties:
        textProperty:
          anyOf:
            - type: string
            - type: 'null'
          title: Textproperty
      additionalProperties: false
      type: object
      required:
        - textProperty
      title: RubricContent
    AgentDetails:
      properties:
        name:
          type: string
          title: Name
          description: Name
        instructions:
          type: string
          title: Instructions
          default: ''
        toolDeclarations:
          items: {}
          type: array
          title: Tooldeclarations
      additionalProperties: false
      type: object
      required:
        - name
      title: AgentDetails
    PartMediaResolution:
      properties:
        level:
          anyOf:
            - $ref: '#/components/schemas/PartMediaResolutionLevel'
            - type: 'null'
        numTokens:
          anyOf:
            - type: integer
            - type: 'null'
          title: Numtokens
      additionalProperties: false
      type: object
      title: PartMediaResolution
    CodeExecutionResult:
      properties:
        outcome:
          anyOf:
            - $ref: '#/components/schemas/Outcome'
            - type: 'null'
        output:
          anyOf:
            - type: string
            - type: 'null'
          title: Output
        id:
          anyOf:
            - type: string
            - type: 'null'
          title: Id
          description: Identifier
      additionalProperties: false
      type: object
      title: CodeExecutionResult
    ExecutableCode:
      properties:
        code:
          anyOf:
            - type: string
            - type: 'null'
          title: Code
        language:
          anyOf:
            - $ref: '#/components/schemas/Language'
            - type: 'null'
          description: Application definition language
        id:
          anyOf:
            - type: string
            - type: 'null'
          title: Id
          description: Identifier
      additionalProperties: false
      type: object
      title: ExecutableCode
    FileData:
      properties:
        displayName:
          anyOf:
            - type: string
            - type: 'null'
          title: Displayname
        fileUri:
          anyOf:
            - type: string
            - type: 'null'
          title: Fileuri
          description: File URI
        mimeType:
          anyOf:
            - type: string
            - type: 'null'
          title: Mimetype
          description: Content MIME type
      additionalProperties: false
      type: object
      title: FileData
    Blob:
      properties:
        data:
          anyOf:
            - type: string
              contentEncoding: base64
              contentMediaType: application/octet-stream
            - type: 'null'
          title: Data
        displayName:
          anyOf:
            - type: string
            - type: 'null'
          title: Displayname
        mimeType:
          anyOf:
            - type: string
            - type: 'null'
          title: Mimetype
          description: Content MIME type
      additionalProperties: false
      type: object
      title: Blob
    VideoMetadata:
      properties:
        endOffset:
          anyOf:
            - type: string
            - type: 'null'
          title: Endoffset
        fps:
          anyOf:
            - type: number
            - type: 'null'
          title: Fps
        startOffset:
          anyOf:
            - type: string
            - type: 'null'
          title: Startoffset
      additionalProperties: false
      type: object
      title: VideoMetadata
    ToolCall:
      properties:
        id:
          anyOf:
            - type: string
            - type: 'null'
          title: Id
          description: Identifier
        toolType:
          anyOf:
            - $ref: '#/components/schemas/ToolType'
            - type: 'null'
        args:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Args
          description: Function arguments
      additionalProperties: false
      type: object
      title: ToolCall
    ToolResponse:
      properties:
        id:
          anyOf:
            - type: string
            - type: 'null'
          title: Id
          description: Identifier
        toolType:
          anyOf:
            - $ref: '#/components/schemas/ToolType'
            - type: 'null'
        response:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Response
          description: Response content
      additionalProperties: false
      type: object
      title: ToolResponse
    PartialArg:
      properties:
        boolValue:
          anyOf:
            - type: boolean
            - type: 'null'
          title: Boolvalue
        jsonPath:
          anyOf:
            - type: string
            - type: 'null'
          title: Jsonpath
        nullValue:
          anyOf:
            - type: string
              const: NULL_VALUE
            - type: 'null'
          title: Nullvalue
        numberValue:
          anyOf:
            - type: number
            - type: 'null'
          title: Numbervalue
        stringValue:
          anyOf:
            - type: string
            - type: 'null'
          title: Stringvalue
        willContinue:
          anyOf:
            - type: boolean
            - type: 'null'
          title: Willcontinue
      additionalProperties: false
      type: object
      title: PartialArg
    FunctionResponseScheduling:
      type: string
      enum:
        - SCHEDULING_UNSPECIFIED
        - SILENT
        - WHEN_IDLE
        - INTERRUPT
      title: FunctionResponseScheduling
    FunctionResponsePart:
      properties:
        inlineData:
          anyOf:
            - $ref: '#/components/schemas/FunctionResponseBlob'
            - type: 'null'
          description: Inline binary content with Base64-encoded data
        fileData:
          anyOf:
            - $ref: '#/components/schemas/FunctionResponseFileData'
            - type: 'null'
          description: File content referenced by URI
      additionalProperties: false
      type: object
      title: FunctionResponsePart
    PartMediaResolutionLevel:
      type: string
      enum:
        - MEDIA_RESOLUTION_UNSPECIFIED
        - MEDIA_RESOLUTION_LOW
        - MEDIA_RESOLUTION_MEDIUM
        - MEDIA_RESOLUTION_HIGH
        - MEDIA_RESOLUTION_ULTRA_HIGH
      title: PartMediaResolutionLevel
    Outcome:
      type: string
      enum:
        - OUTCOME_UNSPECIFIED
        - OUTCOME_OK
        - OUTCOME_FAILED
        - OUTCOME_DEADLINE_EXCEEDED
      title: Outcome
    Language:
      type: string
      enum:
        - LANGUAGE_UNSPECIFIED
        - PYTHON
      title: Language
    ToolType:
      type: string
      enum:
        - TOOL_TYPE_UNSPECIFIED
        - GOOGLE_SEARCH_WEB
        - GOOGLE_SEARCH_IMAGE
        - URL_CONTEXT
        - GOOGLE_MAPS
        - FILE_SEARCH
      title: ToolType
    FunctionResponseBlob:
      properties:
        mimeType:
          anyOf:
            - type: string
            - type: 'null'
          title: Mimetype
          description: Content MIME type
        data:
          anyOf:
            - type: string
              contentEncoding: base64
              contentMediaType: application/octet-stream
            - type: 'null'
          title: Data
        displayName:
          anyOf:
            - type: string
            - type: 'null'
          title: Displayname
      additionalProperties: false
      type: object
      title: FunctionResponseBlob
    FunctionResponseFileData:
      properties:
        fileUri:
          anyOf:
            - type: string
            - type: 'null'
          title: Fileuri
          description: File URI
        mimeType:
          anyOf:
            - type: string
            - type: 'null'
          title: Mimetype
          description: Content MIME type
        displayName:
          anyOf:
            - type: string
            - type: 'null'
          title: Displayname
      additionalProperties: false
      type: object
      title: FunctionResponseFileData
  securitySchemes:
    RuntimeBearer:
      type: http
      scheme: bearer
      description: >-
        Optional locally without a gateway; cloud deployments use the Runtime
        API key or user-pool JWT required by that deployment, never the model
        API key

````