Observability

VeADK provides application logs and traces to diagnose model calls, tools, and agent execution. Logs record individual events; traces connect operations within a request so you can inspect timing and relationships Tracing uses OpenTelemetry spans and generative AI attributes. Coverage depends on the instrumentation provided by the runtime and tools. External backends also require compatible protocols, endpoints, and credentials See Application logging for log levels and formatting

Core concepts

span

A span is a traceable unit of execution that records a name, start/end times, attributes, and parent-child relationships. VeADK builds a tree of spans for each request, covering nodes such as agent runs, model calls, and tool executions, and annotates them per the generative-AI semantic conventions with attributes like gen_ai.operation.name and gen_ai.span.kind.

The tracer `OpentelemetryTracer`

OpentelemetryTracer is the unified tracing entry point. It holds a list of exporters, wires each one into the tracing pipeline on initialization, and automatically attaches an in-memory exporter for local retention and dumping.

exporter

An exporter sends span data to a specific backend platform. Each exporter targets one backend and can be used alone or combined. VeADK ships the following exporters:

Exporter	Class	Target platform	Docs
APMPlus	`APMPlusExporter`	Volcengine APMPlus, traces and metrics	APMPlus
Cozeloop	`CozeloopExporter`	Cozeloop, trace observation and evaluation	Cozeloop
TLS	`TLSExporter`	Volcengine TLS log service	TLS
In-memory	`InMemoryExporter`	In-process memory, local debugging and dumping	In-memory exporter

Traces may contain user input, model responses, and tool data. Before enabling external exporters, confirm the destination, access controls, and retention policy. force_export() requests a flush; it does not prove that the backend received the data

Attaching a tracer to an agent

Complete model configuration first. This example stores traces locally and does not require an external observability backend. Running it prints a model response and the path to a JSON trace file

app.py

import asyncio
from pathlib import Path

from veadk import Agent, Runner
from veadk.tracing.telemetry.opentelemetry_tracer import OpentelemetryTracer

tracer = OpentelemetryTracer()
agent = Agent(name="trace_demo", tracers=[tracer])
runner = Runner(agent=agent, app_name="trace_demo", user_id="demo-user")
response = asyncio.run(
    runner.run("Explain the purpose of an agent in one sentence", session_id="demo-session")
)
print(response)
Path("traces").mkdir(exist_ok=True)
path = tracer.dump(user_id="demo-user", session_id="demo-session", path="traces")
print(path)

After configuring each backend’s endpoint, credentials, and target resource, one tracer can report to multiple platforms. This configuration fragment replaces the tracer above:

from veadk.tracing.telemetry.exporters.apmplus_exporter import APMPlusExporter
from veadk.tracing.telemetry.exporters.cozeloop_exporter import CozeloopExporter
from veadk.tracing.telemetry.exporters.tls_exporter import TLSExporter
from veadk.tracing.telemetry.opentelemetry_tracer import OpentelemetryTracer

tracer = OpentelemetryTracer(
    exporters=[APMPlusExporter(), CozeloopExporter(), TLSExporter()]
)

Tracer parameters

OpentelemetryTracer accepts the following fields:

Parameter	Type	Default	Description
`name`	`str`	`veadk_opentelemetry_tracer`	Tracer identifier, used for logging and naming dump files.
`exporters`	`list[BaseExporter]`	`[]`	List of exporters. `InMemoryExporter` may not be added explicitly, or validation fails; the in-memory exporter is attached automatically.
`apmplus_managed_externally`	`bool`	—	Read-only property. Whether a global `TracerProvider` already existed at initialization. When `True`, VeADK reuses the external provider instead of overriding it and skips registering the APMPlus span processor; this does not establish that the external provider reports to APMPlus.

InMemoryExporter is managed automatically by OpentelemetryTracer and retains completed, sampled spans. Do not add it to the exporters list manually, or initialization fails.

Reusing a preconfigured global TracerProvider

By default, OpentelemetryTracer creates and sets the global TracerProvider during initialization. If a global TracerProvider already exists at that point (for example, set by another library or the application itself), VeADK reuses it instead of overriding it. When a global provider already exists, the APMPlusExporter object remains in the exporter list, but its span processor is not registered by this tracer. Cozeloop, TLS, and in-memory exporters are still attached. This also applies to exporters created with ENABLE_APMPLUS=true apmplus_managed_externally only indicates that a provider already existed at initialization. It does not inspect that provider’s destination or confirm delivery to APMPlus. If your application already configures OpenTelemetry, configure the APMPlus destination there and verify the Trace ID in the backend

Enabling exporters via environment variables

Besides constructing exporters explicitly, Agent attaches the matching exporter at runtime based on the following environment variables (enabled when set to true). If the agent provides no tracers, an OpentelemetryTracer is created automatically:

Environment variable	Exporter enabled
`ENABLE_APMPLUS`	`APMPlusExporter`
`ENABLE_COZELOOP`	`CozeloopExporter`
`ENABLE_TLS`	`TLSExporter`

When all three variables are false (the default), Agent creates no tracer. To trace in that case, construct an OpentelemetryTracer explicitly and pass it via tracers.

Content tracing

By default, spans record the input and output content of the agent, model, and tools. In scenarios involving sensitive data, you can disable content capture and reduce content captured by VeADK. This is controlled by OpenTelemetryConfig.trace_content.

Config	Environment variable	Type	Default	Description
`trace_content`	`OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT`	`bool`	`True`	Whether to write prompt, completion, and tool input/output content into spans. Set to `false` to disable these content fields; configure third-party instrumentation and application logs separately.

# Disable content capture
export OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT=false

Local dumping

The in-memory exporter built into OpentelemetryTracer retains spans by session_id. Call dump to export the current session’s spans to a local JSON file for offline analysis:

path = tracer.dump(user_id="demo-user", session_id="demo-session", path="traces")
print(f"trace written to {path}")

The exported JSON contains, for each span, its name, span_id, trace_id, start/end times, attributes, and parent span. See In-memory exporter for details.

Overview

Get started

Core components

Extensions

Frontend

Security

Deployment

Observability

Core concepts

span

The tracer `OpentelemetryTracer`

exporter

Attaching a tracer to an agent

Tracer parameters

Reusing a preconfigured global TracerProvider

Enabling exporters via environment variables

Content tracing

Local dumping

​Core concepts

​span

​The tracer OpentelemetryTracer

​exporter

​Attaching a tracer to an agent

​Tracer parameters

​Reusing a preconfigured global TracerProvider

​Enabling exporters via environment variables

​Content tracing

​Local dumping

Core concepts

span

The tracer `OpentelemetryTracer`

exporter

Attaching a tracer to an agent

Tracer parameters

Reusing a preconfigured global TracerProvider

Enabling exporters via environment variables

Content tracing

Local dumping