> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veadk.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

Observability records the full execution path of an agent so you can understand its internal behavior, debug issues, analyze performance, and keep optimizing. VeADK's built-in tracing captures each request — from receiving user input, through model inference, tool calls, memory and knowledge-base reads/writes, to producing a response — as structured span data, and reports it to Volcengine or third-party platforms through exporters.

VeADK tracing follows the [OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/) generative-AI semantic conventions, with field names aligned to standard span attributes. Trace data can therefore be imported directly into any OpenTelemetry-compatible system for analysis and visualization.

Runtime logs use Python's standard `logging` package. See [Application logging](/productions/veadk/preview/en/components/observability/logging) for levels, formatting, and trace correlation.

## Core concepts

### span

A span is a traceable unit of execution that records a name, start/end times, attributes, and parent-child relationships. VeADK builds a tree of spans for each request, covering nodes such as agent runs, model calls, and tool executions, and annotates them per the generative-AI semantic conventions with attributes like `gen_ai.operation.name` and `gen_ai.span.kind`.

### The tracer `OpentelemetryTracer`

`OpentelemetryTracer` is the unified tracing entry point. It holds a list of exporters, wires each one into the tracing pipeline on initialization, and automatically attaches an in-memory exporter for local retention and dumping.

### exporter

An exporter sends span data to a specific backend platform. Each exporter targets one backend and can be used alone or combined. VeADK ships the following exporters:

| Exporter | Class | Target platform | Docs |
| :- | :- | :- | :- |
| APMPlus | `APMPlusExporter` | Volcengine APMPlus, traces and metrics | [APMPlus](/productions/veadk/preview/en/components/observability/apmplus) |
| Cozeloop | `CozeloopExporter` | Cozeloop, trace observation and evaluation | [Cozeloop](/productions/veadk/preview/en/components/observability/cozeloop) |
| TLS | `TLSExporter` | Volcengine TLS log service | [TLS](/productions/veadk/preview/en/components/observability/tls) |
| In-memory | `InMemoryExporter` | In-process memory, local debugging and dumping | [In-memory exporter](/productions/veadk/preview/en/components/observability/inmemory) |

## Attaching a tracer to an agent

Tracing is wired in through the `tracers` field of `Agent`. Construct an exporter for the target backend, hand it to `OpentelemetryTracer`, then attach the tracer to the agent:

```python lines theme={null}
import asyncio

from veadk import Agent, Runner
from veadk.memory.short_term_memory import ShortTermMemory
from veadk.tools.demo_tools import get_city_weather
from veadk.tracing.telemetry.exporters.apmplus_exporter import APMPlusExporter
from veadk.tracing.telemetry.opentelemetry_tracer import OpentelemetryTracer

exporters = [APMPlusExporter()]
tracer = OpentelemetryTracer(exporters=exporters)

agent = Agent(tools=[get_city_weather], tracers=[tracer])

runner = Runner(agent=agent, short_term_memory=ShortTermMemory())

asyncio.run(runner.run(messages="How is the weather in Beijing?", session_id="session_id_demo"))
```

A single tracer can hold multiple exporters, reporting the same spans to several platforms:

```python lines theme={null}
from veadk.tracing.telemetry.exporters.apmplus_exporter import APMPlusExporter
from veadk.tracing.telemetry.exporters.cozeloop_exporter import CozeloopExporter
from veadk.tracing.telemetry.exporters.tls_exporter import TLSExporter
from veadk.tracing.telemetry.opentelemetry_tracer import OpentelemetryTracer

tracer = OpentelemetryTracer(
    exporters=[APMPlusExporter(), CozeloopExporter(), TLSExporter()]
)
```

### Tracer parameters

`OpentelemetryTracer` accepts the following fields:

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `name` | `str` | `veadk_opentelemetry_tracer` | Tracer identifier, used for logging and naming dump files. |
| `exporters` | `list[BaseExporter]` | `[]` | List of exporters. `InMemoryExporter` may not be added explicitly, or validation fails; the in-memory exporter is attached automatically. |
| `apmplus_managed_externally` | `bool` | — | Read-only property. Whether a global `TracerProvider` already existed at initialization. When `True`, VeADK reuses the external provider instead of overriding it and automatically removes `APMPlusExporter`. |

<Note>
  `InMemoryExporter` is managed automatically by `OpentelemetryTracer` and records every span. Do not add it to the `exporters` list manually, or initialization fails.
</Note>

### Reusing a preconfigured global TracerProvider

By default, `OpentelemetryTracer` creates and sets the global `TracerProvider` during initialization. If a global `TracerProvider` already exists at that point (for example, set by another library or the application itself), VeADK reuses it instead of overriding it.

In this case VeADK automatically removes `APMPlusExporter` from the exporters list, because the existing provider is assumed to be responsible for APMPlus trace reporting. Other exporters (such as `CozeloopExporter` and `TLSExporter`) are still registered with that provider. The VeADK in-memory exporter is also attached as usual.

<Note>
  This behavior also applies when `ENABLE_APMPLUS=true` auto-creates the exporter: if a global `TracerProvider` already exists, the auto-created `APMPlusExporter` is also removed.
</Note>

You can check the `apmplus_managed_externally` property to determine whether the tracer detected an external provider:

```python lines theme={null}
tracer = OpentelemetryTracer(exporters=[APMPlusExporter()])
if tracer.apmplus_managed_externally:
    # VeADK reused an existing global TracerProvider; APMPlusExporter was removed
    pass
```

## Enabling exporters via environment variables

Besides constructing exporters explicitly, `Agent` attaches the matching exporter at runtime based on the following environment variables (enabled when set to `true`). If the agent provides no `tracers`, an `OpentelemetryTracer` is created automatically:

| Environment variable | Exporter enabled |
| :- | :- |
| `ENABLE_APMPLUS` | `APMPlusExporter` |
| `ENABLE_COZELOOP` | `CozeloopExporter` |
| `ENABLE_TLS` | `TLSExporter` |

<Tip>
  When all three variables are `false` (the default), `Agent` creates no tracer. To trace in that case, construct an `OpentelemetryTracer` explicitly and pass it via `tracers`.
</Tip>

## Content tracing

By default, spans record the input and output content of the agent, model, and tools. In scenarios involving sensitive data, you can disable content capture and keep only the non-content span structure and timing. This is controlled by `OpenTelemetryConfig.trace_content`.

| Config | Environment variable | Type | Default | Description |
| :- | :- | :- | :- | :- |
| `trace_content` | `OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT` | `bool` | `True` | Whether to write prompt, completion, and tool input/output content into spans. Set to `false` to keep only non-content trace information. |

```bash lines theme={null}
# Disable content capture
export OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT=false
```

## Local dumping

The in-memory exporter built into `OpentelemetryTracer` retains spans by `session_id`. Call `dump` to export the current session's spans to a local JSON file for offline analysis:

```python lines theme={null}
path = tracer.dump(user_id="user-1", session_id="session_id_demo")
print(f"trace written to {path}")
```

The exported JSON contains, for each span, its name, `span_id`, `trace_id`, start/end times, attributes, and parent span. See [In-memory exporter](/productions/veadk/preview/en/components/observability/inmemory) for details.