Distributed Tracing & Observability

Agent Substrate has first-class support for the W3C Distributed Tracing standard and exports native OpenTelemetry spans directly to Datadog, Honeycomb, Grafana, or Jaeger.

Span Hierarchy

Every agent turn emits a clean hierarchical trace tree:

• agent.orchestrator.execute (Root Span)
  • wal.journal.commit (Time to persist turn to PostgreSQL)
  • tool.mcp.postgres_query (Latency, query arguments, return row count)
  • llm.claude-3-7-sonnet.stream (Prompt tokens, completion tokens, time to first token)
  • guardrail.safety_filter (PII checks, injection detection latency)

Configuring OTel Export

Configure your exporter with a single call:

from substrate.infrastructure.observability import configure_otel

configure_otel(
    service_name="prod-agent-substrate",
    otlp_endpoint="http://otel-collector.monitoring:4317",
    sample_rate=1.0  # 100% of production runs captured
)