Agent Tracing and Evaluation Setup

Capture structured traces of every LLM and tool call and run a small regression eval suite on each change.

What to log per span

span:
  trace_id: uuid
  kind: llm | tool | retrieval
  name: model or tool name
  input: redacted prompt/args
  output: redacted response/result
  tokens: {in: int, out: int}
  latency_ms: int
  error: string|null
  prompt_version: git sha or tag

Steps

1. Wrap your LLM client and tool dispatcher to emit spans (OpenTelemetry GenAI semantic conventions, or a tool such as Langfuse or Arize Phoenix, both of which can be self-hosted).
2. Redact secrets and personal data before export.
3. Build a golden set of 20-100 real tasks with expected outcomes.
4. On every prompt/model change, run the set and compare pass rate, cost, and latency against the previous version.
5. Promote failing production traces into the golden set.

Failure modes

• Logging full prompts can store PII or secrets; redact before export.
• Eval sets that never change stop catching regressions; add real failures to them.
• High-volume tracing adds latency and cost if exported synchronously.

Markdown version: /blueprints/agent-observability.md

Related blueprints