LLM Tool Call Trace Debugging Runbook Builder

Create a debugging runbook for AI apps where tool calls, traces, permissions, and model steps need root cause analysis.

Prompt Template

You are a senior AI application debugging lead. Build a practical runbook for diagnosing failures in an LLM-powered app that uses tool calls.

Application type: [chat app, agent workflow, support copilot, coding agent, internal automation]
Model and orchestration stack: [model provider, framework, agent runtime, prompt layer]
Tools available: [browser, database, CRM, Slack, filesystem, code execution, internal API, retrieval]
Observed failure: [wrong tool called, missing tool call, bad arguments, permission denial, timeout, stale result, hallucinated action]
Expected behavior: [what should have happened]
Trace data available: [messages, tool arguments, tool outputs, spans, logs, evals, request IDs]
Recent changes: [prompt, model, tool schema, permissions, retrieval, deployment, feature flag]
Environment: [local, staging, production, tenant, user role]
Safety constraints: [PII, secrets, destructive tools, customer data, audit requirements]
Reproduction status: [reproducible, intermittent, only in production, unknown]
Team tools: [OpenTelemetry, LangSmith, Datadog, Honeycomb, custom logs, spreadsheets]

Create:
1. Triage checklist that separates model reasoning issues, schema issues, tool runtime failures, permission failures, and stale context.
2. Trace fields to inspect, including request ID, conversation state, tool name, arguments, auth context, latency, retries, and output summary.
3. Reproduction plan using sanitized fixtures and deterministic test cases where possible.
4. Root-cause decision tree for wrong tool selection, malformed arguments, missing calls, duplicate calls, and unsafe calls.
5. Instrumentation gap checklist for spans, redacted payloads, user role, policy decision, and tool result status.
6. Safe replay workflow that avoids re-sending emails, charging cards, deleting data, or mutating production systems.
7. Regression test plan with golden traces, negative tests, prompt changes, and tool schema versioning.
8. Dashboard and alert recommendations for tool error rate, denial rate, retry loops, latency, and unexpected tool usage.
9. Incident handoff template for engineering, product, security, and support.
10. Fix validation checklist before rollout.

Be concrete. Bias toward steps that help an engineer find the failing boundary quickly instead of guessing from chat transcripts alone.

Example Output

# Tool Call Debugging Runbook

Failure Summary

Issue: The agent called update_contact with a stale account ID after retrieval returned two similarly named companies.

Expected: Ask a clarifying question or select the account from the active CRM context.

Trace Checks

| Field | Finding | Next Step |

|---|---|---|

| tool.name | update_contact | Confirm this tool was allowed for user role |

| tool.args.account_id | acc_184 | Compare against retrieved account candidates |

| policy.decision | allow | Verify tenant and role scope |

| retrieval.top_k | Two near-duplicate accounts | Add disambiguation test case |

Safe Replay

Replay the conversation in staging with mutation tools replaced by dry-run mocks. Assert that the model either asks for clarification or calls update_contact_dry_run with the active account ID.

Fix Validation

Add a schema description requiring active account confirmation when multiple accounts match, then add a golden trace regression test for duplicate company names.