Two days, built for engineers.

Debugging Intelligence: How Do You Debug a System That Is Thinking?

Nishant Gupta

Traditional debugging assumes deterministic software: given the same input and state, we expect the same behavior. AI systems break that assumption. A production AI application can return HTTP 200, meet every latency SLO, successfully execute every tool call and still be completely wrong. As AI evolves from models that generate responses into agents that plan, retrieve context, invoke tools, maintain state, and execute long-running workflows, debugging must evolve with it. The failure may no longer be a bad line of code. It could be missing context, an incorrect plan, a stale memory, a failed retrieval, an inappropriate tool call, an orchestration issue, or a sequence of individually reasonable decisions that collectively produce the wrong outcome. In this talk, we'll explore a systems approach to debugging non-deterministic AI. We'll examine how distributed tracing, execution trajectories, context inspection, tool-call telemetry, replay, failure attribution, and online and offline evaluations can help engineers understand why an AI system behaved the way it did. We'll introduce a practical debugging loop: Observe → Evaluate → Diagnose → Replay → Verify and show how engineering teams can move beyond monitoring infrastructure health toward diagnosing behavioral correctness. Attendees will leave with a framework for debugging production AI systems where the most important question is no longer simply, "Which line of code failed?" but "Which decision caused the system to diverge from the desired outcome?"