Observability for AI Systems

Your LLM is behaving oddly in production. You have no idea why. Here's how to instrument it.

What to Log

Every API call:

• Input (prompt/user query)
• Model used and parameters
• Latency
• Cost (tokens used)
• Output (full LLM response)
• Timestamp and user ID

Retrieval metrics (if using RAG):

• Documents retrieved and their scores
• Chunk text and source
• Retrieval latency

User feedback:

• Was the answer helpful? (binary or rating)
• Did the user flag incorrect information?
• Time spent on the page

System metrics:

• Error rates
• API quota usage
• Cache hit rates
• Queue depth

The Monitoring Dashboard

Create alerts for:

• High error rate (sudden spikes)
• Cost spikes (unusual token usage)
• Latency increases
• Degraded answer quality (user feedback score drop)

Real-time dashboards matter less than alerts. You can't watch 24/7, but you need to know when something breaks.

Debugging Process

When something breaks:

1. Check logs. Find the failed request's exact prompt, model used, tokens consumed.
2. Reproduce offline. Call the API with the same prompt and see if you get the same output.
3. Vary inputs. Try similar prompts. Is it specific to one input or systematic?
4. Check the retrieval. If using RAG, did you get the right documents? That's usually the problem.
5. Examine the prompt. Did something change in the system prompt or example?

The same debugging discipline you use for code applies to LLM systems. Logs are your debugger.

The Difference from Regular Services

Traditional services: broken = error thrown. LLM services: broken = wrong answer, returned confidently.

You won't get exceptions. You'll get silent failures masquerading as correct answers. Logging and feedback loops are your only defenses.