Evaluating LLM Outputs: Beyond Vibes

How do you know if your prompt changes actually worked? Here are evaluation frameworks.

The Evaluation Types

Retrieval metrics. Are you getting the right documents?

• Precision: Of retrieved docs, how many were relevant?
• Recall: Of all relevant docs, how many did you retrieve?
• NDCG: Ranking quality (top results most relevant)

Generation quality. Is the LLM output good?

• Exact match: Does it match expected output exactly?
• BLEU/ROUGE: String similarity to reference answers
• Semantic similarity: Do answers mean the same thing?

User-focused metrics. Does the user actually care?

• Satisfaction surveys: Simple thumbs up/down
• Task success: Did the answer actually help?
• Latency: How fast was it?

Implementation

For retrieval: Compare your results to ground truth annotations. Use open-source tools (pytrec-eval) to compute metrics automatically.

For generation: Set up a test set with expected outputs. Compute BLEU/ROUGE. Use sentence embedding similarity (all-MiniLM-L6-v2) to measure semantic closeness.

For user focus: Add "was this helpful?" buttons. Track task completion rates.

The Iteration Loop

1. Define metric that matters for your use case
2. Establish baseline (current system performance)
3. Make a change
4. Measure impact
5. Keep improvements, revert non-improvements
6. Repeat

Without this loop, you're optimizing for vibes, not quality.

Most prompt improvements fail this test. Measure before celebrating.