Home
cd ../playbooks
Developer ToolsIntermediate

LLM Error Analysis

Build evaluators grounded in how your LLM system actually fails, by reading ~100 traces and letting failure categories emerge instead of borrowing generic ones — a 7-step process (collect, read/note, group, label, rate, decide, iterate) that notes the first thing that went wrong, fixes obvious prompt/tool/code gaps before building any evaluator, and prioritizes by real failure frequency.

15 minutes
By Hamel Husain (hamelsmu)Source
#llm-evals#ai-engineering#error-analysis#testing#machine-learning#observability

Most LLM eval projects start by scoring 'helpfulness' and 'coherence' — borrowed research metrics that look rigorous and measure nothing your product actually gets wrong. Error analysis flips it: read your real traces first, let the failure categories emerge from what you see, and most of the top failures turn out to have a one-line fix that needs no evaluator at all.

Who it's for: AI engineers starting an eval project and unsure where to begin, teams whose LLM product fails in ways their metrics don't capture, developers who want failure categories grounded in real traces instead of generic scores, anyone re-establishing evals after a model switch, prompt rewrite, or production incident

Example

"Help me figure out how our LLM agent actually fails" → ~100 full traces collected and read one at a time, each judged pass/fail with a note on the first thing that went wrong, similar notes grouped into 5-10 grounded, actionable categories (Missing Query Constraints, Wrong Tone, Fabricated Facts), failure rates computed to rank them, and the top categories triaged — obvious prompt/tool/code gaps fixed directly, evaluators reserved only for failures worth iterating on

CLAUDE.md Template

New here? 3-minute setup guide → | Already set up? Copy the template below.

Or install from your terminal:
npx ccplaybooks add llm-error-analysis
# LLM Error Analysis

Systematically identify and categorize the failure modes of an LLM pipeline by reading its traces, so you build evaluators grounded in how the system actually fails rather than in borrowed, generic metrics. Use when starting a new eval project, after a significant pipeline change (new features, model switch, prompt rewrite), when production metrics drop, or after an incident.

## Overview

1. Collect ~100 representative traces.
2. Read each one, judge pass/fail, and note what went wrong.
3. Group similar failures into categories.
4. Label every trace against those categories.
5. Compute failure rates to decide what to fix first.

Get new playbooks like this one

One email a week with new Claude Code workflows. Free, like everything here.

No spam. Unsubscribe anytime.

README.md

What This Does

The foundational error-analysis methodology from Hamel Husain's eval work — the step that should come before any evaluator, judge, or dashboard gets built. Rather than borrowing generic quality metrics ("helpfulness", "hallucination", "coherence") that look rigorous and miss what a product actually gets wrong, it has you read ~100 real (or synthetic) traces, judge each pass/fail, and note the first thing that went wrong (errors cascade, so fixing the root cause makes the downstream symptoms disappear). Failure categories are allowed to emerge from what you observe instead of being imposed from a pre-defined list, which is the difference between evaluators that catch real problems and ones that score well on paper.

The seven steps run from trace collection (full traces, with diversity via embedding clustering) through reading-and-noting, grouping into 5–10 distinct and actionable categories (with explicit split-vs-group heuristics and cautious LLM-assisted clustering only after a human has reviewed the first batch), binary labeling, and failure-rate computation to prioritize. The step that saves the most wasted effort is the triage: before building any evaluator, check whether a failure just needs an obvious fix — a missing prompt instruction, an unexposed tool, a plain code bug — and do that first, reserving evaluators for failures you'll actually iterate on repeatedly and preferring code-based checks over LLM judges for anything objective. It includes a trace-sampling strategy table and an anti-patterns list, and pairs directly with the companion eval-audit playbook.


Quick Start

Step 1: Create a Project Folder

mkdir error-analysis && cd error-analysis

Step 2: Download the Template

Click Download above, then:

mv ~/Downloads/CLAUDE.md ./

Step 3: Analyze Your Failures

claude

Bring a sample of your LLM pipeline's traces (or ask Claude to help generate synthetic ones if you have no production data). It will walk the seven steps with you — reading traces, surfacing grounded failure categories, labeling, and triaging what to fix directly vs. what warrants an evaluator.


Tips & Best Practices

  • Resist the urge to pre-define failure categories — reading traces first and letting categories emerge is the whole method, and starting from a fixed list bakes in confirmation bias that hides your real failures.
  • Note only the first thing that went wrong in each trace; chasing every downstream symptom wastes effort on issues that vanish once the root cause is fixed.
  • Expect many top failures to need no evaluator at all — a missing prompt instruction or an unexposed tool is a direct fix, and building a judge for it instead is exactly the wasted work this method is designed to prevent.

Limitations

  • It needs a human (ideally a domain expert) in the loop for the first 30–50 traces — this is deliberate, since grounding categories in domain knowledge is the point, but it means the method isn't fully automatable.
  • The ~100-trace saturation target scales with system complexity; a simple pipeline saturates sooner and a complex multi-agent one may need more, so treat the number as a heuristic, not a quota.
  • It produces grounded failure categories and a fix/evaluator triage — actually writing and validating the evaluators for the failures that warrant them is the next step, covered by the companion eval-audit playbook and judge-writing methodology.

$Related Playbooks

Developer Tools

LLM Eval Audit

Audit an existing LLM evaluation pipeline against six diagnostic areas — error analysis, evaluator design, judge validation, human review, labeled data, and pipeline hygiene — surfacing unvalidated judges, Likert-scale noise, vanity similarity metrics, and the accuracy-under-class-imbalance trap, as an impact-ordered findings report where every problem links to a concrete fix.

10 minutes
Advanced
Developer Tools

Long-Horizon Agent Prompting

Design the launch prompt for an agent working autonomously for hours or days — a pseudo-formal task brief with an exact success predicate, an enumerated non-counting-outcomes list, adversarial verification, and effort floors, because persistence pressure against a loose spec produces confident non-solutions.

10 minutes
Advanced
Developer Tools

Loopy: Build and Reuse Agent Loops

Mine your codebase and coding history for repeated work, turn it into bounded agent loops, and audit the loops you already have

10 minutes
Intermediate
Developer Tools

LLM Icon Finder

Find and download AI/LLM model brand icons from the lobe-icons library — Claude, GPT, Gemini, and more — as ready-to-use SVG, PNG, or WEBP URLs.

5 minutes
Beginner
Developer Tools

LLM Coding Guardrails

Four behavioral guidelines that reduce the most common Claude coding mistakes: silent assumptions, overcomplication, scope creep, and vague success criteria.

2 minutes
Beginner
Developer Tools

Multi-Stage Prompt Pipeline

A methodology for building Claude-to-Claude prompt pipelines where one stage's output feeds the next — a four-purpose taxonomy (Research/Plan/Do/Refine), per-prompt provenance folders, XML-structured outputs with required metadata tags, a fixed human-scannable SUMMARY.md, dependency-aware layered execution, and per-stage validation before anything chains forward.

10 minutes
Advanced
Developer Tools

Minimalist UI Design Skill: Editorial, Monochrome Interfaces

A frontend design system that enforces warm monochrome, bento grids, and editorial typography — the Notion/Linear look, banned from gradients, heavy shadows, and Inter

5 minutes
Beginner
Developer Tools

Mermaid Tools

Extract Mermaid diagrams from markdown files and render them to high-quality PNG images using bundled scripts — turning text diagrams into shareable visuals.

5 minutes
Beginner
Developer Tools

Minimalist MVP Scoping

Scope a weekend-shippable MVP using the manual → processized → productized progression. Build as little as possible; charge from day one.

5 minutes
Beginner
Developer Tools

MCP Server Hub Manager

Access 1200+ AI Agent tools via Model Context Protocol (MCP)

10 minutes
Advanced
Developer Tools

n8n Workflow Builder

Automate document workflows with n8n - 7800+ workflow templates

10 minutes
Advanced
Developer Tools

MCP Server Builder

Guide for creating high-quality Model Context Protocol (MCP) servers that enable LLMs to interact with external services and APIs.

30 minutes
Advanced

Browse all Developer Tools playbooks →