Home
cd ../playbooks
Developer ToolsAdvanced

LLM Eval Audit

Audit an existing LLM evaluation pipeline against six diagnostic areas — error analysis, evaluator design, judge validation, human review, labeled data, and pipeline hygiene — surfacing unvalidated judges, Likert-scale noise, vanity similarity metrics, and the accuracy-under-class-imbalance trap, as an impact-ordered findings report where every problem links to a concrete fix.

10 minutes
By Hamel Husain (hamelsmu)Source
#llm-evals#ai-engineering#llm-as-judge#testing#error-analysis#machine-learning

A production LLM judge that was never validated against human labels can report 90% accuracy while catching zero real failures — because when 90% of traces pass, a judge that blindly says 'Pass' scores 90% too, and raw accuracy will never reveal that the judge is useless.

Who it's for: ML and AI engineers who inherited an eval system and don't trust it, teams whose LLM product metrics look fine but quality complaints keep coming, anyone running LLM-as-judge evaluators without having validated them, engineers unsure whether their evals measure real failure modes or borrowed research metrics

Example

"We have evals but I don't trust them — audit the pipeline" → The eval artifacts inspected across six areas, surfacing the real problems (judges validated with raw accuracy instead of TPR/TNR, Likert scales no one can calibrate, ROUGE used as the primary quality metric, failure categories brainstormed rather than observed from traces), delivered as an impact-ordered report where each finding names the specific issue in THIS pipeline and links to its fix

CLAUDE.md Template

New here? 3-minute setup guide → | Already set up? Copy the template below.

Or install from your terminal:
npx ccplaybooks add llm-eval-audit
# LLM Eval Audit

Audit an existing LLM eval pipeline and surface its problems — missing error analysis, unvalidated judges, vanity metrics — as a prioritized list with concrete fixes. Use when inheriting an eval system, when unsure whether your evals are trustworthy, or as a starting point when there's no eval infrastructure yet. (To build an evaluator from scratch instead, start with error analysis.)

## Overview

1. Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards.
2. Run the diagnostic checks across six areas.
3. Produce a findings report ordered by impact, each finding linked to a fix.

If an observability platform (Phoenix, Braintrust, LangSmith, or similar) is connected, pull traces, evaluator definitions, and experiment results from it. Otherwise ask for local files: CSVs, JSON trace exports, notebooks, eval scripts. If there are no artifacts at all, jump to "No Eval Infrastructure."

Get new playbooks like this one

One email a week with new Claude Code workflows. Free, like everything here.

No spam. Unsubscribe anytime.

README.md

What This Does

A diagnostic audit for an existing LLM eval pipeline, drawn from Hamel Husain's widely-cited eval methodology. It inspects the actual artifacts (traces, evaluator configs, judge prompts, labeled data, dashboards) across six areas and produces an impact-ordered findings report where each problem is tied to a specific fix — not a generic checklist, but a read of what this pipeline actually does wrong. The six areas are error analysis (were failure categories observed from real traces, or brainstormed from research vocabulary?), evaluator design (binary pass/fail vs. uncalibratable Likert scales; one-failure-mode judges vs. holistic "is this good?" prompts; code checks reserved for objective criteria; similarity metrics like ROUGE/BERTScore not masquerading as quality measures), judge validation (validated against human labels at all; measured with TPR/TNR rather than accuracy; a real train/dev/test split with no few-shot leakage), human review (domain experts not general annotators; full traces not just outputs), labeled-data sufficiency with smart sampling strategies, and pipeline hygiene (error analysis and judges refreshed after significant changes).

The single highest-value check is judge validation: the audit specifically catches the class-imbalance accuracy trap, where an unvalidated judge reports high accuracy while catching none of the failures that matter, and insists on TPR/TNR instead. It also handles the from-scratch case — if there's no eval infrastructure at all, it refuses to recommend building judges or dashboards before error analysis on real traces is done, because evaluators built without grounding in observed failures measure the wrong things. It pairs naturally with the companion error-analysis playbook, which is the first step it keeps pointing to.


Quick Start

Step 1: Create a Project Folder

mkdir eval-audit && cd eval-audit

Step 2: Download the Template

Click Download above, then:

mv ~/Downloads/CLAUDE.md ./

Step 3: Audit Your Eval Pipeline

claude

Point Claude at your eval artifacts (or an observability platform, or just your eval scripts and CSVs) and ask it to audit the pipeline. It will work through the six areas against what you actually have and return an impact-ordered findings report with a concrete fix per problem.


Tips & Best Practices

  • Prioritize fixing judge validation if the audit flags it — an unvalidated judge isn't a minor gap, it means every metric downstream of that judge is unreliable, and the TPR/TNR fix is cheap relative to the false confidence it removes.
  • Take the "fix obvious problems before building evaluators" ordering seriously — many failures turn out to be a missing prompt instruction, an unexposed tool, or a plain code bug, and building a judge for those instead of fixing them is wasted effort.
  • Re-run the audit after any significant pipeline change (model switch, prompt rewrite, new feature); eval trustworthiness decays as the system evolves, and a one-time audit gives false comfort.

Limitations

  • An audit of an existing pipeline, not a build guide — when the finding is "no error analysis has been done," the actual work shifts to the companion error-analysis methodology, which this playbook defers to rather than reproducing.
  • Its value depends on being given real artifacts to inspect; run as a checklist against nothing, it degrades into the generic advice it explicitly warns against.
  • Judge-validation targets (~50 Pass / ~50 Fail, ~100 traces for saturation) are rules of thumb that scale with system complexity, not hard thresholds — treat them as starting points, not guarantees.

$Related Playbooks

Developer Tools

Long-Horizon Agent Prompting

Design the launch prompt for an agent working autonomously for hours or days — a pseudo-formal task brief with an exact success predicate, an enumerated non-counting-outcomes list, adversarial verification, and effort floors, because persistence pressure against a loose spec produces confident non-solutions.

10 minutes
Advanced
Developer Tools

Loopy: Build and Reuse Agent Loops

Mine your codebase and coding history for repeated work, turn it into bounded agent loops, and audit the loops you already have

10 minutes
Intermediate
Developer Tools

LLM Icon Finder

Find and download AI/LLM model brand icons from the lobe-icons library — Claude, GPT, Gemini, and more — as ready-to-use SVG, PNG, or WEBP URLs.

5 minutes
Beginner
Developer Tools

LLM Coding Guardrails

Four behavioral guidelines that reduce the most common Claude coding mistakes: silent assumptions, overcomplication, scope creep, and vague success criteria.

2 minutes
Beginner
Developer Tools

Multi-Stage Prompt Pipeline

A methodology for building Claude-to-Claude prompt pipelines where one stage's output feeds the next — a four-purpose taxonomy (Research/Plan/Do/Refine), per-prompt provenance folders, XML-structured outputs with required metadata tags, a fixed human-scannable SUMMARY.md, dependency-aware layered execution, and per-stage validation before anything chains forward.

10 minutes
Advanced
Developer Tools

Minimalist UI Design Skill: Editorial, Monochrome Interfaces

A frontend design system that enforces warm monochrome, bento grids, and editorial typography — the Notion/Linear look, banned from gradients, heavy shadows, and Inter

5 minutes
Beginner
Developer Tools

Mermaid Tools

Extract Mermaid diagrams from markdown files and render them to high-quality PNG images using bundled scripts — turning text diagrams into shareable visuals.

5 minutes
Beginner
Developer Tools

Minimalist MVP Scoping

Scope a weekend-shippable MVP using the manual → processized → productized progression. Build as little as possible; charge from day one.

5 minutes
Beginner
Developer Tools

MCP Server Hub Manager

Access 1200+ AI Agent tools via Model Context Protocol (MCP)

10 minutes
Advanced
Developer Tools

n8n Workflow Builder

Automate document workflows with n8n - 7800+ workflow templates

10 minutes
Advanced
Developer Tools

MCP Server Builder

Guide for creating high-quality Model Context Protocol (MCP) servers that enable LLMs to interact with external services and APIs.

30 minutes
Advanced
Developer Tools

Nuxt UI Component Guidance

Build interfaces correctly with @nuxt/ui v4 — five core rules (UApp wrapper, semantic colors, reading generated theme files for real slot names, override priority, icon naming), a task-to-focus-area routing table, the Nuxt UI MCP server for live component API lookups, and full installation for both Nuxt and standalone Vue.

5 minutes
Beginner

Browse all Developer Tools playbooks →