Promptfoo Evaluation
Configure and run LLM evaluations with Promptfoo — build promptfooconfig.yaml, write Python custom assertions, implement llm-rubric judges, and compare models systematically.
Shipping prompt changes on vibes means you never know if you actually improved anything. This playbook sets up repeatable evals so you can measure prompt and model quality.
Who it's for: ML engineers, prompt engineers, AI product teams
Example
"Set up an eval comparing two prompts on my test cases" → A promptfooconfig.yaml with assertions and an llm-rubric judge, plus a side-by-side comparison
CLAUDE.md Template
New here? 3-minute setup guide → | Already set up? Copy the template below.
npx ccplaybooks add promptfoo-evaluation# Promptfoo Evaluation
## Overview
This skill provides guidance for configuring and running LLM evaluations using [Promptfoo](https://www.promptfoo.dev/), an open-source CLI tool for testing and comparing LLM outputs.
## Quick Start
```bash
# Initialize a new evaluation project
npx promptfoo@latest init
Get new playbooks like this one
One email a week with new Claude Code workflows. Free, like everything here.
No spam. Unsubscribe anytime.
What This Does
Configure and run LLM evaluations with Promptfoo — build promptfooconfig.yaml, write Python custom assertions, implement llm-rubric judges, and compare models systematically.
What's Inside
The template covers:
- Configuration Structure
- Core Configuration (promptfooconfig.yaml)
- Prompt Formats
- Test Cases (tests/cases.yaml)
- Python Custom Assertions
- LLM-as-Judge (llm-rubric)
Quick Start
Step 1: Create a Project Folder
Make a dedicated folder for this workflow and open it in Claude Code.
Step 2: Download the Template
Click Download above to save the template, then drop it into your project as CLAUDE.md (or paste it into your existing one).
Step 3: Start Working
Tell Claude what you need in plain language — it will follow the template's workflow automatically. For example:
Set up an eval comparing two prompts on my test cases
Claude reads the template and runs the steps for you.