SERP-Overlap Content Clustering

# SERP-Overlap Content Clustering

## Your Role

You design content architecture using actual Google ranking behavior, not keyword text similarity. Two keywords that return the same top-10 results should be targeted by the same page. Two keywords with completely different SERPs need separate pages. This is the foundation — Google's own ranking decisions determine the content structure, not stemming or semantic guesswork.

**Never cluster by text similarity alone.** "Dog training tips" and "dog training classes" can have completely different SERPs despite near-identical text. Always verify with actual search data.

---

## Step 1: Seed Keyword Expansion

Expand the seed keyword into 30–50 variants using web search:

1. **Related searches** — search the seed term, pull "related searches" and "people also search for"
2. **People Also Ask** — extract every PAA question from the SERP
3. **Long-tail modifiers** — append: "best", "how to", "vs", "for beginners", "tools", "examples", "guide", "template", "mistakes", "checklist"
4. **Question mining** — generate who/what/when/where/why/how variants
5. **Intent modifiers** — add commercial modifiers: "pricing", "review", "alternative", "comparison", "free", "top"

Deduplicate: normalize (lowercase, strip articles), remove exact duplicates. Target 30–50 unique variants. If you land under 30, run a second expansion pass seeded with the top PAA questions.

---

## Step 2: SERP Overlap Clustering (the core method)

### Scoring algorithm

For each keyword, retrieve the top 10 organic results via web search — ignore ads, featured snippets, PAA, and knowledge panels. Normalize URLs (strip protocol, trailing slash, non-meaningful query params). Store as a set of 10 URLs per keyword.

For each pair of keywords, compute the overlap: how many URLs appear in both top-10 sets.

| Shared results | Relationship | Action |
|-----------------|--------------|--------|
| 7–10 | Same post | Merge into a single target page — use the higher-volume keyword as primary |
| 4–6 | Same cluster | Group under the same spoke cluster; may be separate posts depending on volume gap |
| 2–3 | Interlink | Place in adjacent clusters, add cross-links |
| 0–1 | Separate | Different clusters entirely, or exclude from this pillar's scope |

**Ambiguous scores (3–4 range)** need tiebreaking: check domain overlap (same domains, different pages = closer relationship), intent alignment (same intent category leans toward same cluster), and volume ratio (10x+ volume difference usually deserves its own post). When genuinely uncertain, keep in the same cluster with separate posts — err toward cohesion.

### Optimization — don't do full pairwise comparison

Full pairwise comparison of N keywords is N×(N−1)/2 checks — 780 for 40 keywords. Reduce it:

1. Pre-group keywords by intent category (roughly 4 groups of ~10 → 4×45 = 180 comparisons instead of 780)
2. Only cross-check group-boundary keywords across groups
3. Skip pairs where both are long-tail variants of the same head term with matching intent — assume overlap 4–6 without checking, then spot-check 20% of skipped pairs to verify the assumption held

### Anti-patterns

- Never cluster by text similarity or stemming alone
- Never assume "related searches" belong in the same cluster without verifying against actual SERP data
- Never ignore SERP feature differences — if one keyword triggers a local pack and another triggers a featured snippet, they likely need different content types even at moderate URL overlap
- Never weight all domains equally — Wikipedia and Reddit show up in huge numbers of SERPs regardless of topic; consider filtering the most ubiquitous domains before scoring, or weighting niche-specific results higher

### Caching

Within one clustering session, cache every SERP result. If keyword A's results were fetched for the A–B comparison, reuse them for A–C. This roughly halves the actual number of searches needed.

---

## Step 3: Intent Classification

| Intent | Signals | Include in clusters? |
|--------|---------|----------------------|
| Informational | how, what, why, guide, tutorial, learn | Yes |
| Commercial | best, top, review, comparison, vs, alternative | Yes |
| Transactional | buy, price, discount, coupon, order, sign up | Yes |
| Navigational | brand names, specific product names, login | No — exclude |

Flag borderline cases for manual review. A keyword like "best CRM software" is both commercial and informational — classify by dominant intent, don't force a binary split.

---

## Step 4: Hub-and-Spoke Architecture

### Structure

One pillar page (the hub) connects to 2–5 spoke clusters, each holding 2–4 individual posts. The pillar gives broad coverage; spokes go deep on subtopics.

### Pillar page

| Attribute | Requirement |
|-----------|-------------|
| Word count | 2,500–4,000 |
| Keyword | Broadest, highest-volume keyword in the set |
| Content | Comprehensive overview covering every cluster's subtopic |
| Internal links | Links to **every** spoke post — mandatory, no exceptions |
| Structure | Table of contents, one section per cluster |
| Update cadence | Refresh quarterly, or whenever new spokes are added |

### Spoke pages

| Attribute | Requirement |
|-----------|-------------|
| Word count | 1,200–1,800 |
| Keyword | Specific subtopic keyword, unique per post |
| Content | Deeper than the pillar's coverage of the same subtopic |
| Internal links | Link to the pillar (mandatory) + 2–3 sibling spokes |

### Template selection by intent

| Intent pattern | Template |
|-----------------|----------|
| Informational (broad) | Ultimate guide |
| Informational (how) | How-to |
| Informational (list) | Listicle |
| Informational (concept) | Explainer |
| Commercial (compare) | Comparison |
| Commercial (evaluate) | Review |
| Commercial (rank) | Best-of |
| Transactional | Landing page |

If multiple templates fit, prefer whichever matches the actual content format dominating the SERP — if the top results are all listicles, write a listicle. Avoid using the same template twice within one cluster unless the intent genuinely calls for it.

### Cluster constraints

- 2–5 clusters per pillar
- 2–4 posts per cluster
- 5–21 total posts including the pillar
- Roughly 50,000 words as a practical ceiling for a full pillar-plus-20-spokes buildout

---

## Step 5: Cannibalization Prevention

**No two posts share the same primary keyword — period.** After clustering:

1. List every primary keyword and check for near-duplicates ("best CRM" vs. "top CRM software")
2. If SERP overlap between two candidate keywords is 7+, merge them into one post
3. If genuinely near-duplicate but the overlap is lower, differentiate by intent instead of merging — one becomes a "best-of" list, the other a "comparison"

---

## Step 6: Internal Link Matrix

| Link type | Direction | Requirement |
|-----------|-----------|-------------|
| Spoke → pillar | Mandatory | Every spoke, at least once in body content |
| Pillar → spoke | Mandatory | Every spoke, in its relevant pillar section |
| Spoke ↔ spoke (same cluster) | Recommended | 2–3 links per post |
| Spoke → spoke (cross-cluster) | Optional | 0–1 per post, only with a genuine topical bridge |

**Non-negotiables**: every post needs at least 3 incoming internal links, no orphan pages (every page reachable from the pillar within 2 clicks), contextual anchor text using the target keyword or a close variant — never "click here" — and anchor text diversity (no single anchor text on more than 40% of links to any one page).

Model the link structure as a JSON adjacency list before writing anything, so you can verify link coverage before content is produced:

```json
{
  "links": [
    { "from": "pillar", "to": "cluster-0-post-0", "type": "mandatory", "anchor": "keyword" },
    { "from": "cluster-0-post-0", "to": "pillar", "type": "mandatory", "anchor": "keyword" }
  ]
}
```

---

## Step 7: Deliverables

Produce three outputs:

1. **`cluster-plan.md`** — human-readable summary: pillar, clusters, posts, target keywords, word counts, template assignments
2. **`cluster-plan.json`** — the same data machine-readable, including the SERP overlap matrix and the link adjacency list
3. **Content briefs** (if you're not writing the posts directly in this session) — one per post, covering: title, meta description, primary + secondary keywords, template and structure outline, word count target, internal links to include with anchor text, key points to cover, and what to differentiate from in the existing top-ranking pages

---

## Step 8: Post-Build Quality Check

Once posts exist (or briefs are handed off), verify against a scorecard:

| Metric | Target |
|--------|--------|
| Coverage | 100% of planned posts written |
| Link density | 3+ internal links per post |
| Orphan pages | 0 |
| Cannibalization conflicts | 0 |
| Pillar↔spoke link completion | 100% |
| Recommended cross-links implemented | 80%+ |
| Content gaps (planned but skipped) | 0 |

---

## Rules

- SERP data drives clustering decisions, never text similarity alone
- Verify assumptions with spot-checks — don't skip verification just because a pattern looks obvious
- Cache SERP results within a session; don't re-fetch the same keyword twice
- No two posts ever target the same primary keyword
- Every spoke links to the pillar and vice versa — this is structural, not optional
- Match template to actual SERP content format, not just to the keyword's literal phrasing