# paper-gen Redesign: MAP-REDUCE Over the Full Daily Corpus

> **Provenance.** Authored 2026-07-04 in response to the editorial *"Five Hundred Papers a
> Week, Five Survive"* (`blog-posts/eschaton-2026-07-04-research-velocity.md`). Produced by a
> 20-agent design workflow: 7 readers mapped the real pipeline + consumers, 3 competing
> architectures were generated and adversarially judged across feasibility / cost /
> editorial-impact lenses (scores 39 / 38.3 / 37.3), then synthesized into this plan.
> The plan takes the winning **batched MAP-REDUCE** spine and grafts the **embeddings-first
> cost lever** and the **local-flywheel** angle from the runners-up.

## 1. Problem restatement

The 2026-07-04 editorial "Five Hundred Papers a Week, Five Survive" criticizes paper-gen's own
funnel: ~440 unique papers are fetched, embedded into Qdrant, and entity-counted every morning —
then **discarded down to a hard `TOP_PAPERS_COUNT=5` slice** with no synthesis over the remainder.
The entity layer is closed-vocabulary (regex plateaued at exactly its vocab size:
models=42=`len(MODEL_PATTERNS)`), so genuinely new names (Claude 5, a fresh benchmark) are
structurally invisible, and the editorial rediscovers cross-paper through-lines like
"modularization" by hand from ≤8 title-only bullets. The directive: stop narrowing and
discarding — do **more entity extraction** (open-vocabulary) and a **genuine MAP-REDUCE over the
main points of the whole corpus**, so compounding output becomes compounding knowledge and the
editorial is fed real emergent themes.

This plan takes the winning **batched MAP-REDUCE** architecture as the spine, grafts the runner-up
**embeddings-first cost lever** (cluster free with MiniLM before spending any LLM tokens) and the
**local-flywheel map angle** (extractions become golden-path gold; MAP can route to local Ollama),
and adopts the adversarial lenses' corrections: separate flags per concern, a batch-coverage floor,
a whole-block wall-clock budget, correct Qdrant vector sourcing, per-call model routing (never a
global `CLAUDE_HEADLESS_MODEL` export), and a literal-substring span guard on claims.

---

## 2. Target architecture

Two new pipeline stages, both fail-open and flag-gated, inserted into `paper_generator.py`'s
existing numbered pipeline. Neither touches the top-5 curation contract.

### MAP phase — `paper_extractor.py` (embeddings-first, then batched extraction)

Runs after Step 2 (papers already embedded/stored in Qdrant `arxiv_papers`), reading the same
`papers_dicts` list the orchestrator builds at `paper_generator.py:95-107`. **Cost is controlled
embeddings-first**: the day's MiniLM vectors already exist, so the MAP never re-embeds for
partitioning.

**Execution strategy (three stacked cheapness levers):**

1. **Regex floor (free):** run the existing `entity_extractor.extract_all()` closed-vocab passes on
   100% of papers to seed the `entities` buckets. Whatever the LLM finds beyond these becomes
   `novelty_terms` — the open-vocab delta.
2. **Batched LLM extraction:** batch `papers_dicts` into groups of `EXTRACT_BATCH_SIZE` (start
   conservative at **8**, not 20 — the adversarial consensus is that 20-object deeply-nested JSON
   drifts/truncates on Haiku), so ~440 papers cost ~**55 calls** at batch 8 (tune upward only after
   measuring parse-success in shadow). Each batch prompt sends `title + abstract[:1200]` (raised
   from the curator's 500 so claim numbers past char 500 are actually captured) + first 3 authors.
   Route through `unified-gen/llm_service.get_llm_service().generate(prompt, model=EXTRACT_MODEL)` —
   **per-call `model=` argument, never a shell `CLAUDE_HEADLESS_MODEL` export** (an export would
   silently flip the shipped curator call off its Opus default). Pin `EXTRACT_MODEL='haiku-4-5'`
   (~15× cheaper; structured output is Voice-Gate-exempt).
3. **Optional local routing:** `ENABLE_LOCAL_ROUTING` fans the MAP to `ollama_client` (llama3.2)
   first, fail-open to Haiku. **Default OFF** until a matched extraction benchmark exists
   (llama3.2:3b schema fidelity on dense abstracts is unproven and undetectable without a benchmark).

**Reliability guards:** per-batch `try/except` skips a bad batch (never the run); a **coverage
floor** logs a warning if a batch returns fewer objects than papers sent; `arxiv_id` is validated
against the batch's input set and mismatched rows are dropped (mirrors the curator's silent-drop
guard); parsing reuses the lifted `_parse_claude_response` fence-strip + first-`[`/last-`]` slice
logic. **Exact-sha per-paper dedup cache** in `paper-state/extractions/` (capture-gen pattern) so
re-runs and cross-day duplicate papers cost 0 calls. Writes `paper-state/extractions/{date}.jsonl`
**atomically** (temp + rename) so a launchd SIGKILL can't corrupt it.

**Per-paper extraction JSON schema (`extraction_schema.py::PaperExtraction`):**

```json
{
  "arxiv_id": "2607.01234",
  "entities": {
    "models":       ["Claude 5", "DeepSeek-V4"],
    "methods":      ["mixture-of-depths"],
    "datasets":     ["FineWeb-2"],
    "benchmarks":   ["GPQA-Diamond"],
    "tasks":        ["long-context retrieval"],
    "institutions": ["Anthropic"]
  },
  "claims": [
    {
      "text": "MoD variant matches dense baseline at 40% fewer FLOPs on GPQA-Diamond",
      "type": "improvement",                     // sota|improvement|capability|limitation|theory
      "benchmark": "GPQA-Diamond",
      "metric": "FLOPs",
      "delta": "-40%",
      "span": "…40% fewer FLOPs on GPQA-Diamond…"  // MUST be a literal substring of the abstract
    }
  ],
  "contribution_type": "new-method",             // new-model|new-method|benchmark|analysis|theory|application|survey
  "capability_vs_deployment": "capability",      // capability|deployment|both|neither
  "one_line": "Mixture-of-Depths cuts inference cost without accuracy loss.",
  "extraction_source": "haiku|local|regex",
  "confidence": 0.0
}
```

Entities are **open-vocabulary free text** (the whole point). Each claim's `span` is validated as a
**literal substring** of the source abstract before the claim is kept (defends against hallucinated
scores; crossref-gen-style provenance, enforced by string containment).

### REDUCE phase — `theme_reducer.py`

Runs before Step 7 (digest render), wrapped in the same fail-open block. Steps:

1. **Flatten** `{date}.jsonl` to a claim list `(arxiv_id, claim_text, entities)` (~600–1300
   claims/day; hard-cap at `MAX_CLAIMS=1500`).
2. **Embed claim texts** with `embedding_service.embed_texts` (batched MiniLM 384-dim, free).
   Claims are embedded, **not** the paper-level Qdrant vectors — claims are finer-grained than
   abstracts.
3. **Cluster into emergent themes.** Primary path:
   `sklearn.cluster.AgglomerativeClustering(metric='cosine', linkage='average',
   distance_threshold=THEME_CLUSTER_THRESHOLD≈0.35, n_clusters=None)` (sklearn 1.8.0 confirmed in
   `paper-gen/venv`) so theme count is data-driven. **Deterministic stdlib fallback**: greedy
   online clustering reusing `relevance_gate.cosine` (assign to first cluster with centroid cosine
   > 1−threshold, else start new) so a missing sklearn or OOM can never crash the run.
4. **Rank + prune**: order clusters by (distinct paper count desc, then mean intra-cluster
   tightness); drop clusters below `MIN_PAPERS_PER_THEME=2`; keep top `MAX_THEMES=10`. This is a
   **coherence floor gated before LLM spend** — incoherent clusters never reach the reduce call.
5. **Exemplars**: per surviving theme pick 3–5 representative claims by **within-cluster
   cosine-to-centroid ranking over the assigned claim vectors** (a hand-rolled loop, NOT
   `search_with_mmr` — that primitive queries the whole 30-day Qdrant collection and would return
   papers outside today's set; the adversarial lenses flagged this as wired-wrong in every proposal
   that cited it).
6. **One REDUCE LLM call** via `llm_service.generate(model=REDUCE_MODEL)` on **Sonnet-4.x**
   (prose-adjacent → must stay on the Voice-Gate allowlist; `model_used` stamped). Input: top
   clusters as `{top_entities, 3-5 representative claims, paper_count}`. Output: per-theme
   `{label, synthesis, significance}` + a global `through_line` + a `capability_vs_deployment` rollup.
7. **Entity momentum**: canonicalize open-vocab entities (lowercase/punct-strip, then
   embedding-merge surface forms with cosine > `ENTITY_MERGE_COSINE=0.9`, **keeping the raw surface
   form for provenance**). Feed a **new open-vocab bucket** through the existing `EntityTracker`
   temporal machinery (`daily_history`/`growth_pct`/`get_trending`) to classify rising / emerging /
   falling — the only way to surface never-before-seen names. Note the divide-by-near-zero noise on
   day-1 sparse terms (acknowledged, self-correcting as the index warms).
8. **Assemble + write** `paper-state/day-shape/{date}.json`, mirror to
   `papers/day-shape-{date}.json` + `papers/day-shape-index.json`, and post to the blackboard
   (`bb.post('paper-gen','day_shape',…)` + `track_publish`).

**Day-shape artifact schema:**

```json
{
  "date": "2026-07-04",
  "through_line": "The day's dominant thread is modularization — routing and mixture architectures displacing monolithic scaling.",
  "themes": [
    {
      "label": "Conditional-compute architectures",
      "synthesis": "Multiple groups cut inference FLOPs via depth/expert routing without accuracy loss.",
      "significance": "high",
      "paper_count": 7,
      "arxiv_ids": ["2607.01234", "..."],
      "top_entities": ["mixture-of-depths", "MoE", "GPQA-Diamond"],
      "representative_claims": ["…", "…"]
    }
  ],
  "entity_momentum": {
    "rising":   [{"name": "mixture-of-depths", "type": "method", "growth_pct": 210, "arxiv_ids": ["..."]}],
    "emerging": [{"name": "DeepSeek-V4", "type": "model", "arxiv_ids": ["..."]}],
    "falling":  []
  },
  "capability_vs_deployment": {"capability": 18, "deployment": 22, "both": 5, "neither": 3},
  "paper_count": 441,
  "model_used": "claude-sonnet-4-6"
}
```

---

## 3. File-level change list

### New modules (under `paper-gen/`)

| Module | Role | Writes |
|---|---|---|
| `paper_extractor.py` | MAP: regex-floor + batched open-vocab entity/claim/contribution extraction over ALL ~440 papers; exact-sha cached; atomic write | `paper-state/extractions/{date}.jsonl` |
| `theme_reducer.py` | REDUCE: MiniLM-embed + agglomerative-cluster claims → themes, within-cluster exemplars, single Sonnet reduce, canonicalized open-vocab entity momentum | `paper-state/day-shape/{date}.json`, `papers/day-shape-{date}.json`, `papers/day-shape-index.json` |
| `extraction_schema.py` | Shared `PaperExtraction`/`Claim` dataclasses, JSON validator (drops malformed rows), lifted `_parse_claude_response`, span-substring check | — |
| `paper-gen/tests/test_extractor.py`, `tests/test_reducer.py` | schema conformance, regex fallback, span validation, cluster determinism, fail-open | — |
| `scripts/ci/check-paper-extraction.py` | CI guard: validate day-shape/extraction JSON against schema + capture-gen-style benchmark-leak check on claim text; fail-closed | — |

### Existing files to modify

- **`paper-gen/paper_generator.py`** — insert flag-gated, `try/except` fail-open **Step 3.5 (MAP)**
  after Step 2 and **Step 5.5 (REDUCE)** before Step 7; renumber prints to `/10` (8 existing + 2
  inserts). Wrap the whole MAP+REDUCE block in a **wall-clock budget**
  (`MAPREDUCE_BUDGET_SECONDS`, e.g. 480) checked between batches so a slow run bails to Step 7
  rather than risking the launchd `-k 60s 30m` SIGKILL that would take the real digest down. Pass
  `day_shape` into the digest history + blackboard.
- **`paper-gen/paper_config.py`** — add: `PAPER_MAP_ENABLED=False`, `PAPER_REDUCE_ENABLED=False`,
  `EDITORIAL_SYNTHESIS_ENABLED=False` (three **independent** flags — one flag conflating
  MAP/REDUCE/editorial-consumption breaks the shadow guarantee); `EXTRACT_BATCH_SIZE=8`,
  `EXTRACT_MODEL='haiku-4-5'`, `REDUCE_MODEL='sonnet-4-6'`, `EXTRACT_ABSTRACT_CHARS=1200`,
  `THEME_CLUSTER_THRESHOLD=0.35`, `MAX_THEMES=10`, `MIN_PAPERS_PER_THEME=2`, `MAX_CLAIMS=1500`,
  `ENTITY_MERGE_COSINE=0.9`, `MAPREDUCE_BUDGET_SECONDS=480`, `ENABLE_LOCAL_EXTRACTION=False`, plus
  output paths.
- **`paper-gen/entity_tracker.py`** — add an open-vocab entity bucket to `_load_index`/`_update_index`
  (default-on-missing migration for the ~1.3MB `entity_index.json` and ~114 historical daily files);
  **persist the per-entity `arxiv_id` provenance lists `record_day` already collects but discards**
  (fixes `get_novel_terms` returning `papers=[]`). Both consumers (`paper_generator` and
  `stats_rollup`) must tolerate the new bucket.
- **`paper-gen/stats_rollup.py`** — add **additive** `open_entities` + `themes` keys under
  `current.daily_stats`; existing keys and `papers.js` reads unchanged.
- **`paper-gen/paper_curator.py`** — (behind `PAPER_MAP_ENABLED` only) replace the arbitrary
  `papers[:50]` slice with a MAP-scored, cluster-diverse top-50 so the single curation call sees
  best-of-corpus. Keep this gated **separately** from editorial consumption.
- **`blog-gen/content_planner.py`** — **ship independently and unconditionally** the field-drift fix:
  map paper-gen's `keyInsight`/`whyInteresting`/`content` into a real `signals['papers'][].summary`
  (the editorial has silently been running on titles alone). Separately, behind
  `EDITORIAL_SYNTHESIS_ENABLED`, attach `signals['paper_synthesis']` from day-shape.
- **`blog-gen/editorial_writer.py`** — render a new `### Today's Research Through-Line` prompt section
  from `signals['paper_synthesis']` **above** the raw paper bullets in `_format_signals_for_prompt`.
- **`scripts/papers.js` + `papers.html`** — additive, `Array.isArray`-guarded themes +
  entity-momentum panel (reads the already-fetched `stats.json` additive keys, or
  `/papers/day-shape-{date}.json`); no change to existing panels. Reuse the existing
  `marked.parse`+`DOMPurify.sanitize` path.
- **`scripts/ci/verify.sh`** — wire in `check-paper-extraction.py`.

### New data artifacts → consumers

| Artifact | Consumer |
|---|---|
| `paper-state/extractions/{date}.jsonl` | `theme_reducer.py`, `entity_tracker.py`, capture-gen (flywheel) |
| `paper-state/day-shape/{date}.json` | `blog-gen` content_planner/editorial_writer |
| `papers/day-shape-{date}.json` + `-index.json` | `scripts/papers.js` (additive) |
| `papers/stats.json` additive `open_entities`/`themes` | `scripts/papers.js` renderTrendsPanel |
| `swarm-state/agents/paper-gen/day_shape.json` | digest-gen / watchdog via `bb.read` + `track_publish` |

All frontend artifacts are **same-origin static JSON** (satisfies CSP `connect-src 'self'`).
`paper-state/` is already gitignored; add a retention prune for `day-shape-{date}.json` (mirror
`PAPERS_RETENTION_DAYS`) so tracked frontend mirrors don't accumulate unboundedly.

---

## 4. Cost & latency budget

- **Today:** 1 Opus curation call/day.
- **New:** ~**55 MAP calls/day** (batch 8) on Haiku-4-5 (~15× cheaper than the Opus default;
  ~8 papers × ~1200 tok in + small JSON out) + **1 REDUCE call** on Sonnet-4.x + **0 extra embedding
  cost** (MiniLM free/batched; Qdrant vectors already exist).
- **$ posture:** low-single-dollars/day incremental at API rates — an order of magnitude more
  *calls* but on a far cheaper *tier*. The exact-sha extraction cache zeroes re-runs and cross-day
  duplicates. Naive per-paper Opus (440 calls) would instantly blow the post-2026-06-15 $200
  Max-20x credit pool; Haiku batching keeps this comfortably inside it.
- **Binding constraint is rate-limit contention, not dollars.** `llm_service` is fully blocking with
  no 429/retry handling. **Keep the MAP serial in v1** (no ThreadPoolExecutor — 4 concurrent
  `claude -p` processes share one OAuth keychain/credit pool and a single 429 silently drops a batch,
  quietly degrading "MAP over ALL 440"). Revisit concurrency only after adding retry to `llm_service`.
- **Wall-clock:** serial MAP at batch 8 ≈ 6–12 min realistic (claude CLI cold-start ~15–40s/call, not
  10s). This is why `MAPREDUCE_BUDGET_SECONDS=480` bails gracefully — it must never brush the 30-min
  launchd deadline. If budget is tight, raise `EXTRACT_BATCH_SIZE` toward 12–15 once shadow
  parse-success is confirmed, trading fidelity for fewer calls.

---

## 5. Rollout & safety

Mirrors `relevance_gate.py`'s shadow-first / fail-open template exactly, with the lenses' corrections:

1. **Three independent flags default `False`.** In shadow, MAP + REDUCE **run and write**
   extractions/day-shape/blackboard, but `PAPER_MAP_ENABLED=False` keeps the curator `[:50]`
   replacement dormant and `EDITORIAL_SYNTHESIS_ENABLED=False` keeps the editorial injection dormant.
   The `content_planner` summary bug-fix ships **outside** all flags (pure correctness). Top-5
   `papers/index.json` curation is never touched.
2. **Total fail-open:** the entire MAP+REDUCE block is wrapped in `try/except` printing `mapreduce
   skipped (fail-open)` and continuing to Step 7 — it can never crash or shrink the digest.
   Cascades: sklearn/OOM → greedy stdlib clusterer; local LLM down → Haiku; Haiku batch fails → skip
   that batch. **Plus** the wall-clock budget catches *slowness* (which `try/except` cannot).
3. **Purely additive contracts:** new state dir, new same-origin JSON, additive-only `stats.json`
   keys. `papers.html`/`digest.html`/`trends.html` all ignore unknown keys — nothing breaks even
   with flags on.
4. **CI guard** `scripts/ci/check-paper-extraction.py` in `verify.sh`: validates schema, runs a
   capture-gen-style benchmark-leak check on claim text, **and asserts every claim `span` is a
   literal substring of its source abstract**. Fail-closed. Add a `check:invariants` entry so the
   atomic-write + fail-open wrapper can't silently regress.
5. **Voice Gate stays green:** the REDUCE call stamps `model_used` on the Sonnet 4.x allowlist;
   structured MAP output is exempt (`check-editorial-model.js` only gates prose frontmatter). The
   published editorial is still written entirely by blog-gen on the allowlist.
6. **State-migration safety:** `entity_tracker._load_index` defaults the new open-vocab bucket when
   absent; the ~114 historical daily files are read-tolerant.

**Flip order after several days of shadow review:** MAP first (populate provenance + golden paths +
entity momentum), then REDUCE (day-shape + curator top-50), then `EDITORIAL_SYNTHESIS_ENABLED` last
(a change to the sacred voice's inputs) — exactly as `RELEVANCE_GATE_ENABLED` is flipped only after
hold-queue review.

---

## 6. Editorial integration

The editorial is the primary win, delivered in two independent increments:

1. **Unconditional bug-fix (ship day 1):** `content_planner.load_signals` currently reads
   `p.get('summary')`/`p.get('categories')` — keys `papers/index.json` never contains — so the
   editorial runs on titles alone. Map paper-gen's real `keyInsight`/`whyInteresting`/`content` into
   `signals['papers'][].summary`. Free, high-value, independent of the whole redesign.
2. **Synthesis injection (behind `EDITORIAL_SYNTHESIS_ENABLED`):** when day-shape exists,
   `content_planner` attaches `signals['paper_synthesis'] = {through_line, themes[]}`;
   `editorial_writer._format_signals_for_prompt` renders a dedicated `### Today's Research
   Through-Line` section **above** the raw paper bullets. The Synthesist then anchors on a computed
   cross-paper narrative (e.g. "modularization") instead of rediscovering it by hand each day.

Note the date-alignment gotcha: paper-gen's 6 AM day-shape is "today's fetch" while the editorial
frames "yesterday's papers." Have `content_planner` load the day-shape whose date matches the papers
it's already summarizing (resolve by the `index.json` date, not `date.today()`) to avoid an off-by-one.

A future digest `day_shape` section is a clean additive follow-on (every digest consumer ignores
unknown section keys) — out of scope for v1.

---

## 7. Flywheel angle

Every gate-passing **structured** extraction in `paper-state/extractions/{date}.jsonl` carries
per-paper provenance and is a natural golden-path candidate. Add the file to **capture-gen's `SOURCE
ALLOWLIST`** so extractions flow into `data/golden-paths.jsonl` under the existing benchmark-leak +
exact-sha guards. Each extraction becomes a `{messages:[user,assistant], meta}` pair (abstract →
structured JSON), gated on `confidence` + schema-validity + the `MAX_TARGET_CHARS` cap (split
multi-claim extractions if oversized).

**Honest scope (matching the existing flywheel posture):** the promoted pointer is a **0.5B
tool-calling router**, and the weekly gate scores a *router* benchmark — so these extraction
golden-paths will **capture** now but not usefully **promote** until a task-matched *extraction*
benchmark + fine-tune target exists. This is the correct North-Star trajectory: the expensive MAP
starts on Haiku, its gate-passing outputs accrue as training data, and once a matched benchmark
lands the MAP migrates local (`ENABLE_LOCAL_EXTRACTION`), cost falls, and extraction quality
compounds — without ever risking the editorial voice. Until then, keep local extraction **opt-in,
default off** (no benchmark to catch llama3.2 fidelity loss).

---

## 8. Phased milestones

**M0 — Free correctness wins (no LLM, ship immediately).**
- `content_planner` summary field-drift fix; persist `entity_tracker` per-entity `arxiv_id` provenance.
- *Verify:* run blog-gen `--dry-run`, confirm editorial prompt now contains real summaries (not
  empty); confirm `get_novel_terms` returns non-empty `papers[]`. `npm run check:bylines`,
  `check:editorial-model` stay green.

**M1 — MAP in shadow (first shippable slice of the redesign).**
- `extraction_schema.py` + `paper_extractor.py`; Step 3.5 in `paper_generator.py` behind
  `PAPER_MAP_ENABLED=False` (runs + writes in shadow, mutates nothing); exact-sha cache; atomic
  write; CI guard + `verify.sh` wiring.
- *Verify:* `python paper_generator.py --dry-run` (or a `--date` backfill) produces
  `extractions/{date}.jsonl`; `check-paper-extraction.py` passes (schema + span-substring + leak).
  Eyeball 2–3 days of open-vocab entities/claims for fidelity; measure per-batch parse-success to
  tune `EXTRACT_BATCH_SIZE`.

**M2 — REDUCE in shadow.**
- `theme_reducer.py`; Step 5.5 behind `PAPER_REDUCE_ENABLED=False`; agglomerative + stdlib-fallback
  clustering; within-cluster exemplars; single Sonnet reduce; entity-momentum via canonicalized
  open-vocab bucket; day-shape + frontend mirror + blackboard write; wall-clock budget.
- *Verify:* day-shape JSON validates; `through_line` is coherent on several shadow days; `model_used`
  on the Sonnet allowlist; confirm clustering degrades to stdlib when sklearn is force-disabled;
  confirm the budget bails cleanly without SIGKILL.

**M3 — Frontend surface (additive, safe with flags off).**
- `papers.js`/`papers.html` themes + entity-momentum panel; `stats_rollup` additive keys.
- *Verify:* `npm run check:noeval`, `check:syntax`, smoke-test — `papers.html`/`digest.html`/
  `trends.html` still parse; panel renders when day-shape present and disappears cleanly when absent
  (Array.isArray guard).

**M4 — Editorial consumption (flip last).**
- Wire `signals['paper_synthesis']` + `### Today's Research Through-Line` behind
  `EDITORIAL_SYNTHESIS_ENABLED`.
- *Verify:* blog-gen `--dry-run` shows the through-line section above the bullets; Voice Gate green;
  after several days review, flip `PAPER_MAP_ENABLED` → `PAPER_REDUCE_ENABLED` →
  `EDITORIAL_SYNTHESIS_ENABLED` in order.

**M5 — Flywheel capture.**
- Add `extractions/{date}.jsonl` to capture-gen's `SOURCE ALLOWLIST` with a confidence/schema floor.
- *Verify:* `capture_main.py --dry-run` harvests extraction pairs into golden-paths under the
  leak/dedup guards; confirm the router benchmark still gates to ITERATE (no false promote) —
  documents the honest-scope gap until an extraction benchmark exists.

**M6 (deferred) — Local MAP + extraction benchmark.**
- Build a task-matched extraction benchmark; enable `ENABLE_LOCAL_EXTRACTION`; only then consider
  MAP concurrency (after adding 429/retry to `llm_service`).

---

**Load-bearing corrections adopted from the adversarial lenses:** separate flags (not one); batch
size 8 with a coverage floor (not 20 with none); abstract budget 1200 chars (not 500) so claim
numbers survive; within-cluster cosine exemplars (not the mis-wired global `search_with_mmr`);
per-call `model=` (not a global `CLAUDE_HEADLESS_MODEL` export that flips the shipped curator);
serial MAP in v1 (no concurrency into the shared credit pool); a whole-block wall-clock budget under
the 30-min launchd deadline; literal-substring span validation on claims; atomic writes; and keeping
local extraction opt-in until a matched benchmark exists.