fix: batch summary writes to avoid output truncation on large harvests

When a harvest produced 15+ articles, asking the agent to emit the entire
.summaries.json in a single response hit the model's output-token ceiling —
the response was truncated and the tail articles silently got no summary
(their 原文 kept the __SUMMARY_<slug>__ sentinel forever). Reproduced every
time a high-volume source ran; ≤9-article sources never hit it.

Add scripts/summary-add.js to incrementally merge small batches (≤4) into
.summaries.json (keyed by url, overwrite-no-dup), with a --status mode that
diffs manifest vs summaries so interrupted runs can resume. Rewrite SKILL.md
Flow A Step 2 + Guardrails to mandate batching, and surface a batchHint in
harvest.js's manifest so the agent sees the guidance on first contact.

SKILL.md

@@ -15,6 +15,8 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau
 | Preview latest (read-only) | `node scripts/harvest.js <source_id> <count> --preview` |
 | Preview with full body | `node scripts/harvest.js <source_id> <count> --preview --full` |
 | Finalize Chinese summaries | `node scripts/finalize.js "<dayDir>"` |
+| Merge a summary batch | `node scripts/summary-add.js "<dayDir>"` |
+| Check summary coverage | `node scripts/summary-add.js "<dayDir>" --status` |
 | Add source | `node scripts/source-manage.js add` |
 | List sources | `node scripts/source-manage.js list` |
 | Edit source | `node scripts/source-manage.js edit <id>` |
@@ -235,19 +237,20 @@ It prints a **compact manifest** to stdout (no full body — only a `bodyPreview
 
 > **Browser sources and any source with a recipe** are auto-handled: before fetching, `harvest.js` calls `ensure-chromium.js`, which health-checks the CDP port and launches Chromium itself (cross-platform binary discovery, headless by default) if it isn't up — a no-op if already running. No manual `chromium ... &` or `sleep` is needed. To debug launch failures: `node scripts/ensure-chromium.js` (prints binary path, port, and stderr-log tail on failure).
 
-#### Step 2. Generate Chinese summaries → write `.summaries.json`
-From the compact manifest, for each article write one entry to `<dayDir>/.summaries.json`:
+#### Step 2. Generate Chinese summaries → write `.summaries.json` (batched)
+
+**Why batch:** writing all summaries in a *single* response hits the model's output-token ceiling once you have ~10+ articles — the response is truncated and the remaining articles silently get no summary (their 原文 keeps the `__SUMMARY_<slug>__` sentinel forever). This failure is **not rare** — it reproduces every time a source yields many articles. Avoid it by working in small batches that each get merged into `.summaries.json` by `summary-add.js`.
+
+Each summary entry has this shape:
 
 ```json
-[
-  {
-    "url": "https://theconversation.com/...-286430",
-    "titleZh": "特朗普新规或拖慢联邦科研",
-    "category": "科技/AI",
-    "tags": ["科技/AI", "The Conversation"],
-    "summaryZh": "## 事件概述\n...\n\n## 背景\n...\n\n## 关键引语\n> \"...\"\n\n## 影响分析\n..."
-  }
-]
+{
+  "url": "https://theconversation.com/...-286430",
+  "titleZh": "特朗普新规或拖慢联邦科研",
+  "category": "科技/AI",
+  "tags": ["科技/AI", "The Conversation"],
+  "summaryZh": "## 事件概述\n...\n\n## 背景\n...\n\n## 关键引语\n> \"...\"\n\n## 影响分析\n..."
+}
 ```
 
 - `titleZh`: concise Chinese title (≤30 chars)
@@ -255,6 +258,23 @@ From the compact manifest, for each article write one entry to `<dayDir>/.summar
 - `summaryZh`: the 4-section markdown body (事件概述 / 背景 / 关键引语 / 影响分析)
 - The `bodyPreview` in the manifest is enough to write the summary. **Do not re-read the full 原文.md** — if you need more, read a targeted slice.
 
+**Procedure** (loop until every article in the manifest has a summary):
+
+1. Let `N` = number of articles in the manifest. If `N ≤ 8`, you may write all entries to `.summaries.json` directly (one Write) and skip to Step 3. Otherwise work in batches of **≤4**.
+2. For each batch: use `Write` to create `<dayDir>/.summaries-batch.json` — a JSON array of ≤4 summary entries — then merge it:
+   ```bash
+   node scripts/summary-add.js "<dayDir>"
+   ```
+   This merges the batch into `.summaries.json` (keyed by `url`, so re-supplying an entry overwrites it — no duplicates), then deletes the batch file. Repeat per batch.
+3. Once all batches are done, verify coverage and fill any gaps:
+   ```bash
+   node scripts/summary-add.js "<dayDir>" --status
+   ```
+   Lists done vs. missing against `.manifest.json`. Write any missing articles as another batch and merge. When `--status` reports `missing: 0`, proceed to Step 3.
+
+> **Interrupted run?** Do not re-run `harvest.js` (it dedups via the registry, so it won't re-fetch). Instead run `node scripts/summary-add.js "<dayDir>" --status` to see which articles still need summaries, then resume from there.
+
+
 #### Step 3. Run finalize (script stitches summaries in)
 ```bash
 node scripts/finalize.js "<dayDir>"
@@ -276,6 +296,7 @@ No files are written. stdout is a compact JSON list (one call, no re-fetching /
 
 - **`count` is the target number of NEW articles**, not raw links. `harvest.js` already accounts for registry dedup: it scrolls to collect extra candidate links (`count + seen + buffer`, capped) so the *new* count can still reach `count` even when most of the homepage was already archived. If the run returns **fewer than `count`**, the stderr prints `⚠️ … stream may be exhausted` — that's a genuine source limit (no fresh articles to fetch), **not** a bug; re-running won't help. Wait for new articles or broaden the source instead. Don't re-run `harvest.js` for the same source/day just to "get more" — it deduplicates via the registry, so a second run only fetches genuinely new articles (and merges them into the pending `.manifest.json`, which is safe).
 - **Never `Read` the 原文.md files** to write summaries — the `bodyPreview` in the manifest is sufficient. Targeted slice reads are OK if truly needed.
+- **Summaries in batches of ≤4, never one big response.** A single response holding 10+ summaries hits the output-token ceiling and is truncated — the tail articles silently get no summary and keep the `__SUMMARY_<slug>__` sentinel. Write each batch to `.summaries-batch.json` and merge via `node scripts/summary-add.js "<dayDir>"`; repeat per batch. `summary-add.js --status` shows what's left. This is mandatory for `N > 8`; recommended for any multi-article run.
 - **One `finalize.js` call** after writing `.summaries.json`. Do not hand-write file I/O — the scripts handle all original/registry/index/summary writes.
 
 ---

scripts/harvest.js

@@ -626,7 +626,11 @@ Examples:
     nextStep: `Read this manifest, write Chinese summaries to ${path.join(dayDir, '.summaries.json')}, then run: node scripts/finalize.js "${dayDir}"`,
     summariesContract: {
       path: path.join(dayDir, '.summaries.json'),
-      format: 'array of { url, titleZh, category, tags[], summaryZh } — one per article url; summaryZh = the 4 sections markdown (事件概述/背景/关键引语/影响分析)'
+      format: 'array of { url, titleZh, category, tags[], summaryZh } — one per article url; summaryZh = the 4 sections markdown (事件概述/背景/关键引语/影响分析)',
+      batchHint: `If ${mergedArticles.length} articles is more than ~8, do NOT write all summaries in one response — ` +
+        `it will hit the output-token ceiling and get truncated. Work in batches of ≤4: write each batch to ` +
+        `${path.join(dayDir, '.summaries-batch.json')} and merge with "node scripts/summary-add.js \\"${dayDir}\\"" ` +
+        `(repeat per batch), then run finalize.js once. Check gaps with "node scripts/summary-add.js \\"${dayDir}\\" --status".`
     },
     articles: mergedArticles
   };

scripts/summary-add.js

+#!/usr/bin/env node
+/**
+ * summary-add.js - Incrementally merge a batch of Chinese summaries into .summaries.json.
+ *
+ * Problem this solves: when a harvest produces many articles (15+), asking the agent
+ * to emit the entire .summaries.json in ONE response hits the model's output-token
+ * ceiling — the response is truncated mid-stream and only some articles get summaries
+ * (the rest keep the __SUMMARY_<slug>__ sentinel forever). Splitting summary work into
+ * small batches that are each merged into .summaries.json here removes that ceiling:
+ * each batch is a tiny Write + one call to this script, so no single response can be
+ * truncated, and an interrupted run resumes from where it left off.
+ *
+ * Usage:
+ *   node summary-add.js <dayDir> [batchFile]     merge a batch (default <dayDir>/.summaries-batch.json)
+ *   node summary-add.js <dayDir> --status        diff .manifest.json vs .summaries.json (done / missing)
+ *   node summary-add.js -h | --help
+ *
+ * Batch file format (JSON array, one entry per article):
+ *   [{ "url": "...", "titleZh": "...", "category": "...", "tags": [...], "summaryZh": "..." }]
+ *
+ * Merge semantics:
+ *   - Entries are keyed by `url`. A url already present in .summaries.json is OVERWRITTEN
+ *     (so a corrected summary for an article replaces the old one, no duplicates).
+ *   - On success the batch file is deleted (it has been absorbed into .summaries.json).
+ *   - On parse error the batch file is LEFT IN PLACE so the agent can fix and retry.
+ *
+ * Exit codes: 0 success, 1 usage/IO error, 2 batch parse error (batch file kept).
+ */
+
+const fs = require('fs');
+const path = require('path');
+
+const DEFAULT_BATCH = '.summaries-batch.json';
+const SUMMARIES = '.summaries.json';
+const MANIFEST = '.manifest.json';
+
+function printHelp() {
+  console.log(`summary-add.js — Incrementally merge Chinese summary batches into .summaries.json.
+
+Solves the output-truncation failure where many articles (15+) exceed a single
+model response: write summaries in small batches (≤4 each), merging each batch
+here, then run finalize.js once at the end.
+
+Usage:
+  node summary-add.js <dayDir> [batchFile]    Merge a batch into <dayDir>/.summaries.json
+  node summary-add.js <dayDir> --status       Show done vs missing summaries (needs .manifest.json)
+  node summary-add.js -h | --help             Show this help
+
+Arguments:
+  dayDir      Path to the YYYYMMDD directory produced by harvest.js
+  batchFile   JSON array of { url, titleZh, category, tags[], summaryZh }
+              (default: <dayDir>/.summaries-batch.json)
+
+Merge:
+  - Keyed by url; existing entry for a url is overwritten (no duplicates).
+  - Batch file is deleted after a successful merge.
+  - Batch file is KEPT on parse error (fix and retry).
+
+Example:
+  # Agent writes 3 summaries to .summaries-batch.json, then:
+  node scripts/summary-add.js "/path/to/vault/Source/20260724"
+  # Repeat per batch, then run finalize.js once.`);
+}
+
+function loadJson(p, label) {
+  if (!fs.existsSync(p)) {
+    console.error(`✗ ${label} not found: ${p}`);
+    process.exit(1);
+  }
+  try {
+    return JSON.parse(fs.readFileSync(p, 'utf8'));
+  } catch (e) {
+    console.error(`✗ Failed to parse ${label}: ${e.message}`);
+    process.exit(1);
+  }
+}
+
+function isValidEntry(s) {
+  return s && typeof s === 'object' &&
+    typeof s.url === 'string' && s.url &&
+    typeof s.titleZh === 'string' && s.titleZh.trim();
+}
+
+// ── --status: diff .manifest.json urls against .summaries.json urls ──────────
+function status(dayDir) {
+  const manifestPath = path.join(dayDir, MANIFEST);
+  const summariesPath = path.join(dayDir, SUMMARIES);
+
+  if (!fs.existsSync(manifestPath)) {
+    console.error(`✗ ${MANIFEST} not found in ${dayDir}`);
+    console.error(`  (already finalized? manifest is removed by finalize.js)`);
+    process.exit(1);
+  }
+
+  const manifest = loadJson(manifestPath, MANIFEST);
+  const manifestArticles = Array.isArray(manifest.articles) ? manifest.articles : [];
+
+  let summaries = [];
+  if (fs.existsSync(summariesPath)) {
+    try { summaries = JSON.parse(fs.readFileSync(summariesPath, 'utf8')); }
+    catch (e) {
+      console.error(`✗ Failed to parse ${SUMMARIES}: ${e.message}`);
+      process.exit(1);
+    }
+  }
+  if (!Array.isArray(summaries)) summaries = [];
+
+  const doneUrls = new Set(summaries.filter(isValidEntry).map(s => s.url));
+  const done = manifestArticles.filter(a => doneUrls.has(a.url));
+  const missing = manifestArticles.filter(a => !doneUrls.has(a.url));
+
+  console.error(`📊 Summary status for ${dayDir}`);
+  console.error(`   Manifest articles: ${manifestArticles.length}`);
+  console.error(`   Summaries written: ${done.length}`);
+  console.error(`   Missing:           ${missing.length}`);
+  if (missing.length) {
+    console.error(`\n   ❌ Missing summaries:`);
+    for (const a of missing) {
+      console.error(`     - ${a.titleEn || a.url}`);
+      console.error(`       ${a.url}`);
+    }
+    console.error(`\n   Write the missing summaries (≤4 per batch) and run:`);
+    console.error(`     node scripts/summary-add.js "${dayDir}"`);
+  } else if (manifestArticles.length > 0) {
+    console.error(`\n   ✅ All ${manifestArticles.length} articles have summaries — run finalize.js:`);
+    console.error(`     node scripts/finalize.js "${dayDir}"`);
+  }
+}
+
+// ── merge: absorb a batch file into .summaries.json ──────────────────────────
+function merge(dayDir, batchFile) {
+  const batchPath = path.isAbsolute(batchFile) ? batchFile : path.join(dayDir, batchFile);
+  const summariesPath = path.join(dayDir, SUMMARIES);
+
+  if (!fs.existsSync(batchPath)) {
+    console.error(`✗ Batch file not found: ${batchPath}`);
+    console.error(`  Write a JSON array of summaries there first.`);
+    process.exit(1);
+  }
+
+  // Parse the batch FIRST; on failure keep it so the agent can fix and retry.
+  let batch;
+  try {
+    batch = JSON.parse(fs.readFileSync(batchPath, 'utf8'));
+  } catch (e) {
+    console.error(`✗ Failed to parse batch file (${batchPath}): ${e.message}`);
+    console.error(`  Batch file KEPT — fix the JSON and re-run.`);
+    process.exit(2);
+  }
+  if (!Array.isArray(batch)) {
+    console.error(`✗ Batch file must be a JSON array, got ${typeof batch} (${batchPath})`);
+    console.error(`  Batch file KEPT — fix and re-run.`);
+    process.exit(2);
+  }
+
+  // Validate entries; collect good ones, report bad ones (don't abort the whole batch).
+  const incoming = [];
+  let bad = 0;
+  for (const s of batch) {
+    if (!isValidEntry(s)) {
+      console.error(`  ⚠️  Skipping invalid entry (needs url + non-empty titleZh): ${JSON.stringify(s).slice(0, 120)}`);
+      bad++;
+      continue;
+    }
+    incoming.push({
+      url: s.url,
+      titleZh: s.titleZh.trim(),
+      category: s.category || '',
+      tags: Array.isArray(s.tags) ? s.tags : [],
+      summaryZh: typeof s.summaryZh === 'string' ? s.summaryZh : ''
+    });
+  }
+  if (incoming.length === 0) {
+    console.error(`✗ No valid entries in batch file (${batchPath}).`);
+    console.error(`  Batch file KEPT — fix and re-run.`);
+    process.exit(2);
+  }
+
+  // Load existing .summaries.json (if any) and merge by url (overwrite).
+  let existing = [];
+  if (fs.existsSync(summariesPath)) {
+    try {
+      const parsed = JSON.parse(fs.readFileSync(summariesPath, 'utf8'));
+      if (Array.isArray(parsed)) existing = parsed;
+    } catch (e) {
+      console.error(`✗ Existing ${SUMMARIES} is corrupt: ${e.message}`);
+      console.error(`  Refusing to merge — inspect ${summariesPath} manually.`);
+      process.exit(1);
+    }
+  }
+
+  const byUrl = new Map();
+  for (const s of existing) if (isValidEntry(s)) byUrl.set(s.url, s);
+  for (const s of incoming) byUrl.set(s.url, s); // overwrite
+  const merged = [...byUrl.values()];
+
+  fs.writeFileSync(summariesPath, JSON.stringify(merged, null, 2));
+  fs.unlinkSync(batchPath); // batch absorbed — clean up
+
+  // Report against the manifest if present (so the agent knows what's left).
+  const manifestPath = path.join(dayDir, MANIFEST);
+  let manifestCount = null, missingCount = null;
+  if (fs.existsSync(manifestPath)) {
+    try {
+      const m = JSON.parse(fs.readFileSync(manifestPath, 'utf8'));
+      const urls = Array.isArray(m.articles) ? m.articles.map(a => a.url) : [];
+      manifestCount = urls.length;
+      const done = new Set(merged.map(s => s.url));
+      missingCount = urls.filter(u => !done.has(u)).length;
+    } catch (e) { /* manifest is informational only */ }
+  }
+
+  console.error(`✓ Merged ${incoming.length} summar${incoming.length === 1 ? 'y' : 'ies'}${bad ? ` (skipped ${bad} invalid)` : ''}`);
+  console.error(`   ${SUMMARIES}: ${merged.length} total (was ${existing.length})`);
+  if (manifestCount !== null) {
+    console.error(`   Manifest: ${manifestCount} articles | missing: ${missingCount}`);
+    if (missingCount > 0) {
+      console.error(`   Write the next batch (≤4) and re-run, or check gaps:`);
+      console.error(`     node scripts/summary-add.js "${dayDir}" --status`);
+    } else {
+      console.error(`   ✅ All ${manifestCount} covered — run finalize.js:`);
+      console.error(`     node scripts/finalize.js "${dayDir}"`);
+    }
+  }
+}
+
+function main() {
+  const args = process.argv.slice(2);
+  if (args.includes('-h') || args.includes('--help')) {
+    printHelp();
+    process.exit(0);
+  }
+
+  const positional = args.filter(a => !a.startsWith('-'));
+  const flags = args.filter(a => a.startsWith('-'));
+
+  if (positional.length === 0) {
+    console.error('Usage: node summary-add.js <dayDir> [batchFile|--status]');
+    console.error('Run "node summary-add.js --help" for details.');
+    process.exit(1);
+  }
+
+  const dayDir = positional[0];
+  if (!fs.existsSync(dayDir) || !fs.statSync(dayDir).isDirectory()) {
+    console.error(`✗ Day directory does not exist: ${dayDir}`);
+    process.exit(1);
+  }
+
+  if (flags.includes('--status')) {
+    status(dayDir);
+  } else {
+    const batchFile = positional[1] || DEFAULT_BATCH;
+    merge(dayDir, batchFile);
+  }
+}
+
+if (require.main === module) {
+  main();
+}