Commit 57447c63 authored by 谢宇轩's avatar 谢宇轩

fix: batch summary writes to avoid output truncation on large harvests

When a harvest produced 15+ articles, asking the agent to emit the entire
.summaries.json in a single response hit the model's output-token ceiling —
the response was truncated and the tail articles silently got no summary
(their 原文 kept the __SUMMARY_<slug>__ sentinel forever). Reproduced every
time a high-volume source ran; ≤9-article sources never hit it.

Add scripts/summary-add.js to incrementally merge small batches (≤4) into
.summaries.json (keyed by url, overwrite-no-dup), with a --status mode that
diffs manifest vs summaries so interrupted runs can resume. Rewrite SKILL.md
Flow A Step 2 + Guardrails to mandate batching, and surface a batchHint in
harvest.js's manifest so the agent sees the guidance on first contact.
parent 8e8fe30b
......@@ -15,6 +15,8 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau
| Preview latest (read-only) | `node scripts/harvest.js <source_id> <count> --preview` |
| Preview with full body | `node scripts/harvest.js <source_id> <count> --preview --full` |
| Finalize Chinese summaries | `node scripts/finalize.js "<dayDir>"` |
| Merge a summary batch | `node scripts/summary-add.js "<dayDir>"` |
| Check summary coverage | `node scripts/summary-add.js "<dayDir>" --status` |
| Add source | `node scripts/source-manage.js add` |
| List sources | `node scripts/source-manage.js list` |
| Edit source | `node scripts/source-manage.js edit <id>` |
......@@ -235,19 +237,20 @@ It prints a **compact manifest** to stdout (no full body — only a `bodyPreview
> **Browser sources and any source with a recipe** are auto-handled: before fetching, `harvest.js` calls `ensure-chromium.js`, which health-checks the CDP port and launches Chromium itself (cross-platform binary discovery, headless by default) if it isn't up — a no-op if already running. No manual `chromium ... &` or `sleep` is needed. To debug launch failures: `node scripts/ensure-chromium.js` (prints binary path, port, and stderr-log tail on failure).
#### Step 2. Generate Chinese summaries → write `.summaries.json`
From the compact manifest, for each article write one entry to `<dayDir>/.summaries.json`:
#### Step 2. Generate Chinese summaries → write `.summaries.json` (batched)
**Why batch:** writing all summaries in a *single* response hits the model's output-token ceiling once you have ~10+ articles — the response is truncated and the remaining articles silently get no summary (their 原文 keeps the `__SUMMARY_<slug>__` sentinel forever). This failure is **not rare** — it reproduces every time a source yields many articles. Avoid it by working in small batches that each get merged into `.summaries.json` by `summary-add.js`.
Each summary entry has this shape:
```json
[
{
"url": "https://theconversation.com/...-286430",
"titleZh": "特朗普新规或拖慢联邦科研",
"category": "科技/AI",
"tags": ["科技/AI", "The Conversation"],
"summaryZh": "## 事件概述\n...\n\n## 背景\n...\n\n## 关键引语\n> \"...\"\n\n## 影响分析\n..."
}
]
{
"url": "https://theconversation.com/...-286430",
"titleZh": "特朗普新规或拖慢联邦科研",
"category": "科技/AI",
"tags": ["科技/AI", "The Conversation"],
"summaryZh": "## 事件概述\n...\n\n## 背景\n...\n\n## 关键引语\n> \"...\"\n\n## 影响分析\n..."
}
```
- `titleZh`: concise Chinese title (≤30 chars)
......@@ -255,6 +258,23 @@ From the compact manifest, for each article write one entry to `<dayDir>/.summar
- `summaryZh`: the 4-section markdown body (事件概述 / 背景 / 关键引语 / 影响分析)
- The `bodyPreview` in the manifest is enough to write the summary. **Do not re-read the full 原文.md** — if you need more, read a targeted slice.
**Procedure** (loop until every article in the manifest has a summary):
1. Let `N` = number of articles in the manifest. If `N ≤ 8`, you may write all entries to `.summaries.json` directly (one Write) and skip to Step 3. Otherwise work in batches of **≤4**.
2. For each batch: use `Write` to create `<dayDir>/.summaries-batch.json` — a JSON array of ≤4 summary entries — then merge it:
```bash
node scripts/summary-add.js "<dayDir>"
```
This merges the batch into `.summaries.json` (keyed by `url`, so re-supplying an entry overwrites it — no duplicates), then deletes the batch file. Repeat per batch.
3. Once all batches are done, verify coverage and fill any gaps:
```bash
node scripts/summary-add.js "<dayDir>" --status
```
Lists done vs. missing against `.manifest.json`. Write any missing articles as another batch and merge. When `--status` reports `missing: 0`, proceed to Step 3.
> **Interrupted run?** Do not re-run `harvest.js` (it dedups via the registry, so it won't re-fetch). Instead run `node scripts/summary-add.js "<dayDir>" --status` to see which articles still need summaries, then resume from there.
#### Step 3. Run finalize (script stitches summaries in)
```bash
node scripts/finalize.js "<dayDir>"
......@@ -276,6 +296,7 @@ No files are written. stdout is a compact JSON list (one call, no re-fetching /
- **`count` is the target number of NEW articles**, not raw links. `harvest.js` already accounts for registry dedup: it scrolls to collect extra candidate links (`count + seen + buffer`, capped) so the *new* count can still reach `count` even when most of the homepage was already archived. If the run returns **fewer than `count`**, the stderr prints `⚠️ … stream may be exhausted` — that's a genuine source limit (no fresh articles to fetch), **not** a bug; re-running won't help. Wait for new articles or broaden the source instead. Don't re-run `harvest.js` for the same source/day just to "get more" — it deduplicates via the registry, so a second run only fetches genuinely new articles (and merges them into the pending `.manifest.json`, which is safe).
- **Never `Read` the 原文.md files** to write summaries — the `bodyPreview` in the manifest is sufficient. Targeted slice reads are OK if truly needed.
- **Summaries in batches of ≤4, never one big response.** A single response holding 10+ summaries hits the output-token ceiling and is truncated — the tail articles silently get no summary and keep the `__SUMMARY_<slug>__` sentinel. Write each batch to `.summaries-batch.json` and merge via `node scripts/summary-add.js "<dayDir>"`; repeat per batch. `summary-add.js --status` shows what's left. This is mandatory for `N > 8`; recommended for any multi-article run.
- **One `finalize.js` call** after writing `.summaries.json`. Do not hand-write file I/O — the scripts handle all original/registry/index/summary writes.
---
......
......@@ -626,7 +626,11 @@ Examples:
nextStep: `Read this manifest, write Chinese summaries to ${path.join(dayDir, '.summaries.json')}, then run: node scripts/finalize.js "${dayDir}"`,
summariesContract: {
path: path.join(dayDir, '.summaries.json'),
format: 'array of { url, titleZh, category, tags[], summaryZh } — one per article url; summaryZh = the 4 sections markdown (事件概述/背景/关键引语/影响分析)'
format: 'array of { url, titleZh, category, tags[], summaryZh } — one per article url; summaryZh = the 4 sections markdown (事件概述/背景/关键引语/影响分析)',
batchHint: `If ${mergedArticles.length} articles is more than ~8, do NOT write all summaries in one response — ` +
`it will hit the output-token ceiling and get truncated. Work in batches of ≤4: write each batch to ` +
`${path.join(dayDir, '.summaries-batch.json')} and merge with "node scripts/summary-add.js \\"${dayDir}\\"" ` +
`(repeat per batch), then run finalize.js once. Check gaps with "node scripts/summary-add.js \\"${dayDir}\\" --status".`
},
articles: mergedArticles
};
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment