Refactor: scripts own file I/O, agent only produces summaries

Root cause: harvest.js fetched content but wrote NO files — it dumped a
giant manifest (with full body) to stdout and left all document I/O to
the agent. This caused silent save failures (agent hand-wrote fragile
Node scripts) and token waste (full body in context → 13 python re-parses).

Architecture change — scripts do all deterministic I/O, full body never
enters the agent context:

  Archive:  harvest.js <src> <n>  → writes 原文.md + registry + index
                                     (sentinel placeholder for 中文标题)
                                     + compact manifest (bodyPreview only)
            agent writes .summaries.json (中文标题 + 四段摘要 + 分类)
            finalize.js <dayDir>    → writes 摘要.md, patches sentinels,
                                     updates registry/index, cleans up

  Preview:  harvest.js <src> <n> --preview [--full]  → read-only, no writes

Changes:
- harvest.js: wire up generateOriginalDoc/updateIndex/appendRegistry
  (were defined but never called); compact manifest (no full body);
  sentinel placeholder links; --preview/--full read-only mode; real
  asset count (was "articles with images"); empty-alt guard (no bare **);
  manifest merge across runs (fixes dangling-sentinel data loss);
  --help; module exports for finalize.js
- finalize.js (new): consumes .manifest.json + .summaries.json, writes
  summary docs, patches sentinel links, updates registry/index, cleans
  up temp files; missing summary → warn + skip (non-fatal); --help
- fetcher-direct.js: author extraction via JSON-LD → <meta name=author>
  → rel=author (was matching arbitrary JSON-LD names → empty authors);
  body noise filter (share buttons, boilerplate); image tracking-pixel
  + duplicate filtering
- fetcher-browser.js: CDP port 9223 → 9222 (match docs/chromium launch)
- SKILL.md: 3-step archive workflow, --preview flow, First-Run Setup
  onboarding, guardrails, .summaries.json contract, sentinel docs
- README.md: architecture diagram, division of labor, task flows, limits

README.md

@@ -21,21 +21,30 @@
 ## 工作原理
 
 ```
-config.json ──▶ harvest.js ──▶ fetcher-*.js ──▶ 文章数据(JSON)
+config.json ──▶ harvest.js ──▶ fetcher-*.js ──▶ 文章数据
   (数据源)       (主调度器)      (direct/browser)     │
        ▲              │                               ▼
-       │         dedup 去重                         Agent(LLM)
-       │         (registry)                    翻译/摘要/分类
-                                                   │
-                                                   ▼
-                                              Obsidian vault
+       │         dedup 去重                  写盘: 原文.md + registry + index
+       │         (registry)                  (中文标题用占位哨兵, 全文已落盘)
+       │                                            │
+       │                              stdout: 紧凑 manifest (无全文, 仅预览)
+       │                                            │
+       │                                            ▼
+       │                                       Agent(LLM)
+       │                                  读 manifest → 写 .summaries.json
+       │                                  (中文标题 + 四段摘要 + 分类)
+       │                                            │
+       └────────────────────────────────────────────┘
+                          finalize.js
+            写摘要.md / 修补原文哨兵链接 / 更新 registry+index
 ```
 
 **脚本与 Agent 分工**：
-- **脚本**（`scripts/`）：确定性工作 —— HTTP 请求、HTML 解析、图片下载、去重、文件写入
-- **Agent**（LLM）：智能工作 —— 中英翻译、生成中文摘要、文章分类
+- **脚本**（`scripts/`）：确定性工作 —— HTTP 请求、HTML 解析、图片下载、去重、**写原文.md / registry / index**（中文标题先用哨兵占位，全文直接落盘）
+- **Agent**（LLM）：智能工作 —— 读紧凑 manifest（无全文，仅 `bodyPreview` ~4000 字），生成中文标题 + 四段摘要 + 分类，写入 `.summaries.json`
+- **finalize.js**：读 `.summaries.json`，写中文摘要文档，把原文里的哨兵替换成真实标题，更新 registry/index，清理临时文件
 
-`harvest.js` 完成采集后输出 JSON manifest，提示 Agent 接管翻译与文档生成。
+> 全文 body 由脚本直接写盘，**永不进入 Agent 上下文**——这是降低 token 消耗的关键。
 
 ---
 
@@ -52,7 +61,8 @@ news-harvester/
 │   ├── config-template.json          # config.json 的模板（首次运行自动复制）
 │   └── source-management.md          # 数据源管理详细说明
 ├── scripts/
-│   ├── harvest.js                    # 主入口：node harvest.js <source_id> <count>
+│   ├── harvest.js                    # 主入口：node harvest.js <source_id> <count> [--preview] [--full]
+│   ├── finalize.js                   # 补全中文摘要：node finalize.js <dayDir>
 │   ├── source-manage.js              # 数据源增删改查
 │   ├── fetcher-direct.js             # 纯 HTTP 抓取（开放站点）
 │   └── fetcher-browser.js            # Chromium CDP 抓取（JS 渲染站点）
@@ -129,26 +139,38 @@ node scripts/source-manage.js add \
 
 或让 Agent 自然语言完成：「添加 The Guardian 作为新闻源」。
 
-#### 任务 2：采集最新文章，返回 JSON（不归档）
+#### 任务 2：采集最新文章，返回数据（不归档）
 
-适合只取数据、不落盘的场景。直接调用 fetcher：
+适合只取数据、不落盘的场景。用 `--preview` 一次调用取齐文章清单（标题/日期/作者/url/字数/图片），不写任何文件：
 
 ```bash
-# 抓取首页文章链接
-node -e "require('./scripts/fetcher-direct').fetchHomeLinks('https://theconversation.com/us','theconversation\\\\.com/[\\\\w-]+-\\\\d+$').then(l=>console.log(JSON.stringify(l,null,2)))"
+# 预览最新 3 篇（含正文预览 ~4000 字，无全文）
+node scripts/harvest.js the-conversation 3 --preview
 
-# 抓取单篇文章全文
-node scripts/fetcher-direct.js https://theconversation.com/some-article-12345
+# 预览并附带完整正文（用户明确要全文时）
+node scripts/harvest.js the-conversation 3 --preview --full
 ```
 
-#### 任务 3：完整采集并归档到 Obsidian
+#### 任务 3：完整采集并归档到 Obsidian（三步）
 
 ```bash
+# Step 1: 采集 + 写原文/registry/index（脚本包揽确定性 I/O）
 node scripts/harvest.js the-conversation 5
 ```
 
-`harvest.js` 会：抓取链接 → 去重 → 逐篇抓取全文 → 下载图片 → 输出 JSON manifest。
-然后 Agent 接管：生成中文摘要 → 写入 Obsidian 文档 → 更新 index.md / registry.json。
+脚本输出**紧凑 manifest**（无全文，仅 `bodyPreview`）。Agent 读 manifest 后：
+
+```bash
+# Step 2: Agent 生成中文摘要, 写入 <dayDir>/.summaries.json
+#   [{"url":"...","titleZh":"中文标题","category":"...","tags":[...],"summaryZh":"## 事件概述\n..."}]
+```
+
+```bash
+# Step 3: 补全中文摘要文档 + 修补原文链接 + 更新 registry/index
+node scripts/finalize.js "<dayDir>"
+```
+
+> Agent **不要手写文件 I/O**——原文、registry、index 由 `harvest.js` 写；摘要文档由 `finalize.js` 写。Agent 只产出 `.summaries.json`。
 
 ### 无头模式调用示例
 
@@ -214,13 +236,14 @@ YYYYMMDD/
 ## 限制与注意
 
 - **正文提取依赖正则**：网站改版可能导致提取失败（正文 < 100 字会被跳过）
-- **中文标题需 Agent 翻译**：`harvest.js` 只生成占位标题 `[需翻译] ...`，真正翻译由 Agent 完成
+- **中文标题/摘要由 Agent 生成**：`harvest.js` 在原文/registry/index 中用哨兵 `__SUMMARY_<slug>__` 占位，`finalize.js` 读取 Agent 写的 `.summaries.json` 后替换为真实标题并生成摘要文档
 - **browser 方法依赖 Chromium CDP**（端口 9222），需预先启动：
   ```bash
   chromium --no-sandbox --remote-debugging-port=9222 \
     --user-data-dir=/tmp/chromium-harvest --disable-gpu "about:blank" &>/dev/null &
   ```
 - **图片下载无重试**：失败仅跳过
+- **Token 卫生**：归档任务为「harvest → 写 .summaries.json → finalize」三步；不要为同一源/日重跑 harvest 仅为"多取几篇"（registry 已去重，二次跑只取真正的新文章并安全合并进待处理 manifest）；不要用 Read 反复读原文大文件——manifest 里的 `bodyPreview`（~4000 字）足够写摘要
 
 ---
 

scripts/fetcher-browser.js

@@ -7,7 +7,9 @@
 const http = require('http');
 const WebSocket = require('ws');
 
-const CDP_PORT = parseInt(process.env.CDP_PORT_OVERRIDE || '9223');
+// Chromium CDP debug port — must match the --remote-debugging-port used to start
+// Chromium (9222 per SKILL.md / README). Override via env if you run elsewhere.
+const CDP_PORT = parseInt(process.env.CDP_PORT_OVERRIDE || '9222');
 
 function getWsUrl() {
   return new Promise((resolve, reject) => {

scripts/fetcher-direct.js

@@ -30,6 +30,45 @@ function fetchHtml(url) {
   });
 }
 
+// Extract author names from JSON-LD, then <meta name="author"> (one tag per
+// author, as on The Conversation), then rel="author" links. Returns '' if none.
+function extractAuthors(html) {
+  // 1. JSON-LD author (Reuters/Guardian embed ld+json blocks)
+  const ldBlocks = [...html.matchAll(/<script[^>]*type=["']application\/ld\+json["'][^>]*>([\s\S]*?)<\/script>/gi)];
+  for (const b of ldBlocks) {
+    try {
+      const data = JSON.parse(b[1]);
+      const candidates = [];
+      const collect = (obj) => {
+        if (!obj) return;
+        if (Array.isArray(obj)) { obj.forEach(collect); return; }
+        candidates.push(obj);
+        if (obj['@graph']) collect(obj['@graph']);
+      };
+      collect(data);
+      for (const obj of candidates) {
+        let author = obj.author || (obj.mainEntity && obj.mainEntity.author);
+        if (author) {
+          const names = (Array.isArray(author) ? author : [author])
+            .map(a => typeof a === 'string' ? a : (a && a.name) || '')
+            .filter(Boolean);
+          if (names.length) return [...new Set(names)].slice(0, 3).join(', ');
+        }
+      }
+    } catch (e) {}
+  }
+  // 2. <meta name="author" content="..."> — multiple tags for co-authors
+  const metaMatches = [...html.matchAll(/<meta[^>]+name=["']author["'][^>]+content=["']([^"']+)["']/gi)];
+  if (metaMatches.length) {
+    const names = [...new Set(metaMatches.map(m => m[1].trim()).filter(Boolean))];
+    if (names.length) return names.slice(0, 3).join(', ');
+  }
+  // 3. rel="author" link text
+  const relMatch = html.match(/<a[^>]+rel=["']author["'][^>]*>([^<]+)<\/a>/i);
+  if (relMatch && relMatch[1].trim()) return relMatch[1].trim();
+  return '';
+}
+
 function extractArticle(html, url) {
   // Title
   const titleMatch = html.match(/<title[^>]*>(.*?)<\/title>/is);
@@ -42,16 +81,17 @@ function extractArticle(html, url) {
   const date = dateMatch ? dateMatch[1] : '';
 
   // Authors
-  const authorMatches = html.match(/["']name["']\s*:\s*["']([^"']{3,60})["']/g) || [];
-  const authors = [...new Set(authorMatches.map(m => m.match(/["']name["']\s*:\s*["']([^"']+)["']/)[1])
-    .filter(a => a.length < 60 && !a.includes('{')))].slice(0, 3).join(', ');
+  const authors = extractAuthors(html);
 
-  // Images
+  // Images — skip logos/icons and tracking pixels (e.g. facebook.com/tr), dedupe by src
   const imgMatches = [...html.matchAll(/<img[^>]+src=["'](https?:\/\/[^"']+)["'][^>]*>/gi)];
+  const seenSrc = new Set();
   const imgs = imgMatches.map(m => {
     const altMatch = m[0].match(/alt=["']([^"']*)["']/i);
     return { src: m[1], alt: altMatch ? altMatch[1] : '', width: 500 };
-  }).filter(i => !i.src.includes('logo') && !i.src.includes('icon') && i.src.length > 50).slice(0, 6);
+  }).filter(i => !i.src.includes('logo') && !i.src.includes('icon')
+    && !i.src.includes('facebook.com/tr') && i.src.length > 50
+    && !seenSrc.has(i.src) && seenSrc.add(i.src)).slice(0, 6);
 
   // Article body - try structured extraction
   let body = '';
@@ -66,12 +106,17 @@ function extractArticle(html, url) {
     // Extract all paragraphs, clean HTML
     const paraMatches = [...html.matchAll(/<p[^>]*>(.*?)<\/p>/gis)];
     const noise = ['Sign up', 'Newsletter', 'Subscribe', 'Follow us', 'Read more', 'Click here',
-      'opens new tab', 'Thomson Reuters', 'All quotes delayed', 'Access unmatched'];
+      'opens new tab', 'Thomson Reuters', 'All quotes delayed', 'Access unmatched',
+      'Copy link', 'Bluesky', 'Write an article and join'];
     body = paraMatches.map(m => {
       let text = m[1].replace(/<[^>]+>/g, '').replace(/&nbsp;/g, ' ')
         .replace(/&amp;/g, '&').replace(/&lt;/g, '<').replace(/&gt;/g, '>').trim();
       return text;
-    }).filter(t => t.length > 60 && !noise.some(n => t.includes(n))).join('\n\n');
+    }).filter(t => {
+      const words = t.replace(/\s+/g, ' ').trim().split(/\s+/);
+      // drop share-button blobs / fragments (much whitespace, few real words)
+      return words.length >= 8 && t.length > 60 && !noise.some(n => t.includes(n));
+    }).join('\n\n');
   }
 
   return { title, date, authors, imgs, body: body.substring(0, 15000) };

scripts/finalize.js

+#!/usr/bin/env node
+/**
+ * finalize.js - Patch Chinese summaries into an already-archived day directory.
+ * Usage: node finalize.js <dayDir>
+ *
+ * Reads:
+ *   <dayDir>/.manifest.json   (written by harvest.js — maps url → slug → original filename)
+ *   <dayDir>/.summaries.json  (written by the agent — Chinese title + 4-section summary)
+ *
+ * For each article with a summary:
+ *   1. Write <dayDir>/<中文标题>.md           (the Chinese summary doc)
+ *   2. Patch the sentinel __SUMMARY_<slug>__ in the original 原文.md     (related + footer wikilink)
+ *   3. Update the registry entry (titleZh / category / tags)
+ * Finally rewrites index.md with real titles and removes the two temp files.
+ *
+ * Missing summary for an article → warn and leave its sentinel in place (non-fatal).
+ * Missing .summaries.json → error and exit.
+ */
+
+const fs = require('fs');
+const path = require('path');
+
+const {
+  generateSummaryDoc,
+  updateIndex,
+  sentinelFor,
+  originalFilename
+} = require('./harvest');
+
+function loadJson(p, label) {
+  if (!fs.existsSync(p)) {
+    console.error(`✗ ${label} not found: ${p}`);
+    process.exit(1);
+  }
+  try {
+    return JSON.parse(fs.readFileSync(p, 'utf8'));
+  } catch (e) {
+    console.error(`✗ Failed to parse ${label}: ${e.message}`);
+    process.exit(1);
+  }
+}
+
+function printHelp() {
+  console.log(`finalize.js — Patch Chinese summaries into an archived day directory.
+
+Usage:
+  node finalize.js <dayDir>
+
+Arguments:
+  dayDir   Path to the YYYYMMDD directory produced by harvest.js
+           (e.g. /path/to/vault/The Conversation/20260724)
+
+Prerequisites:
+  - <dayDir>/.manifest.json   (written by harvest.js)
+  - <dayDir>/.summaries.json  (written by the agent — Chinese title + 4-section summary)
+
+What it does:
+  1. Writes <dayDir>/<中文标题>.md for each article with a summary
+  2. Replaces sentinel placeholders in the 原文.md files with real titles
+  3. Updates registry.json and index.md with real titles/categories/tags
+  4. Removes .manifest.json and .summaries.json
+
+Options:
+  -h, --help   Show this help message
+
+Example:
+  node finalize.js "/Users/me/obsidian/Harvester/The Conversation/20260724"`);
+}
+
+function main() {
+  const args = process.argv.slice(2);
+  if (args.includes('-h') || args.includes('--help')) {
+    printHelp();
+    process.exit(0);
+  }
+
+  const dayDir = args.find(a => !a.startsWith('-'));
+  if (!dayDir) {
+    console.error('Usage: node finalize.js <dayDir>');
+    console.error('Example: node finalize.js /path/to/vault/The Conversation/20260724');
+    console.error('Run "node finalize.js --help" for details.');
+    process.exit(1);
+  }
+  if (!fs.existsSync(dayDir) || !fs.statSync(dayDir).isDirectory()) {
+    console.error(`✗ Day directory does not exist: ${dayDir}`);
+    process.exit(1);
+  }
+
+  const manifestPath = path.join(dayDir, '.manifest.json');
+  const summariesPath = path.join(dayDir, '.summaries.json');
+
+  const manifest = loadJson(manifestPath, '.manifest.json');
+  const summaries = loadJson(summariesPath, '.summaries.json');
+
+  const sourceName = manifest.sourceName || 'Unknown';
+  // Index summaries by url for O(1) lookup
+  const summaryByUrl = new Map();
+  for (const s of (Array.isArray(summaries) ? summaries : [])) {
+    if (s && s.url) summaryByUrl.set(s.url, s);
+  }
+
+  console.error(`\n🖌️  Finalizing ${dayDir}`);
+  console.error(`   Articles in manifest: ${manifest.articles.length}`);
+  console.error(`   Summaries provided:   ${summaryByUrl.size}`);
+
+  // Build the registry path (same convention as harvest.js)
+  const regPath = path.join(dayDir, 'registry.json');
+  let registry = { date: path.basename(dayDir), source: manifest.sourceId, articles: [] };
+  if (fs.existsSync(regPath)) {
+    try { registry = JSON.parse(fs.readFileSync(regPath, 'utf8')); } catch (e) {}
+  }
+
+  let patched = 0, skipped = 0;
+
+  for (const article of manifest.articles) {
+    const { url, slug, titleEn, originalFilename: origName } = article;
+    const summary = summaryByUrl.get(url);
+
+    if (!summary) {
+      console.error(`  ⚠️  No summary for: ${titleEn} — leaving sentinel in place`);
+      skipped++;
+      continue;
+    }
+
+    const titleZh = summary.titleZh;
+    if (!titleZh || typeof titleZh !== 'string' || !titleZh.trim()) {
+      console.error(`  ⚠️  Empty titleZh for: ${titleEn} — skipping`);
+      skipped++;
+      continue;
+    }
+
+    const sentinel = sentinelFor(slug);
+    const summaryFilePath = path.join(dayDir, `${titleZh}.md`);
+    const origFilePath = path.join(dayDir, origName || originalFilename(titleEn, sourceName));
+
+    // 1. Write the Chinese summary doc
+    const summaryDoc = generateSummaryDoc({
+      titleZh,
+      titleEn,
+      date: article.date,
+      authors: article.authors,
+      source: url,
+      category: summary.category || article.category,
+      tags: summary.tags || article.tags,
+      summaryZh: summary.summaryZh
+    }, sourceName);
+    fs.writeFileSync(summaryFilePath, summaryDoc);
+
+    // 2. Patch the sentinel in the original doc (frontmatter related + footer wikilink)
+    if (fs.existsSync(origFilePath)) {
+      const orig = fs.readFileSync(origFilePath, 'utf8');
+      // Replace all occurrences of the sentinel with the real title.
+      const patchedOrig = orig.split(sentinel).join(titleZh);
+      fs.writeFileSync(origFilePath, patchedOrig);
+    } else {
+      console.error(`  ⚠️  Original doc missing: ${origName} — summary written but link unpatched`);
+    }
+
+    // 3. Update the registry entry for this url
+    const regEntry = registry.articles.find(a => a.url === url);
+    if (regEntry) {
+      regEntry.titleZh = titleZh;
+      if (summary.category) regEntry.category = summary.category;
+      if (summary.tags) regEntry.tags = summary.tags;
+    }
+
+    patched++;
+    console.error(`  ✓ ${titleEn} → ${titleZh}`);
+  }
+
+  // Persist the patched registry
+  fs.writeFileSync(regPath, JSON.stringify(registry, null, 2));
+
+  // 4. Rewrite index.md using real titles (from the now-updated registry).
+  //    Index links resolve to summary docs where a titleZh exists, else sentinel.
+  const indexArticles = registry.articles.map(a => ({
+    titleZh: a.titleZh,
+    date: a.date || '',
+    category: a.category || '国际',
+    tags: a.tags || []
+  }));
+  updateIndex(dayDir, indexArticles, sourceName);
+
+  // 5. Cleanup temp files
+  try { fs.unlinkSync(manifestPath); } catch (e) {}
+  try { fs.unlinkSync(summariesPath); } catch (e) {}
+
+  console.error(`\n✅ Done: ${patched} finalized, ${skipped} skipped`);
+  console.error(`   Registry + index updated, temp files removed`);
+}
+
+if (require.main === module) {
+  main();
+}