feat: support source helper method

README.md

@@ -55,18 +55,25 @@ news-harvester/
 ├── SKILL.md                          # Skill 定义（Agent 加载入口）
 ├── README.md                         # 本文件
 ├── .gitignore
-├── package.json                      # Node 依赖（ws，仅 browser 方法用）
+├── package.json                      # Node 依赖（ws + js-yaml）
 ├── config.json                       # ⚠️ 运行时生成，不入库
+├── helpers/                          # 数据源 recipe（per-source spec，入库共享）
+│   ├── reuters.md                    # 路透社抓取规格（生产用）
+│   └── nikkei-tech.md                # 日经亚洲科技抓取规格（生产用）
 ├── references/
 │   ├── config-template.json          # config.json 的模板（首次运行自动复制）
-│   └── source-management.md          # 数据源管理详细说明
+│   ├── source-management.md          # 数据源管理详细说明
+│   └── helper-example.md             # helper.md 参考示例（带完整字段注释）
 ├── scripts/
 │   ├── harvest.js                    # 主入口：node harvest.js <source_id> <count> [--preview] [--full]
 │   ├── finalize.js                   # 补全中文摘要：node finalize.js <dayDir>
-│   ├── source-manage.js              # 数据源增删改查
+│   ├── source-manage.js              # 数据源增删改查 + recipe scaffold
 │   ├── ensure-chromium.js            # 跨平台自动发现并启动 Chromium（browser 源用）
-│   ├── fetcher-direct.js             # 纯 HTTP 抓取（开放站点）
-│   └── fetcher-browser.js            # Chromium CDP 抓取（JS 渲染站点）
+│   ├── inspect-source.js             # 诊断工具：分析页面结构，辅助编写 helper.md
+│   ├── cdp-client.js                 # 共享 CDP 基元（WebSocket/导航/滚动/求值）
+│   ├── fetcher-helper.js             # spec 驱动抓取（有 recipe 时用）
+│   ├── fetcher-direct.js             # 纯 HTTP 抓取（legacy direct 源）
+│   └── fetcher-browser.js            # Chromium CDP 抓取（legacy browser 源，无 recipe 时用）
 └── data/                             # ⚠️ 运行时去重数据，不入库
     └── registry.json
 ```
@@ -216,10 +223,71 @@ claude -p \
 | `name` | 显示名 | `路透社` |
 | `homepageUrl` | 首页 URL（抓取链接入口） | `https://www.reuters.com/` |
 | `method` | `direct`（纯 HTTP）或 `browser`（需 Chromium） | `direct` |
-| `articleUrlPattern` | 文章 URL 匹配正则 | `theconversation\\.com/[\\w-]+-\\d+$` |
+| `articleUrlPattern` | 文章 URL 匹配正则 — **legacy direct 源专用**；有 recipe(`helpers/<id>.md`) 时被忽略 | `theconversation\\.com/[\\w-]+-\\d+$` |
 | `vaultFolder` | vault 下的子文件夹名 | `The Conversation` |
 | `enabled` | 是否启用 | `true` |
 
+> browser 源应配 recipe（`helpers/<id>.md`）而非依赖 `articleUrlPattern`；recipe 的 `urlPattern`/选择器是解析真相源，详见 `SKILL.md` 的 **Source recipes** 小节。
+
+---
+
+## 数据源 Recipe（`helpers/<id>.md`）
+
+每个数据源可以有一份 **recipe**——`helpers/<source-id>.md` 文件，用 YAML frontmatter 声明如何定位文章列表、如何提取正文与元数据。`harvest.js` 发现 recipe 时用 spec 驱动的 `fetcher-helper.js`；没有 recipe 则回退 legacy fetcher。**`source-manage.js list` 的 `RECIPE` 列显示每个源走哪条路径**（`✅` = 有 recipe，`—` = legacy）。
+
+> **优先级**：有 recipe 时，其 `urlPattern`/选择器是解析真相源，`config.json` 的 `articleUrlPattern` **被忽略**（且不应再出现在该源的 config 条目里）。recipe 也会强制启用 Chromium（因为 `fetcher-helper.js` 驱动浏览器），与 `method` 无关。
+
+### 为什么用 recipe
+
+旧方案靠一个硬编码正则扫描整页 `<a>`，导航/侧栏/页脚的链接混进来，且 SPA 站点懒加载的首屏只有几篇。recipe 用 `listSelector` 把链接采集收窄到主文章流容器、用 `scrollSteps` 触发懒加载、用收紧的 `urlPattern` 校验——直接解决"分析新 CDP 源找不到高质量数据"的问题。recipe 是纯 markdown，站点改版时改文件即可，不动代码。
+
+### 添加新 browser 源（inspect → recipe → verify）
+
+```bash
+# 1. 加入 config
+node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser
+
+# 2. 诊断页面结构（输出按容器分组的链接 + 可粘贴的 recipe 骨架）
+node scripts/inspect-source.js https://example.com/ --scroll 3 --pattern 'example\.com/.+'
+
+# 3. scaffold recipe 模板，然后填入 listSelector + urlPattern
+node scripts/source-manage.js helper mysrc
+#   编辑 helpers/mysrc.md ...
+
+# 4. 只读验证
+node scripts/harvest.js mysrc 3 --preview
+```
+
+### recipe 字段速查
+
+所有字段除 `urlPattern` 外都可选，缺省值 = legacy 行为。完整说明见 `references/source-management.md`（Helper recipes）和带注释的 `references/helper-example.md`。
+
+| 阶段 | 字段 | 作用 |
+|------|------|------|
+| 发现 | `listSelector` | 主文章列表容器 CSS 选择器（收窄范围，排除导航/侧栏）|
+| 发现 | `linkSelector` | 容器内候选链接（默认 `a[href]`）|
+| 发现 | `urlPattern` | **必填**，文章 URL 校验正则，取代 config 的 `articleUrlPattern` |
+| 发现 | `excludeUrlPattern` | 排除正则（如 podcasts/section 页）|
+| 发现 | `scrollSteps` | 向下滚动次数，触发 SPA 懒加载 |
+| 提取 | `bodySelectors` | 正文容器选择器数组，按序首个满足 `bodyMinParagraphs` 即用 |
+| 提取 | `dateSelector` / `authorSelector` | 元数据选择器；找不到时自动回退 JSON-LD |
+| 提取 | `excludeImage` / `imageMinWidth` | 图片过滤（排除 logo/头像/跟踪像素）|
+| 提取 | `titleStripSuffix` | 从 `document.title` 剥离的后缀正则 |
+
+> **正则引用**：regex 字段用单引号包裹（`'reuters\.com/.+'`）。双引号会让 js-yaml 把 `\.` 当非法转义报错。
+>
+> **JSON-LD 兜底**：`dateSelector`/`authorSelector` 找不到时，`fetcher-helper.js` 自动读 `<script type="application/ld+json">` 的 `datePublished`/`author.name`。很多 SPA 站点（如 Nikkei）只在 JSON-LD 放元数据，所以通常不用自定义这两个选择器。
+
+### 管理 recipe
+
+```bash
+node scripts/source-manage.js helper <id>            # scaffold 模板（已存在则拒绝）
+node scripts/source-manage.js helper <id> --force    # 覆盖
+node scripts/source-manage.js helper <id> --show     # 查看 recipe 路径 + 是否存在
+```
+
+recipe 存放在 `helpers/`（入库共享，跨机器一致），区别于 `config.json`（机器特定，不入库）。
+
 ---
 
 ## 归档产物结构

SKILL.md

@@ -5,7 +5,7 @@ description: Harvest and archive news articles from configured sources (Reuters,
 
 # News Harvester Skill
 
-Fetch latest articles from configured news sources, archive them to Obsidian vault with Chinese summaries + original text. Supports browser-based (Chromium CDP) and direct HTTP fetching. Deduplicates via per-day registry with a sliding window.
+Fetch latest articles from configured news sources, archive them to Obsidian vault with Chinese summaries + original text. Supports browser-based (Chromium CDP), direct HTTP, and per-source **recipe** (`helpers/<id>.md`) fetching — a recipe declares the selectors/scroll/URL-filter spec and takes precedence over the legacy fetchers. Deduplicates via per-day registry with a sliding window.
 
 ## Quick Reference
 
@@ -18,6 +18,8 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau
 | Add source | `node scripts/source-manage.js add` |
 | List sources | `node scripts/source-manage.js list` |
 | Edit source | `node scripts/source-manage.js edit <id>` |
+| Scaffold a source recipe | `node scripts/source-manage.js helper <id>` |
+| Inspect a source's page structure | `node scripts/inspect-source.js <url> --scroll 3 [--pattern '<regex>'] [--write <id>]` |
 | Ensure Chromium (browser sources) | `node scripts/ensure-chromium.js` (or `--check`) |
 
 > **All commands assume `cd <skill-root>` first** (the directory containing this `SKILL.md`). Scripts live in the `scripts/` subfolder — **always prefix with `scripts/`**, e.g. `node scripts/harvest.js`, never `node harvest.js` (that fails with `MODULE_NOT_FOUND`).
@@ -54,7 +56,7 @@ node scripts/source-manage.js add --id the-conversation --name "The Conversation
 If the user wants a different first source (e.g. `reuters` with `browser` method), add that instead and note the Chromium prerequisite below applies.
 
 **3. Check prerequisites for the source's `method`.**
-- `direct` → nothing extra.
+- `direct` → nothing extra (plain HTTP via `fetcher-direct.js`).
 - `browser` → `harvest.js` auto-launches Chromium on first browser-source use (see `scripts/ensure-chromium.js`). It cross-platform-discovers the binary (macOS `.app` bundles, Linux PATH, Windows install dirs) — no manual `chromium ... &` needed. To verify it can find a browser without running a full harvest:
   ```bash
   node scripts/ensure-chromium.js --check     # is CDP already up? exit 0/1
@@ -62,6 +64,8 @@ If the user wants a different first source (e.g. `reuters` with `browser` method
   ```
   If discovery can't find your browser, set `config.chromium.executablePath` to its full path (see Configuration).
 
+> **Chromium is launched whenever `method: browser` OR the source has a recipe** (`helpers/<id>.md`). A `direct` source that gains a recipe also uses Chromium, since `fetcher-helper.js` drives the browser. See Source recipes below.
+
 **4. Smoke test (read-only, no vault writes).** Confirm the fetch pipeline works end-to-end before archiving anything:
 ```bash
 node scripts/harvest.js the-conversation 2 --preview
@@ -117,6 +121,35 @@ To point a source at a different vault subfolder, set `sources[].vaultFolder`. A
 
 ---
 
+## Source recipes (`helpers/<id>.md`)
+
+Each source can have a **recipe** — a `helpers/<source-id>.md` file whose YAML frontmatter declares how to find its article list and extract article content (selectors, scroll, URL filters). When `harvest.js` sees one, it uses the spec-driven `fetcher-helper.js` instead of the legacy fetcher. **No recipe → legacy path, zero behavior change** — so existing sources keep working and you migrate one at a time.
+
+> **Precedence**: when a recipe exists, its `urlPattern`/selectors are the source of truth and `config.json`'s `articleUrlPattern` is **ignored**. Sources with a recipe therefore don't need (and shouldn't carry) `articleUrlPattern` in config — it's a legacy-direct-source field only. `source-manage.js list` shows a `RECIPE` column so you can tell at a glance which sources are on which path.
+
+Recipes solve the "can't find high-quality data on new CDP sources" problem: instead of one hard-coded regex scanning the whole page (which mixes nav/sidebar/footer links with real articles), a recipe scopes link collection to a `listSelector` container, triggers lazy-load via `scrollSteps`, and validates URLs with a tightened `urlPattern`. The recipe is just markdown — edit it to adapt to a site redesign without touching code.
+
+**Adding a new browser source** (3-step closed loop: add → inspect --write → verify):
+```bash
+# 1. Add to config (browser sources should get a recipe — articleUrlPattern is legacy-only)
+node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser --folder "My Source"
+
+# 2. Inspect + auto-write the recipe in one shot (measures listSelector + infers urlPattern from real URLs)
+node scripts/inspect-source.js https://example.com/ --scroll 3 --write mysrc
+
+# 3. Verify (read-only)
+node scripts/harvest.js mysrc 3 --preview
+```
+**Convergence check**: preview passes when the fetched links are real articles (not nav/section pages) and `charCount > 500`. If it fails — links are nav/section → tighten `urlPattern` in `helpers/mysrc.md`; too few links → raise `scrollSteps`; empty body → fix `bodySelectors`. Re-run preview after each edit. **Do not hand-edit the recipe blindly** — re-run `inspect` (without `--write`) to see the container groups if unsure.
+
+**Once verified, archive** by following Flow A below (harvest writes originals → you write `.summaries.json` → finalize). The harvest stderr prints the `Day dir` and the exact next command — follow that, don't parse the stdout manifest.
+
+Full schema and authoring notes: `references/source-management.md` (Helper recipes).
+
+> **Roadmap (model B)**: the current split keeps direct sources on legacy `fetcher-direct.js`. The eventual goal is for *every* source to have a recipe (moving `method`/`homepageUrl`/`urlPattern` into the recipe and reducing `config.json` to a pure index of `id`/`name`/`vaultFolder`/`enabled`), with `fetcher-helper.js` gaining a direct-HTTP mode. Direct sources are intentionally untouched this round to avoid disrupting working fetchers.
+
+---
+
 ## Workflow
 
 Two flows depending on intent: **archive** (fetch + save to Obsidian) or **preview** (read-only, just return article data).
@@ -137,7 +170,7 @@ The script fetches links, deduplicates, fetches each article, downloads images,
 
 It prints a **compact manifest** to stdout (no full body — only a `bodyPreview` ~4000 chars per article). Read it.
 
-> **Browser sources** are auto-handled: before fetching, `harvest.js` calls `ensure-chromium.js`, which health-checks the CDP port and launches Chromium itself (cross-platform binary discovery, headless by default) if it isn't up — a no-op if already running. No manual `chromium ... &` or `sleep` is needed. To debug launch failures: `node scripts/ensure-chromium.js` (prints binary path, port, and stderr-log tail on failure).
+> **Browser sources and any source with a recipe** are auto-handled: before fetching, `harvest.js` calls `ensure-chromium.js`, which health-checks the CDP port and launches Chromium itself (cross-platform binary discovery, headless by default) if it isn't up — a no-op if already running. No manual `chromium ... &` or `sleep` is needed. To debug launch failures: `node scripts/ensure-chromium.js` (prints binary path, port, and stderr-log tail on failure).
 
 #### Step 2. Generate Chinese summaries → write `.summaries.json`
 From the compact manifest, for each article write one entry to `<dayDir>/.summaries.json`:

package-lock.json

@@ -9,9 +9,38 @@
       "version": "1.0.0",
       "license": "ISC",
       "dependencies": {
+        "js-yaml": "^5.2.2",
         "ws": "^8.19.0"
       }
     },
+    "node_modules/argparse": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz",
+      "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==",
+      "license": "Python-2.0"
+    },
+    "node_modules/js-yaml": {
+      "version": "5.2.2",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-5.2.2.tgz",
+      "integrity": "sha512-dayzUzKkJ1MkuUtZglSebU43utNXH0OWQByK9rKOOuYIO8M5TV1y+n8ALMdG0rdzBnfNkOmZEqrURepb0ejqBw==",
+      "funding": [
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/puzrin"
+        },
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/nodeca"
+        }
+      ],
+      "license": "MIT",
+      "dependencies": {
+        "argparse": "^2.0.1"
+      },
+      "bin": {
+        "js-yaml": "bin/js-yaml.mjs"
+      }
+    },
     "node_modules/ws": {
       "version": "8.19.0",
       "resolved": "https://registry.npmjs.org/ws/-/ws-8.19.0.tgz",

package.json

@@ -11,6 +11,7 @@
   "license": "ISC",
   "type": "commonjs",
   "dependencies": {
+    "js-yaml": "^5.2.2",
     "ws": "^8.19.0"
   }
 }

references/helper-example.md

+---
+# ── Source identity ──
+# source 必须与 config.json 的 source.id 一致；harvest.js 据此加载本文件。
+source: reuters
+
+# ── 发现阶段：在列表页定位文章链接 ──
+# listSelector: 主文章列表容器的 CSS 选择器。
+#   关键字段——把链接采集范围从"整页"收窄到这个容器，排除导航/侧栏/页脚。
+#   null = 全页扫描（=legacy 行为，不推荐）。用 `node scripts/inspect-source.js <url>` 找到最大容器组。
+listSelector: 'div[data-testid="Title"]'
+
+# linkSelector: 容器内候选 <a> 元素。一般不用改。
+linkSelector: 'a[href]'
+
+# urlPattern: 【必填】文章 URL 校验正则（regex source，不是带斜杠的字面量）。
+#   匹配此正则的 href 才算文章。有 helper 时此字段取代 config.articleUrlPattern（后者被忽略）。
+#   注意：用单引号包裹（双引号会让 js-yaml 把 \. 当非法转义报错）。
+urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$'
+
+# excludeUrlPattern: 排除正则——匹配 urlPattern 但不该算文章的 URL（如 podcasts/newsletter）。
+excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
+
+# scrollSteps: 向下滚动次数，触发 SPA 懒加载。0 = 不滚（只拿首屏）。
+#   Reuters 首屏仅 ~10 篇，滚 3 次能拿全当天列表。
+scrollSteps: 3
+scrollWaitMs: 2000        # 每次滚动后等待毫秒（等新内容渲染）
+waitMs: 15000             # 页面初始加载后等待毫秒（等 JS/React 渲染完）
+maxLinks: 30              # 链接上限
+
+# ── 提取阶段：在文章页定位正文/元数据 ──
+# bodySelectors: 正文容器选择器，按序尝试，首个满足 ≥ bodyMinParagraphs 的即用。
+bodySelectors:
+  - '[data-testid^="paragraph"]'
+  - '[class*="articleBodyContent"] p'
+  - '[class*="article-body"] p'
+  - 'article p'
+bodyMinParagraphs: 3
+
+# dateSelector / authorSelector: CSS 选择器。找不到时自动回退到页面的
+#   <script type="application/ld+json"> 的 datePublished / author.name（JSON-LD 兜底）。
+#   很多 SPA 站点（Nikkei）只在 JSON-LD 放元数据，所以通常不用自定义。
+dateSelector: 'time[datetime]'
+authorSelector: '[rel="author"], [class*="author"] a'
+
+# imageSelector: 图片元素（一般不动）。imageMinWidth 过滤图标/头像/跟踪像素。
+# excludeImage: 图片 URL 子串黑名单（小写匹配）。
+imageSelector: 'img'
+imageMinWidth: 200
+excludeImage:
+  - logo
+  - icon
+  - avatar
+  - profile
+
+# noisePrefixes: 正文段落过滤——以这些前缀开头的 <p> 会被丢弃（如 Reuters 的"Reporting by"署名块）。
+noisePrefixes:
+  - 'Reporting by'
+  - 'Editing by'
+  - 'Access unmatched'
+  - 'Browse an unrivalled'
+  - 'Screen for heightened'
+  - 'Sign up here'
+  - 'Reuters, the news'
+  - 'All quotes delayed'
+  - 'See here for'
+
+# titleStripSuffix: 从 document.title 剥离的后缀正则（清理 "| Reuters" 等）。
+titleStripSuffix: ' \| Reuters$'
+maxBodyChars: 15000
+---
+
+# Reuters 抓取备忘
+
+这一段（frontmatter 下方）是给人看的备忘，脚本不解析。记录：
+- 主文章流容器是 `div[data-testid="Title"]`（inspect 实测 59 篇）。
+- 文章 URL 以日期 `-YYYY-MM-DD/` 结尾，是过滤导航/section 页的关键。
+- 反爬：headless 会被拦，需 config.chromium.headless:false。
+
+---
+# 字段速查（所有字段除 urlPattern 外都可选，缺省值 = legacy 行为）
+#
+# 发现阶段:  listSelector / linkSelector / urlPattern(必填) / excludeUrlPattern
+#            scrollSteps / scrollWaitMs / waitMs / maxLinks
+# 提取阶段:  bodySelectors / bodyMinParagraphs / dateSelector / authorSelector
+#            imageSelector / imageMinWidth / excludeImage / noisePrefixes
+#            titleStripSuffix / maxBodyChars
+#
+# 完整说明见 references/source-management.md 的 "Helper recipes" 小节。
+# 写新 recipe 流程: inspect → scaffold(source-manage helper <id>) → 填字段 → preview 验证

references/source-management.md

@@ -20,11 +20,11 @@ Prompt the user for (or infer from context):
 | `name` | Display name | `路透社` |
 | `homepageUrl` | URL to scrape for links | `https://www.reuters.com/` |
 | `method` | `browser` (needs JS) or `direct` (plain HTTP) | `browser` |
-| `articleUrlPattern` | Regex to match article URLs | `reuters\\.com/.+/\\d{4}-\\d{2}-\\d{2}/` |
+| `articleUrlPattern` | Regex to match article URLs — **legacy direct sources only**; ignored when a recipe (`helpers/<id>.md`) exists | `theconversation\\.com/[\\w-]+-\\d+$` |
 | `vaultFolder` | Subfolder under vaultPath | `路透社` |
 | `enabled` | Start enabled? | `true` |
 
-Auto-detect `articleUrlPattern` if not provided:
+Auto-detect `articleUrlPattern` if not provided (legacy path only — browser sources should get a recipe instead, see Helper recipes below):
 - Reuters: `reuters\\.com/.+/\\d{4}-\\d{2}-\\d{2}/`
 - The Conversation: `theconversation\\.com/.+-\\d+$`
 - Generic: derive from homepage domain + path depth
@@ -42,7 +42,7 @@ Usage: `node source-manage.js edit <id>`
 
 Show current values, prompt for field to change, update `config.json`.
 
-Editable fields: `name`, `homepageUrl`, `method`, `articleUrlPattern`, `vaultFolder`, `enabled`, `countPerRun` (if set as default).
+Editable fields: `name`, `homepageUrl`, `method`, `articleUrlPattern`, `vaultFolder`, `enabled`.
 
 ## Enable / Disable Source
 
@@ -51,3 +51,92 @@ Toggle `enabled` field:
 node source-manage.js enable reuters
 node source-manage.js disable reuters
 ```
+
+---
+
+## Helper recipes (`helpers/<id>.md`)
+
+A source opts into the spec-driven fetcher by having a `helpers/<source-id>.md`
+file. When `harvest.js` finds one, it uses `fetcher-helper.js` (which reads the
+recipe's selectors/scroll/filter) instead of the legacy `fetcher-browser.js`/
+`fetcher-direct.js`. **No helper.md → legacy path, zero behavior change.**
+
+> **Precedence**: when a recipe exists, its `urlPattern`/selectors are the
+> source of truth and `config.json`'s `articleUrlPattern` is **ignored** (and
+> should be omitted from that source's config entry to avoid drift). A recipe
+> also forces Chromium use regardless of `method` (since `fetcher-helper.js`
+> drives the browser). `source-manage.js list` shows a `RECIPE` column (`✅`/
+> `—`) indicating which path each source is on.
+
+### Authoring workflow (for a new browser source)
+
+1. **Add the source** to config (see Add Source above).
+2. **Inspect the page** to discover where article links actually live:
+   ```bash
+   node scripts/inspect-source.js <homepageUrl> --scroll 3 --pattern '<domain>\.com/.+'
+   ```
+   This prints links grouped by container, biggest group first, plus a
+   ready-to-paste helper.md skeleton.
+3. **Scaffold + edit the recipe**:
+   ```bash
+   node scripts/source-manage.js helper <id>     # writes helpers/<id>.md template
+   ```
+   Fill in `listSelector` (from the biggest inspect group) and tighten
+   `urlPattern` (from the sample URLs inspect printed).
+4. **Verify** (read-only):
+   ```bash
+   node scripts/harvest.js <id> 3 --preview
+   ```
+   Confirm the links are real articles (not nav/section pages) and the body
+   char counts look right. Iterate on the recipe until clean.
+
+### Frontmatter schema
+
+All fields optional except `urlPattern`; defaults reproduce legacy behaviour.
+
+**Discovery** (find the article-link list):
+
+| Field | Default | Purpose |
+|-------|---------|---------|
+| `listSelector` | `null` (whole page) | CSS selector for the article-list container — scopes link collection away from nav/sidebar/footer |
+| `linkSelector` | `a[href]` | Candidate links inside each container |
+| `urlPattern` | **required** | Regex source; an href must match to count as an article |
+| `excludeUrlPattern` | `null` | Regex source; drop matching hrefs (section pages, podcasts, etc.) |
+| `scrollSteps` | `0` | Bottom-scroll iterations to trigger lazy-loaded lists |
+| `scrollWaitMs` | `1500` | Pause after each scroll |
+| `waitMs` | `12000` | Initial render wait after load |
+| `maxLinks` | `30` | Link cap |
+
+**Extraction** (single article body + metadata):
+
+| Field | Default | Purpose |
+|-------|---------|---------|
+| `bodySelectors` | (6 selectors) | Tried in order; first yielding ≥ `bodyMinParagraphs` wins |
+| `bodyMinParagraphs` | `3` | Min paragraph count for a selector to qualify |
+| `dateSelector` | `time[datetime]` | Falls back to JSON-LD `datePublished` |
+| `authorSelector` | `[rel="author"], [class*="author"] a` | Falls back to JSON-LD `author.name` |
+| `imageSelector` | `img` | |
+| `imageMinWidth` | `200` | Drops icons/avatars/tracking pixels |
+| `excludeImage` | `["logo","icon","avatar","profile"]` | Substring blocklist on image URL |
+| `noisePrefixes` | (Reuters boilerplate) | Paragraphs starting with these are dropped from body |
+| `titleStripSuffix` | ` \| Reuters\| ...` | Regex source, stripped from `document.title` |
+| `maxBodyChars` | `15000` | Body truncation |
+
+> **Regex quoting**: use single-quoted YAML scalars for regex fields (`'reuters\.com/.+'`).
+> Double quotes make js-yaml reject `\.` as an unknown escape. This matches how
+> you'd write the regex in code.
+>
+> **JSON-LD fallback**: if `dateSelector`/`authorSelector` find nothing,
+> `fetcher-helper.js` automatically reads `<script type="application/ld+json">`
+> (`datePublished` / `author.name`). Many SPA sites (Nikkei) only embed metadata
+> there, so you usually don't need a custom selector.
+
+### Managing recipes
+
+```bash
+node scripts/source-manage.js helper <id>            # scaffold a template (refuses if exists)
+node scripts/source-manage.js helper <id> --force    # overwrite
+node scripts/source-manage.js helper <id> --show     # print path + exists/missing
+```
+Recipes live in `helpers/` (committed to git, shared across machines) — unlike
+`config.json` which is machine-specific.

scripts/cdp-client.js

+/**
+ * cdp-client.js — Shared Chrome DevTools Protocol primitives.
+ *
+ * Previously every CDP caller (fetcher-browser.js) hand-rolled the same
+ * WebSocket boilerplate: open connection, send a command with an id, attach a
+ * message listener matching that id, parse the result. That pattern was
+ * duplicated 4× and the Page.enable call didn't even check the id (resolving
+ * on the first message of any kind). This module is the single source of truth
+ * for those operations so fetcher-helper.js (and a future fetcher-browser
+ * refactor) can build on top.
+ *
+ * Lifecycle (browser launch/discovery) stays in ensure-chromium.js — this
+ * module only handles per-tab CDP commands, assuming a browser is already up.
+ */
+
+const http = require('http');
+const fs = require('fs');
+const path = require('path');
+const WebSocket = require('ws');
+
+const SKILL_DIR = path.dirname(__dirname);
+const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
+const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
+
+// ── Utilities ────────────────────────────────────────────────────────────────
+
+function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
+
+// Monotonic id for CDP commands. Per-tab sessions are sequential, so a simple
+// counter is fine and avoids the random-id collisions the old code risked.
+let _msgId = 0;
+function nextId() { return ++_msgId; }
+
+// ── Port resolution ──────────────────────────────────────────────────────────
+// Precedence: env override > config.chromium.cdpPort > default 9222.
+// (Mirrors the old private resolveCdpPort in fetcher-browser.js.)
+function resolveCdpPort() {
+  if (process.env.CDP_PORT_OVERRIDE) return parseInt(process.env.CDP_PORT_OVERRIDE, 10);
+  try {
+    const cfgPath = fs.existsSync(CONFIG_PATH) ? CONFIG_PATH : TEMPLATE_PATH;
+    const cfg = JSON.parse(fs.readFileSync(cfgPath, 'utf8'));
+    if (cfg.chromium && cfg.chromium.cdpPort) return parseInt(cfg.chromium.cdpPort, 10);
+  } catch (e) { /* fall through to default */ }
+  return 9222;
+}
+
+// ── WebSocket URL discovery ───────────────────────────────────────────────────
+// Fetch /json and pick the first `page` tab's webSocketDebuggerUrl. Retries a
+// few times: right after ensure-chromium spawns the browser, the port may
+// accept connections a beat before /json responds.
+function getWsUrl() {
+  const port = resolveCdpPort();
+  const MAX_ATTEMPTS = 6;
+  const STEP_MS = 500;
+  return new Promise((resolve, reject) => {
+    let attempt = 0;
+    const tryOnce = () => {
+      attempt++;
+      http.get(`http://localhost:${port}/json`, (res) => {
+        let data = '';
+        res.on('data', d => data += d);
+        res.on('end', () => {
+          try {
+            const tabs = JSON.parse(data);
+            // Prefer a page tab to avoid grabbing an iframe/service_worker target.
+            const tab = tabs.find(t => t.type === 'page') || tabs[0];
+            if (!tab || !tab.webSocketDebuggerUrl) {
+              return reject(new Error(`CDP /json returned no page tab on :${port}`));
+            }
+            resolve(tab.webSocketDebuggerUrl);
+          } catch (e) {
+            if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
+            reject(e);
+          }
+        });
+      }).on('error', (e) => {
+        if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
+        reject(new Error(`Cannot reach CDP on :${port} after ${MAX_ATTEMPTS} tries (${e.message}). Run: node scripts/ensure-chromium.js`));
+      });
+    };
+    tryOnce();
+  });
+}
+
+// ── Session management ───────────────────────────────────────────────────────
+
+// Open a WebSocket to the page target and resolve once connected.
+// Returns { ws, close }. The caller owns the lifecycle (close when done).
+function openSession() {
+  return getWsUrl().then(wsUrl => new Promise((resolve, reject) => {
+    const ws = new WebSocket(wsUrl);
+    ws.on('open', () => resolve({ ws, close: () => ws.close() }));
+    ws.on('error', reject);
+  }));
+}
+
+// ── Command sending ──────────────────────────────────────────────────────────
+
+// Send a CDP command and await its response, matched by id. Rejects on CDP
+// error. Concurrent calls are safe — each attaches its own id-matched listener.
+function cdpSend(ws, method, params = {}) {
+  return new Promise((resolve, reject) => {
+    const id = nextId();
+    const handler = (raw) => {
+      const msg = JSON.parse(raw.toString());
+      if (msg.id === id) {
+        ws.removeListener('message', handler);
+        if (msg.error) reject(new Error(`CDP ${method} error: ${JSON.stringify(msg.error)}`));
+        else resolve(msg.result);
+      }
+    };
+    ws.on('message', handler);
+    ws.send(JSON.stringify({ id, method, params }));
+  });
+}
+
+// Evaluate a JS expression in the page and return the value (returnByValue).
+// Returns undefined if the expression yielded no value.
+async function cdpEval(ws, expression) {
+  const result = await cdpSend(ws, 'Runtime.evaluate', {
+    expression,
+    returnByValue: true
+  });
+  return result && result.result && result.result.value;
+}
+
+// ── Navigation ───────────────────────────────────────────────────────────────
+
+// Enable Page domain, navigate to url, wait for loadEventFired (racing a 25s
+// timeout), then sleep waitMs for JS/React to finish rendering.
+// Fixes the old Page.enable bug where ws.once('message') resolved on ANY
+// message instead of the enable response — here cdpSend id-checks it.
+async function navigateAndWait(ws, url, waitMs = 12000) {
+  await cdpSend(ws, 'Page.enable');
+
+  // Attach the load listener BEFORE sending navigate, since loadEventFired can
+  // fire before the navigate response arrives.
+  let loaded = false;
+  const loadPromise = new Promise(resolve => {
+    const handler = (raw) => {
+      const msg = JSON.parse(raw.toString());
+      if (msg.method === 'Page.loadEventFired' && !loaded) {
+        loaded = true;
+        ws.removeListener('message', handler);
+        resolve();
+      }
+    };
+    ws.on('message', handler);
+  });
+
+  await cdpSend(ws, 'Page.navigate', { url });
+  await Promise.race([loadPromise, sleep(25000)]);
+  await sleep(waitMs);
+}
+
+// ── Scrolling + lazy-load collection ─────────────────────────────────────────
+
+// Scroll to the bottom of the page `steps` times, waiting `waitMs` after each
+// scroll, and run `collectExpr` (a JS expression returning a JSON string of an
+// array) after the initial load and after each scroll. Results are merged,
+// deduplicating by `dedupKey` (default: JSON.stringify of the element).
+//
+// Returns the merged array (of parsed objects). This is what lets SPA list
+// pages that lazy-load article cards yield their full set instead of just the
+// first screenful.
+async function scrollAndCollect(ws, { steps = 0, waitMs = 1500, collectExpr, dedupKey }) {
+  const keyFn = dedupKey || (item => JSON.stringify(item));
+  const seen = new Set();
+  const merged = [];
+
+  const absorb = (raw) => {
+    if (!raw) return;
+    let arr;
+    try { arr = JSON.parse(raw); } catch (e) { return; }
+    if (!Array.isArray(arr)) return;
+    for (const item of arr) {
+      const k = keyFn(item);
+      if (!seen.has(k)) { seen.add(k); merged.push(item); }
+    }
+  };
+
+  absorb(await cdpEval(ws, collectExpr));
+
+  for (let i = 0; i < steps; i++) {
+    await cdpEval(ws, 'window.scrollTo(0, document.body.scrollHeight)');
+    await sleep(waitMs);
+    absorb(await cdpEval(ws, collectExpr));
+  }
+
+  return merged;
+}
+
+module.exports = {
+  sleep,
+  resolveCdpPort,
+  getWsUrl,
+  openSession,
+  cdpSend,
+  cdpEval,
+  navigateAndWait,
+  scrollAndCollect
+};

scripts/fetcher-helper.js

+/**
+ * fetcher-helper.js — Spec-driven fetcher for sources with a helper.md recipe.
+ *
+ * Each helper.md is a YAML-frontmattered markdown file whose frontmatter is a
+ * structured spec (selectors / scroll / filter). This module parses that spec
+ * and drives Chromium via the shared cdp-client primitives — no per-source JS.
+ *
+ * Two operations, mirroring the legacy fetcher contract:
+ *   fetchHomeLinks(url, spec) → string[]   (absolute article URLs, deduped)
+ *   fetchPage(url, spec)      → { title, date, authors, imgs, body }
+ *
+ * The 2-arg `(url, spec)` signature aligns with how harvest.js calls legacy
+ * fetchers `(url, ctx)`, so the dispatch branch in harvest.js stays minimal.
+ *
+ * Usage (CLI, for manual testing):
+ *   node fetcher-helper.js <helperPath> links <url>
+ *   node fetcher-helper.js <helperPath> page  <url>
+ */
+
+const fs = require('fs');
+const path = require('path');
+const yaml = require('js-yaml');
+
+const { openSession, navigateAndWait, scrollAndCollect, cdpEval } = require('./cdp-client');
+
+// ── Spec defaults ────────────────────────────────────────────────────────────
+// Defaults reproduce the existing hard-coded behaviour of fetcher-browser.js,
+// so a minimal helper.md (just urlPattern + listSelector) already improves on
+// legacy without forcing the author to redeclare every field.
+
+const DEFAULTS = {
+  // ── Discovery ──
+  listSelector: null,          // CSS selector for the article-list container(s); null = whole page
+  linkSelector: 'a[href]',     // candidate links inside each container
+  urlPattern: null,            // REQUIRED — regex source; article URL validation
+  excludeUrlPattern: null,     // regex source; URLs to drop even if they match urlPattern
+  scrollSteps: 0,              // bottom-scroll iterations to trigger lazy-load
+  scrollWaitMs: 1500,          // pause after each scroll
+  waitMs: 12000,               // initial render wait after load
+  maxLinks: 30,
+
+  // ── Extraction ──
+  bodySelectors: [
+    '[data-testid^="paragraph"]',
+    '[class*="articleBodyContent"] p',
+    '[class*="article-body"] p',
+    '[class*="ArticleBody"] p',
+    '[class*="story-body"] p',
+    'article p'
+  ],
+  bodyMinParagraphs: 3,        // a selector must yield ≥ this many nodes to "win"
+  dateSelector: 'time[datetime]',
+  authorSelector: '[rel="author"], [class*="author"] a',
+  imageSelector: 'img',
+  imageMinWidth: 200,
+  excludeImage: ['logo', 'icon', 'avatar', 'profile'],
+  noisePrefixes: [
+    'Reporting by', 'Editing by', 'Access unmatched', 'Browse an unrivalled',
+    'Screen for heightened', 'Sign up here', 'Reuters, the news',
+    'All quotes delayed', 'See here for'
+  ],
+  titleStripSuffix: ' \\| Reuters| \\| The Conversation| - Bloomberg$',  // regex source, applied with /g
+  maxBodyChars: 15000
+};
+
+// Parse a helper.md file into a fully-defaulted spec object.
+// Throws if frontmatter is missing or urlPattern is absent.
+function loadSpec(helperPath) {
+  const raw = fs.readFileSync(helperPath, 'utf8');
+  const fm = raw.match(/^---\s*\r?\n([\s\S]*?)\r?\n---\s*(?:\r?\n|$)/);
+  if (!fm) throw new Error(`No YAML frontmatter found in ${helperPath}`);
+  const parsed = yaml.load(fm[1]) || {};
+  const spec = { ...DEFAULTS, ...parsed };
+  if (!spec.urlPattern) {
+    throw new Error(`helper.md ${path.basename(helperPath)} missing required 'urlPattern'`);
+  }
+  return spec;
+}
+
+// ── Expression builders ──────────────────────────────────────────────────────
+// Each builder returns a JS expression string to be evaluated in the page via
+// Runtime.evaluate. Spec values are embedded via JSON.stringify so quoting and
+// backslashes survive the round-trip (YAML → js-yaml → JS string → JSON → expr).
+
+function buildDiscoveryExpr(spec) {
+  const listSel = spec.listSelector ? JSON.stringify(spec.listSelector) : 'null';
+  const linkSel = JSON.stringify(spec.linkSelector);
+  const urlPattern = JSON.stringify(spec.urlPattern);
+  const excludePattern = spec.excludeUrlPattern ? JSON.stringify(spec.excludeUrlPattern) : 'null';
+  // listSelector may match one big container OR many card containers; handle
+  // both by iterating every match and collecting links within each.
+  return `(function(){
+    var roots = ${listSel} ? Array.from(document.querySelectorAll(${listSel})) : [document];
+    var urlRe = new RegExp(${urlPattern});
+    var exRe = ${excludePattern} ? new RegExp(${excludePattern}) : null;
+    var out = [], seen = {};
+    for (var r = 0; r < roots.length; r++) {
+      var links = Array.from(roots[r].querySelectorAll(${linkSel}));
+      for (var i = 0; i < links.length; i++) {
+        var href = links[i].href;
+        if (!href) continue;
+        try { var u = new URL(href); u.search = ''; u.hash = ''; href = u.toString(); }
+        catch (e) { continue; }
+        if (!urlRe.test(href)) continue;
+        if (exRe && exRe.test(href)) continue;
+        if (seen[href]) continue;
+        seen[href] = true;
+        out.push({ url: href, text: (links[i].innerText || links[i].textContent || '').trim().substring(0, 200) });
+      }
+    }
+    return JSON.stringify(out);
+  })()`;
+}
+
+function buildExtractExpr(spec) {
+  const bodySelectors = JSON.stringify(spec.bodySelectors);
+  const bodyMinParagraphs = spec.bodyMinParagraphs;
+  const dateSelector = JSON.stringify(spec.dateSelector);
+  const authorSelector = JSON.stringify(spec.authorSelector);
+  const imageMinWidth = spec.imageMinWidth;
+  const excludeImage = JSON.stringify(spec.excludeImage);
+  const noisePrefixes = JSON.stringify(spec.noisePrefixes);
+  const titleStripRe = JSON.stringify(spec.titleStripSuffix);
+  const maxBodyChars = spec.maxBodyChars;
+  return `(function(){
+    var title = document.title.replace(new RegExp(${titleStripRe}, 'g'), '').trim();
+    // JSON-LD fallback: many sites (Nikkei, Reuters) embed datePublished/author
+    // only in <script type="application/ld+json">, not in <time> or visible
+    // author links. Parse it once up front so the CSS-selector results can
+    // fall back to it below.
+    var ld = null;
+    var ldEl = document.querySelector('script[type="application/ld+json"]');
+    if (ldEl) { try { ld = JSON.parse(ldEl.textContent); } catch (e) {} }
+    function ldGet(obj, key) {
+      if (!obj) return null;
+      if (Array.isArray(obj)) { for (var i=0;i<obj.length;i++){ var v=ldGet(obj[i],key); if(v) return v; } return null; }
+      if (obj[key]) return obj[key];
+      if (obj['@graph']) return ldGet(obj['@graph'], key);
+      return null;
+    }
+    var ldDate = ldGet(ld, 'datePublished');
+    var ldAuthor = ldGet(ld, 'author');
+    var ldAuthorNames = [];
+    if (Array.isArray(ldAuthor)) {
+      ldAuthorNames = ldAuthor.map(function(a){ return typeof a === 'string' ? a : (a && a.name) || ''; }).filter(Boolean);
+    } else if (ldAuthor) {
+      ldAuthorNames = [typeof ldAuthor === 'string' ? ldAuthor : (ldAuthor.name || '')].filter(Boolean);
+    }
+
+    var dateEl = document.querySelector(${dateSelector});
+    var date = dateEl ? dateEl.getAttribute('datetime') : '';
+    if (!date && ldDate) date = ldDate;
+    var authors = Array.from(document.querySelectorAll(${authorSelector}))
+      .map(function(a){ return a.innerText.trim(); }).filter(Boolean)
+      .filter(function(v,i,a){ return a.indexOf(v) === i; }).join(', ');
+    if (!authors && ldAuthorNames.length) authors = ldAuthorNames.join(', ');
+    var imgs = Array.from(document.querySelectorAll('img')).map(function(img){
+      return { src: img.src, alt: img.alt || '', width: img.naturalWidth || img.width || 0 };
+    }).filter(function(i){
+      return i.src && i.src.indexOf('http') === 0 && i.width > ${imageMinWidth}
+        && !${excludeImage}.some(function(x){ return i.src.toLowerCase().indexOf(x) !== -1; });
+    }).slice(0, 6);
+    var body = '';
+    var sels = ${bodySelectors};
+    for (var s = 0; s < sels.length; s++) {
+      var els = document.querySelectorAll(sels[s]);
+      if (els.length >= ${bodyMinParagraphs}) {
+        body = Array.from(els).map(function(e){ return e.innerText.trim(); })
+          .filter(function(t){ return t.length > 40; }).join('\\n\\n');
+        break;
+      }
+    }
+    if (!body) {
+      body = Array.from(document.querySelectorAll('p'))
+        .map(function(e){ return e.innerText.trim(); })
+        .filter(function(t){ return t.length > 60
+          && !${noisePrefixes}.some(function(n){ return t.indexOf(n) === 0; }); })
+        .join('\\n\\n');
+    }
+    return JSON.stringify({ title: title, date: date, authors: authors, imgs: imgs, body: body.substring(0, ${maxBodyChars}) });
+  })()`;
+}
+
+// ── Public API ───────────────────────────────────────────────────────────────
+
+// Discover article links on a listing page. Returns absolute URL strings
+// (deduped, query/hash stripped, capped at spec.maxLinks) — the same shape
+// legacy fetchHomeLinks returns so harvest.js needs no changes downstream.
+async function fetchHomeLinks(url, spec) {
+  const { ws, close } = await openSession();
+  try {
+    await navigateAndWait(ws, url, spec.waitMs);
+    const items = await scrollAndCollect(ws, {
+      steps: spec.scrollSteps,
+      waitMs: spec.scrollWaitMs,
+      collectExpr: buildDiscoveryExpr(spec),
+      dedupKey: item => item.url
+    });
+    return items.map(i => i.url).slice(0, spec.maxLinks);
+  } finally {
+    close();
+  }
+}
+
+// Extract a single article's content. Returns { title, date, authors, imgs, body }
+// or null if evaluation yielded nothing.
+async function fetchPage(url, spec) {
+  const { ws, close } = await openSession();
+  try {
+    await navigateAndWait(ws, url, spec.waitMs);
+    const result = await cdpEval(ws, buildExtractExpr(spec));
+    return result ? JSON.parse(result) : null;
+  } finally {
+    close();
+  }
+}
+
+module.exports = { loadSpec, fetchHomeLinks, fetchPage };
+
+// ── CLI (manual testing) ─────────────────────────────────────────────────────
+//   node fetcher-helper.js helpers/reuters.md links https://www.reuters.com/
+//   node fetcher-helper.js helpers/reuters.md page  https://www.reuters.com/world/...
+if (require.main === module) {
+  const [helperPath, mode, url] = process.argv.slice(2);
+  if (!helperPath || !mode || !url) {
+    console.error('Usage: node fetcher-helper.js <helperPath> links|page <url>');
+    process.exit(1);
+  }
+  const spec = loadSpec(helperPath);
+  console.error(`Loaded spec: ${path.basename(helperPath)} (urlPattern=${spec.urlPattern}, listSelector=${spec.listSelector || '(whole page)'})`);
+  const fn = mode === 'links' ? fetchHomeLinks : fetchPage;
+  fn(url, spec).then(data => {
+    process.stdout.write(JSON.stringify(data, null, 2));
+  }).catch(e => { console.error('Error:', e.message); process.exit(1); });
+}

scripts/harvest.js

@@ -360,13 +360,27 @@ Examples:
   console.error(`\n🗞️  Harvesting ${source.name} (max ${count} articles) — ${mode}`);
   console.error(`   Method: ${source.method}`);
 
-  const fetcher = source.method === 'browser'
+  // Strategy dispatch: if a helpers/<id>.md recipe exists for this source, use
+  // the spec-driven fetcher-helper; otherwise fall back to the legacy fetcher
+  // for its method. Existing sources keep working unchanged; a source opts in
+  // simply by gaining a helper.md file. See references/source-management.md
+  // (Helper recipes) for the schema.
+  const helperPath = path.join(SKILL_DIR, 'helpers', `${source.id}.md`);
+  let fetcher, spec = null;
+  if (fs.existsSync(helperPath)) {
+    const fetcherHelper = require('./fetcher-helper');
+    spec = fetcherHelper.loadSpec(helperPath);
+    fetcher = fetcherHelper;
+    console.error(`   Recipe: helpers/${source.id}.md (listSelector=${spec.listSelector || 'whole page'}, scroll=${spec.scrollSteps})`);
+  } else {
+    fetcher = source.method === 'browser'
       ? require('./fetcher-browser')
       : require('./fetcher-direct');
+  }
 
-  // For browser sources, ensure a Chromium CDP endpoint is up before fetching.
-  // ensure-chromium.js is a no-op if already running, so this is safe every run.
-  if (source.method === 'browser') {
+  // CDP-based fetchers (browser method, or any source with a helper recipe)
+  // need a Chromium endpoint. ensure-chromium.js is a no-op if already running.
+  if (source.method === 'browser' || spec) {
     const { ensureChromium } = require('./ensure-chromium');
     console.error('   Ensuring Chromium (CDP)...');
     try {
@@ -378,6 +392,17 @@ Examples:
     }
   }
 
+  // Call-site adapters: legacy fetchers take (url, pattern) / (url[, waitMs]),
+  // while fetcher-helper takes (url, spec). The closures hide that difference
+  // so the main flow below is identical for both paths. NB: spec must NOT be
+  // passed to legacy fetchPage (it would be coerced to waitMs=NaN).
+  const getLinks = () => spec
+    ? fetcher.fetchHomeLinks(source.homepageUrl, spec)
+    : fetcher.fetchHomeLinks(source.homepageUrl, source.articleUrlPattern);
+  const getPage = (url) => spec
+    ? fetcher.fetchPage(url, spec)
+    : fetcher.fetchPage(url);
+
   // Build dedup set (skip in preview — preview shows latest regardless)
   const seen = preview ? new Set() : buildDedupSet(config, sourceId);
   console.error(`   Registry: ${seen.size} URLs in last ${config.registryWindowDays || 7} days`);
@@ -386,7 +411,7 @@ Examples:
   console.error(`\n📡 Fetching homepage: ${source.homepageUrl}`);
   let links;
   try {
-    links = await fetcher.fetchHomeLinks(source.homepageUrl, source.articleUrlPattern);
+    links = await getLinks();
   } catch (e) {
     console.error('Failed to fetch homepage:', e.message);
     process.exit(1);
@@ -408,7 +433,7 @@ Examples:
       console.error(`\n[${i+1}/${newLinks.length}] ${url}`);
       let articleData;
       try {
-        articleData = await fetcher.fetchPage(url);
+        articleData = await getPage(url);
       } catch (e) {
         console.error(`  ✗ Fetch failed: ${e.message}`);
         continue;
@@ -463,7 +488,7 @@ Examples:
 
     let articleData;
     try {
-      articleData = await fetcher.fetchPage(url);
+      articleData = await getPage(url);
     } catch (e) {
       console.error(`  ✗ Fetch failed: ${e.message}`);
       continue;
@@ -569,8 +594,8 @@ Examples:
   // stdout shows only THIS run's new articles (the agent summarizes what it just fetched)
   process.stdout.write(JSON.stringify({ ...manifest, articles: newArticles }, null, 2));
   console.error(`\n✅ Done: ${processed.length} processed this run, ${skipped} skipped`);
-  console.error(`   Manifest (${mergedArticles.length} total pending): ${manifestPath}`);
-  console.error(`   Next: generate .summaries.json, then run finalize.js`);
+  console.error(`   Day dir: ${dayDir}`);
+  console.error(`   Next: write .summaries.json to ${path.join(dayDir, '.summaries.json')}, then run: node scripts/finalize.js "${dayDir}"`);
 }
 
 // Run only when invoked directly (not when required by finalize.js)

scripts/inspect-source.js

+#!/usr/bin/env node
+/**
+ * inspect-source.js — Diagnose a news source's page structure to author a
+ * helper.md recipe. This is the tool you reach for when adding a new browser
+ * source: it shows you WHERE article links live on the page so you can pick a
+ * good listSelector and urlPattern instead of guessing.
+ *
+ * Usage:
+ *   node scripts/inspect-source.js <url> [--pattern REGEX] [--scroll N] [--wait MS] [--selector CSS]
+ *
+ * What it does:
+ *   1. Launches Chromium (via ensure-chromium) and navigates to <url>.
+ *   2. Optionally scrolls N times (default 0) to trigger lazy-loaded lists.
+ *   3. Collects every <a[href]> on the page, grouped by the "container
+ *      signature" of its nearest meaningful ancestor — a short label derived
+ *      from the ancestor's tag + data-testid/class/id. This reveals which page
+ *      region holds the real article stream (usually one big group) vs.
+ *      nav/footer/sidebars (many small groups).
+ *   4. Filters links by --pattern if given.
+ *   5. Prints each group: count + a sample of link texts and URLs, sorted by
+ *      count descending (the biggest group is almost always the main list).
+ *   6. Prints a ready-to-paste helper.md skeleton using the largest matching
+ *      group's container signature as listSelector.
+ *
+ * Examples:
+ *   node scripts/inspect-source.js https://www.reuters.com/ --scroll 3
+ *   node scripts/inspect-source.js https://asia.nikkei.com/business/tech --pattern 'nikkei\\.com/.+'
+ *   node scripts/inspect-source.js https://www.reuters.com/ --selector 'main'
+ *
+ * Exit code 0 on success, 1 on error.
+ */
+
+const fs = require('fs');
+const path = require('path');
+
+const SKILL_DIR = path.dirname(__dirname);
+const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
+const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
+
+function loadConfig() {
+  const cfgPath = fs.existsSync(CONFIG_PATH) ? CONFIG_PATH : TEMPLATE_PATH;
+  return JSON.parse(fs.readFileSync(cfgPath, 'utf8'));
+}
+
+const { openSession, navigateAndWait, scrollAndCollect, cdpEval, sleep } = require('./cdp-client');
+
+// ── Arg parsing ──────────────────────────────────────────────────────────────
+
+function parseArgs(argv) {
+  const out = { scroll: 0, wait: 12000, pattern: null, selector: null, url: null, write: null, force: false };
+  const positional = [];
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === '--scroll') out.scroll = parseInt(argv[++i] || '0', 10);
+    else if (a === '--wait') out.wait = parseInt(argv[++i] || '12000', 10);
+    else if (a === '--pattern') out.pattern = argv[++i];
+    else if (a === '--selector') out.selector = argv[++i];
+    else if (a === '--write') out.write = argv[++i];
+    else if (a === '--force') out.force = true;
+    else if (a === '-h' || a === '--help') { out.help = true; }
+    else positional.push(a);
+  }
+  out.url = positional[0];
+  return out;
+}
+
+function printHelp() {
+  console.log(`inspect-source.js — Diagnose a news source's page structure for helper.md authoring.
+
+Usage:
+  node scripts/inspect-source.js <url> [options]
+
+Options:
+  --pattern REGEX   Only show links whose URL matches this regex (try the site's article URL shape)
+  --scroll N        Scroll to the bottom N times to trigger lazy-loaded lists (default 0)
+  --wait MS         Initial render wait in ms (default 12000)
+  --selector CSS    Restrict link collection to descendants of this container (default: whole page)
+  --write <id>      Write a ready-to-use helpers/<id>.md using the measured listSelector + urlPattern
+                    (source <id> must already exist in config.json; refuses if helper exists unless --force)
+  --force           With --write, overwrite an existing helpers/<id>.md
+  -h, --help        Show this help
+
+Output:
+  - Links grouped by container signature, biggest group first
+  - With --write: writes helpers/<id>.md and prints a one-line confirmation + next step
+  - Without --write: prints a ready-to-paste helper.md skeleton
+
+Examples:
+  node scripts/inspect-source.js https://www.reuters.com/ --scroll 3
+  node scripts/inspect-source.js https://www.reuters.com/markets/ --scroll 3 --write reuters-markets
+  node scripts/inspect-source.js https://asia.nikkei.com/business/tech --pattern 'nikkei\\\\.com/.+'`);
+}
+
+// ── Page-side collection expression ──────────────────────────────────────────
+// Returns a JSON string of [{ url, text, container }]. `container` is a short
+// signature of the nearest ancestor that has an identifying attribute, so links
+// from the same article stream share a container label and cluster together.
+
+function buildCollectExpr(rootSelector, patternStr) {
+  const rootExpr = rootSelector ? JSON.stringify(rootSelector) : 'null';
+  const patExpr = patternStr ? JSON.stringify(patternStr) : 'null';
+  return `(function(){
+    var roots = ${rootExpr} ? Array.from(document.querySelectorAll(${rootExpr})) : [document];
+    var pat = ${patExpr} ? new RegExp(${patExpr}) : null;
+    var out = [];
+    // Build a short signature for an element: tag + first useful attr.
+    function sig(el){
+      if (!el || el === document.body) return 'body';
+      var t = el.tagName.toLowerCase();
+      var dt = el.getAttribute('data-testid');
+      if (dt) return t + '[data-testid="' + dt + '"]';
+      var id = el.id;
+      if (id) return t + '#' + id;
+      var cls = (el.className && typeof el.className === 'string')
+        ? el.className.trim().split(/\\s+/)[0] : '';
+      if (cls) return t + '.' + cls;
+      return sig(el.parentElement) || t;
+    }
+    for (var r = 0; r < roots.length; r++) {
+      var links = Array.from(roots[r].querySelectorAll('a[href]'));
+      for (var i = 0; i < links.length; i++) {
+        var href = links[i].href;
+        if (!href || href.indexOf('http') !== 0) continue;
+        if (pat && !pat.test(href)) continue;
+        var text = (links[i].innerText || links[i].textContent || '').trim().replace(/\\s+/g,' ').substring(0, 120);
+        // Walk up to the nearest ancestor with an identifying attribute (skip <a> itself).
+        var node = links[i].parentElement;
+        var c = '';
+        var hops = 0;
+        while (node && hops < 6) {
+          var s = sig(node);
+          if (s !== node.tagName.toLowerCase() && s !== 'body') { c = s; break; }
+          node = node.parentElement; hops++;
+        }
+        if (!c) c = sig(links[i].parentElement) || 'body';
+        out.push({ url: href, text: text, container: c });
+      }
+    }
+    return JSON.stringify(out);
+  })()`;
+}
+
+// ── Reporting ────────────────────────────────────────────────────────────────
+
+function groupByContainer(items) {
+  const groups = {};
+  for (const it of items) {
+    (groups[it.container] = groups[it.container] || []).push(it);
+  }
+  return Object.entries(groups)
+    .map(([container, links]) => ({ container, links, count: links.length }))
+    .sort((a, b) => b.count - a.count);
+}
+
+function printGroups(groups, maxGroups, samplePerGroup) {
+  if (!groups.length) {
+    console.log('\n  (no links found — try --scroll, a longer --wait, or a different --selector)');
+    return;
+  }
+  console.log(`\n📦 ${groups.length} container group(s), ${groups.reduce((n,g)=>n+g.count,0)} link(s) total\n`);
+  for (let i = 0; i < Math.min(groups.length, maxGroups); i++) {
+    const g = groups[i];
+    console.log(`  ┌─ [${g.count}] ${g.container}`);
+    for (const l of g.links.slice(0, samplePerGroup)) {
+      const label = l.text ? `"${l.text}"` : '(no text)';
+      console.log(`  │   ${label}`);
+      console.log(`  │   → ${l.url}`);
+    }
+    if (g.count > samplePerGroup) console.log(`  │   … +${g.count - samplePerGroup} more`);
+    console.log(`  └─`);
+  }
+}
+
+// Build a complete helper.md content string from measured data.
+// Uses the largest group's container as listSelector and infers a urlPattern
+// from the sample URLs. The rest of the fields use the same defaults as
+// `source-manage.js helper` so the file is immediately usable — the agent
+// (or human) only needs to verify via preview and tune if needed.
+function buildHelperContent(groups, patternStr, url, sourceId, sourceName) {
+  // Pick the best container: skip obvious nav/header/footer/menu groups, then
+  // take the largest remaining. Without this, a nav dropdown (50+ section
+  // links) would win over the real article stream (which may be split across
+  // several card-type containers of 6-10 links each).
+  const NAVISH = /nav|header|footer|dropdown|menu|breadcrumb|pagination|ticker|sidebar/i;
+  let best = groups.find(g => !NAVISH.test(g.container));
+  if (!best) best = groups[0]; // fall back to overall largest if everything looks navish
+  if (!best) return null;
+  // Infer a urlPattern: take the first matching link, strip trailing slug,
+  // keep domain + path to the last segment that looks like an article id.
+  const sampleUrl = best.links[0].url;
+  let inferredPattern = '';
+  try {
+    const u = new URL(sampleUrl);
+    const segs = u.pathname.split('/').filter(Boolean);
+    if (segs.length) {
+      // Heuristic: drop the last segment (likely the article slug) and anchor
+      // on the path prefix. Author should tighten this in the real helper.md.
+      const prefix = segs.slice(0, -1).join('/');
+      inferredPattern = prefix
+        ? `${u.host.replace(/\./g, '\\.')}/${prefix}/.+`
+        : `${u.host.replace(/\./g, '\\.')}/.+`;
+    }
+  } catch (e) { /* leave empty */ }
+
+  const listSel = best.container;
+  const finalPattern = patternStr || inferredPattern;
+  const displayName = sourceName || sourceId;
+  const content = `---
+source: ${sourceId}
+listSelector: ${JSON.stringify(listSel)}
+linkSelector: 'a[href]'
+urlPattern: '${finalPattern}'
+excludeUrlPattern: null
+scrollSteps: ${patternStr ? 0 : 3}
+scrollWaitMs: 2000
+waitMs: 15000
+bodySelectors:
+  - '[data-testid^="paragraph"]'
+  - '[class*="articleBodyContent"] p'
+  - '[class*="article-body"] p'
+  - 'article p'
+bodyMinParagraphs: 3
+dateSelector: 'time[datetime]'
+authorSelector: '[rel="author"], [class*="author"] a'
+imageSelector: 'img'
+imageMinWidth: 200
+excludeImage:
+  - logo
+  - icon
+  - avatar
+  - profile
+titleStripSuffix: ''
+maxBodyChars: 15000
+---
+# ${displayName} 抓取备忘
+
+- 主文章流容器 \`${listSel}\` 实测 ${best.count} 个候选链接（inspect 自动生成）。
+- urlPattern \`${finalPattern}\` 由样本 URL 推断，如 preview 漏抓/误抓请手编收紧。
+- 首屏可能不全，scrollSteps=${patternStr ? 0 : 3} 触发懒加载；反爬严重时调高 waitMs。
+- 验证: node scripts/harvest.js ${sourceId} 3 --preview（链接为真实文章且 charCount>500 即通过）`;
+  return { content, best };
+}
+
+// ── Main ─────────────────────────────────────────────────────────────────────
+
+async function main() {
+  const args = parseArgs(process.argv.slice(2));
+  if (args.help || !args.url) { printHelp(); process.exit(args.help ? 0 : 1); }
+
+  const config = loadConfig();
+
+  // --write <id>: validate the source exists in config before doing any browser work.
+  let writeSource = null;
+  if (args.write) {
+    writeSource = config.sources.find(s => s.id === args.write);
+    if (!writeSource) {
+      console.error(`✗ Source "${args.write}" not found in config.json. Add it first:\n  node scripts/source-manage.js add --id ${args.write} --name "..." --url ${args.url} --method browser`);
+      process.exit(1);
+    }
+  }
+
+  // Ensure Chromium is up (same path as harvest.js).
+  const { ensureChromium } = require('./ensure-chromium');
+  console.error(`🔬 Inspecting: ${args.url}`);
+  console.error(`   scroll=${args.scroll} wait=${args.wait} pattern=${args.pattern || '(none)'} selector=${args.selector || '(whole page)'}${args.write ? ` write=${args.write}` : ''}`);
+  console.error('   Ensuring Chromium (CDP)...');
+  try {
+    const r = await ensureChromium(config);
+    console.error(`   CDP ready on :${r.port}${r.alreadyRunning ? ' (already running)' : ` (pid ${r.pid})`}`);
+  } catch (e) {
+    console.error(`   Chromium startup failed:\n${e.message}`);
+    process.exit(1);
+  }
+
+  const { ws, close } = await openSession();
+  try {
+    console.error(`   Navigating + rendering (${args.wait}ms)...`);
+    await navigateAndWait(ws, args.url, args.wait);
+
+    const items = await scrollAndCollect(ws, {
+      steps: args.scroll,
+      waitMs: 1500,
+      collectExpr: buildCollectExpr(args.selector, args.pattern),
+      dedupKey: it => it.url
+    });
+
+    const groups = groupByContainer(items);
+    printGroups(groups, 15, 4);
+
+    const built = buildHelperContent(groups, args.pattern, args.url, args.write || '<source-id>', writeSource && writeSource.name);
+    if (!built) return;
+    const { content, best } = built;
+
+    if (args.write) {
+      // Write a ready-to-use helper.md, refusing if it exists without --force.
+      const helperPath = path.join(SKILL_DIR, 'helpers', `${args.write}.md`);
+      if (fs.existsSync(helperPath) && !args.force) {
+        console.error(`\n✗ helpers/${args.write}.md already exists. Use --force to overwrite, or edit it manually.`);
+        process.exit(1);
+      }
+      if (!fs.existsSync(path.join(SKILL_DIR, 'helpers'))) fs.mkdirSync(path.join(SKILL_DIR, 'helpers'), { recursive: true });
+      fs.writeFileSync(helperPath, content);
+      console.error(`\n✅ Wrote helpers/${args.write}.md (listSelector=${best.container}, ${best.count} links measured)`);
+      console.error(`   Next: node scripts/harvest.js ${args.write} 3 --preview   # verify links are real articles + charCount>500`);
+    } else {
+      console.log(`\n📝 Suggested helper.md skeleton (best group: ${best.count} links in ${best.container}):\n`);
+      console.log(content);
+      console.log(`\n   Save to helpers/<source-id>.md (or re-run with --write <id>), then test: node scripts/harvest.js <source-id> 3 --preview`);
+    }
+  } finally {
+    close();
+  }
+}
+
+main().catch(e => { console.error('Fatal:', e.message); process.exit(1); });

scripts/source-manage.js

@@ -7,6 +7,8 @@
  *   node source-manage.js edit <id> --field value
  *   node source-manage.js enable <id>
  *   node source-manage.js disable <id>
+ *   node source-manage.js helper <id>          # scaffold/overwrite helpers/<id>.md
+ *   node source-manage.js helper <id> --show    # print current helper.md path + status
  */
 
 const fs = require('fs');
@@ -15,6 +17,7 @@ const path = require('path');
 const SKILL_DIR = path.dirname(__dirname);
 const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
 const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
+const HELPERS_DIR = path.join(SKILL_DIR, 'helpers');
 
 function loadConfig() {
   if (!fs.existsSync(CONFIG_PATH)) {
@@ -50,18 +53,21 @@ const config = loadConfig();
 switch (command) {
   case 'list': {
     console.log('\n📋 Configured Sources\n');
-    console.log('ID'.padEnd(20) + 'NAME'.padEnd(20) + 'METHOD'.padEnd(10) + 'STATUS');
-    console.log('─'.repeat(60));
+    console.log('ID'.padEnd(20) + 'NAME'.padEnd(20) + 'METHOD'.padEnd(10) + 'RECIPE'.padEnd(10) + 'STATUS');
+    console.log('─'.repeat(70));
     for (const s of config.sources) {
       const status = s.enabled ? '✅ enabled' : '❌ disabled';
+      const hasRecipe = fs.existsSync(path.join(HELPERS_DIR, `${s.id}.md`));
+      const recipe = hasRecipe ? '✅' : '—';
       console.log(
         s.id.padEnd(20) +
         s.name.padEnd(20) +
         s.method.padEnd(10) +
+        recipe.padEnd(10) +
         status
       );
     }
-    console.log();
+    console.log('\n   RECIPE ✅ = helpers/<id>.md exists (spec-driven fetcher); — = legacy fetcher\n');
     break;
   }
 
@@ -104,6 +110,11 @@ switch (command) {
     console.log(`   Method: ${newSource.method}`);
     console.log(`   URL Pattern: ${newSource.articleUrlPattern}`);
     console.log(`   Vault Folder: ${newSource.vaultFolder}`);
+    if (newSource.method === 'browser') {
+      console.log(`\n   💡 browser sources should get a recipe (articleUrlPattern is legacy-only):`);
+      console.log(`      node scripts/inspect-source.js ${newSource.homepageUrl} --scroll 3 --write ${id}`);
+      console.log(`      node scripts/harvest.js ${id} 3 --preview   # verify, then archive`);
+    }
     console.log(`\n   Run: node harvest.js ${id} 5\n`);
     break;
   }
@@ -147,13 +158,84 @@ switch (command) {
     break;
   }
 
+  case 'helper': {
+    const id = args._0;
+    const source = config.sources.find(s => s.id === id);
+    if (!source) { console.error(`Source "${id}" not found`); process.exit(1); }
+    const helperPath = path.join(HELPERS_DIR, `${id}.md`);
+
+    if (args.show !== undefined) {
+      const exists = fs.existsSync(helperPath);
+      console.log(`\n   helper: ${helperPath}`);
+      console.log(`   status: ${exists ? '✅ exists' : '❌ none (source uses legacy fetcher)'}`);
+      break;
+    }
+
+    // Scaffold a helper.md template. Does NOT overwrite without --force.
+    if (fs.existsSync(helperPath) && args.force === undefined) {
+      console.error(`helpers/${id}.md already exists. Use --force to overwrite.`);
+      process.exit(1);
+    }
+    if (!fs.existsSync(HELPERS_DIR)) fs.mkdirSync(HELPERS_DIR, { recursive: true });
+    // Derive a starter urlPattern from the source's homepage domain.
+    let domainPattern = 'example\\.com/.+';
+    try {
+      const u = new URL(source.homepageUrl);
+      domainPattern = u.hostname.replace('www.', '').replace(/\./g, '\\.') + '/.+';
+    } catch (e) { /* keep default */ }
+
+    const template = `---
+source: ${id}
+listSelector: null
+linkSelector: 'a[href]'
+urlPattern: '${domainPattern}'
+excludeUrlPattern: null
+scrollSteps: 0
+scrollWaitMs: 1500
+waitMs: 12000
+bodySelectors:
+  - '[class*="article-body"] p'
+  - 'article p'
+bodyMinParagraphs: 3
+dateSelector: 'time[datetime]'
+authorSelector: '[rel="author"], [class*="author"] a'
+imageSelector: 'img'
+imageMinWidth: 200
+excludeImage:
+  - logo
+  - icon
+  - avatar
+  - profile
+titleStripSuffix: ''
+maxBodyChars: 15000
+---
+# ${source.name} 抓取备忘
+
+- 先运行 inspect 定位真实结构:
+  node scripts/inspect-source.js ${source.homepageUrl} --scroll 3 --pattern '${domainPattern}'
+- 用 inspect 输出的最大容器签名填入 listSelector
+- 用 inspect 输出的 URL 样本收紧 urlPattern
+- 验证: node scripts/harvest.js ${id} 3 --preview
+`;
+    fs.writeFileSync(helperPath, template);
+    console.log(`\n✅ Scaffolded helpers/${id}.md`);
+    console.log(`   Next: run inspect, then fill in listSelector + urlPattern:`);
+    console.log(`   node scripts/inspect-source.js ${source.homepageUrl} --scroll 3 --pattern '${domainPattern}'\n`);
+    break;
+  }
+
   default:
     console.log(`
 Usage:
   node source-manage.js list
-  node source-manage.js add --id <id> --name <name> --url <homepage> --method browser|direct
-  node source-manage.js edit <id> --name "New Name" --channel 123456
+  node source-manage.js add --id <id> --name <name> --url <homepage> --method browser|direct [--pattern REGEX] [--folder NAME]
+  node source-manage.js edit <id> --name "New Name" --url https://... --method browser
   node source-manage.js enable <id>
   node source-manage.js disable <id>
+  node source-manage.js helper <id> [--force|--show]   # scaffold a helpers/<id>.md recipe
+
+Helper recipes: a source opts into the spec-driven fetcher by having a
+helpers/<id>.md file. Scaffold one with "helper <id>", inspect the page with
+scripts/inspect-source.js, then fill in listSelector + urlPattern.
     `);
 }