feat: support source helper method

README.md

@@ -55,18 +55,25 @@ news-harvester/
 ├── SKILL.md                          # Skill 定义（Agent 加载入口）
 ├── README.md                         # 本文件
 ├── .gitignore
-├── package.json                      # Node 依赖（ws，仅 browser 方法用）
+├── package.json                      # Node 依赖（ws + js-yaml）
 ├── config.json                       # ⚠️ 运行时生成，不入库
+├── helpers/                          # 数据源 recipe（per-source spec，入库共享）
+│   ├── reuters.md                    # 路透社抓取规格（生产用）
+│   └── nikkei-tech.md                # 日经亚洲科技抓取规格（生产用）
 ├── references/
 │   ├── config-template.json          # config.json 的模板（首次运行自动复制）
-│   └── source-management.md          # 数据源管理详细说明
+│   ├── source-management.md          # 数据源管理详细说明
+│   └── helper-example.md             # helper.md 参考示例（带完整字段注释）
 ├── scripts/
 │   ├── harvest.js                    # 主入口：node harvest.js <source_id> <count> [--preview] [--full]
 │   ├── finalize.js                   # 补全中文摘要：node finalize.js <dayDir>
-│   ├── source-manage.js              # 数据源增删改查
+│   ├── source-manage.js              # 数据源增删改查 + recipe scaffold
 │   ├── ensure-chromium.js            # 跨平台自动发现并启动 Chromium（browser 源用）
-│   ├── fetcher-direct.js             # 纯 HTTP 抓取（开放站点）
-│   └── fetcher-browser.js            # Chromium CDP 抓取（JS 渲染站点）
+│   ├── inspect-source.js             # 诊断工具：分析页面结构，辅助编写 helper.md
+│   ├── cdp-client.js                 # 共享 CDP 基元（WebSocket/导航/滚动/求值）
+│   ├── fetcher-helper.js             # spec 驱动抓取（有 recipe 时用）
+│   ├── fetcher-direct.js             # 纯 HTTP 抓取（legacy direct 源）
+│   └── fetcher-browser.js            # Chromium CDP 抓取（legacy browser 源，无 recipe 时用）
 └── data/                             # ⚠️ 运行时去重数据，不入库
     └── registry.json
 ```
@@ -216,10 +223,71 @@ claude -p \
 | `name` | 显示名 | `路透社` |
 | `homepageUrl` | 首页 URL（抓取链接入口） | `https://www.reuters.com/` |
 | `method` | `direct`（纯 HTTP）或 `browser`（需 Chromium） | `direct` |
-| `articleUrlPattern` | 文章 URL 匹配正则 | `theconversation\\.com/[\\w-]+-\\d+$` |
+| `articleUrlPattern` | 文章 URL 匹配正则 — **legacy direct 源专用**；有 recipe(`helpers/<id>.md`) 时被忽略 | `theconversation\\.com/[\\w-]+-\\d+$` |
 | `vaultFolder` | vault 下的子文件夹名 | `The Conversation` |
 | `enabled` | 是否启用 | `true` |
 
+> browser 源应配 recipe（`helpers/<id>.md`）而非依赖 `articleUrlPattern`；recipe 的 `urlPattern`/选择器是解析真相源，详见 `SKILL.md` 的 **Source recipes** 小节。
+
+---
+
+## 数据源 Recipe（`helpers/<id>.md`）
+
+每个数据源可以有一份 **recipe**——`helpers/<source-id>.md` 文件，用 YAML frontmatter 声明如何定位文章列表、如何提取正文与元数据。`harvest.js` 发现 recipe 时用 spec 驱动的 `fetcher-helper.js`；没有 recipe 则回退 legacy fetcher。**`source-manage.js list` 的 `RECIPE` 列显示每个源走哪条路径**（`✅` = 有 recipe，`—` = legacy）。
+
+> **优先级**：有 recipe 时，其 `urlPattern`/选择器是解析真相源，`config.json` 的 `articleUrlPattern` **被忽略**（且不应再出现在该源的 config 条目里）。recipe 也会强制启用 Chromium（因为 `fetcher-helper.js` 驱动浏览器），与 `method` 无关。
+
+### 为什么用 recipe
+
+旧方案靠一个硬编码正则扫描整页 `<a>`，导航/侧栏/页脚的链接混进来，且 SPA 站点懒加载的首屏只有几篇。recipe 用 `listSelector` 把链接采集收窄到主文章流容器、用 `scrollSteps` 触发懒加载、用收紧的 `urlPattern` 校验——直接解决"分析新 CDP 源找不到高质量数据"的问题。recipe 是纯 markdown，站点改版时改文件即可，不动代码。
+
+### 添加新 browser 源（inspect → recipe → verify）
+
+```bash
+# 1. 加入 config
+node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser
+
+# 2. 诊断页面结构（输出按容器分组的链接 + 可粘贴的 recipe 骨架）
+node scripts/inspect-source.js https://example.com/ --scroll 3 --pattern 'example\.com/.+'
+
+# 3. scaffold recipe 模板，然后填入 listSelector + urlPattern
+node scripts/source-manage.js helper mysrc
+#   编辑 helpers/mysrc.md ...
+
+# 4. 只读验证
+node scripts/harvest.js mysrc 3 --preview
+```
+
+### recipe 字段速查
+
+所有字段除 `urlPattern` 外都可选，缺省值 = legacy 行为。完整说明见 `references/source-management.md`（Helper recipes）和带注释的 `references/helper-example.md`。
+
+| 阶段 | 字段 | 作用 |
+|------|------|------|
+| 发现 | `listSelector` | 主文章列表容器 CSS 选择器（收窄范围，排除导航/侧栏）|
+| 发现 | `linkSelector` | 容器内候选链接（默认 `a[href]`）|
+| 发现 | `urlPattern` | **必填**，文章 URL 校验正则，取代 config 的 `articleUrlPattern` |
+| 发现 | `excludeUrlPattern` | 排除正则（如 podcasts/section 页）|
+| 发现 | `scrollSteps` | 向下滚动次数，触发 SPA 懒加载 |
+| 提取 | `bodySelectors` | 正文容器选择器数组，按序首个满足 `bodyMinParagraphs` 即用 |
+| 提取 | `dateSelector` / `authorSelector` | 元数据选择器；找不到时自动回退 JSON-LD |
+| 提取 | `excludeImage` / `imageMinWidth` | 图片过滤（排除 logo/头像/跟踪像素）|
+| 提取 | `titleStripSuffix` | 从 `document.title` 剥离的后缀正则 |
+
+> **正则引用**：regex 字段用单引号包裹（`'reuters\.com/.+'`）。双引号会让 js-yaml 把 `\.` 当非法转义报错。
+>
+> **JSON-LD 兜底**：`dateSelector`/`authorSelector` 找不到时，`fetcher-helper.js` 自动读 `<script type="application/ld+json">` 的 `datePublished`/`author.name`。很多 SPA 站点（如 Nikkei）只在 JSON-LD 放元数据，所以通常不用自定义这两个选择器。
+
+### 管理 recipe
+
+```bash
+node scripts/source-manage.js helper <id>            # scaffold 模板（已存在则拒绝）
+node scripts/source-manage.js helper <id> --force    # 覆盖
+node scripts/source-manage.js helper <id> --show     # 查看 recipe 路径 + 是否存在
+```
+
+recipe 存放在 `helpers/`（入库共享，跨机器一致），区别于 `config.json`（机器特定，不入库）。
+
 ---
 
 ## 归档产物结构

SKILL.md

@@ -5,7 +5,7 @@ description: Harvest and archive news articles from configured sources (Reuters,
 
 # News Harvester Skill
 
-Fetch latest articles from configured news sources, archive them to Obsidian vault with Chinese summaries + original text. Supports browser-based (Chromium CDP) and direct HTTP fetching. Deduplicates via per-day registry with a sliding window.
+Fetch latest articles from configured news sources, archive them to Obsidian vault with Chinese summaries + original text. Supports browser-based (Chromium CDP), direct HTTP, and per-source **recipe** (`helpers/<id>.md`) fetching — a recipe declares the selectors/scroll/URL-filter spec and takes precedence over the legacy fetchers. Deduplicates via per-day registry with a sliding window.
 
 ## Quick Reference
 
@@ -18,6 +18,8 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau
 | Add source | `node scripts/source-manage.js add` |
 | List sources | `node scripts/source-manage.js list` |
 | Edit source | `node scripts/source-manage.js edit <id>` |
+| Scaffold a source recipe | `node scripts/source-manage.js helper <id>` |
+| Inspect a source's page structure | `node scripts/inspect-source.js <url> --scroll 3 [--pattern '<regex>'] [--write <id>]` |
 | Ensure Chromium (browser sources) | `node scripts/ensure-chromium.js` (or `--check`) |
 
 > **All commands assume `cd <skill-root>` first** (the directory containing this `SKILL.md`). Scripts live in the `scripts/` subfolder — **always prefix with `scripts/`**, e.g. `node scripts/harvest.js`, never `node harvest.js` (that fails with `MODULE_NOT_FOUND`).
@@ -54,7 +56,7 @@ node scripts/source-manage.js add --id the-conversation --name "The Conversation
 If the user wants a different first source (e.g. `reuters` with `browser` method), add that instead and note the Chromium prerequisite below applies.
 
 **3. Check prerequisites for the source's `method`.**
-- `direct` → nothing extra.
+- `direct` → nothing extra (plain HTTP via `fetcher-direct.js`).
 - `browser` → `harvest.js` auto-launches Chromium on first browser-source use (see `scripts/ensure-chromium.js`). It cross-platform-discovers the binary (macOS `.app` bundles, Linux PATH, Windows install dirs) — no manual `chromium ... &` needed. To verify it can find a browser without running a full harvest:
   ```bash
   node scripts/ensure-chromium.js --check     # is CDP already up? exit 0/1
@@ -62,6 +64,8 @@ If the user wants a different first source (e.g. `reuters` with `browser` method
   ```
   If discovery can't find your browser, set `config.chromium.executablePath` to its full path (see Configuration).
 
+> **Chromium is launched whenever `method: browser` OR the source has a recipe** (`helpers/<id>.md`). A `direct` source that gains a recipe also uses Chromium, since `fetcher-helper.js` drives the browser. See Source recipes below.
+
 **4. Smoke test (read-only, no vault writes).** Confirm the fetch pipeline works end-to-end before archiving anything:
 ```bash
 node scripts/harvest.js the-conversation 2 --preview
@@ -117,6 +121,35 @@ To point a source at a different vault subfolder, set `sources[].vaultFolder`. A
 
 ---
 
+## Source recipes (`helpers/<id>.md`)
+
+Each source can have a **recipe** — a `helpers/<source-id>.md` file whose YAML frontmatter declares how to find its article list and extract article content (selectors, scroll, URL filters). When `harvest.js` sees one, it uses the spec-driven `fetcher-helper.js` instead of the legacy fetcher. **No recipe → legacy path, zero behavior change** — so existing sources keep working and you migrate one at a time.
+
+> **Precedence**: when a recipe exists, its `urlPattern`/selectors are the source of truth and `config.json`'s `articleUrlPattern` is **ignored**. Sources with a recipe therefore don't need (and shouldn't carry) `articleUrlPattern` in config — it's a legacy-direct-source field only. `source-manage.js list` shows a `RECIPE` column so you can tell at a glance which sources are on which path.
+
+Recipes solve the "can't find high-quality data on new CDP sources" problem: instead of one hard-coded regex scanning the whole page (which mixes nav/sidebar/footer links with real articles), a recipe scopes link collection to a `listSelector` container, triggers lazy-load via `scrollSteps`, and validates URLs with a tightened `urlPattern`. The recipe is just markdown — edit it to adapt to a site redesign without touching code.
+
+**Adding a new browser source** (3-step closed loop: add → inspect --write → verify):
+```bash
+# 1. Add to config (browser sources should get a recipe — articleUrlPattern is legacy-only)
+node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser --folder "My Source"
+
+# 2. Inspect + auto-write the recipe in one shot (measures listSelector + infers urlPattern from real URLs)
+node scripts/inspect-source.js https://example.com/ --scroll 3 --write mysrc
+
+# 3. Verify (read-only)
+node scripts/harvest.js mysrc 3 --preview
+```
+**Convergence check**: preview passes when the fetched links are real articles (not nav/section pages) and `charCount > 500`. If it fails — links are nav/section → tighten `urlPattern` in `helpers/mysrc.md`; too few links → raise `scrollSteps`; empty body → fix `bodySelectors`. Re-run preview after each edit. **Do not hand-edit the recipe blindly** — re-run `inspect` (without `--write`) to see the container groups if unsure.
+
+**Once verified, archive** by following Flow A below (harvest writes originals → you write `.summaries.json` → finalize). The harvest stderr prints the `Day dir` and the exact next command — follow that, don't parse the stdout manifest.
+
+Full schema and authoring notes: `references/source-management.md` (Helper recipes).
+
+> **Roadmap (model B)**: the current split keeps direct sources on legacy `fetcher-direct.js`. The eventual goal is for *every* source to have a recipe (moving `method`/`homepageUrl`/`urlPattern` into the recipe and reducing `config.json` to a pure index of `id`/`name`/`vaultFolder`/`enabled`), with `fetcher-helper.js` gaining a direct-HTTP mode. Direct sources are intentionally untouched this round to avoid disrupting working fetchers.
+
+---
+
 ## Workflow
 
 Two flows depending on intent: **archive** (fetch + save to Obsidian) or **preview** (read-only, just return article data).
@@ -137,7 +170,7 @@ The script fetches links, deduplicates, fetches each article, downloads images,
 
 It prints a **compact manifest** to stdout (no full body — only a `bodyPreview` ~4000 chars per article). Read it.
 
-> **Browser sources** are auto-handled: before fetching, `harvest.js` calls `ensure-chromium.js`, which health-checks the CDP port and launches Chromium itself (cross-platform binary discovery, headless by default) if it isn't up — a no-op if already running. No manual `chromium ... &` or `sleep` is needed. To debug launch failures: `node scripts/ensure-chromium.js` (prints binary path, port, and stderr-log tail on failure).
+> **Browser sources and any source with a recipe** are auto-handled: before fetching, `harvest.js` calls `ensure-chromium.js`, which health-checks the CDP port and launches Chromium itself (cross-platform binary discovery, headless by default) if it isn't up — a no-op if already running. No manual `chromium ... &` or `sleep` is needed. To debug launch failures: `node scripts/ensure-chromium.js` (prints binary path, port, and stderr-log tail on failure).
 
 #### Step 2. Generate Chinese summaries → write `.summaries.json`
 From the compact manifest, for each article write one entry to `<dayDir>/.summaries.json`:

package-lock.json

@@ -9,9 +9,38 @@
       "version": "1.0.0",
       "license": "ISC",
       "dependencies": {
+        "js-yaml": "^5.2.2",
         "ws": "^8.19.0"
       }
     },
+    "node_modules/argparse": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz",
+      "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==",
+      "license": "Python-2.0"
+    },
+    "node_modules/js-yaml": {
+      "version": "5.2.2",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-5.2.2.tgz",
+      "integrity": "sha512-dayzUzKkJ1MkuUtZglSebU43utNXH0OWQByK9rKOOuYIO8M5TV1y+n8ALMdG0rdzBnfNkOmZEqrURepb0ejqBw==",
+      "funding": [
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/puzrin"
+        },
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/nodeca"
+        }
+      ],
+      "license": "MIT",
+      "dependencies": {
+        "argparse": "^2.0.1"
+      },
+      "bin": {
+        "js-yaml": "bin/js-yaml.mjs"
+      }
+    },
     "node_modules/ws": {
       "version": "8.19.0",
       "resolved": "https://registry.npmjs.org/ws/-/ws-8.19.0.tgz",

package.json

@@ -11,6 +11,7 @@
   "license": "ISC",
   "type": "commonjs",
   "dependencies": {
+    "js-yaml": "^5.2.2",
     "ws": "^8.19.0"
   }
 }

references/helper-example.md

+---
+# ── Source identity ──
+# source 必须与 config.json 的 source.id 一致；harvest.js 据此加载本文件。
+source: reuters
+
+# ── 发现阶段：在列表页定位文章链接 ──
+# listSelector: 主文章列表容器的 CSS 选择器。
+#   关键字段——把链接采集范围从"整页"收窄到这个容器，排除导航/侧栏/页脚。
+#   null = 全页扫描（=legacy 行为，不推荐）。用 `node scripts/inspect-source.js <url>` 找到最大容器组。
+listSelector: 'div[data-testid="Title"]'
+
+# linkSelector: 容器内候选 <a> 元素。一般不用改。
+linkSelector: 'a[href]'
+
+# urlPattern: 【必填】文章 URL 校验正则（regex source，不是带斜杠的字面量）。
+#   匹配此正则的 href 才算文章。有 helper 时此字段取代 config.articleUrlPattern（后者被忽略）。
+#   注意：用单引号包裹（双引号会让 js-yaml 把 \. 当非法转义报错）。
+urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$'
+
+# excludeUrlPattern: 排除正则——匹配 urlPattern 但不该算文章的 URL（如 podcasts/newsletter）。
+excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
+
+# scrollSteps: 向下滚动次数，触发 SPA 懒加载。0 = 不滚（只拿首屏）。
+#   Reuters 首屏仅 ~10 篇，滚 3 次能拿全当天列表。
+scrollSteps: 3
+scrollWaitMs: 2000        # 每次滚动后等待毫秒（等新内容渲染）
+waitMs: 15000             # 页面初始加载后等待毫秒（等 JS/React 渲染完）
+maxLinks: 30              # 链接上限
+
+# ── 提取阶段：在文章页定位正文/元数据 ──
+# bodySelectors: 正文容器选择器，按序尝试，首个满足 ≥ bodyMinParagraphs 的即用。
+bodySelectors:
+  - '[data-testid^="paragraph"]'
+  - '[class*="articleBodyContent"] p'
+  - '[class*="article-body"] p'
+  - 'article p'
+bodyMinParagraphs: 3
+
+# dateSelector / authorSelector: CSS 选择器。找不到时自动回退到页面的
+#   <script type="application/ld+json"> 的 datePublished / author.name（JSON-LD 兜底）。
+#   很多 SPA 站点（Nikkei）只在 JSON-LD 放元数据，所以通常不用自定义。
+dateSelector: 'time[datetime]'
+authorSelector: '[rel="author"], [class*="author"] a'
+
+# imageSelector: 图片元素（一般不动）。imageMinWidth 过滤图标/头像/跟踪像素。
+# excludeImage: 图片 URL 子串黑名单（小写匹配）。
+imageSelector: 'img'
+imageMinWidth: 200
+excludeImage:
+  - logo
+  - icon
+  - avatar
+  - profile
+
+# noisePrefixes: 正文段落过滤——以这些前缀开头的 <p> 会被丢弃（如 Reuters 的"Reporting by"署名块）。
+noisePrefixes:
+  - 'Reporting by'
+  - 'Editing by'
+  - 'Access unmatched'
+  - 'Browse an unrivalled'
+  - 'Screen for heightened'
+  - 'Sign up here'
+  - 'Reuters, the news'
+  - 'All quotes delayed'
+  - 'See here for'
+
+# titleStripSuffix: 从 document.title 剥离的后缀正则（清理 "| Reuters" 等）。
+titleStripSuffix: ' \| Reuters$'
+maxBodyChars: 15000
+---
+
+# Reuters 抓取备忘
+
+这一段（frontmatter 下方）是给人看的备忘，脚本不解析。记录：
+- 主文章流容器是 `div[data-testid="Title"]`（inspect 实测 59 篇）。
+- 文章 URL 以日期 `-YYYY-MM-DD/` 结尾，是过滤导航/section 页的关键。
+- 反爬：headless 会被拦，需 config.chromium.headless:false。
+
+---
+# 字段速查（所有字段除 urlPattern 外都可选，缺省值 = legacy 行为）
+#
+# 发现阶段:  listSelector / linkSelector / urlPattern(必填) / excludeUrlPattern
+#            scrollSteps / scrollWaitMs / waitMs / maxLinks
+# 提取阶段:  bodySelectors / bodyMinParagraphs / dateSelector / authorSelector
+#            imageSelector / imageMinWidth / excludeImage / noisePrefixes
+#            titleStripSuffix / maxBodyChars
+#
+# 完整说明见 references/source-management.md 的 "Helper recipes" 小节。
+# 写新 recipe 流程: inspect → scaffold(source-manage helper <id>) → 填字段 → preview 验证

references/source-management.md

@@ -20,11 +20,11 @@ Prompt the user for (or infer from context):
 | `name` | Display name | `路透社` |
 | `homepageUrl` | URL to scrape for links | `https://www.reuters.com/` |
 | `method` | `browser` (needs JS) or `direct` (plain HTTP) | `browser` |
-| `articleUrlPattern` | Regex to match article URLs | `reuters\\.com/.+/\\d{4}-\\d{2}-\\d{2}/` |
+| `articleUrlPattern` | Regex to match article URLs — **legacy direct sources only**; ignored when a recipe (`helpers/<id>.md`) exists | `theconversation\\.com/[\\w-]+-\\d+$` |
 | `vaultFolder` | Subfolder under vaultPath | `路透社` |
 | `enabled` | Start enabled? | `true` |
 
-Auto-detect `articleUrlPattern` if not provided:
+Auto-detect `articleUrlPattern` if not provided (legacy path only — browser sources should get a recipe instead, see Helper recipes below):
 - Reuters: `reuters\\.com/.+/\\d{4}-\\d{2}-\\d{2}/`
 - The Conversation: `theconversation\\.com/.+-\\d+$`
 - Generic: derive from homepage domain + path depth
@@ -42,7 +42,7 @@ Usage: `node source-manage.js edit <id>`
 
 Show current values, prompt for field to change, update `config.json`.
 
-Editable fields: `name`, `homepageUrl`, `method`, `articleUrlPattern`, `vaultFolder`, `enabled`, `countPerRun` (if set as default).
+Editable fields: `name`, `homepageUrl`, `method`, `articleUrlPattern`, `vaultFolder`, `enabled`.
 
 ## Enable / Disable Source
 
@@ -51,3 +51,92 @@ Toggle `enabled` field:
 node source-manage.js enable reuters
 node source-manage.js disable reuters
 ```
+
+---
+
+## Helper recipes (`helpers/<id>.md`)
+
+A source opts into the spec-driven fetcher by having a `helpers/<source-id>.md`
+file. When `harvest.js` finds one, it uses `fetcher-helper.js` (which reads the
+recipe's selectors/scroll/filter) instead of the legacy `fetcher-browser.js`/
+`fetcher-direct.js`. **No helper.md → legacy path, zero behavior change.**
+
+> **Precedence**: when a recipe exists, its `urlPattern`/selectors are the
+> source of truth and `config.json`'s `articleUrlPattern` is **ignored** (and
+> should be omitted from that source's config entry to avoid drift). A recipe
+> also forces Chromium use regardless of `method` (since `fetcher-helper.js`
+> drives the browser). `source-manage.js list` shows a `RECIPE` column (`✅`/
+> `—`) indicating which path each source is on.
+
+### Authoring workflow (for a new browser source)
+
+1. **Add the source** to config (see Add Source above).
+2. **Inspect the page** to discover where article links actually live:
+   ```bash
+   node scripts/inspect-source.js <homepageUrl> --scroll 3 --pattern '<domain>\.com/.+'
+   ```
+   This prints links grouped by container, biggest group first, plus a
+   ready-to-paste helper.md skeleton.
+3. **Scaffold + edit the recipe**:
+   ```bash
+   node scripts/source-manage.js helper <id>     # writes helpers/<id>.md template
+   ```
+   Fill in `listSelector` (from the biggest inspect group) and tighten
+   `urlPattern` (from the sample URLs inspect printed).
+4. **Verify** (read-only):
+   ```bash
+   node scripts/harvest.js <id> 3 --preview
+   ```
+   Confirm the links are real articles (not nav/section pages) and the body
+   char counts look right. Iterate on the recipe until clean.
+
+### Frontmatter schema
+
+All fields optional except `urlPattern`; defaults reproduce legacy behaviour.
+
+**Discovery** (find the article-link list):
+
+| Field | Default | Purpose |
+|-------|---------|---------|
+| `listSelector` | `null` (whole page) | CSS selector for the article-list container — scopes link collection away from nav/sidebar/footer |
+| `linkSelector` | `a[href]` | Candidate links inside each container |
+| `urlPattern` | **required** | Regex source; an href must match to count as an article |
+| `excludeUrlPattern` | `null` | Regex source; drop matching hrefs (section pages, podcasts, etc.) |
+| `scrollSteps` | `0` | Bottom-scroll iterations to trigger lazy-loaded lists |
+| `scrollWaitMs` | `1500` | Pause after each scroll |
+| `waitMs` | `12000` | Initial render wait after load |
+| `maxLinks` | `30` | Link cap |
+
+**Extraction** (single article body + metadata):
+
+| Field | Default | Purpose |
+|-------|---------|---------|
+| `bodySelectors` | (6 selectors) | Tried in order; first yielding ≥ `bodyMinParagraphs` wins |
+| `bodyMinParagraphs` | `3` | Min paragraph count for a selector to qualify |
+| `dateSelector` | `time[datetime]` | Falls back to JSON-LD `datePublished` |
+| `authorSelector` | `[rel="author"], [class*="author"] a` | Falls back to JSON-LD `author.name` |
+| `imageSelector` | `img` | |
+| `imageMinWidth` | `200` | Drops icons/avatars/tracking pixels |
+| `excludeImage` | `["logo","icon","avatar","profile"]` | Substring blocklist on image URL |
+| `noisePrefixes` | (Reuters boilerplate) | Paragraphs starting with these are dropped from body |
+| `titleStripSuffix` | ` \| Reuters\| ...` | Regex source, stripped from `document.title` |
+| `maxBodyChars` | `15000` | Body truncation |
+
+> **Regex quoting**: use single-quoted YAML scalars for regex fields (`'reuters\.com/.+'`).
+> Double quotes make js-yaml reject `\.` as an unknown escape. This matches how
+> you'd write the regex in code.
+>
+> **JSON-LD fallback**: if `dateSelector`/`authorSelector` find nothing,
+> `fetcher-helper.js` automatically reads `<script type="application/ld+json">`
+> (`datePublished` / `author.name`). Many SPA sites (Nikkei) only embed metadata
+> there, so you usually don't need a custom selector.
+
+### Managing recipes
+
+```bash
+node scripts/source-manage.js helper <id>            # scaffold a template (refuses if exists)
+node scripts/source-manage.js helper <id> --force    # overwrite
+node scripts/source-manage.js helper <id> --show     # print path + exists/missing
+```
+Recipes live in `helpers/` (committed to git, shared across machines) — unlike
+`config.json` which is machine-specific.

scripts/cdp-client.js

+/**
+ * cdp-client.js — Shared Chrome DevTools Protocol primitives.
+ *
+ * Previously every CDP caller (fetcher-browser.js) hand-rolled the same
+ * WebSocket boilerplate: open connection, send a command with an id, attach a
+ * message listener matching that id, parse the result. That pattern was
+ * duplicated 4× and the Page.enable call didn't even check the id (resolving
+ * on the first message of any kind). This module is the single source of truth
+ * for those operations so fetcher-helper.js (and a future fetcher-browser
+ * refactor) can build on top.
+ *
+ * Lifecycle (browser launch/discovery) stays in ensure-chromium.js — this
+ * module only handles per-tab CDP commands, assuming a browser is already up.
+ */
+
+const http = require('http');
+const fs = require('fs');
+const path = require('path');
+const WebSocket = require('ws');
+
+const SKILL_DIR = path.dirname(__dirname);
+const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
+const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
+
+// ── Utilities ────────────────────────────────────────────────────────────────
+
+function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
+
+// Monotonic id for CDP commands. Per-tab sessions are sequential, so a simple
+// counter is fine and avoids the random-id collisions the old code risked.
+let _msgId = 0;
+function nextId() { return ++_msgId; }
+
+// ── Port resolution ──────────────────────────────────────────────────────────
+// Precedence: env override > config.chromium.cdpPort > default 9222.
+// (Mirrors the old private resolveCdpPort in fetcher-browser.js.)
+function resolveCdpPort() {
+  if (process.env.CDP_PORT_OVERRIDE) return parseInt(process.env.CDP_PORT_OVERRIDE, 10);
+  try {
+    const cfgPath = fs.existsSync(CONFIG_PATH) ? CONFIG_PATH : TEMPLATE_PATH;
+    const cfg = JSON.parse(fs.readFileSync(cfgPath, 'utf8'));
+    if (cfg.chromium && cfg.chromium.cdpPort) return parseInt(cfg.chromium.cdpPort, 10);
+  } catch (e) { /* fall through to default */ }
+  return 9222;
+}
+
+// ── WebSocket URL discovery ───────────────────────────────────────────────────
+// Fetch /json and pick the first `page` tab's webSocketDebuggerUrl. Retries a
+// few times: right after ensure-chromium spawns the browser, the port may
+// accept connections a beat before /json responds.
+function getWsUrl() {
+  const port = resolveCdpPort();
+  const MAX_ATTEMPTS = 6;
+  const STEP_MS = 500;
+  return new Promise((resolve, reject) => {
+    let attempt = 0;
+    const tryOnce = () => {
+      attempt++;
+      http.get(`http://localhost:${port}/json`, (res) => {
+        let data = '';
+        res.on('data', d => data += d);
+        res.on('end', () => {
+          try {
+            const tabs = JSON.parse(data);
+            // Prefer a page tab to avoid grabbing an iframe/service_worker target.
+            const tab = tabs.find(t => t.type === 'page') || tabs[0];
+            if (!tab || !tab.webSocketDebuggerUrl) {
+              return reject(new Error(`CDP /json returned no page tab on :${port}`));
+            }
+            resolve(tab.webSocketDebuggerUrl);
+          } catch (e) {
+            if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
+            reject(e);
+          }
+        });
+      }).on('error', (e) => {
+        if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
+        reject(new Error(`Cannot reach CDP on :${port} after ${MAX_ATTEMPTS} tries (${e.message}). Run: node scripts/ensure-chromium.js`));
+      });
+    };
+    tryOnce();
+  });
+}
+
+// ── Session management ───────────────────────────────────────────────────────
+
+// Open a WebSocket to the page target and resolve once connected.
+// Returns { ws, close }. The caller owns the lifecycle (close when done).
+function openSession() {
+  return getWsUrl().then(wsUrl => new Promise((resolve, reject) => {
+    const ws = new WebSocket(wsUrl);
+    ws.on('open', () => resolve({ ws, close: () => ws.close() }));
+    ws.on('error', reject);
+  }));
+}
+
+// ── Command sending ──────────────────────────────────────────────────────────
+
+// Send a CDP command and await its response, matched by id. Rejects on CDP
+// error. Concurrent calls are safe — each attaches its own id-matched listener.
+function cdpSend(ws, method, params = {}) {
+  return new Promise((resolve, reject) => {
+    const id = nextId();
+    const handler = (raw) => {
+      const msg = JSON.parse(raw.toString());
+      if (msg.id === id) {
+        ws.removeListener('message', handler);
+        if (msg.error) reject(new Error(`CDP ${method} error: ${JSON.stringify(msg.error)}`));
+        else resolve(msg.result);
+      }
+    };
+    ws.on('message', handler);
+    ws.send(JSON.stringify({ id, method, params }));
+  });
+}
+
+// Evaluate a JS expression in the page and return the value (returnByValue).
+// Returns undefined if the expression yielded no value.
+async function cdpEval(ws, expression) {
+  const result = await cdpSend(ws, 'Runtime.evaluate', {
+    expression,
+    returnByValue: true
+  });
+  return result && result.result && result.result.value;
+}
+
+// ── Navigation ───────────────────────────────────────────────────────────────
+
+// Enable Page domain, navigate to url, wait for loadEventFired (racing a 25s
+// timeout), then sleep waitMs for JS/React to finish rendering.
+// Fixes the old Page.enable bug where ws.once('message') resolved on ANY
+// message instead of the enable response — here cdpSend id-checks it.
+async function navigateAndWait(ws, url, waitMs = 12000) {
+  await cdpSend(ws, 'Page.enable');
+
+  // Attach the load listener BEFORE sending navigate, since loadEventFired can
+  // fire before the navigate response arrives.
+  let loaded = false;
+  const loadPromise = new Promise(resolve => {
+    const handler = (raw) => {
+      const msg = JSON.parse(raw.toString());
+      if (msg.method === 'Page.loadEventFired' && !loaded) {
+        loaded = true;
+        ws.removeListener('message', handler);
+        resolve();
+      }
+    };
+    ws.on('message', handler);
+  });
+
+  await cdpSend(ws, 'Page.navigate', { url });
+  await Promise.race([loadPromise, sleep(25000)]);
+  await sleep(waitMs);
+}
+
+// ── Scrolling + lazy-load collection ─────────────────────────────────────────
+
+// Scroll to the bottom of the page `steps` times, waiting `waitMs` after each
+// scroll, and run `collectExpr` (a JS expression returning a JSON string of an
+// array) after the initial load and after each scroll. Results are merged,
+// deduplicating by `dedupKey` (default: JSON.stringify of the element).
+//
+// Returns the merged array (of parsed objects). This is what lets SPA list
+// pages that lazy-load article cards yield their full set instead of just the
+// first screenful.
+async function scrollAndCollect(ws, { steps = 0, waitMs = 1500, collectExpr, dedupKey }) {
+  const keyFn = dedupKey || (item => JSON.stringify(item));
+  const seen = new Set();
+  const merged = [];
+
+  const absorb = (raw) => {
+    if (!raw) return;
+    let arr;
+    try { arr = JSON.parse(raw); } catch (e) { return; }
+    if (!Array.isArray(arr)) return;
+    for (const item of arr) {
+      const k = keyFn(item);
+      if (!seen.has(k)) { seen.add(k); merged.push(item); }
+    }
+  };
+
+  absorb(await cdpEval(ws, collectExpr));
+
+  for (let i = 0; i < steps; i++) {
+    await cdpEval(ws, 'window.scrollTo(0, document.body.scrollHeight)');
+    await sleep(waitMs);
+    absorb(await cdpEval(ws, collectExpr));
+  }
+
+  return merged;
+}
+
+module.exports = {
+  sleep,
+  resolveCdpPort,
+  getWsUrl,
+  openSession,
+  cdpSend,
+  cdpEval,
+  navigateAndWait,
+  scrollAndCollect
+};

scripts/harvest.js

@@ -360,13 +360,27 @@ Examples:
   console.error(`\n🗞️  Harvesting ${source.name} (max ${count} articles) — ${mode}`);
   console.error(`   Method: ${source.method}`);
 
-  const fetcher = source.method === 'browser'
-    ? require('./fetcher-browser')
-    : require('./fetcher-direct');
+  // Strategy dispatch: if a helpers/<id>.md recipe exists for this source, use
+  // the spec-driven fetcher-helper; otherwise fall back to the legacy fetcher
+  // for its method. Existing sources keep working unchanged; a source opts in
+  // simply by gaining a helper.md file. See references/source-management.md
+  // (Helper recipes) for the schema.
+  const helperPath = path.join(SKILL_DIR, 'helpers', `${source.id}.md`);
+  let fetcher, spec = null;
+  if (fs.existsSync(helperPath)) {
+    const fetcherHelper = require('./fetcher-helper');
+    spec = fetcherHelper.loadSpec(helperPath);
+    fetcher = fetcherHelper;
+    console.error(`   Recipe: helpers/${source.id}.md (listSelector=${spec.listSelector || 'whole page'}, scroll=${spec.scrollSteps})`);
+  } else {
+    fetcher = source.method === 'browser'
+      ? require('./fetcher-browser')
+      : require('./fetcher-direct');
+  }
 
-  // For browser sources, ensure a Chromium CDP endpoint is up before fetching.
-  // ensure-chromium.js is a no-op if already running, so this is safe every run.
-  if (source.method === 'browser') {
+  // CDP-based fetchers (browser method, or any source with a helper recipe)
+  // need a Chromium endpoint. ensure-chromium.js is a no-op if already running.
+  if (source.method === 'browser' || spec) {
     const { ensureChromium } = require('./ensure-chromium');
     console.error('   Ensuring Chromium (CDP)...');
     try {
@@ -378,6 +392,17 @@ Examples:
     }
   }
 
+  // Call-site adapters: legacy fetchers take (url, pattern) / (url[, waitMs]),
+  // while fetcher-helper takes (url, spec). The closures hide that difference
+  // so the main flow below is identical for both paths. NB: spec must NOT be
+  // passed to legacy fetchPage (it would be coerced to waitMs=NaN).
+  const getLinks = () => spec
+    ? fetcher.fetchHomeLinks(source.homepageUrl, spec)
+    : fetcher.fetchHomeLinks(source.homepageUrl, source.articleUrlPattern);
+  const getPage = (url) => spec
+    ? fetcher.fetchPage(url, spec)
+    : fetcher.fetchPage(url);
+
   // Build dedup set (skip in preview — preview shows latest regardless)
   const seen = preview ? new Set() : buildDedupSet(config, sourceId);
   console.error(`   Registry: ${seen.size} URLs in last ${config.registryWindowDays || 7} days`);
@@ -386,7 +411,7 @@ Examples:
   console.error(`\n📡 Fetching homepage: ${source.homepageUrl}`);
   let links;
   try {
-    links = await fetcher.fetchHomeLinks(source.homepageUrl, source.articleUrlPattern);
+    links = await getLinks();
   } catch (e) {
     console.error('Failed to fetch homepage:', e.message);
     process.exit(1);
@@ -408,7 +433,7 @@ Examples:
       console.error(`\n[${i+1}/${newLinks.length}] ${url}`);
       let articleData;
       try {
-        articleData = await fetcher.fetchPage(url);
+        articleData = await getPage(url);
       } catch (e) {
         console.error(`  ✗ Fetch failed: ${e.message}`);
         continue;
@@ -463,7 +488,7 @@ Examples:
 
     let articleData;
     try {
-      articleData = await fetcher.fetchPage(url);
+      articleData = await getPage(url);
     } catch (e) {
       console.error(`  ✗ Fetch failed: ${e.message}`);
       continue;
@@ -569,8 +594,8 @@ Examples:
   // stdout shows only THIS run's new articles (the agent summarizes what it just fetched)
   process.stdout.write(JSON.stringify({ ...manifest, articles: newArticles }, null, 2));
   console.error(`\n✅ Done: ${processed.length} processed this run, ${skipped} skipped`);
-  console.error(`   Manifest (${mergedArticles.length} total pending): ${manifestPath}`);
-  console.error(`   Next: generate .summaries.json, then run finalize.js`);
+  console.error(`   Day dir: ${dayDir}`);
+  console.error(`   Next: write .summaries.json to ${path.join(dayDir, '.summaries.json')}, then run: node scripts/finalize.js "${dayDir}"`);
 }
 
 // Run only when invoked directly (not when required by finalize.js)

scripts/source-manage.js

@@ -7,6 +7,8 @@
  *   node source-manage.js edit <id> --field value
  *   node source-manage.js enable <id>
  *   node source-manage.js disable <id>
+ *   node source-manage.js helper <id>          # scaffold/overwrite helpers/<id>.md
+ *   node source-manage.js helper <id> --show    # print current helper.md path + status
  */
 
 const fs = require('fs');
@@ -15,6 +17,7 @@ const path = require('path');
 const SKILL_DIR = path.dirname(__dirname);
 const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
 const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
+const HELPERS_DIR = path.join(SKILL_DIR, 'helpers');
 
 function loadConfig() {
   if (!fs.existsSync(CONFIG_PATH)) {
@@ -50,18 +53,21 @@ const config = loadConfig();
 switch (command) {
   case 'list': {
     console.log('\n📋 Configured Sources\n');
-    console.log('ID'.padEnd(20) + 'NAME'.padEnd(20) + 'METHOD'.padEnd(10) + 'STATUS');
-    console.log('─'.repeat(60));
+    console.log('ID'.padEnd(20) + 'NAME'.padEnd(20) + 'METHOD'.padEnd(10) + 'RECIPE'.padEnd(10) + 'STATUS');
+    console.log('─'.repeat(70));
     for (const s of config.sources) {
       const status = s.enabled ? '✅ enabled' : '❌ disabled';
+      const hasRecipe = fs.existsSync(path.join(HELPERS_DIR, `${s.id}.md`));
+      const recipe = hasRecipe ? '✅' : '—';
       console.log(
         s.id.padEnd(20) +
         s.name.padEnd(20) +
         s.method.padEnd(10) +
+        recipe.padEnd(10) +
         status
       );
     }
-    console.log();
+    console.log('\n   RECIPE ✅ = helpers/<id>.md exists (spec-driven fetcher); — = legacy fetcher\n');
     break;
   }
 
@@ -104,6 +110,11 @@ switch (command) {
     console.log(`   Method: ${newSource.method}`);
     console.log(`   URL Pattern: ${newSource.articleUrlPattern}`);
     console.log(`   Vault Folder: ${newSource.vaultFolder}`);
+    if (newSource.method === 'browser') {
+      console.log(`\n   💡 browser sources should get a recipe (articleUrlPattern is legacy-only):`);
+      console.log(`      node scripts/inspect-source.js ${newSource.homepageUrl} --scroll 3 --write ${id}`);
+      console.log(`      node scripts/harvest.js ${id} 3 --preview   # verify, then archive`);
+    }
     console.log(`\n   Run: node harvest.js ${id} 5\n`);
     break;
   }
@@ -147,13 +158,84 @@ switch (command) {
     break;
   }
 
+  case 'helper': {
+    const id = args._0;
+    const source = config.sources.find(s => s.id === id);
+    if (!source) { console.error(`Source "${id}" not found`); process.exit(1); }
+    const helperPath = path.join(HELPERS_DIR, `${id}.md`);
+
+    if (args.show !== undefined) {
+      const exists = fs.existsSync(helperPath);
+      console.log(`\n   helper: ${helperPath}`);
+      console.log(`   status: ${exists ? '✅ exists' : '❌ none (source uses legacy fetcher)'}`);
+      break;
+    }
+
+    // Scaffold a helper.md template. Does NOT overwrite without --force.
+    if (fs.existsSync(helperPath) && args.force === undefined) {
+      console.error(`helpers/${id}.md already exists. Use --force to overwrite.`);
+      process.exit(1);
+    }
+    if (!fs.existsSync(HELPERS_DIR)) fs.mkdirSync(HELPERS_DIR, { recursive: true });
+    // Derive a starter urlPattern from the source's homepage domain.
+    let domainPattern = 'example\\.com/.+';
+    try {
+      const u = new URL(source.homepageUrl);
+      domainPattern = u.hostname.replace('www.', '').replace(/\./g, '\\.') + '/.+';
+    } catch (e) { /* keep default */ }
+
+    const template = `---
+source: ${id}
+listSelector: null
+linkSelector: 'a[href]'
+urlPattern: '${domainPattern}'
+excludeUrlPattern: null
+scrollSteps: 0
+scrollWaitMs: 1500
+waitMs: 12000
+bodySelectors:
+  - '[class*="article-body"] p'
+  - 'article p'
+bodyMinParagraphs: 3
+dateSelector: 'time[datetime]'
+authorSelector: '[rel="author"], [class*="author"] a'
+imageSelector: 'img'
+imageMinWidth: 200
+excludeImage:
+  - logo
+  - icon
+  - avatar
+  - profile
+titleStripSuffix: ''
+maxBodyChars: 15000
+---
+# ${source.name} 抓取备忘
+
+- 先运行 inspect 定位真实结构:
+  node scripts/inspect-source.js ${source.homepageUrl} --scroll 3 --pattern '${domainPattern}'
+- 用 inspect 输出的最大容器签名填入 listSelector
+- 用 inspect 输出的 URL 样本收紧 urlPattern
+- 验证: node scripts/harvest.js ${id} 3 --preview
+`;
+    fs.writeFileSync(helperPath, template);
+    console.log(`\n✅ Scaffolded helpers/${id}.md`);
+    console.log(`   Next: run inspect, then fill in listSelector + urlPattern:`);
+    console.log(`   node scripts/inspect-source.js ${source.homepageUrl} --scroll 3 --pattern '${domainPattern}'\n`);
+    break;
+  }
+
   default:
     console.log(`
 Usage:
   node source-manage.js list
-  node source-manage.js add --id <id> --name <name> --url <homepage> --method browser|direct
-  node source-manage.js edit <id> --name "New Name" --channel 123456
+  node source-manage.js add --id <id> --name <name> --url <homepage> --method browser|direct [--pattern REGEX] [--folder NAME]
+  node source-manage.js edit <id> --name "New Name" --url https://... --method browser
   node source-manage.js enable <id>
   node source-manage.js disable <id>
+  node source-manage.js helper <id> [--force|--show]   # scaffold a helpers/<id>.md recipe
+
+Helper recipes: a source opts into the spec-driven fetcher by having a
+helpers/<id>.md file. Scaffold one with "helper <id>", inspect the page with
+scripts/inspect-source.js, then fill in listSelector + urlPattern.
     `);
 }