feat: better harvest action

badf6d70 · 谢宇轩 · 625392a1 · badf6d70 · badf6d70 · badf6d70
Commit badf6d70 authored Jul 29, 2026 by 谢宇轩
8 changed files
--- a/README.md
+++ b/README.md
@@ -21,22 +21,33 @@
 ## 工作原理
 ```
-config.json ──▶ harvest.js ──▶ fetcher-*.js ──▶ 文章数据
+config.json ──▶ harvest.js ──▶ 有 helpers/<id>.md ?
-  (数据源)       (主调度器)      (direct/browser)     │
+  (数据源)       (主调度器)        │
-       ▲              │                               ▼
+       ▲              │      ┌─────┴─────┐
-       │         dedup 去重                  写盘: 原文.md + registry + index
+       │              │      │ YES       │ NO
-       │         (registry)                  (中文标题用占位哨兵, 全文已落盘)
+       │              │      ▼           ▼
-       │                                            │
+       │              │  fetcher-helper  fetcher-direct/browser
-       │                              stdout: 紧凑 manifest (无全文, 仅预览)
+       │              │  (spec 驱动)      (legacy)
-       │                                            │
+       │              │      │           │
-       │                                            ▼
+       │              │      └─────┬─────┘
-       │                                       Agent(LLM)
+       │              │            ▼
-       │                                  读 manifest → 写 .summaries.json
+       │         dedup 去重       文章数据
-       │                                  (中文标题 + 四段摘要 + 分类)
+       │         (registry)        │
-       │                                            │
+       │                           ▼
-       └────────────────────────────────────────────┘
+       │                  写盘: 原文.md + registry + index
-                          finalize.js
+       │                  (中文标题用占位哨兵, 全文已落盘)
-            写摘要.md / 修补原文哨兵链接 / 更新 registry+index
+       │                                  │
+       │                  stderr: Day dir + 下一步命令
+       │                  stdout: 紧凑 manifest (无全文, 仅预览)
+       │                                  │
+       │                                  ▼
+       │                             Agent(LLM)
+       │                        读 manifest → 写 .summaries.json
+       │                        (中文标题 + 四段摘要 + 分类)
+       │                                  │
+       └──────────────────────────────────┘
+                      finalize.js
+        写摘要.md / 修补原文哨兵链接 / 更新 registry+index
 ```
 **脚本与 Agent 分工**：
@@ -69,7 +80,7 @@ news-harvester/
 │   ├── finalize.js                   # 补全中文摘要：node finalize.js <dayDir>
 │   ├── source-manage.js              # 数据源增删改查 + recipe scaffold
 │   ├── ensure-chromium.js            # 跨平台自动发现并启动 Chromium（browser 源用）
-│   ├── inspect-source.js             # 诊断工具：分析页面结构，辅助编写 helper.md
+│   ├── inspect-source.js             # 诊断工具：分析页面结构，--write 可一步生成 helper.md
 │   ├── cdp-client.js                 # 共享 CDP 基元（WebSocket/导航/滚动/求值）
 │   ├── fetcher-helper.js             # spec 驱动抓取（有 recipe 时用）
 │   ├── fetcher-direct.js             # 纯 HTTP 抓取（legacy direct 源）
@@ -241,23 +252,63 @@ claude -p \
 旧方案靠一个硬编码正则扫描整页 `<a>`，导航/侧栏/页脚的链接混进来，且 SPA 站点懒加载的首屏只有几篇。recipe 用 `listSelector` 把链接采集收窄到主文章流容器、用 `scrollSteps` 触发懒加载、用收紧的 `urlPattern` 校验——直接解决"分析新 CDP 源找不到高质量数据"的问题。recipe 是纯 markdown，站点改版时改文件即可，不动代码。
-### 添加新 browser 源（inspect → recipe → verify）
+### 工作流程
-```bash
+```
-# 1. 加入 config
+── 添加新源 ─────────────────────────────────────────────────
-node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser
+  source-manage add       inspect --write          harvest --preview
+  (写 config)        ──▶   (写 helpers/<id>.md)  ──▶  (只读验证)
+                                     │
+                                     ▼
+                          ┌── 收敛判定 ──────────────┐
+                          │  链接 = 真实文章?         │
+                          │  charCount > 500?        │
+                          └────┬──────────────┬──────┘
+                            YES              NO
+                             │                │
+                             ▼                ▼
+                          归档流程        手编 helpers/<id>.md
+                                      (收紧 urlPattern /
+                                       调 scrollSteps /
+                                       修 bodySelectors)
+                                           │
+                                           └──▶ 重跑 --preview ─┐
+                                                                 │
+── 归档流程 (Flow A) ─────────────────────────────────────────┘
+  harvest.js <id> <count>
+       │
+       ├─▶ 写 原文.md + registry + index (哨兵标题)
+       └─▶ stderr: Day dir + 下一步命令
+                  │
+                  ▼
+       写 .summaries.json (中文标题 + 四段摘要)
+                  │
+                  ▼
+       finalize.js "<dayDir>"
+       ├─▶ 写 中文摘要.md
+       ├─▶ 修补原文哨兵链接
+       └─▶ 更新 registry + index
+```
+### 添加新 browser 源（3 步闭环：add → inspect --write → verify）
-# 2. 诊断页面结构（输出按容器分组的链接 + 可粘贴的 recipe 骨架）
+```bash
-node scripts/inspect-source.js https://example.com/ --scroll 3 --pattern 'example\.com/.+'
+# 1. 加入 config（browser 源应配 recipe，articleUrlPattern 是 legacy 专用）
+node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser --folder "My Source"
-# 3. scaffold recipe 模板，然后填入 listSelector + urlPattern
+# 2. 诊断页面结构 + 自动写 recipe（实测最大非导航容器作 listSelector，真实 URL 推断 urlPattern）
-node scripts/source-manage.js helper mysrc
+node scripts/inspect-source.js https://example.com/ --scroll 3 --write mysrc
-#   编辑 helpers/mysrc.md ...
-# 4. 只读验证
+# 3. 只读验证
 node scripts/harvest.js mysrc 3 --preview
 ```
+**收敛判定**：preview 通过 = 抓到的链接是真实文章（非 nav/section 页）且 `charCount > 500`。不通过时按症状调 `helpers/mysrc.md`：链接是导航 → 收紧 `urlPattern`；文章太少 → 调大 `scrollSteps`；正文空 → 调 `bodySelectors`。改完重跑 preview。
+> 验证通过后归档：直接跑 `node scripts/harvest.js mysrc <count>`，stderr 会打印 `Day dir` 和下一步命令——按它执行 `.summaries.json` → `finalize.js`，不必 parse stdout。
 ### recipe 字段速查
 所有字段除 `urlPattern` 外都可选，缺省值 = legacy 行为。完整说明见 `references/source-management.md`（Helper recipes）和带注释的 `references/helper-example.md`。
@@ -278,12 +329,55 @@ node scripts/harvest.js mysrc 3 --preview
 >
 > **JSON-LD 兜底**：`dateSelector`/`authorSelector` 找不到时，`fetcher-helper.js` 自动读 `<script type="application/ld+json">` 的 `datePublished`/`author.name`。很多 SPA 站点（如 Nikkei）只在 JSON-LD 放元数据，所以通常不用自定义这两个选择器。
+### helper.md 示例
+下面是一份真实的 recipe（`helpers/reuters.md` 简化版）。frontmatter 是机器执行的结构化 spec，下方备忘是给人看的注释。完整带注释版见 `references/helper-example.md`。
+```yaml
+---
+source: reuters
+listSelector: 'div[data-testid="Title"]'      # 主文章流容器（inspect 实测）
+linkSelector: 'a[href]'
+urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$'   # 文章 URL 以日期结尾
+excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
+scrollSteps: 3                                 # 滚动触发懒加载
+scrollWaitMs: 2000
+waitMs: 15000
+bodySelectors:
+  - '[data-testid^="paragraph"]'              # Reuters 正文优先选择器
+  - '[class*="articleBodyContent"] p'
+  - 'article p'
+bodyMinParagraphs: 3
+dateSelector: 'time[datetime]'
+authorSelector: '[rel="author"], [class*="author"] a'
+imageSelector: 'img'
+imageMinWidth: 200
+excludeImage: [logo, icon, avatar, profile]
+titleStripSuffix: ' \| Reuters$'
+maxBodyChars: 15000
+---
+# Reuters 抓取备忘
+- 主文章流容器是 `div[data-testid="Title"]`（inspect 实测 59 篇）。
+- 文章 URL 以 `-YYYY-MM-DD/` 结尾，是过滤导航/section 页的关键。
+- 反爬：headless 会被拦，需 config.chromium.headless:false。
+- 正文优先 `[data-testid^="paragraph"]`，旧 `article p` 会被"相关阅读"污染。
+```
 ### 管理 recipe
 ```bash
+# 推荐：实测页面结构后一步写出可用 recipe
+node scripts/inspect-source.js <url> --scroll 3 --write <id>        # 写 helpers/<id>.md（已存在则拒绝）
+node scripts/inspect-source.js <url> --scroll 3 --write <id> --force  # 覆盖
+# 备用：scaffold 空模板（不含实测数据，需手填 listSelector/urlPattern）
 node scripts/source-manage.js helper <id>            # scaffold 模板（已存在则拒绝）
 node scripts/source-manage.js helper <id> --force    # 覆盖
 node scripts/source-manage.js helper <id> --show     # 查看 recipe 路径 + 是否存在
+# 诊断：只看页面结构不写盘（打印容器分组 + 可粘贴骨架）
+node scripts/inspect-source.js <url> --scroll 3 --pattern '<regex>'
 ```
 recipe 存放在 `helpers/`（入库共享，跨机器一致），区别于 `config.json`（机器特定，不入库）。

--- a/SKILL.md
+++ b/SKILL.md
@@ -211,7 +211,7 @@ No files are written. stdout is a compact JSON list (one call, no re-fetching /
 ### Guardrails (token hygiene)
- **Don't re-run `harvest.js`** for the same source/day just to "get more" — it deduplicates via the registry, so a second run only fetches genuinely new articles (and merges them into the pending `.manifest.json`, which is safe). If you need more, increase `count` on the first run.
+- **`count` is the target number of NEW articles**, not raw links. `harvest.js` already accounts for registry dedup: it scrolls to collect extra candidate links (`count + seen + buffer`, capped) so the *new* count can still reach `count` even when most of the homepage was already archived. If the run returns **fewer than `count`**, the stderr prints `⚠️ … stream may be exhausted` — that's a genuine source limit (no fresh articles to fetch), **not** a bug; re-running won't help. Wait for new articles or broaden the source instead. Don't re-run `harvest.js` for the same source/day just to "get more" — it deduplicates via the registry, so a second run only fetches genuinely new articles (and merges them into the pending `.manifest.json`, which is safe).
 - **Never `Read` the 原文.md files** to write summaries — the `bodyPreview` in the manifest is sufficient. Targeted slice reads are OK if truly needed.
 - **One `finalize.js` call** after writing `.summaries.json`. Do not hand-write file I/O — the scripts handle all original/registry/index/summary writes.

--- a/helpers/reuters-business.md
+++ b/helpers/reuters-business.md
+---
+source: reuters-business
+listSelector: null
+linkSelector: 'a[href]'
+urlPattern: 'reuters\.com/business/(?:[^/]+/)*[\w-]+-\d{4}-\d{2}-\d{2}/'
+excludeUrlPattern: null
+scrollSteps: 3
+scrollWaitMs: 1500
+waitMs: 12000
+bodySelectors:
+  - '[data-testid^="paragraph"]'
+  - '[class*="article-body"] p'
+  - 'article p'
+bodyMinParagraphs: 3
+dateSelector: 'time[datetime]'
+authorSelector: '[rel="author"], [class*="author"] a'
+imageSelector: 'img'
+imageMinWidth: 200
+excludeImage:
+  - logo
+  - icon
+  - avatar
+  - profile
+titleStripSuffix: ''
+maxBodyChars: 15000
+---
+# Reuters Business 抓取备忘
+- Reuters Business 首页的文章链接位于页面主体 `div[data-testid="Title"]`，但使用全页扫描配合严格 URL 过滤可避免依赖动态 CSS 类名。
+- 文章 URL 形如 `/business/<可选分区>/<slug>-YYYY-MM-DD/`；日期后缀用于排除导航、专题页和分区页。
+- `scrollSteps: 3` 触发首页懒加载，正文优先使用段落测试选择器。
--- a/helpers/reuters-markets.md
+++ b/helpers/reuters-markets.md
+---
+source: reuters-markets
+listSelector: "div.media-story-card-module__placement-container__1ZSB1"
+linkSelector: 'a[href]'
+urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}'
+excludeUrlPattern: null
+scrollSteps: 3
+scrollWaitMs: 2000
+waitMs: 15000
+bodySelectors:
+  - '[data-testid^="paragraph"]'
+  - '[class*="articleBodyContent"] p'
+  - '[class*="article-body"] p'
+  - 'article p'
+bodyMinParagraphs: 3
+dateSelector: 'time[datetime]'
+authorSelector: '[rel="author"], [class*="author"] a'
+imageSelector: 'img'
+imageMinWidth: 200
+excludeImage:
+  - logo
+  - icon
+  - avatar
+  - profile
+titleStripSuffix: ' \| Reuters$'
+maxBodyChars: 15000
+---
+# Reuters Markets 抓取备忘
+- 主文章流容器 `div.media-story-card-module__placement-container__1ZSB1` 实测 6 个候选链接（inspect 自动生成）。
+- urlPattern `reuters\.com/.+-\d{4}-\d{2}-\d{2}` 由样本 URL 推断，如 preview 漏抓/误抓请手编收紧。
+- 首屏可能不全，scrollSteps=0 触发懒加载；反爬严重时调高 waitMs。
+- 验证: node scripts/harvest.js reuters-markets 3 --preview（链接为真实文章且 charCount>500 即通过）
\ No newline at end of file
--- a/helpers/reuters-world.md
+++ b/helpers/reuters-world.md
+---
+source: reuters-world
+listSelector: 'div[data-testid="Title"]'
+linkSelector: 'a[href]'
+urlPattern: 'www\.reuters\.com/(world|business|markets|technology|sustainability)/.+-\d{4}-\d{2}-\d{2}/'
+excludeUrlPattern: null
+scrollSteps: 3
+scrollWaitMs: 2000
+waitMs: 15000
+bodySelectors:
+  - '[data-testid^="paragraph"]'
+  - '[class*="articleBodyContent"] p'
+  - '[class*="article-body"] p'
+  - 'article p'
+bodyMinParagraphs: 3
+dateSelector: 'time[datetime]'
+authorSelector: '[rel="author"], [class*="author"] a'
+imageSelector: 'img'
+imageMinWidth: 200
+excludeImage:
+  - logo
+  - icon
+  - avatar
+  - profile
+titleStripSuffix: ''
+maxBodyChars: 15000
+---
+# 路透社 World 抓取备忘
+- 主文章流容器 `li.link-group-module__item__Za3t6` 实测 30 个候选链接（inspect 自动生成）。
+- urlPattern `www\.reuters\.com/sitemap/.+` 由样本 URL 推断，如 preview 漏抓/误抓请手编收紧。
+- 首屏可能不全，scrollSteps=3 触发懒加载；反爬严重时调高 waitMs。
+- 验证: node scripts/harvest.js reuters-world 3 --preview（链接为真实文章且 charCount>500 即通过）
\ No newline at end of file
--- a/scripts/cdp-client.js
+++ b/scripts/cdp-client.js
@@ -163,7 +163,21 @@ async function navigateAndWait(ws, url, waitMs = 12000) {
 // Returns the merged array (of parsed objects). This is what lets SPA list
 // pages that lazy-load article cards yield their full set instead of just the
 // first screenful.
-async function scrollAndCollect(ws, { steps = 0, waitMs = 1500, collectExpr, dedupKey }) {
+//
+// Two loop modes (mutually exclusive):
+//   • Fixed steps (default): exactly `steps` scroll iterations, no early exit.
+//     Used by preview mode and any caller that just wants a fixed lazy-load
+//     pass. Backward compatible — no `targetCount` ⇒ this mode.
+//   • Target-count: when `targetCount` is set, scroll until `merged.length >=
+//     targetCount` OR `maxSteps` iterations OR `maxStaleSteps` consecutive
+//     scrolls add nothing (stream exhausted). `steps` is ignored in this mode.
+//     Used by archive harvest so the fetcher can keep loading a lazy stream
+//     until enough candidates exist to satisfy the requested count *after*
+//     dedup, instead of giving up at a fixed `scrollSteps` cap.
+async function scrollAndCollect(ws, {
+  steps = 0, waitMs = 1500, collectExpr, dedupKey,
+  targetCount = null, maxSteps = 10, maxStaleSteps = 3
+}) {
  const keyFn = dedupKey || (item => JSON.stringify(item));
  const seen = new Set();
  const merged = [];
@@ -181,6 +195,23 @@ async function scrollAndCollect(ws, { steps = 0, waitMs = 1500, collectExpr, ded
  absorb(await cdpEval(ws, collectExpr));
+  // Target-count mode: bounded by maxSteps, exits early on enough items or
+  // on a stale streak (consecutive scrolls adding zero new items ⇒ the lazy
+  // stream has run dry, so more scrolling won't help).
+  if (targetCount && merged.length < targetCount) {
+    let stale = 0;
+    for (let i = 0; i < maxSteps && merged.length < targetCount; i++) {
+      const before = merged.length;
+      await cdpEval(ws, 'window.scrollTo(0, document.body.scrollHeight)');
+      await sleep(waitMs);
+      absorb(await cdpEval(ws, collectExpr));
+      stale = (merged.length === before) ? stale + 1 : 0;
+      if (stale >= maxStaleSteps) break;
+    }
+    return merged;
+  }
+  // Fixed-steps mode (default / preview): backward compatible.
  for (let i = 0; i < steps; i++) {
    await cdpEval(ws, 'window.scrollTo(0, document.body.scrollHeight)');
    await sleep(waitMs);

--- a/scripts/fetcher-helper.js
+++ b/scripts/fetcher-helper.js
@@ -34,7 +34,9 @@ const DEFAULTS = {
  linkSelector: 'a[href]',     // candidate links inside each container
  urlPattern: null,            // REQUIRED — regex source; article URL validation
  excludeUrlPattern: null,     // regex source; URLs to drop even if they match urlPattern
-  scrollSteps: 0,              // bottom-scroll iterations to trigger lazy-load
+  scrollSteps: 0,              // bottom-scroll iterations to trigger lazy-load (fixed-steps mode)
+  maxScrollSteps: 12,          // scroll ceiling in target-count mode (archive harvest tops up
+                               // candidates until dedup can still satisfy the requested count)
  scrollWaitMs: 1500,          // pause after each scroll
  waitMs: 12000,               // initial render wait after load
  maxLinks: 30,
@@ -186,17 +188,34 @@ function buildExtractExpr(spec) {
 // Discover article links on a listing page. Returns absolute URL strings
 // (deduped, query/hash stripped, capped at spec.maxLinks) — the same shape
 // legacy fetchHomeLinks returns so harvest.js needs no changes downstream.
-async function fetchHomeLinks(url, spec) {
+//
+// Optional `desiredCount` (archive harvest): when > 0, switch scrollAndCollect
+// into target-count mode so a lazy stream keeps loading until ~desiredCount
+// candidates exist (or the stream goes stale / hits maxScrollSteps). This lets
+// the harvest top up candidates enough that, after registry dedup, the
+// requested count of *new* articles can still be satisfied — instead of being
+// capped by the fixed `scrollSteps` lazy-load pass. The slice cap is widened
+// to `max(maxLinks, desiredCount)` so the extra candidates aren't discarded.
+async function fetchHomeLinks(url, spec, desiredCount) {
  const { ws, close } = await openSession();
  try {
    await navigateAndWait(ws, url, spec.waitMs);
-    const items = await scrollAndCollect(ws, {
+    const opts = {
-      steps: spec.scrollSteps,
      waitMs: spec.scrollWaitMs,
      collectExpr: buildDiscoveryExpr(spec),
      dedupKey: item => item.url
-    });
+    };
-    return items.map(i => i.url).slice(0, spec.maxLinks);
+    if (desiredCount && desiredCount > 0) {
+      opts.targetCount = desiredCount;
+      opts.maxSteps = spec.maxScrollSteps;
+    } else {
+      opts.steps = spec.scrollSteps;
+    }
+    const items = await scrollAndCollect(ws, opts);
+    const cap = (desiredCount && desiredCount > 0)
+      ? Math.max(spec.maxLinks || 30, desiredCount)
+      : (spec.maxLinks || 30);
+    return items.map(i => i.url).slice(0, cap);
  } finally {
    close();
  }

--- a/scripts/harvest.js
+++ b/scripts/harvest.js
@@ -26,6 +26,14 @@ const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json')
 // Body preview length sent to the agent (full body stays on disk, never in context)
 const PREVIEW_LEN = 4000;
+// Upper bound on candidate links to request from a fetcher when topping up.
+// The harvest asks for `count + seen.size + buffer` candidates so registry
+// dedup can still leave `count` *new* articles; this caps that figure so a
+// large 7-day registry can't trigger hundreds of scroll iterations. The
+// fetcher's stale-exit stops early when a lazy stream runs dry, so a high
+// ask on a shallow stream is still cheap.
+const MAX_CANDIDATE_LINKS = 100;
 // ── Helpers ─────────────────────────────────────────────────────────────────
 function loadConfig() {
@@ -80,7 +88,8 @@ function sentinelFor(slug) {
 }
 function originalFilename(titleEn, sourceName) {
-  return `${titleEn} - ${sourceName}原文.md`;
+  const safeTitle = titleEn.replace(/[\\/:*?"<>|]/g, '-').replace(/-+/g, '-').trim();
+  return `${safeTitle} - ${sourceName}原文.md`;
 }
 // ── Registry ─────────────────────────────────────────────────────────────────
@@ -179,7 +188,7 @@ ${data.summaryZh || '（摘要生成中）'}
 }
 function generateOriginalDoc(data, sourceName, imgFiles) {
-  const { titleEn, titleZh, date, authors, source: url, body } = data;
+  const { titleEn, titleZh, date, authors, url, body } = data;
  const tags = (data.tags || []).map(t => `  - ${t}`).join('\n');
  // Empty-alt guard: only render the caption line when alt text exists, so we
  // never emit a bare `**` (empty bold) for avatar/tracking images.
@@ -396,8 +405,15 @@ Examples:
  // while fetcher-helper takes (url, spec). The closures hide that difference
  // so the main flow below is identical for both paths. NB: spec must NOT be
  // passed to legacy fetchPage (it would be coerced to waitMs=NaN).
-  const getLinks = () => spec
+  // Call-site adapters: legacy fetchers take (url, pattern) / (url[, waitMs]),
-    ? fetcher.fetchHomeLinks(source.homepageUrl, spec)
+  // while fetcher-helper takes (url, spec[, desiredCount]). The closures hide
+  // that difference so the main flow below is identical for both paths.
+  // NB: spec must NOT be passed to legacy fetchPage (it would be coerced to
+  // waitMs=NaN). desiredCount is only threaded into the spec path — legacy
+  // sources do a single-page fetch with no scroll/pagination, so they can't
+  // top up candidates (that's a separate, larger change).
+  const getLinks = (desiredCount) => spec
+    ? fetcher.fetchHomeLinks(source.homepageUrl, spec, desiredCount)
    : fetcher.fetchHomeLinks(source.homepageUrl, source.articleUrlPattern);
  const getPage = (url) => spec
    ? fetcher.fetchPage(url, spec)
@@ -407,16 +423,30 @@ Examples:
  const seen = preview ? new Set() : buildDedupSet(config, sourceId);
  console.error(`   Registry: ${seen.size} URLs in last ${config.registryWindowDays || 7} days`);
+  // Ask the fetcher for enough candidate links that, after registry dedup
+  // eats the already-archived ones, `count` NEW articles can still remain.
+  // `count + seen.size` covers the worst case (every seen URL reappears in the
+  // candidates); `+ 10` is a buffer for candidates that fetch but are then
+  // dropped as low-content. Capped so a large registry can't trigger hundreds
+  // of scroll iterations — the fetcher's stale-exit stops a shallow stream
+  // early regardless. Preview stays null (fixed scrollSteps, speed-first).
+  const desiredCount = preview
+    ? null
+    : Math.min(count + seen.size + 10, MAX_CANDIDATE_LINKS);
+  if (!preview && desiredCount > count) {
+    console.error(`   Targeting ${desiredCount} candidates to net ${count} new after dedup`);
+  }
  // Fetch homepage links
  console.error(`\n📡 Fetching homepage: ${source.homepageUrl}`);
  let links;
  try {
-    links = await getLinks();
+    links = await getLinks(desiredCount);
  } catch (e) {
    console.error('Failed to fetch homepage:', e.message);
    process.exit(1);
  }
-  console.error(`   Found ${links.length} article links`);
+  console.error(`   Collected ${links.length} candidate links`);
  // Normalize URLs (strip query params for dedup)
  const normalizeUrl = (u) => { try { const p = new URL(u); p.search = ''; return p.toString(); } catch(e) { return u; } };
@@ -424,6 +454,9 @@ Examples:
  const newLinks = normalizedLinks.filter(u => !seen.has(u)).slice(0, count);
  const skipped = normalizedLinks.filter(u => seen.has(u)).length;
  console.error(`   New: ${newLinks.length} | Already seen: ${skipped}`);
+  if (!preview && newLinks.length < count) {
+    console.error(`   ⚠️  Only ${newLinks.length} new of ${count} requested — stream may be exhausted (fewer fresh articles than asked). Re-running won't help; wait for new articles or broaden the source.`);
+  }
  // ── Preview mode: fetch only, print compact list, write nothing ───────────
  if (preview) {