Skip to content

N
news-harvester
Commit badf6d70 authored by 谢宇轩's avatar 谢宇轩
Browse files
Options

feat: better harvest action

parent 625392a1
Hide whitespace changes
Inline Side-by-side
Showing with 315 additions and 41 deletions
README.md
View file @ badf6d70
......@@ -21,22 +21,33 @@
## 工作原理
```
config.json ──▶ harvest.js ──▶ fetcher-*.js ──▶ 文章数据
(数据源) (主调度器) (direct/browser) │
▲ │ ▼
│ dedup 去重 写盘: 原文.md + registry + index
│ (registry) (中文标题用占位哨兵, 全文已落盘)
│ │
│ stdout: 紧凑 manifest (无全文, 仅预览)
│ │
│ ▼
│ Agent(LLM)
│ 读 manifest → 写 .summaries.json
│ (中文标题 + 四段摘要 + 分类)
│ │
└────────────────────────────────────────────┘
finalize.js
写摘要.md / 修补原文哨兵链接 / 更新 registry+index
config.json ──▶ harvest.js ──▶ 有 helpers/<id>.md ?
(数据源) (主调度器) │
▲ │ ┌─────┴─────┐
│ │ │ YES │ NO
│ │ ▼ ▼
│ │ fetcher-helper fetcher-direct/browser
│ │ (spec 驱动) (legacy)
│ │ │ │
│ │ └─────┬─────┘
│ │ ▼
│ dedup 去重 文章数据
│ (registry) │
│ ▼
│ 写盘: 原文.md + registry + index
│ (中文标题用占位哨兵, 全文已落盘)
│ │
│ stderr: Day dir + 下一步命令
│ stdout: 紧凑 manifest (无全文, 仅预览)
│ │
│ ▼
│ Agent(LLM)
│ 读 manifest → 写 .summaries.json
│ (中文标题 + 四段摘要 + 分类)
│ │
└──────────────────────────────────┘
finalize.js
写摘要.md / 修补原文哨兵链接 / 更新 registry+index
```
**脚本与 Agent 分工**：
......@@ -69,7 +80,7 @@ news-harvester/
│ ├── finalize.js # 补全中文摘要：node finalize.js <dayDir>
│ ├── source-manage.js # 数据源增删改查 + recipe scaffold
│ ├── ensure-chromium.js # 跨平台自动发现并启动 Chromium（browser 源用）
│ ├── inspect-source.js # 诊断工具：分析页面结构，辅助编写 helper.md
│ ├── inspect-source.js # 诊断工具：分析页面结构，--write 可一步生成 helper.md
│ ├── cdp-client.js # 共享 CDP 基元（WebSocket/导航/滚动/求值）
│ ├── fetcher-helper.js # spec 驱动抓取（有 recipe 时用）
│ ├── fetcher-direct.js # 纯 HTTP 抓取（legacy direct 源）
......@@ -241,23 +252,63 @@ claude -p \
旧方案靠一个硬编码正则扫描整页 `<a>`，导航/侧栏/页脚的链接混进来，且 SPA 站点懒加载的首屏只有几篇。recipe 用 `listSelector` 把链接采集收窄到主文章流容器、用 `scrollSteps` 触发懒加载、用收紧的 `urlPattern` 校验——直接解决"分析新 CDP 源找不到高质量数据"的问题。recipe 是纯 markdown，站点改版时改文件即可，不动代码。
### 添加新 browser 源（inspect → recipe → verify）
### 工作流程
```bash
# 1. 加入 config
node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser
```
── 添加新源 ─────────────────────────────────────────────────
source-manage add inspect --write harvest --preview
(写 config) ──▶ (写 helpers/<id>.md) ──▶ (只读验证)
│
▼
┌── 收敛判定 ──────────────┐
│ 链接 = 真实文章? │
│ charCount > 500? │
└────┬──────────────┬──────┘
YES NO
│ │
▼ ▼
归档流程 手编 helpers/<id>.md
(收紧 urlPattern /
调 scrollSteps /
修 bodySelectors)
│
└──▶ 重跑 --preview ─┐
│
── 归档流程 (Flow A) ─────────────────────────────────────────┘
harvest.js <id> <count>
│
├─▶ 写 原文.md + registry + index (哨兵标题)
└─▶ stderr: Day dir + 下一步命令
│
▼
写 .summaries.json (中文标题 + 四段摘要)
│
▼
finalize.js "<dayDir>"
├─▶ 写 中文摘要.md
├─▶ 修补原文哨兵链接
└─▶ 更新 registry + index
```
### 添加新 browser 源（3 步闭环：add → inspect --write → verify）
# 2. 诊断页面结构（输出按容器分组的链接 + 可粘贴的 recipe 骨架）
node scripts/inspect-source.js https://example.com/ --scroll 3 --pattern 'example\.com/.+'
```bash
# 1. 加入 config（browser 源应配 recipe，articleUrlPattern 是 legacy 专用）
node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser --folder "My Source"
# 3. scaffold recipe 模板，然后填入 listSelector + urlPattern
node scripts/source-manage.js helper mysrc
# 编辑 helpers/mysrc.md ...
# 2. 诊断页面结构 + 自动写 recipe（实测最大非导航容器作 listSelector，真实 URL 推断 urlPattern）
node scripts/inspect-source.js https://example.com/ --scroll 3 --write mysrc
# 4. 只读验证
# 3. 只读验证
node scripts/harvest.js mysrc 3 --preview
```
**收敛判定**：preview 通过 = 抓到的链接是真实文章（非 nav/section 页）且 `charCount > 500`。不通过时按症状调 `helpers/mysrc.md`：链接是导航 → 收紧 `urlPattern`；文章太少 → 调大 `scrollSteps`；正文空 → 调 `bodySelectors`。改完重跑 preview。
> 验证通过后归档：直接跑 `node scripts/harvest.js mysrc <count>`，stderr 会打印 `Day dir` 和下一步命令——按它执行 `.summaries.json` → `finalize.js`，不必 parse stdout。
### recipe 字段速查
所有字段除 `urlPattern` 外都可选，缺省值 = legacy 行为。完整说明见 `references/source-management.md`（Helper recipes）和带注释的 `references/helper-example.md`。
......@@ -278,12 +329,55 @@ node scripts/harvest.js mysrc 3 --preview
>
> **JSON-LD 兜底**：`dateSelector`/`authorSelector` 找不到时，`fetcher-helper.js` 自动读 `<script type="application/ld+json">` 的 `datePublished`/`author.name`。很多 SPA 站点（如 Nikkei）只在 JSON-LD 放元数据，所以通常不用自定义这两个选择器。
### helper.md 示例
下面是一份真实的 recipe（`helpers/reuters.md` 简化版）。frontmatter 是机器执行的结构化 spec，下方备忘是给人看的注释。完整带注释版见 `references/helper-example.md`。
```yaml
---
source: reuters
listSelector: 'div[data-testid="Title"]' # 主文章流容器（inspect 实测）
linkSelector: 'a[href]'
urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$' # 文章 URL 以日期结尾
excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
scrollSteps: 3 # 滚动触发懒加载
scrollWaitMs: 2000
waitMs: 15000
bodySelectors:
- '[data-testid^="paragraph"]' # Reuters 正文优先选择器
- '[class*="articleBodyContent"] p'
- 'article p'
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
imageSelector: 'img'
imageMinWidth: 200
excludeImage: [logo, icon, avatar, profile]
titleStripSuffix: ' \| Reuters$'
maxBodyChars: 15000
---
# Reuters 抓取备忘
- 主文章流容器是 `div[data-testid="Title"]`（inspect 实测 59 篇）。
- 文章 URL 以 `-YYYY-MM-DD/` 结尾，是过滤导航/section 页的关键。
- 反爬：headless 会被拦，需 config.chromium.headless:false。
- 正文优先 `[data-testid^="paragraph"]`，旧 `article p` 会被"相关阅读"污染。
```
### 管理 recipe
```bash
# 推荐：实测页面结构后一步写出可用 recipe
node scripts/inspect-source.js <url> --scroll 3 --write <id> # 写 helpers/<id>.md（已存在则拒绝）
node scripts/inspect-source.js <url> --scroll 3 --write <id> --force # 覆盖
# 备用：scaffold 空模板（不含实测数据，需手填 listSelector/urlPattern）
node scripts/source-manage.js helper <id> # scaffold 模板（已存在则拒绝）
node scripts/source-manage.js helper <id> --force # 覆盖
node scripts/source-manage.js helper <id> --show # 查看 recipe 路径 + 是否存在
# 诊断：只看页面结构不写盘（打印容器分组 + 可粘贴骨架）
node scripts/inspect-source.js <url> --scroll 3 --pattern '<regex>'
```
recipe 存放在 `helpers/`（入库共享，跨机器一致），区别于 `config.json`（机器特定，不入库）。
......
SKILL.md
View file @ badf6d70
......@@ -211,7 +211,7 @@ No files are written. stdout is a compact JSON list (one call, no re-fetching /
### Guardrails (token hygiene)
- **Don't re-run `harvest.js`** for the same source/day just to "get more" — it deduplicates via the registry, so a second run only fetches genuinely new articles (and merges them into the pending `.manifest.json`, which is safe). If you need more, increase `count` on the first run.
- **`count` is the target number of NEW articles**, not raw links. `harvest.js` already accounts for registry dedup: it scrolls to collect extra candidate links (`count + seen + buffer`, capped) so the *new* count can still reach `count` even when most of the homepage was already archived. If the run returns **fewer than `count`**, the stderr prints `⚠️ … stream may be exhausted` — that's a genuine source limit (no fresh articles to fetch), **not** a bug; re-running won't help. Wait for new articles or broaden the source instead. Don't re-run `harvest.js` for the same source/day just to "get more" — it deduplicates via the registry, so a second run only fetches genuinely new articles (and merges them into the pending `.manifest.json`, which is safe).
- **Never `Read` the 原文.md files** to write summaries — the `bodyPreview` in the manifest is sufficient. Targeted slice reads are OK if truly needed.
- **One `finalize.js` call** after writing `.summaries.json`. Do not hand-write file I/O — the scripts handle all original/registry/index/summary writes.
......
helpers/reuters-business.md 0 → 100644
View file @ badf6d70
---
source: reuters-business
listSelector: null
linkSelector: 'a[href]'
urlPattern: 'reuters\.com/business/(?:[^/]+/)*[\w-]+-\d{4}-\d{2}-\d{2}/'
excludeUrlPattern: null
scrollSteps: 3
scrollWaitMs: 1500
waitMs: 12000
bodySelectors:
- '[data-testid^="paragraph"]'
- '[class*="article-body"] p'
- 'article p'
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
- logo
- icon
- avatar
- profile
titleStripSuffix: ''
maxBodyChars: 15000
---
# Reuters Business 抓取备忘
- Reuters Business 首页的文章链接位于页面主体 `div[data-testid="Title"]`，但使用全页扫描配合严格 URL 过滤可避免依赖动态 CSS 类名。
- 文章 URL 形如 `/business/<可选分区>/<slug>-YYYY-MM-DD/`；日期后缀用于排除导航、专题页和分区页。
- `scrollSteps: 3` 触发首页懒加载，正文优先使用段落测试选择器。
helpers/reuters-markets.md 0 → 100644
View file @ badf6d70
---
source: reuters-markets
listSelector: "div.media-story-card-module__placement-container__1ZSB1"
linkSelector: 'a[href]'
urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}'
excludeUrlPattern: null
scrollSteps: 3
scrollWaitMs: 2000
waitMs: 15000
bodySelectors:
- '[data-testid^="paragraph"]'
- '[class*="articleBodyContent"] p'
- '[class*="article-body"] p'
- 'article p'
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
- logo
- icon
- avatar
- profile
titleStripSuffix: ' \| Reuters$'
maxBodyChars: 15000
---
# Reuters Markets 抓取备忘
- 主文章流容器 `div.media-story-card-module__placement-container__1ZSB1` 实测 6 个候选链接（inspect 自动生成）。
- urlPattern `reuters\.com/.+-\d{4}-\d{2}-\d{2}` 由样本 URL 推断，如 preview 漏抓/误抓请手编收紧。
- 首屏可能不全，scrollSteps=0 触发懒加载；反爬严重时调高 waitMs。
- 验证: node scripts/harvest.js reuters-markets 3 --preview（链接为真实文章且 charCount>500 即通过）
\ No newline at end of file
helpers/reuters-world.md 0 → 100644
View file @ badf6d70
---
source: reuters-world
listSelector: 'div[data-testid="Title"]'
linkSelector: 'a[href]'
urlPattern: 'www\.reuters\.com/(world|business|markets|technology|sustainability)/.+-\d{4}-\d{2}-\d{2}/'
excludeUrlPattern: null
scrollSteps: 3
scrollWaitMs: 2000
waitMs: 15000
bodySelectors:
- '[data-testid^="paragraph"]'
- '[class*="articleBodyContent"] p'
- '[class*="article-body"] p'
- 'article p'
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
- logo
- icon
- avatar
- profile
titleStripSuffix: ''
maxBodyChars: 15000
---
# 路透社 World 抓取备忘
- 主文章流容器 `li.link-group-module__item__Za3t6` 实测 30 个候选链接（inspect 自动生成）。
- urlPattern `www\.reuters\.com/sitemap/.+` 由样本 URL 推断，如 preview 漏抓/误抓请手编收紧。
- 首屏可能不全，scrollSteps=3 触发懒加载；反爬严重时调高 waitMs。
- 验证: node scripts/harvest.js reuters-world 3 --preview（链接为真实文章且 charCount>500 即通过）
\ No newline at end of file
scripts/cdp-client.js
View file @ badf6d70
......@@ -163,7 +163,21 @@ async function navigateAndWait(ws, url, waitMs = 12000) {
// Returns the merged array (of parsed objects). This is what lets SPA list
// pages that lazy-load article cards yield their full set instead of just the
// first screenful.
async function scrollAndCollect(ws, { steps = 0, waitMs = 1500, collectExpr, dedupKey }) {
//
// Two loop modes (mutually exclusive):
// • Fixed steps (default): exactly `steps` scroll iterations, no early exit.
// Used by preview mode and any caller that just wants a fixed lazy-load
// pass. Backward compatible — no `targetCount` ⇒ this mode.
// • Target-count: when `targetCount` is set, scroll until `merged.length >=
// targetCount` OR `maxSteps` iterations OR `maxStaleSteps` consecutive
// scrolls add nothing (stream exhausted). `steps` is ignored in this mode.
// Used by archive harvest so the fetcher can keep loading a lazy stream
// until enough candidates exist to satisfy the requested count *after*
// dedup, instead of giving up at a fixed `scrollSteps` cap.
async function scrollAndCollect(ws, {
steps = 0, waitMs = 1500, collectExpr, dedupKey,
targetCount = null, maxSteps = 10, maxStaleSteps = 3
}) {
const keyFn = dedupKey || (item => JSON.stringify(item));
const seen = new Set();
const merged = [];
......@@ -181,6 +195,23 @@ async function scrollAndCollect(ws, { steps = 0, waitMs = 1500, collectExpr, ded
absorb(await cdpEval(ws, collectExpr));
// Target-count mode: bounded by maxSteps, exits early on enough items or
// on a stale streak (consecutive scrolls adding zero new items ⇒ the lazy
// stream has run dry, so more scrolling won't help).
if (targetCount && merged.length < targetCount) {
let stale = 0;
for (let i = 0; i < maxSteps && merged.length < targetCount; i++) {
const before = merged.length;
await cdpEval(ws, 'window.scrollTo(0, document.body.scrollHeight)');
await sleep(waitMs);
absorb(await cdpEval(ws, collectExpr));
stale = (merged.length === before) ? stale + 1 : 0;
if (stale >= maxStaleSteps) break;
}
return merged;
}
// Fixed-steps mode (default / preview): backward compatible.
for (let i = 0; i < steps; i++) {
await cdpEval(ws, 'window.scrollTo(0, document.body.scrollHeight)');
await sleep(waitMs);
......
scripts/fetcher-helper.js
View file @ badf6d70
......@@ -34,7 +34,9 @@ const DEFAULTS = {
linkSelector: 'a[href]', // candidate links inside each container
urlPattern: null, // REQUIRED — regex source; article URL validation
excludeUrlPattern: null, // regex source; URLs to drop even if they match urlPattern
scrollSteps: 0, // bottom-scroll iterations to trigger lazy-load
scrollSteps: 0, // bottom-scroll iterations to trigger lazy-load (fixed-steps mode)
maxScrollSteps: 12, // scroll ceiling in target-count mode (archive harvest tops up
// candidates until dedup can still satisfy the requested count)
scrollWaitMs: 1500, // pause after each scroll
waitMs: 12000, // initial render wait after load
maxLinks: 30,
......@@ -186,17 +188,34 @@ function buildExtractExpr(spec) {
// Discover article links on a listing page. Returns absolute URL strings
// (deduped, query/hash stripped, capped at spec.maxLinks) — the same shape
// legacy fetchHomeLinks returns so harvest.js needs no changes downstream.
async function fetchHomeLinks(url, spec) {
//
// Optional `desiredCount` (archive harvest): when > 0, switch scrollAndCollect
// into target-count mode so a lazy stream keeps loading until ~desiredCount
// candidates exist (or the stream goes stale / hits maxScrollSteps). This lets
// the harvest top up candidates enough that, after registry dedup, the
// requested count of *new* articles can still be satisfied — instead of being
// capped by the fixed `scrollSteps` lazy-load pass. The slice cap is widened
// to `max(maxLinks, desiredCount)` so the extra candidates aren't discarded.
async function fetchHomeLinks(url, spec, desiredCount) {
const { ws, close } = await openSession();
try {
await navigateAndWait(ws, url, spec.waitMs);
const items = await scrollAndCollect(ws, {
steps: spec.scrollSteps,
const opts = {
waitMs: spec.scrollWaitMs,
collectExpr: buildDiscoveryExpr(spec),
dedupKey: item => item.url
});
return items.map(i => i.url).slice(0, spec.maxLinks);
};
if (desiredCount && desiredCount > 0) {
opts.targetCount = desiredCount;
opts.maxSteps = spec.maxScrollSteps;
} else {
opts.steps = spec.scrollSteps;
}
const items = await scrollAndCollect(ws, opts);
const cap = (desiredCount && desiredCount > 0)
? Math.max(spec.maxLinks || 30, desiredCount)
: (spec.maxLinks || 30);
return items.map(i => i.url).slice(0, cap);
} finally {
close();
}
......
scripts/harvest.js
View file @ badf6d70
......@@ -26,6 +26,14 @@ const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json')
// Body preview length sent to the agent (full body stays on disk, never in context)
const PREVIEW_LEN = 4000;
// Upper bound on candidate links to request from a fetcher when topping up.
// The harvest asks for `count + seen.size + buffer` candidates so registry
// dedup can still leave `count` *new* articles; this caps that figure so a
// large 7-day registry can't trigger hundreds of scroll iterations. The
// fetcher's stale-exit stops early when a lazy stream runs dry, so a high
// ask on a shallow stream is still cheap.
const MAX_CANDIDATE_LINKS = 100;
// ── Helpers ─────────────────────────────────────────────────────────────────
function loadConfig() {
......@@ -80,7 +88,8 @@ function sentinelFor(slug) {
}
function originalFilename(titleEn, sourceName) {
return `${titleEn} - ${sourceName}原文.md`;
const safeTitle = titleEn.replace(/[\\/:*?"<>|]/g, '-').replace(/-+/g, '-').trim();
return `${safeTitle} - ${sourceName}原文.md`;
}
// ── Registry ─────────────────────────────────────────────────────────────────
......@@ -179,7 +188,7 @@ ${data.summaryZh || '（摘要生成中）'}
}
function generateOriginalDoc(data, sourceName, imgFiles) {
const { titleEn, titleZh, date, authors, source: url, body } = data;
const { titleEn, titleZh, date, authors, url, body } = data;
const tags = (data.tags || []).map(t => ` - ${t}`).join('\n');
// Empty-alt guard: only render the caption line when alt text exists, so we
// never emit a bare `**` (empty bold) for avatar/tracking images.
......@@ -396,8 +405,15 @@ Examples:
// while fetcher-helper takes (url, spec). The closures hide that difference
// so the main flow below is identical for both paths. NB: spec must NOT be
// passed to legacy fetchPage (it would be coerced to waitMs=NaN).
const getLinks = () => spec
? fetcher.fetchHomeLinks(source.homepageUrl, spec)
// Call-site adapters: legacy fetchers take (url, pattern) / (url[, waitMs]),
// while fetcher-helper takes (url, spec[, desiredCount]). The closures hide
// that difference so the main flow below is identical for both paths.
// NB: spec must NOT be passed to legacy fetchPage (it would be coerced to
// waitMs=NaN). desiredCount is only threaded into the spec path — legacy
// sources do a single-page fetch with no scroll/pagination, so they can't
// top up candidates (that's a separate, larger change).
const getLinks = (desiredCount) => spec
? fetcher.fetchHomeLinks(source.homepageUrl, spec, desiredCount)
: fetcher.fetchHomeLinks(source.homepageUrl, source.articleUrlPattern);
const getPage = (url) => spec
? fetcher.fetchPage(url, spec)
......@@ -407,16 +423,30 @@ Examples:
const seen = preview ? new Set() : buildDedupSet(config, sourceId);
console.error(` Registry: ${seen.size} URLs in last ${config.registryWindowDays || 7} days`);
// Ask the fetcher for enough candidate links that, after registry dedup
// eats the already-archived ones, `count` NEW articles can still remain.
// `count + seen.size` covers the worst case (every seen URL reappears in the
// candidates); `+ 10` is a buffer for candidates that fetch but are then
// dropped as low-content. Capped so a large registry can't trigger hundreds
// of scroll iterations — the fetcher's stale-exit stops a shallow stream
// early regardless. Preview stays null (fixed scrollSteps, speed-first).
const desiredCount = preview
? null
: Math.min(count + seen.size + 10, MAX_CANDIDATE_LINKS);
if (!preview && desiredCount > count) {
console.error(` Targeting ${desiredCount} candidates to net ${count} new after dedup`);
}
// Fetch homepage links
console.error(`\n📡 Fetching homepage: ${source.homepageUrl}`);
let links;
try {
links = await getLinks();
links = await getLinks(desiredCount);
} catch (e) {
console.error('Failed to fetch homepage:', e.message);
process.exit(1);
}
console.error(` Found ${links.length} article links`);
console.error(` Collected ${links.length} candidate links`);
// Normalize URLs (strip query params for dedup)
const normalizeUrl = (u) => { try { const p = new URL(u); p.search = ''; return p.toString(); } catch(e) { return u; } };
......@@ -424,6 +454,9 @@ Examples:
const newLinks = normalizedLinks.filter(u => !seen.has(u)).slice(0, count);
const skipped = normalizedLinks.filter(u => seen.has(u)).length;
console.error(` New: ${newLinks.length} | Already seen: ${skipped}`);
if (!preview && newLinks.length < count) {
console.error(` ⚠️ Only ${newLinks.length} new of ${count} requested — stream may be exhausted (fewer fresh articles than asked). Re-running won't help; wait for new articles or broaden the source.`);
}
// ── Preview mode: fetch only, print compact list, write nothing ───────────
if (preview) {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment