Commit 6e0d6021 authored by 谢宇轩's avatar 谢宇轩

feat: commit Reuters section bundles, untrack local helpers

Export reuters-business, reuters-markets, reuters-world as shareable
.nhsource.json bundles (the committed unit for sources). Untrack the
helpers/ directory — recipes/fixtures are now installed from bundles
or authored locally; only bundles are committed. Add helpers/ to
.gitignore so product files stay on disk for harvest but out of git.
parent ba940690
......@@ -12,5 +12,10 @@ node_modules/
*.log
/tmp/
# Local source recipes & fixtures (product files)
# These are installed from shareable bundles (bundles/*.nhsource.json) or
# authored locally — the bundle is the committed unit, not the helper.
helpers/
# IDE / agent workspace files
.zcode/
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-07-30T04:12:38.724Z",
"source": {
"id": "reuters-business",
"name": "Reuters Business",
"homepageUrl": "https://www.reuters.com/business/",
"method": "browser",
"vaultFolder": "Reuters Business"
},
"files": {
"helpers/reuters-business.md": "---\nsource: reuters-business\nlistSelector: 'div[data-testid=\"Title\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$'\nexcludeUrlPattern: 'reuters\\.com/(podcasts|newsletter)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[data-testid^=\"paragraph\"]'\n - '[class*=\"articleBodyContent\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[rel=\"author\"], [class*=\"author\"] a'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\ntitleStripSuffix: ' \\| Reuters$'\nmaxBodyChars: 15000\n---\n# Reuters Business 抓取备忘\n\n## 列表页链接收集(`/business/`)\n\n- **listSelector: `div[data-testid=\"Title\"]`**。实测全页 20 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定(hash 不变)。\n- **urlPattern: `reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/business/`、`/business/finance/`)无日期后缀。放宽为全站匹配(不限 `/business/`),因为板块页会推荐其他分区的文章(如 `/world/asia-pacific/...`)。\n- **excludeUrlPattern** 排除 `/podcasts/`、`/newsletter/`(它们也带日期但非正文报道)。\n- `scrollSteps: 3` 触发懒加载拿全当天列表;首屏仅 ~10 篇。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:\n- 文章:`og:type=article`、`@type=NewsArticle`、正文 4000+ 字符\n- 分类页 `/business/`:`og:type=website`、`@type=CollectionPage`、正文 0\n\nReuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。\n\n## 其他\n\n- 反爬:headless 会被拦,需 `config.chromium.headless: false`。\n- 正文优先 `[data-testid^=\"paragraph\"]`,旧 `article p` 会被\"相关阅读\"\"订阅推广\"段落污染。\n"
}
}
\ No newline at end of file
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-07-30T04:12:38.760Z",
"source": {
"id": "reuters-markets",
"name": "Reuters Markets",
"homepageUrl": "https://www.reuters.com/markets/",
"method": "browser",
"vaultFolder": "Reuters Markets"
},
"files": {
"helpers/reuters-markets.md": "---\nsource: reuters-markets\nlistSelector: '[data-testid=\"MediaStoryCard\"], [data-testid=\"BasicCard\"], [data-testid=\"HeroCard\"], [data-testid=\"HubCard\"], [data-testid=\"AuthorCard\"], [data-testid=\"OurColumnists\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$'\nexcludeUrlPattern: 'reuters\\.com/(podcasts|newsletter)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[data-testid^=\"paragraph\"]'\n - '[class*=\"articleBodyContent\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[rel=\"author\"], [class*=\"author\"] a'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\ntitleStripSuffix: ' \\| Reuters$'\nmaxBodyChars: 15000\n---\n# Reuters Markets 抓取备忘\n\n## 列表页链接收集(`/markets/`)\n\n- **listSelector 用复合 `data-testid` 选择器**:`MediaStoryCard`、`BasicCard`、`HeroCard`、`HubCard`、`AuthorCard`、`OurColumnists` 六类卡片容器。实测全页 20 篇文章链接分散在这六类容器里(`MediaStoryCard` 18 + `BasicCard` 12 + `HeroCard`/`HubCard`/`AuthorCard`/`OurColumnists` 各 2-3,去重后 20)。\n- **为什么不用旧 `div.media-story-card-module__placement-container__1ZSB1`**:旧选择器依赖带 hash 的 CSS Modules 类名,hash 随构建变化 → 过期后只匹配 9 篇,漏掉 11 篇。`data-testid` 无 hash,稳定得多。Markets 页面布局与 Business/World 不同(`div[data-testid=\"Title\"]` 在此页 0 篇),必须用 Markets 专属的复合容器。\n- **urlPattern: `reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/markets/`、`/markets/currencies/`)无日期后缀。放宽为全站匹配,因为 Markets 板块会推荐其他分区文章(如 `/world/asia-pacific/...`、`/business/energy/...`、`/commentary/...`)。\n- **excludeUrlPattern** 排除 `/podcasts/`、`/newsletter/`(它们也带日期但非正文报道)。\n- `scrollSteps: 3` 触发懒加载拿全当天列表。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:\n- 文章:`og:type=article`、`@type=NewsArticle`、正文 4000+ 字符\n- 分类页 `/markets/`:`og:type=website`、正文 0\n\nReuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。\n\n## 其他\n\n- 反爬:headless 会被拦,需 `config.chromium.headless: false`。\n- 正文优先 `[data-testid^=\"paragraph\"]`,旧 `article p` 会被\"相关阅读\"\"订阅推广\"段落污染。\n"
}
}
\ No newline at end of file
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-07-30T04:12:38.797Z",
"source": {
"id": "reuters-world",
"name": "路透社 World",
"homepageUrl": "https://www.reuters.com/world/",
"method": "browser",
"vaultFolder": "路透社 World"
},
"files": {
"helpers/reuters-world.md": "---\nsource: reuters-world\nlistSelector: 'div[data-testid=\"Title\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$'\nexcludeUrlPattern: 'reuters\\.com/(podcasts|newsletter)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[data-testid^=\"paragraph\"]'\n - '[class*=\"articleBodyContent\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[rel=\"author\"], [class*=\"author\"] a'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\ntitleStripSuffix: ' \\| Reuters$'\nmaxBodyChars: 15000\n---\n# 路透社 World 抓取备忘\n\n## 列表页链接收集(`/world/`)\n\n- **listSelector: `div[data-testid=\"Title\"]`**。实测全页 19 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定。\n- **urlPattern: `reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/world/`、`/world/asia-pacific/`)无日期后缀。放宽为全站匹配(不限 `/world/`),因为板块页会推荐其他分区的文章(如 `/business/environment/...`、`/legal/government/...`)。\n- **excludeUrlPattern** 排除 `/podcasts/`、`/newsletter/`(它们也带日期但非正文报道)。\n- `scrollSteps: 3` 触发懒加载拿全当天列表。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:\n- 文章:`og:type=article`、`@type=NewsArticle`、正文 6000+ 字符\n- 分类页 `/world/`:`og:type=website`、正文 0\n\nReuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。\n\n## 其他\n\n- 反爬:headless 会被拦,需 `config.chromium.headless: false`。\n- 正文优先 `[data-testid^=\"paragraph\"]`,旧 `article p` 会被\"相关阅读\"\"订阅推广\"段落污染。\n"
}
}
\ No newline at end of file
{
"_comment": "Regression fixtures for the nikkei-tech recipe. Article URLs must fetch with isArticle=true (og:type=article + body ≥500 chars); section URLs must fetch with isArticle=false (og:type=website). URLs verified 2026-07-29 against https://asia.nikkei.com/business/tech. Update when Nikkei archives these — pick current articles from the same path depths (2-seg /economy/<slug>, 3-seg /business/tech/semiconductors/<slug>).",
"articles": [
"https://asia.nikkei.com/business/tech/semiconductors/sk-hynix-q2-profit-surges-but-misses-market-forecast-shares-slide",
"https://asia.nikkei.com/business/tech/semiconductors/ai-chip-boom-shifts-bottleneck-to-advanced-packaging-says-at-s-ceo",
"https://asia.nikkei.com/spotlight/society/for-philippine-women-in-taiwan-s-tech-factories-motherhood-is-impossible",
"https://asia.nikkei.com/spotlight/trump-administration/us-bans-new-chinese-humanoid-robots-to-protect-ai-buildout",
"https://asia.nikkei.com/economy/natural-disasters/major-japan-quake-traps-people-inside-kumamoto-shopping-mall-factory",
"https://asia.nikkei.com/business/markets/south-korea-s-kospi-plunges-11-on-china-chip-competition-fears",
"https://asia.nikkei.com/business/companies/panasonic-to-end-tv-production-in-malaysia-cutting-jobs"
],
"sections": [
"https://asia.nikkei.com/business/tech",
"https://asia.nikkei.com/business/tech/semiconductors",
"https://asia.nikkei.com/business/markets",
"https://asia.nikkei.com/economy/natural-disasters",
"https://asia.nikkei.com/spotlight/society"
],
"minLinks": 10
}
---
source: nikkei-tech
listSelector: '[class*="StreamArticleCard"], [class*="SpotlightArticleCard"], [class*="SecondaryArticleCard"]'
linkSelector: 'a[href]'
urlPattern: 'asia\.nikkei\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts)/[^/?#]+/.+'
excludeUrlPattern: '/(?:podcasts|static|infographics|member)/'
scrollSteps: 3
scrollWaitMs: 2000
waitMs: 15000
# 详情页正文验证(见 fetcher-helper.js buildExtractExpr):放宽 urlPattern 后,
# 靠 og:type / JSON-LD @type / 正文字符数在 fetchPage 阶段剔除分类页,而不是
# 靠 URL 末段连字符数猜测。实测文章 og:type=article + 正文>3000字符;分类页
# og:type=website + 正文 0,判别力极强。
validateArticle: true
bodyMinChars: 500
bodySelectors:
- '[class*="ArticleBody"] p'
- '[class*="article-body"] p'
- 'article p'
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[class*="author"] a, [rel="author"]'
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
- logo
- icon
- avatar
- profile
- placeholder
- outbrainimg.com
titleStripSuffix: ' - Nikkei Asia$'
maxBodyChars: 15000
---
# 日经亚洲科技 抓取备忘
## 列表页链接收集(`/business/tech`)
- **listSelector 用复合卡片选择器**:`[class*="StreamArticleCard"], [class*="SpotlightArticleCard"], [class*="SecondaryArticleCard"]`。实测文章卡片分散在这三类容器里(StreamArticleCard 27 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 33 篇)。旧规则只锁定 `StreamArticleCard`,丢了 6 篇在 `SpotlightArticleCard`/`SecondaryArticleCard` 里的真实文章(含 SK Hynix、Philippine women、humanoid robots、Kumamoto quake、Panasonic、KOSPI 等)。
- **为什么不直接 `listSelector: null`(全页)**:实测全页扫描会被导航栏的 `/location/<country>` 国家下拉链接(`/location/east-asia/china` 等 ~30 条)在 `maxLinks: 30` 上限内塞满,导致真实文章一条都收不到。卡片容器范围是必须的,只要把三类 ArticleCard 容器都纳入即可覆盖全部文章且避开 nav。
- **urlPattern 扩展顶层段白名单**。旧 pattern 只覆盖 `/business/``/spotlight/`;实测 `/economy/``/politics/``/editor-s-picks/``/location/``/opinion/``/features/``/life-arts/` 下都出现文章。`/[^/?#]+/.+` 要求至少 3 段路径(顶层/子段/slug),同时覆盖 2 段深文章(如 `/economy/<slug>`)和 3 段深文章(如 `/business/tech/semiconductors/<slug>`)。
- **excludeUrlPattern** 剔除 `/podcasts/``/static/``/infographics/``/member/` 等非文章路径。
- **删除"末段 ≥5 连字符"硬规则**。旧 pattern 用 `(?:-[^/?#]*){5,}$` 猜测"长 slug=文章",但分类页 slug 也可能很长(如 `artificial-intelligence``wealth-management`),短标题文章又会被误伤。改由详情页验证区分。卡片里仍会混入 ~18 条子分类页链接(如 `/business/tech/semiconductors``/spotlight/trump-administration`,作为卡片的 tag 链接出现),由 `validateArticle` 在 fetchPage 阶段剔除。
- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页(非无限滚动),query strip 自动排除分页链接。
## 详情页正文验证(取代 URL 猜测)
`validateArticle: true``fetchPage` 在提取正文时额外采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 的候选直接 skip。判别逻辑:
```
isArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})
&& bodyChars >= 500
&& bodyParagraphs >= 3
```
实测信号(2026-07-29):
| 页面类型 | og:type | JSON-LD @type | 正文段落 | 正文字符 |
|----------|---------|---------------|----------|----------|
| 文章(SK Hynix Q2) | article | NewsArticle | 11 | 3671 |
| 文章(Philippine women) | article | NewsArticle | 30 | 5647 |
| 分类页(/business/tech/semiconductors) | website | Thing | 0 | 0 |
正文验证发生在候选链接被收集**之后**(fetchPage 阶段),所以放宽 urlPattern 混入的少量分类页会在这一步被剔除,不会进入 registry 或落盘。
## 其他
- 正文优先 `[class*="ArticleBody"]`(Nikkei 的 CSS Modules 正文容器)。
- 部分文章有付费墙,正文可能不完整——bodyMinChars=500 + bodyMinParagraphs=3 的下限会过滤掉被付费墙截断到极短的页面;preview 验证后若 charCount 过低需排除。
- 回归测试:`node scripts/recipe-test.js nikkei-tech`(固定文章 URL 必须 isArticle,固定分类页 URL 必须非 isArticle,列表页链接数 ≥10 且不含 exclude 路径)。
---
source: reuters-business
listSelector: 'div[data-testid="Title"]'
linkSelector: 'a[href]'
urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$'
excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
scrollSteps: 3
scrollWaitMs: 2000
waitMs: 15000
validateArticle: true
bodyMinChars: 500
bodySelectors:
- '[data-testid^="paragraph"]'
- '[class*="articleBodyContent"] p'
- '[class*="article-body"] p'
- 'article p'
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
- logo
- icon
- avatar
- profile
titleStripSuffix: ' \| Reuters$'
maxBodyChars: 15000
---
# Reuters Business 抓取备忘
## 列表页链接收集(`/business/`)
- **listSelector: `div[data-testid="Title"]`**。实测全页 20 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定(hash 不变)。
- **urlPattern: `reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/business/``/business/finance/`)无日期后缀。放宽为全站匹配(不限 `/business/`),因为板块页会推荐其他分区的文章(如 `/world/asia-pacific/...`)。
- **excludeUrlPattern** 排除 `/podcasts/``/newsletter/`(它们也带日期但非正文报道)。
- `scrollSteps: 3` 触发懒加载拿全当天列表;首屏仅 ~10 篇。
## 详情页正文验证
`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:
- 文章:`og:type=article``@type=NewsArticle`、正文 4000+ 字符
- 分类页 `/business/`:`og:type=website``@type=CollectionPage`、正文 0
Reuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。
## 其他
- 反爬:headless 会被拦,需 `config.chromium.headless: false`
- 正文优先 `[data-testid^="paragraph"]`,旧 `article p` 会被"相关阅读""订阅推广"段落污染。
---
source: reuters-markets
listSelector: '[data-testid="MediaStoryCard"], [data-testid="BasicCard"], [data-testid="HeroCard"], [data-testid="HubCard"], [data-testid="AuthorCard"], [data-testid="OurColumnists"]'
linkSelector: 'a[href]'
urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$'
excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
scrollSteps: 3
scrollWaitMs: 2000
waitMs: 15000
validateArticle: true
bodyMinChars: 500
bodySelectors:
- '[data-testid^="paragraph"]'
- '[class*="articleBodyContent"] p'
- '[class*="article-body"] p'
- 'article p'
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
- logo
- icon
- avatar
- profile
titleStripSuffix: ' \| Reuters$'
maxBodyChars: 15000
---
# Reuters Markets 抓取备忘
## 列表页链接收集(`/markets/`)
- **listSelector 用复合 `data-testid` 选择器**:`MediaStoryCard``BasicCard``HeroCard``HubCard``AuthorCard``OurColumnists` 六类卡片容器。实测全页 20 篇文章链接分散在这六类容器里(`MediaStoryCard` 18 + `BasicCard` 12 + `HeroCard`/`HubCard`/`AuthorCard`/`OurColumnists` 各 2-3,去重后 20)。
- **为什么不用旧 `div.media-story-card-module__placement-container__1ZSB1`**:旧选择器依赖带 hash 的 CSS Modules 类名,hash 随构建变化 → 过期后只匹配 9 篇,漏掉 11 篇。`data-testid` 无 hash,稳定得多。Markets 页面布局与 Business/World 不同(`div[data-testid="Title"]` 在此页 0 篇),必须用 Markets 专属的复合容器。
- **urlPattern: `reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/markets/``/markets/currencies/`)无日期后缀。放宽为全站匹配,因为 Markets 板块会推荐其他分区文章(如 `/world/asia-pacific/...``/business/energy/...``/commentary/...`)。
- **excludeUrlPattern** 排除 `/podcasts/``/newsletter/`(它们也带日期但非正文报道)。
- `scrollSteps: 3` 触发懒加载拿全当天列表。
## 详情页正文验证
`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:
- 文章:`og:type=article``@type=NewsArticle`、正文 4000+ 字符
- 分类页 `/markets/`:`og:type=website`、正文 0
Reuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。
## 其他
- 反爬:headless 会被拦,需 `config.chromium.headless: false`
- 正文优先 `[data-testid^="paragraph"]`,旧 `article p` 会被"相关阅读""订阅推广"段落污染。
---
source: reuters-world
listSelector: 'div[data-testid="Title"]'
linkSelector: 'a[href]'
urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$'
excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
scrollSteps: 3
scrollWaitMs: 2000
waitMs: 15000
validateArticle: true
bodyMinChars: 500
bodySelectors:
- '[data-testid^="paragraph"]'
- '[class*="articleBodyContent"] p'
- '[class*="article-body"] p'
- 'article p'
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
- logo
- icon
- avatar
- profile
titleStripSuffix: ' \| Reuters$'
maxBodyChars: 15000
---
# 路透社 World 抓取备忘
## 列表页链接收集(`/world/`)
- **listSelector: `div[data-testid="Title"]`**。实测全页 19 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定。
- **urlPattern: `reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/world/``/world/asia-pacific/`)无日期后缀。放宽为全站匹配(不限 `/world/`),因为板块页会推荐其他分区的文章(如 `/business/environment/...``/legal/government/...`)。
- **excludeUrlPattern** 排除 `/podcasts/``/newsletter/`(它们也带日期但非正文报道)。
- `scrollSteps: 3` 触发懒加载拿全当天列表。
## 详情页正文验证
`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:
- 文章:`og:type=article``@type=NewsArticle`、正文 6000+ 字符
- 分类页 `/world/`:`og:type=website`、正文 0
Reuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。
## 其他
- 反爬:headless 会被拦,需 `config.chromium.headless: false`
- 正文优先 `[data-testid^="paragraph"]`,旧 `article p` 会被"相关阅读""订阅推广"段落污染。
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment