Commit ba940690 authored by 谢宇轩's avatar 谢宇轩

feat: update Reuters section helpers + nikkei-tech recipe

Reuters:
- Remove homepage source (reuters) from config; keep business/markets/world
- reuters-markets: fix stale hashed CSS class (only 9/20 articles) →
  composite data-testid listSelector (MediaStoryCard/BasicCard/HeroCard/
  HubCard/AuthorCard/OurColumnists), now captures all 20
- All three: add validateArticle (og:type/JSON-LD/body), broaden
  urlPattern to site-wide date-suffix match, exclude podcasts/newsletter
- Verified: Business 20, Markets 20 (was 9), World 19 article links

nikkei-tech: bring recipe + fixtures into tree (composite listSelector,
validateArticle, 7-article/5-section regression fixtures).
parent 8ab2efa0
{
"_comment": "Regression fixtures for the nikkei-tech recipe. Article URLs must fetch with isArticle=true (og:type=article + body ≥500 chars); section URLs must fetch with isArticle=false (og:type=website). URLs verified 2026-07-29 against https://asia.nikkei.com/business/tech. Update when Nikkei archives these — pick current articles from the same path depths (2-seg /economy/<slug>, 3-seg /business/tech/semiconductors/<slug>).",
"articles": [
"https://asia.nikkei.com/business/tech/semiconductors/sk-hynix-q2-profit-surges-but-misses-market-forecast-shares-slide",
"https://asia.nikkei.com/business/tech/semiconductors/ai-chip-boom-shifts-bottleneck-to-advanced-packaging-says-at-s-ceo",
"https://asia.nikkei.com/spotlight/society/for-philippine-women-in-taiwan-s-tech-factories-motherhood-is-impossible",
"https://asia.nikkei.com/spotlight/trump-administration/us-bans-new-chinese-humanoid-robots-to-protect-ai-buildout",
"https://asia.nikkei.com/economy/natural-disasters/major-japan-quake-traps-people-inside-kumamoto-shopping-mall-factory",
"https://asia.nikkei.com/business/markets/south-korea-s-kospi-plunges-11-on-china-chip-competition-fears",
"https://asia.nikkei.com/business/companies/panasonic-to-end-tv-production-in-malaysia-cutting-jobs"
],
"sections": [
"https://asia.nikkei.com/business/tech",
"https://asia.nikkei.com/business/tech/semiconductors",
"https://asia.nikkei.com/business/markets",
"https://asia.nikkei.com/economy/natural-disasters",
"https://asia.nikkei.com/spotlight/society"
],
"minLinks": 10
}
---
source: nikkei-tech
listSelector: '[class*="StreamArticleCard"], [class*="SpotlightArticleCard"], [class*="SecondaryArticleCard"]'
linkSelector: 'a[href]'
urlPattern: 'asia\.nikkei\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts)/[^/?#]+/.+'
excludeUrlPattern: '/(?:podcasts|static|infographics|member)/'
scrollSteps: 3
scrollWaitMs: 2000
waitMs: 15000
# 详情页正文验证(见 fetcher-helper.js buildExtractExpr):放宽 urlPattern 后,
# 靠 og:type / JSON-LD @type / 正文字符数在 fetchPage 阶段剔除分类页,而不是
# 靠 URL 末段连字符数猜测。实测文章 og:type=article + 正文>3000字符;分类页
# og:type=website + 正文 0,判别力极强。
validateArticle: true
bodyMinChars: 500
bodySelectors:
- '[class*="ArticleBody"] p'
- '[class*="article-body"] p'
- 'article p'
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[class*="author"] a, [rel="author"]'
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
- logo
- icon
- avatar
- profile
- placeholder
- outbrainimg.com
titleStripSuffix: ' - Nikkei Asia$'
maxBodyChars: 15000
---
# 日经亚洲科技 抓取备忘
## 列表页链接收集(`/business/tech`)
- **listSelector 用复合卡片选择器**:`[class*="StreamArticleCard"], [class*="SpotlightArticleCard"], [class*="SecondaryArticleCard"]`。实测文章卡片分散在这三类容器里(StreamArticleCard 27 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 33 篇)。旧规则只锁定 `StreamArticleCard`,丢了 6 篇在 `SpotlightArticleCard`/`SecondaryArticleCard` 里的真实文章(含 SK Hynix、Philippine women、humanoid robots、Kumamoto quake、Panasonic、KOSPI 等)。
- **为什么不直接 `listSelector: null`(全页)**:实测全页扫描会被导航栏的 `/location/<country>` 国家下拉链接(`/location/east-asia/china` 等 ~30 条)在 `maxLinks: 30` 上限内塞满,导致真实文章一条都收不到。卡片容器范围是必须的,只要把三类 ArticleCard 容器都纳入即可覆盖全部文章且避开 nav。
- **urlPattern 扩展顶层段白名单**。旧 pattern 只覆盖 `/business/``/spotlight/`;实测 `/economy/``/politics/``/editor-s-picks/``/location/``/opinion/``/features/``/life-arts/` 下都出现文章。`/[^/?#]+/.+` 要求至少 3 段路径(顶层/子段/slug),同时覆盖 2 段深文章(如 `/economy/<slug>`)和 3 段深文章(如 `/business/tech/semiconductors/<slug>`)。
- **excludeUrlPattern** 剔除 `/podcasts/``/static/``/infographics/``/member/` 等非文章路径。
- **删除"末段 ≥5 连字符"硬规则**。旧 pattern 用 `(?:-[^/?#]*){5,}$` 猜测"长 slug=文章",但分类页 slug 也可能很长(如 `artificial-intelligence``wealth-management`),短标题文章又会被误伤。改由详情页验证区分。卡片里仍会混入 ~18 条子分类页链接(如 `/business/tech/semiconductors``/spotlight/trump-administration`,作为卡片的 tag 链接出现),由 `validateArticle` 在 fetchPage 阶段剔除。
- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页(非无限滚动),query strip 自动排除分页链接。
## 详情页正文验证(取代 URL 猜测)
`validateArticle: true``fetchPage` 在提取正文时额外采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 的候选直接 skip。判别逻辑:
```
isArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})
&& bodyChars >= 500
&& bodyParagraphs >= 3
```
实测信号(2026-07-29):
| 页面类型 | og:type | JSON-LD @type | 正文段落 | 正文字符 |
|----------|---------|---------------|----------|----------|
| 文章(SK Hynix Q2) | article | NewsArticle | 11 | 3671 |
| 文章(Philippine women) | article | NewsArticle | 30 | 5647 |
| 分类页(/business/tech/semiconductors) | website | Thing | 0 | 0 |
正文验证发生在候选链接被收集**之后**(fetchPage 阶段),所以放宽 urlPattern 混入的少量分类页会在这一步被剔除,不会进入 registry 或落盘。
## 其他
- 正文优先 `[class*="ArticleBody"]`(Nikkei 的 CSS Modules 正文容器)。
- 部分文章有付费墙,正文可能不完整——bodyMinChars=500 + bodyMinParagraphs=3 的下限会过滤掉被付费墙截断到极短的页面;preview 验证后若 charCount 过低需排除。
- 回归测试:`node scripts/recipe-test.js nikkei-tech`(固定文章 URL 必须 isArticle,固定分类页 URL 必须非 isArticle,列表页链接数 ≥10 且不含 exclude 路径)。
---
source: reuters-business
listSelector: null
listSelector: 'div[data-testid="Title"]'
linkSelector: 'a[href]'
urlPattern: 'reuters\.com/business/(?:[^/]+/)*[\w-]+-\d{4}-\d{2}-\d{2}/'
excludeUrlPattern: null
urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$'
excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
scrollSteps: 3
scrollWaitMs: 1500
waitMs: 12000
scrollWaitMs: 2000
waitMs: 15000
validateArticle: true
bodyMinChars: 500
bodySelectors:
- '[data-testid^="paragraph"]'
- '[class*="articleBodyContent"] p'
- '[class*="article-body"] p'
- 'article p'
bodyMinParagraphs: 3
......@@ -21,11 +24,27 @@ excludeImage:
- icon
- avatar
- profile
titleStripSuffix: ''
titleStripSuffix: ' \| Reuters$'
maxBodyChars: 15000
---
# Reuters Business 抓取备忘
- Reuters Business 首页的文章链接位于页面主体 `div[data-testid="Title"]`,但使用全页扫描配合严格 URL 过滤可避免依赖动态 CSS 类名。
- 文章 URL 形如 `/business/<可选分区>/<slug>-YYYY-MM-DD/`;日期后缀用于排除导航、专题页和分区页。
- `scrollSteps: 3` 触发首页懒加载,正文优先使用段落测试选择器。
## 列表页链接收集(`/business/`)
- **listSelector: `div[data-testid="Title"]`**。实测全页 20 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定(hash 不变)。
- **urlPattern: `reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/business/``/business/finance/`)无日期后缀。放宽为全站匹配(不限 `/business/`),因为板块页会推荐其他分区的文章(如 `/world/asia-pacific/...`)。
- **excludeUrlPattern** 排除 `/podcasts/``/newsletter/`(它们也带日期但非正文报道)。
- `scrollSteps: 3` 触发懒加载拿全当天列表;首屏仅 ~10 篇。
## 详情页正文验证
`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:
- 文章:`og:type=article``@type=NewsArticle`、正文 4000+ 字符
- 分类页 `/business/`:`og:type=website``@type=CollectionPage`、正文 0
Reuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。
## 其他
- 反爬:headless 会被拦,需 `config.chromium.headless: false`
- 正文优先 `[data-testid^="paragraph"]`,旧 `article p` 会被"相关阅读""订阅推广"段落污染。
---
source: reuters-markets
listSelector: "div.media-story-card-module__placement-container__1ZSB1"
listSelector: '[data-testid="MediaStoryCard"], [data-testid="BasicCard"], [data-testid="HeroCard"], [data-testid="HubCard"], [data-testid="AuthorCard"], [data-testid="OurColumnists"]'
linkSelector: 'a[href]'
urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}'
excludeUrlPattern: null
urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$'
excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
scrollSteps: 3
scrollWaitMs: 2000
waitMs: 15000
validateArticle: true
bodyMinChars: 500
bodySelectors:
- '[data-testid^="paragraph"]'
- '[class*="articleBodyContent"] p'
......@@ -27,7 +29,23 @@ maxBodyChars: 15000
---
# Reuters Markets 抓取备忘
- 主文章流容器 `div.media-story-card-module__placement-container__1ZSB1` 实测 6 个候选链接(inspect 自动生成)。
- urlPattern `reuters\.com/.+-\d{4}-\d{2}-\d{2}` 由样本 URL 推断,如 preview 漏抓/误抓请手编收紧。
- 首屏可能不全,scrollSteps=0 触发懒加载;反爬严重时调高 waitMs。
- 验证: node scripts/harvest.js reuters-markets 3 --preview(链接为真实文章且 charCount>500 即通过)
\ No newline at end of file
## 列表页链接收集(`/markets/`)
- **listSelector 用复合 `data-testid` 选择器**:`MediaStoryCard``BasicCard``HeroCard``HubCard``AuthorCard``OurColumnists` 六类卡片容器。实测全页 20 篇文章链接分散在这六类容器里(`MediaStoryCard` 18 + `BasicCard` 12 + `HeroCard`/`HubCard`/`AuthorCard`/`OurColumnists` 各 2-3,去重后 20)。
- **为什么不用旧 `div.media-story-card-module__placement-container__1ZSB1`**:旧选择器依赖带 hash 的 CSS Modules 类名,hash 随构建变化 → 过期后只匹配 9 篇,漏掉 11 篇。`data-testid` 无 hash,稳定得多。Markets 页面布局与 Business/World 不同(`div[data-testid="Title"]` 在此页 0 篇),必须用 Markets 专属的复合容器。
- **urlPattern: `reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/markets/``/markets/currencies/`)无日期后缀。放宽为全站匹配,因为 Markets 板块会推荐其他分区文章(如 `/world/asia-pacific/...``/business/energy/...``/commentary/...`)。
- **excludeUrlPattern** 排除 `/podcasts/``/newsletter/`(它们也带日期但非正文报道)。
- `scrollSteps: 3` 触发懒加载拿全当天列表。
## 详情页正文验证
`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:
- 文章:`og:type=article``@type=NewsArticle`、正文 4000+ 字符
- 分类页 `/markets/`:`og:type=website`、正文 0
Reuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。
## 其他
- 反爬:headless 会被拦,需 `config.chromium.headless: false`
- 正文优先 `[data-testid^="paragraph"]`,旧 `article p` 会被"相关阅读""订阅推广"段落污染。
......@@ -2,11 +2,13 @@
source: reuters-world
listSelector: 'div[data-testid="Title"]'
linkSelector: 'a[href]'
urlPattern: 'www\.reuters\.com/(world|business|markets|technology|sustainability)/.+-\d{4}-\d{2}-\d{2}/'
excludeUrlPattern: null
urlPattern: 'reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$'
excludeUrlPattern: 'reuters\.com/(podcasts|newsletter)/'
scrollSteps: 3
scrollWaitMs: 2000
waitMs: 15000
validateArticle: true
bodyMinChars: 500
bodySelectors:
- '[data-testid^="paragraph"]'
- '[class*="articleBodyContent"] p'
......@@ -22,12 +24,27 @@ excludeImage:
- icon
- avatar
- profile
titleStripSuffix: ''
titleStripSuffix: ' \| Reuters$'
maxBodyChars: 15000
---
# 路透社 World 抓取备忘
- 主文章流容器 `li.link-group-module__item__Za3t6` 实测 30 个候选链接(inspect 自动生成)。
- urlPattern `www\.reuters\.com/sitemap/.+` 由样本 URL 推断,如 preview 漏抓/误抓请手编收紧。
- 首屏可能不全,scrollSteps=3 触发懒加载;反爬严重时调高 waitMs。
- 验证: node scripts/harvest.js reuters-world 3 --preview(链接为真实文章且 charCount>500 即通过)
\ No newline at end of file
## 列表页链接收集(`/world/`)
- **listSelector: `div[data-testid="Title"]`**。实测全页 19 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定。
- **urlPattern: `reuters\.com/.+-\d{4}-\d{2}-\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/world/``/world/asia-pacific/`)无日期后缀。放宽为全站匹配(不限 `/world/`),因为板块页会推荐其他分区的文章(如 `/business/environment/...``/legal/government/...`)。
- **excludeUrlPattern** 排除 `/podcasts/``/newsletter/`(它们也带日期但非正文报道)。
- `scrollSteps: 3` 触发懒加载拿全当天列表。
## 详情页正文验证
`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:
- 文章:`og:type=article``@type=NewsArticle`、正文 6000+ 字符
- 分类页 `/world/`:`og:type=website`、正文 0
Reuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。
## 其他
- 反爬:headless 会被拦,需 `config.chromium.headless: false`
- 正文优先 `[data-testid^="paragraph"]`,旧 `article p` 会被"相关阅读""订阅推广"段落污染。
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment