Commit be725014 authored by 谢宇轩's avatar 谢宇轩

fix: edit img select and nhsource bundles

parent 518b835c
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-07-30T08:41:39.136Z",
"exportedAt": "2026-08-04T04:39:33.227Z",
"source": {
"id": "nikkei-markets",
"name": "日经亚洲市场",
......@@ -10,6 +10,6 @@
"vaultFolder": "日经亚洲/市场"
},
"files": {
"helpers/nikkei-markets.md": "---\nsource: nikkei-markets\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲市场 抓取备忘\n\n## 列表页链接收集(`/business/markets`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/business/markets` 页面文章卡片分散在这三类容器里(StreamArticleCard 24 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 30 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。`/[^/?#]+/.+` 要求至少 3 段路径\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/currencies`、`/business/markets/property` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
"helpers/nikkei-markets.md": "---\nsource: nikkei-markets\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲市场 抓取备忘\n\n## 列表页链接收集(`/business/markets`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/business/markets` 页面文章卡片分散在这三类容器里(StreamArticleCard 24 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 30 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets/commodities`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/currencies`、`/business/markets/property` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
}
}
\ No newline at end of file
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-07-29T07:04:52.699Z",
"exportedAt": "2026-08-04T04:39:33.187Z",
"source": {
"id": "nikkei-tech",
"name": "日经亚洲科技",
......@@ -10,7 +10,7 @@
"vaultFolder": "日经亚洲/科技"
},
"files": {
"helpers/nikkei-tech.md": "---\nsource: nikkei-tech\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\n# 详情页正文验证(见 fetcher-helper.js buildExtractExpr):放宽 urlPattern 后,\n# 靠 og:type / JSON-LD @type / 正文字符数在 fetchPage 阶段剔除分类页,而不是\n# 靠 URL 末段连字符数猜测。实测文章 og:type=article + 正文>3000字符;分类页\n# og:type=website + 正文 0,判别力极强。\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲科技 抓取备忘\n\n## 列表页链接收集(`/business/tech`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测文章卡片分散在这三类容器里(StreamArticleCard 27 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 33 篇)。旧规则只锁定 `StreamArticleCard`,丢了 6 篇在 `SpotlightArticleCard`/`SecondaryArticleCard` 里的真实文章(含 SK Hynix、Philippine women、humanoid robots、Kumamoto quake、Panasonic、KOSPI 等)。\n- **为什么不直接 `listSelector: null`(全页)**:实测全页扫描会被导航栏的 `/location/<country>` 国家下拉链接(`/location/east-asia/china` 等 ~30 条)在 `maxLinks: 30` 上限内塞满,导致真实文章一条都收不到。卡片容器范围是必须的,只要把三类 ArticleCard 容器都纳入即可覆盖全部文章且避开 nav。\n- **urlPattern 扩展顶层段白名单**。旧 pattern 只覆盖 `/business/`、`/spotlight/`;实测 `/economy/`、`/politics/`、`/editor-s-picks/`、`/location/`、`/opinion/`、`/features/`、`/life-arts/` 下都出现文章。`/[^/?#]+/.+` 要求至少 3 段路径(顶层/子段/slug),同时覆盖 2 段深文章(如 `/economy/<slug>`)和 3 段深文章(如 `/business/tech/semiconductors/<slug>`)\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **删除\"末段 ≥5 连字符\"硬规则**。旧 pattern 用 `(?:-[^/?#]*){5,}$` 猜测\"长 slug=文章\",但分类页 slug 也可能很长(如 `artificial-intelligence`、`wealth-management`),短标题文章又会被误伤。改由详情页验证区分。卡片里仍会混入 ~18 条子分类页链接(如 `/business/tech/semiconductors`、`/spotlight/trump-administration`,作为卡片的 tag 链接出现),由 `validateArticle` 在 fetchPage 阶段剔除。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页(非无限滚动),query strip 自动排除分页链接。\n\n## 详情页正文验证(取代 URL 猜测)\n\n`validateArticle: true` 让 `fetchPage` 在提取正文时额外采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 的候选直接 skip。判别逻辑:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n实测信号(2026-07-29):\n| 页面类型 | og:type | JSON-LD @type | 正文段落 | 正文字符 |\n|----------|---------|---------------|----------|----------|\n| 文章(SK Hynix Q2) | article | NewsArticle | 11 | 3671 |\n| 文章(Philippine women) | article | NewsArticle | 30 | 5647 |\n| 分类页(/business/tech/semiconductors) | website | Thing | 0 | 0 |\n\n正文验证发生在候选链接被收集**之后**(fetchPage 阶段),所以放宽 urlPattern 混入的少量分类页会在这一步被剔除,不会进入 registry 或落盘。\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei 的 CSS Modules 正文容器)。\n- 部分文章有付费墙,正文可能不完整——bodyMinChars=500 + bodyMinParagraphs=3 的下限会过滤掉被付费墙截断到极短的页面;preview 验证后若 charCount 过低需排除。\n- 回归测试:`node scripts/recipe-test.js nikkei-tech`(固定文章 URL 必须 isArticle,固定分类页 URL 必须非 isArticle,列表页链接数 ≥10 且不含 exclude 路径)。\n",
"helpers/nikkei-tech.md": "---\nsource: nikkei-tech\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\n# 详情页正文验证(见 fetcher-helper.js buildExtractExpr):放宽 urlPattern 后,\n# 靠 og:type / JSON-LD @type / 正文字符数在 fetchPage 阶段剔除分类页,而不是\n# 靠 URL 末段连字符数猜测。实测文章 og:type=article + 正文>3000字符;分类页\n# og:type=website + 正文 0,判别力极强。\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲科技 抓取备忘\n\n## 列表页链接收集(`/business/tech`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测文章卡片分散在这三类容器里(StreamArticleCard 27 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 33 篇)。旧规则只锁定 `StreamArticleCard`,丢了 6 篇在 `SpotlightArticleCard`/`SecondaryArticleCard` 里的真实文章(含 SK Hynix、Philippine women、humanoid robots、Kumamoto quake、Panasonic、KOSPI 等)。\n- **为什么不直接 `listSelector: null`(全页)**:实测全页扫描会被导航栏的 `/location/<country>` 国家下拉链接(`/location/east-asia/china` 等 ~30 条)在 `maxLinks: 30` 上限内塞满,导致真实文章一条都收不到。卡片容器范围是必须的,只要把三类 ArticleCard 容器都纳入即可覆盖全部文章且避开 nav。\n- **urlPattern 扩展顶层段白名单**。旧 pattern 只覆盖 `/business/`、`/spotlight/`;实测 `/economy/`、`/politics/`、`/editor-s-picks/`、`/location/`、`/opinion/`、`/features/`、`/life-arts/`、`/techasia/` 下都出现文章。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/tech/semiconductors/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **删除\"末段 ≥5 连字符\"硬规则**。旧 pattern 用 `(?:-[^/?#]*){5,}$` 猜测\"长 slug=文章\",但分类页 slug 也可能很长(如 `artificial-intelligence`、`wealth-management`),短标题文章又会被误伤。改由详情页验证区分。卡片里仍会混入 ~18 条子分类页链接(如 `/business/tech/semiconductors`、`/spotlight/trump-administration`,作为卡片的 tag 链接出现),由 `validateArticle` 在 fetchPage 阶段剔除。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页(非无限滚动),query strip 自动排除分页链接。\n\n## 详情页正文验证(取代 URL 猜测)\n\n`validateArticle: true` 让 `fetchPage` 在提取正文时额外采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 的候选直接 skip。判别逻辑:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n实测信号(2026-07-29):\n| 页面类型 | og:type | JSON-LD @type | 正文段落 | 正文字符 |\n|----------|---------|---------------|----------|----------|\n| 文章(SK Hynix Q2) | article | NewsArticle | 11 | 3671 |\n| 文章(Philippine women) | article | NewsArticle | 30 | 5647 |\n| 分类页(/business/tech/semiconductors) | website | Thing | 0 | 0 |\n\n正文验证发生在候选链接被收集**之后**(fetchPage 阶段),所以放宽 urlPattern 混入的少量分类页会在这一步被剔除,不会进入 registry 或落盘。\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei 的 CSS Modules 正文容器)。\n- 部分文章有付费墙,正文可能不完整——bodyMinChars=500 + bodyMinParagraphs=3 的下限会过滤掉被付费墙截断到极短的页面;preview 验证后若 charCount 过低需排除。\n- 回归测试:`node scripts/recipe-test.js nikkei-tech`(固定文章 URL 必须 isArticle,固定分类页 URL 必须非 isArticle,列表页链接数 ≥10 且不含 exclude 路径)。\n",
"helpers/nikkei-tech.fixtures.json": "{\n \"_comment\": \"Regression fixtures for the nikkei-tech recipe. Article URLs must fetch with isArticle=true (og:type=article + body ≥500 chars); section URLs must fetch with isArticle=false (og:type=website). URLs verified 2026-07-29 against https://asia.nikkei.com/business/tech. Update when Nikkei archives these — pick current articles from the same path depths (2-seg /economy/<slug>, 3-seg /business/tech/semiconductors/<slug>).\",\n \"articles\": [\n \"https://asia.nikkei.com/business/tech/semiconductors/sk-hynix-q2-profit-surges-but-misses-market-forecast-shares-slide\",\n \"https://asia.nikkei.com/business/tech/semiconductors/ai-chip-boom-shifts-bottleneck-to-advanced-packaging-says-at-s-ceo\",\n \"https://asia.nikkei.com/spotlight/society/for-philippine-women-in-taiwan-s-tech-factories-motherhood-is-impossible\",\n \"https://asia.nikkei.com/spotlight/trump-administration/us-bans-new-chinese-humanoid-robots-to-protect-ai-buildout\",\n \"https://asia.nikkei.com/economy/natural-disasters/major-japan-quake-traps-people-inside-kumamoto-shopping-mall-factory\",\n \"https://asia.nikkei.com/business/markets/south-korea-s-kospi-plunges-11-on-china-chip-competition-fears\",\n \"https://asia.nikkei.com/business/companies/panasonic-to-end-tv-production-in-malaysia-cutting-jobs\"\n ],\n \"sections\": [\n \"https://asia.nikkei.com/business/tech\",\n \"https://asia.nikkei.com/business/tech/semiconductors\",\n \"https://asia.nikkei.com/business/markets\",\n \"https://asia.nikkei.com/economy/natural-disasters\",\n \"https://asia.nikkei.com/spotlight/society\"\n ],\n \"minLinks\": 10\n}\n"
}
}
\ No newline at end of file
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-07-30T08:41:39.076Z",
"exportedAt": "2026-08-04T04:39:33.259Z",
"source": {
"id": "nikkei-world",
"name": "日经亚洲世界",
......@@ -10,6 +10,6 @@
"vaultFolder": "日经亚洲/世界"
},
"files": {
"helpers/nikkei-world.md": "---\nsource: nikkei-world\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲世界 抓取备忘\n\n## 列表页链接收集(`/location`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/location` 页面文章卡片分散在这三类容器里(StreamArticleCard 26 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 32 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。`/[^/?#]+/.+` 要求至少 3 段路径\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/commodities` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
"helpers/nikkei-world.md": "---\nsource: nikkei-world\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲世界 抓取备忘\n\n## 列表页链接收集(`/location`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/location` 页面文章卡片分散在这三类容器里(StreamArticleCard 26 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 32 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/commodities` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
}
}
\ No newline at end of file
......@@ -64,6 +64,11 @@ const DEFAULTS = {
imageSelector: 'img',
imageMinWidth: 200,
excludeImage: ['logo', 'icon', 'avatar', 'profile'],
// CSS selector: any <img> whose closest ancestor (or itself) matches is
// dropped during extraction. Use this to exclude "recommended articles" /
// "related articles" thumbnails that share the same CDN host as the hero
// image (so src-substring excludeImage can't tell them apart). null = off.
excludeImageSelector: null,
noisePrefixes: [
'Reporting by', 'Editing by', 'Access unmatched', 'Browse an unrivalled',
'Screen for heightened', 'Sign up here', 'Reuters, the news',
......@@ -129,6 +134,19 @@ function buildExtractExpr(spec) {
const authorSelector = JSON.stringify(spec.authorSelector);
const imageMinWidth = spec.imageMinWidth;
const excludeImage = JSON.stringify(spec.excludeImage);
// imageSelector scopes which <img> nodes are candidates for download. The
// default 'img' (whole document) is kept for backward compat, but recipes
// should narrow it to the article-body container so bottom-of-page
// "recommended articles" thumbnails aren't pulled into assets/. Supported
// as a single CSS selector or a comma-separated list (querySelectorAll
// handles both); null/empty falls back to 'img'.
const imageSelector = JSON.stringify(spec.imageSelector || 'img');
// excludeImageSelector: drop an <img> if it (or any ancestor) matches this
// CSS selector. Comma-separated lists are valid. null → no container-based
// exclusion (existing behaviour). Used to filter "recommended articles"
// thumbnails that share a CDN host with the hero image.
const excludeImageSelector = spec.excludeImageSelector
? JSON.stringify(spec.excludeImageSelector) : 'null';
const noisePrefixes = JSON.stringify(spec.noisePrefixes);
const titleStripRe = JSON.stringify(spec.titleStripSuffix);
const maxBodyChars = spec.maxBodyChars;
......@@ -169,7 +187,15 @@ function buildExtractExpr(spec) {
.map(function(a){ return a.innerText.trim(); }).filter(Boolean)
.filter(function(v,i,a){ return a.indexOf(v) === i; }).join(', ');
if (!authors && ldAuthorNames.length) authors = ldAuthorNames.join(', ');
var imgs = Array.from(document.querySelectorAll('img')).map(function(img){
var exclImgSel = ${excludeImageSelector};
var imgs = Array.from(document.querySelectorAll(${imageSelector}))
.filter(function(img){
// Container-based exclusion: drop if the img (or any ancestor) sits
// inside a "recommended/related articles" block. Skipped when no
// excludeImageSelector is configured.
return !exclImgSel || !img.closest(exclImgSel);
})
.map(function(img){
return { src: img.src, alt: img.alt || '', width: img.naturalWidth || img.width || 0 };
}).filter(function(i){
return i.src && i.src.indexOf('http') === 0 && i.width > ${imageMinWidth}
......
......@@ -222,6 +222,8 @@ bodySelectors:
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
# Scope to the article-body container (e.g. '[class*="ArticleBody"] img, article img')
# so bottom-of-page "recommended articles" thumbnails aren't downloaded into assets/.
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
......@@ -229,6 +231,9 @@ excludeImage:
- icon
- avatar
- profile
# excludeImageSelector drops imgs whose ancestor matches — use for
# "recommended/related articles" thumbnails sharing the hero image's CDN host.
# excludeImageSelector: '[class*="RelatedArticle"], [class*="RecommendedArticle"]'
titleStripSuffix: ''
maxBodyChars: 15000
---
......
......@@ -201,6 +201,8 @@ bodySelectors:
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
# Scope to the article-body container (e.g. '[class*="ArticleBody"] img, article img')
# so bottom-of-page "recommended articles" thumbnails aren't downloaded into assets/.
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
......@@ -208,6 +210,9 @@ excludeImage:
- icon
- avatar
- profile
# excludeImageSelector drops imgs whose ancestor matches — use for
# "recommended/related articles" thumbnails sharing the hero image's CDN host.
# excludeImageSelector: '[class*="RelatedArticle"], [class*="RecommendedArticle"]'
titleStripSuffix: ''
maxBodyChars: 15000
---
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment