Commit be725014 authored by 谢宇轩's avatar 谢宇轩

fix: edit img select and nhsource bundles

parent 518b835c
{ {
"format": "news-harvester-source", "format": "news-harvester-source",
"version": 1, "version": 1,
"exportedAt": "2026-07-30T08:41:39.136Z", "exportedAt": "2026-08-04T04:39:33.227Z",
"source": { "source": {
"id": "nikkei-markets", "id": "nikkei-markets",
"name": "日经亚洲市场", "name": "日经亚洲市场",
...@@ -10,6 +10,6 @@ ...@@ -10,6 +10,6 @@
"vaultFolder": "日经亚洲/市场" "vaultFolder": "日经亚洲/市场"
}, },
"files": { "files": {
"helpers/nikkei-markets.md": "---\nsource: nikkei-markets\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲市场 抓取备忘\n\n## 列表页链接收集(`/business/markets`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/business/markets` 页面文章卡片分散在这三类容器里(StreamArticleCard 24 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 30 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。`/[^/?#]+/.+` 要求至少 3 段路径\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/currencies`、`/business/markets/property` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n" "helpers/nikkei-markets.md": "---\nsource: nikkei-markets\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲市场 抓取备忘\n\n## 列表页链接收集(`/business/markets`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/business/markets` 页面文章卡片分散在这三类容器里(StreamArticleCard 24 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 30 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets/commodities`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/currencies`、`/business/markets/property` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
} }
} }
\ No newline at end of file
{ {
"format": "news-harvester-source", "format": "news-harvester-source",
"version": 1, "version": 1,
"exportedAt": "2026-07-29T07:04:52.699Z", "exportedAt": "2026-08-04T04:39:33.187Z",
"source": { "source": {
"id": "nikkei-tech", "id": "nikkei-tech",
"name": "日经亚洲科技", "name": "日经亚洲科技",
...@@ -10,7 +10,7 @@ ...@@ -10,7 +10,7 @@
"vaultFolder": "日经亚洲/科技" "vaultFolder": "日经亚洲/科技"
}, },
"files": { "files": {
"helpers/nikkei-tech.md": "---\nsource: nikkei-tech\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\n# 详情页正文验证(见 fetcher-helper.js buildExtractExpr):放宽 urlPattern 后,\n# 靠 og:type / JSON-LD @type / 正文字符数在 fetchPage 阶段剔除分类页,而不是\n# 靠 URL 末段连字符数猜测。实测文章 og:type=article + 正文>3000字符;分类页\n# og:type=website + 正文 0,判别力极强。\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲科技 抓取备忘\n\n## 列表页链接收集(`/business/tech`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测文章卡片分散在这三类容器里(StreamArticleCard 27 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 33 篇)。旧规则只锁定 `StreamArticleCard`,丢了 6 篇在 `SpotlightArticleCard`/`SecondaryArticleCard` 里的真实文章(含 SK Hynix、Philippine women、humanoid robots、Kumamoto quake、Panasonic、KOSPI 等)。\n- **为什么不直接 `listSelector: null`(全页)**:实测全页扫描会被导航栏的 `/location/<country>` 国家下拉链接(`/location/east-asia/china` 等 ~30 条)在 `maxLinks: 30` 上限内塞满,导致真实文章一条都收不到。卡片容器范围是必须的,只要把三类 ArticleCard 容器都纳入即可覆盖全部文章且避开 nav。\n- **urlPattern 扩展顶层段白名单**。旧 pattern 只覆盖 `/business/`、`/spotlight/`;实测 `/economy/`、`/politics/`、`/editor-s-picks/`、`/location/`、`/opinion/`、`/features/`、`/life-arts/` 下都出现文章。`/[^/?#]+/.+` 要求至少 3 段路径(顶层/子段/slug),同时覆盖 2 段深文章(如 `/economy/<slug>`)和 3 段深文章(如 `/business/tech/semiconductors/<slug>`)\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **删除\"末段 ≥5 连字符\"硬规则**。旧 pattern 用 `(?:-[^/?#]*){5,}$` 猜测\"长 slug=文章\",但分类页 slug 也可能很长(如 `artificial-intelligence`、`wealth-management`),短标题文章又会被误伤。改由详情页验证区分。卡片里仍会混入 ~18 条子分类页链接(如 `/business/tech/semiconductors`、`/spotlight/trump-administration`,作为卡片的 tag 链接出现),由 `validateArticle` 在 fetchPage 阶段剔除。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页(非无限滚动),query strip 自动排除分页链接。\n\n## 详情页正文验证(取代 URL 猜测)\n\n`validateArticle: true` 让 `fetchPage` 在提取正文时额外采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 的候选直接 skip。判别逻辑:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n实测信号(2026-07-29):\n| 页面类型 | og:type | JSON-LD @type | 正文段落 | 正文字符 |\n|----------|---------|---------------|----------|----------|\n| 文章(SK Hynix Q2) | article | NewsArticle | 11 | 3671 |\n| 文章(Philippine women) | article | NewsArticle | 30 | 5647 |\n| 分类页(/business/tech/semiconductors) | website | Thing | 0 | 0 |\n\n正文验证发生在候选链接被收集**之后**(fetchPage 阶段),所以放宽 urlPattern 混入的少量分类页会在这一步被剔除,不会进入 registry 或落盘。\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei 的 CSS Modules 正文容器)。\n- 部分文章有付费墙,正文可能不完整——bodyMinChars=500 + bodyMinParagraphs=3 的下限会过滤掉被付费墙截断到极短的页面;preview 验证后若 charCount 过低需排除。\n- 回归测试:`node scripts/recipe-test.js nikkei-tech`(固定文章 URL 必须 isArticle,固定分类页 URL 必须非 isArticle,列表页链接数 ≥10 且不含 exclude 路径)。\n", "helpers/nikkei-tech.md": "---\nsource: nikkei-tech\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\n# 详情页正文验证(见 fetcher-helper.js buildExtractExpr):放宽 urlPattern 后,\n# 靠 og:type / JSON-LD @type / 正文字符数在 fetchPage 阶段剔除分类页,而不是\n# 靠 URL 末段连字符数猜测。实测文章 og:type=article + 正文>3000字符;分类页\n# og:type=website + 正文 0,判别力极强。\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲科技 抓取备忘\n\n## 列表页链接收集(`/business/tech`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测文章卡片分散在这三类容器里(StreamArticleCard 27 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 33 篇)。旧规则只锁定 `StreamArticleCard`,丢了 6 篇在 `SpotlightArticleCard`/`SecondaryArticleCard` 里的真实文章(含 SK Hynix、Philippine women、humanoid robots、Kumamoto quake、Panasonic、KOSPI 等)。\n- **为什么不直接 `listSelector: null`(全页)**:实测全页扫描会被导航栏的 `/location/<country>` 国家下拉链接(`/location/east-asia/china` 等 ~30 条)在 `maxLinks: 30` 上限内塞满,导致真实文章一条都收不到。卡片容器范围是必须的,只要把三类 ArticleCard 容器都纳入即可覆盖全部文章且避开 nav。\n- **urlPattern 扩展顶层段白名单**。旧 pattern 只覆盖 `/business/`、`/spotlight/`;实测 `/economy/`、`/politics/`、`/editor-s-picks/`、`/location/`、`/opinion/`、`/features/`、`/life-arts/`、`/techasia/` 下都出现文章。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/tech/semiconductors/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **删除\"末段 ≥5 连字符\"硬规则**。旧 pattern 用 `(?:-[^/?#]*){5,}$` 猜测\"长 slug=文章\",但分类页 slug 也可能很长(如 `artificial-intelligence`、`wealth-management`),短标题文章又会被误伤。改由详情页验证区分。卡片里仍会混入 ~18 条子分类页链接(如 `/business/tech/semiconductors`、`/spotlight/trump-administration`,作为卡片的 tag 链接出现),由 `validateArticle` 在 fetchPage 阶段剔除。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页(非无限滚动),query strip 自动排除分页链接。\n\n## 详情页正文验证(取代 URL 猜测)\n\n`validateArticle: true` 让 `fetchPage` 在提取正文时额外采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 的候选直接 skip。判别逻辑:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n实测信号(2026-07-29):\n| 页面类型 | og:type | JSON-LD @type | 正文段落 | 正文字符 |\n|----------|---------|---------------|----------|----------|\n| 文章(SK Hynix Q2) | article | NewsArticle | 11 | 3671 |\n| 文章(Philippine women) | article | NewsArticle | 30 | 5647 |\n| 分类页(/business/tech/semiconductors) | website | Thing | 0 | 0 |\n\n正文验证发生在候选链接被收集**之后**(fetchPage 阶段),所以放宽 urlPattern 混入的少量分类页会在这一步被剔除,不会进入 registry 或落盘。\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei 的 CSS Modules 正文容器)。\n- 部分文章有付费墙,正文可能不完整——bodyMinChars=500 + bodyMinParagraphs=3 的下限会过滤掉被付费墙截断到极短的页面;preview 验证后若 charCount 过低需排除。\n- 回归测试:`node scripts/recipe-test.js nikkei-tech`(固定文章 URL 必须 isArticle,固定分类页 URL 必须非 isArticle,列表页链接数 ≥10 且不含 exclude 路径)。\n",
"helpers/nikkei-tech.fixtures.json": "{\n \"_comment\": \"Regression fixtures for the nikkei-tech recipe. Article URLs must fetch with isArticle=true (og:type=article + body ≥500 chars); section URLs must fetch with isArticle=false (og:type=website). URLs verified 2026-07-29 against https://asia.nikkei.com/business/tech. Update when Nikkei archives these — pick current articles from the same path depths (2-seg /economy/<slug>, 3-seg /business/tech/semiconductors/<slug>).\",\n \"articles\": [\n \"https://asia.nikkei.com/business/tech/semiconductors/sk-hynix-q2-profit-surges-but-misses-market-forecast-shares-slide\",\n \"https://asia.nikkei.com/business/tech/semiconductors/ai-chip-boom-shifts-bottleneck-to-advanced-packaging-says-at-s-ceo\",\n \"https://asia.nikkei.com/spotlight/society/for-philippine-women-in-taiwan-s-tech-factories-motherhood-is-impossible\",\n \"https://asia.nikkei.com/spotlight/trump-administration/us-bans-new-chinese-humanoid-robots-to-protect-ai-buildout\",\n \"https://asia.nikkei.com/economy/natural-disasters/major-japan-quake-traps-people-inside-kumamoto-shopping-mall-factory\",\n \"https://asia.nikkei.com/business/markets/south-korea-s-kospi-plunges-11-on-china-chip-competition-fears\",\n \"https://asia.nikkei.com/business/companies/panasonic-to-end-tv-production-in-malaysia-cutting-jobs\"\n ],\n \"sections\": [\n \"https://asia.nikkei.com/business/tech\",\n \"https://asia.nikkei.com/business/tech/semiconductors\",\n \"https://asia.nikkei.com/business/markets\",\n \"https://asia.nikkei.com/economy/natural-disasters\",\n \"https://asia.nikkei.com/spotlight/society\"\n ],\n \"minLinks\": 10\n}\n" "helpers/nikkei-tech.fixtures.json": "{\n \"_comment\": \"Regression fixtures for the nikkei-tech recipe. Article URLs must fetch with isArticle=true (og:type=article + body ≥500 chars); section URLs must fetch with isArticle=false (og:type=website). URLs verified 2026-07-29 against https://asia.nikkei.com/business/tech. Update when Nikkei archives these — pick current articles from the same path depths (2-seg /economy/<slug>, 3-seg /business/tech/semiconductors/<slug>).\",\n \"articles\": [\n \"https://asia.nikkei.com/business/tech/semiconductors/sk-hynix-q2-profit-surges-but-misses-market-forecast-shares-slide\",\n \"https://asia.nikkei.com/business/tech/semiconductors/ai-chip-boom-shifts-bottleneck-to-advanced-packaging-says-at-s-ceo\",\n \"https://asia.nikkei.com/spotlight/society/for-philippine-women-in-taiwan-s-tech-factories-motherhood-is-impossible\",\n \"https://asia.nikkei.com/spotlight/trump-administration/us-bans-new-chinese-humanoid-robots-to-protect-ai-buildout\",\n \"https://asia.nikkei.com/economy/natural-disasters/major-japan-quake-traps-people-inside-kumamoto-shopping-mall-factory\",\n \"https://asia.nikkei.com/business/markets/south-korea-s-kospi-plunges-11-on-china-chip-competition-fears\",\n \"https://asia.nikkei.com/business/companies/panasonic-to-end-tv-production-in-malaysia-cutting-jobs\"\n ],\n \"sections\": [\n \"https://asia.nikkei.com/business/tech\",\n \"https://asia.nikkei.com/business/tech/semiconductors\",\n \"https://asia.nikkei.com/business/markets\",\n \"https://asia.nikkei.com/economy/natural-disasters\",\n \"https://asia.nikkei.com/spotlight/society\"\n ],\n \"minLinks\": 10\n}\n"
} }
} }
\ No newline at end of file
{ {
"format": "news-harvester-source", "format": "news-harvester-source",
"version": 1, "version": 1,
"exportedAt": "2026-07-30T08:41:39.076Z", "exportedAt": "2026-08-04T04:39:33.259Z",
"source": { "source": {
"id": "nikkei-world", "id": "nikkei-world",
"name": "日经亚洲世界", "name": "日经亚洲世界",
...@@ -10,6 +10,6 @@ ...@@ -10,6 +10,6 @@
"vaultFolder": "日经亚洲/世界" "vaultFolder": "日经亚洲/世界"
}, },
"files": { "files": {
"helpers/nikkei-world.md": "---\nsource: nikkei-world\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲世界 抓取备忘\n\n## 列表页链接收集(`/location`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/location` 页面文章卡片分散在这三类容器里(StreamArticleCard 26 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 32 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。`/[^/?#]+/.+` 要求至少 3 段路径\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/commodities` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n" "helpers/nikkei-world.md": "---\nsource: nikkei-world\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲世界 抓取备忘\n\n## 列表页链接收集(`/location`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/location` 页面文章卡片分散在这三类容器里(StreamArticleCard 26 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 32 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/commodities` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
} }
} }
\ No newline at end of file
...@@ -64,6 +64,11 @@ const DEFAULTS = { ...@@ -64,6 +64,11 @@ const DEFAULTS = {
imageSelector: 'img', imageSelector: 'img',
imageMinWidth: 200, imageMinWidth: 200,
excludeImage: ['logo', 'icon', 'avatar', 'profile'], excludeImage: ['logo', 'icon', 'avatar', 'profile'],
// CSS selector: any <img> whose closest ancestor (or itself) matches is
// dropped during extraction. Use this to exclude "recommended articles" /
// "related articles" thumbnails that share the same CDN host as the hero
// image (so src-substring excludeImage can't tell them apart). null = off.
excludeImageSelector: null,
noisePrefixes: [ noisePrefixes: [
'Reporting by', 'Editing by', 'Access unmatched', 'Browse an unrivalled', 'Reporting by', 'Editing by', 'Access unmatched', 'Browse an unrivalled',
'Screen for heightened', 'Sign up here', 'Reuters, the news', 'Screen for heightened', 'Sign up here', 'Reuters, the news',
...@@ -129,6 +134,19 @@ function buildExtractExpr(spec) { ...@@ -129,6 +134,19 @@ function buildExtractExpr(spec) {
const authorSelector = JSON.stringify(spec.authorSelector); const authorSelector = JSON.stringify(spec.authorSelector);
const imageMinWidth = spec.imageMinWidth; const imageMinWidth = spec.imageMinWidth;
const excludeImage = JSON.stringify(spec.excludeImage); const excludeImage = JSON.stringify(spec.excludeImage);
// imageSelector scopes which <img> nodes are candidates for download. The
// default 'img' (whole document) is kept for backward compat, but recipes
// should narrow it to the article-body container so bottom-of-page
// "recommended articles" thumbnails aren't pulled into assets/. Supported
// as a single CSS selector or a comma-separated list (querySelectorAll
// handles both); null/empty falls back to 'img'.
const imageSelector = JSON.stringify(spec.imageSelector || 'img');
// excludeImageSelector: drop an <img> if it (or any ancestor) matches this
// CSS selector. Comma-separated lists are valid. null → no container-based
// exclusion (existing behaviour). Used to filter "recommended articles"
// thumbnails that share a CDN host with the hero image.
const excludeImageSelector = spec.excludeImageSelector
? JSON.stringify(spec.excludeImageSelector) : 'null';
const noisePrefixes = JSON.stringify(spec.noisePrefixes); const noisePrefixes = JSON.stringify(spec.noisePrefixes);
const titleStripRe = JSON.stringify(spec.titleStripSuffix); const titleStripRe = JSON.stringify(spec.titleStripSuffix);
const maxBodyChars = spec.maxBodyChars; const maxBodyChars = spec.maxBodyChars;
...@@ -169,12 +187,20 @@ function buildExtractExpr(spec) { ...@@ -169,12 +187,20 @@ function buildExtractExpr(spec) {
.map(function(a){ return a.innerText.trim(); }).filter(Boolean) .map(function(a){ return a.innerText.trim(); }).filter(Boolean)
.filter(function(v,i,a){ return a.indexOf(v) === i; }).join(', '); .filter(function(v,i,a){ return a.indexOf(v) === i; }).join(', ');
if (!authors && ldAuthorNames.length) authors = ldAuthorNames.join(', '); if (!authors && ldAuthorNames.length) authors = ldAuthorNames.join(', ');
var imgs = Array.from(document.querySelectorAll('img')).map(function(img){ var exclImgSel = ${excludeImageSelector};
return { src: img.src, alt: img.alt || '', width: img.naturalWidth || img.width || 0 }; var imgs = Array.from(document.querySelectorAll(${imageSelector}))
}).filter(function(i){ .filter(function(img){
return i.src && i.src.indexOf('http') === 0 && i.width > ${imageMinWidth} // Container-based exclusion: drop if the img (or any ancestor) sits
&& !${excludeImage}.some(function(x){ return i.src.toLowerCase().indexOf(x) !== -1; }); // inside a "recommended/related articles" block. Skipped when no
}).slice(0, 6); // excludeImageSelector is configured.
return !exclImgSel || !img.closest(exclImgSel);
})
.map(function(img){
return { src: img.src, alt: img.alt || '', width: img.naturalWidth || img.width || 0 };
}).filter(function(i){
return i.src && i.src.indexOf('http') === 0 && i.width > ${imageMinWidth}
&& !${excludeImage}.some(function(x){ return i.src.toLowerCase().indexOf(x) !== -1; });
}).slice(0, 6);
var body = ''; var body = '';
var bodyParagraphs = 0; var bodyParagraphs = 0;
var sels = ${bodySelectors}; var sels = ${bodySelectors};
......
...@@ -222,6 +222,8 @@ bodySelectors: ...@@ -222,6 +222,8 @@ bodySelectors:
bodyMinParagraphs: 3 bodyMinParagraphs: 3
dateSelector: 'time[datetime]' dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a' authorSelector: '[rel="author"], [class*="author"] a'
# Scope to the article-body container (e.g. '[class*="ArticleBody"] img, article img')
# so bottom-of-page "recommended articles" thumbnails aren't downloaded into assets/.
imageSelector: 'img' imageSelector: 'img'
imageMinWidth: 200 imageMinWidth: 200
excludeImage: excludeImage:
...@@ -229,6 +231,9 @@ excludeImage: ...@@ -229,6 +231,9 @@ excludeImage:
- icon - icon
- avatar - avatar
- profile - profile
# excludeImageSelector drops imgs whose ancestor matches — use for
# "recommended/related articles" thumbnails sharing the hero image's CDN host.
# excludeImageSelector: '[class*="RelatedArticle"], [class*="RecommendedArticle"]'
titleStripSuffix: '' titleStripSuffix: ''
maxBodyChars: 15000 maxBodyChars: 15000
--- ---
......
...@@ -201,6 +201,8 @@ bodySelectors: ...@@ -201,6 +201,8 @@ bodySelectors:
bodyMinParagraphs: 3 bodyMinParagraphs: 3
dateSelector: 'time[datetime]' dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a' authorSelector: '[rel="author"], [class*="author"] a'
# Scope to the article-body container (e.g. '[class*="ArticleBody"] img, article img')
# so bottom-of-page "recommended articles" thumbnails aren't downloaded into assets/.
imageSelector: 'img' imageSelector: 'img'
imageMinWidth: 200 imageMinWidth: 200
excludeImage: excludeImage:
...@@ -208,6 +210,9 @@ excludeImage: ...@@ -208,6 +210,9 @@ excludeImage:
- icon - icon
- avatar - avatar
- profile - profile
# excludeImageSelector drops imgs whose ancestor matches — use for
# "recommended/related articles" thumbnails sharing the hero image's CDN host.
# excludeImageSelector: '[class*="RelatedArticle"], [class*="RecommendedArticle"]'
titleStripSuffix: '' titleStripSuffix: ''
maxBodyChars: 15000 maxBodyChars: 15000
--- ---
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment