Commit be725014 authored by 谢宇轩's avatar 谢宇轩

fix: edit img select and nhsource bundles

parent 518b835c
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-07-30T08:41:39.136Z",
"exportedAt": "2026-08-04T04:39:33.227Z",
"source": {
"id": "nikkei-markets",
"name": "日经亚洲市场",
......@@ -10,6 +10,6 @@
"vaultFolder": "日经亚洲/市场"
},
"files": {
"helpers/nikkei-markets.md": "---\nsource: nikkei-markets\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲市场 抓取备忘\n\n## 列表页链接收集(`/business/markets`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/business/markets` 页面文章卡片分散在这三类容器里(StreamArticleCard 24 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 30 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。`/[^/?#]+/.+` 要求至少 3 段路径\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/currencies`、`/business/markets/property` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
"helpers/nikkei-markets.md": "---\nsource: nikkei-markets\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲市场 抓取备忘\n\n## 列表页链接收集(`/business/markets`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/business/markets` 页面文章卡片分散在这三类容器里(StreamArticleCard 24 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 30 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets/commodities`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/currencies`、`/business/markets/property` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
}
}
\ No newline at end of file
This diff is collapsed.
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-07-30T08:41:39.076Z",
"exportedAt": "2026-08-04T04:39:33.259Z",
"source": {
"id": "nikkei-world",
"name": "日经亚洲世界",
......@@ -10,6 +10,6 @@
"vaultFolder": "日经亚洲/世界"
},
"files": {
"helpers/nikkei-world.md": "---\nsource: nikkei-world\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲世界 抓取备忘\n\n## 列表页链接收集(`/location`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/location` 页面文章卡片分散在这三类容器里(StreamArticleCard 26 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 32 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。`/[^/?#]+/.+` 要求至少 3 段路径\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/commodities` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
"helpers/nikkei-world.md": "---\nsource: nikkei-world\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲世界 抓取备忘\n\n## 列表页链接收集(`/location`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/location` 页面文章卡片分散在这三类容器里(StreamArticleCard 26 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 32 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/commodities` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
}
}
\ No newline at end of file
......@@ -64,6 +64,11 @@ const DEFAULTS = {
imageSelector: 'img',
imageMinWidth: 200,
excludeImage: ['logo', 'icon', 'avatar', 'profile'],
// CSS selector: any <img> whose closest ancestor (or itself) matches is
// dropped during extraction. Use this to exclude "recommended articles" /
// "related articles" thumbnails that share the same CDN host as the hero
// image (so src-substring excludeImage can't tell them apart). null = off.
excludeImageSelector: null,
noisePrefixes: [
'Reporting by', 'Editing by', 'Access unmatched', 'Browse an unrivalled',
'Screen for heightened', 'Sign up here', 'Reuters, the news',
......@@ -129,6 +134,19 @@ function buildExtractExpr(spec) {
const authorSelector = JSON.stringify(spec.authorSelector);
const imageMinWidth = spec.imageMinWidth;
const excludeImage = JSON.stringify(spec.excludeImage);
// imageSelector scopes which <img> nodes are candidates for download. The
// default 'img' (whole document) is kept for backward compat, but recipes
// should narrow it to the article-body container so bottom-of-page
// "recommended articles" thumbnails aren't pulled into assets/. Supported
// as a single CSS selector or a comma-separated list (querySelectorAll
// handles both); null/empty falls back to 'img'.
const imageSelector = JSON.stringify(spec.imageSelector || 'img');
// excludeImageSelector: drop an <img> if it (or any ancestor) matches this
// CSS selector. Comma-separated lists are valid. null → no container-based
// exclusion (existing behaviour). Used to filter "recommended articles"
// thumbnails that share a CDN host with the hero image.
const excludeImageSelector = spec.excludeImageSelector
? JSON.stringify(spec.excludeImageSelector) : 'null';
const noisePrefixes = JSON.stringify(spec.noisePrefixes);
const titleStripRe = JSON.stringify(spec.titleStripSuffix);
const maxBodyChars = spec.maxBodyChars;
......@@ -169,12 +187,20 @@ function buildExtractExpr(spec) {
.map(function(a){ return a.innerText.trim(); }).filter(Boolean)
.filter(function(v,i,a){ return a.indexOf(v) === i; }).join(', ');
if (!authors && ldAuthorNames.length) authors = ldAuthorNames.join(', ');
var imgs = Array.from(document.querySelectorAll('img')).map(function(img){
return { src: img.src, alt: img.alt || '', width: img.naturalWidth || img.width || 0 };
}).filter(function(i){
return i.src && i.src.indexOf('http') === 0 && i.width > ${imageMinWidth}
&& !${excludeImage}.some(function(x){ return i.src.toLowerCase().indexOf(x) !== -1; });
}).slice(0, 6);
var exclImgSel = ${excludeImageSelector};
var imgs = Array.from(document.querySelectorAll(${imageSelector}))
.filter(function(img){
// Container-based exclusion: drop if the img (or any ancestor) sits
// inside a "recommended/related articles" block. Skipped when no
// excludeImageSelector is configured.
return !exclImgSel || !img.closest(exclImgSel);
})
.map(function(img){
return { src: img.src, alt: img.alt || '', width: img.naturalWidth || img.width || 0 };
}).filter(function(i){
return i.src && i.src.indexOf('http') === 0 && i.width > ${imageMinWidth}
&& !${excludeImage}.some(function(x){ return i.src.toLowerCase().indexOf(x) !== -1; });
}).slice(0, 6);
var body = '';
var bodyParagraphs = 0;
var sels = ${bodySelectors};
......
......@@ -222,6 +222,8 @@ bodySelectors:
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
# Scope to the article-body container (e.g. '[class*="ArticleBody"] img, article img')
# so bottom-of-page "recommended articles" thumbnails aren't downloaded into assets/.
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
......@@ -229,6 +231,9 @@ excludeImage:
- icon
- avatar
- profile
# excludeImageSelector drops imgs whose ancestor matches — use for
# "recommended/related articles" thumbnails sharing the hero image's CDN host.
# excludeImageSelector: '[class*="RelatedArticle"], [class*="RecommendedArticle"]'
titleStripSuffix: ''
maxBodyChars: 15000
---
......
......@@ -201,6 +201,8 @@ bodySelectors:
bodyMinParagraphs: 3
dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a'
# Scope to the article-body container (e.g. '[class*="ArticleBody"] img, article img')
# so bottom-of-page "recommended articles" thumbnails aren't downloaded into assets/.
imageSelector: 'img'
imageMinWidth: 200
excludeImage:
......@@ -208,6 +210,9 @@ excludeImage:
- icon
- avatar
- profile
# excludeImageSelector drops imgs whose ancestor matches — use for
# "recommended/related articles" thumbnails sharing the hero image's CDN host.
# excludeImageSelector: '[class*="RelatedArticle"], [class*="RecommendedArticle"]'
titleStripSuffix: ''
maxBodyChars: 15000
---
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment