Commit be725014 authored by 谢宇轩's avatar 谢宇轩

fix: edit img select and nhsource bundles

parent 518b835c
{ {
"format": "news-harvester-source", "format": "news-harvester-source",
"version": 1, "version": 1,
"exportedAt": "2026-07-30T08:41:39.136Z", "exportedAt": "2026-08-04T04:39:33.227Z",
"source": { "source": {
"id": "nikkei-markets", "id": "nikkei-markets",
"name": "日经亚洲市场", "name": "日经亚洲市场",
...@@ -10,6 +10,6 @@ ...@@ -10,6 +10,6 @@
"vaultFolder": "日经亚洲/市场" "vaultFolder": "日经亚洲/市场"
}, },
"files": { "files": {
"helpers/nikkei-markets.md": "---\nsource: nikkei-markets\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲市场 抓取备忘\n\n## 列表页链接收集(`/business/markets`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/business/markets` 页面文章卡片分散在这三类容器里(StreamArticleCard 24 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 30 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。`/[^/?#]+/.+` 要求至少 3 段路径\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/currencies`、`/business/markets/property` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n" "helpers/nikkei-markets.md": "---\nsource: nikkei-markets\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲市场 抓取备忘\n\n## 列表页链接收集(`/business/markets`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/business/markets` 页面文章卡片分散在这三类容器里(StreamArticleCard 24 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 30 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets/commodities`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/currencies`、`/business/markets/property` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
} }
} }
\ No newline at end of file
This diff is collapsed.
{ {
"format": "news-harvester-source", "format": "news-harvester-source",
"version": 1, "version": 1,
"exportedAt": "2026-07-30T08:41:39.076Z", "exportedAt": "2026-08-04T04:39:33.259Z",
"source": { "source": {
"id": "nikkei-world", "id": "nikkei-world",
"name": "日经亚洲世界", "name": "日经亚洲世界",
...@@ -10,6 +10,6 @@ ...@@ -10,6 +10,6 @@
"vaultFolder": "日经亚洲/世界" "vaultFolder": "日经亚洲/世界"
}, },
"files": { "files": {
"helpers/nikkei-world.md": "---\nsource: nikkei-world\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+/.+'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲世界 抓取备忘\n\n## 列表页链接收集(`/location`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/location` 页面文章卡片分散在这三类容器里(StreamArticleCard 26 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 32 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。`/[^/?#]+/.+` 要求至少 3 段路径\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/commodities` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n" "helpers/nikkei-world.md": "---\nsource: nikkei-world\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲世界 抓取备忘\n\n## 列表页链接收集(`/location`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/location` 页面文章卡片分散在这三类容器里(StreamArticleCard 26 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 32 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets`),由 `validateArticle: true` 在 fetchPage 阶段剔除\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/commodities` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
} }
} }
\ No newline at end of file
...@@ -64,6 +64,11 @@ const DEFAULTS = { ...@@ -64,6 +64,11 @@ const DEFAULTS = {
imageSelector: 'img', imageSelector: 'img',
imageMinWidth: 200, imageMinWidth: 200,
excludeImage: ['logo', 'icon', 'avatar', 'profile'], excludeImage: ['logo', 'icon', 'avatar', 'profile'],
// CSS selector: any <img> whose closest ancestor (or itself) matches is
// dropped during extraction. Use this to exclude "recommended articles" /
// "related articles" thumbnails that share the same CDN host as the hero
// image (so src-substring excludeImage can't tell them apart). null = off.
excludeImageSelector: null,
noisePrefixes: [ noisePrefixes: [
'Reporting by', 'Editing by', 'Access unmatched', 'Browse an unrivalled', 'Reporting by', 'Editing by', 'Access unmatched', 'Browse an unrivalled',
'Screen for heightened', 'Sign up here', 'Reuters, the news', 'Screen for heightened', 'Sign up here', 'Reuters, the news',
...@@ -129,6 +134,19 @@ function buildExtractExpr(spec) { ...@@ -129,6 +134,19 @@ function buildExtractExpr(spec) {
const authorSelector = JSON.stringify(spec.authorSelector); const authorSelector = JSON.stringify(spec.authorSelector);
const imageMinWidth = spec.imageMinWidth; const imageMinWidth = spec.imageMinWidth;
const excludeImage = JSON.stringify(spec.excludeImage); const excludeImage = JSON.stringify(spec.excludeImage);
// imageSelector scopes which <img> nodes are candidates for download. The
// default 'img' (whole document) is kept for backward compat, but recipes
// should narrow it to the article-body container so bottom-of-page
// "recommended articles" thumbnails aren't pulled into assets/. Supported
// as a single CSS selector or a comma-separated list (querySelectorAll
// handles both); null/empty falls back to 'img'.
const imageSelector = JSON.stringify(spec.imageSelector || 'img');
// excludeImageSelector: drop an <img> if it (or any ancestor) matches this
// CSS selector. Comma-separated lists are valid. null → no container-based
// exclusion (existing behaviour). Used to filter "recommended articles"
// thumbnails that share a CDN host with the hero image.
const excludeImageSelector = spec.excludeImageSelector
? JSON.stringify(spec.excludeImageSelector) : 'null';
const noisePrefixes = JSON.stringify(spec.noisePrefixes); const noisePrefixes = JSON.stringify(spec.noisePrefixes);
const titleStripRe = JSON.stringify(spec.titleStripSuffix); const titleStripRe = JSON.stringify(spec.titleStripSuffix);
const maxBodyChars = spec.maxBodyChars; const maxBodyChars = spec.maxBodyChars;
...@@ -169,12 +187,20 @@ function buildExtractExpr(spec) { ...@@ -169,12 +187,20 @@ function buildExtractExpr(spec) {
.map(function(a){ return a.innerText.trim(); }).filter(Boolean) .map(function(a){ return a.innerText.trim(); }).filter(Boolean)
.filter(function(v,i,a){ return a.indexOf(v) === i; }).join(', '); .filter(function(v,i,a){ return a.indexOf(v) === i; }).join(', ');
if (!authors && ldAuthorNames.length) authors = ldAuthorNames.join(', '); if (!authors && ldAuthorNames.length) authors = ldAuthorNames.join(', ');
var imgs = Array.from(document.querySelectorAll('img')).map(function(img){ var exclImgSel = ${excludeImageSelector};
return { src: img.src, alt: img.alt || '', width: img.naturalWidth || img.width || 0 }; var imgs = Array.from(document.querySelectorAll(${imageSelector}))
}).filter(function(i){ .filter(function(img){
return i.src && i.src.indexOf('http') === 0 && i.width > ${imageMinWidth} // Container-based exclusion: drop if the img (or any ancestor) sits
&& !${excludeImage}.some(function(x){ return i.src.toLowerCase().indexOf(x) !== -1; }); // inside a "recommended/related articles" block. Skipped when no
}).slice(0, 6); // excludeImageSelector is configured.
return !exclImgSel || !img.closest(exclImgSel);
})
.map(function(img){
return { src: img.src, alt: img.alt || '', width: img.naturalWidth || img.width || 0 };
}).filter(function(i){
return i.src && i.src.indexOf('http') === 0 && i.width > ${imageMinWidth}
&& !${excludeImage}.some(function(x){ return i.src.toLowerCase().indexOf(x) !== -1; });
}).slice(0, 6);
var body = ''; var body = '';
var bodyParagraphs = 0; var bodyParagraphs = 0;
var sels = ${bodySelectors}; var sels = ${bodySelectors};
......
...@@ -222,6 +222,8 @@ bodySelectors: ...@@ -222,6 +222,8 @@ bodySelectors:
bodyMinParagraphs: 3 bodyMinParagraphs: 3
dateSelector: 'time[datetime]' dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a' authorSelector: '[rel="author"], [class*="author"] a'
# Scope to the article-body container (e.g. '[class*="ArticleBody"] img, article img')
# so bottom-of-page "recommended articles" thumbnails aren't downloaded into assets/.
imageSelector: 'img' imageSelector: 'img'
imageMinWidth: 200 imageMinWidth: 200
excludeImage: excludeImage:
...@@ -229,6 +231,9 @@ excludeImage: ...@@ -229,6 +231,9 @@ excludeImage:
- icon - icon
- avatar - avatar
- profile - profile
# excludeImageSelector drops imgs whose ancestor matches — use for
# "recommended/related articles" thumbnails sharing the hero image's CDN host.
# excludeImageSelector: '[class*="RelatedArticle"], [class*="RecommendedArticle"]'
titleStripSuffix: '' titleStripSuffix: ''
maxBodyChars: 15000 maxBodyChars: 15000
--- ---
......
...@@ -201,6 +201,8 @@ bodySelectors: ...@@ -201,6 +201,8 @@ bodySelectors:
bodyMinParagraphs: 3 bodyMinParagraphs: 3
dateSelector: 'time[datetime]' dateSelector: 'time[datetime]'
authorSelector: '[rel="author"], [class*="author"] a' authorSelector: '[rel="author"], [class*="author"] a'
# Scope to the article-body container (e.g. '[class*="ArticleBody"] img, article img')
# so bottom-of-page "recommended articles" thumbnails aren't downloaded into assets/.
imageSelector: 'img' imageSelector: 'img'
imageMinWidth: 200 imageMinWidth: 200
excludeImage: excludeImage:
...@@ -208,6 +210,9 @@ excludeImage: ...@@ -208,6 +210,9 @@ excludeImage:
- icon - icon
- avatar - avatar
- profile - profile
# excludeImageSelector drops imgs whose ancestor matches — use for
# "recommended/related articles" thumbnails sharing the hero image's CDN host.
# excludeImageSelector: '[class*="RelatedArticle"], [class*="RecommendedArticle"]'
titleStripSuffix: '' titleStripSuffix: ''
maxBodyChars: 15000 maxBodyChars: 15000
--- ---
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment