Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Sign in
Toggle navigation
N
news-harvester
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Back End
news-harvester
Commits
edea36c0
Commit
edea36c0
authored
Aug 05, 2026
by
谢宇轩
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
feat: update reuters bundles
parent
5cdadaef
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
9 additions
and
9 deletions
+9
-9
reuters-business.nhsource.json
bundles/reuters-business.nhsource.json
+3
-3
reuters-markets.nhsource.json
bundles/reuters-markets.nhsource.json
+3
-3
reuters-world.nhsource.json
bundles/reuters-world.nhsource.json
+3
-3
No files found.
bundles/reuters-business.nhsource.json
View file @
edea36c0
{
"format"
:
"news-harvester-source"
,
"version"
:
1
,
"exportedAt"
:
"2026-0
7-30T04:12:38.724
Z"
,
"exportedAt"
:
"2026-0
8-05T06:40:26.452
Z"
,
"source"
:
{
"id"
:
"reuters-business"
,
"name"
:
"
Reuters B
usiness"
,
"name"
:
"
reuters b
usiness"
,
"homepageUrl"
:
"https://www.reuters.com/business/"
,
"method"
:
"browser"
,
"vaultFolder"
:
"
Reuters Business
"
"vaultFolder"
:
"
路透社/商业
"
},
"files"
:
{
"helpers/reuters-business.md"
:
"---
\n
source: reuters-business
\n
listSelector: 'div[data-testid=
\"
Title
\"
]'
\n
linkSelector: 'a[href]'
\n
urlPattern: 'reuters
\\
.com/.+-
\\
d{4}-
\\
d{2}-
\\
d{2}/?$'
\n
excludeUrlPattern: 'reuters
\\
.com/(podcasts|newsletter)/'
\n
scrollSteps: 3
\n
scrollWaitMs: 2000
\n
waitMs: 15000
\n
validateArticle: true
\n
bodyMinChars: 500
\n
bodySelectors:
\n
- '[data-testid^=
\"
paragraph
\"
]'
\n
- '[class*=
\"
articleBodyContent
\"
] p'
\n
- '[class*=
\"
article-body
\"
] p'
\n
- 'article p'
\n
bodyMinParagraphs: 3
\n
dateSelector: 'time[datetime]'
\n
authorSelector: '[rel=
\"
author
\"
], [class*=
\"
author
\"
] a'
\n
imageSelector: 'img'
\n
imageMinWidth: 200
\n
excludeImage:
\n
- logo
\n
- icon
\n
- avatar
\n
- profile
\n
titleStripSuffix: '
\\
| Reuters$'
\n
maxBodyChars: 15000
\n
---
\n
# Reuters Business 抓取备忘
\n\n
## 列表页链接收集(`/business/`)
\n\n
- **listSelector: `div[data-testid=
\"
Title
\"
]`**。实测全页 20 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定(hash 不变)。
\n
- **urlPattern: `reuters
\\
.com/.+-
\\
d{4}-
\\
d{2}-
\\
d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/business/`、`/business/finance/`)无日期后缀。放宽为全站匹配(不限 `/business/`),因为板块页会推荐其他分区的文章(如 `/world/asia-pacific/...`)。
\n
- **excludeUrlPattern** 排除 `/podcasts/`、`/newsletter/`(它们也带日期但非正文报道)。
\n
- `scrollSteps: 3` 触发懒加载拿全当天列表;首屏仅 ~10 篇。
\n\n
## 详情页正文验证
\n\n
`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:
\n
- 文章:`og:type=article`、`@type=NewsArticle`、正文 4000+ 字符
\n
- 分类页 `/business/`:`og:type=website`、`@type=CollectionPage`、正文 0
\n\n
Reuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。
\n\n
## 其他
\n\n
- 反爬:headless 会被拦,需 `config.chromium.headless: false`。
\n
- 正文优先 `[data-testid^=
\"
paragraph
\"
]`,旧 `article p` 会被
\"
相关阅读
\"\"
订阅推广
\"
段落污染。
\n
"
...
...
bundles/reuters-markets.nhsource.json
View file @
edea36c0
{
"format"
:
"news-harvester-source"
,
"version"
:
1
,
"exportedAt"
:
"2026-0
7-30T04:12:38.760
Z"
,
"exportedAt"
:
"2026-0
8-05T06:40:26.498
Z"
,
"source"
:
{
"id"
:
"reuters-markets"
,
"name"
:
"
Reuters M
arkets"
,
"name"
:
"
reuters m
arkets"
,
"homepageUrl"
:
"https://www.reuters.com/markets/"
,
"method"
:
"browser"
,
"vaultFolder"
:
"
Reuters Markets
"
"vaultFolder"
:
"
路透社/市场
"
},
"files"
:
{
"helpers/reuters-markets.md"
:
"---
\n
source: reuters-markets
\n
listSelector: '[data-testid=
\"
MediaStoryCard
\"
], [data-testid=
\"
BasicCard
\"
], [data-testid=
\"
HeroCard
\"
], [data-testid=
\"
HubCard
\"
], [data-testid=
\"
AuthorCard
\"
], [data-testid=
\"
OurColumnists
\"
]'
\n
linkSelector: 'a[href]'
\n
urlPattern: 'reuters
\\
.com/.+-
\\
d{4}-
\\
d{2}-
\\
d{2}/?$'
\n
excludeUrlPattern: 'reuters
\\
.com/(podcasts|newsletter)/'
\n
scrollSteps: 3
\n
scrollWaitMs: 2000
\n
waitMs: 15000
\n
validateArticle: true
\n
bodyMinChars: 500
\n
bodySelectors:
\n
- '[data-testid^=
\"
paragraph
\"
]'
\n
- '[class*=
\"
articleBodyContent
\"
] p'
\n
- '[class*=
\"
article-body
\"
] p'
\n
- 'article p'
\n
bodyMinParagraphs: 3
\n
dateSelector: 'time[datetime]'
\n
authorSelector: '[rel=
\"
author
\"
], [class*=
\"
author
\"
] a'
\n
imageSelector: 'img'
\n
imageMinWidth: 200
\n
excludeImage:
\n
- logo
\n
- icon
\n
- avatar
\n
- profile
\n
titleStripSuffix: '
\\
| Reuters$'
\n
maxBodyChars: 15000
\n
---
\n
# Reuters Markets 抓取备忘
\n\n
## 列表页链接收集(`/markets/`)
\n\n
- **listSelector 用复合 `data-testid` 选择器**:`MediaStoryCard`、`BasicCard`、`HeroCard`、`HubCard`、`AuthorCard`、`OurColumnists` 六类卡片容器。实测全页 20 篇文章链接分散在这六类容器里(`MediaStoryCard` 18 + `BasicCard` 12 + `HeroCard`/`HubCard`/`AuthorCard`/`OurColumnists` 各 2-3,去重后 20)。
\n
- **为什么不用旧 `div.media-story-card-module__placement-container__1ZSB1`**:旧选择器依赖带 hash 的 CSS Modules 类名,hash 随构建变化 → 过期后只匹配 9 篇,漏掉 11 篇。`data-testid` 无 hash,稳定得多。Markets 页面布局与 Business/World 不同(`div[data-testid=
\"
Title
\"
]` 在此页 0 篇),必须用 Markets 专属的复合容器。
\n
- **urlPattern: `reuters
\\
.com/.+-
\\
d{4}-
\\
d{2}-
\\
d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/markets/`、`/markets/currencies/`)无日期后缀。放宽为全站匹配,因为 Markets 板块会推荐其他分区文章(如 `/world/asia-pacific/...`、`/business/energy/...`、`/commentary/...`)。
\n
- **excludeUrlPattern** 排除 `/podcasts/`、`/newsletter/`(它们也带日期但非正文报道)。
\n
- `scrollSteps: 3` 触发懒加载拿全当天列表。
\n\n
## 详情页正文验证
\n\n
`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:
\n
- 文章:`og:type=article`、`@type=NewsArticle`、正文 4000+ 字符
\n
- 分类页 `/markets/`:`og:type=website`、正文 0
\n\n
Reuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。
\n\n
## 其他
\n\n
- 反爬:headless 会被拦,需 `config.chromium.headless: false`。
\n
- 正文优先 `[data-testid^=
\"
paragraph
\"
]`,旧 `article p` 会被
\"
相关阅读
\"\"
订阅推广
\"
段落污染。
\n
"
...
...
bundles/reuters-world.nhsource.json
View file @
edea36c0
{
"format"
:
"news-harvester-source"
,
"version"
:
1
,
"exportedAt"
:
"2026-0
7-30T04:12:38.797
Z"
,
"exportedAt"
:
"2026-0
8-05T06:40:26.534
Z"
,
"source"
:
{
"id"
:
"reuters-world"
,
"name"
:
"
路透社 W
orld"
,
"name"
:
"
reuters w
orld"
,
"homepageUrl"
:
"https://www.reuters.com/world/"
,
"method"
:
"browser"
,
"vaultFolder"
:
"路透社
World
"
"vaultFolder"
:
"路透社
/世界
"
},
"files"
:
{
"helpers/reuters-world.md"
:
"---
\n
source: reuters-world
\n
listSelector: 'div[data-testid=
\"
Title
\"
]'
\n
linkSelector: 'a[href]'
\n
urlPattern: 'reuters
\\
.com/.+-
\\
d{4}-
\\
d{2}-
\\
d{2}/?$'
\n
excludeUrlPattern: 'reuters
\\
.com/(podcasts|newsletter)/'
\n
scrollSteps: 3
\n
scrollWaitMs: 2000
\n
waitMs: 15000
\n
validateArticle: true
\n
bodyMinChars: 500
\n
bodySelectors:
\n
- '[data-testid^=
\"
paragraph
\"
]'
\n
- '[class*=
\"
articleBodyContent
\"
] p'
\n
- '[class*=
\"
article-body
\"
] p'
\n
- 'article p'
\n
bodyMinParagraphs: 3
\n
dateSelector: 'time[datetime]'
\n
authorSelector: '[rel=
\"
author
\"
], [class*=
\"
author
\"
] a'
\n
imageSelector: 'img'
\n
imageMinWidth: 200
\n
excludeImage:
\n
- logo
\n
- icon
\n
- avatar
\n
- profile
\n
titleStripSuffix: '
\\
| Reuters$'
\n
maxBodyChars: 15000
\n
---
\n
# 路透社 World 抓取备忘
\n\n
## 列表页链接收集(`/world/`)
\n\n
- **listSelector: `div[data-testid=
\"
Title
\"
]`**。实测全页 19 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定。
\n
- **urlPattern: `reuters
\\
.com/.+-
\\
d{4}-
\\
d{2}-
\\
d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/world/`、`/world/asia-pacific/`)无日期后缀。放宽为全站匹配(不限 `/world/`),因为板块页会推荐其他分区的文章(如 `/business/environment/...`、`/legal/government/...`)。
\n
- **excludeUrlPattern** 排除 `/podcasts/`、`/newsletter/`(它们也带日期但非正文报道)。
\n
- `scrollSteps: 3` 触发懒加载拿全当天列表。
\n\n
## 详情页正文验证
\n\n
`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:
\n
- 文章:`og:type=article`、`@type=NewsArticle`、正文 6000+ 字符
\n
- 分类页 `/world/`:`og:type=website`、正文 0
\n\n
Reuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。
\n\n
## 其他
\n\n
- 反爬:headless 会被拦,需 `config.chromium.headless: false`。
\n
- 正文优先 `[data-testid^=
\"
paragraph
\"
]`,旧 `article p` 会被
\"
相关阅读
\"\"
订阅推广
\"
段落污染。
\n
"
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment