Skip to content

N
News Hub Skills
Commit d5995b60 authored by 谢宇轩's avatar 谢宇轩
Browse files
Options

Initial commit

parents
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1907 additions and 0 deletions
.gitignore 0 → 100644
View file @ d5995b60
# Machine-specific runtime config (auto-generated from references/config-template.json)
config.json
config.json.bak
# Runtime dedup registry (harvester only)
data/
# Dependencies
node_modules/
# Logs / temp
*.log
/tmp/
# Local source recipes & fixtures (product files)
# These are installed from shareable bundles (bundles/*.nhsource.json) or
# authored locally — the bundle is the committed unit, not the helper.
helpers/
# IDE / agent workspace files
.zcode/
README.md 0 → 100644
View file @ d5995b60
# News Hub Skills
Two independent skills for the news harvesting pipeline:
## news-source-analyzer (无状态分析器)
Analyze news websites, author fetch recipes, and produce installable source bundles (`.nhsource.json`). Stateless — no vault dependency, no source registry. Works on any machine with Chromium installed.
## news-harvester (采集器)
Harvest and archive news articles from configured sources to Obsidian vault with Chinese summaries + original text. Sources are installed from bundles produced by the analyzer. Manages vault path, push config, and source registry.
## Data flow
```
Analyzer (any machine) Harvester (collection machine)
───────────────── ─────────────────────────────
inspect → recipe.md install bundle.nhsource.json
↓ → helpers/<id>.md + config.sources
preview → verify article extraction ↓
↓ harvest → summaries → finalize → push
recipe-test → regression lock ↓
↓ Obsidian vault + News Hub
pack → bundle.nhsource.json
│
└───── manual/scripted transfer ─────→ harvester
```
The `.nhsource.json` bundle is the only interface between the two skills. No shared runtime state, no shared config, no shared browser profile.
## Shared scripts
Four CDP/Chromium infrastructure scripts (`ensure-chromium.js`, `cdp-client.js`, `fetcher-helper.js`, `block-check.js`) are duplicated in both skills as standalone copies. These are stable low-level CDP wrappers with low change frequency. When modifying one, sync the copy in the other skill.
skills/news-harvester/SKILL.md 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-harvester/bundles/nikkei-markets.nhsource.json 0 → 100644
View file @ d5995b60
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-08-04T04:39:33.227Z",
"source": {
"id": "nikkei-markets",
"name": "日经亚洲市场",
"homepageUrl": "https://asia.nikkei.com/business/markets",
"method": "browser",
"vaultFolder": "日经亚洲/市场"
},
"files": {
"helpers/nikkei-markets.md": "---\nsource: nikkei-markets\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲市场 抓取备忘\n\n## 列表页链接收集(`/business/markets`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/business/markets` 页面文章卡片分散在这三类容器里(StreamArticleCard 24 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 30 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets/commodities`),由 `validateArticle: true` 在 fetchPage 阶段剔除。\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/currencies`、`/business/markets/property` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
}
}
\ No newline at end of file
skills/news-harvester/bundles/nikkei-tech.nhsource.json 0 → 100644
View file @ d5995b60
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-08-04T04:39:33.187Z",
"source": {
"id": "nikkei-tech",
"name": "日经亚洲科技",
"homepageUrl": "https://asia.nikkei.com/business/tech",
"method": "browser",
"vaultFolder": "日经亚洲/科技"
},
"files": {
"helpers/nikkei-tech.md": "---\nsource: nikkei-tech\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\n# 详情页正文验证(见 fetcher-helper.js buildExtractExpr):放宽 urlPattern 后,\n# 靠 og:type / JSON-LD @type / 正文字符数在 fetchPage 阶段剔除分类页,而不是\n# 靠 URL 末段连字符数猜测。实测文章 og:type=article + 正文>3000字符;分类页\n# og:type=website + 正文 0,判别力极强。\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲科技 抓取备忘\n\n## 列表页链接收集(`/business/tech`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测文章卡片分散在这三类容器里(StreamArticleCard 27 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 33 篇)。旧规则只锁定 `StreamArticleCard`,丢了 6 篇在 `SpotlightArticleCard`/`SecondaryArticleCard` 里的真实文章(含 SK Hynix、Philippine women、humanoid robots、Kumamoto quake、Panasonic、KOSPI 等)。\n- **为什么不直接 `listSelector: null`(全页)**:实测全页扫描会被导航栏的 `/location/<country>` 国家下拉链接(`/location/east-asia/china` 等 ~30 条)在 `maxLinks: 30` 上限内塞满,导致真实文章一条都收不到。卡片容器范围是必须的,只要把三类 ArticleCard 容器都纳入即可覆盖全部文章且避开 nav。\n- **urlPattern 扩展顶层段白名单**。旧 pattern 只覆盖 `/business/`、`/spotlight/`;实测 `/economy/`、`/politics/`、`/editor-s-picks/`、`/location/`、`/opinion/`、`/features/`、`/life-arts/`、`/techasia/` 下都出现文章。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/tech/semiconductors/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets`),由 `validateArticle: true` 在 fetchPage 阶段剔除。\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **删除\"末段 ≥5 连字符\"硬规则**。旧 pattern 用 `(?:-[^/?#]*){5,}$` 猜测\"长 slug=文章\",但分类页 slug 也可能很长(如 `artificial-intelligence`、`wealth-management`),短标题文章又会被误伤。改由详情页验证区分。卡片里仍会混入 ~18 条子分类页链接(如 `/business/tech/semiconductors`、`/spotlight/trump-administration`,作为卡片的 tag 链接出现),由 `validateArticle` 在 fetchPage 阶段剔除。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页(非无限滚动),query strip 自动排除分页链接。\n\n## 详情页正文验证(取代 URL 猜测)\n\n`validateArticle: true` 让 `fetchPage` 在提取正文时额外采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 的候选直接 skip。判别逻辑:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n实测信号(2026-07-29):\n| 页面类型 | og:type | JSON-LD @type | 正文段落 | 正文字符 |\n|----------|---------|---------------|----------|----------|\n| 文章(SK Hynix Q2) | article | NewsArticle | 11 | 3671 |\n| 文章(Philippine women) | article | NewsArticle | 30 | 5647 |\n| 分类页(/business/tech/semiconductors) | website | Thing | 0 | 0 |\n\n正文验证发生在候选链接被收集**之后**(fetchPage 阶段),所以放宽 urlPattern 混入的少量分类页会在这一步被剔除,不会进入 registry 或落盘。\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei 的 CSS Modules 正文容器)。\n- 部分文章有付费墙,正文可能不完整——bodyMinChars=500 + bodyMinParagraphs=3 的下限会过滤掉被付费墙截断到极短的页面;preview 验证后若 charCount 过低需排除。\n- 回归测试:`node scripts/recipe-test.js nikkei-tech`(固定文章 URL 必须 isArticle,固定分类页 URL 必须非 isArticle,列表页链接数 ≥10 且不含 exclude 路径)。\n",
"helpers/nikkei-tech.fixtures.json": "{\n \"_comment\": \"Regression fixtures for the nikkei-tech recipe. Article URLs must fetch with isArticle=true (og:type=article + body ≥500 chars); section URLs must fetch with isArticle=false (og:type=website). URLs verified 2026-07-29 against https://asia.nikkei.com/business/tech. Update when Nikkei archives these — pick current articles from the same path depths (2-seg /economy/<slug>, 3-seg /business/tech/semiconductors/<slug>).\",\n \"articles\": [\n \"https://asia.nikkei.com/business/tech/semiconductors/sk-hynix-q2-profit-surges-but-misses-market-forecast-shares-slide\",\n \"https://asia.nikkei.com/business/tech/semiconductors/ai-chip-boom-shifts-bottleneck-to-advanced-packaging-says-at-s-ceo\",\n \"https://asia.nikkei.com/spotlight/society/for-philippine-women-in-taiwan-s-tech-factories-motherhood-is-impossible\",\n \"https://asia.nikkei.com/spotlight/trump-administration/us-bans-new-chinese-humanoid-robots-to-protect-ai-buildout\",\n \"https://asia.nikkei.com/economy/natural-disasters/major-japan-quake-traps-people-inside-kumamoto-shopping-mall-factory\",\n \"https://asia.nikkei.com/business/markets/south-korea-s-kospi-plunges-11-on-china-chip-competition-fears\",\n \"https://asia.nikkei.com/business/companies/panasonic-to-end-tv-production-in-malaysia-cutting-jobs\"\n ],\n \"sections\": [\n \"https://asia.nikkei.com/business/tech\",\n \"https://asia.nikkei.com/business/tech/semiconductors\",\n \"https://asia.nikkei.com/business/markets\",\n \"https://asia.nikkei.com/economy/natural-disasters\",\n \"https://asia.nikkei.com/spotlight/society\"\n ],\n \"minLinks\": 10\n}\n"
}
}
\ No newline at end of file
skills/news-harvester/bundles/nikkei-world.nhsource.json 0 → 100644
View file @ d5995b60
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-08-04T04:39:33.259Z",
"source": {
"id": "nikkei-world",
"name": "日经亚洲世界",
"homepageUrl": "https://asia.nikkei.com/location",
"method": "browser",
"vaultFolder": "日经亚洲/世界"
},
"files": {
"helpers/nikkei-world.md": "---\nsource: nikkei-world\nlistSelector: '[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'asia\\.nikkei\\.com/(?:business|spotlight|economy|politics|editor-s-picks|location|opinion|features|life-arts|techasia)/[^/?#]+(?:/.+)?'\nexcludeUrlPattern: '/(?:podcasts|static|infographics|member)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[class*=\"ArticleBody\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[class*=\"author\"] a, [rel=\"author\"]'\nimageSelector: 'img'\n# 文章底部\"推荐/相关文章\"卡片缩略图与 hero 头图共用 images.ft.com CDN,\n# src 子串排除(excludeImage)无法区分,改用容器选择器整块排除。\n# 实测容器:RelatedArticle* / RelatedArticleFeature* / LatestOnTopicBottom* / Outbrain(.ob-rec-image)。\n# hero 头图在 NewsArticleHeaderImage 内,不在排除范围 → 保留。\nexcludeImageSelector: '[class*=\"RelatedArticle\"], [class*=\"RelatedArticleFeature\"], [class*=\"LatestOnTopicBottom\"], .ob-rec-image'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\n - placeholder\n - outbrainimg.com\ntitleStripSuffix: ' - Nikkei Asia$'\nmaxBodyChars: 15000\n---\n# 日经亚洲世界 抓取备忘\n\n## 列表页链接收集(`/location`)\n\n- **listSelector 用复合卡片选择器**:`[class*=\"StreamArticleCard\"], [class*=\"SpotlightArticleCard\"], [class*=\"SecondaryArticleCard\"]`。实测 `/location` 页面文章卡片分散在这三类容器里(StreamArticleCard 26 篇 + SpotlightArticleCard 3 篇 + SecondaryArticleCard 3 篇 = 32 篇)。\n- 与 nikkei-tech 同属 Nikkei 模板,卡片容器类型一致,复用同一复合选择器。\n- **urlPattern** 覆盖所有可能的文章路径段:`business`、`spotlight`、`economy`、`politics`、`editor-s-picks`、`location`、`opinion`、`features`、`life-arts`、`techasia`。第三段 `(?:/.+)?` 可选——Nikkei 的 URL 结构不统一:`/business/markets/currencies/<slug>` 是 3 段,但 `/opinion/<slug>`、`/economy/<slug>`、`/politics/<slug>` 是 2 段。2 段 URL 会混入分类页(如 `/business/markets`),由 `validateArticle: true` 在 fetchPage 阶段剔除。\n- **excludeUrlPattern** 剔除 `/podcasts/`、`/static/`、`/infographics/`、`/member/` 等非文章路径。\n- **validateArticle: true** 在 fetchPage 阶段用 og:type + JSON-LD + 正文字符数剔除分类页(卡片 tag 链接如 `/business/markets/commodities` 会被判 isArticle=false 并跳过)。\n- `scrollSteps: 3` 触发懒加载;Nikkei 用 `?page=N` 分页,query strip 自动排除分页链接。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 `fetchPage` 采集判别信号,返回 `isArticle` 布尔;harvest 对 `isArticle === false` 直接 skip:\n\n```\nisArticle = (og:type === 'article' || JSON-LD @type ∈ {NewsArticle, Article})\n && bodyChars >= 500\n && bodyParagraphs >= 3\n```\n\n## 其他\n\n- 正文优先 `[class*=\"ArticleBody\"]`(Nikkei CSS Modules 正文容器)。\n- 部分文章有付费墙,bodyMinChars=500 + bodyMinParagraphs=3 过滤被截断到极短的页面。\n"
}
}
\ No newline at end of file
skills/news-harvester/bundles/reuters-business.nhsource.json 0 → 100644
View file @ d5995b60
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-08-05T06:40:26.452Z",
"source": {
"id": "reuters-business",
"name": "路透社商业",
"homepageUrl": "https://www.reuters.com/business/",
"method": "browser",
"vaultFolder": "路透社/商业"
},
"files": {
"helpers/reuters-business.md": "---\nsource: reuters-business\nlistSelector: 'div[data-testid=\"Title\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$'\nexcludeUrlPattern: 'reuters\\.com/(podcasts|newsletter)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[data-testid^=\"paragraph\"]'\n - '[class*=\"articleBodyContent\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[rel=\"author\"], [class*=\"author\"] a'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\ntitleStripSuffix: ' \\| Reuters$'\nmaxBodyChars: 15000\n---\n# Reuters Business 抓取备忘\n\n## 列表页链接收集(`/business/`)\n\n- **listSelector: `div[data-testid=\"Title\"]`**。实测全页 20 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定(hash 不变)。\n- **urlPattern: `reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/business/`、`/business/finance/`)无日期后缀。放宽为全站匹配(不限 `/business/`),因为板块页会推荐其他分区的文章(如 `/world/asia-pacific/...`)。\n- **excludeUrlPattern** 排除 `/podcasts/`、`/newsletter/`(它们也带日期但非正文报道)。\n- `scrollSteps: 3` 触发懒加载拿全当天列表;首屏仅 ~10 篇。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:\n- 文章:`og:type=article`、`@type=NewsArticle`、正文 4000+ 字符\n- 分类页 `/business/`:`og:type=website`、`@type=CollectionPage`、正文 0\n\nReuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。\n\n## 其他\n\n- 反爬:headless 会被拦,需 `config.chromium.headless: false`。\n- 正文优先 `[data-testid^=\"paragraph\"]`,旧 `article p` 会被\"相关阅读\"\"订阅推广\"段落污染。\n"
}
}
\ No newline at end of file
skills/news-harvester/bundles/reuters-markets.nhsource.json 0 → 100644
View file @ d5995b60
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-08-05T06:40:26.498Z",
"source": {
"id": "reuters-markets",
"name": "路透社市场",
"homepageUrl": "https://www.reuters.com/markets/",
"method": "browser",
"vaultFolder": "路透社/市场"
},
"files": {
"helpers/reuters-markets.md": "---\nsource: reuters-markets\nlistSelector: '[data-testid=\"MediaStoryCard\"], [data-testid=\"BasicCard\"], [data-testid=\"HeroCard\"], [data-testid=\"HubCard\"], [data-testid=\"AuthorCard\"], [data-testid=\"OurColumnists\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$'\nexcludeUrlPattern: 'reuters\\.com/(podcasts|newsletter)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[data-testid^=\"paragraph\"]'\n - '[class*=\"articleBodyContent\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[rel=\"author\"], [class*=\"author\"] a'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\ntitleStripSuffix: ' \\| Reuters$'\nmaxBodyChars: 15000\n---\n# Reuters Markets 抓取备忘\n\n## 列表页链接收集(`/markets/`)\n\n- **listSelector 用复合 `data-testid` 选择器**:`MediaStoryCard`、`BasicCard`、`HeroCard`、`HubCard`、`AuthorCard`、`OurColumnists` 六类卡片容器。实测全页 20 篇文章链接分散在这六类容器里(`MediaStoryCard` 18 + `BasicCard` 12 + `HeroCard`/`HubCard`/`AuthorCard`/`OurColumnists` 各 2-3,去重后 20)。\n- **为什么不用旧 `div.media-story-card-module__placement-container__1ZSB1`**:旧选择器依赖带 hash 的 CSS Modules 类名,hash 随构建变化 → 过期后只匹配 9 篇,漏掉 11 篇。`data-testid` 无 hash,稳定得多。Markets 页面布局与 Business/World 不同(`div[data-testid=\"Title\"]` 在此页 0 篇),必须用 Markets 专属的复合容器。\n- **urlPattern: `reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/markets/`、`/markets/currencies/`)无日期后缀。放宽为全站匹配,因为 Markets 板块会推荐其他分区文章(如 `/world/asia-pacific/...`、`/business/energy/...`、`/commentary/...`)。\n- **excludeUrlPattern** 排除 `/podcasts/`、`/newsletter/`(它们也带日期但非正文报道)。\n- `scrollSteps: 3` 触发懒加载拿全当天列表。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:\n- 文章:`og:type=article`、`@type=NewsArticle`、正文 4000+ 字符\n- 分类页 `/markets/`:`og:type=website`、正文 0\n\nReuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。\n\n## 其他\n\n- 反爬:headless 会被拦,需 `config.chromium.headless: false`。\n- 正文优先 `[data-testid^=\"paragraph\"]`,旧 `article p` 会被\"相关阅读\"\"订阅推广\"段落污染。\n"
}
}
\ No newline at end of file
skills/news-harvester/bundles/reuters-world.nhsource.json 0 → 100644
View file @ d5995b60
{
"format": "news-harvester-source",
"version": 1,
"exportedAt": "2026-08-05T06:40:26.534Z",
"source": {
"id": "reuters-world",
"name": "路透社世界",
"homepageUrl": "https://www.reuters.com/world/",
"method": "browser",
"vaultFolder": "路透社/世界"
},
"files": {
"helpers/reuters-world.md": "---\nsource: reuters-world\nlistSelector: 'div[data-testid=\"Title\"]'\nlinkSelector: 'a[href]'\nurlPattern: 'reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$'\nexcludeUrlPattern: 'reuters\\.com/(podcasts|newsletter)/'\nscrollSteps: 3\nscrollWaitMs: 2000\nwaitMs: 15000\nvalidateArticle: true\nbodyMinChars: 500\nbodySelectors:\n - '[data-testid^=\"paragraph\"]'\n - '[class*=\"articleBodyContent\"] p'\n - '[class*=\"article-body\"] p'\n - 'article p'\nbodyMinParagraphs: 3\ndateSelector: 'time[datetime]'\nauthorSelector: '[rel=\"author\"], [class*=\"author\"] a'\nimageSelector: 'img'\nimageMinWidth: 200\nexcludeImage:\n - logo\n - icon\n - avatar\n - profile\ntitleStripSuffix: ' \\| Reuters$'\nmaxBodyChars: 15000\n---\n# 路透社 World 抓取备忘\n\n## 列表页链接收集(`/world/`)\n\n- **listSelector: `div[data-testid=\"Title\"]`**。实测全页 19 篇文章链接全部在此容器内(单一容器覆盖全部,无分散问题)。该 data-testid 比 CSS Modules 类名稳定。\n- **urlPattern: `reuters\\.com/.+-\\d{4}-\\d{2}-\\d{2}/?$`**。Reuters 文章 URL 以 `-YYYY-MM-DD/` 结尾,日期后缀是极强的文章标识——分类页(`/world/`、`/world/asia-pacific/`)无日期后缀。放宽为全站匹配(不限 `/world/`),因为板块页会推荐其他分区的文章(如 `/business/environment/...`、`/legal/government/...`)。\n- **excludeUrlPattern** 排除 `/podcasts/`、`/newsletter/`(它们也带日期但非正文报道)。\n- `scrollSteps: 3` 触发懒加载拿全当天列表。\n\n## 详情页正文验证\n\n`validateArticle: true` 让 fetchPage 用 `og:type` + JSON-LD `@type` + 正文长度剔除分类页。实测信号:\n- 文章:`og:type=article`、`@type=NewsArticle`、正文 6000+ 字符\n- 分类页 `/world/`:`og:type=website`、正文 0\n\nReuters 的日期后缀 URL 已能区分大部分文章/分类页,validateArticle 作为兜底防止非标准 URL 混入。\n\n## 其他\n\n- 反爬:headless 会被拦,需 `config.chromium.headless: false`。\n- 正文优先 `[data-testid^=\"paragraph\"]`,旧 `article p` 会被\"相关阅读\"\"订阅推广\"段落污染。\n"
}
}
\ No newline at end of file
skills/news-harvester/package-lock.json 0 → 100644
View file @ d5995b60
{
"name": "news-harvester",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "news-harvester",
"version": "1.0.0",
"license": "ISC",
"dependencies": {
"js-yaml": "^5.2.2",
"ws": "^8.19.0"
}
},
"node_modules/argparse": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz",
"integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==",
"license": "Python-2.0"
},
"node_modules/js-yaml": {
"version": "5.2.3",
"resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-5.2.3.tgz",
"integrity": "sha512-n+mUVyUX5bVv7G/G2zyIHOhdxfuU1dY2NOFzTQUWiMUbFss8b57NFlgCCaggU78wSw5KVS9cllzeLyzyR+n5nw==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/puzrin"
},
{
"type": "github",
"url": "https://github.com/sponsors/nodeca"
}
],
"license": "MIT",
"dependencies": {
"argparse": "^2.0.1"
},
"bin": {
"js-yaml": "bin/js-yaml.mjs"
}
},
"node_modules/ws": {
"version": "8.21.3",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",
"integrity": "sha512-201TZ/kPWxoPr/OKWjquZR1SWKXcvxdH+e1xrx89b3YbmzLMFCLfnaG1HFIgWzJOEWZ7MvpK++odZufgYR50Rw==",
"license": "MIT",
"engines": {
"node": ">=10.0.0"
},
"peerDependencies": {
"bufferutil": "^4.0.1",
"utf-8-validate": ">=5.0.2"
},
"peerDependenciesMeta": {
"bufferutil": {
"optional": true
},
"utf-8-validate": {
"optional": true
}
}
}
}
}
skills/news-harvester/package.json 0 → 100644
View file @ d5995b60
{
"name": "news-harvester",
"version": "1.0.0",
"description": "News harvester — install, harvest, archive, push",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"type": "commonjs",
"dependencies": {
"js-yaml": "^5.2.2",
"ws": "^8.19.0"
}
}
skills/news-harvester/references/config-template.json 0 → 100644
View file @ d5995b60
{
"vaultPath": "/path/to/your/obsidian/vault",
"registryWindowDays": 7,
"push": {
"enabled": false,
"endpoint": "http://localhost:8080/api/v1",
"appId": "",
"appKeyId": "",
"appSecret": "",
"batchSize": 10,
"autoPushAfterFinalize": true,
"retry": { "maxAttempts": 5, "baseDelayMs": 1000 }
},
"chromium": {
"cdpPort": 9222,
"headless": true,
"executablePath": null,
"userDataDir": null
},
"sources": []
}
skills/news-harvester/references/document-formats.md 0 → 100644
View file @ d5995b60
# Document Formats (reference)
These docs are **generated by the scripts**, not hand-written by the agent —
shown here for debugging or when reading archived files. The agent only writes
`.summaries.json`; `finalize.js` produces the Chinese summary docs and patches
the originals.
## Original Doc (written by `harvest.js`)
Filename: `<英文标题> - <SourceName>原文.md`
Between Step 1 (harvest) and Step 3 (finalize), the Chinese-title links in this file are a **sentinel placeholder** `__SUMMARY_<slug>__` — in both the frontmatter `related` field and the footer wikilink. `finalize.js` replaces them with the real `titleZh`, and overwrites the `tags` block + `category` line with the agent-extracted values. Image captions: only rendered when `alt` text exists (empty alt → bare embed, no `**`).
```markdown
---
title: "<英文标题>"
date: <ISO 8601 published timestamp, e.g. 2026-07-30T21:47:28Z>
tags: # agent-extracted (≤3); placeholder [source] until finalize
- <tag1>
- <tag2>
category: <分类> # agent-supplied; overrides the script's keyword guess
source: "<url>"
publisher: <source>
authors: "<authors>"
type: 原文
related: "[[<中文标题>]]" # sentinel __SUMMARY_<slug>__ until finalize
saved: YYYY-MM-DD
---
# <英文标题>
> [!info] Source
> **Published:** <Shanghai time, e.g. 2026-07-31 05:47>
> **URL:** <url>
> **中文摘要：** [[<中文标题>]]
---
## 配图
![[assets/<slug>-1.jpg]]
*<alt text — omitted if empty>*
---
## 正文
<full article body>
---
[[<中文标题>]]
```
## Chinese Summary Doc (written by `finalize.js`)
File: `<dayDir>/summary/<中文标题>.md` (concise title, ≤30 chars). The `summary/` subdirectory is a sibling of `assets/`. Wikilinks from 原文.md and index.md use the bare title `[[<中文标题>]]` — Obsidian resolves them by basename.
```markdown
---
title: <中文标题>
date: <ISO 8601 published timestamp, e.g. 2026-07-30T21:47:28Z>
tags: # agent-extracted from the summary/body (≤3)
- <tag1>
- <tag2>
category: <分类> # agent-supplied; enables Obsidian property/Bases filtering
source: "<original URL>"
publisher: <source name>
authors: "<author names>"
type: 摘要
related: "[[<英文标题> - <SourceName>原文]]"
saved: YYYY-MM-DD
---
# <中文标题>
> [!info] 文章信息
> **发布：** <Shanghai time, e.g. 2026-07-31 05:47>
> **来源：** <source>
> **作者：** <authors>
> **原文：** [[<英文标题> - <SourceName>原文]]
---
## 事件概述
<2-3 sentences summary in Chinese>
## 背景
<context>
## 关键引语
> "<translated quote>"
## 影响分析
<analysis>
---
[[<英文标题> - <SourceName>原文]]
```
## index.md Format
File: `<vaultFolder>/YYYYMMDD/index.md` (written by `harvest.js`, rewritten by `finalize.js`)
Image asset count is a **real file count** of `assets/` (not "articles with images"), so it reflects downloaded files even before finalize.
```markdown
---
date: YYYY-MM-DD
type: daily-index
source: <source name>
total_articles: N
last_updated: "<ISO timestamp>"
---
# <SourceName> Daily Index · YYYY-MM-DD
> **最近更新：** YYYY-MM-DD HH:MM CST
> **当日归档：** N 篇 | **图片资产：** N 张
## 📊 分类统计
| 分类 | 数量 |
|------|------|
| ... | ... |
## 📋 文章列表
| # | 标题 | 原文标题 | 分类 | 标签 | 发布时间 | 状态 |
|---|------|----------|------|------|----------|------|
| 1 | [[中文标题]] | English Title | ... | tag1, tag2 | YYYY-MM-DD | ✅ |
> 发布时间列是紧凑 YYYY-MM-DD(本地时区),便于表格浏览;frontmatter `date:` 存完整 ISO 8601 时间戳(无损,Obsidian 可按任意时区解析)。
## 🗂️ 分类导航
### 分类1（N篇）
- [[文章1]]
```
## registry.json Format
File: `<vaultFolder>/YYYYMMDD/registry.json` (written incrementally by `harvest.js`, patched by `finalize.js`)
`titleZh` is the sentinel `__SUMMARY_<slug>__` until finalize supplies the real title.
```json
{
"date": "YYYYMMDD",
"source": "<source_id>",
"articles": [
{
"url": "https://...",
"titleZh": "中文标题",
"titleEn": "English Title",
"date": "2026-07-24T14:30:00.000Z",
"processedAt": "2026-07-24T15:00:00.000Z",
"category": "地缘政治",
"tags": ["tag1", "tag2"],
"slug": "theconversation-com-some-article-286430",
"originalFilename": "Some Article - The Conversation原文.md"
}
]
}
```
skills/news-harvester/references/push-api-spec.md 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-harvester/references/source-management.md 0 → 100644
View file @ d5995b60
# Source Management
## List Sources
```bash
node scripts/source-manage.js list
```
```
ID NAME METHOD RECIPE STATUS
reuters-world 路透社世界 browser ✅ ✅ enabled
theconversation The Conversation direct — ❌ disabled
```
RECIPE ✅ = `helpers/<id>.md` exists (spec-driven fetcher); — = legacy fetcher.
## Edit Source
```bash
node scripts/source-manage.js edit <id> --name "New Name" --url https://... --method browser
```
Editable fields: `name`, `homepageUrl`, `method`, `articleUrlPattern`, `vaultFolder`, `enabled`.
## Enable / Disable Source
```bash
node scripts/source-manage.js enable <id>
node scripts/source-manage.js disable <id>
```
---
## Installing sources from bundles
Sources are added by installing `.nhsource.json` bundles produced by the **news-source-analyzer** skill. There is no `add` or `export` command in the harvester — use the analyzer to research and pack new sources.
```bash
node scripts/source-manage.js install <pkg.nhsource.json> [--force]
```
`install` writes the recipe + fixtures into `helpers/`, registers the source in `config.json` (enabled), and the source is immediately usable.
### Bundle format
A `.nhsource.json` is plain JSON:
| Field | Contents |
|-------|----------|
| `format` | `"news-harvester-source"` (install validates this) |
| `version` | `1` |
| `exportedAt` | ISO timestamp |
| `source` | `{ id, name, homepageUrl, method, vaultFolder }` — the config entry, minus `vaultPath`, `enabled`, and `articleUrlPattern` |
| `files` | Map of relative path → file content. Includes `helpers/<id>.md` (the recipe) and `helpers/<id>.fixtures.json` (if present). |
### Install behavior
- Writes every file in `files` into `helpers/` (path traversal guarded — only basenames under `helpers/` are written).
- Registers the source in `config.json` as enabled. If `<id>` already exists, the entry is overwritten in place; otherwise it's appended.
- **Conflict handling**: refuses if `<id>` is already in `config.json` or a helper file of the same name exists, unless `--force` is passed. `--force` overwrites the config entry and all helper files.
- **vaultPath caveat**: install succeeds even when `config.vaultPath` is still the placeholder (the source is registered, `--preview` works), but archiving requires a real vault path — install prints a warning in that case.
- After install, the source is immediately usable: `harvest.js <id> 3 --preview`.
Bundles carry no signature/checksum (personal-sharing scope). To vet a bundle before installing, open it in an editor and check the `source` block + the recipe frontmatter in `files["helpers/<id>.md"]`.
### Managing recipes
Recipes live in `helpers/` (installed from bundles, gitignored). Each installed source has a `helpers/<id>.md` file. `source-manage.js list` shows which sources have recipes (✅) vs legacy (—).
If you need to research or create a new recipe, use the **news-source-analyzer** skill — it has `inspect-source.js`, `preview.js`, `recipe-test.js`, and `pack.js` for the full recipe authoring workflow.
skills/news-harvester/scripts/block-check.js 0 → 100644
View file @ d5995b60
/**
* block-check.js — Detect human-verification (CAPTCHA / anti-bot challenge) and
* login-state-loss (auth redirect / paywall login wall) on fetched pages.
*
* When a block is detected, a HumanInterventionError is thrown so harvest.js can
* exit immediately with a friendly message instead of silently archiving empty
* or garbage articles. The check is intentionally conservative: it only fires on
* high-confidence signals (challenge iframes/titles, auth-host redirects, or a
* login-wall CTA paired with a near-empty body) to avoid false positives on
* legitimate articles.
*
* Two entry points:
* • Browser (CDP) sources — evaluateBrowserBlock(ws) runs an in-page probe via
* cdpEval and throws if blocked. Used by fetcher-helper.js / fetcher-browser.js.
* • Direct HTTP sources — checkDirectBlock(html, url) does a string scan of the
* raw HTML. Used by fetcher-direct.js (open sites rarely need login, but a
* Cloudflare-style challenge can still appear).
*/
// Sent to the page as a Runtime.evaluate expression. Serialized from a real
// function via .toString() so regex backslashes survive without double-escaping.
function _browserBlockProbe() {
var url = location.href, host = location.hostname;
var title = (document.title || '').trim();
var bodyText = (document.body && document.body.innerText) ? document.body.innerText.substring(0, 4000) : '';
var bodyLen = bodyText.length;
// 1. CAPTCHA / anti-bot challenge — iframes from known challenge providers.
var challengeIframe = Array.from(document.querySelectorAll('iframe[src]')).some(function (f) {
var s = f.src.toLowerCase();
return s.indexOf('recaptcha') !== -1 || s.indexOf('hcaptcha') !== -1
|| s.indexOf('challenges.cloudflare.com') !== -1
|| s.indexOf('turnstile') !== -1
|| s.indexOf('arkoselabs') !== -1 || s.indexOf('funcaptcha') !== -1;
});
// Challenge page titles (Cloudflare "Just a moment...", "Access Denied", etc.).
var challengeTitle = /^(just a moment|attention required|verify|checking your browser|are you human)/i.test(title)
|| /access denied|请输入验证码|安全验证/i.test(title);
// Challenge body copy.
var challengeText = /please verify (that )?you are human|verify you are human|unusual traffic from your computer|checking your browser before accessing|enable javascript and cookies to continue|our system has detected unusual traffic|请完成安全验证|进行人机验证/i.test(bodyText);
if (challengeIframe || challengeTitle || challengeText) {
return JSON.stringify({ blocked: true, kind: 'captcha', reason: '人机验证/反爬挑战页面', url: url, title: title });
}
// 2. Login-state loss — the article URL redirected to an auth/login host or
// path. A real article page is never served from accounts.* / login.* or a
// /signin path, so this is a high-confidence signal regardless of body length.
var authHost = /^(accounts\.|login\.|signin\.|auth\.|sso\.|chkpt\.)/i.test(host);
var authPath = /\/(signin|sign-in|login|auth|account\/login|subscribe|sso)\b/i.test(location.pathname + location.search);
if (authHost || authPath) {
return JSON.stringify({ blocked: true, kind: 'login', reason: '页面被重定向到登录/认证页面（登录状态可能已失效）', url: url, title: title });
}
// 3. Paywall / login wall — a login/subscribe CTA paired with a near-empty
// body (the page is essentially a gate, not an article). The CTA regex is
// specific ("... to continue/read") to avoid matching normal marketing copy.
var loginCta = /sign in to (continue|read)|log in to (continue|read)|subscribe to (continue|read|keep)|以继续阅读|登录后查看|订阅后查看|登录以继续|登录后可继续阅读/i.test(bodyText);
if (loginCta && bodyLen < 800) {
return JSON.stringify({ blocked: true, kind: 'login', reason: '检测到登录/订阅墙（正文极短且出现登录/订阅提示）', url: url, title: title });
}
return JSON.stringify({ blocked: false, kind: null, reason: '', url: url, title: title });
}
const BROWSER_CHECK_EXPR = '(' + _browserBlockProbe.toString() + ')()';
// Custom error — harvest.js checks `e.name === 'HumanInterventionError'` (or
// `instanceof`) to distinguish a block from an ordinary fetch failure.
class HumanInterventionError extends Error {
constructor(reason, { kind, url, title } = {}) {
super(reason);
this.name = 'HumanInterventionError';
this.kind = kind; // 'captcha' | 'login'
this.blockedUrl = url;
this.pageTitle = title;
}
}
// Parse the JSON string returned by the in-page probe and throw if blocked.
function evaluateBlockResult(raw) {
if (!raw) return;
let info;
try { info = JSON.parse(raw); } catch (e) { return; }
if (info && info.blocked) {
throw new HumanInterventionError(info.reason, { kind: info.kind, url: info.url, title: info.title });
}
}
// Browser (CDP) entry point. Runs the probe in the page via cdpEval; throws
// HumanInterventionError if the page is a challenge/login wall.
async function evaluateBrowserBlock(ws) {
const { cdpEval } = require('./cdp-client');
const raw = await cdpEval(ws, BROWSER_CHECK_EXPR);
evaluateBlockResult(raw);
}
// ── Direct (HTTP) entry point ───────────────────────────────────────────────
// String scan of raw HTML for challenge/login-wall signals. No DOM, so the
// signals are a subset of the browser probe — enough to catch a Cloudflare
// interstitial or a login redirect on an open site.
const DIRECT_CHALLENGE_RE = /please verify (that )?you are human|verify you are human|unusual traffic from your computer|checking your browser before accessing|enable javascript and cookies to continue|our system has detected unusual traffic|请完成安全验证|进行人机验证/i;
const DIRECT_CHALLENGE_TITLE_RE = /<title[^>]*>(just a moment|attention required|verify|checking your browser|are you human|access denied|请输入验证码|安全验证)/i;
const DIRECT_LOGIN_CTA_RE = /sign in to (continue|read)|log in to (continue|read)|subscribe to (continue|read|keep)|以继续阅读|登录后查看|订阅后查看|登录以继续|登录后可继续阅读/i;
function checkDirectBlock(html, url) {
if (!html) return;
const head = html.substring(0, 8000);
if (DIRECT_CHALLENGE_TITLE_RE.test(head) || DIRECT_CHALLENGE_RE.test(head)) {
throw new HumanInterventionError('人机验证/反爬挑战页面', { kind: 'captcha', url });
}
// Login wall: CTA present AND the extracted article body would be tiny. We
// approximate "tiny body" by counting <p> text length in the head slice.
if (DIRECT_LOGIN_CTA_RE.test(head)) {
const paras = head.match(/<p[^>]*>([\s\S]*?)<\/p>/gi) || [];
const textLen = paras.reduce((n, p) => n + p.replace(/<[^>]+>/g, '').trim().length, 0);
if (textLen < 400) {
throw new HumanInterventionError('检测到登录/订阅墙（正文极短且出现登录/订阅提示）', { kind: 'login', url });
}
}
}
module.exports = {
HumanInterventionError,
BROWSER_CHECK_EXPR,
evaluateBrowserBlock,
checkDirectBlock
};
skills/news-harvester/scripts/cdp-client.js 0 → 100644
View file @ d5995b60
/**
* cdp-client.js — Shared Chrome DevTools Protocol primitives.
*
* Previously every CDP caller (fetcher-browser.js) hand-rolled the same
* WebSocket boilerplate: open connection, send a command with an id, attach a
* message listener matching that id, parse the result. That pattern was
* duplicated 4× and the Page.enable call didn't even check the id (resolving
* on the first message of any kind). This module is the single source of truth
* for those operations so fetcher-helper.js (and a future fetcher-browser
* refactor) can build on top.
*
* Lifecycle (browser launch/discovery) stays in ensure-chromium.js — this
* module only handles per-tab CDP commands, assuming a browser is already up.
*/
const http = require('http');
const fs = require('fs');
const path = require('path');
const WebSocket = require('ws');
const SKILL_DIR = path.dirname(__dirname);
const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
// ── Utilities ────────────────────────────────────────────────────────────────
function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
// Monotonic id for CDP commands. Per-tab sessions are sequential, so a simple
// counter is fine and avoids the random-id collisions the old code risked.
let _msgId = 0;
function nextId() { return ++_msgId; }
// ── Port resolution ──────────────────────────────────────────────────────────
// Precedence: env override > config.chromium.cdpPort > default 9222.
// (Mirrors the old private resolveCdpPort in fetcher-browser.js.)
function resolveCdpPort() {
if (process.env.CDP_PORT_OVERRIDE) return parseInt(process.env.CDP_PORT_OVERRIDE, 10);
try {
const cfgPath = fs.existsSync(CONFIG_PATH) ? CONFIG_PATH : TEMPLATE_PATH;
const cfg = JSON.parse(fs.readFileSync(cfgPath, 'utf8'));
if (cfg.chromium && cfg.chromium.cdpPort) return parseInt(cfg.chromium.cdpPort, 10);
} catch (e) { /* fall through to default */ }
return 9222;
}
// ── WebSocket URL discovery ───────────────────────────────────────────────────
// Fetch /json and pick the first `page` tab's webSocketDebuggerUrl. Retries a
// few times: right after ensure-chromium spawns the browser, the port may
// accept connections a beat before /json responds.
function getWsUrl() {
const port = resolveCdpPort();
const MAX_ATTEMPTS = 6;
const STEP_MS = 500;
return new Promise((resolve, reject) => {
let attempt = 0;
const tryOnce = () => {
attempt++;
http.get(`http://localhost:${port}/json`, (res) => {
let data = '';
res.on('data', d => data += d);
res.on('end', () => {
try {
const tabs = JSON.parse(data);
// Prefer a page tab to avoid grabbing an iframe/service_worker target.
const tab = tabs.find(t => t.type === 'page') || tabs[0];
if (!tab || !tab.webSocketDebuggerUrl) {
return reject(new Error(`CDP /json returned no page tab on :${port}`));
}
resolve(tab.webSocketDebuggerUrl);
} catch (e) {
if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
reject(e);
}
});
}).on('error', (e) => {
if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
reject(new Error(`Cannot reach CDP on :${port} after ${MAX_ATTEMPTS} tries (${e.message}). Run: node scripts/ensure-chromium.js`));
});
};
tryOnce();
});
}
// ── Session management ───────────────────────────────────────────────────────
// Open a WebSocket to the page target and resolve once connected.
// Returns { ws, close }. The caller owns the lifecycle (close when done).
function openSession() {
return getWsUrl().then(wsUrl => new Promise((resolve, reject) => {
const ws = new WebSocket(wsUrl);
ws.on('open', () => resolve({ ws, close: () => ws.close() }));
ws.on('error', reject);
}));
}
// ── Command sending ──────────────────────────────────────────────────────────
// Send a CDP command and await its response, matched by id. Rejects on CDP
// error. Concurrent calls are safe — each attaches its own id-matched listener.
function cdpSend(ws, method, params = {}) {
return new Promise((resolve, reject) => {
const id = nextId();
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.id === id) {
ws.removeListener('message', handler);
if (msg.error) reject(new Error(`CDP ${method} error: ${JSON.stringify(msg.error)}`));
else resolve(msg.result);
}
};
ws.on('message', handler);
ws.send(JSON.stringify({ id, method, params }));
});
}
// Evaluate a JS expression in the page and return the value (returnByValue).
// Returns undefined if the expression yielded no value.
async function cdpEval(ws, expression) {
const result = await cdpSend(ws, 'Runtime.evaluate', {
expression,
returnByValue: true
});
return result && result.result && result.result.value;
}
// ── Navigation ───────────────────────────────────────────────────────────────
// Enable Page domain, navigate to url, wait for loadEventFired (racing a 25s
// timeout), then sleep waitMs for JS/React to finish rendering.
// Fixes the old Page.enable bug where ws.once('message') resolved on ANY
// message instead of the enable response — here cdpSend id-checks it.
async function navigateAndWait(ws, url, waitMs = 12000) {
await cdpSend(ws, 'Page.enable');
// Attach the load listener BEFORE sending navigate, since loadEventFired can
// fire before the navigate response arrives.
let loaded = false;
const loadPromise = new Promise(resolve => {
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.method === 'Page.loadEventFired' && !loaded) {
loaded = true;
ws.removeListener('message', handler);
resolve();
}
};
ws.on('message', handler);
});
await cdpSend(ws, 'Page.navigate', { url });
await Promise.race([loadPromise, sleep(25000)]);
await sleep(waitMs);
}
// ── Scrolling + lazy-load collection ─────────────────────────────────────────
// Scroll to the bottom of the page `steps` times, waiting `waitMs` after each
// scroll, and run `collectExpr` (a JS expression returning a JSON string of an
// array) after the initial load and after each scroll. Results are merged,
// deduplicating by `dedupKey` (default: JSON.stringify of the element).
//
// Returns the merged array (of parsed objects). This is what lets SPA list
// pages that lazy-load article cards yield their full set instead of just the
// first screenful.
//
// Two loop modes (mutually exclusive):
// • Fixed steps (default): exactly `steps` scroll iterations, no early exit.
// Used by preview mode and any caller that just wants a fixed lazy-load
// pass. Backward compatible — no `targetCount` ⇒ this mode.
// • Target-count: when `targetCount` is set, scroll until `merged.length >=
// targetCount` OR `maxSteps` iterations OR `maxStaleSteps` consecutive
// scrolls add nothing (stream exhausted). `steps` is ignored in this mode.
// Used by archive harvest so the fetcher can keep loading a lazy stream
// until enough candidates exist to satisfy the requested count *after*
// dedup, instead of giving up at a fixed `scrollSteps` cap.
async function scrollAndCollect(ws, {
steps = 0, waitMs = 1500, collectExpr, dedupKey,
targetCount = null, maxSteps = 10, maxStaleSteps = 3
}) {
const keyFn = dedupKey || (item => JSON.stringify(item));
const seen = new Set();
const merged = [];
const absorb = (raw) => {
if (!raw) return;
let arr;
try { arr = JSON.parse(raw); } catch (e) { return; }
if (!Array.isArray(arr)) return;
for (const item of arr) {
const k = keyFn(item);
if (!seen.has(k)) { seen.add(k); merged.push(item); }
}
};
absorb(await cdpEval(ws, collectExpr));
// Target-count mode: bounded by maxSteps, exits early on enough items or
// on a stale streak (consecutive scrolls adding zero new items ⇒ the lazy
// stream has run dry, so more scrolling won't help).
if (targetCount && merged.length < targetCount) {
let stale = 0;
for (let i = 0; i < maxSteps && merged.length < targetCount; i++) {
const before = merged.length;
await cdpEval(ws, 'window.scrollTo(0, document.body.scrollHeight)');
await sleep(waitMs);
absorb(await cdpEval(ws, collectExpr));
stale = (merged.length === before) ? stale + 1 : 0;
if (stale >= maxStaleSteps) break;
}
return merged;
}
// Fixed-steps mode (default / preview): backward compatible.
for (let i = 0; i < steps; i++) {
await cdpEval(ws, 'window.scrollTo(0, document.body.scrollHeight)');
await sleep(waitMs);
absorb(await cdpEval(ws, collectExpr));
}
return merged;
}
module.exports = {
sleep,
resolveCdpPort,
getWsUrl,
openSession,
cdpSend,
cdpEval,
navigateAndWait,
scrollAndCollect
};
skills/news-harvester/scripts/ensure-chromium.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-harvester/scripts/fetcher-browser.js 0 → 100644
View file @ d5995b60
/**
* fetcher-browser.js
* Fetch article content using Chromium CDP (for JS-heavy sites like Reuters)
* Usage: node fetcher-browser.js <url> [waitMs]
*/
const http = require('http');
const fs = require('fs');
const path = require('path');
const WebSocket = require('ws');
const { evaluateBrowserBlock } = require('./block-check');
const SKILL_DIR = path.dirname(__dirname);
const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
// Resolve the CDP port lazily so config.json changes are picked up per run.
// Precedence: env override > config.chromium.cdpPort > default 9222.
function resolveCdpPort() {
if (process.env.CDP_PORT_OVERRIDE) return parseInt(process.env.CDP_PORT_OVERRIDE, 10);
try {
const cfgPath = fs.existsSync(CONFIG_PATH) ? CONFIG_PATH : TEMPLATE_PATH;
const cfg = JSON.parse(fs.readFileSync(cfgPath, 'utf8'));
if (cfg.chromium && cfg.chromium.cdpPort) return parseInt(cfg.chromium.cdpPort, 10);
} catch (e) { /* fall through to default */ }
return 9222;
}
// Retry the /json endpoint a few times: right after ensure-chromium spawns the
// browser, the port may accept connections a beat before /json responds.
function getWsUrl() {
const port = resolveCdpPort();
const MAX_ATTEMPTS = 6;
const STEP_MS = 500;
return new Promise((resolve, reject) => {
let attempt = 0;
const tryOnce = () => {
attempt++;
http.get(`http://localhost:${port}/json`, (res) => {
let data = '';
res.on('data', d => data += d);
res.on('end', () => {
try {
const tabs = JSON.parse(data);
// 优先找 page 类型的 tab，避免拿到 iframe/service_worker
const tab = tabs.find(t => t.type === 'page') || tabs[0];
if (!tab || !tab.webSocketDebuggerUrl) {
return reject(new Error(`CDP /json returned no page tab on :${port}`));
}
resolve(tab.webSocketDebuggerUrl);
} catch (e) {
if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
reject(e);
}
});
}).on('error', (e) => {
if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
reject(new Error(`Cannot reach CDP on :${port} after ${MAX_ATTEMPTS} tries (${e.message}). Run: node scripts/ensure-chromium.js`));
});
};
tryOnce();
});
}
async function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
async function fetchPage(url, waitMs = 12000) {
const wsUrl = await getWsUrl();
const ws = new WebSocket(wsUrl);
await new Promise((resolve, reject) => { ws.on('open', resolve); ws.on('error', reject); });
// Enable Page events
await new Promise(resolve => {
ws.send(JSON.stringify({ id: 1, method: 'Page.enable' }));
ws.once('message', () => resolve());
});
// Navigate + wait for load
let loaded = false;
const loadPromise = new Promise(resolve => {
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.method === 'Page.loadEventFired' && !loaded) {
loaded = true;
ws.removeListener('message', handler);
resolve();
}
};
ws.on('message', handler);
});
ws.send(JSON.stringify({ id: 2, method: 'Page.navigate', params: { url } }));
await Promise.race([loadPromise, sleep(25000)]);
await sleep(waitMs); // wait for React/JS to fully render
// Human-verification / login-state-loss guard. Throws HumanInterventionError
// (propagates to harvest.js, which exits with a friendly message) before we
// try to extract content from what might be a challenge or login page.
await evaluateBrowserBlock(ws);
// Extract content
const result = await new Promise((resolve) => {
const id = Math.floor(Math.random() * 100000) + 90000;
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.id === id) {
ws.removeListener('message', handler);
resolve(msg.result && msg.result.result && msg.result.result.value);
}
};
ws.on('message', handler);
ws.send(JSON.stringify({ id, method: 'Runtime.evaluate', params: {
expression: `(function(){
var title = document.title.replace(/ \\| Reuters| \\| The Conversation| - Bloomberg$/g,'').trim();
var dateEl = document.querySelector('time[datetime]');
var date = dateEl ? dateEl.getAttribute('datetime') : '';
var authors = Array.from(document.querySelectorAll('[rel="author"], [class*="author"] a'))
.map(a => a.innerText.trim()).filter(Boolean)
.filter((v,i,a) => a.indexOf(v)===i).join(', ');
var imgs = Array.from(document.querySelectorAll('img')).map(img => ({
src: img.src,
alt: img.alt || '',
width: img.naturalWidth || img.width || 0
})).filter(i => i.src && i.src.startsWith('http') && i.width > 200
&& !i.src.includes('logo') && !i.src.includes('icon') && !i.src.includes('avatar')
&& !i.src.includes('profile')).slice(0, 6);
var body = '';
var selectors = [
'[data-testid^="paragraph"]',
'[class*="articleBodyContent"] p',
'[class*="article-body"] p',
'[class*="ArticleBody"] p',
'[class*="story-body"] p',
'article p'
];
for (var sel of selectors) {
var els = document.querySelectorAll(sel);
if (els.length >= 3) {
body = Array.from(els).map(e => e.innerText.trim()).filter(t => t.length > 40).join('\\n\\n');
break;
}
}
if (!body) {
var noise = ['Reporting by','Editing by','Access unmatched','Browse an unrivalled',
'Screen for heightened','Sign up here','Reuters, the news','All quotes delayed','See here for'];
body = Array.from(document.querySelectorAll('p'))
.map(e => e.innerText.trim())
.filter(t => t.length > 60 && !noise.some(n => t.startsWith(n)))
.join('\\n\\n');
}
return JSON.stringify({ title, date, authors, imgs, body: body.substring(0, 15000) });
})()`
}}));
});
ws.close();
return result ? JSON.parse(result) : null;
}
// Extract article links from homepage
async function fetchHomeLinks(url, pattern, waitMs = 15000) {
const wsUrl = await getWsUrl();
const ws = new WebSocket(wsUrl);
await new Promise((resolve, reject) => { ws.on('open', resolve); ws.on('error', reject); });
await new Promise(resolve => {
ws.send(JSON.stringify({ id: 1, method: 'Page.enable' }));
ws.once('message', () => resolve());
});
let loaded = false;
const loadPromise = new Promise(resolve => {
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.method === 'Page.loadEventFired' && !loaded) {
loaded = true;
ws.removeListener('message', handler);
resolve();
}
};
ws.on('message', handler);
});
ws.send(JSON.stringify({ id: 2, method: 'Page.navigate', params: { url } }));
await Promise.race([loadPromise, sleep(25000)]);
await sleep(waitMs);
// Human-verification / login-state-loss guard (see fetchPage above).
await evaluateBrowserBlock(ws);
const result = await new Promise((resolve) => {
const id = Math.floor(Math.random() * 100000) + 80000;
const patternStr = pattern.toString();
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.id === id) {
ws.removeListener('message', handler);
resolve(msg.result && msg.result.result && msg.result.result.value);
}
};
ws.on('message', handler);
ws.send(JSON.stringify({ id, method: 'Runtime.evaluate', params: {
expression: `JSON.stringify(
Array.from(document.querySelectorAll('a[href]'))
.map(a => a.href)
.filter(h => h && new RegExp(${JSON.stringify(pattern)}).test(h))
.filter((h, i, arr) => arr.indexOf(h) === i)
.slice(0, 30)
)`
}}));
});
ws.close();
return result ? JSON.parse(result) : [];
}
module.exports = { fetchPage, fetchHomeLinks };
// CLI usage
if (require.main === module) {
const url = process.argv[2];
const waitMs = parseInt(process.argv[3] || '12000');
if (!url) { console.error('Usage: node fetcher-browser.js <url> [waitMs]'); process.exit(1); }
fetchPage(url, waitMs).then(data => {
process.stdout.write(JSON.stringify(data, null, 2));
}).catch(e => { console.error('Error:', e.message); process.exit(1); });
}
skills/news-harvester/scripts/fetcher-direct.js 0 → 100644
View file @ d5995b60
/**
* fetcher-direct.js
* Fetch article content via direct HTTP (for open sites like The Conversation)
* Usage: node fetcher-direct.js <url>
*/
const https = require('https');
const http = require('http');
const { checkDirectBlock } = require('./block-check');
function fetchHtml(url) {
return new Promise((resolve, reject) => {
const lib = url.startsWith('https') ? https : http;
const req = lib.get(url, {
headers: {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml',
'Accept-Language': 'en-US,en;q=0.9'
}
}, (res) => {
// Handle redirects
if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
return resolve(fetchHtml(res.headers.location));
}
let data = '';
res.on('data', d => data += d);
res.on('end', () => resolve(data));
});
req.on('error', reject);
req.setTimeout(15000, () => { req.destroy(); reject(new Error('Timeout')); });
});
}
// Extract author names from JSON-LD, then <meta name="author"> (one tag per
// author, as on The Conversation), then rel="author" links. Returns '' if none.
function extractAuthors(html) {
// 1. JSON-LD author (Reuters/Guardian embed ld+json blocks)
const ldBlocks = [...html.matchAll(/<script[^>]*type=["']application\/ld\+json["'][^>]*>([\s\S]*?)<\/script>/gi)];
for (const b of ldBlocks) {
try {
const data = JSON.parse(b[1]);
const candidates = [];
const collect = (obj) => {
if (!obj) return;
if (Array.isArray(obj)) { obj.forEach(collect); return; }
candidates.push(obj);
if (obj['@graph']) collect(obj['@graph']);
};
collect(data);
for (const obj of candidates) {
let author = obj.author || (obj.mainEntity && obj.mainEntity.author);
if (author) {
const names = (Array.isArray(author) ? author : [author])
.map(a => typeof a === 'string' ? a : (a && a.name) || '')
.filter(Boolean);
if (names.length) return [...new Set(names)].slice(0, 3).join(', ');
}
}
} catch (e) {}
}
// 2. <meta name="author" content="..."> — multiple tags for co-authors
const metaMatches = [...html.matchAll(/<meta[^>]+name=["']author["'][^>]+content=["']([^"']+)["']/gi)];
if (metaMatches.length) {
const names = [...new Set(metaMatches.map(m => m[1].trim()).filter(Boolean))];
if (names.length) return names.slice(0, 3).join(', ');
}
// 3. rel="author" link text
const relMatch = html.match(/<a[^>]+rel=["']author["'][^>]*>([^<]+)<\/a>/i);
if (relMatch && relMatch[1].trim()) return relMatch[1].trim();
return '';
}
function extractArticle(html, url) {
// Title
const titleMatch = html.match(/<title[^>]*>(.*?)<\/title>/is);
let title = titleMatch ? titleMatch[1].trim() : '';
title = title.replace(/ \| The Conversation| \| Reuters/g, '').trim();
// Date — capture the full ISO 8601 timestamp (incl. fractional seconds and
// timezone offsets like -05:00 / +09:00). The earlier [\d\-T:Z+] class missed
// '.' and the second '-' / '+' in offsets, truncating those values.
const dateMatch = html.match(/["']datePublished["']\s*:\s*["']([^"']+)["']/) ||
html.match(/<time[^>]+datetime=["']([^"']+)["']/i);
const date = dateMatch ? dateMatch[1] : '';
// Authors
const authors = extractAuthors(html);
// Images — skip logos/icons and tracking pixels (e.g. facebook.com/tr), dedupe by src
const imgMatches = [...html.matchAll(/<img[^>]+src=["'](https?:\/\/[^"']+)["'][^>]*>/gi)];
const seenSrc = new Set();
const imgs = imgMatches.map(m => {
const altMatch = m[0].match(/alt=["']([^"']*)["']/i);
return { src: m[1], alt: altMatch ? altMatch[1] : '', width: 500 };
}).filter(i => !i.src.includes('logo') && !i.src.includes('icon')
&& !i.src.includes('facebook.com/tr') && i.src.length > 50
&& !seenSrc.has(i.src) && seenSrc.add(i.src)).slice(0, 6);
// Article body - try structured extraction
let body = '';
// Try JSON-LD articleBody
const jsonLdMatch = html.match(/"articleBody"\s*:\s*"((?:[^"\\]|\\.)*)"/);
if (jsonLdMatch) {
body = jsonLdMatch[1].replace(/\\n/g, '\n').replace(/\\"/g, '"').replace(/\\\//g, '/');
}
if (!body) {
// Extract all paragraphs, clean HTML
const paraMatches = [...html.matchAll(/<p[^>]*>(.*?)<\/p>/gis)];
const noise = ['Sign up', 'Newsletter', 'Subscribe', 'Follow us', 'Read more', 'Click here',
'opens new tab', 'Thomson Reuters', 'All quotes delayed', 'Access unmatched',
'Copy link', 'Bluesky', 'Write an article and join'];
body = paraMatches.map(m => {
let text = m[1].replace(/<[^>]+>/g, '').replace(/&nbsp;/g, ' ')
.replace(/&amp;/g, '&').replace(/&lt;/g, '<').replace(/&gt;/g, '>').trim();
return text;
}).filter(t => {
const words = t.replace(/\s+/g, ' ').trim().split(/\s+/);
// drop share-button blobs / fragments (much whitespace, few real words)
return words.length >= 8 && t.length > 60 && !noise.some(n => t.includes(n));
}).join('\n\n');
}
return { title, date, authors, imgs, body: body.substring(0, 15000) };
}
function extractLinks(html, baseUrl, pattern) {
// Special case: The Conversation uses Next.js __NEXT_DATA__ JSON
const nextDataMatch = html.match(/<script id="__NEXT_DATA__" type="application\/json">([\s\S]*?)<\/script>/);
if (nextDataMatch) {
try {
const nextData = JSON.parse(nextDataMatch[1]);
const blocks = nextData?.props?.pageProps?.blocks || [];
const urls = [];
for (const block of blocks) {
for (const article of (block.blocks || [])) {
const url = article.url || '';
if (url && url.includes('theconversation.com') && /-\d+$/.test(url)) {
if (!urls.includes(url)) urls.push(url);
}
}
}
if (urls.length > 0) return urls.slice(0, 30);
} catch (e) {}
}
// Generic: find links by regex pattern
const regex = new RegExp(`href=["']((?:https?://)?[^"']*${pattern}[^"']*)["']`, 'gi');
const matches = [...html.matchAll(regex)];
const links = matches.map(m => {
let href = m[1];
if (href.startsWith('/')) {
const base = new URL(baseUrl);
href = `${base.protocol}//${base.host}${href}`;
}
return href;
}).filter(h => h.startsWith('http'));
return [...new Set(links)].slice(0, 30);
}
async function fetchPage(url) {
const html = await fetchHtml(url);
checkDirectBlock(html, url);
return extractArticle(html, url);
}
async function fetchHomeLinks(url, pattern) {
const html = await fetchHtml(url);
checkDirectBlock(html, url);
return extractLinks(html, url, pattern);
}
module.exports = { fetchPage, fetchHomeLinks };
// CLI
if (require.main === module) {
const url = process.argv[2];
if (!url) { console.error('Usage: node fetcher-direct.js <url>'); process.exit(1); }
fetchPage(url).then(data => {
process.stdout.write(JSON.stringify(data, null, 2));
}).catch(e => { console.error('Error:', e.message); process.exit(1); });
}
skills/news-harvester/scripts/fetcher-helper.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-harvester/scripts/finalize.js 0 → 100644
View file @ d5995b60
#!/usr/bin/env node
/**
* finalize.js - Patch Chinese summaries into an already-archived day directory.
* Usage: node finalize.js <dayDir>
*
* Reads:
* <dayDir>/.manifest.json (written by harvest.js — maps url → slug → original filename)
* <dayDir>/.summaries.json (written by the agent — Chinese title + 4-section summary)
*
* For each article with a summary:
* 1. Write <dayDir>/<中文标题>.md (the Chinese summary doc)
* 2. Patch the sentinel __SUMMARY_<slug>__ in the original 原文.md (related + footer wikilink)
* 3. Update the registry entry (titleZh / category / tags)
* Finally rewrites index.md with real titles and removes the two temp files.
*
* Missing summary for an article → warn and leave its sentinel in place (non-fatal).
* Missing .summaries.json → error and exit.
*/
const fs = require('fs');
const path = require('path');
const {
generateSummaryDoc,
updateIndex,
sentinelFor,
originalFilename
} = require('./harvest');
function loadJson(p, label) {
if (!fs.existsSync(p)) {
console.error(`✗ ${label} not found: ${p}`);
process.exit(1);
}
try {
return JSON.parse(fs.readFileSync(p, 'utf8'));
} catch (e) {
console.error(`✗ Failed to parse ${label}: ${e.message}`);
process.exit(1);
}
}
function printHelp() {
console.log(`finalize.js — Patch Chinese summaries into an archived day directory.
Usage:
node finalize.js <dayDir>
Arguments:
dayDir Path to the YYYYMMDD directory produced by harvest.js
(e.g. /path/to/vault/The Conversation/20260724)
Prerequisites:
- <dayDir>/.manifest.json (written by harvest.js)
- <dayDir>/.summaries.json (written by the agent — Chinese title + 4-section summary)
What it does:
1. Writes <dayDir>/<中文标题>.md for each article with a summary
2. Replaces sentinel placeholders in the 原文.md files with real titles
3. Updates registry.json and index.md with real titles/categories/tags
4. Removes .manifest.json and .summaries.json
Options:
-h, --help Show this help message
Example:
node finalize.js "/Users/me/obsidian/Harvester/The Conversation/20260724"`);
}
function main() {
const args = process.argv.slice(2);
if (args.includes('-h') || args.includes('--help')) {
printHelp();
process.exit(0);
}
const dayDir = args.find(a => !a.startsWith('-'));
if (!dayDir) {
console.error('Usage: node finalize.js <dayDir>');
console.error('Example: node finalize.js /path/to/vault/The Conversation/20260724');
console.error('Run "node finalize.js --help" for details.');
process.exit(1);
}
if (!fs.existsSync(dayDir) || !fs.statSync(dayDir).isDirectory()) {
console.error(`✗ Day directory does not exist: ${dayDir}`);
process.exit(1);
}
const manifestPath = path.join(dayDir, '.manifest.json');
const summariesPath = path.join(dayDir, '.summaries.json');
const manifest = loadJson(manifestPath, '.manifest.json');
const summaries = loadJson(summariesPath, '.summaries.json');
const sourceName = manifest.sourceName || 'Unknown';
// Index summaries by url for O(1) lookup
const summaryByUrl = new Map();
for (const s of (Array.isArray(summaries) ? summaries : [])) {
if (s && s.url) summaryByUrl.set(s.url, s);
}
console.error(`\n🖌️ Finalizing ${dayDir}`);
console.error(` Articles in manifest: ${manifest.articles.length}`);
console.error(` Summaries provided: ${summaryByUrl.size}`);
// Build the registry path (same convention as harvest.js)
const regPath = path.join(dayDir, 'registry.json');
let registry = { date: path.basename(dayDir), source: manifest.sourceId, articles: [] };
if (fs.existsSync(regPath)) {
try { registry = JSON.parse(fs.readFileSync(regPath, 'utf8')); } catch (e) {}
}
let patched = 0, skipped = 0;
for (const article of manifest.articles) {
const { url, slug, titleEn, originalFilename: origName } = article;
const summary = summaryByUrl.get(url);
if (!summary) {
console.error(` ⚠️ No summary for: ${titleEn} — leaving sentinel in place`);
skipped++;
continue;
}
const titleZh = summary.titleZh;
if (!titleZh || typeof titleZh !== 'string' || !titleZh.trim()) {
console.error(` ⚠️ Empty titleZh for: ${titleEn} — skipping`);
skipped++;
continue;
}
const sentinel = sentinelFor(slug);
// Summary docs live in a `summary/` subdirectory (sibling of `assets/`).
// Wikilinks stay bare-title ([[中文标题]]); Obsidian resolves them by basename.
const summaryDir = path.join(dayDir, 'summary');
fs.mkdirSync(summaryDir, { recursive: true });
const summaryFilePath = path.join(summaryDir, `${titleZh}.md`);
const origFilePath = path.join(dayDir, origName || originalFilename(titleEn, sourceName));
// 1. Write the Chinese summary doc
const summaryDoc = generateSummaryDoc({
titleZh,
titleEn,
date: article.date,
authors: article.authors,
source: url,
category: summary.category || article.category,
tags: summary.tags || article.tags,
summaryZh: summary.summaryZh
}, sourceName);
fs.writeFileSync(summaryFilePath, summaryDoc);
// 2. Patch the original doc: replace the title sentinel, then overwrite the
// frontmatter `tags:` block and `category:` line with the agent-extracted
// values so 原文.md carries the same tags/category as the summary doc.
if (fs.existsSync(origFilePath)) {
const orig = fs.readFileSync(origFilePath, 'utf8');
// Replace all occurrences of the sentinel with the real title.
let patchedOrig = orig.split(sentinel).join(titleZh);
// Overwrite the tags block (tags:\n - x\n - y\n) with agent-extracted tags.
const finalTags = (summary.tags && summary.tags.length) ? summary.tags : article.tags;
const newTagsYaml = (finalTags || []).map(t => ` - ${t}`).join('\n');
patchedOrig = patchedOrig.replace(/(^tags:\n)(?: - .*\n)+/m, `$1${newTagsYaml}\n`);
// Overwrite the category line if the agent supplied one.
if (summary.category) {
patchedOrig = patchedOrig.replace(/^category: .*/m, `category: ${summary.category}`);
}
fs.writeFileSync(origFilePath, patchedOrig);
} else {
console.error(` ⚠️ Original doc missing: ${origName} — summary written but link unpatched`);
}
// 3. Update the registry entry for this url
const regEntry = registry.articles.find(a => a.url === url);
if (regEntry) {
regEntry.titleZh = titleZh;
if (summary.category) regEntry.category = summary.category;
if (summary.tags) regEntry.tags = summary.tags;
}
patched++;
console.error(` ✓ ${titleEn} → ${titleZh}`);
}
// Persist the patched registry
fs.writeFileSync(regPath, JSON.stringify(registry, null, 2));
// 4. Rewrite index.md using real titles (from the now-updated registry).
// Index links resolve to summary docs where a titleZh exists, else sentinel.
const indexArticles = registry.articles.map(a => ({
titleZh: a.titleZh,
titleEn: a.titleEn || '',
date: a.date || '',
category: a.category || '国际',
tags: a.tags || []
}));
updateIndex(dayDir, indexArticles, sourceName);
// 5. Cleanup temp files
try { fs.unlinkSync(manifestPath); } catch (e) {}
try { fs.unlinkSync(summariesPath); } catch (e) {}
console.error(`\n✅ Done: ${patched} finalized, ${skipped} skipped`);
console.error(` Registry + index updated, temp files removed`);
// 6. Auto-push to News Hub if enabled and configured. Push failures do NOT
// abort finalize (the archive is already complete and re-pushable), so
// a backend hiccup never leaves the local archive in a bad state.
maybeAutoPush(dayDir);
}
// Chain into push.js when config.push.enabled && autoPushAfterFinalize.
// Runs push.js as a child process so a push crash can't take down finalize.
function maybeAutoPush(dayDir) {
const cfgPath = path.join(__dirname, '..', 'config.json');
let cfg;
try { cfg = JSON.parse(fs.readFileSync(cfgPath, 'utf8')); }
catch (e) { return; } // no config → skip silently
const p = cfg.push;
if (!p || !p.enabled || !p.autoPushAfterFinalize) return;
if (!p.endpoint || !p.appId || !p.appKeyId || !p.appSecret) {
console.error(`\n⚠️ Push enabled but config incomplete (endpoint/appId/appKeyId/appSecret) — skipping.`);
console.error(` Run manually once configured: node scripts/push.js "${dayDir}"`);
return;
}
console.error(`\n📤 Auto-pushing to News Hub (config.push.autoPushAfterFinalize = true)`);
const { spawnSync } = require('child_process');
const r = spawnSync(process.execPath, [path.join(__dirname, 'push.js'), dayDir], {
stdio: ['ignore', 'inherit', 'inherit'], // stdout/stderr → this process
});
if (r.status !== 0) {
console.error(`\n⚠️ Push exited with code ${r.status} — local archive is unaffected.`);
console.error(` Fix the issue and re-run: node scripts/push.js "${dayDir}"`);
}
}
if (require.main === module) {
main();
}
skills/news-harvester/scripts/harvest.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-harvester/scripts/push.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-harvester/scripts/source-manage.js 0 → 100644
View file @ d5995b60
#!/usr/bin/env node
/**
* source-manage.js - Manage installed news sources
*
* This is the harvester-side source manager. It does NOT add/export/helper
* — sources are added exclusively by installing .nhsource.json bundles
* produced by the news-source-analyzer skill. This keeps the harvester
* focused on the collection pipeline.
*
* Usage:
* node source-manage.js list
* node source-manage.js edit <id> --field value
* node source-manage.js enable <id>
* node source-manage.js disable <id>
* node source-manage.js install <pkg.nhsource.json> [--force] # install a shared bundle
*/
const fs = require('fs');
const path = require('path');
const SKILL_DIR = path.dirname(__dirname);
const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
const HELPERS_DIR = path.join(SKILL_DIR, 'helpers');
function loadConfig() {
if (!fs.existsSync(CONFIG_PATH)) {
const template = JSON.parse(fs.readFileSync(TEMPLATE_PATH, 'utf8'));
fs.writeFileSync(CONFIG_PATH, JSON.stringify(template, null, 2));
}
return JSON.parse(fs.readFileSync(CONFIG_PATH, 'utf8'));
}
function saveConfig(config) {
fs.writeFileSync(CONFIG_PATH, JSON.stringify(config, null, 2));
}
function parseArgs(argv) {
const args = {};
for (let i = 0; i < argv.length; i++) {
if (argv[i].startsWith('--')) {
args[argv[i].slice(2)] = argv[i+1] || true;
i++;
} else {
args['_' + (args._count || 0)] = argv[i];
args._count = (args._count || 0) + 1;
}
}
return args;
}
const command = process.argv[2];
const args = parseArgs(process.argv.slice(3));
const config = loadConfig();
switch (command) {
case 'list': {
console.log('\n📋 Configured Sources\n');
console.log('ID'.padEnd(20) + 'NAME'.padEnd(20) + 'METHOD'.padEnd(10) + 'RECIPE'.padEnd(10) + 'STATUS');
console.log('─'.repeat(70));
for (const s of config.sources) {
const status = s.enabled ? '✅ enabled' : '❌ disabled';
const hasRecipe = fs.existsSync(path.join(HELPERS_DIR, `${s.id}.md`));
const recipe = hasRecipe ? '✅' : '—';
console.log(
s.id.padEnd(20) +
s.name.padEnd(20) +
s.method.padEnd(10) +
recipe.padEnd(10) +
status
);
}
console.log('\n RECIPE ✅ = helpers/<id>.md exists (spec-driven fetcher); — = legacy fetcher\n');
break;
}
case 'edit': {
const id = args._0;
const source = config.sources.find(s => s.id === id);
if (!source) { console.error(`Source "${id}" not found`); process.exit(1); }
const editableFields = ['name', 'homepageUrl', 'method', 'articleUrlPattern', 'vaultFolder', 'enabled'];
let changed = false;
for (const field of editableFields) {
if (args[field] !== undefined) {
const oldVal = source[field];
source[field] = args[field] === 'true' ? true : args[field] === 'false' ? false : args[field];
console.log(` ${field}: ${oldVal} → ${source[field]}`);
changed = true;
}
}
if (!changed) {
console.log(`\nCurrent config for "${id}":`);
console.log(JSON.stringify(source, null, 2));
console.log('\nUsage: node source-manage.js edit <id> --field value');
} else {
saveConfig(config);
console.log(`\n✅ Updated source "${id}"`);
}
break;
}
case 'enable':
case 'disable': {
const id = args._0;
const source = config.sources.find(s => s.id === id);
if (!source) { console.error(`Source "${id}" not found`); process.exit(1); }
source.enabled = command === 'enable';
saveConfig(config);
console.log(`✅ Source "${id}" ${command}d`);
break;
}
case 'install': {
// Unpack a .nhsource.json: write recipe/fixtures into helpers/, add the
// source to config.json. Refuses on id conflict unless --force (matching
// the "helper" command's refuse-with-force convention).
const pkgPath = args._0;
if (!pkgPath) { console.error('Usage: node source-manage.js install <pkg.nhsource.json> [--force]'); process.exit(1); }
const absPkg = path.resolve(pkgPath);
if (!fs.existsSync(absPkg)) { console.error(`Bundle not found: ${absPkg}`); process.exit(1); }
let bundle;
try { bundle = JSON.parse(fs.readFileSync(absPkg, 'utf8')); }
catch (e) { console.error(`Failed to parse bundle: ${e.message}`); process.exit(1); }
if (bundle.format !== 'news-harvester-source') {
console.error(`Not a news-harvester source bundle (format="${bundle.format}").`);
process.exit(1);
}
const src = bundle.source || {};
const id = src.id;
if (!id) { console.error('Bundle is missing source.id'); process.exit(1); }
const existing = config.sources.find(s => s.id === id);
if (existing && args.force === undefined) {
console.error(`Source "${id}" already exists in config.json. Use --force to overwrite (config entry + helper files).`);
process.exit(1);
}
// Write helper files (refuse to overwrite an existing file unless --force,
// same convention as the id-conflict check above).
if (!fs.existsSync(HELPERS_DIR)) fs.mkdirSync(HELPERS_DIR, { recursive: true });
const written = [];
const files = bundle.files || {};
for (const [relPath, content] of Object.entries(files)) {
// Guard against path traversal: only allow writing into helpers/.
const dest = path.join(HELPERS_DIR, path.basename(relPath));
if (dest.indexOf(HELPERS_DIR) !== 0) {
console.error(` ⚠️ Skipping unsafe path in bundle: ${relPath}`);
continue;
}
if (fs.existsSync(dest) && args.force === undefined && !existing) {
console.error(`helpers/${path.basename(dest)} already exists. Use --force to overwrite.`);
process.exit(1);
}
fs.writeFileSync(dest, content);
written.push(`helpers/${path.basename(dest)}`);
}
// Upsert the config entry. recipe sources don't carry articleUrlPattern
// (it's legacy-direct-only and ignored when a recipe exists).
const entry = {
id,
name: src.name || id,
homepageUrl: src.homepageUrl || '',
method: src.method || 'browser',
vaultFolder: src.vaultFolder || src.name || id,
enabled: true
};
if (existing) {
Object.assign(existing, entry);
} else {
config.sources.push(entry);
}
saveConfig(config);
console.log(`\n✅ Installed source "${entry.name}" (id: ${id})`);
console.log(` Method: ${entry.method} | Vault folder: ${entry.vaultFolder}`);
if (written.length) console.log(` Files: ${written.join(', ')}`);
// Warn if vaultPath is still the placeholder — preview works, archive won't.
if (config.vaultPath === '/path/to/your/obsidian/vault' || !config.vaultPath) {
console.error(`\n ⚠️ config.vaultPath is not set. Preview works, but archiving needs a real vault path —`);
console.error(` set it in config.json before running a full harvest.`);
}
console.log(`\n Next:`);
if (written.includes(`helpers/${id}.fixtures.json`)) {
console.log(` node scripts/recipe-test.js ${id} # verify the recipe still matches the live site`);
}
console.log(` node scripts/harvest.js ${id} 3 --preview # fetch latest articles (read-only)\n`);
break;
}
default:
console.log(`
Usage:
node source-manage.js list
node source-manage.js edit <id> --name "New Name" --url https://... --method browser
node source-manage.js enable <id>
node source-manage.js disable <id>
node source-manage.js install <pkg.nhsource.json> [--force] # install a shared bundle (recipe + config entry)
Sources are added by installing .nhsource.json bundles (produced by the
news-source-analyzer skill). There is no "add" or "export" command —
use the analyzer to research and pack new sources.
Helper recipes: each installed source has a helpers/<id>.md file (written
by install). "list" shows which sources have recipes (✅) vs legacy (—).
`);
}
skills/news-harvester/scripts/summary-add.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/SKILL.md 0 → 100644
View file @ d5995b60
---
name: news-source-analyzer
description: Analyze news websites and produce installable source bundles (.nhsource.json) for the news-harvester. Use this skill whenever the user wants to research a new news source, inspect a website's page structure, author or debug a fetch recipe, verify a recipe works, or pack a source bundle. Stateless — no vault dependency, no source registry. Triggers on phrases like "分析数据源", "研究新闻网站", "inspect source", "create recipe", "make bundle", "打包数据源", "添加数据源", "new source", "recipe", "selector", "listSelector", "urlPattern".
---
# News Source Analyzer Skill
Analyze news websites, author fetch recipes, and produce installable `.nhsource.json` bundles. This skill is **stateless** — it does not manage the harvester's vault, push config, or source registry. Its sole output is a bundle file that the harvester installs in one command.
> **Skill root** = the directory containing this `SKILL.md`. Prefix every command with `cd "<skill-root>" &&`, or scripts fail with `MODULE_NOT_FOUND`.
## Quick Reference
| Task | Command |
|------|---------|
| Inspect page structure | `node scripts/inspect-source.js <url> --scroll 3` |
| Inspect + write recipe | `node scripts/inspect-source.js <url> --scroll 3 --write <id> [--name "名称"]` |
| Preview articles (read-only) | `node scripts/preview.js --url <url> --recipe <path> [--count 3] [--full]` |
| Run recipe regression tests | `node scripts/recipe-test.js <id> --url <homepageUrl>` |
| Pack a bundle | `node scripts/pack.js --id <id> --name <name> --url <url> --method <browser\|direct> --folder <folder> --recipe <path> [--fixtures <path>] [--out <path>]` |
| Ensure Chromium | `node scripts/ensure-chromium.js` (or `--check`, `--list`, `--login`) |
> **All commands must be prefixed with `cd "<skill-root>" &&`**. Scripts live in `scripts/` — always reference them as `node scripts/inspect-source.js`, never `node inspect-source.js`.
---
## First-Run Setup
**Pre-flight gate** — run this at the start of a task. If it prints `OK`, skip setup:
```bash
cd "<skill-root>" && node -e "const c=require('./config.json'); console.log(c.chromium?'OK':'UNCONFIGURED')" 2>/dev/null || echo "UNCONFIGURED (no config.json)"
```
### Onboarding (only if UNCONFIGURED)
This skill only needs the `chromium` block in `config.json` (no vault, no push, no sources). The config is auto-created from `references/config-template.json` on first script run. To pre-configure:
1. Check what browsers are available:
```bash
node scripts/ensure-chromium.js --list
```
2. If a source needs login (subscription/paywall), launch a windowed browser to log in:
```bash
node scripts/ensure-chromium.js --login --url <homepage>
```
Log in inside the window, then **quit the browser fully** (Cmd+Q / Ctrl+Q) to persist cookies.
3. If `--login` used the default profile, persist it in `config.json`:
```json
{ "chromium": { "userDataDir": "/Users/<you>/.news-harvester/analyzer-profile" } }
```
4. Smoke test:
```bash
node scripts/inspect-source.js https://theconversation.com/us --scroll 1
```
If you see container groups with article links, setup is complete.
---
## Configuration
| Field | Required? | Description |
|-------|-----------|-------------|
| `chromium.cdpPort` | Optional | CDP debug port (default `9222`) |
| `chromium.headless` | Optional | Launch headless (`true`, default) or windowed (`false`) |
| `chromium.executablePath` | Optional | Full path to browser binary. `null` = auto-discover |
| `chromium.userDataDir` | Optional | Browser profile dir for login state. Set to a fixed profile you've logged into once |
---
## Workflow
The 4-step closed loop: **inspect → preview → test → pack**.
### Step 1. Inspect + write recipe
```bash
node scripts/inspect-source.js <homepageUrl> --scroll 3 --write <id> [--name "显示名称"]
```
- Navigates to the URL, scrolls to trigger lazy-loading, collects every `<a[href]>`, and groups them by container signature.
- The biggest non-nav group becomes `listSelector`; `urlPattern` is inferred from sample URLs.
- Writes `helpers/<id>.md` directly (refuses if it exists unless `--force`).
- Without `--write`: prints a paste-ready skeleton to stdout.
### Step 2. Preview (read-only verification)
```bash
node scripts/preview.js --url <homepageUrl> --recipe helpers/<id>.md --count 3
```
- Fetches candidate links using the recipe, then fetches each article page.
- Prints a compact JSON manifest (titles + metadata + bodyPreview) to stdout.
- No files written. Use `--full` to include complete article bodies.
**Convergence check**: preview passes when fetched links are real articles (not nav/section pages) and `charCount > 500`. Diagnose failures:
| Symptom | Fix |
|---------|-----|
| Too few candidates | `listSelector` too narrow. Use composite selectors (comma-separated). |
| Candidates are nav/section pages | Tighten `urlPattern`, or set `validateArticle: true`. |
| Empty body | Fix `bodySelectors`. |
| 0 candidates, all nav | `listSelector: null` is wrong — scope to card containers. |
Re-run preview after each recipe edit. Deep troubleshooting: `references/recipe-guide.md` → *Troubleshooting recipes*.
### Step 3. Lock with regression tests
Create `helpers/<id>.fixtures.json`:
```json
{
"articles": ["https://.../real-article-1", "https://.../real-article-2"],
"sections": ["https://.../section-page-1"],
"minLinks": 10
}
```
Then:
```bash
node scripts/recipe-test.js <id> --url <homepageUrl>
```
Asserts: every article URL fetches `isArticle=true` with `bodyChars ≥ bodyMinChars`; every section URL fetches `isArticle=false`; homepage yields ≥ `minLinks` and zero `excludeUrlPattern` matches.
### Step 4. Pack into a bundle
```bash
node scripts/pack.js --id <id> --name <name> --url <homepageUrl> \
--method browser --folder <vaultFolder> \
--recipe helpers/<id>.md [--fixtures helpers/<id>.fixtures.json] \
[--out bundles/<id>.nhsource.json]
```
Produces a self-contained `.nhsource.json` bundle. The harvester installs it with:
```bash
node scripts/source-manage.js install <bundle.nhsource.json>
```
---
## Category Detection
Preview output includes category classification based on URL path and title keywords:
| Keywords | Category |
|----------|---------|
| asia-pacific, pakistan, afghanistan, 巴, 阿富汗 | 地缘政治 |
| ai, anthropic, openai, tech, 科技, AI | 科技/AI |
| china, 中国, 中美 | 中美关系 |
| business, finance, housing, economy, 经济, 财经 | 财经 |
| sustainability, feminist, society, 社会 | 社会 |
| ukraine, russia, middle-east, war | 国际冲突 |
| default | 国际 |
Category emoji: 地缘政治→🌏, 科技/AI→🤖, 中美关系→🇨🇳, 财经→💰, 社会→🧑‍🤝‍🧑, 国际冲突→⚔️, 国际→📰
---
## Human-intervention exits (CAPTCHA / login loss)
`preview.js` detects CAPTCHA/anti-bot challenges and login-state loss (exit code `70`). When this happens:
- **Do not retry** the same command — the block is deterministic until the user intervenes.
- Relay the printed message to the user.
- Suggest: `node scripts/ensure-chromium.js --login --url <homepage>` to pass the challenge or re-login in a windowed browser.
---
## References
| File | When to read |
|------|--------------|
| `references/recipe-guide.md` | Recipe frontmatter schema, authoring workflow, troubleshooting (composite listSelector, `validateArticle`), fixtures format. |
| `references/helper-example.md` | Annotated recipe template with every field commented. Start here when authoring a recipe by hand. |
skills/news-source-analyzer/package-lock.json 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/package.json 0 → 100644
View file @ d5995b60
{
"name": "news-source-analyzer",
"version": "1.0.0",
"description": "Stateless news source analyzer — inspect, recipe, pack",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"type": "commonjs",
"dependencies": {
"js-yaml": "^5.2.2",
"ws": "^8.19.0"
}
}
skills/news-source-analyzer/references/config-template.json 0 → 100644
View file @ d5995b60
{
"chromium": {
"cdpPort": 9222,
"headless": true,
"executablePath": null,
"userDataDir": null
}
}
skills/news-source-analyzer/references/helper-example.md 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/references/recipe-guide.md 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/scripts/block-check.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/scripts/cdp-client.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/scripts/ensure-chromium.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/scripts/fetcher-helper.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/scripts/inspect-source.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/scripts/pack.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/scripts/preview.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
skills/news-source-analyzer/scripts/recipe-test.js 0 → 100644
View file @ d5995b60
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment