Two independent skills for the news harvesting pipeline:
## news-source-analyzer (无状态分析器)
Analyze news websites, author fetch recipes, and produce installable source bundles (`.nhsource.json`). Stateless — no vault dependency, no source registry. Works on any machine with Chromium installed.
## news-harvester (采集器)
Harvest and archive news articles from configured sources to Obsidian vault with Chinese summaries + original text. Sources are installed from bundles produced by the analyzer. Manages vault path, push config, and source registry.
The `.nhsource.json` bundle is the only interface between the two skills. No shared runtime state, no shared config, no shared browser profile.
## Shared scripts
Four CDP/Chromium infrastructure scripts (`ensure-chromium.js`, `cdp-client.js`, `fetcher-helper.js`, `block-check.js`) are duplicated in both skills as standalone copies. These are stable low-level CDP wrappers with low change frequency. When modifying one, sync the copy in the other skill.
description:Harvest and archive news articles from configured sources to Obsidian vault with Chinese summaries + original text. Sources are installed from .nhsource.json bundles produced by the news-source-analyzer skill. Use this skill whenever the user wants to fetch latest news, run a harvest job, archive articles, push to News Hub, install a source bundle, list/edit/enable/disable sources, or check archived content. Triggers on phrases like "抓取新闻", "harvest news", "fetch articles", "run harvest", "install bundle", "push to news hub", "采集新闻", "归档", "推送", "install source", "list sources", "news sources".
---
# News Harvester Skill
Fetch latest articles from configured news sources, archive them to Obsidian vault with Chinese summaries + original text. Sources are installed exclusively from `.nhsource.json` bundles (produced by the **news-source-analyzer** skill) — there is no source-authoring capability in this skill.
> **Skill root** = the directory containing this `SKILL.md`. Prefix every command with `cd "<skill-root>" &&`, or scripts fail with `MODULE_NOT_FOUND`.
> **All commands must be prefixed with `cd "<skill-root>" &&`**. Scripts live in `scripts/` — always reference them as `node scripts/harvest.js`, never `node harvest.js`.
Config file: `config.json` (auto-generated from `references/config-template.json` on first run).
---
## First-Run Setup
**Pre-flight gate** — run this at the start of a task. If it prints `OK`, skip setup:
```bash
cd"<skill-root>"&& node -e"const c=require('./config.json'); console.log(c.vaultPath==='/path/to/your/obsidian/vault'?'UNCONFIGURED':'OK')" 2>/dev/null ||echo"UNCONFIGURED (no config.json)"
```
### Onboarding (only if UNCONFIGURED)
**1. Set the Obsidian vault path.** Ask the user for the absolute path to their Obsidian vault root. Verify it exists and is writable:
Log in, **quit the browser fully** (Cmd+Q), then persist `chromium.userDataDir` in `config.json`. After that, `harvest.js` auto-launches Chromium headless with this profile — login state intact.
Articles are archived to `<vaultPath>/<vaultFolder>/YYYYMMDD/`. Within a day directory: 原文.md files + `registry.json` + `index.md` at the root, images in `assets/`, and Chinese summary docs in `summary/`.
---
## Installing sources from bundles
Sources are added by installing `.nhsource.json` bundles (produced by the **news-source-analyzer** skill):
This writes the recipe + fixtures into `helpers/`, registers the source in `config.json` (enabled), and the source is immediately usable. See `references/source-management.md` for bundle format and install behavior.
---
## Workflow
Two flows: **archive** (fetch + save to Obsidian) or **preview** (read-only).
> **Stay lean.** If the user already named a source (e.g. "reuters"), use it directly — do **not** run `source-manage.js list` first. The archive flow is exactly **3 script calls**: `harvest.js` → write `.summaries.json` → `finalize.js`.
### How to work (read once)
- **Scripts do all deterministic I/O** (fetch, parse, download images, write 原文.md / registry / index). The agent does only the intelligent part — Chinese translation + 4-section summaries — and hands them back as a JSON file for `finalize.js` to stitch in. The full article body is written to disk by the script and **never enters the agent's context**.
- **`count` is the target number of NEW articles**, not raw links. If a run returns fewer than `count`, the stderr prints `⚠️ … stream may be exhausted` — that's a genuine source limit, not a bug. Don't re-run for the same source/day just to "get more".
- **Never `Read` the 原文.md files** to write summaries — the `bodyPreview` in the manifest is sufficient.
- **Extract `tags` from the summary/body** (≤3 semantic keywords), not from the category/source.
- **One `finalize.js` call** after writing `.summaries.json`.
### Flow A — Archive to Obsidian (3 steps)
#### Step 1. Run harvest
```bash
node scripts/harvest.js <source_id><count>
```
Fetches links, deduplicates, fetches each article, downloads images, and writes to disk:
- `<英文标题> - <SourceName>原文.md`
- `registry.json` (dedup record, written incrementally)
- `index.md` (daily index, sentinel titles until finalize)
- `.manifest.json` (temp — consumed by finalize)
Prints a **compact manifest** to stdout (bodyPreview ~4000 chars per article). Read it.
#### Step 2. Generate Chinese summaries → write `.summaries.json`
2. `node scripts/summary-add.js "<dayDir>"` — merges batch into `.summaries.json`.
3. If output says `missing: 0` → go to Step 3. Otherwise write the next batch.
> **Interrupted run?** Do not re-run `harvest.js`. Run `node scripts/summary-add.js "<dayDir>" --status` to see which articles still need summaries, then resume batching.
#### Step 3. Run finalize
```bash
node scripts/finalize.js "<dayDir>"
```
Writes each `<中文标题>.md`, patches sentinel links in 原文.md files, updates `registry.json` + `index.md`, and removes `.manifest.json` + `.summaries.json`.
> **Auto-push (optional).** If `config.push.enabled` is `true`, finalize auto-chains into `push.js`. A push failure never aborts finalize — re-push with `node scripts/push.js "<dayDir>"` once the backend is reachable.
`harvest.js` detects two situations where it cannot proceed autonomously and exits immediately (exit code `70`):
| Situation | What to do |
|-----------|------------|
| **人机验证 / 反爬挑战** (CAPTCHA, Cloudflare) | Wait and retry; optionally `node scripts/ensure-chromium.js --login --url <homepage>` to pass the challenge in a windowed browser. |
| **登录状态失效** (login wall, expired cookies) | Run `node scripts/ensure-chromium.js --login --url <homepage>`, log in, quit browser (Cmd+Q), then re-run harvest. |
When `harvest.js` prints the `🚫 采集中止 — 需要人工介入` block, **do not retry** — relay the printed 建议操作 to the user and stop. Articles already archived before the block are kept.
"helpers/nikkei-tech.fixtures.json":"{\n\"_comment\": \"Regression fixtures for the nikkei-tech recipe. Article URLs must fetch with isArticle=true (og:type=article + body ≥500 chars); section URLs must fetch with isArticle=false (og:type=website). URLs verified 2026-07-29 against https://asia.nikkei.com/business/tech. Update when Nikkei archives these — pick current articles from the same path depths (2-seg /economy/<slug>, 3-seg /business/tech/semiconductors/<slug>).\",\n\"articles\": [\n\"https://asia.nikkei.com/business/tech/semiconductors/sk-hynix-q2-profit-surges-but-misses-market-forecast-shares-slide\",\n\"https://asia.nikkei.com/business/tech/semiconductors/ai-chip-boom-shifts-bottleneck-to-advanced-packaging-says-at-s-ceo\",\n\"https://asia.nikkei.com/spotlight/society/for-philippine-women-in-taiwan-s-tech-factories-motherhood-is-impossible\",\n\"https://asia.nikkei.com/spotlight/trump-administration/us-bans-new-chinese-humanoid-robots-to-protect-ai-buildout\",\n\"https://asia.nikkei.com/economy/natural-disasters/major-japan-quake-traps-people-inside-kumamoto-shopping-mall-factory\",\n\"https://asia.nikkei.com/business/markets/south-korea-s-kospi-plunges-11-on-china-chip-competition-fears\",\n\"https://asia.nikkei.com/business/companies/panasonic-to-end-tv-production-in-malaysia-cutting-jobs\"\n ],\n\"sections\": [\n\"https://asia.nikkei.com/business/tech\",\n\"https://asia.nikkei.com/business/tech/semiconductors\",\n\"https://asia.nikkei.com/business/markets\",\n\"https://asia.nikkei.com/economy/natural-disasters\",\n\"https://asia.nikkei.com/spotlight/society\"\n ],\n\"minLinks\": 10\n}\n"
These docs are **generated by the scripts**, not hand-written by the agent —
shown here for debugging or when reading archived files. The agent only writes
`.summaries.json`; `finalize.js` produces the Chinese summary docs and patches
the originals.
## Original Doc (written by `harvest.js`)
Filename: `<英文标题> - <SourceName>原文.md`
Between Step 1 (harvest) and Step 3 (finalize), the Chinese-title links in this file are a **sentinel placeholder**`__SUMMARY_<slug>__` — in both the frontmatter `related` field and the footer wikilink. `finalize.js` replaces them with the real `titleZh`, and overwrites the `tags` block + `category` line with the agent-extracted values. Image captions: only rendered when `alt` text exists (empty alt → bare embed, no `**`).
tags: # agent-extracted (≤3); placeholder [source] until finalize
-<tag1>
-<tag2>
category: <分类> # agent-supplied; overrides the script's keyword guess
source: "<url>"
publisher: <source>
authors: "<authors>"
type: 原文
related: "[[<中文标题>]]" # sentinel __SUMMARY_<slug>__ until finalize
saved: YYYY-MM-DD
---
# <英文标题>
> [!info] Source
> **Published:** <Shanghai time, e.g. 2026-07-31 05:47>
> **URL:** <url>
> **中文摘要：** [[<中文标题>]]
---
## 配图
![[assets/<slug>-1.jpg]]
*<alt text — omitted if empty>*
---
## 正文
<fullarticlebody>
---
[[<中文标题>]]
```
## Chinese Summary Doc (written by `finalize.js`)
File: `<dayDir>/summary/<中文标题>.md` (concise title, ≤30 chars). The `summary/` subdirectory is a sibling of `assets/`. Wikilinks from 原文.md and index.md use the bare title `[[<中文标题>]]` — Obsidian resolves them by basename.
Sources are added by installing `.nhsource.json` bundles produced by the **news-source-analyzer** skill. There is no `add` or `export` command in the harvester — use the analyzer to research and pack new sources.
| `source` | `{ id, name, homepageUrl, method, vaultFolder }` — the config entry, minus `vaultPath`, `enabled`, and `articleUrlPattern` |
| `files` | Map of relative path → file content. Includes `helpers/<id>.md` (the recipe) and `helpers/<id>.fixtures.json` (if present). |
### Install behavior
- Writes every file in `files` into `helpers/` (path traversal guarded — only basenames under `helpers/` are written).
- Registers the source in `config.json` as enabled. If `<id>` already exists, the entry is overwritten in place; otherwise it's appended.
-**Conflict handling**: refuses if `<id>` is already in `config.json` or a helper file of the same name exists, unless `--force` is passed. `--force` overwrites the config entry and all helper files.
-**vaultPath caveat**: install succeeds even when `config.vaultPath` is still the placeholder (the source is registered, `--preview` works), but archiving requires a real vault path — install prints a warning in that case.
- After install, the source is immediately usable: `harvest.js <id> 3 --preview`.
Bundles carry no signature/checksum (personal-sharing scope). To vet a bundle before installing, open it in an editor and check the `source` block + the recipe frontmatter in `files["helpers/<id>.md"]`.
### Managing recipes
Recipes live in `helpers/` (installed from bundles, gitignored). Each installed source has a `helpers/<id>.md` file. `source-manage.js list` shows which sources have recipes (✅) vs legacy (—).
If you need to research or create a new recipe, use the **news-source-analyzer** skill — it has `inspect-source.js`, `preview.js`, `recipe-test.js`, and `pack.js` for the full recipe authoring workflow.
varchallengeTitle=/^(just a moment|attention required|verify|checking your browser|are you human)/i.test(title)
||/access denied|请输入验证码|安全验证/i.test(title);
// Challenge body copy.
varchallengeText=/please verify (that )?you are human|verify you are human|unusual traffic from your computer|checking your browser before accessing|enable javascript and cookies to continue|our system has detected unusual traffic|请完成安全验证|进行人机验证/i.test(bodyText);
// Browser (CDP) entry point. Runs the probe in the page via cdpEval; throws
// HumanInterventionError if the page is a challenge/login wall.
asyncfunctionevaluateBrowserBlock(ws){
const{cdpEval}=require('./cdp-client');
constraw=awaitcdpEval(ws,BROWSER_CHECK_EXPR);
evaluateBlockResult(raw);
}
// ── Direct (HTTP) entry point ───────────────────────────────────────────────
// String scan of raw HTML for challenge/login-wall signals. No DOM, so the
// signals are a subset of the browser probe — enough to catch a Cloudflare
// interstitial or a login redirect on an open site.
constDIRECT_CHALLENGE_RE=/please verify (that )?you are human|verify you are human|unusual traffic from your computer|checking your browser before accessing|enable javascript and cookies to continue|our system has detected unusual traffic|请完成安全验证|进行人机验证/i;
constDIRECT_CHALLENGE_TITLE_RE=/<title[^>]*>(just a moment|attention required|verify|checking your browser|are you human|access denied|请输入验证码|安全验证)/i;
constDIRECT_LOGIN_CTA_RE=/sign in to (continue|read)|log in to (continue|read)|subscribe to (continue|read|keep)|以继续阅读|登录后查看|订阅后查看|登录以继续|登录后可继续阅读/i;
console.error(` ⚠️ Only ${newLinks.length} new candidates of ${count} requested — stream may be exhausted (fewer fresh articles than asked). Re-running won't help; wait for new articles or broaden the source.`);
description:Analyze news websites and produce installable source bundles (.nhsource.json) for the news-harvester. Use this skill whenever the user wants to research a new news source, inspect a website's page structure, author or debug a fetch recipe, verify a recipe works, or pack a source bundle. Stateless — no vault dependency, no source registry. Triggers on phrases like "分析数据源", "研究新闻网站", "inspect source", "create recipe", "make bundle", "打包数据源", "添加数据源", "new source", "recipe", "selector", "listSelector", "urlPattern".
---
# News Source Analyzer Skill
Analyze news websites, author fetch recipes, and produce installable `.nhsource.json` bundles. This skill is **stateless** — it does not manage the harvester's vault, push config, or source registry. Its sole output is a bundle file that the harvester installs in one command.
> **Skill root** = the directory containing this `SKILL.md`. Prefix every command with `cd "<skill-root>" &&`, or scripts fail with `MODULE_NOT_FOUND`.
> **All commands must be prefixed with `cd "<skill-root>" &&`**. Scripts live in `scripts/` — always reference them as `node scripts/inspect-source.js`, never `node inspect-source.js`.
---
## First-Run Setup
**Pre-flight gate** — run this at the start of a task. If it prints `OK`, skip setup:
```bash
cd"<skill-root>"&& node -e"const c=require('./config.json'); console.log(c.chromium?'OK':'UNCONFIGURED')" 2>/dev/null ||echo"UNCONFIGURED (no config.json)"
```
### Onboarding (only if UNCONFIGURED)
This skill only needs the `chromium` block in `config.json` (no vault, no push, no sources). The config is auto-created from `references/config-template.json` on first script run. To pre-configure:
1. Check what browsers are available:
```bash
node scripts/ensure-chromium.js --list
```
2. If a source needs login (subscription/paywall), launch a windowed browser to log in:
Asserts: every article URL fetches `isArticle=true` with `bodyChars ≥ bodyMinChars`; every section URL fetches `isArticle=false`; homepage yields ≥ `minLinks` and zero `excludeUrlPattern` matches.
A recipe (`helpers/<id>.md`) is a YAML frontmatter file that declares the selectors, scroll behavior, and URL filters for a news source. When the harvester's `harvest.js` finds a recipe, it uses `fetcher-helper.js` (which reads the recipe's spec) instead of the legacy `fetcher-browser.js`/`fetcher-direct.js`.
> **Precedence**: when a recipe exists, its `urlPattern`/selectors are the source of truth and the config's `articleUrlPattern` is **ignored**. A recipe also forces Chromium use regardless of `method` (since `fetcher-helper.js` drives the browser).
## Authoring workflow
The canonical 4-step flow for analyzing a new source:
1.**Inspect + auto-write the recipe** — `inspect --write` measures `listSelector` (biggest container group) and infers `urlPattern` from real URLs, then writes `helpers/<id>.md` directly:
Confirm the links are real articles (not nav/section pages) and the body char counts look right. Iterate on the recipe until clean — see *Troubleshooting recipes* below.
3. **Lock with regression tests**(needs `helpers/<id>.fixtures.json`):
| `scrollSteps` | `0` | Bottom-scroll iterations to trigger lazy-loaded lists (fixed-steps mode; used by `--preview` and when `maxScrollSteps` doesn't apply) |
| `maxScrollSteps` | `12` | Scroll ceiling in target-count mode (archive harvest). When the harvest needs more candidates than a fixed pass yields, it keeps scrolling up to this many iterations, exiting early once enough candidates are collected or the stream goes stale (3 consecutive scrolls add nothing) |
| `scrollWaitMs` | `1500` | Pause after each scroll |
| `bodySelectors` | (6 selectors) | Tried in order; first yielding ≥ `bodyMinParagraphs` wins |
| `bodyMinParagraphs` | `3` | Min paragraph count for a selector to qualify |
| `validateArticle` | `false` | When `true`, `fetchPage` probes `og:type` + JSON-LD `@type` + body length and returns `isArticle`; `harvest.js` skips candidates where `isArticle === false`. Use with a *broad* `urlPattern` (so candidate URLs aren't pre-filtered by fragile slug heuristics) to reject section/category pages that slip through once the page is actually fetched. Off by default → existing recipes unchanged. |
| `bodyMinChars` | `500` | Floor on body length for`isArticle`(only consulted when `validateArticle: true`) |
| `dateSelector` | `time[datetime]` | Falls back to JSON-LD `datePublished` |
| `authorSelector` | `[rel="author"], [class*="author"] a` | Falls back to JSON-LD `author.name` |
> (`datePublished` / `author.name`). Many SPA sites (Nikkei) only embed metadata
> there, so you usually don't need a custom selector.
## Troubleshooting recipes
### Convergence check
Preview passes when fetched links are real articles (not nav/section pages) and `charCount > 500`. Diagnose failures against the candidate count `preview.js` prints (`Collected N candidate links`):
| Symptom | Fix |
|---------|-----|
| Too few candidates | `listSelector` too narrow. Re-run `inspect`(without `--write`) to see container groups;if articles live in several card classes, use a **composite `listSelector`**(comma-separated, e.g. `[class*="StreamArticleCard"], [class*="SpotlightArticleCard"]`). |
| Candidates are nav/section pages | Tighten `urlPattern`, or (if article & section URLs share a path shape)set`validateArticle: true` so `fetchPage` rejects section pages by `og:type`/body. |
| Empty body | Fix `bodySelectors`. |
| 0 candidates, all `/location/`-style nav | `listSelector: null`(whole page) is wrong — nav floods the `maxLinks` cap. Scope to card containers. |
### Pitfall: articles spread across multiple card containers
`inspect-source.js` reports the *single largest* container group as `listSelector`, but on many sites (Nikkei, large news hubs) real articles are spread across several card-container classes of similar size, and no one container holds them all. Setting `listSelector` to just the biggest group silently drops the rest.
Symptom: `inspect` shows ~30 article-like links on the page but `preview` collects only ~12. Fix: write `listSelector` as a **comma-separated composite** of every article-card class prefix, e.g.
Re-run `inspect`(without `--write`) to enumerate which `*ArticleCard*` classes carry links if unsure.
Why not `listSelector: null`(whole page)? On sites with country/topic nav dropdowns (Nikkei's `/location/<country>` links, ~30 entries), a whole-page scan floods the `maxLinks` cap before real articles are reached. Scoping to card containers avoids this. `inspect` without `--write` shows the container groups so you can pick the right composite.
### When to use `validateArticle` (and when not to)
Set `validateArticle: true` when article and section/category URLs on the site **share a path shape** — i.e. you can't write a `urlPattern` that admits articles and rejects section pages purely from the URL. Nikkei is the canonical case: `/business/tech/semiconductors/<slug>`(article) and `/business/tech/semiconductors`(section) differ only by the trailing slug, and slug length is an unreliable signal. With `validateArticle`, `fetchPage` probes `og:type` + JSON-LD `@type` + body length, marks the section page `isArticle=false`(via `og:type=website` / `@type=Thing` / empty body), and `harvest.js` skips it. This lets you use a *broad*`urlPattern`(no fragile "slug must have N hyphens" guesswork) and still keep section pages out.
Don't set it when the URL alone cleanly separates articles from sections (e.g. Reuters'`/world/<slug>/<date>/` vs. `/world/`) — a tight `urlPattern` is cheaper than a per-candidate page fetch. `validateArticle` defaults to `false`, so existing recipes are unaffected.
-`articles`: URLs that must fetch as `isArticle=true` with `bodyChars ≥ bodyMinChars`
-`sections`: URLs that must fetch as `isArticle=false` (only checked when `validateArticle: true`)
-`minLinks`: homepage must yield at least this many candidate links
Pick fixture URLs across the path depths the site uses (2-seg and 3-seg) so depth regressions surface. When the site archives a fixture article, swap in a current one from the same path shape.
varchallengeTitle=/^(just a moment|attention required|verify|checking your browser|are you human)/i.test(title)
||/access denied|请输入验证码|安全验证/i.test(title);
// Challenge body copy.
varchallengeText=/please verify (that )?you are human|verify you are human|unusual traffic from your computer|checking your browser before accessing|enable javascript and cookies to continue|our system has detected unusual traffic|请完成安全验证|进行人机验证/i.test(bodyText);
// Browser (CDP) entry point. Runs the probe in the page via cdpEval; throws
// HumanInterventionError if the page is a challenge/login wall.
asyncfunctionevaluateBrowserBlock(ws){
const{cdpEval}=require('./cdp-client');
constraw=awaitcdpEval(ws,BROWSER_CHECK_EXPR);
evaluateBlockResult(raw);
}
// ── Direct (HTTP) entry point ───────────────────────────────────────────────
// String scan of raw HTML for challenge/login-wall signals. No DOM, so the
// signals are a subset of the browser probe — enough to catch a Cloudflare
// interstitial or a login redirect on an open site.
constDIRECT_CHALLENGE_RE=/please verify (that )?you are human|verify you are human|unusual traffic from your computer|checking your browser before accessing|enable javascript and cookies to continue|our system has detected unusual traffic|请完成安全验证|进行人机验证/i;
constDIRECT_CHALLENGE_TITLE_RE=/<title[^>]*>(just a moment|attention required|verify|checking your browser|are you human|access denied|请输入验证码|安全验证)/i;
constDIRECT_LOGIN_CTA_RE=/sign in to (continue|read)|log in to (continue|read)|subscribe to (continue|read|keep)|以继续阅读|登录后查看|订阅后查看|登录以继续|登录后可继续阅读/i;
console.error(`\n✅ Wrote helpers/${args.write}.md (listSelector=${best.container}, ${best.count} links measured)`);
console.error(` Next: node scripts/preview.js --url ${args.url} --recipe helpers/${args.write}.md --count 3 # verify links are real articles + charCount>500`);
}else{
console.log(`\n📝 Suggested helper.md skeleton (best group: ${best.count} links in ${best.container}):\n`);
console.log(content);
console.log(`\n Save to helpers/<source-id>.md (or re-run with --write <id>), then test: node scripts/preview.js --url <url> --recipe helpers/<source-id>.md --count 3`);
