Commit 45dcc7de authored by 谢宇轩's avatar 谢宇轩

feat: skill 文档精简 + CDP 登录引导 + tag/category 改造 + 日期修复

SKILL.md 精简 (476→296行):
- Document Formats 移到 references/document-formats.md (每次调用少加载1/3上下文)
- 摘要 batching 改成单一决策树 (N≤8 一次写 / N>8 批≤4),消除4处重复
- 添加新源流程统一为 inspect --write 4步闭环,消除与 source-management.md 的冲突
- 新增 References 小节索引三个 references 文件

CDP 登录引导 (跨平台):
- ensure-chromium.js 新增 --list (检测平台+可用浏览器) 和 --login (窗口化登录)
- --login 用固定 profile (~/.news-harvester/chromium-profile),端口占用时报告退出
- SKILL.md 的 Dedicated profile 从 macOS-only 手写命令改为跨平台3步流程
- --no-sandbox 仅在 root 下添加 (修掉 macOS 的不受支持标记警告)
- config-template.json 加 browser method 示例源

tag/category 改造:
- tags 从 [category, source] 改为 agent 从摘要提取的 ≤3 语义关键词
- 新增 frontmatter category 字段 (独立于 tags,支持 Obsidian 筛选)
- finalize 回写原文.md 的 tags+category frontmatter (与摘要.md 一致)
- index.md 文章列表新增"原文标题"列,标签显示提取的
- 摘要文件移到 <dayDir>/summary/ 子目录 (与 assets/ 同级,wikilink 保持裸标题)

日期修复:
- frontmatter date: 存完整 ISO 8601 时间戳 (原 substring(0,10) 截断导致跨时区差一天)
- fetcher-direct.js 正则放宽,捕获含毫秒/时区偏移的完整 ISO
- registry article 加 date 字段 (之前未存,导致 index 日期列空)
- callout 发布时间显示上海时区 (UTC+8),来源/URL 各自另起一行
- index 表格用紧凑 YYYY-MM-DD (dateCell),存储用完整 ISO (dateShort)
parent 1d7e3152
...@@ -26,11 +26,11 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau ...@@ -26,11 +26,11 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau
| Export a source as a shareable bundle | `node scripts/source-manage.js export <id> [--out <path.nhsource.json>]` | | Export a source as a shareable bundle | `node scripts/source-manage.js export <id> [--out <path.nhsource.json>]` |
| Install a source from a bundle | `node scripts/source-manage.js install <pkg.nhsource.json> [--force]` | | Install a source from a bundle | `node scripts/source-manage.js install <pkg.nhsource.json> [--force]` |
| Ensure Chromium (browser sources) | `node scripts/ensure-chromium.js` (or `--check`) | | Ensure Chromium (browser sources) | `node scripts/ensure-chromium.js` (or `--check`) |
| Detect available CDP browsers | `node scripts/ensure-chromium.js --list` |
| Launch login window (set port + profile) | `node scripts/ensure-chromium.js --login [--url <url>]` |
> **All commands assume `cd <skill-root>` first** (the directory containing this `SKILL.md`). Scripts live in the `scripts/` subfolder — **always prefix with `scripts/`**, e.g. `node scripts/harvest.js`, never `node harvest.js` (that fails with `MODULE_NOT_FOUND`). > **All commands assume `cd <skill-root>` first** (the directory containing this `SKILL.md`). Scripts live in the `scripts/` subfolder — **always prefix with `scripts/`**, e.g. `node scripts/harvest.js`, never `node harvest.js` (that fails with `MODULE_NOT_FOUND`).
**Division of labor:** scripts do all deterministic I/O (fetch, parse, download images, write 原文.md / registry / index). The agent does only the intelligent part — Chinese translation + 4-section summaries — and hands them back as a JSON file for `finalize.js` to stitch in. The full article body is written to disk by the script and **never enters the agent's context**.
Config file: `config.json` (auto-generated from `references/config-template.json` on first run — see First-Run Setup / Configuration below) Config file: `config.json` (auto-generated from `references/config-template.json` on first run — see First-Run Setup / Configuration below)
--- ---
...@@ -67,7 +67,7 @@ If the user wants a different first source (e.g. `reuters` with `browser` method ...@@ -67,7 +67,7 @@ If the user wants a different first source (e.g. `reuters` with `browser` method
node scripts/ensure-chromium.js --check # is CDP already up? exit 0/1 node scripts/ensure-chromium.js --check # is CDP already up? exit 0/1
node scripts/ensure-chromium.js # launch + readiness poll (no-op if already up) node scripts/ensure-chromium.js # launch + readiness poll (no-op if already up)
``` ```
If discovery can't find your browser, set `config.chromium.executablePath` to its full path (see Configuration). If discovery can't find your browser, set `config.chromium.executablePath` to its full path (see Configuration). **If the source needs login** (subscription/paywall), run `node scripts/ensure-chromium.js --login --url <homepage>` to open a windowed browser, log in, and persist the profile — see *Dedicated profile (login state)* below.
> **Chromium is launched whenever `method: browser` OR the source has a recipe** (`helpers/<id>.md`). A `direct` source that gains a recipe also uses Chromium, since `fetcher-helper.js` drives the browser. See Source recipes below. > **Chromium is launched whenever `method: browser` OR the source has a recipe** (`helpers/<id>.md`). A `direct` source that gains a recipe also uses Chromium, since `fetcher-helper.js` drives the browser. See Source recipes below.
...@@ -79,9 +79,7 @@ If this returns article data, setup is complete. If it errors (empty list, parse ...@@ -79,9 +79,7 @@ If this returns article data, setup is complete. If it errors (empty list, parse
**5. (Optional) full test archive.** Only if the user wants to verify vault writes — run Flow A once with `count 1-2` and confirm files land in `<vaultPath>/<vaultFolder>/YYYYMMDD/`. Skip otherwise; the `--preview` smoke test is enough to consider setup done. **5. (Optional) full test archive.** Only if the user wants to verify vault writes — run Flow A once with `count 1-2` and confirm files land in `<vaultPath>/<vaultFolder>/YYYYMMDD/`. Skip otherwise; the `--preview` smoke test is enough to consider setup done.
### After onboarding Once `vaultPath` is real and the smoke test passes, **skip this section** on future runs and go straight to Workflow.
Once `vaultPath` is real and the smoke test passes, **skip this section** on future runs and go straight to Workflow. The check is idempotent and cheap — re-running `node scripts/source-manage.js list` is always safe.
--- ---
...@@ -102,46 +100,44 @@ Before running a harvest, the agent MUST verify/edit these fields: ...@@ -102,46 +100,44 @@ Before running a harvest, the agent MUST verify/edit these fields:
#### Dedicated profile (login state) #### Dedicated profile (login state)
For sources that need cookies/login (subscriptions, paywalled sites), point `chromium.userDataDir` at a fixed profile and log into it once. After that, every Chromium `ensure-chromium` auto-launches loads that profile — your cookies/environment carry over: For sources that need cookies/login (subscriptions, paywalled sites), log into a **fixed profile** once; after that, every headless `harvest.js` auto-launch reuses that profile and stays logged in. The launcher auto-discovers the browser binary per-platform (macOS `.app` bundles, Linux PATH, Windows install dirs), so this flow is identical on every OS — no hand-written per-OS command.
**Onboarding a login-required source (3 steps):**
```bash ```bash
# 1. First time: start Chromium windowed with the fixed profile, log in to the source. # 1. Detect the environment — platform + which Chromium/Chrome binaries are installed.
# Use the SAME path you'll put in config.json (expand $HOME — JSON doesn't). # Report this to the user so they know what will be launched.
"/Applications/Chromium.app/Contents/MacOS/Chromium" \ node scripts/ensure-chromium.js --list
--remote-debugging-port=9222 \
--user-data-dir="$HOME/.news-harvester/chromium-profile" \
--no-first-run --no-default-browser-check &
# Log in inside the window, then quit. The profile persists on disk.
# 2. In config.json (absolute path, no $HOME):
# "chromium": { "userDataDir": "/Users/<you>/.news-harvester/chromium-profile" }
# 3. Later runs: harvest.js auto-launches Chromium with this profile —
# headless by default, login state intact. No manual start needed.
```
On Linux replace the binary path with `chromium` (or your distro's name). If a site's anti-bot flags headless, set `chromium.headless: false`. # 2. Launch a WINDOWED browser on the source homepage for interactive login.
node scripts/ensure-chromium.js --login --url https://www.reuters.com/
```
The `--login` command opens a visible browser (headless off) with CDP enabled on `chromium.cdpPort`. It uses `config.chromium.userDataDir` if set, else a fixed default (`~/.news-harvester/chromium-profile`) — **never** the ephemeral tmp dir, so login state persists. Guide the user to:
1. Log into the site inside the window that opened.
2. **Quit the browser fully** (Cmd+Q on macOS, Ctrl+Q on Linux/Windows) when done — the cookies are saved to the profile.
To point a source at a different vault subfolder, set `sources[].vaultFolder`. Articles are archived to `<vaultPath>/<vaultFolder>/YYYYMMDD/`. If `--login` finds the CDP port already occupied (e.g. a headless harvest instance is running), it reports that and exits without launching — close the other browser first, then re-run.
--- **3. Persist the profile in config.json.** If `--login` used the *default* profile path (config had no `userDataDir`), it prints a ready-to-paste snippet. Write it in (absolute path, no `$HOME`):
```json
"chromium": { "userDataDir": "/Users/<you>/.news-harvester/chromium-profile" }
```
After this, `harvest.js` auto-launches Chromium headless with this profile — login state intact, no manual start needed.
## Source recipes (`helpers/<id>.md`) > If a site's anti-bot flags headless, set `chromium.headless: false` (the harvest auto-launch will then keep a window). `--login` always launches windowed regardless, since you need to interact with it.
Each source can have a **recipe** — a `helpers/<source-id>.md` file whose YAML frontmatter declares how to find its article list and extract article content (selectors, scroll, URL filters). When `harvest.js` sees one, it uses the spec-driven `fetcher-helper.js` instead of the legacy fetcher. **No recipe → legacy path, zero behavior change** — so existing sources keep working and you migrate one at a time. To point a source at a different vault subfolder, set `sources[].vaultFolder`. Articles are archived to `<vaultPath>/<vaultFolder>/YYYYMMDD/`. Within a day directory: 原文.md files + `registry.json` + `index.md` at the root, images in `assets/`, and Chinese summary docs (摘要) in `summary/`.
> **Precedence**: when a recipe exists, its `urlPattern`/selectors are the source of truth and `config.json`'s `articleUrlPattern` is **ignored**. Sources with a recipe therefore don't need (and shouldn't carry) `articleUrlPattern` in config — it's a legacy-direct-source field only. `source-manage.js list` shows a `RECIPE` column so you can tell at a glance which sources are on which path. ---
Recipes solve the "can't find high-quality data on new CDP sources" problem: instead of one hard-coded regex scanning the whole page (which mixes nav/sidebar/footer links with real articles), a recipe scopes link collection to a `listSelector` container, triggers lazy-load via `scrollSteps`, and validates URLs with a `urlPattern`. Two layers of filtering keep the candidate pool clean: ## Adding a new source
1. **Link collection** (`listSelector` + `urlPattern`): scope to the article-card containers and match a URL shape. This is the coarse filter — it runs once on the list page. A source is a config entry + an optional **recipe** (`helpers/<id>.md`). Browser sources should get a recipe — `articleUrlPattern` is legacy-direct-only and ignored when a recipe exists. `source-manage.js list` shows a `RECIPE` column (`✅`/`—`) so you can tell which path each source is on.
2. **Article validation** (`validateArticle`, optional): when a candidate page is actually fetched, probe `og:type` + JSON-LD `@type` + body length and reject non-article pages (section/category pages that share an article's URL shape). This is the fine filter — it lets you use a *broad* `urlPattern` (no fragile "slug must have N hyphens" guesswork) and still keep section pages out of the archive.
The recipe is just markdown — edit it to adapt to a site redesign without touching code. **4-step closed loop** (add → inspect --write → verify → lock):
**Adding a new browser source** (4-step closed loop: add → inspect --write → verify → lock with regression tests):
```bash ```bash
# 1. Add to config (browser sources should get a recipe — articleUrlPattern is legacy-only) # 1. Add to config
node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser --folder "My Source" node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser --folder "My Source"
# 2. Inspect + auto-write the recipe in one shot (measures listSelector + infers urlPattern from real URLs) # 2. Inspect + auto-write the recipe in one shot (measures listSelector + infers urlPattern from real URLs)
...@@ -154,24 +150,21 @@ node scripts/harvest.js mysrc 3 --preview ...@@ -154,24 +150,21 @@ node scripts/harvest.js mysrc 3 --preview
node scripts/recipe-test.js mysrc # needs helpers/mysrc.fixtures.json — see below node scripts/recipe-test.js mysrc # needs helpers/mysrc.fixtures.json — see below
``` ```
**Convergence check (step 3)**: preview passes when the fetched links are real articles (not nav/section pages) and `charCount > 500`. Diagnose failures against the candidate count `harvest.js` prints (`Collected N candidate links`): ### Convergence check (step 3)
- **Too few candidates** → the `listSelector` is too narrow. Run `inspect` (without `--write`) and check whether articles live in *multiple* card-container classes. If so, use a **composite `listSelector`** (see pitfall below).
- **Candidates are nav/section pages** → either tighten `urlPattern`, or (preferred for sites where article and section URLs share a shape) set `validateArticle: true` and let `fetchPage` reject section pages by `og:type`/body.
- **Empty body** → fix `bodySelectors`.
- **0 candidates, all `/location/`-style nav links** → `listSelector: null` (whole page) is almost always wrong; nav dropdowns flood the `maxLinks` cap before real articles are reached. Scope to card containers instead.
Re-run preview after each edit. **Do not hand-edit the recipe blindly** — re-run `inspect` (without `--write`) to see the container groups if unsure.
#### Pitfall: articles spread across multiple card containers Preview passes when fetched links are real articles (not nav/section pages) and `charCount > 500`. Diagnose failures against the candidate count `harvest.js` prints (`Collected N candidate links`):
`inspect-source.js` reports the *single largest* container group as `listSelector`, but on many sites (Nikkei, large news hubs) real articles are spread across several card-container classes of similar size, and no one container holds them all. Setting `listSelector` to just the biggest group silently drops the rest.
Symptom: `inspect` shows ~30 article-like links on the page but `harvest --preview` collects only ~12. Fix: write `listSelector` as a **comma-separated composite** of every article-card class prefix, e.g. `[class*="StreamArticleCard"], [class*="SpotlightArticleCard"], [class*="SecondaryArticleCard"]`. Re-run `inspect` to enumerate which `*ArticleCard*` classes carry links if unsure. | Symptom | Fix |
|---------|-----|
| Too few candidates | `listSelector` too narrow. Re-run `inspect` (without `--write`) to see container groups; if articles live in several card classes, use a **composite `listSelector`** (comma-separated, e.g. `[class*="StreamArticleCard"], [class*="SpotlightArticleCard"]`). |
| Candidates are nav/section pages | Tighten `urlPattern`, or (if article & section URLs share a path shape) set `validateArticle: true` so `fetchPage` rejects section pages by `og:type`/body. |
| Empty body | Fix `bodySelectors`. |
| 0 candidates, all `/location/`-style nav | `listSelector: null` (whole page) is wrong — nav floods the `maxLinks` cap. Scope to card containers. |
#### When to use `validateArticle` (and when not to) Re-run preview after each edit. Don't hand-edit blindly — re-run `inspect` (without `--write`) to see the container groups if unsure. Deep troubleshooting (composite listSelector rationale, when to use `validateArticle`) is in `references/source-management.md` → *Troubleshooting recipes*.
Set `validateArticle: true` when article and section/category URLs on the site **share a path shape** — i.e. you can't write a `urlPattern` that admits articles and rejects section pages purely from the URL. Nikkei is the canonical case: `/business/tech/semiconductors/<slug>` (article) and `/business/tech/semiconductors` (section) differ only by the trailing slug, and slug length is an unreliable signal. With `validateArticle`, `fetchPage` marks the section page `isArticle=false` (via `og:type=website` / `@type=Thing` / empty body) and `harvest.js` skips it.
Don't set it when the URL alone cleanly separates articles from sections (e.g. Reuters' `/world/<slug>/<date>/` vs. `/world/`) — a tight `urlPattern` is cheaper than a per-candidate page fetch. `validateArticle` defaults to `false`, so existing recipes are unaffected. ### Locking with regression tests (step 4)
#### Locking a stable recipe with regression tests (step 4)
Once a recipe converges, capture a fixture set so a future site redesign is caught instead of silently dropping articles. Create `helpers/<id>.fixtures.json`: Once a recipe converges, capture a fixture set so a future site redesign is caught instead of silently dropping articles. Create `helpers/<id>.fixtures.json`:
```json ```json
{ {
...@@ -184,34 +177,20 @@ Then `node scripts/recipe-test.js <id>` asserts: every article URL fetches `isAr ...@@ -184,34 +177,20 @@ Then `node scripts/recipe-test.js <id>` asserts: every article URL fetches `isAr
**Once verified, archive** by following Flow A below (harvest writes originals → you write `.summaries.json` → finalize). The harvest stderr prints the `Day dir` and the exact next command — follow that, don't parse the stdout manifest. **Once verified, archive** by following Flow A below (harvest writes originals → you write `.summaries.json` → finalize). The harvest stderr prints the `Day dir` and the exact next command — follow that, don't parse the stdout manifest.
Full schema and authoring notes: `references/source-management.md` (Helper recipes). Full recipe schema and authoring notes: `references/source-management.md`.
> **Roadmap (model B)**: the current split keeps direct sources on legacy `fetcher-direct.js`. The eventual goal is for *every* source to have a recipe (moving `method`/`homepageUrl`/`urlPattern` into the recipe and reducing `config.json` to a pure index of `id`/`name`/`vaultFolder`/`enabled`), with `fetcher-helper.js` gaining a direct-HTTP mode. Direct sources are intentionally untouched this round to avoid disrupting working fetchers.
--- ---
## Sharing sources (export / install) ## Sharing sources (export / install)
A recipe (`helpers/<id>.md` + optional `helpers/<id>.fixtures.json`) captures everything site-specific about a source — so a finished, verified recipe is portable. `source-manage.js` can pack a source into a single self-contained **`.nhsource.json`** bundle that another user installs in one command, with no manual recipe authoring on their end. A recipe (`helpers/<id>.md` + optional `helpers/<id>.fixtures.json`) captures everything site-specific about a source — so a finished, verified recipe is portable. `source-manage.js` packs a source into a single self-contained **`.nhsource.json`** bundle that another user installs in one command.
```bash ```bash
# Pack: config entry + recipe + fixtures → one shareable file node scripts/source-manage.js export <id> [--out <path.nhsource.json>] # pack (default: ./<id>.nhsource.json)
node scripts/source-manage.js export <id> [--out <path.nhsource.json>] node scripts/source-manage.js install <pkg.nhsource.json> [--force] # install on another machine/vault
# Default output: ./<id>.nhsource.json in the cwd.
# Install (on another machine / in another vault):
node scripts/source-manage.js install <pkg.nhsource.json> [--force]
``` ```
**What's in a bundle**: the source's config metadata (`id`/`name`/`homepageUrl`/`method`/`vaultFolder`) plus its `helpers/<id>.md` and `helpers/<id>.fixtures.json` (if present). It deliberately **excludes** `vaultPath` (machine-specific, global), `enabled` (the installer's choice — always set `true`), and `articleUrlPattern` (legacy-direct-only; a recipe source doesn't need it). `install` writes the recipe + fixtures into `helpers/`, registers the source in `config.json` (enabled), and the source is **immediately usable** — run `node scripts/harvest.js <id> 3 --preview` right after. If fixtures came with the bundle, `node scripts/recipe-test.js <id>` verifies the recipe still matches the live site. Bundle format, install conflict handling (`--force`), and the `vaultPath` caveat are in `references/source-management.md` → *Sharing sources*.
**Install behavior**: writes the recipe + fixtures into `helpers/` and registers the source in `config.json` (enabled). The installed source is **immediately usable** — run `node scripts/harvest.js <id> 3 --preview` right after. If `helpers/<id>.fixtures.json` came with the bundle, `node scripts/recipe-test.js <id>` verifies the recipe still matches the live site.
**Conflicts**: `install` refuses if `<id>` already exists in `config.json` or a helper file of the same name is present; `--force` overwrites both (printing what changed). This matches the `helper` command's refuse-with-force convention.
**vaultPath caveat**: install works even before `vaultPath` is configured (the source is registered, preview works), but archiving needs a real vault path — install prints a warning if `vaultPath` is still the placeholder.
> Bundles are plain JSON — open them in any editor to inspect before installing, or share via any channel (email, chat, git). No signing/checksum (personal-sharing scope); if you need integrity, verify the file's `exportedAt` + `source` fields by eye.
--- ---
...@@ -219,7 +198,15 @@ node scripts/source-manage.js install <pkg.nhsource.json> [--force] ...@@ -219,7 +198,15 @@ node scripts/source-manage.js install <pkg.nhsource.json> [--force]
Two flows depending on intent: **archive** (fetch + save to Obsidian) or **preview** (read-only, just return article data). Two flows depending on intent: **archive** (fetch + save to Obsidian) or **preview** (read-only, just return article data).
> **Stay lean.** If the user already named a source (e.g. "reuters"), use it directly — do **not** run `source-manage.js list` first. The archive flow is exactly **3 script calls**: `harvest.js` → write `.summaries.json` → `finalize.js`. The pre-flight gate above is the only config check; don't re-verify the vault or re-list sources mid-flow. > **Stay lean.** If the user already named a source (e.g. "reuters"), use it directly — do **not** run `source-manage.js list` first. The archive flow is exactly **3 script calls**: `harvest.js` → write `.summaries.json` → `finalize.js`.
### How to work (read once)
- **Scripts do all deterministic I/O** (fetch, parse, download images, write 原文.md / registry / index). The agent does only the intelligent part — Chinese translation + 4-section summaries — and hands them back as a JSON file for `finalize.js` to stitch in. The full article body is written to disk by the script and **never enters the agent's context**.
- **`count` is the target number of NEW articles**, not raw links. `harvest.js` scrolls for extra candidates so the *new* count can reach `count` even when most of the homepage was already archived. If a run returns **fewer than `count`**, the stderr prints `⚠️ … stream may be exhausted` — that's a genuine source limit (no fresh articles), **not** a bug; re-running won't help. Don't re-run `harvest.js` for the same source/day just to "get more" — it dedups via the registry, so a second run only fetches genuinely new articles (merging into the pending `.manifest.json`, which is safe).
- **Never `Read` the 原文.md files** to write summaries — the `bodyPreview` in the manifest is sufficient. Targeted slice reads are OK if truly needed.
- **Extract `tags` from the summary/body** (≤3 semantic keywords), not from the category/source. The script's harvest-time placeholder tags are overwritten by yours at finalize. `category` is separate (a classification, not a tag).
- **One `finalize.js` call** after writing `.summaries.json`. Do not hand-write file I/O — the scripts handle all original/registry/index/summary writes.
### Flow A — Archive to Obsidian (3 steps) ### Flow A — Archive to Obsidian (3 steps)
...@@ -237,9 +224,7 @@ It prints a **compact manifest** to stdout (no full body — only a `bodyPreview ...@@ -237,9 +224,7 @@ It prints a **compact manifest** to stdout (no full body — only a `bodyPreview
> **Browser sources and any source with a recipe** are auto-handled: before fetching, `harvest.js` calls `ensure-chromium.js`, which health-checks the CDP port and launches Chromium itself (cross-platform binary discovery, headless by default) if it isn't up — a no-op if already running. No manual `chromium ... &` or `sleep` is needed. To debug launch failures: `node scripts/ensure-chromium.js` (prints binary path, port, and stderr-log tail on failure). > **Browser sources and any source with a recipe** are auto-handled: before fetching, `harvest.js` calls `ensure-chromium.js`, which health-checks the CDP port and launches Chromium itself (cross-platform binary discovery, headless by default) if it isn't up — a no-op if already running. No manual `chromium ... &` or `sleep` is needed. To debug launch failures: `node scripts/ensure-chromium.js` (prints binary path, port, and stderr-log tail on failure).
#### Step 2. Generate Chinese summaries → write `.summaries.json` (batched) #### Step 2. Generate Chinese summaries → write `.summaries.json`
**Why batch:** writing all summaries in a *single* response hits the model's output-token ceiling once you have ~10+ articles — the response is truncated and the remaining articles silently get no summary (their 原文 keeps the `__SUMMARY_<slug>__` sentinel forever). This failure is **not rare** — it reproduces every time a source yields many articles. Avoid it by working in small batches that each get merged into `.summaries.json` by `summary-add.js`.
Each summary entry has this shape: Each summary entry has this shape:
...@@ -248,33 +233,33 @@ Each summary entry has this shape: ...@@ -248,33 +233,33 @@ Each summary entry has this shape:
"url": "https://theconversation.com/...-286430", "url": "https://theconversation.com/...-286430",
"titleZh": "特朗普新规或拖慢联邦科研", "titleZh": "特朗普新规或拖慢联邦科研",
"category": "科技/AI", "category": "科技/AI",
"tags": ["科技/AI", "The Conversation"], "tags": ["科研经费", "联邦政策", "大学"],
"summaryZh": "## 事件概述\n...\n\n## 背景\n...\n\n## 关键引语\n> \"...\"\n\n## 影响分析\n..." "summaryZh": "## 事件概述\n...\n\n## 背景\n...\n\n## 关键引语\n> \"...\"\n\n## 影响分析\n..."
} }
``` ```
- `titleZh`: concise Chinese title (≤30 chars) - `titleZh`: concise Chinese title (≤30 chars)
- `category` / `tags`: the agent's classification (overrides the script's keyword guess) - `category`: the article's category — overrides the script's keyword guess. Written to the frontmatter `category` field (enables Obsidian Bases/property filtering). See Category Detection below for the allowed values.
- `tags`: **≤3 keywords extracted from the summary/body** (semantic, not the category or source name). The script's harvest-time placeholder (`[source.name]`) is overwritten by these at finalize. These flow into both the 摘要.md and 原文.md frontmatter `tags`, and the index table.
- `summaryZh`: the 4-section markdown body (事件概述 / 背景 / 关键引语 / 影响分析) - `summaryZh`: the 4-section markdown body (事件概述 / 背景 / 关键引语 / 影响分析)
- The `bodyPreview` in the manifest is enough to write the summary. **Do not re-read the full 原文.md** — if you need more, read a targeted slice. - The `bodyPreview` in the manifest is enough to write the summary. **Do not re-read the full 原文.md** — if you need more, read a targeted slice.
**Procedure** (loop until every article in the manifest has a summary): **Batching decision tree** (writing all summaries in one response hits the output-token ceiling once you have ~10+ articles — the response is truncated and tail articles silently get no summary):
```
N = manifest 文章数
N ≤ 8 → Write 全部条目到 <dayDir>/.summaries.json (单次 Write) → Step 3
N > 8 → 循环(每批 ≤4 条):
1. Write <dayDir>/.summaries-batch.json (JSON 数组, ≤4 条)
2. node scripts/summary-add.js "<dayDir>" (合并进 .summaries.json, 按 url 覆盖, 删 batch 文件)
3. 回到 1, 直到所有文章都有摘要
完成后 → node scripts/summary-add.js "<dayDir>" --status → 若 missing=0 进 Step 3, 否则补缺
```
1. Let `N` = number of articles in the manifest. If `N ≤ 8`, you may write all entries to `.summaries.json` directly (one Write) and skip to Step 3. Otherwise work in batches of **≤4**. `summary-add.js` merges each batch into `.summaries.json` (keyed by `url` — re-supplying an entry overwrites it, no duplicates), then deletes the batch file.
2. For each batch: use `Write` to create `<dayDir>/.summaries-batch.json` — a JSON array of ≤4 summary entries — then merge it:
```bash
node scripts/summary-add.js "<dayDir>"
```
This merges the batch into `.summaries.json` (keyed by `url`, so re-supplying an entry overwrites it — no duplicates), then deletes the batch file. Repeat per batch.
3. Once all batches are done, verify coverage and fill any gaps:
```bash
node scripts/summary-add.js "<dayDir>" --status
```
Lists done vs. missing against `.manifest.json`. Write any missing articles as another batch and merge. When `--status` reports `missing: 0`, proceed to Step 3.
> **Interrupted run?** Do not re-run `harvest.js` (it dedups via the registry, so it won't re-fetch). Instead run `node scripts/summary-add.js "<dayDir>" --status` to see which articles still need summaries, then resume from there. > **Interrupted run?** Do not re-run `harvest.js` (it dedups via the registry, so it won't re-fetch). Instead run `node scripts/summary-add.js "<dayDir>" --status` to see which articles still need summaries, then resume from there.
#### Step 3. Run finalize (script stitches summaries in) #### Step 3. Run finalize (script stitches summaries in)
```bash ```bash
node scripts/finalize.js "<dayDir>" node scripts/finalize.js "<dayDir>"
...@@ -292,171 +277,6 @@ node scripts/harvest.js <source_id> <count> --preview --full # ... + full body ...@@ -292,171 +277,6 @@ node scripts/harvest.js <source_id> <count> --preview --full # ... + full body
No files are written. stdout is a compact JSON list (one call, no re-fetching / re-parsing). Use `--full` only when the user explicitly wants the complete article text. No files are written. stdout is a compact JSON list (one call, no re-fetching / re-parsing). Use `--full` only when the user explicitly wants the complete article text.
### Guardrails (token hygiene)
- **`count` is the target number of NEW articles**, not raw links. `harvest.js` already accounts for registry dedup: it scrolls to collect extra candidate links (`count + seen + buffer`, capped) so the *new* count can still reach `count` even when most of the homepage was already archived. If the run returns **fewer than `count`**, the stderr prints `⚠️ … stream may be exhausted` — that's a genuine source limit (no fresh articles to fetch), **not** a bug; re-running won't help. Wait for new articles or broaden the source instead. Don't re-run `harvest.js` for the same source/day just to "get more" — it deduplicates via the registry, so a second run only fetches genuinely new articles (and merges them into the pending `.manifest.json`, which is safe).
- **Never `Read` the 原文.md files** to write summaries — the `bodyPreview` in the manifest is sufficient. Targeted slice reads are OK if truly needed.
- **Summaries in batches of ≤4, never one big response.** A single response holding 10+ summaries hits the output-token ceiling and is truncated — the tail articles silently get no summary and keep the `__SUMMARY_<slug>__` sentinel. Write each batch to `.summaries-batch.json` and merge via `node scripts/summary-add.js "<dayDir>"`; repeat per batch. `summary-add.js --status` shows what's left. This is mandatory for `N > 8`; recommended for any multi-article run.
- **One `finalize.js` call** after writing `.summaries.json`. Do not hand-write file I/O — the scripts handle all original/registry/index/summary writes.
---
## Document Formats
> These docs are **generated by the scripts**, not hand-written by the agent. Shown here for reference (e.g. when debugging or reading archived files). The agent only writes `.summaries.json`; `finalize.js` produces the Chinese summary docs and patches the originals.
### Original Doc (written by `harvest.js`)
Filename: `<英文标题> - <SourceName>原文.md`
Between Step 1 (harvest) and Step 3 (finalize), the Chinese-title links in this file are a **sentinel placeholder** `__SUMMARY_<slug>__` — in both the frontmatter `related` field and the footer wikilink. `finalize.js` replaces them with the real `titleZh`. Image captions: only rendered when `alt` text exists (empty alt → bare embed, no `**`).
```markdown
---
title: "<英文标题>"
date: YYYY-MM-DD
tags: [...]
source: "<url>"
publisher: <source>
authors: "<authors>"
type: 原文
related: "[[<中文标题>]]" # sentinel __SUMMARY_<slug>__ until finalize
saved: YYYY-MM-DD
---
# <英文标题>
> [!info] Source
> **Published:** YYYY-MM-DD | **URL:** <url>
> **中文摘要:** [[<中文标题>]]
---
## 配图
![[assets/<slug>-1.jpg]]
*<alt text — omitted if empty>*
---
## 正文
<full article body>
---
[[<中文标题>]]
```
### Chinese Summary Doc (written by `finalize.js`)
Filename: `<中文标题>.md` (concise, ≤30 chars)
```markdown
---
title: <中文标题>
date: YYYY-MM-DD
tags:
- <tag1>
- <tag2>
source: "<original URL>"
publisher: <source name>
authors: "<author names>"
type: 摘要
related: "[[<英文标题> - <SourceName>原文]]"
saved: YYYY-MM-DD
---
# <中文标题>
> [!info] 文章信息
> **发布:** YYYY-MM-DD | **来源:** <source>
> **作者:** <authors>
> **原文:** [[<英文标题> - <SourceName>原文]]
---
## 事件概述
<2-3 sentences summary in Chinese>
## 背景
<context>
## 关键引语
> "<translated quote>"
## 影响分析
<analysis>
---
[[<英文标题> - <SourceName>原文]]
```
---
## index.md Format
File: `<vaultFolder>/YYYYMMDD/index.md` (written by `harvest.js`, rewritten by `finalize.js`)
Image asset count is a **real file count** of `assets/` (not "articles with images"), so it reflects downloaded files even before finalize.
```markdown
---
date: YYYY-MM-DD
type: daily-index
source: <source name>
total_articles: N
last_updated: "<ISO timestamp>"
---
# <SourceName> Daily Index · YYYY-MM-DD
> **最近更新:** YYYY-MM-DD HH:MM CST
> **当日归档:** N 篇 | **图片资产:** N 张
## 📊 分类统计
| 分类 | 数量 |
|------|------|
| ... | ... |
## 📋 文章列表
| # | 标题 | 分类 | 标签 | 发布时间 | 状态 |
|---|------|------|------|----------|------|
| 1 | [[中文标题]] | ... | ... | YYYY-MM-DD | ✅ |
## 🗂️ 分类导航
### 分类1(N篇)
- [[文章1]]
```
---
## registry.json Format
File: `<vaultFolder>/YYYYMMDD/registry.json` (written incrementally by `harvest.js`, patched by `finalize.js`)
`titleZh` is the sentinel `__SUMMARY_<slug>__` until finalize supplies the real title.
```json
{
"date": "YYYYMMDD",
"source": "<source_id>",
"articles": [
{
"url": "https://...",
"titleZh": "中文标题",
"titleEn": "English Title",
"processedAt": "2026-07-24T14:30:00.000Z",
"category": "地缘政治",
"tags": ["tag1", "tag2"],
"slug": "theconversation-com-some-article-286430",
"originalFilename": "Some Article - The Conversation原文.md"
}
]
}
```
---
## Source Management
Read `references/source-management.md` for detailed instructions on add/list/edit operations.
--- ---
## Category Detection ## Category Detection
...@@ -474,3 +294,13 @@ Assign category based on URL path and title keywords: ...@@ -474,3 +294,13 @@ Assign category based on URL path and title keywords:
| default | 国际 | | default | 国际 |
Category emoji mapping: 地缘政治→🌏, 科技/AI→🤖, 中美关系→🇨🇳, 财经→💰, 社会→🧑‍🤝‍🧑, 国际冲突→⚔️, 国际→📰 Category emoji mapping: 地缘政治→🌏, 科技/AI→🤖, 中美关系→🇨🇳, 财经→💰, 社会→🧑‍🤝‍🧑, 国际冲突→⚔️, 国际→📰
---
## References
| File | When to read |
|------|--------------|
| `references/document-formats.md` | Debugging archived files — schema of 原文/摘要/index.md/registry.json (all script-generated; the agent only writes `.summaries.json`). |
| `references/source-management.md` | Adding/editing sources in depth — recipe frontmatter schema, troubleshooting (composite listSelector, `validateArticle`), bundle format & install behavior. |
| `references/helper-example.md` | Annotated recipe template with every field commented. Start here when authoring a recipe by hand. |
...@@ -16,6 +16,14 @@ ...@@ -16,6 +16,14 @@
"articleUrlPattern": "theconversation\\.com/[\\w-]+-\\d+$", "articleUrlPattern": "theconversation\\.com/[\\w-]+-\\d+$",
"vaultFolder": "The Conversation", "vaultFolder": "The Conversation",
"enabled": true "enabled": true
},
{
"id": "reuters-world",
"name": "路透社 World",
"homepageUrl": "https://www.reuters.com/world/",
"method": "browser",
"vaultFolder": "路透社 World",
"enabled": true
} }
] ]
} }
# Document Formats (reference)
These docs are **generated by the scripts**, not hand-written by the agent —
shown here for debugging or when reading archived files. The agent only writes
`.summaries.json`; `finalize.js` produces the Chinese summary docs and patches
the originals.
## Original Doc (written by `harvest.js`)
Filename: `<英文标题> - <SourceName>原文.md`
Between Step 1 (harvest) and Step 3 (finalize), the Chinese-title links in this file are a **sentinel placeholder** `__SUMMARY_<slug>__` — in both the frontmatter `related` field and the footer wikilink. `finalize.js` replaces them with the real `titleZh`, and overwrites the `tags` block + `category` line with the agent-extracted values. Image captions: only rendered when `alt` text exists (empty alt → bare embed, no `**`).
```markdown
---
title: "<英文标题>"
date: <ISO 8601 published timestamp, e.g. 2026-07-30T21:47:28Z>
tags: # agent-extracted (≤3); placeholder [source] until finalize
- <tag1>
- <tag2>
category: <分类> # agent-supplied; overrides the script's keyword guess
source: "<url>"
publisher: <source>
authors: "<authors>"
type: 原文
related: "[[<中文标题>]]" # sentinel __SUMMARY_<slug>__ until finalize
saved: YYYY-MM-DD
---
# <英文标题>
> [!info] Source
> **Published:** <Shanghai time, e.g. 2026-07-31 05:47>
> **URL:** <url>
> **中文摘要:** [[<中文标题>]]
---
## 配图
![[assets/<slug>-1.jpg]]
*<alt text — omitted if empty>*
---
## 正文
<full article body>
---
[[<中文标题>]]
```
## Chinese Summary Doc (written by `finalize.js`)
File: `<dayDir>/summary/<中文标题>.md` (concise title, ≤30 chars). The `summary/` subdirectory is a sibling of `assets/`. Wikilinks from 原文.md and index.md use the bare title `[[<中文标题>]]` — Obsidian resolves them by basename.
```markdown
---
title: <中文标题>
date: <ISO 8601 published timestamp, e.g. 2026-07-30T21:47:28Z>
tags: # agent-extracted from the summary/body (≤3)
- <tag1>
- <tag2>
category: <分类> # agent-supplied; enables Obsidian property/Bases filtering
source: "<original URL>"
publisher: <source name>
authors: "<author names>"
type: 摘要
related: "[[<英文标题> - <SourceName>原文]]"
saved: YYYY-MM-DD
---
# <中文标题>
> [!info] 文章信息
> **发布:** <Shanghai time, e.g. 2026-07-31 05:47>
> **来源:** <source>
> **作者:** <authors>
> **原文:** [[<英文标题> - <SourceName>原文]]
---
## 事件概述
<2-3 sentences summary in Chinese>
## 背景
<context>
## 关键引语
> "<translated quote>"
## 影响分析
<analysis>
---
[[<英文标题> - <SourceName>原文]]
```
## index.md Format
File: `<vaultFolder>/YYYYMMDD/index.md` (written by `harvest.js`, rewritten by `finalize.js`)
Image asset count is a **real file count** of `assets/` (not "articles with images"), so it reflects downloaded files even before finalize.
```markdown
---
date: YYYY-MM-DD
type: daily-index
source: <source name>
total_articles: N
last_updated: "<ISO timestamp>"
---
# <SourceName> Daily Index · YYYY-MM-DD
> **最近更新:** YYYY-MM-DD HH:MM CST
> **当日归档:** N 篇 | **图片资产:** N 张
## 📊 分类统计
| 分类 | 数量 |
|------|------|
| ... | ... |
## 📋 文章列表
| # | 标题 | 原文标题 | 分类 | 标签 | 发布时间 | 状态 |
|---|------|----------|------|------|----------|------|
| 1 | [[中文标题]] | English Title | ... | tag1, tag2 | YYYY-MM-DD | ✅ |
> 发布时间列是紧凑 YYYY-MM-DD(本地时区),便于表格浏览;frontmatter `date:` 存完整 ISO 8601 时间戳(无损,Obsidian 可按任意时区解析)。
## 🗂️ 分类导航
### 分类1(N篇)
- [[文章1]]
```
## registry.json Format
File: `<vaultFolder>/YYYYMMDD/registry.json` (written incrementally by `harvest.js`, patched by `finalize.js`)
`titleZh` is the sentinel `__SUMMARY_<slug>__` until finalize supplies the real title.
```json
{
"date": "YYYYMMDD",
"source": "<source_id>",
"articles": [
{
"url": "https://...",
"titleZh": "中文标题",
"titleEn": "English Title",
"date": "2026-07-24T14:30:00.000Z",
"processedAt": "2026-07-24T15:00:00.000Z",
"category": "地缘政治",
"tags": ["tag1", "tag2"],
"slug": "theconversation-com-some-article-286430",
"originalFilename": "Some Article - The Conversation原文.md"
}
]
}
```
...@@ -85,5 +85,9 @@ maxBodyChars: 15000 ...@@ -85,5 +85,9 @@ maxBodyChars: 15000
# imageSelector / imageMinWidth / excludeImage / noisePrefixes # imageSelector / imageMinWidth / excludeImage / noisePrefixes
# titleStripSuffix / maxBodyChars # titleStripSuffix / maxBodyChars
# #
# 完整说明见 references/source-management.md 的 "Helper recipes" 小节。 # 完整说明见 references/source-management.md 的 "Authoring workflow" 小节。
# 写新 recipe 流程: inspect → scaffold(source-manage helper <id>) → 填字段 → preview 验证 # 写新 recipe 流程(一步生成): node scripts/inspect-source.js <url> --scroll 3 --write <id>
# → 自动测 listSelector + 推断 urlPattern,直接写好 helpers/<id>.md
# → node scripts/harvest.js <id> 3 --preview 验证
# → node scripts/recipe-test.js <id> 锁定回归测试
# (source-manage.js helper <id> 仅在想要空白模板手填时用)
...@@ -20,6 +20,11 @@ Prompt the user for (or infer from context): ...@@ -20,6 +20,11 @@ Prompt the user for (or infer from context):
| `name` | Display name | `路透社` | | `name` | Display name | `路透社` |
| `homepageUrl` | URL to scrape for links | `https://www.reuters.com/` | | `homepageUrl` | URL to scrape for links | `https://www.reuters.com/` |
| `method` | `browser` (needs JS) or `direct` (plain HTTP) | `browser` | | `method` | `browser` (needs JS) or `direct` (plain HTTP) | `browser` |
> A `browser` source that needs login (subscription/paywall): first run
> `node scripts/ensure-chromium.js --login --url <homepage>` to open a windowed
> browser and persist a logged-in profile, then set `chromium.userDataDir` in
> config.json. See `SKILL.md` → *Dedicated profile (login state)*.
| `articleUrlPattern` | Regex to match article URLs — **legacy direct sources only**; ignored when a recipe (`helpers/<id>.md`) exists | `theconversation\\.com/[\\w-]+-\\d+$` | | `articleUrlPattern` | Regex to match article URLs — **legacy direct sources only**; ignored when a recipe (`helpers/<id>.md`) exists | `theconversation\\.com/[\\w-]+-\\d+$` |
| `vaultFolder` | Subfolder under vaultPath | `路透社` | | `vaultFolder` | Subfolder under vaultPath | `路透社` |
| `enabled` | Start enabled? | `true` | | `enabled` | Start enabled? | `true` |
...@@ -70,25 +75,32 @@ recipe's selectors/scroll/filter) instead of the legacy `fetcher-browser.js`/ ...@@ -70,25 +75,32 @@ recipe's selectors/scroll/filter) instead of the legacy `fetcher-browser.js`/
### Authoring workflow (for a new browser source) ### Authoring workflow (for a new browser source)
This is the canonical 4-step flow — identical to the one in `SKILL.md` (single
source of truth, no divergence):
1. **Add the source** to config (see Add Source above). 1. **Add the source** to config (see Add Source above).
2. **Inspect the page** to discover where article links actually live: 2. **Inspect + auto-write the recipe in one shot**`inspect --write` measures
```bash `listSelector` (biggest container group) and infers `urlPattern` from real
node scripts/inspect-source.js <homepageUrl> --scroll 3 --pattern '<domain>\.com/.+' URLs, then writes `helpers/<id>.md` directly:
```
This prints links grouped by container, biggest group first, plus a
ready-to-paste helper.md skeleton.
3. **Scaffold + edit the recipe**:
```bash ```bash
node scripts/source-manage.js helper <id> # writes helpers/<id>.md template node scripts/inspect-source.js <homepageUrl> --scroll 3 --write <id>
``` ```
Fill in `listSelector` (from the biggest inspect group) and tighten (Use `--force` to overwrite an existing recipe.)
`urlPattern` (from the sample URLs inspect printed). 3. **Verify** (read-only):
4. **Verify** (read-only):
```bash ```bash
node scripts/harvest.js <id> 3 --preview node scripts/harvest.js <id> 3 --preview
``` ```
Confirm the links are real articles (not nav/section pages) and the body Confirm the links are real articles (not nav/section pages) and the body
char counts look right. Iterate on the recipe until clean. char counts look right. Iterate on the recipe until clean — see
*Troubleshooting recipes* below.
4. **Lock with regression tests** (see *Locking a stable recipe* in `SKILL.md`):
```bash
node scripts/recipe-test.js <id> # needs helpers/<id>.fixtures.json
```
> `node scripts/source-manage.js helper <id>` still scaffolds a blank template
> if you'd rather hand-author from scratch — but `inspect --write` is the
> default path and does the measuring for you.
### Frontmatter schema ### Frontmatter schema
...@@ -134,6 +146,51 @@ All fields optional except `urlPattern`; defaults reproduce legacy behaviour. ...@@ -134,6 +146,51 @@ All fields optional except `urlPattern`; defaults reproduce legacy behaviour.
> (`datePublished` / `author.name`). Many SPA sites (Nikkei) only embed metadata > (`datePublished` / `author.name`). Many SPA sites (Nikkei) only embed metadata
> there, so you usually don't need a custom selector. > there, so you usually don't need a custom selector.
### Troubleshooting recipes
The quick convergence-check table lives in `SKILL.md` → *Convergence check*.
The deeper rationale for the two most common pitfalls is here.
#### Pitfall: articles spread across multiple card containers
`inspect-source.js` reports the *single largest* container group as
`listSelector`, but on many sites (Nikkei, large news hubs) real articles are
spread across several card-container classes of similar size, and no one
container holds them all. Setting `listSelector` to just the biggest group
silently drops the rest.
Symptom: `inspect` shows ~30 article-like links on the page but
`harvest --preview` collects only ~12. Fix: write `listSelector` as a
**comma-separated composite** of every article-card class prefix, e.g.
`[class*="StreamArticleCard"], [class*="SpotlightArticleCard"], [class*="SecondaryArticleCard"]`.
Re-run `inspect` (without `--write`) to enumerate which `*ArticleCard*` classes
carry links if unsure.
Why not `listSelector: null` (whole page)? On sites with country/topic nav
dropdowns (Nikkei's `/location/<country>` links, ~30 entries), a whole-page scan
floods the `maxLinks` cap before real articles are reached. Scoping to card
containers avoids this. `inspect` without `--write` shows the container groups
so you can pick the right composite.
#### When to use `validateArticle` (and when not to)
Set `validateArticle: true` when article and section/category URLs on the site
**share a path shape** — i.e. you can't write a `urlPattern` that admits articles
and rejects section pages purely from the URL. Nikkei is the canonical case:
`/business/tech/semiconductors/<slug>` (article) and
`/business/tech/semiconductors` (section) differ only by the trailing slug, and
slug length is an unreliable signal. With `validateArticle`, `fetchPage` probes
`og:type` + JSON-LD `@type` + body length, marks the section page
`isArticle=false` (via `og:type=website` / `@type=Thing` / empty body), and
`harvest.js` skips it. This lets you use a *broad* `urlPattern` (no fragile
"slug must have N hyphens" guesswork) and still keep section pages out.
Don't set it when the URL alone cleanly separates articles from sections (e.g.
Reuters' `/world/<slug>/<date>/` vs. `/world/`) — a tight `urlPattern` is
cheaper than a per-candidate page fetch. `validateArticle` defaults to `false`,
so existing recipes are unaffected. See `helpers/nikkei-tech.md` for a worked
example with measured signals.
### Managing recipes ### Managing recipes
```bash ```bash
......
...@@ -88,6 +88,9 @@ async function waitForCdp(port, totalMs = 30000, stepMs = 500) { ...@@ -88,6 +88,9 @@ async function waitForCdp(port, totalMs = 30000, stepMs = 500) {
// ── Binary discovery ──────────────────────────────────────────────────────── // ── Binary discovery ────────────────────────────────────────────────────────
// Each candidate is { name, path }. `name` is a friendly label for --list; the
// first existing `path` wins in discoverBinary() (backward compat with harvest.js).
function macCandidates() { function macCandidates() {
const apps = [ const apps = [
['Chromium', 'Chromium.app', 'Chromium'], ['Chromium', 'Chromium.app', 'Chromium'],
...@@ -96,11 +99,12 @@ function macCandidates() { ...@@ -96,11 +99,12 @@ function macCandidates() {
['Microsoft Edge', 'Microsoft Edge.app', 'Microsoft Edge'], ['Microsoft Edge', 'Microsoft Edge.app', 'Microsoft Edge'],
]; ];
const out = []; const out = [];
for (const [, bundle, bin] of apps) { for (const [name, bundle, bin] of apps) {
out.push(`/Applications/${bundle}/Contents/MacOS/${bin}`); out.push({ name, path: `/Applications/${bundle}/Contents/MacOS/${bin}` });
out.push(path.join(os.homedir(), 'Applications', bundle, 'Contents', 'MacOS', bin)); out.push({ name, path: path.join(os.homedir(), 'Applications', bundle, 'Contents', 'MacOS', bin) });
} }
out.push('/opt/homebrew/bin/chromium', '/usr/local/bin/chromium'); out.push({ name: 'Chromium (Homebrew)', path: '/opt/homebrew/bin/chromium' });
out.push({ name: 'Chromium (Homebrew)', path: '/usr/local/bin/chromium' });
return out; return out;
} }
...@@ -112,10 +116,14 @@ function linuxCandidates() { ...@@ -112,10 +116,14 @@ function linuxCandidates() {
try { try {
const p = execFileSync('command', ['-v', n], { stdio: ['ignore', 'pipe', 'ignore'], shell: true }) const p = execFileSync('command', ['-v', n], { stdio: ['ignore', 'pipe', 'ignore'], shell: true })
.toString().trim(); .toString().trim();
if (p) found.push(p); if (p) found.push({ name: n, path: p });
} catch (e) { /* not on PATH */ } } catch (e) { /* not on PATH */ }
} }
return [...found, '/usr/bin/chromium', '/usr/bin/chromium-browser', '/snap/bin/chromium']; return [...found,
{ name: 'chromium', path: '/usr/bin/chromium' },
{ name: 'chromium-browser', path: '/usr/bin/chromium-browser' },
{ name: 'chromium (snap)', path: '/snap/bin/chromium' },
];
} }
function windowsCandidates() { function windowsCandidates() {
...@@ -123,20 +131,33 @@ function windowsCandidates() { ...@@ -123,20 +131,33 @@ function windowsCandidates() {
const pf86 = process.env['ProgramFiles(x86)'] || 'C:\\Program Files (x86)'; const pf86 = process.env['ProgramFiles(x86)'] || 'C:\\Program Files (x86)';
const local = process.env['LOCALAPPDATA'] || ''; const local = process.env['LOCALAPPDATA'] || '';
return [ return [
path.join(pf, 'Google', 'Chrome', 'Application', 'chrome.exe'), { name: 'Google Chrome', path: path.join(pf, 'Google', 'Chrome', 'Application', 'chrome.exe') },
path.join(pf86, 'Google', 'Chrome', 'Application', 'chrome.exe'), { name: 'Google Chrome', path: path.join(pf86, 'Google', 'Chrome', 'Application', 'chrome.exe') },
path.join(local, 'Google', 'Chrome', 'Application', 'chrome.exe'), { name: 'Google Chrome', path: path.join(local, 'Google', 'Chrome', 'Application', 'chrome.exe') },
path.join(pf, 'Chromium', 'Application', 'chrome.exe'), { name: 'Chromium', path: path.join(pf, 'Chromium', 'Application', 'chrome.exe') },
]; ];
} }
function discoverBinary() { function candidatesForPlatform() {
const cands = process.platform === 'darwin' ? macCandidates() return process.platform === 'darwin' ? macCandidates()
: process.platform === 'win32' ? windowsCandidates() : process.platform === 'win32' ? windowsCandidates()
: linuxCandidates(); : linuxCandidates();
for (const p of cands) { }
try { if (fs.existsSync(p) && fs.statSync(p).isFile()) return p; } catch (e) {}
// All browsers actually present on this machine (for --list / agent detection).
function discoverAll() {
const out = [];
for (const c of candidatesForPlatform()) {
try { if (fs.existsSync(c.path) && fs.statSync(c.path).isFile()) out.push(c); } catch (e) {}
} }
return out;
}
// First existing binary (backward compat — used by harvest.js + ensureChromium).
function discoverBinary() {
const found = discoverAll();
if (found.length) return found[0].path;
const cands = candidatesForPlatform().map(c => c.path);
throw new Error( throw new Error(
'Could not find a Chromium/Chrome binary. Tried:\n ' + cands.join('\n ') + 'Could not find a Chromium/Chrome binary. Tried:\n ' + cands.join('\n ') +
'\nSet config.chromium.executablePath to the full path of your browser.' '\nSet config.chromium.executablePath to the full path of your browser.'
...@@ -157,7 +178,7 @@ function cleanStaleLocks(userDataDir) { ...@@ -157,7 +178,7 @@ function cleanStaleLocks(userDataDir) {
// ── Launch ────────────────────────────────────────────────────────────────── // ── Launch ──────────────────────────────────────────────────────────────────
function spawnBrowser(bin, opts) { function spawnBrowser(bin, opts) {
const { cdpPort, headless, userDataDir } = opts; const { cdpPort, headless, userDataDir, url = 'about:blank', windowsHide = true } = opts;
fs.mkdirSync(userDataDir, { recursive: true }); fs.mkdirSync(userDataDir, { recursive: true });
cleanStaleLocks(userDataDir); cleanStaleLocks(userDataDir);
...@@ -168,17 +189,23 @@ function spawnBrowser(bin, opts) { ...@@ -168,17 +189,23 @@ function spawnBrowser(bin, opts) {
'--no-default-browser-check', '--no-default-browser-check',
'--disable-extensions', '--disable-extensions',
'--disable-gpu', '--disable-gpu',
'--no-sandbox',
]; ];
// --no-sandbox is only needed when running as root (typical in Linux/Docker
// containers, where Chromium refuses to start with the sandbox as root). On
// macOS/Windows/regular-user Linux it's unnecessary and Chromium shows an
// "unsupported command-line flag" warning banner — so omit it there.
if (typeof process.getuid === 'function' && process.getuid() === 0) {
args.push('--no-sandbox');
}
if (headless) args.push('--headless=new'); if (headless) args.push('--headless=new');
args.push('about:blank'); args.push(url);
const logPath = path.join(userDataDir, 'chromium-stderr.log'); const logPath = path.join(userDataDir, 'chromium-stderr.log');
const logFd = fs.openSync(logPath, 'a'); const logFd = fs.openSync(logPath, 'a');
const child = spawn(bin, args, { const child = spawn(bin, args, {
detached: true, detached: true,
stdio: ['ignore', 'ignore', logFd], stdio: ['ignore', 'ignore', logFd],
windowsHide: true, windowsHide,
}); });
child.on('error', (e) => { child.on('error', (e) => {
console.error(`Chromium spawn error: ${e.message}`); console.error(`Chromium spawn error: ${e.message}`);
...@@ -239,6 +266,12 @@ USAGE ...@@ -239,6 +266,12 @@ USAGE
node ensure-chromium.js Ensure Chromium is up on the CDP port node ensure-chromium.js Ensure Chromium is up on the CDP port
(launches it if not; no-op if already running) (launches it if not; no-op if already running)
node ensure-chromium.js --check Only health-check; exit 0 if up, 1 if not node ensure-chromium.js --check Only health-check; exit 0 if up, 1 if not
node ensure-chromium.js --list Detect platform + all available browsers,
print current config; exit 0
node ensure-chromium.js --login [--url <url>] [--port <port>] [--profile <dir>]
Launch a WINDOWED browser on the CDP port for
interactive login. Persisted to a fixed profile
so headless harvests keep your cookies.
node ensure-chromium.js --port 9333 Override the CDP port for this run node ensure-chromium.js --port 9333 Override the CDP port for this run
node ensure-chromium.js --help Show this help node ensure-chromium.js --help Show this help
...@@ -257,39 +290,178 @@ WHAT IT DOES ...@@ -257,39 +290,178 @@ WHAT IT DOES
5. Polls /json/version until ready (up to 30s, every 500ms). 5. Polls /json/version until ready (up to 30s, every 500ms).
6. On timeout, prints binary path / port / userDataDir / stderr-log tail. 6. On timeout, prints binary path / port / userDataDir / stderr-log tail.
--LIST
Prints the platform, every Chromium/Chrome binary found on this machine, and
the current chromium config (port / headless / executablePath / userDataDir).
Use this to report the environment to a new user before launching a login.
--LOGIN
Launches a WINDOWED browser (headless=false) with CDP enabled so the user can
log into a site interactively. The profile persists, so later headless harvests
reuse the same cookies/login state.
--url <url> Page to open (default about:blank; pass the source homepage)
--port <port> CDP port (default config.chromium.cdpPort / 9222)
--profile <dir> Profile dir (default config.chromium.userDataDir, else
~/.news-harvester/chromium-profile — a fixed path, NOT the
ephemeral tmp dir, so login state survives)
If the CDP port is already occupied (e.g. a headless harvest instance), --login
reports it and exits 0 WITHOUT launching (avoiding a SingletonLock clash).
If the profile used is the default path (config had no userDataDir), --login
prints a ready-to-paste config.json fragment so the agent/user can persist it.
CONFIG (config.json, all optional — see SKILL.md "Configuration") CONFIG (config.json, all optional — see SKILL.md "Configuration")
chromium.cdpPort CDP port (default 9222; env CDP_PORT_OVERRIDE wins) chromium.cdpPort CDP port (default 9222; env CDP_PORT_OVERRIDE wins)
chromium.headless true (default) | false (keep a window) chromium.headless true (default) | false (keep a window)
chromium.executablePath full path to browser binary (null = auto-discover) chromium.executablePath full path to browser binary (null = auto-discover)
chromium.userDataDir browser profile dir (null = <tmp>/news-harvester-chromium). chromium.userDataDir browser profile dir (null = <tmp>/news-harvester-chromium).
Set this to a fixed profile you've logged into once Set this to a fixed profile you've logged into once
(windowed, manual start) so future auto-launches keep (via --login) so future auto-launches keep your
your cookies/login state. cookies/login state.
EXIT CODES EXIT CODES
0 CDP is up (already running or just launched) 0 CDP is up (already running or just launched); --list ok; --login ok/already-running
1 CDP not up (--check), or launch failed 1 CDP not up (--check), or launch failed
EXAMPLES EXAMPLES
# Diagnose: is Chromium reachable? # Diagnose: is Chromium reachable?
node scripts/ensure-chromium.js --check node scripts/ensure-chromium.js --check
# Detect what browsers are installed (new-user onboarding step 1)
node scripts/ensure-chromium.js --list
# Launch (auto-discovers Chrome on a Mac) # Launch (auto-discovers Chrome on a Mac)
node scripts/ensure-chromium.js node scripts/ensure-chromium.js
# Open a windowed browser on the source homepage for interactive login
node scripts/ensure-chromium.js --login --url https://www.reuters.com/
# Use a non-default port for both this launcher and harvest.js # Use a non-default port for both this launcher and harvest.js
CDP_PORT_OVERRIDE=9333 node scripts/ensure-chromium.js CDP_PORT_OVERRIDE=9333 node scripts/ensure-chromium.js
CDP_PORT_OVERRIDE=9333 node scripts/harvest.js reuters 2 --preview`; CDP_PORT_OVERRIDE=9333 node scripts/harvest.js reuters 2 --preview`;
// ── --list: detect platform + available browsers + config ───────────────────
function platformLabel() {
return process.platform === 'darwin' ? 'macOS'
: process.platform === 'win32' ? 'Windows'
: 'Linux';
}
async function cmdList(config) {
const opts = resolveOptions(config, null);
console.log(`Platform: ${platformLabel()} (${process.platform})`);
const found = discoverAll();
if (found.length === 0) {
console.log('Browsers: none found in standard locations.');
console.log(' Set config.chromium.executablePath to your browser binary.');
} else {
console.log('Browsers found:');
for (const b of found) console.log(` • ${b.name}${b.path}`);
}
console.log('\nCurrent chromium config:');
console.log(` cdpPort: ${opts.cdpPort}`);
console.log(` headless: ${opts.headless}`);
console.log(` executablePath: ${opts.executablePath || '(auto-discover)'}`);
console.log(` userDataDir: ${opts.userDataDir || '(ephemeral tmp — set a fixed path for login persistence)'}`);
process.exit(0);
}
// ── --login: windowed browser for interactive login ─────────────────────────
const DEFAULT_PROFILE = path.join(os.homedir(), '.news-harvester', 'chromium-profile');
async function cmdLogin(config, flagUrl, flagPort, flagProfile) {
const chromium = (config && config.chromium) || {};
const port = parseInt(flagPort || chromium.cdpPort || '9222', 10);
// A persistent profile is MANDATORY for login; never fall back to the tmp dir.
const profile = flagProfile || chromium.userDataDir || DEFAULT_PROFILE;
const url = flagUrl || 'about:blank';
// 1. Port already occupied? Report and exit 0 (don't clash).
try {
const v = await cdpVersion(port, 1500);
console.log(`ℹ️ A browser is already running on CDP port :${port}${v.Browser}`);
console.log(' If that is a headless harvest instance, close it first (Cmd+Q / Ctrl+Q), then re-run --login.');
console.log(' (Not reusing it: it may be a different profile, and a second instance would hit a SingletonLock.)');
process.exit(0);
} catch (e) { /* port free — proceed to launch */ }
// 2. Discover binary.
const bin = chromium.executablePath || discoverBinary();
if (!fs.existsSync(bin)) {
console.error(`❌ Browser binary not found: ${bin}`);
console.error(' Set config.chromium.executablePath to the full path of your browser.');
process.exit(1);
}
// 3. Launch windowed (headless=false, window visible, open the target url).
const { pid, logPath } = spawnBrowser(bin, {
cdpPort: port, headless: false, userDataDir: profile, url, windowsHide: false,
});
// 4. Short readiness poll (15s) — catches silent launch failures.
try {
const v = await waitForCdp(port, 15000, 500);
console.log(`✅ Windowed browser launched (pid ${pid}) on :${port}${v.Browser}`);
console.log(` binary: ${bin}`);
console.log(` profile: ${profile}`);
console.log(` log: ${logPath}`);
} catch (e) {
console.error(`❌ Browser did not become ready on :${port} within 15s.`);
console.error(` binary: ${bin}`);
console.error(` profile: ${profile}`);
console.error(` log: ${logPath}`);
console.error(` stderr tail:\n${tailFile(logPath)}`);
process.exit(1);
}
// 5. Guide the user through login.
const siteHint = url !== 'about:blank' ? ` at ${url}` : '';
console.log(`\n🔐 A browser window has opened${siteHint}.`);
console.log(' 1. Log into the site inside that window (it uses the profile above).');
console.log(' 2. When done, QUIT the browser fully: Cmd+Q (macOS) / Ctrl+Q (Linux/Windows).');
console.log(' 3. The login state (cookies) is persisted in the profile, so future headless');
console.log(' harvests auto-launch with this same profile and stay logged in.');
// 6. If the profile is the default path (config had no userDataDir), prompt to persist it.
const configHadProfile = !!(chromium.userDataDir);
if (!configHadProfile) {
console.log(`\n📝 To make this profile permanent, add userDataDir to config.json's "chromium" block:`);
console.log(` "chromium": { "userDataDir": "${profile}" }`);
console.log(' (This script does not edit config.json — write the snippet above in yourself.)');
}
process.exit(0);
}
// ── CLI ─────────────────────────────────────────────────────────────────────
function flagValue(argv, name) {
const i = argv.indexOf(name);
return i >= 0 ? argv[i + 1] : null;
}
if (require.main === module) { if (require.main === module) {
const argv = process.argv.slice(2); const argv = process.argv.slice(2);
if (argv.includes('--help') || argv.includes('-h')) { if (argv.includes('--help') || argv.includes('-h')) {
console.log(HELP); console.log(HELP);
process.exit(0); process.exit(0);
} }
if (argv.includes('--list')) {
cmdList(loadConfig());
return;
}
if (argv.includes('--login')) {
cmdLogin(
loadConfig(),
flagValue(argv, '--url'),
flagValue(argv, '--port'),
flagValue(argv, '--profile')
);
return;
}
const checkOnly = argv.includes('--check'); const checkOnly = argv.includes('--check');
const portIdx = argv.indexOf('--port'); const portOverride = flagValue(argv, '--port');
const portOverride = portIdx >= 0 ? argv[portIdx + 1] : null;
(async () => { (async () => {
const config = loadConfig(); const config = loadConfig();
...@@ -322,4 +494,4 @@ if (require.main === module) { ...@@ -322,4 +494,4 @@ if (require.main === module) {
})(); })();
} }
module.exports = { ensureChromium, cdpVersion, resolveOptions, discoverBinary }; module.exports = { ensureChromium, cdpVersion, resolveOptions, discoverBinary, discoverAll };
...@@ -75,9 +75,11 @@ function extractArticle(html, url) { ...@@ -75,9 +75,11 @@ function extractArticle(html, url) {
let title = titleMatch ? titleMatch[1].trim() : ''; let title = titleMatch ? titleMatch[1].trim() : '';
title = title.replace(/ \| The Conversation| \| Reuters/g, '').trim(); title = title.replace(/ \| The Conversation| \| Reuters/g, '').trim();
// Date // Date — capture the full ISO 8601 timestamp (incl. fractional seconds and
const dateMatch = html.match(/["']datePublished["']\s*:\s*["']([\d\-T:Z+]+)["']/) || // timezone offsets like -05:00 / +09:00). The earlier [\d\-T:Z+] class missed
html.match(/<time[^>]+datetime=["']([\d\-T:Z+]+)["']/i); // '.' and the second '-' / '+' in offsets, truncating those values.
const dateMatch = html.match(/["']datePublished["']\s*:\s*["']([^"']+)["']/) ||
html.match(/<time[^>]+datetime=["']([^"']+)["']/i);
const date = dateMatch ? dateMatch[1] : ''; const date = dateMatch ? dateMatch[1] : '';
// Authors // Authors
......
...@@ -130,7 +130,11 @@ function main() { ...@@ -130,7 +130,11 @@ function main() {
} }
const sentinel = sentinelFor(slug); const sentinel = sentinelFor(slug);
const summaryFilePath = path.join(dayDir, `${titleZh}.md`); // Summary docs live in a `summary/` subdirectory (sibling of `assets/`).
// Wikilinks stay bare-title ([[中文标题]]); Obsidian resolves them by basename.
const summaryDir = path.join(dayDir, 'summary');
fs.mkdirSync(summaryDir, { recursive: true });
const summaryFilePath = path.join(summaryDir, `${titleZh}.md`);
const origFilePath = path.join(dayDir, origName || originalFilename(titleEn, sourceName)); const origFilePath = path.join(dayDir, origName || originalFilename(titleEn, sourceName));
// 1. Write the Chinese summary doc // 1. Write the Chinese summary doc
...@@ -146,11 +150,21 @@ function main() { ...@@ -146,11 +150,21 @@ function main() {
}, sourceName); }, sourceName);
fs.writeFileSync(summaryFilePath, summaryDoc); fs.writeFileSync(summaryFilePath, summaryDoc);
// 2. Patch the sentinel in the original doc (frontmatter related + footer wikilink) // 2. Patch the original doc: replace the title sentinel, then overwrite the
// frontmatter `tags:` block and `category:` line with the agent-extracted
// values so 原文.md carries the same tags/category as the summary doc.
if (fs.existsSync(origFilePath)) { if (fs.existsSync(origFilePath)) {
const orig = fs.readFileSync(origFilePath, 'utf8'); const orig = fs.readFileSync(origFilePath, 'utf8');
// Replace all occurrences of the sentinel with the real title. // Replace all occurrences of the sentinel with the real title.
const patchedOrig = orig.split(sentinel).join(titleZh); let patchedOrig = orig.split(sentinel).join(titleZh);
// Overwrite the tags block (tags:\n - x\n - y\n) with agent-extracted tags.
const finalTags = (summary.tags && summary.tags.length) ? summary.tags : article.tags;
const newTagsYaml = (finalTags || []).map(t => ` - ${t}`).join('\n');
patchedOrig = patchedOrig.replace(/(^tags:\n)(?: - .*\n)+/m, `$1${newTagsYaml}\n`);
// Overwrite the category line if the agent supplied one.
if (summary.category) {
patchedOrig = patchedOrig.replace(/^category: .*/m, `category: ${summary.category}`);
}
fs.writeFileSync(origFilePath, patchedOrig); fs.writeFileSync(origFilePath, patchedOrig);
} else { } else {
console.error(` ⚠️ Original doc missing: ${origName} — summary written but link unpatched`); console.error(` ⚠️ Original doc missing: ${origName} — summary written but link unpatched`);
...@@ -175,6 +189,7 @@ function main() { ...@@ -175,6 +189,7 @@ function main() {
// Index links resolve to summary docs where a titleZh exists, else sentinel. // Index links resolve to summary docs where a titleZh exists, else sentinel.
const indexArticles = registry.articles.map(a => ({ const indexArticles = registry.articles.map(a => ({
titleZh: a.titleZh, titleZh: a.titleZh,
titleEn: a.titleEn || '',
date: a.date || '', date: a.date || '',
category: a.category || '国际', category: a.category || '国际',
tags: a.tags || [] tags: a.tags || []
......
...@@ -56,8 +56,39 @@ function todayStr() { ...@@ -56,8 +56,39 @@ function todayStr() {
return `${d.getFullYear()}${String(d.getMonth()+1).padStart(2,'0')}${String(d.getDate()).padStart(2,'0')}`; return `${d.getFullYear()}${String(d.getMonth()+1).padStart(2,'0')}${String(d.getDate()).padStart(2,'0')}`;
} }
// Return the full ISO 8601 timestamp for frontmatter `date:` fields. Storing
// the complete timestamp (not a truncated YYYY-MM-DD) is lossless and lets
// Obsidian render it in any timezone — truncating the raw UTC string caused
// off-by-one errors for articles published near UTC midnight.
function dateShort(dateStr) { function dateShort(dateStr) {
return (dateStr || '').substring(0, 10) || todayStr().replace(/(\d{4})(\d{2})(\d{2})/, '$1-$2-$3'); return (dateStr && dateStr.trim()) || todayStr().replace(/(\d{4})(\d{2})(\d{2})/, '$1-$2-$3');
}
// Compact YYYY-MM-DD for the index.md table cell (full ISO is too wide there).
// Falls back to the raw string if it isn't a valid date.
function dateCell(dateStr) {
if (!dateStr) return '';
const d = new Date(dateStr);
if (isNaN(d.getTime())) return dateStr.substring(0, 10);
const y = d.getFullYear();
const m = String(d.getMonth() + 1).padStart(2, '0');
const day = String(d.getDate()).padStart(2, '0');
return `${y}-${m}-${day}`;
}
// Format a published timestamp for display in the Shanghai timezone (UTC+8),
// e.g. "2026-07-31 05:47". Used in the info callouts of 原文.md / 摘要.md.
function dateShanghai(dateStr) {
if (!dateStr) return '—';
const d = new Date(dateStr);
if (isNaN(d.getTime())) return dateStr;
const parts = new Intl.DateTimeFormat('en-CA', {
timeZone: 'Asia/Shanghai',
year: 'numeric', month: '2-digit', day: '2-digit',
hour: '2-digit', minute: '2-digit', hour12: false
}).formatToParts(d);
const get = t => (parts.find(p => p.type === t) || {}).value || '';
return `${get('year')}-${get('month')}-${get('day')} ${get('hour')}:${get('minute')}`;
} }
function getCategory(url, title) { function getCategory(url, title) {
...@@ -162,6 +193,7 @@ title: "${titleZh}" ...@@ -162,6 +193,7 @@ title: "${titleZh}"
date: ${dateShort(date)} date: ${dateShort(date)}
tags: tags:
${tagsYaml} ${tagsYaml}
category: ${category}
source: "${url}" source: "${url}"
publisher: ${sourceName} publisher: ${sourceName}
authors: "${authors || ''}" authors: "${authors || ''}"
...@@ -173,7 +205,8 @@ saved: ${todayStr().replace(/(\d{4})(\d{2})(\d{2})/, '$1-$2-$3')} ...@@ -173,7 +205,8 @@ saved: ${todayStr().replace(/(\d{4})(\d{2})(\d{2})/, '$1-$2-$3')}
# ${titleZh} # ${titleZh}
> [!info] 文章信息 > [!info] 文章信息
> **发布:** ${dateShort(date)} | **来源:** ${sourceName} > **发布:** ${dateShanghai(date)}
> **来源:** ${sourceName}
> **作者:** ${authors || '—'} > **作者:** ${authors || '—'}
> **原文:** [[${origName}]] > **原文:** [[${origName}]]
...@@ -188,7 +221,7 @@ ${data.summaryZh || '(摘要生成中)'} ...@@ -188,7 +221,7 @@ ${data.summaryZh || '(摘要生成中)'}
} }
function generateOriginalDoc(data, sourceName, imgFiles) { function generateOriginalDoc(data, sourceName, imgFiles) {
const { titleEn, titleZh, date, authors, url, body } = data; const { titleEn, titleZh, date, authors, url, body, category } = data;
const tags = (data.tags || []).map(t => ` - ${t}`).join('\n'); const tags = (data.tags || []).map(t => ` - ${t}`).join('\n');
// Empty-alt guard: only render the caption line when alt text exists, so we // Empty-alt guard: only render the caption line when alt text exists, so we
// never emit a bare `**` (empty bold) for avatar/tracking images. // never emit a bare `**` (empty bold) for avatar/tracking images.
...@@ -204,6 +237,7 @@ title: "${titleEn}" ...@@ -204,6 +237,7 @@ title: "${titleEn}"
date: ${dateShort(date)} date: ${dateShort(date)}
tags: tags:
${tags} ${tags}
category: ${category}
source: "${url}" source: "${url}"
publisher: ${sourceName} publisher: ${sourceName}
authors: "${authors || ''}" authors: "${authors || ''}"
...@@ -215,7 +249,8 @@ saved: ${todayStr().replace(/(\d{4})(\d{2})(\d{2})/, '$1-$2-$3')} ...@@ -215,7 +249,8 @@ saved: ${todayStr().replace(/(\d{4})(\d{2})(\d{2})/, '$1-$2-$3')}
# ${titleEn} # ${titleEn}
> [!info] Source > [!info] Source
> **Published:** ${dateShort(date)} | **URL:** ${url} > **Published:** ${dateShanghai(date)}
> **URL:** ${url}
> **中文摘要:** [[${titleZh}]] > **中文摘要:** [[${titleZh}]]
--- ---
...@@ -258,7 +293,7 @@ function updateIndex(dayDir, articles, sourceName) { ...@@ -258,7 +293,7 @@ function updateIndex(dayDir, articles, sourceName) {
articles.forEach(a => { catCount[a.category] = (catCount[a.category] || 0) + 1; }); articles.forEach(a => { catCount[a.category] = (catCount[a.category] || 0) + 1; });
const catRows = Object.entries(catCount).map(([c, n]) => `| ${CATEGORY_EMOJI[c] || '📰'} ${c} | ${n} |`).join('\n'); const catRows = Object.entries(catCount).map(([c, n]) => `| ${CATEGORY_EMOJI[c] || '📰'} ${c} | ${n} |`).join('\n');
const tableRows = articles.map((a, i) => const tableRows = articles.map((a, i) =>
`| ${i+1} | [[${a.titleZh}]] | ${a.category} | ${(a.tags||[]).slice(0,3).join(', ')} | ${dateShort(a.date)} | ✅ |` `| ${i+1} | [[${a.titleZh}]] | ${a.titleEn || ''} | ${a.category} | ${(a.tags||[]).slice(0,3).join(', ')} | ${dateCell(a.date)} | ✅ |`
).join('\n'); ).join('\n');
const catNav = Object.entries(catCount).map(([cat, n]) => { const catNav = Object.entries(catCount).map(([cat, n]) => {
...@@ -293,8 +328,8 @@ ${catRows} ...@@ -293,8 +328,8 @@ ${catRows}
## 📋 文章列表 ## 📋 文章列表
| # | 标题 | 分类 | 标签 | 发布时间 | 状态 | | # | 标题 | 原文标题 | 分类 | 标签 | 发布时间 | 状态 |
|---|------|------|------|----------|------| |---|------|----------|------|------|----------|------|
${tableRows} ${tableRows}
--- ---
...@@ -451,17 +486,25 @@ Examples: ...@@ -451,17 +486,25 @@ Examples:
// Normalize URLs (strip query params for dedup) // Normalize URLs (strip query params for dedup)
const normalizeUrl = (u) => { try { const p = new URL(u); p.search = ''; return p.toString(); } catch(e) { return u; } }; const normalizeUrl = (u) => { try { const p = new URL(u); p.search = ''; return p.toString(); } catch(e) { return u; } };
const normalizedLinks = [...new Set(links.map(u => normalizeUrl(u)))]; const normalizedLinks = [...new Set(links.map(u => normalizeUrl(u)))];
const newLinks = normalizedLinks.filter(u => !seen.has(u)).slice(0, count); // Keep ALL new candidates rather than slicing to `count`: recipe sources with
// validateArticle reject section/category pages only at fetch time, so capping
// candidates here would shortchange the article target (section pages mixed
// into the first `count` candidates eat slots the buffer was meant to backfill).
// The fetcher already capped collection at desiredCount (count + seen + buffer
// ≤ MAX_CANDIDATE_LINKS), so this list is bounded; the fetch loops below stop
// once `count` articles are successfully archived.
const newLinks = normalizedLinks.filter(u => !seen.has(u));
const skipped = normalizedLinks.filter(u => seen.has(u)).length; const skipped = normalizedLinks.filter(u => seen.has(u)).length;
console.error(` New: ${newLinks.length} | Already seen: ${skipped}`); console.error(` New: ${newLinks.length} candidates | Already seen: ${skipped}`);
if (!preview && newLinks.length < count) { if (!preview && newLinks.length < count) {
console.error(` ⚠️ Only ${newLinks.length} new of ${count} requested — stream may be exhausted (fewer fresh articles than asked). Re-running won't help; wait for new articles or broaden the source.`); console.error(` ⚠️ Only ${newLinks.length} new candidates of ${count} requested — stream may be exhausted (fewer fresh articles than asked). Re-running won't help; wait for new articles or broaden the source.`);
} }
// ── Preview mode: fetch only, print compact list, write nothing ─────────── // ── Preview mode: fetch only, print compact list, write nothing ───────────
if (preview) { if (preview) {
const out = []; const out = [];
for (let i = 0; i < newLinks.length; i++) { for (let i = 0; i < newLinks.length; i++) {
if (out.length >= count) break; // cap on successful articles, not candidates
const url = newLinks[i]; const url = newLinks[i];
console.error(`\n[${i+1}/${newLinks.length}] ${url}`); console.error(`\n[${i+1}/${newLinks.length}] ${url}`);
let articleData; let articleData;
...@@ -521,6 +564,7 @@ Examples: ...@@ -521,6 +564,7 @@ Examples:
const processed = []; const processed = [];
for (let i = 0; i < newLinks.length; i++) { for (let i = 0; i < newLinks.length; i++) {
if (processed.length >= count) break; // cap on successful articles, not candidates
const url = newLinks[i]; const url = newLinks[i];
console.error(`\n[${i+1}/${newLinks.length}] ${url}`); console.error(`\n[${i+1}/${newLinks.length}] ${url}`);
...@@ -545,7 +589,7 @@ Examples: ...@@ -545,7 +589,7 @@ Examples:
const { title: titleEn, date, authors, imgs = [], body } = articleData; const { title: titleEn, date, authors, imgs = [], body } = articleData;
const slug = slugFromUrl(url); const slug = slugFromUrl(url);
const category = getCategory(url, titleEn); const category = getCategory(url, titleEn);
const tags = [category, source.name]; const tags = [source.name]; // placeholder; finalize overwrites with agent-extracted tags
const titleZh = sentinelFor(slug); // patched by finalize.js const titleZh = sentinelFor(slug); // patched by finalize.js
const origName = originalFilename(titleEn, source.name); const origName = originalFilename(titleEn, source.name);
...@@ -571,7 +615,7 @@ Examples: ...@@ -571,7 +615,7 @@ Examples:
// Append to registry (dedup record, written immediately so a crash keeps it) // Append to registry (dedup record, written immediately so a crash keeps it)
appendRegistry(config, source, today, { appendRegistry(config, source, today, {
url, titleZh, titleEn, processedAt: new Date().toISOString(), url, titleZh, titleEn, date, processedAt: new Date().toISOString(),
category, tags, slug, originalFilename: origName category, tags, slug, originalFilename: origName
}); });
...@@ -583,6 +627,10 @@ Examples: ...@@ -583,6 +627,10 @@ Examples:
await sleep(1000); await sleep(1000);
} }
if (!preview && processed.length < count) {
console.error(` ⚠️ Archived ${processed.length} of ${count} requested remaining candidates were non-article/low-content pages; stream ran out of real articles. Re-running won't help.`);
}
// Compact manifest for the agent: NO full body, only a preview slice. // Compact manifest for the agent: NO full body, only a preview slice.
const newArticles = processed.map(a => ({ const newArticles = processed.map(a => ({
url: a.url, url: a.url,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment