Commit c95a5227 authored by 谢宇轩's avatar 谢宇轩

feat: robust cross-platform Chromium launcher + skill flow cleanup

Chromium launch was fragile: the documented `chromium ... & sleep 5` assumed
`chromium` was on PATH, but on macOS the binary lives inside an .app bundle
and the command silently failed (&>/dev/null swallowed the error). sleep 5
was not a readiness check, getWsUrl() had zero retries, and the health check
depended on python3.

Changes:
- Add scripts/ensure-chromium.js: cross-platform binary discovery (macOS
  .app bundles, Linux PATH, Windows install dirs), stale-SingletonLock
  cleanup, spawn with CDP, poll /json/version until ready (30s), and a
  diagnostic stderr-log tail on timeout. Pure node http, no python3.
  Supports config.chromium.{cdpPort,headless,executablePath,userDataDir}.
- harvest.js: auto-call ensureChromium before browser-source fetch (no-op
  if already running). Sources needing login can set chromium.userDataDir
  to a logged-in profile; auto-launches then carry the cookies.
- fetcher-browser.js: getWsUrl() now retries (6x500ms) and reads the CDP
  port from config (env > config.chromium.cdpPort > 9222).
- config-template.json: add optional chromium block.
- SKILL.md / README.md: replace the manual launch command with the
  auto-launch explanation; add --help docs and dedicated-profile guidance.

Also clean up the skill flow to cut a 16-step run down to ~4:
- All commands now use the `scripts/` prefix (bare `node harvest.js` failed
  with MODULE_NOT_FOUND — 5x in the trace).
- First-Run Setup becomes a single pre-flight gate: print OK -> skip to
  Workflow, don't re-check vault/source/config mid-task.
- Workflow adds a "stay lean" note: if the user named a source, don't run
  source-manage.js list; the archive flow is exactly 3 script calls.
parent eb5973f6
......@@ -64,6 +64,7 @@ news-harvester/
│ ├── harvest.js # 主入口:node harvest.js <source_id> <count> [--preview] [--full]
│ ├── finalize.js # 补全中文摘要:node finalize.js <dayDir>
│ ├── source-manage.js # 数据源增删改查
│ ├── ensure-chromium.js # 跨平台自动发现并启动 Chromium(browser 源用)
│ ├── fetcher-direct.js # 纯 HTTP 抓取(开放站点)
│ └── fetcher-browser.js # Chromium CDP 抓取(JS 渲染站点)
└── data/ # ⚠️ 运行时去重数据,不入库
......@@ -78,7 +79,14 @@ news-harvester/
- [Node.js](https://nodejs.org/) ≥ 18
- 一个可写的 Obsidian vault 目录
- (可选)Chromium —— 仅当使用 `browser` 方法的源(如 reuters)时需要
- (可选)Chromium / Google Chrome / Edge —— 仅当使用 `browser` 方法的源(如 reuters)时需要。脚本会跨平台自动发现已安装的浏览器(macOS `.app`、Linux PATH、Windows 安装目录),推荐手动启动,指定profile 目录和端口
```sh
"/Applications/Chromium.app/Contents/MacOS/Chromium" \
--remote-debugging-port=9222 \
--user-data-dir="$HOME/.news-harvester/chromium-profile" \
--no-first-run --no-default-browser-check &
```
### 1. 克隆并安装依赖
......@@ -240,11 +248,7 @@ YYYYMMDD/
- **正文提取依赖正则**:网站改版可能导致提取失败(正文 < 100 字会被跳过)
- **中文标题/摘要由 Agent 生成**`harvest.js` 在原文/registry/index 中用哨兵 `__SUMMARY_<slug>__` 占位,`finalize.js` 读取 Agent 写的 `.summaries.json` 后替换为真实标题并生成摘要文档
- **browser 方法依赖 Chromium CDP**(端口 9222),需预先启动:
```bash
chromium --no-sandbox --remote-debugging-port=9222 \
--user-data-dir=/tmp/chromium-harvest --disable-gpu "about:blank" &>/dev/null &
```
- **browser 方法自动启动 Chromium**`harvest.js` 在抓取前调用 `ensure-chromium.js`,自动发现并启动浏览器(跨平台:macOS `.app` bundle、Linux PATH、Windows 安装目录),默认 `headless=new`,轮询 `/json/version` 直到就绪。无需手动 `chromium ... &`。若发现失败可在 `config.json``chromium.executablePath` 指定路径;调试用 `node scripts/ensure-chromium.js`(失败时打印二进制路径/端口/stderr 日志尾)。需要登录态的源:设 `chromium.userDataDir` 指向一个登录过的固定 profile,后续自动启动即带 cookie/环境。配置项见 SKILL.md 的 Configuration
- **图片下载无重试**:失败仅跳过
- **Token 卫生**:归档任务为「harvest → 写 .summaries.json → finalize」三步;不要为同一源/日重跑 harvest 仅为"多取几篇"(registry 已去重,二次跑只取真正的新文章并安全合并进待处理 manifest);不要用 Read 反复读原文大文件——manifest 里的 `bodyPreview`(~4000 字)足够写摘要
......
......@@ -11,15 +11,16 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau
| Task | Command |
|------|---------|
| Archive latest articles | `node harvest.js <source_id> <count>` |
| Preview latest (read-only) | `node harvest.js <source_id> <count> --preview` |
| Preview with full body | `node harvest.js <source_id> <count> --preview --full` |
| Finalize Chinese summaries | `node finalize.js <dayDir>` |
| Add source | `node source-manage.js add` |
| List sources | `node source-manage.js list` |
| Edit source | `node source-manage.js edit <id>` |
| Archive latest articles | `node scripts/harvest.js <source_id> <count>` |
| Preview latest (read-only) | `node scripts/harvest.js <source_id> <count> --preview` |
| Preview with full body | `node scripts/harvest.js <source_id> <count> --preview --full` |
| Finalize Chinese summaries | `node scripts/finalize.js "<dayDir>"` |
| Add source | `node scripts/source-manage.js add` |
| List sources | `node scripts/source-manage.js list` |
| Edit source | `node scripts/source-manage.js edit <id>` |
| Ensure Chromium (browser sources) | `node scripts/ensure-chromium.js` (or `--check`) |
> Scripts are invoked from the skill root directory (`scripts/` is relative to this skill's root).
> **All commands assume `cd <skill-root>` first** (the directory containing this `SKILL.md`). Scripts live in the `scripts/` subfolder — **always prefix with `scripts/`**, e.g. `node scripts/harvest.js`, never `node harvest.js` (that fails with `MODULE_NOT_FOUND`).
**Division of labor:** scripts do all deterministic I/O (fetch, parse, download images, write 原文.md / registry / index). The agent does only the intelligent part — Chinese translation + 4-section summaries — and hands them back as a JSON file for `finalize.js` to stitch in. The full article body is written to disk by the script and **never enters the agent's context**.
......@@ -29,17 +30,15 @@ Config file: `config.json` (auto-generated from `references/config-template.json
## First-Run Setup
The agent MUST proactively check configuration state at the start of **any** news-harvester task. If unconfigured, run this onboarding **before** doing anything else — do not wait for the user to ask. This is the single biggest cause of silent failures (articles written into a non-existent `/path/to/...` dir).
**Detect unconfigured state** — either condition triggers onboarding:
- `config.json` does not exist, OR
- `vaultPath` still equals the template placeholder `/path/to/your/obsidian/vault`
**Pre-flight gate — run this ONE command at the start of a task.** If it prints `OK`, the skill is already configured: **skip straight to Workflow** and do not re-check vault/source/config. Only if it prints `UNCONFIGURED` do the onboarding below.
```bash
node -e "const c=require('./config.json'); console.log(c.vaultPath==='/path/to/your/obsidian/vault'?'UNCONFIGURED':'OK')" 2>/dev/null || echo "UNCONFIGURED (no config.json)"
```
### Onboarding steps
This is the only config check needed per run — don't re-verify vault writability or re-list sources mid-task.
### Onboarding (only if UNCONFIGURED)
**1. Set the Obsidian vault path.** Ask the user for the absolute path to their Obsidian vault root (e.g. `~/obsidian/MyVault`). Verify it exists and is writable; create it if they intend a fresh folder:
```bash
......@@ -47,27 +46,25 @@ mkdir -p "<vaultPath>" && test -w "<vaultPath>" && echo "writable"
```
Then write the real path into `config.json` (replace the placeholder). Never leave the placeholder in place.
**2. Confirm the first test source.** The template ships `the-conversation` (method `direct`, no Chromium needed) — the ideal first test source. `node source-manage.js list` should show it enabled. If absent, add it:
**2. Confirm the first test source.** The template ships `the-conversation` (method `direct`, no Chromium needed) — the ideal first test source. `node scripts/source-manage.js list` should show it enabled. If absent, add it:
```bash
node source-manage.js add --id the-conversation --name "The Conversation" \
node scripts/source-manage.js add --id the-conversation --name "The Conversation" \
--url https://theconversation.com/us --method direct --folder "The Conversation"
```
If the user wants a different first source (e.g. `reuters` with `browser` method), add that instead and note the Chromium prerequisite below applies.
**3. Check prerequisites for the source's `method`.**
- `direct` → nothing extra.
- `browser` → Chromium must be running on port 9222:
```bash
curl -s http://localhost:9222/json/version 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK:', d.get('Browser',''))"
```
If not running:
- `browser``harvest.js` auto-launches Chromium on first browser-source use (see `scripts/ensure-chromium.js`). It cross-platform-discovers the binary (macOS `.app` bundles, Linux PATH, Windows install dirs) — no manual `chromium ... &` needed. To verify it can find a browser without running a full harvest:
```bash
chromium --no-sandbox --remote-debugging-port=9222 --user-data-dir=/tmp/chromium-harvest --disable-gpu "about:blank" &>/dev/null & sleep 5
node scripts/ensure-chromium.js --check # is CDP already up? exit 0/1
node scripts/ensure-chromium.js # launch + readiness poll (no-op if already up)
```
If discovery can't find your browser, set `config.chromium.executablePath` to its full path (see Configuration).
**4. Smoke test (read-only, no vault writes).** Confirm the fetch pipeline works end-to-end before archiving anything:
```bash
node harvest.js the-conversation 2 --preview
node scripts/harvest.js the-conversation 2 --preview
```
If this returns article data, setup is complete. If it errors (empty list, parse failure, network), fix the source config before proceeding — do not move on to a full archive.
......@@ -75,7 +72,7 @@ If this returns article data, setup is complete. If it errors (empty list, parse
### After onboarding
Once `vaultPath` is real and the smoke test passes, **skip this section** on future runs and go straight to Workflow. The check is idempotent and cheap — re-running `node source-manage.js list` is always safe.
Once `vaultPath` is real and the smoke test passes, **skip this section** on future runs and go straight to Workflow. The check is idempotent and cheap — re-running `node scripts/source-manage.js list` is always safe.
---
......@@ -89,6 +86,32 @@ Before running a harvest, the agent MUST verify/edit these fields:
|-------|-----------|-------------|
| `vaultPath` | ✅ Yes | Absolute path to the Obsidian vault root. Must exist and be writable. Replace the template placeholder `/path/to/your/obsidian/vault`. |
| `registryWindowDays` | Optional | Dedup sliding window in days (default `7`). |
| `chromium.cdpPort` | Optional | CDP debug port (default `9222`). Only relevant for `browser` sources. |
| `chromium.headless` | Optional | Launch headless (`true`, default) or with a window (`false`). Set `false` if a site's anti-bot detects headless. |
| `chromium.executablePath` | Optional | Full path to the browser binary. Set only if auto-discovery fails (non-standard install location). `null` = auto-discover. |
| `chromium.userDataDir` | Optional | Browser profile dir (default `<os.tmpdir>/news-harvester-chromium`). Set to a fixed profile you've logged into once so auto-launched instances keep your cookies. See *Dedicated profile (login state)* below. |
#### Dedicated profile (login state)
For sources that need cookies/login (subscriptions, paywalled sites), point `chromium.userDataDir` at a fixed profile and log into it once. After that, every Chromium `ensure-chromium` auto-launches loads that profile — your cookies/environment carry over:
```bash
# 1. First time: start Chromium windowed with the fixed profile, log in to the source.
# Use the SAME path you'll put in config.json (expand $HOME — JSON doesn't).
"/Applications/Chromium.app/Contents/MacOS/Chromium" \
--remote-debugging-port=9222 \
--user-data-dir="$HOME/.news-harvester/chromium-profile" \
--no-first-run --no-default-browser-check &
# Log in inside the window, then quit. The profile persists on disk.
# 2. In config.json (absolute path, no $HOME):
# "chromium": { "userDataDir": "/Users/<you>/.news-harvester/chromium-profile" }
# 3. Later runs: harvest.js auto-launches Chromium with this profile —
# headless by default, login state intact. No manual start needed.
```
On Linux replace the binary path with `chromium` (or your distro's name). If a site's anti-bot flags headless, set `chromium.headless: false`.
To point a source at a different vault subfolder, set `sources[].vaultFolder`. Articles are archived to `<vaultPath>/<vaultFolder>/YYYYMMDD/`.
......@@ -98,11 +121,13 @@ To point a source at a different vault subfolder, set `sources[].vaultFolder`. A
Two flows depending on intent: **archive** (fetch + save to Obsidian) or **preview** (read-only, just return article data).
> **Stay lean.** If the user already named a source (e.g. "reuters"), use it directly — do **not** run `source-manage.js list` first. The archive flow is exactly **3 script calls**: `harvest.js` → write `.summaries.json` → `finalize.js`. The pre-flight gate above is the only config check; don't re-verify the vault or re-list sources mid-flow.
### Flow A — Archive to Obsidian (3 steps)
#### Step 1. Run harvest (script writes originals + registry + index)
```bash
node harvest.js <source_id> <count>
node scripts/harvest.js <source_id> <count>
```
The script fetches links, deduplicates, fetches each article, downloads images, and **writes to disk itself**:
- `<英文标题> - <SourceName>原文.md` (full body + images, with a sentinel placeholder for the Chinese title link)
......@@ -112,11 +137,7 @@ The script fetches links, deduplicates, fetches each article, downloads images,
It prints a **compact manifest** to stdout (no full body — only a `bodyPreview` ~4000 chars per article). Read it.
> **Prerequisite** for `browser` sources: Chromium must be running on port 9222:
> ```bash
> curl -s http://localhost:9222/json/version 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK:', d.get('Browser',''))"
> ```
> If not running: `chromium --no-sandbox --remote-debugging-port=9222 --user-data-dir=/tmp/chromium-harvest --disable-gpu "about:blank" &>/dev/null & sleep 5`
> **Browser sources** are auto-handled: before fetching, `harvest.js` calls `ensure-chromium.js`, which health-checks the CDP port and launches Chromium itself (cross-platform binary discovery, headless by default) if it isn't up — a no-op if already running. No manual `chromium ... &` or `sleep` is needed. To debug launch failures: `node scripts/ensure-chromium.js` (prints binary path, port, and stderr-log tail on failure).
#### Step 2. Generate Chinese summaries → write `.summaries.json`
From the compact manifest, for each article write one entry to `<dayDir>/.summaries.json`:
......@@ -140,7 +161,7 @@ From the compact manifest, for each article write one entry to `<dayDir>/.summar
#### Step 3. Run finalize (script stitches summaries in)
```bash
node finalize.js "<dayDir>"
node scripts/finalize.js "<dayDir>"
```
This writes each `<中文标题>.md`, patches the sentinel links in the 原文.md files, updates `registry.json` + `index.md` with real titles, and removes `.manifest.json` + `.summaries.json`.
......@@ -149,8 +170,8 @@ This writes each `<中文标题>.md`, patches the sentinel links in the 原文.m
For "返回最新文章数据" / "看看最新文章" requests where the user does **not** ask to save:
```bash
node harvest.js <source_id> <count> --preview # titles + metadata + bodyPreview
node harvest.js <source_id> <count> --preview --full # ... + full body
node scripts/harvest.js <source_id> <count> --preview # titles + metadata + bodyPreview
node scripts/harvest.js <source_id> <count> --preview --full # ... + full body
```
No files are written. stdout is a compact JSON list (one call, no re-fetching / re-parsing). Use `--full` only when the user explicitly wants the complete article text.
......
{
"vaultPath": "/path/to/your/obsidian/vault",
"registryWindowDays": 7,
"chromium": {
"cdpPort": 9222,
"headless": true,
"executablePath": null,
"userDataDir": null
},
"sources": [
{
"id": "the-conversation",
......
#!/usr/bin/env node
/**
* ensure-chromium.js - Robust cross-platform Chromium launcher for CDP
*
* Why this exists: the old documented command `chromium ... & sleep 5` assumed
* `chromium` was on PATH. On macOS, Chromium/Chrome ship as .app bundles whose
* real binary lives in /Applications/<Browser>.app/Contents/MacOS/<Binary>, so
* `chromium` is not found and the command silently fails (&>/dev/null swallows
* the error). `sleep 5` is also not a readiness check — a slow first launch or
* a crash leaves you waiting 5s for nothing.
*
* This script:
* 1. Health-checks the CDP port (pure node http, no python3 dependency).
* 2. If not up, discovers the browser binary per-platform (or uses
* config.chromium.executablePath).
* 3. Cleans stale SingletonLock files left by a previous crash.
* 4. Spawns the browser with CDP enabled (headless=new by default, configurable).
* 5. Polls /json/version until ready (up to 30s).
* 6. On timeout, dumps the browser's stderr log tail for diagnosis.
*
* CLI:
* node ensure-chromium.js # ensure running (no-op if already up)
* node ensure-chromium.js --check # only check, exit 0 if up / 1 if not
* node ensure-chromium.js --port 9333 # override CDP port for this run
*
* Used programmatically by harvest.js for browser-method sources.
*/
const fs = require('fs');
const path = require('path');
const http = require('http');
const os = require('os');
const { spawn, execFileSync } = require('child_process');
const SKILL_DIR = path.dirname(__dirname);
const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
// ── Config ──────────────────────────────────────────────────────────────────
function loadConfig() {
if (!fs.existsSync(CONFIG_PATH)) {
const template = JSON.parse(fs.readFileSync(TEMPLATE_PATH, 'utf8'));
fs.writeFileSync(CONFIG_PATH, JSON.stringify(template, null, 2));
}
return JSON.parse(fs.readFileSync(CONFIG_PATH, 'utf8'));
}
function resolveOptions(config, portOverride) {
const chromium = (config && config.chromium) || {};
const cdpPort = parseInt(portOverride || process.env.CDP_PORT_OVERRIDE || chromium.cdpPort || '9222', 10);
const headless = chromium.headless !== false; // default true, set false to keep a window
const userDataDir = chromium.userDataDir || path.join(os.tmpdir(), 'news-harvester-chromium');
return { cdpPort, headless, userDataDir, executablePath: chromium.executablePath || null };
}
// ── CDP health check (no python3) ───────────────────────────────────────────
function cdpVersion(port, timeoutMs = 1500) {
return new Promise((resolve, reject) => {
const req = http.get(`http://localhost:${port}/json/version`, (res) => {
let data = '';
res.on('data', (d) => (data += d));
res.on('end', () => {
try { resolve(JSON.parse(data)); }
catch (e) { reject(new Error(`bad /json/version response: ${data.slice(0, 120)}`)); }
});
});
req.on('error', reject);
req.setTimeout(timeoutMs, () => { req.destroy(); reject(new Error('health-check timeout')); });
});
}
async function waitForCdp(port, totalMs = 30000, stepMs = 500) {
const deadline = Date.now() + totalMs;
let lastErr;
while (Date.now() < deadline) {
try {
const v = await cdpVersion(port, 1500);
return v;
} catch (e) {
lastErr = e;
await new Promise((r) => setTimeout(r, stepMs));
}
}
throw new Error(`CDP not ready after ${totalMs / 1000}s (last: ${lastErr && lastErr.message})`);
}
// ── Binary discovery ────────────────────────────────────────────────────────
function macCandidates() {
const apps = [
['Chromium', 'Chromium.app', 'Chromium'],
['Google Chrome', 'Google Chrome.app', 'Google Chrome'],
['Brave Browser', 'Brave Browser.app', 'Brave Browser'],
['Microsoft Edge', 'Microsoft Edge.app', 'Microsoft Edge'],
];
const out = [];
for (const [, bundle, bin] of apps) {
out.push(`/Applications/${bundle}/Contents/MacOS/${bin}`);
out.push(path.join(os.homedir(), 'Applications', bundle, 'Contents', 'MacOS', bin));
}
out.push('/opt/homebrew/bin/chromium', '/usr/local/bin/chromium');
return out;
}
function linuxCandidates() {
// `command -v` resolves PATH-based binaries; fall back to common fixed paths.
const names = ['chromium', 'chromium-browser', 'google-chrome', 'google-chrome-stable', 'brave-browser'];
const found = [];
for (const n of names) {
try {
const p = execFileSync('command', ['-v', n], { stdio: ['ignore', 'pipe', 'ignore'], shell: true })
.toString().trim();
if (p) found.push(p);
} catch (e) { /* not on PATH */ }
}
return [...found, '/usr/bin/chromium', '/usr/bin/chromium-browser', '/snap/bin/chromium'];
}
function windowsCandidates() {
const pf = process.env['ProgramFiles'] || 'C:\\Program Files';
const pf86 = process.env['ProgramFiles(x86)'] || 'C:\\Program Files (x86)';
const local = process.env['LOCALAPPDATA'] || '';
return [
path.join(pf, 'Google', 'Chrome', 'Application', 'chrome.exe'),
path.join(pf86, 'Google', 'Chrome', 'Application', 'chrome.exe'),
path.join(local, 'Google', 'Chrome', 'Application', 'chrome.exe'),
path.join(pf, 'Chromium', 'Application', 'chrome.exe'),
];
}
function discoverBinary() {
const cands = process.platform === 'darwin' ? macCandidates()
: process.platform === 'win32' ? windowsCandidates()
: linuxCandidates();
for (const p of cands) {
try { if (fs.existsSync(p) && fs.statSync(p).isFile()) return p; } catch (e) {}
}
throw new Error(
'Could not find a Chromium/Chrome binary. Tried:\n ' + cands.join('\n ') +
'\nSet config.chromium.executablePath to the full path of your browser.'
);
}
// ── Stale-lock cleanup ──────────────────────────────────────────────────────
// A crashed Chromium leaves SingletonLock / SingletonCookie / SingletonSocket in
// the user-data-dir, which make a fresh process exit silently. Remove them.
function cleanStaleLocks(userDataDir) {
for (const name of ['SingletonLock', 'SingletonCookie', 'SingletonSocket']) {
const f = path.join(userDataDir, name);
try { if (fs.existsSync(f)) fs.unlinkSync(f); } catch (e) { /* best effort */ }
}
}
// ── Launch ──────────────────────────────────────────────────────────────────
function spawnBrowser(bin, opts) {
const { cdpPort, headless, userDataDir } = opts;
fs.mkdirSync(userDataDir, { recursive: true });
cleanStaleLocks(userDataDir);
const args = [
`--remote-debugging-port=${cdpPort}`,
`--user-data-dir=${userDataDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-extensions',
'--disable-gpu',
'--no-sandbox',
];
if (headless) args.push('--headless=new');
args.push('about:blank');
const logPath = path.join(userDataDir, 'chromium-stderr.log');
const logFd = fs.openSync(logPath, 'a');
const child = spawn(bin, args, {
detached: true,
stdio: ['ignore', 'ignore', logFd],
windowsHide: true,
});
child.on('error', (e) => {
console.error(`Chromium spawn error: ${e.message}`);
});
// Detach so it survives this process exiting.
child.unref();
return { pid: child.pid, logPath };
}
function tailFile(filePath, lines = 25) {
try {
const content = fs.readFileSync(filePath, 'utf8');
return content.split('\n').filter(Boolean).slice(-lines).join('\n');
} catch (e) { return '(no log file)'; }
}
// ── Main ────────────────────────────────────────────────────────────────────
async function ensureChromium(config, portOverride) {
const opts = resolveOptions(config, portOverride);
// 1. Already up? (idempotent — safe to call every run)
try {
const v = await cdpVersion(opts.cdpPort, 1500);
return { port: opts.cdpPort, alreadyRunning: true, browser: v.Browser };
} catch (e) { /* not up yet, fall through to launch */ }
// 2. Discover binary
const bin = opts.executablePath || discoverBinary();
if (!opts.executablePath && !fs.existsSync(bin)) {
// discoverBinary already throws, but guard explicit paths too
throw new Error(`executablePath not found: ${bin}`);
}
// 3. Launch
const { pid, logPath } = spawnBrowser(bin, opts);
// 4. Wait for readiness
try {
const v = await waitForCdp(opts.cdpPort, 30000, 500);
return { port: opts.cdpPort, pid, browser: v.Browser, bin, logPath };
} catch (e) {
throw new Error(
`Chromium did not become ready on port ${opts.cdpPort}.\n` +
` binary: ${bin}\n` +
` userData: ${opts.userDataDir}\n` +
` log: ${logPath}\n` +
` stderr tail:\n${tailFile(logPath)}`
);
}
}
// ── CLI ─────────────────────────────────────────────────────────────────────
const HELP = `ensure-chromium.js — robust cross-platform Chromium launcher for CDP
USAGE
node ensure-chromium.js Ensure Chromium is up on the CDP port
(launches it if not; no-op if already running)
node ensure-chromium.js --check Only health-check; exit 0 if up, 1 if not
node ensure-chromium.js --port 9333 Override the CDP port for this run
node ensure-chromium.js --help Show this help
WHAT IT DOES
1. Health-checks http://localhost:<port>/json/version (pure node, no python3).
2. If not up, discovers the browser binary per platform:
macOS — /Applications/{Chromium,Google Chrome,Brave,Edge}.app/Contents/MacOS/*
+ /opt/homebrew/bin/chromium, /usr/local/bin/chromium
Linux — chromium, chromium-browser, google-chrome[-stable], brave-browser
(via command -v) + /usr/bin/*, /snap/bin/chromium
Win — Program Files\\Google\\Chrome\\Application\\chrome.exe (+x86, %LOCALAPPDATA%)
Override discovery with config.chromium.executablePath.
3. Cleans stale SingletonLock files left by a crashed browser.
4. Spawns the browser (headless=new by default; --no-sandbox --disable-gpu,
remote-debugging-port=<port>, user-data-dir=<dir>, about:blank).
5. Polls /json/version until ready (up to 30s, every 500ms).
6. On timeout, prints binary path / port / userDataDir / stderr-log tail.
CONFIG (config.json, all optional — see SKILL.md "Configuration")
chromium.cdpPort CDP port (default 9222; env CDP_PORT_OVERRIDE wins)
chromium.headless true (default) | false (keep a window)
chromium.executablePath full path to browser binary (null = auto-discover)
chromium.userDataDir browser profile dir (null = <tmp>/news-harvester-chromium).
Set this to a fixed profile you've logged into once
(windowed, manual start) so future auto-launches keep
your cookies/login state.
EXIT CODES
0 CDP is up (already running or just launched)
1 CDP not up (--check), or launch failed
EXAMPLES
# Diagnose: is Chromium reachable?
node scripts/ensure-chromium.js --check
# Launch (auto-discovers Chrome on a Mac)
node scripts/ensure-chromium.js
# Use a non-default port for both this launcher and harvest.js
CDP_PORT_OVERRIDE=9333 node scripts/ensure-chromium.js
CDP_PORT_OVERRIDE=9333 node scripts/harvest.js reuters 2 --preview`;
if (require.main === module) {
const argv = process.argv.slice(2);
if (argv.includes('--help') || argv.includes('-h')) {
console.log(HELP);
process.exit(0);
}
const checkOnly = argv.includes('--check');
const portIdx = argv.indexOf('--port');
const portOverride = portIdx >= 0 ? argv[portIdx + 1] : null;
(async () => {
const config = loadConfig();
const opts = resolveOptions(config, portOverride);
if (checkOnly) {
try {
const v = await cdpVersion(opts.cdpPort, 1500);
console.log(`✅ Chromium CDP up on :${opts.cdpPort}${v.Browser}`);
process.exit(0);
} catch (e) {
console.log(`❌ Chromium CDP not running on :${opts.cdpPort} (${e.message})`);
process.exit(1);
}
}
try {
const r = await ensureChromium(config, portOverride);
if (r.alreadyRunning) {
console.log(`✅ Chromium already running on :${r.port}${r.browser}`);
} else {
console.log(`✅ Chromium launched (pid ${r.pid}) on :${r.port}${r.browser}`);
console.log(` binary: ${r.bin}`);
console.log(` log: ${r.logPath}`);
}
} catch (e) {
console.error(`❌ ${e.message}`);
process.exit(1);
}
})();
}
module.exports = { ensureChromium, cdpVersion, resolveOptions, discoverBinary };
......@@ -5,26 +5,59 @@
*/
const http = require('http');
const fs = require('fs');
const path = require('path');
const WebSocket = require('ws');
// Chromium CDP debug port — must match the --remote-debugging-port used to start
// Chromium (9222 per SKILL.md / README). Override via env if you run elsewhere.
const CDP_PORT = parseInt(process.env.CDP_PORT_OVERRIDE || '9222');
const SKILL_DIR = path.dirname(__dirname);
const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
// Resolve the CDP port lazily so config.json changes are picked up per run.
// Precedence: env override > config.chromium.cdpPort > default 9222.
function resolveCdpPort() {
if (process.env.CDP_PORT_OVERRIDE) return parseInt(process.env.CDP_PORT_OVERRIDE, 10);
try {
const cfgPath = fs.existsSync(CONFIG_PATH) ? CONFIG_PATH : TEMPLATE_PATH;
const cfg = JSON.parse(fs.readFileSync(cfgPath, 'utf8'));
if (cfg.chromium && cfg.chromium.cdpPort) return parseInt(cfg.chromium.cdpPort, 10);
} catch (e) { /* fall through to default */ }
return 9222;
}
// Retry the /json endpoint a few times: right after ensure-chromium spawns the
// browser, the port may accept connections a beat before /json responds.
function getWsUrl() {
const port = resolveCdpPort();
const MAX_ATTEMPTS = 6;
const STEP_MS = 500;
return new Promise((resolve, reject) => {
http.get(`http://localhost:${CDP_PORT}/json`, (res) => {
let data = '';
res.on('data', d => data += d);
res.on('end', () => {
try {
const tabs = JSON.parse(data);
// 优先找 page 类型的 tab,避免拿到 iframe/service_worker
const tab = tabs.find(t => t.type === 'page') || tabs[0];
resolve(tab.webSocketDebuggerUrl);
} catch (e) { reject(e); }
let attempt = 0;
const tryOnce = () => {
attempt++;
http.get(`http://localhost:${port}/json`, (res) => {
let data = '';
res.on('data', d => data += d);
res.on('end', () => {
try {
const tabs = JSON.parse(data);
// 优先找 page 类型的 tab,避免拿到 iframe/service_worker
const tab = tabs.find(t => t.type === 'page') || tabs[0];
if (!tab || !tab.webSocketDebuggerUrl) {
return reject(new Error(`CDP /json returned no page tab on :${port}`));
}
resolve(tab.webSocketDebuggerUrl);
} catch (e) {
if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
reject(e);
}
});
}).on('error', (e) => {
if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
reject(new Error(`Cannot reach CDP on :${port} after ${MAX_ATTEMPTS} tries (${e.message}). Run: node scripts/ensure-chromium.js`));
});
}).on('error', reject);
};
tryOnce();
});
}
......
......@@ -364,6 +364,20 @@ Examples:
? require('./fetcher-browser')
: require('./fetcher-direct');
// For browser sources, ensure a Chromium CDP endpoint is up before fetching.
// ensure-chromium.js is a no-op if already running, so this is safe every run.
if (source.method === 'browser') {
const { ensureChromium } = require('./ensure-chromium');
console.error(' Ensuring Chromium (CDP)...');
try {
const r = await ensureChromium(config);
console.error(` CDP ready on :${r.port}${r.alreadyRunning ? ' (already running)' : ` (pid ${r.pid})`}`);
} catch (e) {
console.error(` Chromium startup failed:\n${e.message}`);
process.exit(1);
}
}
// Build dedup set (skip in preview — preview shows latest regardless)
const seen = preview ? new Set() : buildDedupSet(config, sourceId);
console.error(` Registry: ${seen.size} URLs in last ${config.registryWindowDays || 7} days`);
......@@ -381,7 +395,7 @@ Examples:
// Normalize URLs (strip query params for dedup)
const normalizeUrl = (u) => { try { const p = new URL(u); p.search = ''; return p.toString(); } catch(e) { return u; } };
const normalizedLinks = links.map(u => normalizeUrl(u));
const normalizedLinks = [...new Set(links.map(u => normalizeUrl(u)))];
const newLinks = normalizedLinks.filter(u => !seen.has(u)).slice(0, count);
const skipped = normalizedLinks.filter(u => seen.has(u)).length;
console.error(` New: ${newLinks.length} | Already seen: ${skipped}`);
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment