Skip to content

N
News Hub Skills
Commit a891ad4e authored by 谢宇轩's avatar 谢宇轩
Browse files
Options

feat: support manage tab in cdp mode

parent d5995b60
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 702 additions and 356 deletions
.gitignore
View file @ a891ad4e
......@@ -19,3 +19,4 @@ helpers/
# IDE / agent workspace files
.zcode/
AGENTS.md
skills/news-harvester/SKILL.md
View file @ a891ad4e
......@@ -89,6 +89,8 @@ If this returns article data, setup is complete.
| `chromium.headless` | Optional | Headless (`true`, default) or windowed (`false`). |
| `chromium.executablePath` | Optional | Full path to browser binary. `null` = auto-discover. |
| `chromium.userDataDir` | Optional | Browser profile dir for login state. |
| `chromium.maxConcurrentTabs` | Optional | Global max concurrent browser tabs across all harvest processes (default `5`). Enforced via CDP `Target.getTargets` browser self-check. |
| `sources[].concurrency` | Optional | Per-source concurrent article fetch count (default `1` = serial). Capped by `chromium.maxConcurrentTabs`. See [Source Management](references/source-management.md#concurrency). |
#### Dedicated profile (login state)
......
skills/news-harvester/references/config-template.json
View file @ a891ad4e
......@@ -15,7 +15,8 @@
"cdpPort": 9222,
"headless": true,
"executablePath": null,
"userDataDir": null
"userDataDir": null,
"maxConcurrentTabs": 5
},
"sources": []
}
skills/news-harvester/references/source-management.md
View file @ a891ad4e
......@@ -12,7 +12,7 @@ reuters-world 路透社世界 browser ✅ ✅ enabled
theconversation The Conversation direct — ❌ disabled
```
RECIPE ✅ = `helpers/<id>.md` exists (spec-driven fetcher); — = legacy fetcher.
RECIPE ✅ = `helpers/<id>.md` exists (spec-driven fetcher-helper); — = no recipe (direct sources only, uses fetcher-direct).
## Edit Source
......@@ -20,7 +20,17 @@ RECIPE ✅ = `helpers/<id>.md` exists (spec-driven fetcher); — = legacy fetche
node scripts/source-manage.js edit <id> --name "New Name" --url https://... --method browser
```
Editable fields: `name`, `homepageUrl`, `method`, `articleUrlPattern`, `vaultFolder`, `enabled`.
Editable fields: `name`, `homepageUrl`, `method`, `articleUrlPattern`, `vaultFolder`, `enabled`, `concurrency`.
### Concurrency
Each source has a `concurrency` field (default `1` = serial). Set it to `2` or higher to fetch multiple articles in parallel using separate browser tabs:
```bash
node scripts/source-manage.js edit <id> --concurrency 3
```
The global tab limit (`chromium.maxConcurrentTabs`, default `5`) is enforced by the browser itself via CDP `Target.getTargets` — even when multiple harvest processes run simultaneously, the total open tabs never stays above the limit. A source's `concurrency` is silently capped to the global limit.
## Enable / Disable Source
......@@ -50,7 +60,7 @@ A `.nhsource.json` is plain JSON:
| `format` | `"news-harvester-source"` (install validates this) |
| `version` | `1` |
| `exportedAt` | ISO timestamp |
| `source` | `{ id, name, homepageUrl, method, vaultFolder }` — the config entry, minus `vaultPath`, `enabled`, and `articleUrlPattern` |
| `source` | `{ id, name, homepageUrl, method, vaultFolder }` — the config entry, minus `vaultPath`, `enabled`, `articleUrlPattern`, and `concurrency` (all set by install) |
| `files` | Map of relative path → file content. Includes `helpers/<id>.md` (the recipe) and `helpers/<id>.fixtures.json` (if present). |
### Install behavior
......
skills/news-harvester/scripts/block-check.js
View file @ a891ad4e
......@@ -11,7 +11,7 @@
*
* Two entry points:
* • Browser (CDP) sources — evaluateBrowserBlock(ws) runs an in-page probe via
* cdpEval and throws if blocked. Used by fetcher-helper.js / fetcher-browser.js.
* cdpEval and throws if blocked. Used by fetcher-helper.js.
* • Direct HTTP sources — checkDirectBlock(html, url) does a string scan of the
* raw HTML. Used by fetcher-direct.js (open sites rarely need login, but a
* Cloudflare-style challenge can still appear).
......@@ -26,9 +26,17 @@ function _browserBlockProbe() {
var bodyLen = bodyText.length;
// 1. CAPTCHA / anti-bot challenge — iframes from known challenge providers.
// Note: recaptcha/api2/aframe is a passive background iframe used by
// reCAPTCHA v3 for ad-fraud prevention — it does NOT block the user and is
// present on many news sites (Reuters, Nikkei, etc.). Only match the active
// challenge iframes (anchor/bframe/enterprise) that constitute a visible
// verification widget.
var challengeIframe = Array.from(document.querySelectorAll('iframe[src]')).some(function (f) {
var s = f.src.toLowerCase();
return s.indexOf('recaptcha') !== -1 || s.indexOf('hcaptcha') !== -1
return s.indexOf('recaptcha/api2/anchor') !== -1
|| s.indexOf('recaptcha/api2/bframe') !== -1
|| s.indexOf('recaptcha/enterprise') !== -1
|| s.indexOf('hcaptcha') !== -1
|| s.indexOf('challenges.cloudflare.com') !== -1
|| s.indexOf('turnstile') !== -1
|| s.indexOf('arkoselabs') !== -1 || s.indexOf('funcaptcha') !== -1;
......
skills/news-harvester/scripts/cdp-client.js
View file @ a891ad4e
This diff is collapsed.
skills/news-harvester/scripts/ensure-chromium.js
View file @ a891ad4e
......@@ -256,6 +256,41 @@ function looksLikeDisplayFailure(log) {
return /Missing X server|\$DISPLAY|platform failed to initialize|ozone/i.test(log);
}
// ── Startup-tab cleanup ─────────────────────────────────────────────────────
// Chromium opens an initial tab (about:blank or chrome://newtab) on launch.
// With the multi-tab architecture, openSession() creates its own tabs, so this
// startup tab is an orphan that wastes a tab slot and inflates Target.getTargets
// counts. Close it via the HTTP /json/close/<id> endpoint (no WebSocket needed).
// Only closes about:blank / chrome://newtab — never touches real harvest tabs.
const STARTUP_TAB_URLS = new Set(['about:blank', 'chrome://newtab/', 'chrome://newtab']);
function closeStartupTabs(port) {
return new Promise((resolve) => {
// agent: false → don't pool sockets in the global HTTP agent. Without this
// the keep-alive sockets linger and keep the Node event loop alive, hanging
// ensure-chromium.js (and any process that calls ensureChromium()).
http.get(`http://localhost:${port}/json`, { agent: false }, (res) => {
let data = '';
res.on('data', (d) => (data += d));
res.on('end', () => {
try {
const tabs = JSON.parse(data);
const orphans = tabs.filter(t => t.type === 'page' && STARTUP_TAB_URLS.has(t.url));
if (orphans.length === 0) return resolve(0);
let remaining = orphans.length;
for (const tab of orphans) {
http.get(`http://localhost:${port}/json/close/${tab.id}`, { agent: false }, () => {
if (--remaining === 0) resolve(orphans.length);
}).on('error', () => {
if (--remaining === 0) resolve(orphans.length);
});
}
} catch (e) { resolve(0); }
});
}).on('error', () => resolve(0));
});
}
// ── Main ────────────────────────────────────────────────────────────────────
async function ensureChromium(config, portOverride) {
......@@ -264,6 +299,7 @@ async function ensureChromium(config, portOverride) {
// 1. Already up? (idempotent — safe to call every run)
try {
const v = await cdpVersion(opts.cdpPort, 1500);
await closeStartupTabs(opts.cdpPort); // clean orphan tabs from prior runs
return { port: opts.cdpPort, alreadyRunning: true, browser: v.Browser };
} catch (e) { /* not up yet, fall through to launch */ }
......@@ -280,6 +316,7 @@ async function ensureChromium(config, portOverride) {
// 4. Wait for readiness
try {
const v = await waitForCdp(opts.cdpPort, 30000, 500);
await closeStartupTabs(opts.cdpPort); // close the initial about:blank tab
return { port: opts.cdpPort, pid, browser: v.Browser, bin, logPath };
} catch (e) {
const logTail = tailFile(logPath);
......@@ -527,6 +564,7 @@ if (require.main === module) {
console.log(` binary: ${r.bin}`);
console.log(` log: ${r.logPath}`);
}
process.exit(0);
} catch (e) {
console.error(`❌ ${e.message}`);
process.exit(1);
......
skills/news-harvester/scripts/fetcher-browser.js deleted 100644 → 0
View file @ d5995b60
/**
* fetcher-browser.js
* Fetch article content using Chromium CDP (for JS-heavy sites like Reuters)
* Usage: node fetcher-browser.js <url> [waitMs]
*/
const http = require('http');
const fs = require('fs');
const path = require('path');
const WebSocket = require('ws');
const { evaluateBrowserBlock } = require('./block-check');
const SKILL_DIR = path.dirname(__dirname);
const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
// Resolve the CDP port lazily so config.json changes are picked up per run.
// Precedence: env override > config.chromium.cdpPort > default 9222.
function resolveCdpPort() {
if (process.env.CDP_PORT_OVERRIDE) return parseInt(process.env.CDP_PORT_OVERRIDE, 10);
try {
const cfgPath = fs.existsSync(CONFIG_PATH) ? CONFIG_PATH : TEMPLATE_PATH;
const cfg = JSON.parse(fs.readFileSync(cfgPath, 'utf8'));
if (cfg.chromium && cfg.chromium.cdpPort) return parseInt(cfg.chromium.cdpPort, 10);
} catch (e) { /* fall through to default */ }
return 9222;
}
// Retry the /json endpoint a few times: right after ensure-chromium spawns the
// browser, the port may accept connections a beat before /json responds.
function getWsUrl() {
const port = resolveCdpPort();
const MAX_ATTEMPTS = 6;
const STEP_MS = 500;
return new Promise((resolve, reject) => {
let attempt = 0;
const tryOnce = () => {
attempt++;
http.get(`http://localhost:${port}/json`, (res) => {
let data = '';
res.on('data', d => data += d);
res.on('end', () => {
try {
const tabs = JSON.parse(data);
// 优先找 page 类型的 tab，避免拿到 iframe/service_worker
const tab = tabs.find(t => t.type === 'page') || tabs[0];
if (!tab || !tab.webSocketDebuggerUrl) {
return reject(new Error(`CDP /json returned no page tab on :${port}`));
}
resolve(tab.webSocketDebuggerUrl);
} catch (e) {
if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
reject(e);
}
});
}).on('error', (e) => {
if (attempt < MAX_ATTEMPTS) return setTimeout(tryOnce, STEP_MS);
reject(new Error(`Cannot reach CDP on :${port} after ${MAX_ATTEMPTS} tries (${e.message}). Run: node scripts/ensure-chromium.js`));
});
};
tryOnce();
});
}
async function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
async function fetchPage(url, waitMs = 12000) {
const wsUrl = await getWsUrl();
const ws = new WebSocket(wsUrl);
await new Promise((resolve, reject) => { ws.on('open', resolve); ws.on('error', reject); });
// Enable Page events
await new Promise(resolve => {
ws.send(JSON.stringify({ id: 1, method: 'Page.enable' }));
ws.once('message', () => resolve());
});
// Navigate + wait for load
let loaded = false;
const loadPromise = new Promise(resolve => {
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.method === 'Page.loadEventFired' && !loaded) {
loaded = true;
ws.removeListener('message', handler);
resolve();
}
};
ws.on('message', handler);
});
ws.send(JSON.stringify({ id: 2, method: 'Page.navigate', params: { url } }));
await Promise.race([loadPromise, sleep(25000)]);
await sleep(waitMs); // wait for React/JS to fully render
// Human-verification / login-state-loss guard. Throws HumanInterventionError
// (propagates to harvest.js, which exits with a friendly message) before we
// try to extract content from what might be a challenge or login page.
await evaluateBrowserBlock(ws);
// Extract content
const result = await new Promise((resolve) => {
const id = Math.floor(Math.random() * 100000) + 90000;
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.id === id) {
ws.removeListener('message', handler);
resolve(msg.result && msg.result.result && msg.result.result.value);
}
};
ws.on('message', handler);
ws.send(JSON.stringify({ id, method: 'Runtime.evaluate', params: {
expression: `(function(){
var title = document.title.replace(/ \\| Reuters| \\| The Conversation| - Bloomberg$/g,'').trim();
var dateEl = document.querySelector('time[datetime]');
var date = dateEl ? dateEl.getAttribute('datetime') : '';
var authors = Array.from(document.querySelectorAll('[rel="author"], [class*="author"] a'))
.map(a => a.innerText.trim()).filter(Boolean)
.filter((v,i,a) => a.indexOf(v)===i).join(', ');
var imgs = Array.from(document.querySelectorAll('img')).map(img => ({
src: img.src,
alt: img.alt || '',
width: img.naturalWidth || img.width || 0
})).filter(i => i.src && i.src.startsWith('http') && i.width > 200
&& !i.src.includes('logo') && !i.src.includes('icon') && !i.src.includes('avatar')
&& !i.src.includes('profile')).slice(0, 6);
var body = '';
var selectors = [
'[data-testid^="paragraph"]',
'[class*="articleBodyContent"] p',
'[class*="article-body"] p',
'[class*="ArticleBody"] p',
'[class*="story-body"] p',
'article p'
];
for (var sel of selectors) {
var els = document.querySelectorAll(sel);
if (els.length >= 3) {
body = Array.from(els).map(e => e.innerText.trim()).filter(t => t.length > 40).join('\\n\\n');
break;
}
}
if (!body) {
var noise = ['Reporting by','Editing by','Access unmatched','Browse an unrivalled',
'Screen for heightened','Sign up here','Reuters, the news','All quotes delayed','See here for'];
body = Array.from(document.querySelectorAll('p'))
.map(e => e.innerText.trim())
.filter(t => t.length > 60 && !noise.some(n => t.startsWith(n)))
.join('\\n\\n');
}
return JSON.stringify({ title, date, authors, imgs, body: body.substring(0, 15000) });
})()`
}}));
});
ws.close();
return result ? JSON.parse(result) : null;
}
// Extract article links from homepage
async function fetchHomeLinks(url, pattern, waitMs = 15000) {
const wsUrl = await getWsUrl();
const ws = new WebSocket(wsUrl);
await new Promise((resolve, reject) => { ws.on('open', resolve); ws.on('error', reject); });
await new Promise(resolve => {
ws.send(JSON.stringify({ id: 1, method: 'Page.enable' }));
ws.once('message', () => resolve());
});
let loaded = false;
const loadPromise = new Promise(resolve => {
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.method === 'Page.loadEventFired' && !loaded) {
loaded = true;
ws.removeListener('message', handler);
resolve();
}
};
ws.on('message', handler);
});
ws.send(JSON.stringify({ id: 2, method: 'Page.navigate', params: { url } }));
await Promise.race([loadPromise, sleep(25000)]);
await sleep(waitMs);
// Human-verification / login-state-loss guard (see fetchPage above).
await evaluateBrowserBlock(ws);
const result = await new Promise((resolve) => {
const id = Math.floor(Math.random() * 100000) + 80000;
const patternStr = pattern.toString();
const handler = (raw) => {
const msg = JSON.parse(raw.toString());
if (msg.id === id) {
ws.removeListener('message', handler);
resolve(msg.result && msg.result.result && msg.result.result.value);
}
};
ws.on('message', handler);
ws.send(JSON.stringify({ id, method: 'Runtime.evaluate', params: {
expression: `JSON.stringify(
Array.from(document.querySelectorAll('a[href]'))
.map(a => a.href)
.filter(h => h && new RegExp(${JSON.stringify(pattern)}).test(h))
.filter((h, i, arr) => arr.indexOf(h) === i)
.slice(0, 30)
)`
}}));
});
ws.close();
return result ? JSON.parse(result) : [];
}
module.exports = { fetchPage, fetchHomeLinks };
// CLI usage
if (require.main === module) {
const url = process.argv[2];
const waitMs = parseInt(process.argv[3] || '12000');
if (!url) { console.error('Usage: node fetcher-browser.js <url> [waitMs]'); process.exit(1); }
fetchPage(url, waitMs).then(data => {
process.stdout.write(JSON.stringify(data, null, 2));
}).catch(e => { console.error('Error:', e.message); process.exit(1); });
}
skills/news-harvester/scripts/fetcher-helper.js
View file @ a891ad4e
......@@ -25,9 +25,8 @@ const { openSession, navigateAndWait, scrollAndCollect, cdpEval } = require('./c
const { evaluateBrowserBlock } = require('./block-check');
// ── Spec defaults ────────────────────────────────────────────────────────────
// Defaults reproduce the existing hard-coded behaviour of fetcher-browser.js,
// so a minimal helper.md (just urlPattern + listSelector) already improves on
// legacy without forcing the author to redeclare every field.
// Sensible defaults so a minimal helper.md (just urlPattern + listSelector)
// already works well without forcing the author to redeclare every field.
const DEFAULTS = {
// ── Discovery ──
......@@ -283,7 +282,7 @@ async function fetchHomeLinks(url, spec, desiredCount) {
: (spec.maxLinks || 30);
return items.map(i => i.url).slice(0, cap);
} finally {
close();
await close();
}
}
......@@ -297,7 +296,7 @@ async function fetchPage(url, spec) {
const result = await cdpEval(ws, buildExtractExpr(spec));
return result ? JSON.parse(result) : null;
} finally {
close();
await close();
}
}
......
skills/news-harvester/scripts/harvest.js
View file @ a891ad4e
This diff is collapsed.
skills/news-harvester/scripts/source-manage.js
View file @ a891ad4e
......@@ -168,7 +168,8 @@ switch (command) {
homepageUrl: src.homepageUrl || '',
method: src.method || 'browser',
vaultFolder: src.vaultFolder || src.name || id,
enabled: true
enabled: true,
concurrency: 1 // serial by default; user can raise to enable multi-tab fetching
};
if (existing) {
Object.assign(existing, entry);
......
skills/news-source-analyzer/references/recipe-guide.md
View file @ a891ad4e
......@@ -2,7 +2,7 @@
## How recipes work
A recipe (`helpers/<id>.md`) is a YAML frontmatter file that declares the selectors, scroll behavior, and URL filters for a news source. When the harvester's `harvest.js` finds a recipe, it uses `fetcher-helper.js` (which reads the recipe's spec) instead of the legacy `fetcher-browser.js`/`fetcher-direct.js`.
A recipe (`helpers/<id>.md`) is a YAML frontmatter file that declares the selectors, scroll behavior, and URL filters for a news source. When the harvester's `harvest.js` finds a recipe, it uses `fetcher-helper.js` (which reads the recipe's spec) to drive Chromium via CDP. Browser-type sources without a recipe are no longer supported (`fetcher-browser.js` has been removed); direct-type sources fall back to `fetcher-direct.js`.
> **Precedence**: when a recipe exists, its `urlPattern`/selectors are the source of truth and the config's `articleUrlPattern` is **ignored**. A recipe also forces Chromium use regardless of `method` (since `fetcher-helper.js` drives the browser).
......
skills/news-source-analyzer/scripts/block-check.js
View file @ a891ad4e
......@@ -11,7 +11,7 @@
*
* Two entry points:
* • Browser (CDP) sources — evaluateBrowserBlock(ws) runs an in-page probe via
* cdpEval and throws if blocked. Used by fetcher-helper.js / fetcher-browser.js.
* cdpEval and throws if blocked. Used by fetcher-helper.js.
* • Direct HTTP sources — checkDirectBlock(html, url) does a string scan of the
* raw HTML. Used by fetcher-direct.js (open sites rarely need login, but a
* Cloudflare-style challenge can still appear).
......@@ -26,9 +26,17 @@ function _browserBlockProbe() {
var bodyLen = bodyText.length;
// 1. CAPTCHA / anti-bot challenge — iframes from known challenge providers.
// Note: recaptcha/api2/aframe is a passive background iframe used by
// reCAPTCHA v3 for ad-fraud prevention — it does NOT block the user and is
// present on many news sites (Reuters, Nikkei, etc.). Only match the active
// challenge iframes (anchor/bframe/enterprise) that constitute a visible
// verification widget.
var challengeIframe = Array.from(document.querySelectorAll('iframe[src]')).some(function (f) {
var s = f.src.toLowerCase();
return s.indexOf('recaptcha') !== -1 || s.indexOf('hcaptcha') !== -1
return s.indexOf('recaptcha/api2/anchor') !== -1
|| s.indexOf('recaptcha/api2/bframe') !== -1
|| s.indexOf('recaptcha/enterprise') !== -1
|| s.indexOf('hcaptcha') !== -1
|| s.indexOf('challenges.cloudflare.com') !== -1
|| s.indexOf('turnstile') !== -1
|| s.indexOf('arkoselabs') !== -1 || s.indexOf('funcaptcha') !== -1;
......
skills/news-source-analyzer/scripts/ensure-chromium.js
View file @ a891ad4e
......@@ -256,6 +256,41 @@ function looksLikeDisplayFailure(log) {
return /Missing X server|\$DISPLAY|platform failed to initialize|ozone/i.test(log);
}
// ── Startup-tab cleanup ─────────────────────────────────────────────────────
// Chromium opens an initial tab (about:blank or chrome://newtab) on launch.
// With the multi-tab architecture, openSession() creates its own tabs, so this
// startup tab is an orphan that wastes a tab slot and inflates Target.getTargets
// counts. Close it via the HTTP /json/close/<id> endpoint (no WebSocket needed).
// Only closes about:blank / chrome://newtab — never touches real harvest tabs.
const STARTUP_TAB_URLS = new Set(['about:blank', 'chrome://newtab/', 'chrome://newtab']);
function closeStartupTabs(port) {
return new Promise((resolve) => {
// agent: false → don't pool sockets in the global HTTP agent. Without this
// the keep-alive sockets linger and keep the Node event loop alive, hanging
// ensure-chromium.js (and any process that calls ensureChromium()).
http.get(`http://localhost:${port}/json`, { agent: false }, (res) => {
let data = '';
res.on('data', (d) => (data += d));
res.on('end', () => {
try {
const tabs = JSON.parse(data);
const orphans = tabs.filter(t => t.type === 'page' && STARTUP_TAB_URLS.has(t.url));
if (orphans.length === 0) return resolve(0);
let remaining = orphans.length;
for (const tab of orphans) {
http.get(`http://localhost:${port}/json/close/${tab.id}`, { agent: false }, () => {
if (--remaining === 0) resolve(orphans.length);
}).on('error', () => {
if (--remaining === 0) resolve(orphans.length);
});
}
} catch (e) { resolve(0); }
});
}).on('error', () => resolve(0));
});
}
// ── Main ────────────────────────────────────────────────────────────────────
async function ensureChromium(config, portOverride) {
......@@ -264,6 +299,7 @@ async function ensureChromium(config, portOverride) {
// 1. Already up? (idempotent — safe to call every run)
try {
const v = await cdpVersion(opts.cdpPort, 1500);
await closeStartupTabs(opts.cdpPort); // clean orphan tabs from prior runs
return { port: opts.cdpPort, alreadyRunning: true, browser: v.Browser };
} catch (e) { /* not up yet, fall through to launch */ }
......@@ -280,6 +316,7 @@ async function ensureChromium(config, portOverride) {
// 4. Wait for readiness
try {
const v = await waitForCdp(opts.cdpPort, 30000, 500);
await closeStartupTabs(opts.cdpPort); // close the initial about:blank tab
return { port: opts.cdpPort, pid, browser: v.Browser, bin, logPath };
} catch (e) {
const logTail = tailFile(logPath);
......@@ -527,6 +564,7 @@ if (require.main === module) {
console.log(` binary: ${r.bin}`);
console.log(` log: ${r.logPath}`);
}
process.exit(0);
} catch (e) {
console.error(`❌ ${e.message}`);
process.exit(1);
......
skills/news-source-analyzer/scripts/fetcher-helper.js
View file @ a891ad4e
......@@ -25,9 +25,8 @@ const { openSession, navigateAndWait, scrollAndCollect, cdpEval } = require('./c
const { evaluateBrowserBlock } = require('./block-check');
// ── Spec defaults ────────────────────────────────────────────────────────────
// Defaults reproduce the existing hard-coded behaviour of fetcher-browser.js,
// so a minimal helper.md (just urlPattern + listSelector) already improves on
// legacy without forcing the author to redeclare every field.
// Sensible defaults so a minimal helper.md (just urlPattern + listSelector)
// already works well without forcing the author to redeclare every field.
const DEFAULTS = {
// ── Discovery ──
......@@ -283,7 +282,7 @@ async function fetchHomeLinks(url, spec, desiredCount) {
: (spec.maxLinks || 30);
return items.map(i => i.url).slice(0, cap);
} finally {
close();
await close();
}
}
......@@ -297,7 +296,7 @@ async function fetchPage(url, spec) {
const result = await cdpEval(ws, buildExtractExpr(spec));
return result ? JSON.parse(result) : null;
} finally {
close();
await close();
}
}
......
skills/news-source-analyzer/scripts/inspect-source.js
View file @ a891ad4e
......@@ -305,7 +305,7 @@ async function main() {
console.log(`\n Save to helpers/<source-id>.md (or re-run with --write <id>), then test: node scripts/preview.js --url <url> --recipe helpers/<source-id>.md --count 3`);
}
} finally {
close();
await close();
}
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment