Commit 518b835c authored by 谢宇轩's avatar 谢宇轩

feat: support push

parent 6d415fe0
...@@ -15,6 +15,8 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau ...@@ -15,6 +15,8 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau
| Preview latest (read-only) | `node scripts/harvest.js <source_id> <count> --preview` | | Preview latest (read-only) | `node scripts/harvest.js <source_id> <count> --preview` |
| Preview with full body | `node scripts/harvest.js <source_id> <count> --preview --full` | | Preview with full body | `node scripts/harvest.js <source_id> <count> --preview --full` |
| Finalize Chinese summaries | `node scripts/finalize.js "<dayDir>"` | | Finalize Chinese summaries | `node scripts/finalize.js "<dayDir>"` |
| Push a day directory to News Hub | `node scripts/push.js "<dayDir>"` |
| Check push sync status | `node scripts/push.js "<dayDir>" --status` |
| Merge a summary batch | `node scripts/summary-add.js "<dayDir>"` | | Merge a summary batch | `node scripts/summary-add.js "<dayDir>"` |
| Check summary coverage | `node scripts/summary-add.js "<dayDir>" --status` | | Check summary coverage | `node scripts/summary-add.js "<dayDir>" --status` |
| Add source | `node scripts/source-manage.js add` | | Add source | `node scripts/source-manage.js add` |
...@@ -93,6 +95,12 @@ Before running a harvest, the agent MUST verify/edit these fields: ...@@ -93,6 +95,12 @@ Before running a harvest, the agent MUST verify/edit these fields:
|-------|-----------|-------------| |-------|-----------|-------------|
| `vaultPath` | ✅ Yes | Absolute path to the Obsidian vault root. Must exist and be writable. Replace the template placeholder `/path/to/your/obsidian/vault`. | | `vaultPath` | ✅ Yes | Absolute path to the Obsidian vault root. Must exist and be writable. Replace the template placeholder `/path/to/your/obsidian/vault`. |
| `registryWindowDays` | Optional | Dedup sliding window in days (default `7`). | | `registryWindowDays` | Optional | Dedup sliding window in days (default `7`). |
| `push.enabled` | Optional | Enable pushing finalized archives to the News Hub backend (default `false`). |
| `push.endpoint` | Optional* | News Hub API base URL, e.g. `http://localhost:8080/api/v1`. *Required when `push.enabled` is `true`. |
| `push.appId` / `appKeyId` / `appSecret` | Optional* | News Hub App credentials (3-header auth). *Required when `push.enabled` is `true`. |
| `push.batchSize` | Optional | Articles per push batch (default `10`, API max 10). |
| `push.autoPushAfterFinalize` | Optional | When `true` (default), `finalize.js` auto-chains into `push.js`. Set `false` to push manually only. |
| `push.retry` | Optional | `{ maxAttempts, baseDelayMs }` for exponential backoff on 5xx/network errors (defaults `5` / `1000`). |
| `chromium.cdpPort` | Optional | CDP debug port (default `9222`). Only relevant for `browser` sources. | | `chromium.cdpPort` | Optional | CDP debug port (default `9222`). Only relevant for `browser` sources. |
| `chromium.headless` | Optional | Launch headless (`true`, default) or with a window (`false`). Set `false` if a site's anti-bot detects headless. | | `chromium.headless` | Optional | Launch headless (`true`, default) or with a window (`false`). Set `false` if a site's anti-bot detects headless. |
| `chromium.executablePath` | Optional | Full path to the browser binary. Set only if auto-discovery fails (non-standard install location). `null` = auto-discover. | | `chromium.executablePath` | Optional | Full path to the browser binary. Set only if auto-discovery fails (non-standard install location). `null` = auto-discover. |
...@@ -266,6 +274,20 @@ node scripts/finalize.js "<dayDir>" ...@@ -266,6 +274,20 @@ node scripts/finalize.js "<dayDir>"
``` ```
This writes each `<中文标题>.md`, patches the sentinel links in the 原文.md files, updates `registry.json` + `index.md` with real titles, and removes `.manifest.json` + `.summaries.json`. This writes each `<中文标题>.md`, patches the sentinel links in the 原文.md files, updates `registry.json` + `index.md` with real titles, and removes `.manifest.json` + `.summaries.json`.
> **Auto-push (optional).** If `config.push.enabled` is `true`, finalize automatically chains into `push.js`, which uploads the day directory to the News Hub backend (images to MinIO via `POST /assets`, then articles via `POST /articles`). A push failure **never** aborts finalize — the local archive is already complete; re-push with `node scripts/push.js "<dayDir>"` once the backend is reachable. To push manually instead, set `push.autoPushAfterFinalize: false`.
### Flow C — Push to News Hub (finalize → remote sync)
After a day directory is finalized, push it to the News Hub backend:
```bash
node scripts/push.js "<dayDir>" # push (resumes from failures via .syncstate.json)
node scripts/push.js "<dayDir>" --status # read-only sync status report
node scripts/push.js "<dayDir>" --force # ignore ledger, re-push all articles
```
`push.js` reads `registry.json` for metadata, parses each 原文.md / 摘要.md (markdown + the four summary sections), uploads images to MinIO (SHA-256 deduped), and batch-pushes articles (≤10/batch, idempotent via `Idempotency-Key`). It writes a `.syncstate.json` ledger per day directory so re-running resumes only failed/missing articles. Exit code `2` means partial failure (some articles still in error). See the backend API at `news-hub/docs/API.md`.
### Flow B — Preview (read-only, no archive) ### Flow B — Preview (read-only, no archive)
For "返回最新文章数据" / "看看最新文章" requests where the user does **not** ask to save: For "返回最新文章数据" / "看看最新文章" requests where the user does **not** ask to save:
......
{ {
"vaultPath": "/path/to/your/obsidian/vault", "vaultPath": "/path/to/your/obsidian/vault",
"registryWindowDays": 7, "registryWindowDays": 7,
"push": {
"enabled": false,
"endpoint": "http://localhost:8080/api/v1",
"appId": "",
"appKeyId": "",
"appSecret": "",
"batchSize": 10,
"autoPushAfterFinalize": true,
"retry": { "maxAttempts": 5, "baseDelayMs": 1000 }
},
"chromium": { "chromium": {
"cdpPort": 9222, "cdpPort": 9222,
"headless": true, "headless": true,
......
...@@ -202,6 +202,37 @@ function main() { ...@@ -202,6 +202,37 @@ function main() {
console.error(`\n✅ Done: ${patched} finalized, ${skipped} skipped`); console.error(`\n✅ Done: ${patched} finalized, ${skipped} skipped`);
console.error(` Registry + index updated, temp files removed`); console.error(` Registry + index updated, temp files removed`);
// 6. Auto-push to News Hub if enabled and configured. Push failures do NOT
// abort finalize (the archive is already complete and re-pushable), so
// a backend hiccup never leaves the local archive in a bad state.
maybeAutoPush(dayDir);
}
// Chain into push.js when config.push.enabled && autoPushAfterFinalize.
// Runs push.js as a child process so a push crash can't take down finalize.
function maybeAutoPush(dayDir) {
const cfgPath = path.join(__dirname, '..', 'config.json');
let cfg;
try { cfg = JSON.parse(fs.readFileSync(cfgPath, 'utf8')); }
catch (e) { return; } // no config → skip silently
const p = cfg.push;
if (!p || !p.enabled || !p.autoPushAfterFinalize) return;
if (!p.endpoint || !p.appId || !p.appKeyId || !p.appSecret) {
console.error(`\n⚠️ Push enabled but config incomplete (endpoint/appId/appKeyId/appSecret) — skipping.`);
console.error(` Run manually once configured: node scripts/push.js "${dayDir}"`);
return;
}
console.error(`\n📤 Auto-pushing to News Hub (config.push.autoPushAfterFinalize = true)`);
const { spawnSync } = require('child_process');
const r = spawnSync(process.execPath, [path.join(__dirname, 'push.js'), dayDir], {
stdio: ['ignore', 'inherit', 'inherit'], // stdout/stderr → this process
});
if (r.status !== 0) {
console.error(`\n⚠️ Push exited with code ${r.status} — local archive is unaffected.`);
console.error(` Fix the issue and re-run: node scripts/push.js "${dayDir}"`);
}
} }
if (require.main === module) { if (require.main === module) {
......
#!/usr/bin/env node
/**
* push.js — Push a finalized day directory to the News Hub backend.
*
* Reads the finalized day directory produced by finalize.js (registry.json +
* 原文.md + summary/<中文标题>.md + assets/), uploads images to MinIO via the
* backend's /assets endpoint (SHA-256 deduped), then batch-pushes articles via
* /articles. Maintains a per-day .syncstate.json ledger for incremental resume.
*
* Usage:
* node scripts/push.js <dayDir> # push a single day directory
* node scripts/push.js <dayDir> --status # show sync status (read-only)
* node scripts/push.js <dayDir> --force # ignore ledger, re-push all
* node scripts/push.js -h | --help
*
* Prerequisites:
* - <dayDir>/registry.json exists (written by harvest.js, patched by finalize.js)
* - <dayDir>/.manifest.json and .summaries.json are ABSENT (= finalize done)
* - config.json has a `push` section (endpoint, appId, appKeyId, appSecret)
*
* Exit codes: 0 success (or all-pending-resolved), 1 usage/IO error,
* 2 partial failure (some articles still error after push).
*/
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const { spawnSync } = require('child_process');
const yaml = require('js-yaml');
const CONFIG_PATH = path.join(__dirname, '..', 'config.json');
const SYNCSTATE = '.syncstate.json';
const MANIFEST = '.manifest.json';
const SUMMARIES = '.summaries.json';
const REGISTRY = 'registry.json';
// Summary section headers (finalize.js produces exactly these four, in order).
const SUMMARY_SECTIONS = ['事件概述', '背景', '关键引语', '影响分析'];
const SUMMARY_KEYS = ['overview', 'background', 'keyQuote', 'impact'];
// ── Helpers ───────────────────────────────────────────────────────────────
function loadConfig() {
if (!fs.existsSync(CONFIG_PATH)) {
console.error(`✗ config.json not found at ${CONFIG_PATH}`);
console.error(` Run a harvest script once to generate it, or copy references/config-template.json.`);
process.exit(1);
}
return JSON.parse(fs.readFileSync(CONFIG_PATH, 'utf8'));
}
function loadJson(p, label) {
if (!fs.existsSync(p)) {
console.error(`✗ ${label} not found: ${p}`);
process.exit(1);
}
try {
return JSON.parse(fs.readFileSync(p, 'utf8'));
} catch (e) {
console.error(`✗ Failed to parse ${label}: ${e.message}`);
process.exit(1);
}
}
function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
function uuid() {
return crypto.randomUUID();
}
// Parse a `---\n...yaml...\n---` frontmatter block from markdown.
// Returns { fm: object|null, body: string } where body is the markdown AFTER
// the frontmatter (frontmatter stripped). Mirrors the pattern in
// fetcher-helper.js:77-82.
function parseFrontmatter(md) {
const m = md.match(/^---\r?\n([\s\S]*?)\r?\n---\r?\n?([\s\S]*)$/);
if (!m) return { fm: null, body: md };
let fm = null;
try { fm = yaml.load(m[1]) || {}; } catch (e) { fm = null; }
return { fm, body: m[2] };
}
// Extract the four summary sections from the summary doc body.
// The body looks like:
// ## 事件概述\n<text>\n\n## 背景\n<text>\n\n## 关键引语\n> "..."\n\n## 影响分析\n<text>
// Returns { overview, background, keyQuote, impact } (missing sections = '').
function parseSummarySections(body) {
const out = { overview: '', background: '', keyQuote: '', impact: '' };
// Match `## <sectionName>` and capture everything until the next `## ` header,
// a horizontal rule (a `---` line surrounded by blank lines), or end of string.
// The `---` guard keeps the trailing footer (wikilink back-link) out of the
// last section, since the summary doc ends with `\n\n---\n\n[[原文]]`.
const names = SUMMARY_SECTIONS.join('|');
const re = new RegExp(
`## (${names})\\r?\\n([\\s\\S]*?)(?=\\r?\\n## (?:${names})|\\r?\\n\\r?\\n---\\r?\\n|$)`, 'g'
);
let m;
while ((m = re.exec(body)) !== null) {
const title = m[1];
const text = m[2].trim();
const idx = SUMMARY_SECTIONS.indexOf(title);
if (idx >= 0) out[SUMMARY_KEYS[idx]] = text;
}
return out;
}
// Parse image embeds from the 原文.md `## 配图` section.
// Format per image:
// ![[assets/<filename>]]
// *<alt text>* ← optional caption line; omitted when alt is empty
// Returns [{ filename, alt }].
function parseImages(originalMd) {
// Isolate the 配图 section: between `## 配图` and the next `## ` header.
const secMatch = originalMd.match(/## 配图\r?\n([\s\S]*?)(?=\r?\n## )/);
if (!secMatch) return [];
const section = secMatch[1];
const imgs = [];
const lines = section.split(/\r?\n/);
for (let i = 0; i < lines.length; i++) {
const embed = lines[i].match(/!\[\[assets\/([^\]]+)\]\]/);
if (!embed) continue;
const filename = embed[1].trim();
// Look at the next non-empty line for a caption `*...*`.
let alt = '';
for (let j = i + 1; j < lines.length; j++) {
const next = lines[j].trim();
if (next === '') continue;
const cap = next.match(/^\*(.+)\*$/);
if (cap) alt = cap[1].trim();
break;
}
imgs.push({ filename, alt });
}
return imgs;
}
// Compute SHA-256 of a file (for local dedup before upload).
function fileSha256(filePath) {
const buf = fs.readFileSync(filePath);
return crypto.createHash('sha256').update(buf).digest('hex');
}
// Locate the Chinese summary doc for an article. finalize.js (current) writes
// `summary/<titleZh>.md`; older layouts placed `<titleZh>.md` directly in the
// day dir. Try the subdir first, then fall back to the root. Returns '' if
// neither exists (the article is still pushed, just with an empty summary).
function readSummaryDoc(dayDir, titleZh) {
if (!titleZh) return '';
const sub = path.join(dayDir, 'summary', `${titleZh}.md`);
if (fs.existsSync(sub)) return fs.readFileSync(sub, 'utf8');
const root = path.join(dayDir, `${titleZh}.md`);
if (fs.existsSync(root)) return fs.readFileSync(root, 'utf8');
return '';
}
// Convert the 原文.md body (markdown AFTER frontmatter) into clean HTML for
// `originalDoc.body`. Strips Obsidian-specific syntax (callouts, wikilinks,
// `![[assets/...]]` embeds) and produces semantic HTML: <h2> for the English
// title, <figure>/<img> for images (src = `asset://<assetId>`), <p> for
// paragraphs, <blockquote> for quotes.
//
// `assetIdByFilename` maps `assets/<filename>` → assetId (from Phase 1 uploads).
// Images without a known assetId get `src=""` (the asset upload failed or the
// file was missing — the article still pushes, just without that image).
function mdToHtmlBody(md, titleEn, assetIdByFilename) {
const lines = md.split(/\r?\n/);
const out = [];
let i = 0;
let paraBuf = [];
const esc = (s) => s.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');
const flush = () => {
if (paraBuf.length) {
const t = paraBuf.join(' ').trim();
if (t) out.push(`<p>${esc(t)}</p>`);
paraBuf = [];
}
};
while (i < lines.length) {
const trimmed = lines[i].trim();
// Skip callout block (> [!info] ... until next ---)
if (trimmed.match(/^>\s*\[!info\]/)) {
while (i < lines.length && lines[i].trim() !== '---') i++;
if (i < lines.length) i++;
continue;
}
// Skip bare wikilinks [[...]] (footer back-links)
if (trimmed.match(/^\[\[.+\]\]$/)) { flush(); i++; continue; }
// Skip horizontal rules
if (trimmed === '---') { flush(); i++; continue; }
// Skip section headers ## 配图 / ## 正文
if (trimmed.match(/^##\s+(配图|正文)$/)) { flush(); i++; continue; }
// # Title → <h2> (use frontmatter titleEn, more authoritative)
if (trimmed.match(/^#\s+(.+)$/)) {
flush();
out.push(`<h2>${esc(titleEn || RegExp.$1)}</h2>`);
i++; continue;
}
// Obsidian image embed ![[assets/<filename>]] + optional *alt* caption
const imgMatch = trimmed.match(/^!\[\[assets\/([^\]]+)\]\]$/);
if (imgMatch) {
flush();
const filename = imgMatch[1];
let alt = '';
if (i + 1 < lines.length) {
const cap = lines[i + 1].trim().match(/^\*(.+)\*$/);
if (cap) { alt = cap[1]; i++; }
}
const assetId = assetIdByFilename ? assetIdByFilename.get(filename) : null;
const src = assetId != null ? `asset://${assetId}` : '';
if (alt) {
out.push(`<figure><img src="${src}" alt="${esc(alt)}"><figcaption>${esc(alt)}</figcaption></figure>`);
} else {
out.push(`<img src="${src}" alt="">`);
}
i++; continue;
}
// Blockquote > text
if (trimmed.match(/^>\s*(.*)$/)) {
flush();
const quotes = [];
while (i < lines.length && lines[i].trim().match(/^>\s*(.*)$/)) {
quotes.push(RegExp.$1); i++;
}
out.push(`<blockquote><p>${esc(quotes.join(' ').trim())}</p></blockquote>`);
continue;
}
// Blank line → flush paragraph
if (trimmed === '') { flush(); i++; continue; }
// Normal text line
paraBuf.push(trimmed);
i++;
}
flush();
return out.join('\n');
}
// ── HTTP client (native fetch, Node 18+/23) ──────────────────────────────
function authHeaders(cfg) {
return {
'X-App-Id': cfg.push.appId,
'X-App-Key-Id': cfg.push.appKeyId,
'X-App-Secret': cfg.push.appSecret,
};
}
async function retryingFetch(url, opts, retryCfg, label) {
const max = (retryCfg && retryCfg.maxAttempts) || 5;
const base = (retryCfg && retryCfg.baseDelayMs) || 1000;
let lastErr;
for (let attempt = 1; attempt <= max; attempt++) {
try {
const res = await fetch(url, opts);
// Retry on 5xx and 429; everything else is a definitive response.
if ((res.status >= 500 || res.status === 429) && attempt < max) {
const delay = base * Math.pow(2, attempt - 1);
console.error(` ↻ ${label}: HTTP ${res.status}, retry ${attempt}/${max} in ${delay}ms`);
await sleep(delay);
continue;
}
return res;
} catch (e) {
lastErr = e;
if (attempt < max) {
const delay = base * Math.pow(2, attempt - 1);
console.error(` ↻ ${label}: ${e.message}, retry ${attempt}/${max} in ${delay}ms`);
await sleep(delay);
continue;
}
}
}
// Exhausted retries — rethrow the last network error, or synthesize one.
throw lastErr || new Error(`${label}: exhausted retries`);
}
// Upload a single image. Returns { assetId, sha256, deduplicated } or null on failure.
async function uploadAsset(cfg, filePath, slug, alt) {
const endpoint = cfg.push.endpoint.replace(/\/$/, '');
const url = `${endpoint}/assets`;
const sha = fileSha256(filePath);
const fd = new FormData();
const buf = fs.readFileSync(filePath);
const filename = path.basename(filePath);
// Infer content type from extension (backend does the same, but set it anyway).
const ext = path.extname(filename).toLowerCase();
const ct = { '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png',
'.gif': 'image/gif', '.webp': 'image/webp', '.svg': 'image/svg+xml' }[ext]
|| 'application/octet-stream';
fd.append('file', new Blob([buf], { type: ct }), filename);
if (slug) fd.append('slug', slug);
if (alt) fd.append('alt', alt);
// originalSrc is unknown after finalize (manifest deleted); omit it.
const res = await retryingFetch(url, {
method: 'POST',
headers: { ...authHeaders(cfg), 'Idempotency-Key': uuid() },
body: fd,
}, cfg.push.retry, `asset ${filename}`);
const text = await res.text();
let body;
try { body = JSON.parse(text); } catch (e) { body = { raw: text }; }
if (!res.ok) {
const code = body && body.error && body.error.code;
const msg = body && body.error && body.error.message || text;
console.error(` ✗ asset ${filename}: HTTP ${res.status} ${code || ''} ${msg}`);
return null;
}
return {
assetId: body.assetId,
sha256: body.sha256 || sha,
deduplicated: !!body.deduplicated,
};
}
// Push a batch of articles (≤10). Returns the parsed response body.
async function pushArticleBatch(cfg, payload) {
const endpoint = cfg.push.endpoint.replace(/\/$/, '');
const url = `${endpoint}/articles`;
const res = await retryingFetch(url, {
method: 'POST',
headers: {
...authHeaders(cfg),
'Content-Type': 'application/json',
'Idempotency-Key': uuid(),
},
body: JSON.stringify(payload),
}, cfg.push.retry, `batch ${payload.clientBatchId}`);
const text = await res.text();
let body;
try { body = JSON.parse(text); } catch (e) { body = { raw: text }; }
if (!res.ok) {
const code = body && body.error && body.error.code;
const msg = body && body.error && body.error.message || text;
const err = new Error(`HTTP ${res.status} ${code || ''} ${msg}`);
err.status = res.status;
err.body = body;
throw err;
}
return body;
}
// ── Ledger (.syncstate.json) ─────────────────────────────────────────────
function loadSyncState(dayDir) {
const p = path.join(dayDir, SYNCSTATE);
if (!fs.existsSync(p)) return null;
try { return JSON.parse(fs.readFileSync(p, 'utf8')); } catch (e) { return null; }
}
function saveSyncState(dayDir, state) {
state.updatedAt = new Date().toISOString();
fs.writeFileSync(path.join(dayDir, SYNCSTATE), JSON.stringify(state, null, 2));
}
function newSyncState(registry) {
return {
source: registry.source,
harvestDate: registry.date,
lastSyncStatus: 'pending',
articles: registry.articles.map(a => ({
url: a.url,
slug: a.slug,
status: 'pending',
articleId: null,
version: null,
error: null,
})),
};
}
// Mark a url's result in the ledger.
function recordResult(state, url, result) {
const entry = state.articles.find(a => a.url === url);
if (!entry) return;
if (result.status === 'error') {
entry.status = 'error';
entry.error = result.error || null;
} else {
entry.status = 'synced';
entry.articleId = result.articleId != null ? String(result.articleId) : null;
entry.version = result.version != null ? result.version : null;
entry.error = null;
}
}
function summarizeState(state) {
let created = 0, updated = 0, unchanged = 0, failed = 0;
for (const a of state.articles) {
// We don't track created/updated/unchanged distinctly in the ledger across
// re-runs (a re-push of an already-synced url returns unchanged). For the
// final summary we report the latest batch's counts directly from the API.
if (a.status === 'error') failed++;
}
return { total: state.articles.length, failed };
}
// ── Core push flow ───────────────────────────────────────────────────────
async function pushDayDir(dayDir, cfg, opts) {
const force = !!opts.force;
// 1. Validate finalized state.
const regPath = path.join(dayDir, REGISTRY);
if (!fs.existsSync(regPath)) {
console.error(`✗ ${REGISTRY} not found in ${dayDir}`);
console.error(` Has harvest.js run for this day?`);
process.exit(1);
}
if (fs.existsSync(path.join(dayDir, MANIFEST)) || fs.existsSync(path.join(dayDir, SUMMARIES))) {
console.error(`✗ Day directory is not finalized yet (`.concat(MANIFEST, ' or ', SUMMARIES, ' still present).'));
console.error(` Run finalize.js first: node scripts/finalize.js "${dayDir}"`);
process.exit(1);
}
const registry = loadJson(regPath, REGISTRY);
const articles = Array.isArray(registry.articles) ? registry.articles : [];
if (articles.length === 0) {
console.error('✗ registry.json has no articles — nothing to push.');
process.exit(0);
}
const source = registry.source;
const harvestDate = registry.date;
console.error(`\n📤 Pushing ${dayDir}`);
console.error(` source: ${source} | harvestDate: ${harvestDate} | articles: ${articles.length}`);
// 2. Load / init ledger, decide what to push.
let state = loadSyncState(dayDir) || newSyncState(registry);
if (force) {
state = newSyncState(registry);
console.error(' --force: ledger reset, re-pushing all articles.');
}
const pending = articles.filter(a => {
const e = state.articles.find(x => x.url === a.url);
return !e || e.status !== 'synced';
});
if (pending.length === 0) {
console.error(' ✅ All articles already synced — nothing to do.');
console.error(' (use --force to re-push)');
state.lastSyncStatus = 'success';
saveSyncState(dayDir, state);
process.stdout.write(JSON.stringify({ dayDir, source, harvestDate, skipped: true, state }, null, 2));
return;
}
console.error(` Pending: ${pending.length} (already synced: ${articles.length - pending.length})`);
// 3. Phase 1 — upload images for pending articles (local sha256 dedup).
const shaCache = new Map(); // sha256 -> assetId (in-process dedup)
// Seed cache from existing synced entries? Not needed: asset ownership is
// per-App on the server and re-upload of identical content returns
// deduplicated:true with the same assetId. Local cache avoids re-reading.
const articleAssets = new Map(); // url -> [{ assetId, alt, originalSrc }]
for (const a of pending) {
const origPath = path.join(dayDir, a.originalFilename);
if (!fs.existsSync(origPath)) {
console.error(` ⚠️ Original doc missing for ${a.titleEn || a.url}: ${a.originalFilename}`);
articleAssets.set(a.url, []);
continue;
}
const origMd = fs.readFileSync(origPath, 'utf8');
const imgs = parseImages(origMd);
const uploaded = [];
for (const img of imgs) {
const imgPath = path.join(dayDir, 'assets', img.filename);
if (!fs.existsSync(imgPath)) {
console.error(` ⚠️ Image missing: assets/${img.filename} (${a.titleEn || a.url})`);
continue;
}
const sha = fileSha256(imgPath);
let assetId;
if (shaCache.has(sha)) {
assetId = shaCache.get(sha);
} else {
console.error(` 📷 Uploading assets/${img.filename} ...`);
const res = await uploadAsset(cfg, imgPath, a.slug, img.alt);
if (!res) {
// Image upload failed — don't block the article; omit this asset.
continue;
}
assetId = res.assetId;
shaCache.set(sha, assetId);
}
uploaded.push({ assetId, filename: img.filename, alt: img.alt || '', originalSrc: '' });
}
articleAssets.set(a.url, uploaded);
}
// 4. Phase 2 — batch push articles (≤ batchSize, default 10).
const batchSize = (cfg.push.batchSize && cfg.push.batchSize > 0) ? cfg.push.batchSize : 10;
const batches = [];
for (let i = 0; i < pending.length; i += batchSize) {
batches.push(pending.slice(i, i + batchSize));
}
const batchResults = [];
for (let bi = 0; bi < batches.length; bi++) {
const batch = batches[bi];
const clientBatchId = `nh-${harvestDate}-${bi}`;
const payload = {
source,
harvestDate,
clientBatchId,
articles: batch.map(a => {
const origPath = path.join(dayDir, a.originalFilename);
const origMd = fs.existsSync(origPath) ? fs.readFileSync(origPath, 'utf8') : '';
const { fm, body: origBody } = parseFrontmatter(origMd);
const summaryMd = readSummaryDoc(dayDir, a.titleZh);
const { body: summaryBody } = parseFrontmatter(summaryMd);
const sections = parseSummarySections(summaryBody);
// Build filename → assetId map for image src resolution in HTML body.
const assetIdByFilename = new Map();
for (const ast of (articleAssets.get(a.url) || [])) {
if (ast.filename) assetIdByFilename.set(ast.filename, ast.assetId);
}
const htmlBody = mdToHtmlBody(origBody, fm && fm.title, assetIdByFilename);
return {
url: a.url,
slug: a.slug,
titleEn: a.titleEn || '',
titleZh: a.titleZh,
publishedAt: a.date || (fm && fm.date) || '',
// --bump-processed-at: override processedAt to now() so the backend's
// LWW (strict greater-than) accepts the update. Use when re-pushing
// corrected content (e.g. summaries were empty on first push) — the
// original processedAt would be judged "unchanged" and skipped.
processedAt: opts.bumpProcessedAt ? new Date().toISOString() : a.processedAt,
category: a.category,
tags: a.tags || [],
authors: a.authors || (fm && fm.authors) || '',
publisher: (fm && fm.publisher) || '',
originalDoc: { markdown: origMd, body: htmlBody },
summaryDoc: { markdown: summaryMd, sections },
assets: articleAssets.get(a.url) || [],
};
}),
};
console.error(`\n ▶ Batch ${bi + 1}/${batches.length} (${batch.length} articles) — ${clientBatchId}`);
let resp;
try {
resp = await pushArticleBatch(cfg, payload);
} catch (e) {
// Whole batch failed (network/5xx exhausted, or 4xx at batch level).
console.error(` ✗ Batch ${clientBatchId} failed: ${e.message}`);
// Mark every article in this batch as error.
for (const a of batch) {
recordResult(state, a.url, { status: 'error', error: { code: 'BATCH_FAILED', message: e.message } });
}
batchResults.push({ clientBatchId, status: 'failed', error: e.message, results: [] });
saveSyncState(dayDir, state);
continue;
}
// Record per-article results from the response.
if (Array.isArray(resp.results)) {
for (const r of resp.results) {
recordResult(state, r.url, r);
}
}
batchResults.push({
clientBatchId,
batchId: resp.batchId,
status: resp.status,
summary: resp.summary,
results: resp.results,
});
console.error(` ✓ Batch ${clientBatchId}: ${resp.status}${JSON.stringify(resp.summary)}`);
saveSyncState(dayDir, state);
}
// 5. Finalize ledger status + output.
const failed = state.articles.filter(a => a.status === 'error').length;
state.lastSyncStatus = failed === 0 ? 'success' : (failed === state.articles.length ? 'failed' : 'partial');
saveSyncState(dayDir, state);
console.error(`\n${failed === 0 ? '✅' : (failed === state.articles.length ? '❌' : '⚠️')} Push ${state.lastSyncStatus}: ${state.articles.length - failed}/${state.articles.length} synced, ${failed} failed`);
if (failed > 0) {
console.error(` Re-run to retry failed articles: node scripts/push.js "${dayDir}"`);
}
const output = {
dayDir, source, harvestDate,
lastSyncStatus: state.lastSyncStatus,
summary: { total: state.articles.length, failed, synced: state.articles.length - failed },
batches: batchResults,
};
process.stdout.write(JSON.stringify(output, null, 2));
// Exit 2 if partial/failed, so callers can detect incomplete pushes.
if (state.lastSyncStatus !== 'success') process.exit(2);
}
// ── --status: read-only ledger report ────────────────────────────────────
function printStatus(dayDir) {
const state = loadSyncState(dayDir);
if (!state) {
console.error(`No ${SYNCSTATE} in ${dayDir} — not pushed yet.`);
process.exit(0);
}
const byStatus = {};
for (const a of state.articles) byStatus[a.status] = (byStatus[a.status] || 0) + 1;
console.error(`📊 Sync status for ${dayDir}`);
console.error(` source: ${state.source} | harvestDate: ${state.harvestDate}`);
console.error(` lastSyncStatus: ${state.lastSyncStatus} | updatedAt: ${state.updatedAt || '?'}`);
console.error(` ${JSON.stringify(byStatus)}`);
const errs = state.articles.filter(a => a.status === 'error');
if (errs.length) {
console.error(`\n ❌ Failed articles:`);
for (const a of errs) {
console.error(` - ${a.slug || a.url}`);
console.error(` ${(a.error && a.error.message) || a.error}`);
}
console.error(`\n Retry: node scripts/push.js "${dayDir}"`);
}
}
// ── CLI ───────────────────────────────────────────────────────────────────
function printHelp() {
console.log(`push.js — Push a finalized day directory to News Hub.
Usage:
node scripts/push.js <dayDir> Push a day directory (incremental)
node scripts/push.js <dayDir> --force Ignore ledger, re-push all articles
node scripts/push.js <dayDir> --bump-processed-at
Override processedAt to now() so the backend's LWW
accepts the update (use when re-pushing corrected
content that was previously pushed with empty/missing
summaries; the original timestamp would be "unchanged")
node scripts/push.js <dayDir> --status Show sync status (read-only)
node scripts/push.js -h | --help Show this help
Arguments:
dayDir Path to the YYYYMMDD directory produced by finalize.js
(e.g. /path/to/vault/Reuters Business/20260731)
Prerequisites:
- <dayDir>/registry.json exists (written by harvest.js, patched by finalize.js)
- <dayDir>/.manifest.json and .summaries.json are ABSENT (= finalize done)
- config.json has a \`push\` section with endpoint + appId/appKeyId/appSecret
What it does:
1. Phase 1: upload each article's images to MinIO (POST /assets, sha256 deduped)
2. Phase 2: batch-push articles (POST /articles, ≤10 per batch, idempotent)
3. Writes <dayDir>/.syncstate.json ledger — re-running resumes from failures
Exit codes:
0 success (all articles synced, or --status printed)
1 usage/IO error (missing files, not finalized, config missing)
2 partial/failed push (some articles still in error state)
Example:
node scripts/push.js "/Users/me/obsidian/Harvester/Reuters Business/20260731"`);
}
function main() {
const args = process.argv.slice(2);
if (args.includes('-h') || args.includes('--help')) {
printHelp();
process.exit(0);
}
const positional = args.filter(a => !a.startsWith('-'));
const flags = args.filter(a => a.startsWith('-'));
if (positional.length === 0) {
console.error('Usage: node scripts/push.js <dayDir> [--force] [--bump-processed-at] [--status]');
console.error('Run "node scripts/push.js --help" for details.');
process.exit(1);
}
const dayDir = positional[0];
if (!fs.existsSync(dayDir) || !fs.statSync(dayDir).isDirectory()) {
console.error(`✗ Day directory does not exist: ${dayDir}`);
process.exit(1);
}
// --status is a pure local ledger read — allow it even when push is disabled,
// so users can inspect sync state without enabling/configuring the backend.
if (flags.includes('--status')) {
printStatus(dayDir);
process.exit(0);
}
const cfg = loadConfig();
if (!cfg.push || !cfg.push.enabled) {
console.error('✗ Push is not enabled in config.json (push.enabled = false).');
console.error(' Configure the push section and set enabled: true to use this command.');
process.exit(1);
}
if (!cfg.push.endpoint || !cfg.push.appId || !cfg.push.appKeyId || !cfg.push.appSecret) {
console.error('✗ Push config incomplete: endpoint/appId/appKeyId/appSecret are all required.');
process.exit(1);
}
pushDayDir(dayDir, cfg, {
force: flags.includes('--force'),
bumpProcessedAt: flags.includes('--bump-processed-at'),
}).catch(e => {
console.error(`Fatal: ${e.message}`);
process.exit(1);
});
}
if (require.main === module) {
main();
}
// Exported for potential finalize.js chaining / testing.
module.exports = {
pushDayDir,
parseFrontmatter,
parseSummarySections,
parseImages,
fileSha256,
readSummaryDoc,
mdToHtmlBody,
};
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment