- 27 Oct, 2026 7 commits
-
-
郭阔 authored
launchd 最小 PATH 不含 /opt/homebrew/bin,keepalive 每小时触发但 4 次全部 exit 127(node: command not found)。对齐 run-web.sh 的 PATH 导出后, launchctl kickstart 实测:访问成功、无墙、实例正常回收。
-
郭阔 authored
- 导航「站点登录」后新增入口 /latest:每源一张可折叠卡片, 摘要行=源名+运行时间+✓n/✗m+已取尽标记,展开为文章标题(链接原文)+成功/失败徽标 - 数据取自各源最新运行 JSON 的业务结果(.result.articles),按运行时间倒序
-
郭阔 authored
- 抽屉改为分行布局:当前定时(chips)/采集频率/第N次逐行+实际时间列, 不再一行平铺;加收起按钮 - 回显:schedule_board 将实际定时按时段归类(与规划器同口径)对应到各次意图; 未配置时段意图的源按实际时间归类生成回显(所见即所排)
-
郭阔 authored
- 操作列由平铺的频率+6时段编辑器改为 ✎排程/
▶ 采集/⏸ 停用(⏵启用)/✕ 一行小按钮, 意图编辑器移到点击后行下方展开的抽屉,表格不再被撑宽 -▶ 采集=立即采集一次(/run),⏸ /⏵=调度与技能总开关(/toggle) - 时间轴上的 06:00 交付线竖线与图例说明移除(规划仍按交付线排,仅不再展示) -
郭阔 authored
- 队列目录 .queue → .queue.port-<port>:同站(同浏览器同账号)互斥, 异站并行(如 nikkei 9315 与 reuters 9408 同时跑,日常即两条流水线) - 站点端口在排队锁之前解析,锁粒度与 runlock 一致 - reuters-world 09:27 实跑验证重试链:attempt1 交付17篇+结构化交付失败 → attempt2 续会话确认 all_seen → 合并17篇并推送成功
-
郭阔 authored
昨夜 nikkei-markets(GLM 无工具会话)与 reuters×3(DataDome 墙)失败时, 模型都通过 structured_output 如实上报 status=FAILED/0交付——但 attempt_needs_retry 只认运行错误(is_error/结构化缺失),业务失败被当'质量失败'直接放弃,attempt2/3 一次没跑。 - attempt_needs_retry 新增 business 判据:status=FAILED 或 0 交付; 取尽豁免优先(all_seen/全非文章页虽自报 FAILED 但业务上是成功,不重跑) - business 失败不走 --resume 续会话(会话确实没干成活,续接只会复述失败), attempt2 同 provider 整场重跑,attempt3 仍切备用 provider - 实测矩阵:四失败信封→重试;成功交付→不重试;取尽→不重试
-
郭阔 authored
- bin/keepalive.sh <site>:解析站点端口/profile,采集进行中自动跳过, 状态写 logs/<site>.keepalive.json - LaunchAgent com.newsharvester.keepalive-reuters(StartInterval 3600,装于本机); 面板调度列表(platform_layer.list_schedules / nh_schedule_list)排除 keepalive*, 不显示为数据源 - 背景:DataDome 按风险评分拦 reuters(出口为数据中心 IP),保温访问维持 cookie 活性降低凌晨采集撞墙概率
-
- 26 Oct, 2026 22 commits
-
-
郭阔 authored
freq dropdown now spawns 第1次..第6次 window selectors (凌晨批次/上午/ 下午/晚间), each run placed independently — e.g. reuters-world 2×/day as overnight+afternoon → 02:00 + 12:00. Sources without explicit windows keep auto semantics (first run in the deadline batch + extras evenly across day_window); legacy string windows normalized. JS toggles run selectors with the freq value; intent persists as a runs list in schedule-plan.json.
-
郭阔 authored
schedule-intent/schedule-gap concatenated str / str (default_conf_dir returns str) → 500 on save; schedule_board already wrapped its copy.
-
郭阔 authored
- freq>=2 with a preferred window now means: first run in the overnight deadline batch + remaining runs spread in the window (reuters-world 2×/day+morning → 04:00 overnight + 07:00 morning) - global min gap is a dropdown on the schedule board (10..120 min), saved via /schedule-gap and immediately re-planned - applied: gap 30min, reuters-world 04:00/07:00
-
郭阔 authored
调度中心 rows now maintain 每日次数 + 时段偏好(自动晨间批次/上午/下午/ 晚间) via dropdowns; POST /schedule-intent persists to schedule-plan.json and immediately re-plans all times. Planner gains windows support: a source with a preferred window gets all its runs evenly placed inside 06:30-11:30 / 12:00-17:30 / 18:00-21:30, outside the morning batch; global gap/deadline math stays automatic. Applied: reuters-technology → morning (06:30), freeing an overnight slot (everything shifts 40min later, more slack before the 06:00 deadline).
-
郭阔 authored
- harvest.sh: global queue lock (runlocks/.queue, mkdir-atomic, stale-PID steal, 180min wait cap) — a source whose time arrives while another is harvesting now waits (phase 排队中) and auto-starts when the previous run exits; runlock is created before queueing so the panel shows queued sources; manual panel trigger no longer refused (it queues) - bin/plan_schedules.py + schedule-plan.json: frequency-driven planner — deadline/safety/min-gap/day-window/avoid all configurable; morning slots laid backward from deadline-safety with exact gaps (wrapping to previous evening when the roster exceeds the night), freq>1 sources spread evenly across the day window, global gap resolution; --dry-run supported - 调度中心:
⚙ 按计划重排全部排程 button (POST /schedule-plan-apply) - applied: 12 sources — 3×4/day (wsj/ft/digitimes-feed latest morning slots) + 9×1/day, 40-min spacing, last start 05:30 vs 06:00 deadline -
郭阔 authored
A fresh attempt2 re-fetched every article (minutes + dollars) just to re-deliver a result whose data was already archived and pushed. attempt2 now extracts session_id from the failed envelope and runs claude --resume with a short re-deliver prompt (no re-collection, no browser); heartbeat is not reset in resume mode. attempt3 (fallback provider) stays a fresh full run — cross-provider resume not trusted.
-
郭阔 authored
When attempt1 delivers data but fails structured reporting, the final attempt fast-exits all_seen with 0 articles — the record then showed nothing delivered, leaving the operator to guess. After the attempt loop, merge earlier attempts' delivered articles (URL-deduped, archive_path required) into the final envelope, recompute counts, and record each attempt's contribution in warnings. Single verdict, full visibility. Unit-tested: 0+3+2(1 dup) → 4 articles, per-attempt notes.
-
郭阔 authored
digitimes updates registry/index incrementally while still fetching, so a mid-fetch registry write was misread as 推送/收尾 and overrode the agent's own correct 采集中 heartbeat. Disk inference has now been wrong in both directions (stale when the agent forgets to flip,超前 on incremental writers) — trust the agent's heartbeat whenever present; disk evidence and neutral 运行中 remain fallbacks only.
-
郭阔 authored
On auto-retry, attempt2 resets the heartbeat to 采集中 and refetches, but attempt1's completion files (registry etc.) are newer than the run start — they overrode the fresh heartbeat and showed 推送/收尾 while the browser was actively fetching. Disk evidence must beat the heartbeat's own mtime to override; attempt resets naturally re-anchor the baseline.
-
郭阔 authored
edit_source checked news-harvester@<name>.timer existence (Linux layout); on macOS schedules are LaunchAgents plists so the edit page always 404ed. Use pl.schedule_state/schedule_times (platform layer) for the guard and current time instead.
-
郭阔 authored
- attempt_needs_retry checked .result type, but the envelope's result is a plain text message even on success — the structured business result lives in .structured_output. Every healthy run was therefore misjudged and silently re-run (attempt2 all_seen fast-exits made it cheap and invisible; reuters-technology went to attempt3 and was then killed by a schedule reload). Criterion now: rc!=0 / empty / invalid JSON / structured_output not an object / is_error. - modify-source.sh refuses to reload a schedule while the source (site) has a live harvest runlock — launchctl bootout terminates the running job (exactly what killed reuters-technology attempt3); --force overrides. Panel schedule-time route already had this guard; CLI is now covered. - reuters-technology record rebuilt from its valid attempt1 envelope (SUCCESS, 10 articles, matching push syncstate).
-
郭阔 authored
The agent occasionally forgets to flip the .phase heartbeat on transition; when the objective disk evidence (registry/.summaries/.manifest mtime) is newer than the heartbeat file, trust the disk. Seen live: heartbeat stuck at 采集中 while manifest had landed 3 minutes earlier.
-
郭阔 authored
Two validation refinements so 'nothing new but junk candidates' stops alarming as FAILED (reuters-markets case: 18 candidates, 16 deduped, the only 2 new ones rejected as non-article pages): - majority-failure rule downgrades to a warning when source_exhausted and every failure is a non-article rejection (og:type/body), same standing as all_seen - preloaded per-URL INVALID_OUTPUT failures no longer set task-failed (task-level no-URL failures and RUNTIME/PERMISSION still fatal); the per-article signal already lives in counts and warnings
-
郭阔 authored
attempt_needs_retry triggers on runtime error / empty output / non-object result (GLM StructuredOutput text-return case) / is_error (1301 etc); content-quality failures do NOT trigger (a provider switch won't fix them). Flow: attempt1 primary → attempt2 same provider (10s backoff) → attempt3 FALLBACK_PROVIDER via resolve_continue_provider with its budget override (ccswitch-gpt=30). Idempotent end-to-end (7-day registry dedup + slug-keyed push), so re-runs only fill gaps; per-attempt raw outputs kept as *.attemptN. Verified decision matrix: success/runtime-error/invalid-output/api-error/ business-fail.
-
郭阔 authored
Disabled/fully-off sources (skill off + schedule not active) no longer appear; transitional state (skill off but schedule still active) stays visible until reconciled.
-
郭阔 authored
-
郭阔 authored
systemd is-active semantics leaked to macOS (merged rows hardcoded active=unknown; scheduled rows showed systemd 'inactive'). Map every row to schedule_state: 调度中 / 已禁用 / 未排程.
-
郭阔 authored
Replace guessed phases with ground truth: harvest.sh seeds logs/<source>.phase=采集中 and the prompt requires the agent to overwrite it the moment it enters each stage (采集中/摘要生成/推送/收尾); trap deletes it on exit. Panel reads the phase file first, falls back to factual disk signals (registry/.summaries/.manifest mtimes), and shows neutral 运行中 when no signal exists — no more inference, no more (推定) labels.
-
郭阔 authored
Redundant runs (all articles deduped by the 7-day registry) burned ~19 turns / $0.6 verifying nothing new. Prompt now instructs: when every candidate is already in the registry, return immediately with empty articles + source_exhausted=true + exhaustion_reason=all_seen — which the result contract already accepts as SUCCESS (exhaustion is not failure), also dodging the GLM StructuredOutput misreport for this case.
-
郭阔 authored
A redundant run (all articles deduped) never rewrites .manifest.json, so the disk-signal inference stayed at 采集中 during the whole summary/finalize phase while the browser sat idle. After 10 minutes with no fetch signals, report 摘要生成(推定) instead.
-
郭阔 authored
active_harvests() only read launchd/systemd service state, so runs started via the panel button or a terminal (not launchd) were invisible on the homepage. Union with runlock-based detection: each port-<port>.<pid>.lock created by harvest.sh itself, PID liveness-checked, source name recovered from the PID's command line — launcher-independent.
-
郭阔 authored
Lock files are port-<port>.<pid>.lock (two dots); ${f##*.} yielded the literal 'lock' instead of the PID, so the liveness check always failed and ANY exiting harvest deleted sibling LIVE locks — with all locks gone the last exiter then quit the site-shared browser out from under a still-running sibling harvest (nikkei-world 09:24 exit killed the browser mid nikkei-markets run; earlier 05:00 run likely degraded the same way until the 60min watchdog TERM'd it at exit=143). Parse correctly: basename minus .lock suffix, then last dot-segment. Self-test: live lock survives, dead-PID lock cleaned.
-
- 23 Oct, 2026 11 commits
-
-
郭阔 authored
上次 trigger time duplicated 最近运行's log mtime — remove it (column now just 下次执行) along with its log-mtime backfill in index().
-
郭阔 authored
launchd exposes no last-trigger either (systemd-only) — fill it from the run log mtime (MM-DD HH:MM); merged unscheduled/disabled rows now carry their parse_last_log too, so only genuinely never-run sources show 从未触发 (verified: 10 ran-today sources show real times, 4 never-run reuters sources remain 从未触发).
-
郭阔 authored
- homepage: drop 成本/Session columns (detail-page info); running badge now shows inferred phase (采集中/摘要生成/推送收尾) via _harvest_phase(): runlock mtime = start, vault-day .manifest/.summaries/.registry mtimes mark phase transitions — zero intrusion into the harvest process - detail page: timeline-style progress bar (采集中→摘要生成→推送/收尾→完成) with pulsing current-stage dot, green done segments, elapsed minutes, 10s auto-refresh while running
-
郭阔 authored
栏目列表 now surfaces running harvests: top banner (⟳ 正在采集: <sources>, browsers stay open until finish), per-row 运行中 badge above 最近运行, and 30s auto-refresh while any harvest is active — previously an in-flight run was invisible until its .last.log landed.
-
郭阔 authored
launchd exposes no next-trigger (systemd-only), so the 栏目列表 下次 column was always '-' on macOS — compute it from schedule times (same _next_fire as the schedule board) and highlight it amber.
-
郭阔 authored
Blue dashed deadline marker at 06:00 on every timeline (交付线 label) + intro copy; multi-time sources show 每日N次 badge. Editorial schedule applied: 02:30 nikkei-world / 03:00 reuters-business / 03:30 bloomberg / 04:00 ft / 04:30 wsj / 05:00+17:00 nikkei-markets (12h interval), all finishing ~35min before the 06:00 deadline.
-
郭阔 authored
app.py honors WEB_HOST/WEB_PORT env (default 127.0.0.1:8080); run-web.sh template exports them from web.conf — set WEB_HOST=0.0.0.0 to reach the panel from other systems on the same network (HTTP + Basic Auth, intranet use intended).
-
郭阔 authored
- new /schedules 调度中心: per-source row with a 24h timeline (green dots at fire times, next fire highlighted amber, red now-line), 定时/下次触发 (countdown)/上次运行 columns, sorted by next fire, 60s auto-refresh - create/update: POST /schedule-time (HH:MM list, validated) → nh_schedule_write + enable (+skill enable for new rows) - delete: POST /schedule-delete/<name> → new platform.sh nh_schedule_remove (bootout + rm plist incl. disabled + entry script + per-source conf; skill source and history kept; digest/web protected) - guards: active harvests block time changes/deletes - nav link 调度中心; schedule_times() string parsed to list
-
郭阔 authored
Clearer purpose naming: the table's function is per-source login verification (test button + last result), not state management.
-
郭阔 authored
站点凭据表新增 人工登录 按钮:拉起站点共享实例窗口→用户在里面登录→ 后台 login-capture.js 每15秒自动检测(最长15分钟)→成功后记录登录态、 缺配方站点自动生成检测配方并回收浏览器实例;页面自动刷新显示 向导状态(运行中/登录已记录/未检测到)。复用 _login_test 轮询通道, 键 manual:<site>。
-
郭阔 authored
- notify-mail.sh: bash 3.2 (stock macOS) treats expanding an empty array under set -u as unbound variable — drop the MSMTP_ARGS array for direct branch invocation (works identically on Linux bash 5) - daily-digest.sh: empty result .json files (run killed / watchdog timeout, claude never wrote output) pass 'jq empty' but emit nothing on extraction, corrupting the summary array with consecutive commas; treat non-empty-size files as parse errors so killed runs show up honestly as interrupted
-