- 28 Oct, 2026 4 commits
-
-
郭阔 authored
- 抽屉新增"手动时间"输入框(HH:MM 逗号分隔,回填当前值),走 /schedule-time - 服务端硬校验:与其他源任一时间或自身多次间隔不足全局最小间隔(24h 回环绕算) → 拒绝保存并列出全部冲突明细(与谁、差多少),不弹确认框 - 实测:02:10 提交被拦并列出与 business 02:00/markets 02:30 冲突,排程未变; 无冲突时间正常保存
-
郭阔 authored
- bin/wall-audit.sh <site> [--compare]:Cookie 清单(datadome 到期)/登录态/ 墙状态/拦截请求网络证据/出口 IP/系统代理/VPN 网卡,快照存 logs/<site>.wall-audit/ - keepalive 撞墙(exit 3)自动抓一份现场快照 - 首份基线:datadome cookie 有效期 365 天(非隔夜过期),出口 IP 与昨日截图不同
-
郭阔 authored
- harvest.sh 运行前检查 .wall-<port>:熔断期内同站源直接跳过(不碰浏览器/ 不耗 API),last.log 记'
⏸ 站点熔断';12h 过期自动解除 - 触发条件:本次 FAILED 且 0 交付 + 反爬特征(≥3 篇 EMPTY_BODY/CAPTCHA/ LOGIN_REQUIRED,或 0 尝试整场拒绝) - keepalive.sh 每小时确认站点正常(wall=false)即删除熔断标记,自动恢复采集 - 调度中心熔断状态显示灰色徽标(区别于红失败) - 实测:昨夜 reuters-world 墙记录判定触发;模拟熔断标记后 nikkei-world 1 秒跳过、零副作用 -
郭阔 authored
-
- 27 Oct, 2026 11 commits
-
-
郭阔 authored
时间表页面本身就展示全部排程,flash 里重复贴排程计划+应用日志冗长; 成功路径只报"已重排,详见下表",失败路径才附输出末尾供排查 (intent/gap/plan-apply 三处一致)
-
郭阔 authored
- harvest-result.py: 要求N篇仅交付M篇且未取尽时,只要有文章成功入库(succeeded>0) → status=WARNING/exit 0,消息进 warnings;交付 0 篇仍判 FAILED - harvest.sh 报告:WARNING →
⚠ ️ 成功(部分交付)(邮件仍按成功分支发) - 面板全站支持黄色徽标:调度中心上次运行/源详情结果状态与历史/运行详情 - 已重判今日 ft 14:30 记录(14/14 交付误判失败→WARNING)并同步修正 last.log -
郭阔 authored
- 导航「添加栏目」后新增 /guide:markdown 实时渲染仓库文档(单一事实源,文档更新自动同步) - web-venv 补装 markdown(阿里镜像);缺库时降级显示原文并提示
-
郭阔 authored
-
郭阔 authored
launchd 最小 PATH 不含 /opt/homebrew/bin,keepalive 每小时触发但 4 次全部 exit 127(node: command not found)。对齐 run-web.sh 的 PATH 导出后, launchctl kickstart 实测:访问成功、无墙、实例正常回收。
-
郭阔 authored
- 导航「站点登录」后新增入口 /latest:每源一张可折叠卡片, 摘要行=源名+运行时间+✓n/✗m+已取尽标记,展开为文章标题(链接原文)+成功/失败徽标 - 数据取自各源最新运行 JSON 的业务结果(.result.articles),按运行时间倒序
-
郭阔 authored
- 抽屉改为分行布局:当前定时(chips)/采集频率/第N次逐行+实际时间列, 不再一行平铺;加收起按钮 - 回显:schedule_board 将实际定时按时段归类(与规划器同口径)对应到各次意图; 未配置时段意图的源按实际时间归类生成回显(所见即所排)
-
郭阔 authored
- 操作列由平铺的频率+6时段编辑器改为 ✎排程/
▶ 采集/⏸ 停用(⏵启用)/✕ 一行小按钮, 意图编辑器移到点击后行下方展开的抽屉,表格不再被撑宽 -▶ 采集=立即采集一次(/run),⏸ /⏵=调度与技能总开关(/toggle) - 时间轴上的 06:00 交付线竖线与图例说明移除(规划仍按交付线排,仅不再展示) -
郭阔 authored
- 队列目录 .queue → .queue.port-<port>:同站(同浏览器同账号)互斥, 异站并行(如 nikkei 9315 与 reuters 9408 同时跑,日常即两条流水线) - 站点端口在排队锁之前解析,锁粒度与 runlock 一致 - reuters-world 09:27 实跑验证重试链:attempt1 交付17篇+结构化交付失败 → attempt2 续会话确认 all_seen → 合并17篇并推送成功
-
郭阔 authored
昨夜 nikkei-markets(GLM 无工具会话)与 reuters×3(DataDome 墙)失败时, 模型都通过 structured_output 如实上报 status=FAILED/0交付——但 attempt_needs_retry 只认运行错误(is_error/结构化缺失),业务失败被当'质量失败'直接放弃,attempt2/3 一次没跑。 - attempt_needs_retry 新增 business 判据:status=FAILED 或 0 交付; 取尽豁免优先(all_seen/全非文章页虽自报 FAILED 但业务上是成功,不重跑) - business 失败不走 --resume 续会话(会话确实没干成活,续接只会复述失败), attempt2 同 provider 整场重跑,attempt3 仍切备用 provider - 实测矩阵:四失败信封→重试;成功交付→不重试;取尽→不重试
-
郭阔 authored
- bin/keepalive.sh <site>:解析站点端口/profile,采集进行中自动跳过, 状态写 logs/<site>.keepalive.json - LaunchAgent com.newsharvester.keepalive-reuters(StartInterval 3600,装于本机); 面板调度列表(platform_layer.list_schedules / nh_schedule_list)排除 keepalive*, 不显示为数据源 - 背景:DataDome 按风险评分拦 reuters(出口为数据中心 IP),保温访问维持 cookie 活性降低凌晨采集撞墙概率
-
- 26 Oct, 2026 22 commits
-
-
郭阔 authored
freq dropdown now spawns 第1次..第6次 window selectors (凌晨批次/上午/ 下午/晚间), each run placed independently — e.g. reuters-world 2×/day as overnight+afternoon → 02:00 + 12:00. Sources without explicit windows keep auto semantics (first run in the deadline batch + extras evenly across day_window); legacy string windows normalized. JS toggles run selectors with the freq value; intent persists as a runs list in schedule-plan.json.
-
郭阔 authored
schedule-intent/schedule-gap concatenated str / str (default_conf_dir returns str) → 500 on save; schedule_board already wrapped its copy.
-
郭阔 authored
- freq>=2 with a preferred window now means: first run in the overnight deadline batch + remaining runs spread in the window (reuters-world 2×/day+morning → 04:00 overnight + 07:00 morning) - global min gap is a dropdown on the schedule board (10..120 min), saved via /schedule-gap and immediately re-planned - applied: gap 30min, reuters-world 04:00/07:00
-
郭阔 authored
调度中心 rows now maintain 每日次数 + 时段偏好(自动晨间批次/上午/下午/ 晚间) via dropdowns; POST /schedule-intent persists to schedule-plan.json and immediately re-plans all times. Planner gains windows support: a source with a preferred window gets all its runs evenly placed inside 06:30-11:30 / 12:00-17:30 / 18:00-21:30, outside the morning batch; global gap/deadline math stays automatic. Applied: reuters-technology → morning (06:30), freeing an overnight slot (everything shifts 40min later, more slack before the 06:00 deadline).
-
郭阔 authored
- harvest.sh: global queue lock (runlocks/.queue, mkdir-atomic, stale-PID steal, 180min wait cap) — a source whose time arrives while another is harvesting now waits (phase 排队中) and auto-starts when the previous run exits; runlock is created before queueing so the panel shows queued sources; manual panel trigger no longer refused (it queues) - bin/plan_schedules.py + schedule-plan.json: frequency-driven planner — deadline/safety/min-gap/day-window/avoid all configurable; morning slots laid backward from deadline-safety with exact gaps (wrapping to previous evening when the roster exceeds the night), freq>1 sources spread evenly across the day window, global gap resolution; --dry-run supported - 调度中心:
⚙ 按计划重排全部排程 button (POST /schedule-plan-apply) - applied: 12 sources — 3×4/day (wsj/ft/digitimes-feed latest morning slots) + 9×1/day, 40-min spacing, last start 05:30 vs 06:00 deadline -
郭阔 authored
A fresh attempt2 re-fetched every article (minutes + dollars) just to re-deliver a result whose data was already archived and pushed. attempt2 now extracts session_id from the failed envelope and runs claude --resume with a short re-deliver prompt (no re-collection, no browser); heartbeat is not reset in resume mode. attempt3 (fallback provider) stays a fresh full run — cross-provider resume not trusted.
-
郭阔 authored
When attempt1 delivers data but fails structured reporting, the final attempt fast-exits all_seen with 0 articles — the record then showed nothing delivered, leaving the operator to guess. After the attempt loop, merge earlier attempts' delivered articles (URL-deduped, archive_path required) into the final envelope, recompute counts, and record each attempt's contribution in warnings. Single verdict, full visibility. Unit-tested: 0+3+2(1 dup) → 4 articles, per-attempt notes.
-
郭阔 authored
digitimes updates registry/index incrementally while still fetching, so a mid-fetch registry write was misread as 推送/收尾 and overrode the agent's own correct 采集中 heartbeat. Disk inference has now been wrong in both directions (stale when the agent forgets to flip,超前 on incremental writers) — trust the agent's heartbeat whenever present; disk evidence and neutral 运行中 remain fallbacks only.
-
郭阔 authored
On auto-retry, attempt2 resets the heartbeat to 采集中 and refetches, but attempt1's completion files (registry etc.) are newer than the run start — they overrode the fresh heartbeat and showed 推送/收尾 while the browser was actively fetching. Disk evidence must beat the heartbeat's own mtime to override; attempt resets naturally re-anchor the baseline.
-
郭阔 authored
edit_source checked news-harvester@<name>.timer existence (Linux layout); on macOS schedules are LaunchAgents plists so the edit page always 404ed. Use pl.schedule_state/schedule_times (platform layer) for the guard and current time instead.
-
郭阔 authored
- attempt_needs_retry checked .result type, but the envelope's result is a plain text message even on success — the structured business result lives in .structured_output. Every healthy run was therefore misjudged and silently re-run (attempt2 all_seen fast-exits made it cheap and invisible; reuters-technology went to attempt3 and was then killed by a schedule reload). Criterion now: rc!=0 / empty / invalid JSON / structured_output not an object / is_error. - modify-source.sh refuses to reload a schedule while the source (site) has a live harvest runlock — launchctl bootout terminates the running job (exactly what killed reuters-technology attempt3); --force overrides. Panel schedule-time route already had this guard; CLI is now covered. - reuters-technology record rebuilt from its valid attempt1 envelope (SUCCESS, 10 articles, matching push syncstate).
-
郭阔 authored
The agent occasionally forgets to flip the .phase heartbeat on transition; when the objective disk evidence (registry/.summaries/.manifest mtime) is newer than the heartbeat file, trust the disk. Seen live: heartbeat stuck at 采集中 while manifest had landed 3 minutes earlier.
-
郭阔 authored
Two validation refinements so 'nothing new but junk candidates' stops alarming as FAILED (reuters-markets case: 18 candidates, 16 deduped, the only 2 new ones rejected as non-article pages): - majority-failure rule downgrades to a warning when source_exhausted and every failure is a non-article rejection (og:type/body), same standing as all_seen - preloaded per-URL INVALID_OUTPUT failures no longer set task-failed (task-level no-URL failures and RUNTIME/PERMISSION still fatal); the per-article signal already lives in counts and warnings
-
郭阔 authored
attempt_needs_retry triggers on runtime error / empty output / non-object result (GLM StructuredOutput text-return case) / is_error (1301 etc); content-quality failures do NOT trigger (a provider switch won't fix them). Flow: attempt1 primary → attempt2 same provider (10s backoff) → attempt3 FALLBACK_PROVIDER via resolve_continue_provider with its budget override (ccswitch-gpt=30). Idempotent end-to-end (7-day registry dedup + slug-keyed push), so re-runs only fill gaps; per-attempt raw outputs kept as *.attemptN. Verified decision matrix: success/runtime-error/invalid-output/api-error/ business-fail.
-
郭阔 authored
Disabled/fully-off sources (skill off + schedule not active) no longer appear; transitional state (skill off but schedule still active) stays visible until reconciled.
-
郭阔 authored
-
郭阔 authored
systemd is-active semantics leaked to macOS (merged rows hardcoded active=unknown; scheduled rows showed systemd 'inactive'). Map every row to schedule_state: 调度中 / 已禁用 / 未排程.
-
郭阔 authored
Replace guessed phases with ground truth: harvest.sh seeds logs/<source>.phase=采集中 and the prompt requires the agent to overwrite it the moment it enters each stage (采集中/摘要生成/推送/收尾); trap deletes it on exit. Panel reads the phase file first, falls back to factual disk signals (registry/.summaries/.manifest mtimes), and shows neutral 运行中 when no signal exists — no more inference, no more (推定) labels.
-
郭阔 authored
Redundant runs (all articles deduped by the 7-day registry) burned ~19 turns / $0.6 verifying nothing new. Prompt now instructs: when every candidate is already in the registry, return immediately with empty articles + source_exhausted=true + exhaustion_reason=all_seen — which the result contract already accepts as SUCCESS (exhaustion is not failure), also dodging the GLM StructuredOutput misreport for this case.
-
郭阔 authored
A redundant run (all articles deduped) never rewrites .manifest.json, so the disk-signal inference stayed at 采集中 during the whole summary/finalize phase while the browser sat idle. After 10 minutes with no fetch signals, report 摘要生成(推定) instead.
-
郭阔 authored
active_harvests() only read launchd/systemd service state, so runs started via the panel button or a terminal (not launchd) were invisible on the homepage. Union with runlock-based detection: each port-<port>.<pid>.lock created by harvest.sh itself, PID liveness-checked, source name recovered from the PID's command line — launcher-independent.
-
郭阔 authored
Lock files are port-<port>.<pid>.lock (two dots); ${f##*.} yielded the literal 'lock' instead of the PID, so the liveness check always failed and ANY exiting harvest deleted sibling LIVE locks — with all locks gone the last exiter then quit the site-shared browser out from under a still-running sibling harvest (nikkei-world 09:24 exit killed the browser mid nikkei-markets run; earlier 05:00 run likely degraded the same way until the 60min watchdog TERM'd it at exit=143). Parse correctly: basename minus .lock suffix, then last dot-segment. Self-test: live lock survives, dead-PID lock cleaned.
-
- 23 Oct, 2026 3 commits
-
-
郭阔 authored
上次 trigger time duplicated 最近运行's log mtime — remove it (column now just 下次执行) along with its log-mtime backfill in index().
-
郭阔 authored
launchd exposes no last-trigger either (systemd-only) — fill it from the run log mtime (MM-DD HH:MM); merged unscheduled/disabled rows now carry their parse_last_log too, so only genuinely never-run sources show 从未触发 (verified: 10 ran-today sources show real times, 4 never-run reuters sources remain 从未触发).
-
郭阔 authored
- homepage: drop 成本/Session columns (detail-page info); running badge now shows inferred phase (采集中/摘要生成/推送收尾) via _harvest_phase(): runlock mtime = start, vault-day .manifest/.summaries/.registry mtimes mark phase transitions — zero intrusion into the harvest process - detail page: timeline-style progress bar (采集中→摘要生成→推送/收尾→完成) with pulsing current-stage dot, green done segments, elapsed minutes, 10s auto-refresh while running
-