Skip to content

M
macos-runtime
Commit c2d5873b authored by 谢宇轩's avatar 谢宇轩
Browse files
Options

feat: firsr version

parents
Hide whitespace changes
Inline Side-by-side
Showing with 4289 additions and 0 deletions
.gitignore 0 → 100644
View file @ c2d5873b
output/
.wb_config.json
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# Visual Studio Code files
.vscode
.vs
# PyCharm files
.idea
# Eclipse Project settings
*.*project
.settings
# Sublime Text settings
*.sublime-workspace
*.sublime-project
# Editor temporaries
*.swn
*.swo
*.swp
*.swm
*~
*.wav
# IPython notebook checkpoints
.ipynb_checkpoints
# macOS dir files
.DS_Store
README.md 0 → 100644
View file @ c2d5873b
# News Harvester Pro
基于 systemd timer + Claude 无头模式的多栏目新闻抓取与监控方案。
每源抓取通过 `claude -p --output-format json` 捕获完整运行元数据(成本、耗时、turns、token、session_id、权限拒绝),落盘为结构化 JSON;独立的每日汇总服务再用 Claude 无头模式 + `--json-schema` 对当天所有抓取记录做智能分析,产出 Markdown 汇总并判断是否需要告警。
---
## 架构总览
```
┌─────────────────────────────────────────────────────────┐
│ 各栏目 timer(每天不同时间触发) │
│ news-harvester@nikkei-world.timer 09:00 │
│ news-harvester@reuters.timer 09:15 │
│ │ │
│ ▼ │
│ news-harvester@<source>.service │
│ │ │
│ ▼ │
│ harvest.sh <source> <count> │
│ ├─ claude -p --output-format json │
│ ├─ 解析元数据(成本/turns/token/session_id) │
│ ├─ 落盘: logs/<source>.<date>.<time>.json │
│ │ logs/<source>.last.log │
│ └─ 按状态发邮件(✅/❌/⚠️,含成本告警) │
│ │
├─────────────────────────────────────────────────────────┤
│ 每日汇总 timer(22:00 触发) │
│ news-harvester-digest.timer │
│ │ │
│ ▼ │
│ news-harvester-digest.service │
│ │ │
│ ▼ │
│ daily-digest.sh │
│ ├─ 扫描 logs/ 下今天的 *.json │
│ ├─ jq 提取每源元数据 → 紧凑 JSON 数组 │
│ ├─ claude -p --json-schema(结构化分析) │
│ │ → {alert, alert_reasons, digest, total_cost} │
│ ├─ 写 logs/digest.<date>.md │
│ ├─ 发汇总邮件(有告警则标题标 [ALERT]) │
│ └─ 清理超期旧日志 │
└─────────────────────────────────────────────────────────┘
```
---
## 目录结构
```
news-harvester-pro/
├── bin/
│ ├── harvest.sh # 单源抓取(被 @.service 调用)
│ ├── continue.sh # 续采/修复(--resume 恢复上次对话)
│ ├── notify-mail.sh # 通用邮件发送(msmtp)
│ ├── daily-digest.sh # 每日汇总 + 告警(被 digest.service 调用)
│ └── status.sh # 友好查看 timer 状态 / 栏目详情 / 汇总
├── systemd/
│ ├── news-harvester@.service # 模板服务,%i=栏目名
│ ├── news-harvester@nikkei-world.timer # 示例:09:00
│ ├── news-harvester@reuters.timer # 示例:09:15
│ ├── news-harvester-digest.service # 每日汇总服务
│ └── news-harvester-digest.timer # 每天 22:00
├── config/
│ ├── default.conf.example # 全局默认配置(COUNT/MAIL_*/DIGEST_*)
│ ├── msmtprc.example # msmtp SMTP 配置示例
│ └── nikkei-world.conf.example # 单栏目覆盖示例
├── migrate.sh # 智能迁移脚本(备份+安装+保留配置)
├── add-source.sh # 新增栏目脚本(自动错开时间+生成timer/conf)
├── modify-source.sh # 修改栏目定时时间及配置(改OnCalendar+conf+重启timer)
├── deploy.sh # 一键部署到服务器(rsync + 同步 bin + systemd)
├── web/ # Flask Web 管理面板
│ ├── app.py # 单文件 Flask 应用(list/add/edit/detail/run)
│ └── templates/ # Jinja2 模板(base/index/add/edit/detail)
└── README.md
```
服务器上的安装位置:
| 本地文件 | 服务器路径 |
|---------|-----------|
| `bin/*.sh` | `/home/yichu/Documents/News-Harvester/bin/` |
| `web/app.py` + `web/templates/` | `/home/yichu/Documents/News-Harvester/web/` |
| `systemd/news-harvester@.service` | `/etc/systemd/system/` |
| `systemd/news-harvester@<source>.timer` | `/etc/systemd/system/` |
| `systemd/news-harvester-digest.*` | `/etc/systemd/system/` |
| `systemd/news-harvester-web.service` | `/etc/systemd/system/` |
| `config/default.conf` | `/etc/news-harvester/default.conf` |
| `config/<source>.conf` | `/etc/news-harvester/<source>.conf` |
| `config/web.conf` | `/etc/news-harvester/web.conf`(Web 面板认证) |
| `.msmtprc` | `~/.msmtprc`(用户家目录) |
---
## 前置依赖
```bash
sudo pacman -S jq msmtp
# Web 面板(可选):
sudo pacman -S python-flask
# claude CLI 需已安装并配置好 API key
# news-harvester 技能需可用(在 claude 的 skills 目录中)
```
---
## 快速开始
### 方式一:全新安装
```bash
# 1. 把项目放到服务器
scp -r news-harvester-pro/ yichu@server:~/
# 2. 进入项目目录,运行迁移脚本(也会处理全新安装)
cd ~/news-harvester-pro
sudo ./migrate.sh --yes
# 3. 编辑配置
sudo vim /etc/news-harvester/default.conf # 填入 MAIL_TO
install -m600 config/msmtprc.example ~/.msmtprc
vim ~/.msmtprc # 填入 SMTP 账号
# 4. 手动测试一次
sudo systemctl start news-harvester@nikkei-world.service
journalctl -u news-harvester@nikkei-world.service -f
```
### 方式二:从旧版迁移
```bash
cd ~/news-harvester-pro
sudo ./migrate.sh
# 脚本会自动:备份旧文件 → 停用旧 timer → 装新版 → 保留 conf → 恢复原 timer 配置
```
### 从本机部署到服务器
本地改完代码后,用 `deploy.sh` 一键同步:
```bash
# 默认用户 yichu
./deploy.sh 192.168.1.100
# 显式指定用户
./deploy.sh yichu@192.168.1.100
# 只更新 bin 脚本,不动 systemd(改脚本不需要 sudo 时用)
./deploy.sh 192.168.1.100 --no-systemd
# 部署后自动跑一次 nikkei-world 测试并实时看日志
./deploy.sh 192.168.1.100 --test
```
deploy.sh 做的事:
1. `rsync` 整个项目到 `~/news-harvester-pro/`(`--delete` 保持镜像,排除 logs/.git)
2. 复制 `bin/*.sh` → `~/Documents/News-Harvester/bin/` 并赋可执行权限
3. 同步 systemd 模板 service + digest unit → `/etc/systemd/system/` 并 `daemon-reload`
**不动的**:栏目 timer(服务器端可能改过时间)、`/etc/news-harvester/*.conf`、`~/.msmtprc`、logs 目录。
---
## 配置详解
### 配置加载顺序
service 模板按顺序加载两个 `EnvironmentFile`,后者覆盖前者(systemd 规则):
1. `/etc/news-harvester/default.conf` — 全局默认
2. `/etc/news-harvester/<source>.conf` — 单栏目覆盖
### 可用变量
| 变量 | 作用 | 默认值 | 用于 |
|------|------|--------|------|
| `COUNT` | 抓取条数 | 20(脚本兜底) | harvest |
| `MAIL_TO` | 收件人;**留空不发邮件**;支持多个(空格或逗号分隔) | 空 | harvest + digest |
| `MAIL_ON_SUCCESS` | 成功是否发邮件:1=发,0=仅失败发 | 1 | harvest |
| `ALERT_COST_OVER` | 单次成本超此值(美元)则标题标 `[ALERT]` | 空(不判断) | harvest |
| `DIGEST_MAIL_ALWAYS` | 每日汇总:1=每次发,0=仅告警时发 | 1 | digest |
| `DIGEST_RETAIN_DAYS` | 旧日志保留天数 | 30 | digest |
| `DIGEST_BUDGET_USD` | 汇总分析的 claude 预算上限 | 2 | digest |
| `WORK_DIR` | 项目工作目录 | `/home/yichu/Documents/News-Harvester` | 两者 |
| `LOG_DIR` | 日志目录 | `<WORK_DIR>/logs` | 两者 |
### 配置示例
**全局默认**(`/etc/news-harvester/default.conf`):
```ini
COUNT=20
MAIL_TO=you@example.com
MAIL_ON_SUCCESS=1
ALERT_COST_OVER=1
DIGEST_MAIL_ALWAYS=1
DIGEST_RETAIN_DAYS=30
DIGEST_BUDGET_USD=2
```
**单栏目覆盖**(`/etc/news-harvester/nikkei-world.conf`):
```ini
COUNT=30
MAIL_ON_SUCCESS=0 # 成功不单独发邮件,只在 digest 里看
ALERT_COST_OVER=2
```
---
## 日志文件
每次抓取运行会落盘:
| 文件 | 说明 | 保留 |
|------|------|------|
| `logs/<source>.<YYYYMMDD>.<HHMMSS>.json` | 完整 claude JSON 输出(digest 分析的数据源) | 按 `DIGEST_RETAIN_DAYS` 自动清理 |
| `logs/<source>.<YYYYMMDD>.<HHMMSS>.raw` | claude 的 stderr(排错用) | 同上 |
| `logs/<source>.last.log` | 人类可读的上次运行摘要(覆盖) | 始终保留最新 |
| `logs/digest.<YYYYMMDD>.md` | 每日汇总 Markdown | 不自动清理 |
查看某源上次运行:
```bash
cat ~/Documents/News-Harvester/logs/nikkei-world.last.log
```
查看今天的所有 JSON 结果:
```bash
jq -r '. | "✅❌\(.is_error) \(.total_cost_usd)\$ \(.num_turns)turns"' ~/Documents/News-Harvester/logs/*.today*.json
```
---
## 邮件通知
> **多收件人**:`MAIL_TO` 支持空格或逗号分隔多个地址(中英文逗号均可),所有收件人会同时收到同一封邮件。
> ```ini
> MAIL_TO=alice@example.com bob@example.com
> # 或
> MAIL_TO=alice@example.com,bob@example.com
> ```
> systemd `EnvironmentFile` 会保留值里的空格,原样传给脚本;`notify-mail.sh` 内部拆分为多个收件人,`To` 头按 RFC 5322 逗号分隔列出,`msmtp` 收件人作为独立参数传入。
### 抓取通知(harvest.sh)
每次抓取后按状态决定是否发邮件:
| 场景 | 发邮件? | 标题示例 |
|------|---------|---------|
| 成功 + `MAIL_ON_SUCCESS=1` | ✅ 发 | `[News Harvester] SUCCESS: nikkei-world` |
| 成功 + `MAIL_ON_SUCCESS=0` | ❌ 不发 | — |
| 成功但成本超 `ALERT_COST_OVER` | ✅ 发 + `[ALERT]` | `[News Harvester] [ALERT] SUCCESS: nikkei-world` |
| 成功但有权限拒绝 | ✅ 发(标 ⚠️) | 同上 |
| 失败 | ✅ 总是发 | `[News Harvester] FAILED: nikkei-world` |
邮件正文含:栏目、状态、退出码、耗时(wall + api)、成本、turns、输入/输出 token、cache token、session_id、权限拒绝数、result 末尾 60 行。
### 每日汇总通知(daily-digest.sh)
每天 22:00 运行,汇总当天所有源:
| 场景 | 发邮件? | 标题示例 |
|------|---------|---------|
| 有告警(任一源失败/异常) | ✅ 发 | `[News Harvester] [ALERT] 每日汇总: 2026-08-17` |
| 无告警 + `DIGEST_MAIL_ALWAYS=1` | ✅ 发 | `[News Harvester] 每日汇总: 2026-08-17` |
| 无告警 + `DIGEST_MAIL_ALWAYS=0` | ❌ 不发 | — |
| 今天没有任何抓取记录 | ✅ 发(视为告警) | `[News Harvester] [ALERT] 无抓取记录: ...` |
汇总邮件正文是 HTML 格式(全内联样式,无外部 CSS),含:概览卡片(源数/成功/失败/总成本/总耗时)、每源状态表(按状态着色)、AI 异常分析列表、建议操作列表。告警时顶部显示红色告警框。完整 Markdown 存档折叠在 `<details>` 中。
### SMTP 配置(msmtp)
```bash
install -m600 config/msmtprc.example ~/.msmtprc
vim ~/.msmtprc # 填入 host/port/user/password
chmod 600 ~/.msmtprc # 必须 600
```
测试 SMTP:
```bash
echo -e 'To: you@example.com\nSubject: test\n\nhello' | msmtp you@example.com
```
---
## Web 管理面板
除了命令行操作,项目还提供一个 Flask Web 面板,可以在浏览器中查看所有栏目状态、添加/编辑/删除栏目、手动触发抓取和汇总。
### 首次安装
```bash
# 1. 安装依赖
sudo pacman -S python-flask
# 2. 创建认证配置
sudo cp config/web.conf.example /etc/news-harvester/web.conf
sudo chmod 600 /etc/news-harvester/web.conf
sudo nano /etc/news-harvester/web.conf # 修改 WEB_USER / WEB_PASS
# 3. 安装 systemd unit
sudo cp systemd/news-harvester-web.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now news-harvester-web.service
# 4. 通过 SSH 隧道访问(监听 127.0.0.1:8080,不对外暴露)
ssh -L 8080:127.0.0.1:8080 yichu@<server-ip>
# 然后浏览器打开 http://localhost:8080
```
### 功能
| 页面 | 路径 | 功能 |
|------|------|------|
| 栏目列表 | `/` | 所有 source 的 timer 状态、运行时间、上次/下次触发、最近运行摘要(状态/成本/session) |
| 栏目详情 | `/source/<name>` | timer 信息、配置、最近运行日志(.last.log 全文)、Claude JSON 元数据(成本/turns/tokens/session_id/权限拒绝) |
| 添加栏目 | `/add` | 表单创建新栏目,调用 add-source.sh(自动错开时间) |
| 编辑栏目 | `/edit/<name>` | 表单修改运行时间/抓取条数/收件人,调用 modify-source.sh |
| 手动触发 | `/run/<name>` | 一键触发指定栏目的抓取 |
| 手动汇总 | `/digest` | 一键触发每日汇总 |
| 删除栏目 | `/delete/<name>` | 停用并删除 timer/conf(需二次确认) |
### 安全说明
- Web 服务监听 `127.0.0.1:8080`,**不对外网暴露**,需通过 SSH 隧道访问
- 使用 HTTP Basic Auth 认证,密码配置在 `/etc/news-harvester/web.conf`(权限 600)
- 服务以 root 运行(因为需要修改 systemd timer 文件),但只暴露受限的操作路由
- 建议仅在受信任的内网环境使用
### 更新 Web 面板
修改 `web/app.py` 或模板后,重新部署即可:
```bash
./deploy.sh <server-ip>
# deploy.sh 会自动同步 web/ 目录并 daemon-reload
# 如果修改了 app.py,需重启服务:
ssh yichu@<server-ip> 'sudo systemctl restart news-harvester-web.service'
```
---
## 日常操作
### 查看运行状态(status.sh)
友好的终端状态查看工具,带颜色高亮:
```bash
# 所有 timer 概览 + 最近运行摘要
./status.sh
# 某栏目详情(最近一次运行摘要)
./status.sh nikkei-tech
# 某栏目完整日志
./status.sh nikkei-tech --log
# 某栏目最近一次的 claude JSON 输出(jq 美化)
./status.sh nikkei-tech --json
# 某栏目的 systemd journal 日志(最近 50 行)
./status.sh nikkei-tech --journal
# 今天的每日汇总
./status.sh --digest
# 只列出所有栏目的最近日志摘要(一行一个)
./status.sh --all-logs
```
概览模式输出示例:
```
📰 News Harvester — Timer 概览
2026-08-17 18:30:00
TIMER 状态 上次触发 下次触发
────────────────────────────────────────────────────────────────────────────────
news-harvester@bloomberg-technology.timer waiting 2026-08-17 16:53:16 2026-08-18 09:30:00
news-harvester@nikkei-tech.timer waiting 2026-08-17 17:26:44 2026-08-18 09:00:00
news-harvester-digest.timer waiting 2026-08-17 22:00:00 2026-08-18 22:00:00
📊 最近运行
✅ bloomberg-technology 成功 | $0.77
✅ nikkei-tech 成功 | $0.76
✅ reuters-business 成功 | $0.69
```
### 查看所有定时器
```bash
systemctl list-timers 'news-harvester@*' news-harvester-digest.timer
```
### 手动触发某个栏目
```bash
sudo systemctl start news-harvester@nikkei-world.service
journalctl -u news-harvester@nikkei-world.service -f
```
### 手动触发每日汇总
```bash
sudo systemctl start news-harvester-digest.service
journalctl -u news-harvester-digest.service -f
# 查看生成的汇总
cat ~/Documents/News-Harvester/logs/digest.$(date +%Y%m%d).md
```
### 继续采集 / 采集修复 (continue.sh)
当某次采集因模型超时、网络中断等原因没有完成,或者采集数据出现错乱(登录失效、付费墙等),可以用 `continue.sh` 通过 Claude `--resume <session_id>` 恢复上次对话:
```bash
cd ~/Documents/News-Harvester/bin
# 续采(默认):在已有基础上继续完成中断的采集
./continue.sh nikkei-world --mode resume
# 修复:清理上次错误数据后重新完整采集(force 推送)
./continue.sh nikkei-world --mode repair
# 指定预算上限(默认 $15)
./continue.sh nikkei-world --mode resume --budget 10
# 跳过确认
./continue.sh nikkei-world --mode resume --yes
```
两种模式说明:
| 模式 | 说明 | 适用场景 |
|------|------|----------|
| `resume` | 恢复对话,检查当前状态,未完成则继续,已完成则确认返回 | 模型超时、网络中断、进程被杀 |
| `repair` | 先清理上次归档的错误数据,再重新完整采集并 force 推送 | 登录失效、付费墙、数据错乱 |
脚本会自动从最新的 `.json` 日志中提取 `session_id`,无需手动指定。日志文件用 `.continue.` 前缀区分:
- `logs/<source>.continue.<YYYYMMDD>.<HHMMSS>.json` — 完整 JSON
- `logs/<source>.continue.<YYYYMMDD>.<HHMMSS>.raw` — stderr
- `logs/<source>.last.log` — 覆盖写入(与 harvest.sh 共用,status.sh 和 Web 面板自动展示)
邮件通知 subject 带 `[CONTINUE]` 前缀,正文格式与正常采集通知一致。
也可以通过 Web 面板操作:在栏目详情页点击「继续采集」或「采集修复」按钮。
### 查看日志
```bash
# 某个栏目
journalctl -u news-harvester@nikkei-world.service --since today
# 今天所有栏目 + 汇总
journalctl -u 'news-harvester@*' -u news-harvester-digest.service --since today
```
### 修改某栏目的运行时间
#### 方式一:用脚本(推荐)
```bash
sudo ./modify-source.sh <source-name> --time HH:MM [选项]
```
脚本会读取当前时间、修改 timer 的 `OnCalendar`、同步 conf(若指定),然后 `daemon-reload` + 重启 timer 使新时间立即生效。比手动 `systemctl edit` 更省事,且能顺便更新 conf 里的抓取条数、收件人等参数。
选项:
| 选项 | 说明 |
|------|------|
| `--time HH:MM` | 新的运行时间(必填) |
| `--count N` | 更新抓取条数,写入/覆盖 `<source>.conf` |
| `--mail-to ADDR` | 更新收件人 |
| `--mail-on-success 0\|1` | 更新成功是否发邮件 |
| `--yes` / `-y` | 跳过确认 |
示例:
```bash
# 只改时间
sudo ./modify-source.sh nikkei-world --time 10:30
# 改时间 + 顺便更新条数和发信设置
sudo ./modify-source.sh bloomberg --time 14:00 --count 20 --mail-on-success 0
```
#### 方式二:手动 systemctl edit
用 drop-in(不改原文件):
```bash
sudo systemctl edit news-harvester@nikkei-world.timer
```
```ini
[Timer]
OnCalendar=
OnCalendar=*-*-* 09:30:00
```
```bash
sudo systemctl daemon-reload
systemctl restart news-harvester@nikkei-world.timer # 重启使新时间生效
systemctl list-timers news-harvester@nikkei-world.timer # 确认
```
> 第一行空的 `OnCalendar=` 必须有,先清空原值,否则会叠加触发。
**改 `[Timer]` 段内容只需 `daemon-reload` + `restart` timer,不需要重新 `enable`。**
### 新增一个栏目
#### 方式一:用脚本(推荐)
```bash
sudo ./add-source.sh <source-name> [选项]
```
脚本会自动生成 timer(+ 可选 conf)、daemon-reload、enable。**不指定 `--time` 时,自动扫描现有所有栏目 timer,取最晚一个 +15 分钟作为运行时间**,省去手动错开的麻烦。
选项:
| 选项 | 说明 |
|------|------|
| `--time HH:MM` | 指定运行时间(不指定则自动错开) |
| `--count N` | 抓取条数,写入 `<source>.conf` |
| `--mail-to ADDR` | 收件人,写入 `<source>.conf` |
| `--mail-on-success 0\|1` | 成功是否发邮件,写入 `<source>.conf` |
| `--yes` / `-y` | 跳过确认 |
| `--no-enable` | 只创建文件不启用 |
| `--force` | 同名 timer 已存在时覆盖 |
示例:
```bash
# 自动错开时间,其余继承 default.conf
sudo ./add-source.sh nikkei-asia
# 指定时间 + 条数 + 成功不发邮件
sudo ./add-source.sh bloomberg --time 09:30 --count 30 --mail-on-success 0
# 指定收件人(覆盖 default)
sudo ./add-source.sh reuters --time 09:15 --mail-to team@example.com
# 只建文件,稍后手动启用
sudo ./add-source.sh nikkei-markets --time 10:00 --no-enable
```
执行后会打印后续操作命令(手动测试、查看日志等)。
#### 方式二:手动三步
```bash
# 1. 复制 timer 改时间
sudo sed 's/nikkei-world/bloomberg/g; s/09:00:00/09:30:00/' \
/etc/systemd/system/news-harvester@nikkei-world.timer \
| sudo tee /etc/systemd/system/news-harvester@bloomberg.timer
# 2. (可选)单栏目配置
echo -e 'COUNT=30\nMAIL_ON_SUCCESS=0' | sudo tee /etc/news-harvester/bloomberg.conf
# 3. 启用
sudo systemctl daemon-reload
sudo systemctl enable --now news-harvester@bloomberg.timer
```
### 停用 / 删除一个栏目
```bash
sudo systemctl disable --now news-harvester@reuters.timer
sudo rm /etc/systemd/system/news-harvester@reuters.timer
sudo systemctl daemon-reload
```
### 回滚到旧版
```bash
sudo ./migrate.sh --uninstall
# 自动恢复最近一次备份
```
---
## daemon-reload vs enable 速查
| 改动 | 处理方式 |
|------|---------|
| 改 `OnCalendar` / `Persistent` / `AccuracySec` | `daemon-reload` 即可 |
| 改 `[Install]` 的 `WantedBy=` | 重新 `enable` |
| 改 timer **文件名** | `disable` 旧的 + 装新的 + `enable` |
| 改 service 的 `ExecStart` / `EnvironmentFile` | `daemon-reload`(下次运行生效) |
| 改 `Documentation=` / `Description=` | `daemon-reload`(纯元信息) |
记法:**改内容 → daemon-reload;改身份(名字/Install)→ 重新 enable。**
---
## 无头模式说明
本项目使用 Claude CLI 的无头模式,关键点:
- `claude -p` 非交互运行,退出码 0=成功,非 0=失败
- `--output-format json` 返回单个完整 JSON 对象,含:
- `result`(结果文本)、`total_cost_usd`(成本)
- `num_turns`、`duration_api_ms`、`session_id`
- `usage`(input/output/cache token)
- `permission_denials`(权限拒绝列表)
- `is_error`、`subtype`、`terminal_reason`
- `--json-schema` 强制返回结构化输出(digest 用此功能)
- `--max-budget-usd` 硬性预算上限
- `--permission-mode bypassPermissions` + `--allowed-tools` 控制工具权限
harvest.sh 用 `jq` 解析 JSON,提取元数据写入日志和邮件。digest.sh 用 `--json-schema` 让 Claude 返回 `{alert, alert_reasons, digest, total_cost, failed_sources}` 结构化结果。
---
## 注意事项
1. **PATH 问题(最易踩坑)**:systemd 不加载 shell profile,脚本循环 source 了 `.bashrc/.bash_profile/.profile/.zshrc`。若手动测试仍报 `claude: command not found`,先 `which claude` 拿绝对路径替换脚本里的 `claude`,或取消脚本里 `export PATH=...` 注释。
2. **jq 必需**:新版依赖 `jq` 解析 JSON 输出。未安装时 harvest 会降级为无法提取元数据,digest 直接报错退出。
3. **错开时间**:多栏目不要全设同一时刻,避免同时起多个 claude 进程撞限流。每个栏目预算独立(默认 `$10`),digest 预算 `$2`。
4. **时区**:timer 按服务器本地时区触发。确认:`timedatectl`,不对就 `sudo timedatectl set-timezone Asia/Shanghai`。
5. **补跑**:`Persistent=true`,若触发时刻服务器关机,开机后自动补跑。
6. **技能名**:技能真名是 `news-harvester`(注意拼写),脚本提示词已用正确拼写。
7. **成本是估算**:文档明确说明 `total_cost_usd` 是客户端估算,可能与实际账单有差异。
8. **日志清理**:digest 每次运行后自动删除超过 `DIGEST_RETAIN_DAYS` 天的 `.json` 和 `.raw` 文件,`digest.*.md` 和 `*.last.log` 不自动清理。
add-source.sh 0 → 100755
View file @ c2d5873b
#!/bin/bash
# add-source.sh — 新增一个抓取栏目(timer + 可选 conf),自动错开时间
#
# 用法:
# sudo ./add-source.sh <source-name> [选项]
#
# 选项:
# --time HH:MM 运行时间,如 09:30(不指定则自动错开:现有最晚时间 +15 分钟)
# --count N 抓取条数(写入 <source>.conf)
# --mail-to ADDR 收件人(写入 <source>.conf)
# --mail-on-success 0|1 成功是否发邮件(写入 <source>.conf)
# --yes, -y 跳过确认
# --no-enable 只创建文件,不 enable
# --force 同名 timer 已存在时覆盖
# --help, -h 显示帮助
#
# 示例:
# sudo ./add-source.sh bloomberg --time 09:30 --count 30 --mail-on-success 0
# sudo ./add-source.sh nikkei-asia # 自动错开到下一个空档
#
# 环境变量(可覆盖默认路径):
# SYSTEMD_DIR systemd unit 目录(默认 /etc/systemd/system)
# CONF_DIR 栏目配置目录(默认 /etc/news-harvester)
set -euo pipefail
SYSTEMD_DIR="${SYSTEMD_DIR:-/etc/systemd/system}"
CONF_DIR="${CONF_DIR:-/etc/news-harvester}"
SOURCE=""
TIME=""
COUNT=""
MAIL_TO=""
MAIL_ON_SUCCESS=""
ASSUME_NO=0
NO_ENABLE=0
FORCE=0
usage() {
sed -n '3,30p' "$0"
}
# --- 参数解析 ---
while [ $# -gt 0 ]; do
case "$1" in
--time) TIME="${2:?--time 需要值}"; shift 2;;
--count) COUNT="${2:?--count 需要值}"; shift 2;;
--mail-to) MAIL_TO="${2:?--mail-to 需要值}"; shift 2;;
--mail-on-success) MAIL_ON_SUCCESS="${2:?--mail-on-success 需要值}"; shift 2;;
--yes|-y) ASSUME_NO=1; shift;;
--no-enable) NO_ENABLE=1; shift;;
--force) FORCE=1; shift;;
--help|-h) usage; exit 0;;
--) shift; break;;
-*) echo "未知选项: $1" >&2; usage >&2; exit 1;;
*)
if [ -z "$SOURCE" ]; then SOURCE="$1"
else echo "多余的参数: $1" >&2; exit 1; fi
shift;;
esac
done
# --- 辅助函数 ---
err() { echo "❌ $*" >&2; exit 1; }
ok() { echo "✅ $*"; }
warn() { echo "⚠️ $*"; }
info() { echo "ℹ️ $*"; }
confirm() {
[ "$ASSUME_NO" = "1" ] && return 0
local reply
read -rp "$1 [y/N] " reply
case "$reply" in y|Y|yes|YES) return 0 ;; *) return 1 ;; esac
}
# --- 校验 ---
[ -n "$SOURCE" ] || err "请指定栏目名:sudo $0 <source-name> [选项]"
[[ "$SOURCE" =~ ^[a-zA-Z0-9_-]+$ ]] || err "栏目名只能含字母、数字、下划线、连字符: $SOURCE"
# 避免和 digest 冲突
[ "$SOURCE" = "digest" ] && err "栏目名不能用 digest(与汇总服务冲突)"
[ "$(id -u)" -eq 0 ] || err "请用 sudo 运行"
# 检查依赖:service 模板 + harvest.sh
[ -f "$SYSTEMD_DIR/news-harvester@.service" ] || err "未找到 $SYSTEMD_DIR/news-harvester@.service,请先运行 migrate.sh 安装"
[ -f "/home/yichu/Documents/News-Harvester/bin/harvest.sh" ] || warn "harvest.sh 不在默认位置,确认已安装(可忽略若路径不同)"
# 校验时间格式
validate_time() {
[[ "$1" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || err "时间格式应为 HH:MM(24小时制),如 09:30,得到: $1"
}
validate_count() {
[[ "$1" =~ ^[0-9]+$ ]] && [ "$1" -gt 0 ] || err "count 应为正整数: $1"
}
validate_mos() {
case "$1" in 0|1) ;; *) err "--mail-on-success 只能是 0 或 1: $1";; esac
}
[ -n "$COUNT" ] && validate_count "$COUNT"
[ -n "$MAIL_ON_SUCCESS" ] && validate_mos "$MAIL_ON_SUCCESS"
# --- 检查是否已存在 ---
TIMER_PATH="$SYSTEMD_DIR/news-harvester@${SOURCE}.timer"
CONF_PATH="$CONF_DIR/${SOURCE}.conf"
if [ -f "$TIMER_PATH" ] && [ "$FORCE" = "0" ]; then
err "timer 已存在: $TIMER_PATH\n 如需覆盖,加 --force;如需改时间,用:\n sudo systemctl edit --full news-harvester@${SOURCE}.timer"
fi
# --- 自动推算时间 ---
suggest_time() {
# 扫描现有所有 news-harvester@*.timer 的 OnCalendar,取最晚时间 +15 分钟
local max_epoch=0
shopt -s nullglob
for t in "$SYSTEMD_DIR"/news-harvester@*.timer; do
[ -f "$t" ] || continue
local line tc h m epoch
line=$(grep '^OnCalendar=' "$t" 2>/dev/null | head -1)
[ -z "$line" ] && continue
# 形如 OnCalendar=*-*-* 09:00:00,提取 HH:MM:SS
tc=$(printf '%s' "$line" | grep -oE '[0-9]{2}:[0-9]{2}:[0-9]{2}' | head -1)
[ -z "$tc" ] && continue
h=${tc:0:2}; m=${tc:3:2}
epoch=$((10#$h * 60 + 10#$m))
[ "$epoch" -gt "$max_epoch" ] && max_epoch=$epoch
done
shopt -u nullglob
if [ "$max_epoch" -eq 0 ]; then
echo "09:00" # 没有任何现有 timer,默认 09:00
return
fi
local new_epoch=$((max_epoch + 15))
if [ "$new_epoch" -ge 1440 ]; then
# 超过 23:59,wrap 到次日 00:00 起算
new_epoch=$((new_epoch - 1440))
warn "最晚栏目已接近午夜,新栏目自动 wrap 到次日 $new_epoch"
fi
printf '%02d:%02d\n' $((new_epoch / 60)) $((new_epoch % 60))
}
if [ -z "$TIME" ]; then
TIME=$(suggest_time)
AUTO_TIME=1
else
validate_time "$TIME"
AUTO_TIME=0
fi
# --- 展示计划,确认 ---
echo "=== 即将创建栏目: $SOURCE ==="
echo " 运行时间: $TIME"
[ "$AUTO_TIME" = "1" ] && echo " (自动推算:现有最晚栏目 +15 分钟)"
echo " timer 文件: $TIMER_PATH"
if [ -n "$COUNT" ] || [ -n "$MAIL_TO" ] || [ -n "$MAIL_ON_SUCCESS" ]; then
echo " conf 文件: $CONF_PATH"
[ -n "$COUNT" ] && echo " COUNT=$COUNT"
[ -n "$MAIL_TO" ] && echo " MAIL_TO=$MAIL_TO"
[ -n "$MAIL_ON_SUCCESS" ] && echo " MAIL_ON_SUCCESS=$MAIL_ON_SUCCESS"
else
echo " conf 文件: (不创建,继承 default.conf)"
fi
[ "$NO_ENABLE" = "1" ] && echo " 启用: 否(--no-enable)"
echo ""
confirm "确认创建?" || { echo "已取消。"; exit 0; }
# --- 写 timer ---
mkdir -p "$SYSTEMD_DIR"
cat > "$TIMER_PATH" <<EOF
[Unit]
Description=Run News Harvester ($SOURCE) daily at $TIME
[Timer]
OnCalendar=*-*-* ${TIME}:00
Persistent=true
AccuracySec=1min
Unit=news-harvester@${SOURCE}.service
[Install]
WantedBy=timers.target
EOF
ok "已创建 $TIMER_PATH"
# --- 写 conf(仅当有覆盖项) ---
if [ -n "$COUNT" ] || [ -n "$MAIL_TO" ] || [ -n "$MAIL_ON_SUCCESS" ]; then
mkdir -p "$CONF_DIR"
{
echo "# /etc/news-harvester/${SOURCE}.conf — 由 add-source.sh 生成 $(date '+%Y-%m-%d %H:%M:%S')"
echo "# 仅写需要覆盖的字段,其余继承 default.conf"
echo ""
[ -n "$COUNT" ] && echo "COUNT=$COUNT"
[ -n "$MAIL_TO" ] && echo "MAIL_TO=$MAIL_TO"
[ -n "$MAIL_ON_SUCCESS" ] && echo "MAIL_ON_SUCCESS=$MAIL_ON_SUCCESS"
} > "$CONF_PATH"
ok "已创建 $CONF_PATH"
fi
# --- daemon-reload + enable ---
systemctl daemon-reload
ok "systemd 已重新加载"
if [ "$NO_ENABLE" = "0" ]; then
systemctl enable --now "news-harvester@${SOURCE}.timer"
ok "已启用并启动 news-harvester@${SOURCE}.timer"
fi
echo ""
echo "========================================"
ok "栏目 $SOURCE 添加完成!"
echo "========================================"
echo ""
echo "后续操作:"
echo " 查看定时器: systemctl list-timers news-harvester@${SOURCE}.timer"
echo " 手动测试: sudo systemctl start news-harvester@${SOURCE}.service"
echo " 实时日志: journalctl -u news-harvester@${SOURCE}.service -f"
echo " 改时间: sudo systemctl edit --full news-harvester@${SOURCE}.timer (改后 daemon-reload)"
echo " 查看运行结果: cat ~/Documents/News-Harvester/logs/${SOURCE}.last.log"
echo ""
echo "提示:此栏目今天的抓取记录会自动被每日汇总(digest)纳入分析。"
bin/continue.sh 0 → 100755
View file @ c2d5873b
#!/bin/bash
# continue.sh — 续采/修复脚本,通过 claude --resume <session_id> 恢复上次对话
#
# 用法:
# ./continue.sh <source> [选项]
#
# 选项:
# --mode resume|repair resume=中断续采(默认),repair=清理重采
# --budget N 单次预算上限美元(默认 15)
# --yes, -y 跳过确认
# --help, -h 显示帮助
#
# 模式说明:
# resume 上次采集因模型/网络等原因中断,在已有基础上继续完成。
# 如果已经全部完成,检查确认后返回结果总结即可。
# repair 上次采集数据错乱(登录失效/付费墙等),先清理本次产生的错误数据,
# 再重新完整采集归档推送(force 模式覆盖)。
#
# 环境变量(由 service 的 EnvironmentFile 注入,同 harvest.sh):
# MAIL_TO 收件人;留空则不发邮件
# MAIL_ON_SUCCESS 1=成功也发,0=仅失败发(默认 1)
# ALERT_COST_OVER 单次成本(美元)超过此值则在邮件标题标记 [ALERT]
# WORK_DIR 项目工作目录(默认 /home/yichu/Documents/News-Harvester)
# LOG_DIR 日志目录(默认 <WORK_DIR>/logs)
# --- 设置 PATH 和 nvm(等价于 .zshrc 中的配置) ---
# Web 面板通过 sudo -u yichu 调用时不加载 .zshrc,需显式设置
export NVM_DIR="/home/yichu/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
export PATH="/home/yichu/.local/bin:/home/yichu/.nvm/versions/node/v24.14.0/bin:$PATH"
SOURCE=""
MODE="resume"
BUDGET=15
ASSUME_YES=0
DEBUG=0
usage() {
sed -n '3,28p' "$0"
}
# --- 参数解析 ---
while [ $# -gt 0 ]; do
case "$1" in
--mode) MODE="${2:?--mode 需要值}"; shift 2;;
--budget) BUDGET="${2:?--budget 需要值}"; shift 2;;
--yes|-y) ASSUME_YES=1; shift;;
--debug) DEBUG=1; shift;;
--help|-h) usage; exit 0;;
--) shift; break;;
-*) echo "未知选项: $1" >&2; usage >&2; exit 1;;
*)
if [ -z "$SOURCE" ]; then SOURCE="$1"
else echo "多余的参数: $1" >&2; exit 1; fi
shift;;
esac
done
# --- 辅助函数 ---
err() { echo "❌ $*" >&2; exit 1; }
ok() { echo "✅ $*"; }
warn() { echo "⚠️ $*"; }
info() { echo "ℹ️ $*"; }
debug() { [ "$DEBUG" = "1" ] && echo "🔍 [DEBUG] $*" >&2 || true; }
confirm() {
[ "$ASSUME_YES" = "1" ] && return 0
local reply
read -rp "$1 [y/N] " reply
case "$reply" in y|Y|yes|YES) return 0 ;; *) return 1 ;; esac
}
# --- 校验 ---
[ -n "$SOURCE" ] || err "请指定栏目名: $0 <source> [选项]"
[[ "$SOURCE" =~ ^[a-zA-Z0-9_-]+$ ]] || err "栏目名只能含字母、数字、下划线、连字符: $SOURCE"
case "$MODE" in
resume|repair) ;;
*) err "--mode 只能是 resume 或 repair: $MODE";;
esac
[[ "$BUDGET" =~ ^[0-9]+\.?[0-9]*$ ]] || err "--budget 应为正数: $BUDGET"
# --- 加载配置(同 harvest.sh 的 EnvironmentFile 层级) ---
# Web 面板直接调用时不走 systemd,EnvironmentFile 不会自动加载,需手动 source
CONF_DIR="${CONF_DIR:-/etc/news-harvester}"
# 先加载 default.conf,再加载 source.conf(后者覆盖前者)
for _conf in "$CONF_DIR/default.conf" "$CONF_DIR/${SOURCE}.conf"; do
[ -f "$_conf" ] && source "$_conf"
done
MAIL_TO="${MAIL_TO:-}"
MAIL_ON_SUCCESS="${MAIL_ON_SUCCESS:-1}"
ALERT_COST_OVER="${ALERT_COST_OVER:-}"
WORK_DIR="${WORK_DIR:-/home/yichu/Documents/News-Harvester}"
LOG_DIR="${LOG_DIR:-$WORK_DIR/logs}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
mkdir -p "$LOG_DIR"
# --- 从最新 JSON 日志提取 session_id ---
# 匹配普通采集日志(<source>.*.json)和 continue 日志(<source>.continue.*.json)
LATEST_JSON=$(ls -t "$LOG_DIR"/${SOURCE}.*.json 2>/dev/null | head -1)
if [ -z "$LATEST_JSON" ]; then
err "找不到 ${SOURCE} 的 JSON 日志文件,无法获取 session_id\n 请先正常运行一次采集: sudo systemctl start news-harvester@${SOURCE}.service"
fi
if ! command -v jq >/dev/null 2>&1; then
err "需要 jq 来解析 JSON 日志,请安装: sudo pacman -S jq"
fi
if ! jq empty "$LATEST_JSON" 2>/dev/null; then
err "JSON 日志文件损坏,无法解析: $LATEST_JSON"
fi
SESSION_ID=$(jq -r '.session_id // empty' "$LATEST_JSON")
if [ -z "$SESSION_ID" ] || [ "$SESSION_ID" = "null" ]; then
err "JSON 日志中没有 session_id,无法恢复对话: $LATEST_JSON\n 可能上次运行 claude 启动失败,未产生有效 session"
fi
MODE_TAG=""
case "$MODE" in
resume) MODE_TAG="续采";;
repair) MODE_TAG="修复";;
esac
info "找到上次运行日志: $LATEST_JSON"
info "Session ID: $SESSION_ID"
info "模式: ${MODE_TAG} (${MODE})"
debug "当前用户: $(whoami)"
debug "claude 路径: $(command -v claude 2>/dev/null || echo 未找到)"
debug "WORK_DIR=$WORK_DIR LOG_DIR=$LOG_DIR"
echo ""
# --- 组装 prompt ---
if [ "$MODE" = "resume" ]; then
PROMPT="上次的采集任务似乎因为中断（模型超时、网络问题等）没有完成。请执行以下操作：
1. 检查当前 Obsidian 中 ${SOURCE} 的归档状态，确认哪些文章已经成功归档、哪些还缺失。
2. 如果还有未完成的文章，继续抓取和归档剩余部分。
3. 如果已经全部完成且没有问题，检查确认后直接返回结果总结即可。
执行要求：
- 必须在前台同步完成所有抓取和归档，不要把浏览器自动化或任何步骤放到后台任务中运行。
- 不要使用后台 shell、nohup、& 或任何形式的异步执行。
- 如果某篇文章抓取失败，跳过它继续下一篇，不要因此中断整个任务。
- 最终结果中简要说明：本次续采成功归档几篇、失败几篇、各失败原因。"
else
PROMPT="之前的 ${SOURCE} 采集遇到了问题（数据源登录失效、付费墙、数据错乱等），需要修复。请执行以下操作：
1. 先清理之前本次采集归档的错误数据（删除本次产生的错误文章文件）。
2. 重新对 ${SOURCE} 数据源进行完整采集、摘要生成和归档。
3. 归档推送时使用 force 模式，覆盖可能已有的同名文件。
4. 完成后返回结果总结。
执行要求：
- 必须在前台同步完成所有抓取和归档，不要把浏览器自动化或任何步骤放到后台任务中运行。
- 不要使用后台 shell、nohup、& 或任何形式的异步执行。
- 如果某篇文章抓取失败，跳过它继续下一篇，不要因此中断整个任务。
- 最终结果中简要说明：修复后成功归档几篇、失败几篇、各失败原因。"
fi
# --- 确认 ---
echo "=== 即将执行: ${MODE_TAG} [${SOURCE}] ==="
echo " 模式: ${MODE} (${MODE_TAG})"
echo " Session: ${SESSION_ID}"
echo " 预算上限: \$${BUDGET}"
echo " 日志来源: ${LATEST_JSON}"
[ -n "$MAIL_TO" ] && echo " 邮件通知: ${MAIL_TO}" || echo " 邮件通知: (无)"
echo ""
confirm "确认执行?" || { echo "已取消。"; exit 0; }
# --- 准备日志文件 ---
DATE_TAG="$(date +%Y%m%d)"
TIME_TAG="$(date +%H%M%S)"
JSON_OUT="$LOG_DIR/${SOURCE}.continue.${DATE_TAG}.${TIME_TAG}.json"
RAW_OUT="$LOG_DIR/${SOURCE}.continue.${DATE_TAG}.${TIME_TAG}.raw"
HUMAN_LOG="$LOG_DIR/${SOURCE}.last.log"
cd "$WORK_DIR"
START_EPOCH=$(date +%s)
# --- 运行 claude 无头模式,--resume 恢复对话 ---
debug "执行命令: claude -p --resume $SESSION_ID --permission-mode bypassPermissions --allowed-tools Bash Write --output-format json --max-budget-usd $BUDGET <prompt>"
debug "JSON_OUT=$JSON_OUT"
debug "RAW_OUT=$RAW_OUT"
debug "PROMPT 长度: ${#PROMPT} 字符"
set +e
claude -p \
--resume "$SESSION_ID" \
--permission-mode bypassPermissions \
--allowed-tools Bash Write \
--output-format json \
--max-budget-usd "$BUDGET" \
"$PROMPT" \
> "$JSON_OUT" 2> "$RAW_OUT"
RC=$?
set -e
debug "claude 退出码: $RC"
debug "JSON_OUT 大小: $(wc -c < "$JSON_OUT" 2>/dev/null || echo 0) bytes"
debug "RAW_OUT 大小: $(wc -c < "$RAW_OUT" 2>/dev/null || echo 0) bytes"
if [ "$DEBUG" = "1" ]; then
debug "RAW_OUT 内容前 500 字符:"
head -c 500 "$RAW_OUT" 2>/dev/null >&2
echo "" >&2
debug "JSON_OUT 内容前 500 字符:"
head -c 500 "$JSON_OUT" 2>/dev/null >&2
echo "" >&2
fi
# 如果 claude 完全没输出,写入诊断信息到 RAW_OUT
if [ ! -s "$JSON_OUT" ] && [ ! -s "$RAW_OUT" ]; then
{
echo "===== 诊断信息(claude 无任何输出) ====="
echo "时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "退出码: $RC"
echo "当前用户: $(whoami)"
echo "PATH: $PATH"
echo ""
echo "尝试检测 claude 是否可用:"
echo " claude --version:"
claude --version 2>&1 || echo " 失败(exit $?)"
echo ""
echo "which claude:"
which claude 2>&1 || echo " 未找到"
} > "$RAW_OUT"
fi
END_EPOCH=$(date +%s)
WALL_DURATION=$((END_EPOCH - START_EPOCH))
# --- 解析 JSON 元数据(同 harvest.sh) ---
HAVE_JQ=0
command -v jq >/dev/null 2>&1 && [ -s "$JSON_OUT" ] && HAVE_JQ=1
if [ "$HAVE_JQ" = "1" ]; then
if ! jq empty "$JSON_OUT" 2>/dev/null; then
HAVE_JQ=0
fi
fi
if [ "$HAVE_JQ" = "1" ]; then
IS_ERROR=$(jq -r '.is_error // "unknown"' "$JSON_OUT")
SUBTYPE=$(jq -r '.subtype // "unknown"' "$JSON_OUT")
COST=$(jq -r '.total_cost_usd // 0' "$JSON_OUT")
NUM_TURNS=$(jq -r '.num_turns // 0' "$JSON_OUT")
API_MS=$(jq -r '.duration_api_ms // 0' "$JSON_OUT")
SESSION_ID=$(jq -r '.session_id // "n/a"' "$JSON_OUT")
IN_TOK=$(jq -r '.usage.input_tokens // 0' "$JSON_OUT")
OUT_TOK=$(jq -r '.usage.output_tokens // 0' "$JSON_OUT")
CACHE_READ=$(jq -r '.usage.cache_read_input_tokens // 0' "$JSON_OUT")
CACHE_CREATE=$(jq -r '.usage.cache_creation_input_tokens // 0' "$JSON_OUT")
DENIALS=$(jq -r '.permission_denials | length' "$JSON_OUT")
TERMINAL=$(jq -r '.terminal_reason // "n/a"' "$JSON_OUT")
RESULT_TEXT=$(jq -r '.result // "(无输出)"' "$JSON_OUT")
else
IS_ERROR="unknown"; SUBTYPE="unknown"; COST=0; NUM_TURNS=0; API_MS=0
SESSION_ID="n/a"; IN_TOK=0; OUT_TOK=0; CACHE_READ=0; CACHE_CREATE=0
DENIALS=0; TERMINAL="n/a"
RESULT_TEXT="(无法解析 JSON 输出,请查看 .raw 文件;可能 claude 启动失败或未安装 jq)"
fi
# --- 判定状态 ---
if [ "$RC" -eq 0 ] && [ "$IS_ERROR" != "true" ]; then
STATUS_EMOJI="✅"
STATUS_TEXT="成功"
STATUS_TAG="SUCCESS"
else
STATUS_EMOJI="❌"
STATUS_TEXT="失败"
STATUS_TAG="FAILED"
fi
COST_ALERT=0
if [ -n "$ALERT_COST_OVER" ]; then
awk -v c="$COST" -v t="$ALERT_COST_OVER" 'BEGIN{exit !(c+0 > t+0)}' && COST_ALERT=1
fi
if [ "$DENIALS" -gt 0 ] && [ "$STATUS_TAG" = "SUCCESS" ]; then
STATUS_EMOJI="⚠️"
STATUS_TEXT="成功(但有 ${DENIALS} 次权限拒绝)"
fi
# --- 格式化耗时 ---
MINS=$((WALL_DURATION / 60))
SECS=$((WALL_DURATION % 60))
API_SECS=$(awk -v ms="$API_MS" 'BEGIN{printf "%.1f", ms/1000}')
# --- 写人类可读日志 ---
{
echo "===== $(date '+%Y-%m-%d %H:%M:%S') ${MODE_TAG} [${SOURCE}] (mode=${MODE}) ====="
echo "状态: ${STATUS_EMOJI} ${STATUS_TEXT}"
echo "退出码: ${RC}"
echo "耗时: ${MINS}m${SECS}s (wall) / ${API_SECS}s (api)"
echo "成本: \$${COST}"
echo "Turns: ${NUM_TURNS}"
echo "Token: 输入 ${IN_TOK} / 输出 ${OUT_TOK} / cache读 ${CACHE_READ} / cache写 ${CACHE_CREATE}"
echo "Session: ${SESSION_ID}"
echo "拒绝: ${DENIALS} 次权限拒绝"
echo "Terminal: ${TERMINAL}"
echo ""
echo "----- result -----"
echo "$RESULT_TEXT"
echo ""
echo "----- JSON 结果文件 -----"
echo "$JSON_OUT"
} > "$HUMAN_LOG"
# --- 邮件通知 ---
SHOULD_NOTIFY=0
if [ -n "$MAIL_TO" ]; then
if [ "$STATUS_TAG" = "FAILED" ]; then
SHOULD_NOTIFY=1
elif [ "$MAIL_ON_SUCCESS" = "1" ]; then
SHOULD_NOTIFY=1
fi
fi
if [ "$SHOULD_NOTIFY" = "1" ]; then
ALERT_PREFIX=""
if [ "$STATUS_TAG" = "FAILED" ] || [ "$COST_ALERT" = "1" ]; then
ALERT_PREFIX="[ALERT] "
fi
SUBJECT="[News Harvester] [CONTINUE] ${ALERT_PREFIX}${STATUS_TAG}: ${SOURCE} (${MODE_TAG})"
RESULT_TAIL=$(printf '%s\n' "$RESULT_TEXT" | tail -n 60)
BODY="$(cat <<EOF
${STATUS_EMOJI} News Harvester ${MODE_TAG}任务通知
栏目: ${SOURCE}
模式: ${MODE} (${MODE_TAG})
状态: ${STATUS_TEXT}
退出码: ${RC}
耗时: ${MINS}m${SECS}s (wall) / ${API_SECS}s (api)
成本: \$${COST}
Turns: ${NUM_TURNS}
Token: 输入 ${IN_TOK} / 输出 ${OUT_TOK}
cache 读 ${CACHE_READ} / cache 写 ${CACHE_CREATE}
Session: ${SESSION_ID}
权限拒绝: ${DENIALS}
Terminal: ${TERMINAL}
EOF
)"
if [ -n "$ALERT_COST_OVER" ]; then
if [ "$COST_ALERT" = "1" ]; then
BODY+=$'\n成本阈值: \$'"${ALERT_COST_OVER} (本次超标 ⚠️)"
else
BODY+=$'\n成本阈值: \$'"${ALERT_COST_OVER} (本次正常)"
fi
fi
BODY+=$'\n\n----- result 末尾 60 行 -----\n'"${RESULT_TAIL}"
printf '%s' "$BODY" | "$SCRIPT_DIR/notify-mail.sh" "$MAIL_TO" "$SUBJECT" \
2>> "$RAW_OUT" || echo "[notify] 邮件发送失败(已忽略)" >> "$HUMAN_LOG"
fi
exit "$RC"
bin/daily-digest.sh 0 → 100755
View file @ c2d5873b
#!/bin/bash
# 每日抓取汇总脚本 —— 由 news-harvester-digest.service 调用
#
# 设计原则:概况数字 + 每源表格由脚本用 jq 确定性计算;Claude 只负责
# 异常分析和建议(脚本算不出的语义判断),避免 AI 输出与邮件固定区块重复。
#
# 工作流程:
# 1. 扫描 LOG_DIR 下今天的 *.json 结果文件
# 2. jq 提取每源元数据 → 紧凑 JSON 数组
# 3. 脚本计算确定性概况(源数/成功/失败/总成本/总turns/总耗时/总token)
# 4. 调 claude --json-schema,只让它返回 anomalies/recommendations/alert
# 5. 写 logs/digest.<date>.md(纯文本存档)
# 6. 发送 HTML 汇总邮件(全内联样式,无外部 CSS)
# 7. 清理超期旧日志
#
# 环境变量:
# MAIL_TO 收件人;留空不发邮件
# DIGEST_MAIL_ALWAYS 1=每次发,0=仅告警时发(默认 1)
# WORK_DIR 工作目录(默认 /home/yichu/Documents/News-Harvester)
# LOG_DIR 日志目录(默认 <WORK_DIR>/logs)
# DIGEST_RETAIN_DAYS 旧日志保留天数(默认 30)
# DIGEST_BUDGET_USD 汇总分析 claude 预算上限(默认 2)
# COST_ALERT_THRESHOLD 单源成本告警阈值(美元,默认 1)
# --- 设置 PATH 和 nvm(等价于 .zshrc 中的配置) ---
# systemd service 和 web 面板(sudo -u yichu)都是非交互 shell,不加载 .zshrc
export NVM_DIR="/home/yichu/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
export PATH="/home/yichu/.local/bin:/home/yichu/.nvm/versions/node/v24.14.0/bin:$PATH"
# --- 手动加载配置(Web 面板直接调用时不走 systemd EnvironmentFile) ---
# systemd 调用时 EnvironmentFile 已加载,再 source 一次是幂等的
CONF_DIR="${CONF_DIR:-/etc/news-harvester}"
[ -f "$CONF_DIR/default.conf" ] && source "$CONF_DIR/default.conf"
MAIL_TO="${MAIL_TO:-}"
DIGEST_MAIL_ALWAYS="${DIGEST_MAIL_ALWAYS:-1}"
WORK_DIR="${WORK_DIR:-/home/yichu/Documents/News-Harvester}"
LOG_DIR="${LOG_DIR:-$WORK_DIR/logs}"
DIGEST_RETAIN_DAYS="${DIGEST_RETAIN_DAYS:-30}"
DIGEST_BUDGET_USD="${DIGEST_BUDGET_USD:-2}"
COST_ALERT_THRESHOLD="${COST_ALERT_THRESHOLD:-1}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
mkdir -p "$LOG_DIR"
TODAY="$(date +%Y%m%d)"
TODAY_HUMAN="$(date '+%Y-%m-%d')"
SUMMARY_JSON="$LOG_DIR/.digest-summary.${TODAY}.json"
DIGEST_MD="$LOG_DIR/digest.${TODAY}.md"
DIGEST_RAW="$LOG_DIR/.digest-raw.${TODAY}.txt"
CLAUDE_JSON="$LOG_DIR/.digest-claude.${TODAY}.json"
cd "$WORK_DIR"
# --- 辅助函数 ---
html_esc() { sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g; s/"/\&quot;/g; s/\x27/\&#39;/g'; }
# ============================================================
# 1. 收集今天的 JSON 结果文件
# ============================================================
TODAY_FILES=()
shopt -s nullglob
for f in "$LOG_DIR"/*."${TODAY}."*.json; do
TODAY_FILES+=("$f")
done
shopt -u nullglob
if ! command -v jq >/dev/null 2>&1; then
echo "[digest] 需要 jq:sudo pacman -S jq" >&2; exit 1
fi
# ============================================================
# 2. jq 提取每源元数据 → JSON 数组
# ============================================================
build_summary() {
echo -n "["
local first=1
for f in "${TODAY_FILES[@]}"; do
local base src rest ts
base="$(basename "$f")"
src="${base%%.*}"
rest="${base#*.}"
ts="${rest#*.}"; ts="${ts%%.*}"
if ! jq empty "$f" 2>/dev/null; then
[ "$first" = "1" ] || echo -n ","; first=0
jq -nc --arg src "$src" --arg ts "$ts" --arg file "$f" \
'{source:$src,time:$ts,status:"PARSE_ERROR",cost:0,turns:0,duration_ms:0,input_tokens:0,output_tokens:0,denials:0,result:"(JSON 解析失败)",file:$file}'
continue
fi
[ "$first" = "1" ] || echo -n ","; first=0
jq -c --arg src "$src" --arg ts "$ts" '{
source:$src,time:$ts,
is_error:(.is_error//false),
status:(if(.is_error//false) then "FAILED" else "SUCCESS" end),
subtype:(.subtype//"unknown"),
cost:(.total_cost_usd//0),
turns:(.num_turns//0),
duration_ms:(.duration_api_ms//0),
input_tokens:(.usage.input_tokens//0),
output_tokens:(.usage.output_tokens//0),
denials:((.permission_denials//[])|length),
terminal:(.terminal_reason//"n/a"),
result:(.result//"")
}' "$f"
done
echo -n "]"
}
build_summary > "$SUMMARY_JSON"
SOURCE_COUNT="${#TODAY_FILES[@]}"
# --- 没有数据 ---
if [ "$SOURCE_COUNT" -eq 0 ]; then
echo "[digest] 今天 ($TODAY_HUMAN) 没有抓取记录。" | tee "$DIGEST_MD"
if [ -n "$MAIL_TO" ] && [ "$DIGEST_MAIL_ALWAYS" = "1" ]; then
BODY='<p>今天没有发现任何抓取记录。</p><p>可能所有 timer 未触发或脚本未落盘 JSON。</p>'
BODY+='<p>请检查:<br><code>systemctl list-timers "news-harvester@*"</code><br><code>journalctl -u "news-harvester@*" --since today</code></p>'
BODY=$(wrap_html "$BODY" "⚠️ 无抓取记录" "$TODAY_HUMAN" "alert")
printf '%s' "$BODY" | "$SCRIPT_DIR/notify-mail.sh" "$MAIL_TO" "[News Harvester] [ALERT] 无抓取记录: $TODAY_HUMAN" html \
2>> "$DIGEST_RAW" || true
fi
exit 0
fi
# ============================================================
# 3. 脚本计算确定性概况
# ============================================================
TOTAL_COST=$(jq '[.[].cost]|add // 0' "$SUMMARY_JSON" | awk '{printf "%.2f", $0}')
SUCCESS_COUNT=$(jq '[.[]|select(.status=="SUCCESS")]|length' "$SUMMARY_JSON" 2>/dev/null || echo 0)
FAILED_COUNT=$((SOURCE_COUNT - SUCCESS_COUNT))
TOTAL_TURNS=$(jq '[.[].turns]|add // 0' "$SUMMARY_JSON")
TOTAL_DURATION_MS=$(jq '[.[].duration_ms]|add // 0' "$SUMMARY_JSON")
TOTAL_IN_TOK=$(jq '[.[].input_tokens]|add // 0' "$SUMMARY_JSON")
TOTAL_OUT_TOK=$(jq '[.[].output_tokens]|add // 0' "$SUMMARY_JSON")
TOTAL_DENIALS=$(jq '[.[].denials]|add // 0' "$SUMMARY_JSON")
TOTAL_DURATION_S=$(awk -v ms="$TOTAL_DURATION_MS" 'BEGIN{printf "%.1f", ms/1000}')
# 唯一源数(同源可能跑多次)
UNIQUE_SOURCES=$(jq '[.[].source]|unique|length' "$SUMMARY_JSON")
# ============================================================
# 4. 调 claude --json-schema(只分析异常和建议)
# ============================================================
SCHEMA='{"type":"object","properties":{"alert":{"type":"boolean"},"alert_reasons":{"type":"array","items":{"type":"string"}},"anomalies":{"type":"array","items":{"type":"string"},"description":"异常项说明。每条简短。不要重复表格数据,只描述需关注的模式或问题"},"recommendations":{"type":"array","items":{"type":"string"},"description":"建议操作。每条简短可执行。无异常则空数组"},"failed_sources":{"type":"array","items":{"type":"string"}}},"required":["alert","alert_reasons","anomalies","recommendations","failed_sources"]}'
SUMMARY_CONTENT="$(cat "$SUMMARY_JSON")"
set +e
claude -p \
--permission-mode bypassPermissions \
--allowed-tools Bash Read Write \
--output-format json \
--json-schema "$SCHEMA" \
--max-budget-usd "$DIGEST_BUDGET_USD" \
"你是新闻抓取系统的监控助手。下面是今天($TODAY_HUMAN)各数据源的抓取运行数据(JSON 数组)。
重要:概况数字和每源表格由脚本自动生成并展示在邮件中,你**不需要**重复这些。你只负责:
1. anomalies:分析需要关注的异常模式(不是简单重复表格):
- result 文本是否暗示未完成(如只说"正在后台抓取"但无归档结果)
- 同一源当天多次运行(可能首次失败后重跑,产生重复成本)
- 成本/耗时/token 异常偏高的源
- 权限拒绝、解析失败等
2. recommendations:可执行建议(如有)
3. alert:是否存在需要人工介入的异常
4. alert_reasons:告警原因(简短,每条一句话)
5. failed_sources:失败的源名称列表
判断参考:
- status 为 FAILED/PARSE_ERROR → 告警
- 单源成本超 \$$COST_ALERT_THRESHOLD → 标注
- 权限拒绝 > 0 → 标注
- result 提到"失败""错误""error""未完成""后台"等 → 关注
- 同源当日多次运行 → 说明原因
汇总数据:
$SUMMARY_CONTENT" \
> "$CLAUDE_JSON" 2> "$DIGEST_RAW"
RC=$?
set -e
# ============================================================
# 5. 解析 claude 结果(有兜底)
# ============================================================
if [ "$RC" -ne 0 ] || ! jq empty "$CLAUDE_JSON" 2>/dev/null; then
echo "[digest] claude 分析失败(exit=$RC),脚本规则兜底。" >&2
ALERT="true"
FAILED=$(jq -r '[.[]|select(.status=="FAILED" or .status=="PARSE_ERROR")|.source]|unique|join(",")' "$SUMMARY_JSON")
[ -z "$FAILED" ] && ALERT="false"
if [ "$ALERT" = "true" ]; then
ALERT_REASONS='["有源失败: '"${FAILED}"'"]'
ANOMALIES_JSON='["脚本兜底:检测到失败的源,请查看日志排查"]'
else
ALERT_REASONS='[]'; ANOMALIES_JSON='[]'
fi
RECOMMENDATIONS_JSON='[]'
AI_NOTE="(claude 分析失败,以下为脚本规则兜底结果)"
else
ALERT=$(jq -r '.structured_output.alert // false' "$CLAUDE_JSON")
ALERT_REASONS=$(jq -c '.structured_output.alert_reasons // []' "$CLAUDE_JSON")
ANOMALIES_JSON=$(jq -c '.structured_output.anomalies // []' "$CLAUDE_JSON")
RECOMMENDATIONS_JSON=$(jq -c '.structured_output.recommendations // []' "$CLAUDE_JSON")
FAILED=$(jq -r '.structured_output.failed_sources // []|join(",")' "$CLAUDE_JSON")
AI_NOTE=""
fi
# ============================================================
# 6. 写 digest.md(纯文本存档)
# ============================================================
{
echo "# 每日抓取汇总 — ${TODAY_HUMAN}"
echo ""
echo "数据源: ${UNIQUE_SOURCES} 个(共 ${SOURCE_COUNT} 次运行)"
echo "成功/失败: ${SUCCESS_COUNT}/${FAILED_COUNT}"
echo "总成本: \$${TOTAL_COST}"
echo "总 Turns: ${TOTAL_TURNS} 总耗时: ${TOTAL_DURATION_S}s"
echo "总 Token: 输入 ${TOTAL_IN_TOK} / 输出 ${TOTAL_OUT_TOK}"
echo "权限拒绝: ${TOTAL_DENIALS}"
echo "告警: ${ALERT}"
[ -n "$AI_NOTE" ] && echo "(${AI_NOTE})"
echo ""
echo "## 各源详情"
jq -r '.[] | "- \(.source) \(.time) \(.status) | $\(.cost) | \(.turns)turns | \(.duration_ms)ms | 拒绝\(.denials)"' "$SUMMARY_JSON"
echo ""
if [ "$(echo "$ANOMALIES_JSON" | jq 'length')" -gt 0 ]; then
echo "## 异常项"
echo "$ANOMALIES_JSON" | jq -r '.[] | "- \(.)"'
echo ""
fi
if [ "$(echo "$RECOMMENDATIONS_JSON" | jq 'length')" -gt 0 ]; then
echo "## 建议"
echo "$RECOMMENDATIONS_JSON" | jq -r '.[] | "- \(.)"'
echo ""
fi
if [ "$ALERT" = "true" ]; then
echo "## 告警原因"
echo "$ALERT_REASONS" | jq -r '.[] | "- \(.)"'
fi
echo ""
echo "---"
echo "生成时间: $(date '+%Y-%m-%d %H:%M:%S')"
} > "$DIGEST_MD"
# ============================================================
# 7. 发送 HTML 邮件
# ============================================================
# 生成表格行 HTML
build_table_rows() {
jq -r '.[] | [.source,.time,.status,.cost,.turns,.duration_ms,.input_tokens,.output_tokens,.denials] | @tsv' "$SUMMARY_JSON" \
| while IFS=$'\t' read -r src time status cost turns dur_ms in_tok out_tok denials; do
src_e=$(printf '%s' "$src" | html_esc)
case "$status" in
FAILED|PARSE_ERROR) bg="#fde8e8"; icon="❌" ;;
*) if [ "${denials:-0}" -gt 0 ]; then bg="#fff4e0"; icon="⚠️"; else bg="#ffffff"; icon="✅"; fi ;;
esac
dur_s=$(awk -v ms="${dur_ms:-0}" 'BEGIN{printf "%.1f", ms/1000}')
cost_fmt=$(awk -v c="${cost:-0}" 'BEGIN{printf "%.2f", c}')
printf '<tr style="background:%s;">' "$bg"
printf '<td style="padding:8px 12px;border-bottom:1px solid #eee;font-weight:600;">%s</td>' "$src_e"
printf '<td style="padding:8px 12px;border-bottom:1px solid #eee;color:#718096;font-size:12px;">%s</td>' "$time"
printf '<td style="padding:8px 12px;border-bottom:1px solid #eee;">%s %s</td>' "$icon" "$status"
printf '<td style="padding:8px 12px;border-bottom:1px solid #eee;text-align:right;">$%s</td>' "$cost_fmt"
printf '<td style="padding:8px 12px;border-bottom:1px solid #eee;text-align:right;">%s</td>' "$turns"
printf '<td style="padding:8px 12px;border-bottom:1px solid #eee;text-align:right;">%ss</td>' "$dur_s"
printf '<td style="padding:8px 12px;border-bottom:1px solid #eee;text-align:right;font-size:12px;">%s/%s</td>' "$in_tok" "$out_tok"
printf '<td style="padding:8px 12px;border-bottom:1px solid #eee;text-align:right;">%s</td>' "$denials"
printf '</tr>\n'
done
}
# JSON 数组 → HTML 列表
json_to_list() {
local json="$1" empty_msg="$2"
local count
count=$(echo "$json" | jq 'length' 2>/dev/null || echo 0)
if [ "$count" -eq 0 ]; then
echo "<p style=\"color:#a0aec0;font-size:13px;margin:0;\">${empty_msg}</p>"
return
fi
echo '<ul style="margin:0;padding-left:20px;">'
echo "$json" | jq -r '.[] | . | gsub("&";"&amp;") | gsub("<";"&lt;") | gsub(">";"&gt;")' \
| while IFS= read -r line; do
printf '<li style="margin-bottom:6px;line-height:1.5;">%s</li>\n' "$line"
done
echo '</ul>'
}
# 组装 HTML 邮件
if [ -n "$MAIL_TO" ]; then
SHOULD_SEND=0
[ "$ALERT" = "true" ] && SHOULD_SEND=1
[ "$DIGEST_MAIL_ALWAYS" = "1" ] && SHOULD_SEND=1
if [ "$SHOULD_SEND" = "1" ]; then
ALERT_PREFIX=""; [ "$ALERT" = "true" ] && ALERT_PREFIX="[ALERT] "
SUBJECT="[News Harvester] ${ALERT_PREFIX}每日汇总: ${TODAY_HUMAN}"
TABLE_ROWS=$(build_table_rows)
# 告警框
ALERT_BOX=""
if [ "$ALERT" = "true" ]; then
REASONS_LIST=$(json_to_list "$ALERT_REASONS" "无")
ALERT_BOX="<div style=\"margin:0 0 20px;padding:14px 18px;background:#fde8e8;border-left:4px solid #e53e3e;border-radius:4px;\">
<div style=\"font-weight:700;color:#c53030;margin-bottom:8px;\">⚠️ 告警</div>
${REASONS_LIST}
</div>"
fi
# 异常项
ANO_COUNT=$(echo "$ANOMALIES_JSON" | jq 'length' 2>/dev/null || echo 0)
if [ "$ANO_COUNT" -gt 0 ]; then
ANO_LIST=$(json_to_list "$ANOMALIES_JSON" "")
ANO_SECTION="<div style=\"margin:0 0 20px;\">
<div style=\"font-size:15px;font-weight:700;margin-bottom:10px;color:#dd6b20;\">🔍 异常分析${AI_NOTE:+ <span style=\"font-size:12px;font-weight:400;color:#a0aec0;\">${AI_NOTE}</span>}</div>
${ANO_LIST}
</div>"
else
ANO_SECTION="<div style=\"margin:0 0 20px;padding:12px 16px;background:#f0fff4;border-radius:4px;border-left:4px solid #38a169;\">
<div style=\"color:#276749;font-weight:600;\">✅ 未发现异常,所有源正常运行</div>
</div>"
fi
# 建议
REC_COUNT=$(echo "$RECOMMENDATIONS_JSON" | jq 'length' 2>/dev/null || echo 0)
if [ "$REC_COUNT" -gt 0 ]; then
REC_LIST=$(json_to_list "$RECOMMENDATIONS_JSON" "")
REC_SECTION="<div style=\"margin:0 0 20px;\">
<div style=\"font-size:15px;font-weight:700;margin-bottom:10px;color:#2b6cb0;\">💡 建议操作</div>
${REC_LIST}
</div>"
else
REC_SECTION=""
fi
# 概况卡片:成功/失败颜色
if [ "$FAILED_COUNT" -gt 0 ]; then
FAIL_COLOR="#e53e3e"
else
FAIL_COLOR="#a0aec0"
fi
HTML_BODY="$(cat <<EOF
<!DOCTYPE html>
<html lang="zh-CN">
<head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"></head>
<body style="margin:0;padding:0;background:#f4f5f7;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,'Helvetica Neue',Arial,sans-serif;color:#2d3748;">
<div style="max-width:680px;margin:0 auto;padding:24px;">
<div style="background:#fff;border-radius:8px;overflow:hidden;box-shadow:0 1px 3px rgba(0,0,0,0.08);">
<!-- 标题栏 -->
<div style="padding:24px 28px;background:linear-gradient(135deg,#2b6cb0,#2c7a7b);color:#fff;">
<div style="font-size:20px;font-weight:700;">📰 News Harvester 每日汇总</div>
<div style="font-size:14px;opacity:0.9;margin-top:4px;">${TODAY_HUMAN} · ${UNIQUE_SOURCES} 个数据源 · ${SOURCE_COUNT} 次运行</div>
</div>
<div style="padding:24px 28px;">
${ALERT_BOX}
<!-- 概览卡片 -->
<div style="display:flex;flex-wrap:wrap;gap:12px;margin-bottom:24px;">
<div style="flex:1;min-width:100px;background:#f7fafc;border-radius:6px;padding:14px;text-align:center;">
<div style="font-size:22px;font-weight:700;color:#2b6cb0;">${SOURCE_COUNT}</div>
<div style="font-size:11px;color:#718096;margin-top:2px;">总运行</div>
</div>
<div style="flex:1;min-width:100px;background:#f7fafc;border-radius:6px;padding:14px;text-align:center;">
<div style="font-size:22px;font-weight:700;color:#38a169;">${SUCCESS_COUNT}</div>
<div style="font-size:11px;color:#718096;margin-top:2px;">成功</div>
</div>
<div style="flex:1;min-width:100px;background:#f7fafc;border-radius:6px;padding:14px;text-align:center;">
<div style="font-size:22px;font-weight:700;color:${FAIL_COLOR};">${FAILED_COUNT}</div>
<div style="font-size:11px;color:#718096;margin-top:2px;">失败</div>
</div>
<div style="flex:1;min-width:100px;background:#f7fafc;border-radius:6px;padding:14px;text-align:center;">
<div style="font-size:22px;font-weight:700;color:#805ad5;">\$${TOTAL_COST}</div>
<div style="font-size:11px;color:#718096;margin-top:2px;">总成本</div>
</div>
<div style="flex:1;min-width:100px;background:#f7fafc;border-radius:6px;padding:14px;text-align:center;">
<div style="font-size:22px;font-weight:700;color:#4a5568;">${TOTAL_DURATION_S}s</div>
<div style="font-size:11px;color:#718096;margin-top:2px;">总耗时</div>
</div>
</div>
<!-- 每源表格 -->
<div style="font-size:15px;font-weight:700;margin-bottom:10px;color:#2d3748;">各源运行详情</div>
<table style="width:100%;border-collapse:collapse;font-size:13px;">
<thead>
<tr style="background:#edf2f7;">
<th style="padding:10px 12px;text-align:left;border-bottom:2px solid #cbd5e0;">栏目</th>
<th style="padding:10px 12px;text-align:left;border-bottom:2px solid #cbd5e0;">时间</th>
<th style="padding:10px 12px;text-align:left;border-bottom:2px solid #cbd5e0;">状态</th>
<th style="padding:10px 12px;text-align:right;border-bottom:2px solid #cbd5e0;">成本</th>
<th style="padding:10px 12px;text-align:right;border-bottom:2px solid #cbd5e0;">Turns</th>
<th style="padding:10px 12px;text-align:right;border-bottom:2px solid #cbd5e0;">耗时</th>
<th style="padding:10px 12px;text-align:right;border-bottom:2px solid #cbd5e0;">Token</th>
<th style="padding:10px 12px;text-align:right;border-bottom:2px solid #cbd5e0;">拒绝</th>
</tr>
</thead>
<tbody>
${TABLE_ROWS}
</tbody>
</table>
<!-- AI 异常分析 -->
<div style="margin-top:24px;">
${ANO_SECTION}
</div>
<!-- 建议 -->
${REC_SECTION}
<!-- 明细 -->
<details style="margin-top:20px;border-top:1px solid #e2e8f0;padding-top:16px;">
<summary style="cursor:pointer;font-size:13px;color:#718096;font-weight:600;">📄 查看完整存档(Markdown)</summary>
<pre style="margin-top:12px;background:#f7fafc;border-radius:6px;padding:14px;white-space:pre-wrap;word-wrap:break-word;font-size:12px;line-height:1.5;overflow-x:auto;">$(cat "$DIGEST_MD" | html_esc)</pre>
</details>
</div>
<!-- 页脚 -->
<div style="padding:14px 28px;background:#f7fafc;border-top:1px solid #e2e8f0;font-size:12px;color:#a0aec0;text-align:center;">
News Harvester Pro · 生成于 $(date '+%Y-%m-%d %H:%M:%S') · 概况由脚本计算,异常分析由 Claude 生成
</div>
</div>
</div>
</body>
</html>
EOF
)"
printf '%s' "$HTML_BODY" | "$SCRIPT_DIR/notify-mail.sh" "$MAIL_TO" "$SUBJECT" html \
2>> "$DIGEST_RAW" || echo "[digest] 邮件发送失败(已忽略)" >> "$DIGEST_MD"
fi
fi
# ============================================================
# 8. 清理旧日志
# ============================================================
if [ "$DIGEST_RETAIN_DAYS" -gt 0 ]; then
find "$LOG_DIR" -name '*.json' -mtime +"$DIGEST_RETAIN_DAYS" -delete 2>/dev/null || true
find "$LOG_DIR" -name '*.raw' -mtime +"$DIGEST_RETAIN_DAYS" -delete 2>/dev/null || true
find "$LOG_DIR" -name '.digest-*' -mtime +"$DIGEST_RETAIN_DAYS" -delete 2>/dev/null || true
fi
exit 0
bin/harvest.sh 0 → 100755
View file @ c2d5873b
#!/bin/bash
# 通用新闻抓取脚本 —— 由 systemd 模板单元 news-harvester@<source>.service 调用
# 用法: harvest.sh <source-name> [count]
# $1 栏目/源名称,如 nikkei-world、reuters(由 systemd %i 传入)
# $2 抓取条数,默认 20(可被 /etc/news-harvester/<source>.conf 里的 COUNT 覆盖)
#
# 使用 claude 无头模式 --output-format json 捕获完整运行元数据:
# 成本、turns、耗时、session_id、token 用量、权限拒绝、结果文本。
# 每次运行落盘:
# logs/<source>.<YYYYMMDD>.<HHMMSS>.json 完整 JSON 结果(daily-digest 读取)
# logs/<source>.<YYYYMMDD>.<HHMMSS>.raw claude stderr(排错用)
# logs/<source>.last.log 人类可读上次运行摘要(覆盖)
#
# 环境变量(由 service 的 EnvironmentFile 注入):
# MAIL_TO 收件人;留空则不发邮件
# MAIL_ON_SUCCESS 1=成功也发,0=仅失败发(默认 1)
# ALERT_COST_OVER 单次成本(美元)超过此值则在邮件标题标记 [ALERT],留空不判断
# WORK_DIR 项目工作目录(默认 /home/yichu/Documents/News-Harvester)
# LOG_DIR 日志目录(默认 <WORK_DIR>/logs)
# --- 设置 PATH 和 nvm(等价于 .zshrc 中的配置) ---
# systemd service (User=yichu) 非交互 shell 不加载 .zshrc,需显式设置
export NVM_DIR="/home/yichu/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
export PATH="/home/yichu/.local/bin:/home/yichu/.nvm/versions/node/v24.14.0/bin:$PATH"
SOURCE="${1:?用法: harvest.sh <source-name> [count]}"
COUNT="${2:-20}"
MAIL_TO="${MAIL_TO:-}"
MAIL_ON_SUCCESS="${MAIL_ON_SUCCESS:-1}"
ALERT_COST_OVER="${ALERT_COST_OVER:-}"
WORK_DIR="${WORK_DIR:-/home/yichu/Documents/News-Harvester}"
LOG_DIR="${LOG_DIR:-$WORK_DIR/logs}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
mkdir -p "$LOG_DIR"
DATE_TAG="$(date +%Y%m%d)"
TIME_TAG="$(date +%H%M%S)"
JSON_OUT="$LOG_DIR/${SOURCE}.${DATE_TAG}.${TIME_TAG}.json"
RAW_OUT="$LOG_DIR/${SOURCE}.${DATE_TAG}.${TIME_TAG}.raw"
HUMAN_LOG="$LOG_DIR/${SOURCE}.last.log"
cd "$WORK_DIR"
START_EPOCH=$(date +%s)
# --- 运行 claude 无头模式,输出 JSON ---
# 不用 set -e,因为要捕获 claude 的退出码
set +e
claude -p \
--permission-mode bypassPermissions \
--allowed-tools Bash Write \
--output-format json \
--max-budget-usd 15 \
"使用news-harvester获取${SOURCE}的最新${COUNT}条数据，并保存到obsidian中。
执行要求：
- 必须在前台同步完成所有抓取和归档，不要把浏览器自动化或任何步骤放到后台任务中运行。
- 不要使用后台 shell、nohup、& 或任何形式的异步执行。
- 所有文章抓取完毕后再依次生成摘要并写入 Obsidian，全部完成后再返回结果。
- 如果某篇文章抓取失败，跳过它继续下一篇，不要因此中断整个任务。
- 最终结果中简要说明：成功归档几篇、失败几篇、各失败原因。" \
> "$JSON_OUT" 2> "$RAW_OUT"
RC=$?
set -e
# 如果 claude 完全没输出(可能 PATH 未生效),写入诊断信息到 RAW_OUT
if [ ! -s "$JSON_OUT" ] && [ ! -s "$RAW_OUT" ]; then
{
echo "===== 诊断信息(claude 无任何输出) ====="
echo "时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "退出码: $RC"
echo "当前用户: $(whoami)"
echo "PATH: $PATH"
echo ""
echo "尝试检测 claude 是否可用:"
echo " claude --version:"
claude --version 2>&1 || echo " 失败(exit $?)"
echo ""
echo "which claude:"
which claude 2>&1 || echo " 未找到"
} > "$RAW_OUT"
fi
END_EPOCH=$(date +%s)
WALL_DURATION=$((END_EPOCH - START_EPOCH))
# --- 解析 JSON 元数据 ---
HAVE_JQ=0
command -v jq >/dev/null 2>&1 && [ -s "$JSON_OUT" ] && HAVE_JQ=1
if [ "$HAVE_JQ" = "1" ]; then
# 校验是否为合法 JSON(claude 失败时 stdout 可能为空或非 JSON)
if ! jq empty "$JSON_OUT" 2>/dev/null; then
HAVE_JQ=0
fi
fi
if [ "$HAVE_JQ" = "1" ]; then
IS_ERROR=$(jq -r '.is_error // "unknown"' "$JSON_OUT")
SUBTYPE=$(jq -r '.subtype // "unknown"' "$JSON_OUT")
COST=$(jq -r '.total_cost_usd // 0' "$JSON_OUT")
NUM_TURNS=$(jq -r '.num_turns // 0' "$JSON_OUT")
API_MS=$(jq -r '.duration_api_ms // 0' "$JSON_OUT")
SESSION_ID=$(jq -r '.session_id // "n/a"' "$JSON_OUT")
IN_TOK=$(jq -r '.usage.input_tokens // 0' "$JSON_OUT")
OUT_TOK=$(jq -r '.usage.output_tokens // 0' "$JSON_OUT")
CACHE_READ=$(jq -r '.usage.cache_read_input_tokens // 0' "$JSON_OUT")
CACHE_CREATE=$(jq -r '.usage.cache_creation_input_tokens // 0' "$JSON_OUT")
DENIALS=$(jq -r '.permission_denials | length' "$JSON_OUT")
TERMINAL=$(jq -r '.terminal_reason // "n/a"' "$JSON_OUT")
RESULT_TEXT=$(jq -r '.result // "(无输出)"' "$JSON_OUT")
else
IS_ERROR="unknown"; SUBTYPE="unknown"; COST=0; NUM_TURNS=0; API_MS=0
SESSION_ID="n/a"; IN_TOK=0; OUT_TOK=0; CACHE_READ=0; CACHE_CREATE=0
DENIALS=0; TERMINAL="n/a"
RESULT_TEXT="(无法解析 JSON 输出,请查看 .raw 文件;可能 claude 启动失败或未安装 jq)"
fi
# --- 判定状态 ---
# 成功条件:进程退出码 0 且 is_error 非 true
if [ "$RC" -eq 0 ] && [ "$IS_ERROR" != "true" ]; then
STATUS_EMOJI="✅"
STATUS_TEXT="成功"
STATUS_TAG="SUCCESS"
else
STATUS_EMOJI="❌"
STATUS_TEXT="失败"
STATUS_TAG="FAILED"
fi
# 成本告警判断
COST_ALERT=0
if [ -n "$ALERT_COST_OVER" ]; then
awk -v c="$COST" -v t="$ALERT_COST_OVER" 'BEGIN{exit !(c+0 > t+0)}' && COST_ALERT=1
fi
# 权限拒绝也算异常
if [ "$DENIALS" -gt 0 ] && [ "$STATUS_TAG" = "SUCCESS" ]; then
STATUS_EMOJI="⚠️"
STATUS_TEXT="成功(但有 ${DENIALS} 次权限拒绝)"
fi
# --- 格式化耗时 ---
MINS=$((WALL_DURATION / 60))
SECS=$((WALL_DURATION % 60))
API_SECS=$(awk -v ms="$API_MS" 'BEGIN{printf "%.1f", ms/1000}')
# --- 写人类可读日志 ---
{
echo "===== $(date '+%Y-%m-%d %H:%M:%S') 抓取 [$SOURCE] ====="
echo "状态: ${STATUS_EMOJI} ${STATUS_TEXT}"
echo "退出码: ${RC}"
echo "耗时: ${MINS}m${SECS}s (wall) / ${API_SECS}s (api)"
echo "成本: \$${COST}"
echo "Turns: ${NUM_TURNS}"
echo "Token: 输入 ${IN_TOK} / 输出 ${OUT_TOK} / cache读 ${CACHE_READ} / cache写 ${CACHE_CREATE}"
echo "Session: ${SESSION_ID}"
echo "拒绝: ${DENIALS} 次权限拒绝"
echo "Terminal: ${TERMINAL}"
echo ""
echo "----- result -----"
echo "$RESULT_TEXT"
echo ""
echo "----- JSON 结果文件 -----"
echo "$JSON_OUT"
} > "$HUMAN_LOG"
# --- 邮件通知 ---
SHOULD_NOTIFY=0
if [ -n "$MAIL_TO" ]; then
if [ "$STATUS_TAG" = "FAILED" ]; then
SHOULD_NOTIFY=1
elif [ "$MAIL_ON_SUCCESS" = "1" ]; then
SHOULD_NOTIFY=1
fi
fi
if [ "$SHOULD_NOTIFY" = "1" ]; then
ALERT_PREFIX=""
if [ "$STATUS_TAG" = "FAILED" ] || [ "$COST_ALERT" = "1" ]; then
ALERT_PREFIX="[ALERT] "
fi
SUBJECT="[News Harvester] ${ALERT_PREFIX}${STATUS_TAG}: ${SOURCE}"
# 取 result 末尾 60 行,避免邮件过长
RESULT_TAIL=$(printf '%s\n' "$RESULT_TEXT" | tail -n 60)
BODY="$(cat <<EOF
${STATUS_EMOJI} News Harvester 任务通知
栏目: ${SOURCE}
状态: ${STATUS_TEXT}
退出码: ${RC}
耗时: ${MINS}m${SECS}s (wall) / ${API_SECS}s (api)
成本: \$${COST}
Turns: ${NUM_TURNS}
Token: 输入 ${IN_TOK} / 输出 ${OUT_TOK}
cache 读 ${CACHE_READ} / cache 写 ${CACHE_CREATE}
Session: ${SESSION_ID}
权限拒绝: ${DENIALS}
Terminal: ${TERMINAL}
EOF
)"
if [ -n "$ALERT_COST_OVER" ]; then
if [ "$COST_ALERT" = "1" ]; then
BODY+=$'\n成本阈值: \$'"${ALERT_COST_OVER} (本次超标 ⚠️)"
else
BODY+=$'\n成本阈值: \$'"${ALERT_COST_OVER} (本次正常)"
fi
fi
BODY+=$'\n\n----- result 末尾 60 行 -----\n'"${RESULT_TAIL}"
# 通知失败不影响任务退出码
printf '%s' "$BODY" | "$SCRIPT_DIR/notify-mail.sh" "$MAIL_TO" "$SUBJECT" \
2>> "$RAW_OUT" || echo "[notify] 邮件发送失败(已忽略)" >> "$HUMAN_LOG"
fi
exit "$RC"
bin/notify-mail.sh 0 → 100755
View file @ c2d5873b
#!/bin/bash
# 通用邮件发送脚本 —— 被 harvest.sh / daily-digest.sh 调用
# 用法: echo "正文" | notify-mail.sh <to> <subject> [format]
# $1 收件人,支持多个:空格或逗号分隔
# 例: "a@x.com b@y.com"
# "a@x.com,b@y.com,c@z.com"
# $2 主题(建议 ASCII,避免邮件头编码问题)
# $3 格式:text(默认) | html
# 正文从 stdin 读取(UTF-8)
# 环境变量:
# MSMTP_CONFIG 指定 msmtp 配置文件路径(可选)
# 依赖:msmtp(配置文件 ~/.msmtprc,权限 600)
set -euo pipefail
RAW_TO="${1:?用法: notify-mail.sh <to> <subject> [text|html]}"
SUBJECT="${2:?用法: notify-mail.sh <to> <subject> [text|html]}"
FORMAT="${3:-text}"
BODY="$(cat)"
if ! command -v msmtp >/dev/null 2>&1; then
echo "[notify] msmtp 未安装,跳过邮件。安装:sudo pacman -S msmtp" >&2
exit 0
fi
# --- 解析收件人:统一逗号→空格,再拆成数组 ---
NORMALIZED="${RAW_TO//,/ }"
NORMALIZED="${NORMALIZED//,/ }"
read -r -a RECIPIENTS <<< "$NORMALIZED"
if [ "${#RECIPIENTS[@]}" -eq 0 ]; then
echo "[notify] 收件人为空,跳过邮件。" >&2
exit 0
fi
TO_HEADER="$(IFS=,; printf '%s' "${RECIPIENTS[*]}")"
# --- Content-Type ---
case "$FORMAT" in
html|HTML) CT="text/html; charset=UTF-8" ;;
*) CT="text/plain; charset=UTF-8" ;;
esac
# --- msmtp 配置文件 ---
MSMTP_ARGS=()
if [ -n "${MSMTP_CONFIG:-}" ] && [ -f "$MSMTP_CONFIG" ]; then
MSMTP_ARGS+=(--file="$MSMTP_CONFIG")
fi
printf 'To: %s\nSubject: %s\nContent-Type: %s\n\n%s' \
"$TO_HEADER" "$SUBJECT" "$CT" "$BODY" \
| msmtp "${MSMTP_ARGS[@]}" "${RECIPIENTS[@]}"
bin/status.sh 0 → 100755
View file @ c2d5873b
#!/bin/bash
# status.sh — 查看 News Harvester 所有 timer 运行情况(纯文本,无颜色)
#
# 用法:
# ./status.sh 查看所有 timer 概览(按运行时间排序)
# ./status.sh <source> 查看某栏目详情(最近一次运行)
# ./status.sh <source> --log 查看某栏目最近运行的完整日志
# ./status.sh <source> --json 查看最近一次的 claude JSON 输出
# ./status.sh <source> --journal 查看 systemd journal 日志
# ./status.sh --digest 查看今天的每日汇总
# ./status.sh --all-logs 列出所有栏目的最近日志摘要
# ./status.sh --help, -h 显示帮助
# 不用 set -e,这个脚本是只读查看工具,任何一步失败都不应中断
set -uo pipefail
WORK_DIR="${WORK_DIR:-/home/yichu/Documents/News-Harvester}"
LOG_DIR="${LOG_DIR:-$WORK_DIR/logs}"
usage() {
cat <<'EOF'
News Harvester 状态查看工具
用法:
./status.sh 查看所有 timer 概览(按运行时间排序)
./status.sh <source> 查看某栏目详情(最近一次运行)
./status.sh <source> --log 查看某栏目最近运行的完整日志
./status.sh <source> --json 查看最近一次的 claude JSON 输出
./status.sh <source> --journal 查看 systemd journal 日志
./status.sh --digest 查看今天的每日汇总
./status.sh --all-logs 列出所有栏目的最近日志摘要
./status.sh --help, -h 显示本帮助
EOF
}
hr() { echo "------------------------------------------------------------"; }
# ============================================================
# 所有栏目的最近日志摘要
# ============================================================
show_all_last_logs() {
shopt -s nullglob
local found=0
for f in "$LOG_DIR"/*.last.log; do
[ -f "$f" ] || continue
found=1
local src status_line cost_line session_line mtime_fmt
src=$(basename "$f" .last.log)
status_line=$(grep '^状态:' "$f" 2>/dev/null | head -1)
cost_line=$(grep '^成本:' "$f" 2>/dev/null | head -1)
session_line=$(grep '^Session:' "$f" 2>/dev/null | head -1)
# 取日志文件修改时间,格式 YYYY-MM-DD HH:MM
mtime_fmt=$(date -r "$f" '+%Y-%m-%d %H:%M' 2>/dev/null || stat -c '%y' "$f" 2>/dev/null | cut -d. -f1 || echo "?")
local status_text cost_text cost_num cost_fmt
status_text="${status_line#状态: }"
[ -z "$status_text" ] && status_text="${status_line#状态:}"
cost_text="${cost_line#成本: }"
[ -z "$cost_text" ] && cost_text="${cost_line#成本:}"
cost_num=$(echo "$cost_text" | grep -oE '[0-9]+\.?[0-9]*' | head -1)
if [ -n "$cost_num" ]; then
cost_fmt=$(awk -v c="$cost_num" 'BEGIN{printf "$%.2f", c}')
else
cost_fmt="$cost_text"
fi
local session_id
session_id="${session_line#Session: }"
[ -z "$session_id" ] && session_id="${session_line#Session:}"
local icon
if echo "$status_text" | grep -q "成功"; then icon="OK"
elif echo "$status_text" | grep -q "失败"; then icon="FAIL"
else icon="?"; fi
if [ -n "$session_id" ]; then
printf "%-6s %-22s %-16s %s | %s | sid:%s\n" "[$icon]" "$src" "$mtime_fmt" "$status_text" "$cost_fmt" "$session_id"
else
printf "%-6s %-22s %-16s %s | %s\n" "[$icon]" "$src" "$mtime_fmt" "$status_text" "$cost_fmt"
fi
done
shopt -u nullglob
if [ "$found" = "0" ]; then
echo " (暂无运行记录)"
fi
echo ""
echo "查看详情: ./status.sh <source>"
}
# ============================================================
# 概览模式:所有 timer,按运行时间排序
# ============================================================
show_overview() {
echo "News Harvester — Timer 概览"
echo "$(date '+%Y-%m-%d %H:%M:%S')"
echo ""
# 获取所有 timer 名称(只取名字列,避免 list-units 格式问题)
local timer_names
timer_names=$(systemctl list-units 'news-harvester@*.timer' 'news-harvester-digest.timer' \
--no-legend --all 2>/dev/null | awk '{print $1}' || true)
if [ -z "$timer_names" ]; then
echo "未发现任何 news-harvester timer。"
echo "提示:用 sudo ./add-source.sh <source> 添加栏目"
return
fi
printf "%-42s %-8s %-10s %-26s %-26s\n" "TIMER" "状态" "运行时间" "上次触发" "下次触发"
hr
# 收集到临时文件
local tmp
tmp=$(mktemp)
# 用 for 循环遍历 timer 名,避免 while-read 在管道里的坑
for name in $timer_names; do
[ -z "$name" ] && continue
# 状态:从 list-units 拿,或用 is-active
local sub
sub=$(systemctl is-active "$name" 2>/dev/null || echo "unknown")
# 获取 OnCalendar / NextElapse / LastTrigger
# OnCalendar 可能返回多行(drop-in 覆盖时),用 grep 取所有值
local on_cal next_el last_tr
on_cal=$(systemctl show "$name" -p OnCalendar 2>/dev/null | grep '^OnCalendar=' | cut -d= -f2- | tr '\n' ' ')
next_el=$(systemctl show "$name" -p NextElapseUSecRealtime 2>/dev/null | cut -d= -f2-)
last_tr=$(systemctl show "$name" -p LastTriggerUSec 2>/dev/null | cut -d= -f2-)
# 提取 HH:MM 排序键(从 OnCalendar 或 NextElapse 兜底)
# OnCalendar 格式: *-*-* 09:00:00
# NextElapse 格式: Wed 2026-08-27 09:00:00 CST
local short sort_key run_fmt
# 先从 OnCalendar 提取
short=$(echo "$on_cal" | grep -oE '[0-9]{2}:[0-9]{2}:[0-9]{2}' | head -1)
# 如果 OnCalendar 没有,从 NextElapse 提取(去掉星期和日期,只留时间)
if [ -z "$short" ] && [ -n "$next_el" ] && [ "$next_el" != "n/a" ]; then
short=$(echo "$next_el" | grep -oE '[0-9]{2}:[0-9]{2}:[0-9]{2}' | head -1)
fi
if [ -n "$short" ]; then
sort_key="${short%:*}" # HH:MM
run_fmt="${short%:*}" # HH:MM
else
sort_key="99:99"
run_fmt="-"
fi
# digest 排到最后,但运行时间仍用从 OnCalendar 读取的实际值
echo "$name" | grep -q "digest" && sort_key="zz:zz"
local next_fmt last_fmt
if [ -n "$next_el" ] && [ "$next_el" != "n/a" ] && [ "$next_el" != "0" ]; then
next_fmt="$next_el"
else
next_fmt="-"
fi
if [ -n "$last_tr" ] && [ "$last_tr" != "n/a" ] && [ "$last_tr" != "0" ]; then
last_fmt="$last_tr"
else
last_fmt="从未触发"
fi
printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
"$sort_key" "$name" "$sub" "$run_fmt" "$last_fmt" "$next_fmt" >> "$tmp"
done
# 排序后输出
if [ -s "$tmp" ]; then
sort -t$'\t' -k1,1 "$tmp" | while IFS=$'\t' read -r _ name sub run_fmt last_fmt next_fmt; do
[ -z "$name" ] && continue
printf "%-42s %-8s %-10s %-26s %-26s\n" "$name" "$sub" "$run_fmt" "$last_fmt" "$next_fmt"
done
else
echo "(无法获取 timer 信息)"
fi
rm -f "$tmp"
echo ""
echo "最近运行"
hr
printf "%-6s %-22s %-16s %-20s %-10s %s\n" "" "来源" "修改时间" "状态" "成本" "session"
show_all_last_logs
}
# ============================================================
# 栏目详情模式
# ============================================================
show_source_detail() {
local source="$1" mode="${2:-detail}"
local timer_name="news-harvester@${source}.timer"
local service_name="news-harvester@${source}.service"
hr
echo "栏目详情: $source"
hr
if systemctl cat "$timer_name" >/dev/null 2>&1; then
echo "Timer:"
local on_cal next_el last_tr
on_cal=$(systemctl show "$timer_name" -p OnCalendar 2>/dev/null | cut -d= -f2)
next_el=$(systemctl show "$timer_name" -p NextElapseUSecRealtime 2>/dev/null | cut -d= -f2-)
last_tr=$(systemctl show "$timer_name" -p LastTriggerUSec 2>/dev/null | cut -d= -f2-)
printf " 运行时间: %s\n" "$on_cal"
if [ -n "$next_el" ] && [ "$next_el" != "n/a" ] && [ "$next_el" != "0" ]; then
printf " 下次触发: %s\n" "$next_el"
else
printf " 下次触发: -\n"
fi
if [ -n "$last_tr" ] && [ "$last_tr" != "n/a" ] && [ "$last_tr" != "0" ]; then
printf " 上次触发: %s\n" "$last_tr"
else
printf " 上次触发: 从未触发\n"
fi
else
echo " Timer 未安装: $timer_name"
fi
echo ""
if [ -f "/etc/news-harvester/${source}.conf" ]; then
echo "栏目配置 (/etc/news-harvester/${source}.conf):"
grep -v '^#\|^$' "/etc/news-harvester/${source}.conf" | sed 's/^/ /'
echo ""
elif [ -f "/etc/news-harvester/default.conf" ]; then
echo "栏目配置: (继承 default.conf)"
echo ""
fi
local last_log="$LOG_DIR/${source}.last.log"
if [ ! -f "$last_log" ]; then
echo " 暂无运行记录($last_log 不存在)"
return
fi
case "$mode" in
--log|-l)
echo "最近运行日志 ($last_log):"
hr
cat "$last_log"
;;
--json|-j)
local latest_json
latest_json=$(ls -t "$LOG_DIR/${source}."*.json 2>/dev/null | head -1)
if [ -z "$latest_json" ]; then
echo " 暂无 JSON 输出文件"
return
fi
echo "最近 JSON 输出 ($latest_json):"
hr
if command -v jq >/dev/null 2>&1; then
jq . "$latest_json"
else
cat "$latest_json"
fi
;;
--journal)
echo "systemd journal (最近 50 行):"
hr
journalctl -u "$service_name" -n 50 --no-pager 2>/dev/null || \
echo " 无法读取 journal(可能需要 sudo)"
;;
detail|*)
echo "最近运行:"
while IFS= read -r line; do
case "$line" in
=====*) echo " $line" ;;
状态:*) printf " %-12s %s\n" "状态:" "${line#状态: }" ;;
退出码:*) printf " %-12s %s\n" "退出码:" "${line#退出码: }" ;;
耗时:*) printf " %-12s %s\n" "耗时:" "${line#耗时: }" ;;
成本:*) printf " %-12s %s\n" "成本:" "${line#成本: }" ;;
Turns:*) printf " %-12s %s\n" "Turns:" "${line#Turns: }" ;;
Token:*) printf " %-12s %s\n" "Token:" "${line#Token: }" ;;
Session:*) printf " %-12s %s\n" "Session:" "${line#Session: }" ;;
拒绝:*) printf " %-12s %s\n" "权限拒绝:" "${line#拒绝: }" ;;
Terminal:*)printf " %-12s %s\n" "Terminal:" "${line#Terminal: }" ;;
-----*) echo " --- result ---" ;;
*) [ -n "$line" ] && echo " $line" ;;
esac
done < "$last_log"
;;
esac
}
# ============================================================
# 每日汇总模式
# ============================================================
show_digest() {
local today digest_md
today=$(date +%Y%m%d)
digest_md="$LOG_DIR/digest.${today}.md"
hr
echo "每日汇总 — $(date '+%Y-%m-%d')"
hr
if [ -f "$digest_md" ]; then
cat "$digest_md"
else
echo "今天的汇总尚未生成($digest_md 不存在)"
echo "提示:手动运行 sudo systemctl start news-harvester-digest.service"
fi
}
# ============================================================
# 主逻辑
# ============================================================
case "${1:-}" in
--help|-h) usage; exit 0 ;;
--digest|-d) show_digest; exit 0 ;;
--all-logs|-a) show_all_last_logs; exit 0 ;;
"") show_overview; exit 0 ;;
-*)
echo "未知选项: $1" >&2
usage >&2
exit 1
;;
*)
show_source_detail "$1" "${2:-detail}"
;;
esac
config/default.conf.example 0 → 100644
View file @ c2d5873b
# /etc/news-harvester/default.conf
# 所有栏目的全局默认配置。单栏目可在 /etc/news-harvester/<source>.conf 中覆盖。
# 注意:systemd EnvironmentFile 格式,每行 VAR=value,不要加 export,不要加引号包裹。
# === 抓取 ===
# 默认抓取条数(未在此设置时脚本兜底 20)
COUNT=20
# === 邮件通知 ===
# 默认收件人;留空则不发邮件。强烈建议在此设一个兜底地址。
# 支持多个收件人:空格或逗号分隔(中英文逗号均可)。
# 例: MAIL_TO=alice@example.com bob@example.com
# MAIL_TO=alice@example.com,bob@example.com
MAIL_TO=you@example.com
# 成功是否也发邮件:1=每次都发,0=仅失败发
MAIL_ON_SUCCESS=1
# === 成本告警 ===
# 单次抓取成本(美元)超过此值则在邮件标题标记 [ALERT],留空不判断
ALERT_COST_OVER=1
# === 每日汇总(digest 相关) ===
# 无论是否有告警都发汇总邮件:1=每次发,0=仅告警时发
DIGEST_MAIL_ALWAYS=1
# 旧日志保留天数,超过则自动清理
DIGEST_RETAIN_DAYS=30
# 汇总分析的 claude 预算上限(美元)
DIGEST_BUDGET_USD=2
# === 路径(通常不用改) ===
# WORK_DIR=/home/yichu/Documents/News-Harvester
# LOG_DIR=/home/yichu/Documents/News-Harvester/logs
config/msmtprc.example 0 → 100644
View file @ c2d5873b
# ~/.msmtprc —— msmtp 配置文件(放在运行用户 yichu 的家目录)
# 权限必须为 600,否则 msmtp 拒绝使用:chmod 600 ~/.msmtprc
#
# 下面以 Gmail 应用专用密码为例;其他 SMTP 服务替换 host/port/auth_user/password 即可。
# Gmail 需开启两步验证后生成「应用专用密码」,不能直接用登录密码。
defaults
auth on
tls on
tls_starttls on
logfile ~/.msmtp.log
account default
host smtp.gmail.com
port 587
from you@gmail.com
user you@gmail.com
password 你的应用专用密码
# 默认使用的 account
account default : default
config/nikkei-world.conf.example 0 → 100644
View file @ c2d5873b
# /etc/news-harvester/nikkei-world.conf
# 单栏目覆盖配置示例。仅写需要覆盖的字段,其余继承 default.conf。
# 此例:nikkei-world 抓 30 条,成功不单独发邮件(只在 digest 里看),成本超 $2 告警。
COUNT=30
MAIL_ON_SUCCESS=0
ALERT_COST_OVER=2
config/web.conf.example 0 → 100644
View file @ c2d5873b
# /etc/news-harvester/web.conf — Web 面板认证配置
# 由 news-harvester-web.service 通过 EnvironmentFile 加载
# 文件权限应为 600: sudo chmod 600 /etc/news-harvester/web.conf
# Basic Auth 用户名
WEB_USER=admin
# Basic Auth 密码(请修改为强密码)
WEB_PASS=change-me-to-a-strong-password
# 可选:Flask session 密钥(留空则使用默认值)
# WEB_SECRET=your-random-secret-key
deploy.sh 0 → 100755
View file @ c2d5873b
#!/bin/bash
# deploy.sh — 一键同步项目到服务器
#
# 用法:
# ./deploy.sh <server-ip> # 默认用户 yichu
# ./deploy.sh yichu@192.168.1.100 # 显式指定用户
# ./deploy.sh <host> --no-systemd # 只更新 bin 脚本,不动 systemd unit
# ./deploy.sh <host> --no-restart-web # 不重启 web 面板
# ./deploy.sh <host> --test # 部署后自动跑一次 nikkei-world 测试
#
# 做的事:
# 1. rsync 整个项目目录到 ~/news-harvester-pro/(--delete 保持镜像)
# 2. 复制 bin/*.sh 到 ~/Documents/News-Harvester/bin/ 并赋可执行权限
# 3. 同步 systemd 模板 service + digest unit 到 /etc/systemd/system/(需要 sudo)
# ⚠️ 栏目 timer(news-harvester@<source>.timer)不在同步范围,由服务器端维护
# 4. daemon-reload
#
# 不动的东西:
# - /etc/news-harvester/*.conf(用户配置)
# - 各栏目 timer(服务器端可能改过时间)
# - ~/.msmtprc(SMTP 配置)
# - logs 目录
set -euo pipefail
SYNC_SYSTEMD=1
RESTART_WEB=1
RUN_TEST=0
HOST=""
# --- 参数解析 ---
while [ $# -gt 0 ]; do
case "$1" in
--no-systemd) SYNC_SYSTEMD=0; shift;;
--no-restart-web) RESTART_WEB=0; shift;;
--test) RUN_TEST=1; shift;;
--help|-h)
sed -n '3,24p' "$0"; exit 0;;
-*) echo "未知选项: $1" >&2; exit 1;;
*)
if [ -z "$HOST" ]; then HOST="$1"
else echo "多余的参数: $1" >&2; exit 1; fi
shift;;
esac
done
[ -n "$HOST" ] || { echo "用法: $0 <[user@]host> [--no-systemd] [--no-restart-web] [--test]" >&2; exit 1; }
# 默认加用户名(如果没写 user@)
case "$HOST" in
*@*) ;;
*) HOST="yichu@${HOST}" ;;
esac
err() { echo "❌ $*" >&2; exit 1; }
ok() { echo "✅ $*"; }
info() { echo "ℹ️ $*"; }
warn() { echo "⚠️ $*"; }
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_NAME="news-harvester-pro"
# --- 检查 rsync ---
command -v rsync >/dev/null 2>&1 || err "本地需要 rsync。macOS 自带,如缺失:brew install rsync"
echo "=== 部署到 $HOST ==="
echo " 项目: $SCRIPT_DIR"
echo " 同步 bin: 是"
[ "$SYNC_SYSTEMD" = "1" ] && echo " 同步 systemd: 是(需要远程 sudo)" || echo " 同步 systemd: 否(--no-systemd)"
[ "$RESTART_WEB" = "1" ] && echo " 重启 web: 是" || echo " 重启 web: 否(--no-restart-web)"
[ "$RUN_TEST" = "1" ] && echo " 部署后测试: 是(--test)"
echo ""
# --- 1. rsync 项目目录 ---
info "1/4 同步项目目录 → ~/news-harvester-pro/"
rsync -avz --delete \
--exclude='.git' \
--exclude='*.log' \
--exclude='logs/' \
--exclude='__pycache__/' \
--exclude='*.pyc' \
"$SCRIPT_DIR/" "$HOST:~/${PROJECT_NAME}/"
ok "项目目录已同步"
# --- 2. 复制 bin + web 到工作目录 ---
info "2/5 复制 bin/*.sh + web/ → ~/Documents/News-Harvester/"
ssh "$HOST" 'mkdir -p ~/Documents/News-Harvester/bin ~/Documents/News-Harvester/web/templates && cp ~/news-harvester-pro/bin/*.sh ~/Documents/News-Harvester/bin/ && chmod +x ~/Documents/News-Harvester/bin/*.sh && cp ~/news-harvester-pro/web/app.py ~/Documents/News-Harvester/web/ && cp ~/news-harvester-pro/web/templates/*.html ~/Documents/News-Harvester/web/templates/'
ok "bin 脚本 + web 应用已就位"
# --- 3. 同步 systemd unit ---
if [ "$SYNC_SYSTEMD" = "1" ]; then
info "3/5 同步 systemd 模板 + digest + web unit → /etc/systemd/system/"
# 用 -t 分配伪终端,让远程 sudo 能交互式提示输入密码
# 只同步模板 service / digest / web,栏目 timer 由服务器端维护
ssh -t "$HOST" 'sudo cp ~/news-harvester-pro/systemd/news-harvester@.service /etc/systemd/system/ && sudo cp ~/news-harvester-pro/systemd/news-harvester-digest.service /etc/systemd/system/ && sudo cp ~/news-harvester-pro/systemd/news-harvester-digest.timer /etc/systemd/system/ && sudo cp ~/news-harvester-pro/systemd/news-harvester-web.service /etc/systemd/system/ && sudo systemctl daemon-reload'
ok "systemd unit 已同步并 reload"
else
info "3/5 跳过 systemd(--no-systemd)"
fi
# --- 4. 提示 Web 面板首次配置 + 重启 ---
info "4/5 Web 面板检查"
ssh "$HOST" 'test -f /etc/news-harvester/web.conf && echo "EXISTS" || echo "MISSING"' 2>/dev/null | grep -q EXISTS \
&& ok "web.conf 已存在" \
|| warn "web.conf 不存在,首次部署需创建:\n ssh $HOST 'sudo cp ~/news-harvester-pro/config/web.conf.example /etc/news-harvester/web.conf && sudo chmod 600 /etc/news-harvester/web.conf && sudo nano /etc/news-harvester/web.conf'\n 然后:ssh $HOST 'sudo systemctl enable --now news-harvester-web.service'"
# 重启 web 面板使新代码生效(首次未安装时忽略失败)
if [ "$RESTART_WEB" = "1" ]; then
# 用 ssh -t 让 sudo 交互式输密码(同 step 3),不接管道避免死锁;靠退出码判断
if ssh -t "$HOST" 'sudo systemctl restart news-harvester-web.service' 2>/dev/null; then
ok "news-harvester-web.service 已重启(新代码生效)"
else
info "web 服务重启失败(可能未安装),若已安装可手动:ssh $HOST 'sudo systemctl restart news-harvester-web.service'"
fi
else
info "跳过 web 重启(--no-restart-web)"
fi
# --- 5. 完成 ---
info "5/5 完成"
echo ""
echo "========================================"
ok "部署完成!"
echo "========================================"
echo ""
echo "验证:"
echo " 查看定时器: ssh $HOST 'systemctl list-timers \"news-harvester@*\" news-harvester-digest.timer'"
echo " 手动测试: ssh $HOST 'sudo systemctl start news-harvester@nikkei-world.service'"
echo " 看日志: ssh $HOST 'journalctl -u news-harvester@nikkei-world.service -f'"
echo " 测试汇总: ssh $HOST 'sudo systemctl start news-harvester-digest.service'"
echo " Web 面板: ssh -L 8080:127.0.0.1:8080 $HOST 然后浏览器打开 http://localhost:8080"
echo ""
# --- 可选:自动测试 ---
if [ "$RUN_TEST" = "1" ]; then
warn "正在触发测试运行 news-harvester@nikkei-world.service ..."
ssh -t "$HOST" 'sudo systemctl start news-harvester@nikkei-world.service' || warn "触发失败,请手动检查"
info "日志(实时,按 Ctrl+C 退出):"
ssh -t "$HOST" 'journalctl -u news-harvester@nikkei-world.service -f -n 20' || true
fi
migrate.sh 0 → 100755
View file @ c2d5873b
#!/bin/bash
# migrate.sh — 从旧版 news-harvester-systemd 迁移到 news-harvester-pro
#
# 功能:
# 1. 检测旧版安装(旧 harvest.sh / 旧 service / 旧 notify-mail.sh)
# 2. 备份旧文件到带时间戳的目录
# 3. 安装新版 bin / systemd / config
# 4. 保留 /etc/news-harvester/*.conf(不覆盖用户配置)
# 5. 停用旧 timer,daemon-reload,启用新 timer(保持原栏目)
# 6. 可选:安装 digest service + timer
#
# 用法:
# sudo ./migrate.sh # 交互式(会提示确认)
# sudo ./migrate.sh --yes # 跳过所有确认
# sudo ./migrate.sh --uninstall # 卸载新版,恢复旧版备份
#
# 环境变量:
# WORK_DIR 项目工作目录(默认 /home/yichu/Documents/News-Harvester)
# USER_NAME 运行用户(默认 yichu)
# INSTALL_DIGEST 是否安装 digest(1/0,默认 1)
set -euo pipefail
# --- 配置 ---
WORK_DIR="${WORK_DIR:-/home/yichu/Documents/News-Harvester}"
USER_NAME="${USER_NAME:-yichu}"
INSTALL_DIGEST="${INSTALL_DIGEST:-1}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
BACKUP_DIR="${WORK_DIR}/.backup-$(date +%Y%m%d%H%M%S)"
ASSUME_NO=0
UNINSTALL=0
# --- 解析参数 ---
for arg in "$@"; do
case "$arg" in
--yes|-y) ASSUME_NO=1 ;;
--uninstall) UNINSTALL=1 ;;
--help|-h)
sed -n '3,20p' "$0"; exit 0 ;;
*) echo "未知参数: $arg"; exit 1 ;;
esac
done
# --- 辅助函数 ---
confirm() {
[ "$ASSUME_NO" = "1" ] && return 0
local prompt="$1" reply
read -rp "$prompt [y/N] " reply
case "$reply" in y|Y|yes|YES) return 0 ;; *) return 1 ;; esac
}
err() { echo "❌ $*" >&2; exit 1; }
ok() { echo "✅ $*"; }
warn() { echo "⚠️ $*"; }
info() { echo "ℹ️ $*"; }
need_root() {
[ "$(id -u)" -eq 0 ] || err "请用 sudo 运行:sudo $0 $*"
}
# --- 卸载模式 ---
if [ "$UNINSTALL" = "1" ]; then
need_root
echo "=== 卸载 news-harvester-pro ==="
# 找到最近的备份
LATEST_BACKUP=$(ls -d "${WORK_DIR}"/.backup-* 2>/dev/null | sort | tail -1)
if [ -z "$LATEST_BACKUP" ]; then
err "未找到备份目录(${WORK_DIR}/.backup-*),无法恢复。"
fi
warn "将恢复备份: $LATEST_BACKUP"
confirm "确认卸载新版并恢复?" || exit 0
# 停用所有 news-harvester timer
for t in $(systemctl list-unit-files 'news-harvester@*.timer' 'news-harvester-digest.timer' --no-legend 2>/dev/null | awk '{print $1}'); do
systemctl disable --now "$t" 2>/dev/null || true
done
# 删除新版 unit 文件
rm -f /etc/systemd/system/news-harvester@.service
rm -f /etc/systemd/system/news-harvester@*.timer
rm -f /etc/systemd/system/news-harvester-digest.service
rm -f /etc/systemd/system/news-harvester-digest.timer
rm -f /etc/systemd/system/news-harvester@*.service.d/*.conf 2>/dev/null
rmdir /etc/systemd/system/news-harvester@*.service.d 2>/dev/null || true
# 恢复备份的 unit 文件
if [ -d "$LATEST_BACKUP/systemd" ]; then
cp -v "$LATEST_BACKUP/systemd/"* /etc/systemd/system/ 2>/dev/null || true
fi
# 恢复备份的 bin
if [ -d "$LATEST_BACKUP/bin" ]; then
mkdir -p "$WORK_DIR/bin"
cp -v "$LATEST_BACKUP/bin/"* "$WORK_DIR/bin/" 2>/dev/null || true
fi
systemctl daemon-reload
# 重新启用恢复的 timer
for t in /etc/systemd/system/news-harvester@*.timer; do
[ -f "$t" ] && systemctl enable --now "$(basename "$t")" 2>/dev/null || true
done
ok "卸载完成,已恢复 $LATEST_BACKUP"
exit 0
fi
# === 正常安装/迁移 ===
need_root "$@"
# 前置检查:jq / msmtp / claude
echo "=== 前置依赖检查 ==="
command -v jq >/dev/null && ok "jq 已安装" || warn "jq 未安装: sudo pacman -S jq"
command -v msmtp >/dev/null && ok "msmtp 已安装" || warn "msmtp 未安装: sudo pacman -S msmtp"
command -v claude >/dev/null && ok "claude 已安装" || warn "claude 未在 root PATH 中(可能正常,脚本以 $USER_NAME 身份运行时会从用户 profile 加载)"
echo ""
echo "=== 配置确认 ==="
echo " WORK_DIR = $WORK_DIR"
echo " USER_NAME = $USER_NAME"
echo " INSTALL_DIGEST = $INSTALL_DIGEST"
echo " 备份目录 = $BACKUP_DIR"
confirm "确认开始迁移?" || exit 0
# --- 1. 备份旧文件 ---
echo ""
echo "=== 步骤 1/5: 备份旧文件 ==="
mkdir -p "$BACKUP_DIR/bin" "$BACKUP_DIR/systemd" "$BACKUP_DIR/conf"
# 备份旧的 bin 脚本(可能是项目根目录下的 harvest.sh / notify-mail.sh,也可能是 bin/)
for f in harvest.sh notify-mail.sh daily-digest.sh; do
for loc in "$WORK_DIR/$f" "$WORK_DIR/bin/$f"; do
if [ -f "$loc" ]; then
cp -v "$loc" "$BACKUP_DIR/bin/" 2>/dev/null || true
fi
done
done
# 备份旧的 systemd unit
for f in /etc/systemd/system/news-harvester@.service \
/etc/systemd/system/news-harvester@*.timer \
/etc/systemd/system/news-harvester-digest.service \
/etc/systemd/system/news-harvester-digest.timer; do
for loc in $f; do
[ -f "$loc" ] && cp -v "$loc" "$BACKUP_DIR/systemd/" 2>/dev/null || true
done
done
# 备份 drop-in
cp -r /etc/systemd/system/news-harvester@*.service.d "$BACKUP_DIR/systemd/" 2>/dev/null || true
# 备份 conf(这些不覆盖,但留个快照)
if [ -d /etc/news-harvester ]; then
cp -rv /etc/news-harvester "$BACKUP_DIR/conf/" 2>/dev/null || true
fi
ok "备份完成 → $BACKUP_DIR"
# --- 2. 停用旧 timer ---
echo ""
echo "=== 步骤 2/5: 停用旧 timer ==="
for t in $(systemctl list-unit-files 'news-harvester@*.timer' --no-legend 2>/dev/null | awk '{print $1}'); do
info "停用 $t"
systemctl disable --now "$t" 2>/dev/null || true
done
if [ "$INSTALL_DIGEST" = "1" ]; then
systemctl disable --now news-harvester-digest.timer 2>/dev/null || true
fi
# --- 3. 安装新版 bin ---
echo ""
echo "=== 步骤 3/5: 安装新版脚本 ==="
mkdir -p "$WORK_DIR/bin"
install -m755 -o "$USER_NAME" -g "$USER_NAME" "$SCRIPT_DIR/bin/harvest.sh" "$WORK_DIR/bin/harvest.sh"
install -m755 -o "$USER_NAME" -g "$USER_NAME" "$SCRIPT_DIR/bin/notify-mail.sh" "$WORK_DIR/bin/notify-mail.sh"
if [ "$INSTALL_DIGEST" = "1" ]; then
install -m755 -o "$USER_NAME" -g "$USER_NAME" "$SCRIPT_DIR/bin/daily-digest.sh" "$WORK_DIR/bin/daily-digest.sh"
fi
ok "脚本已安装到 $WORK_DIR/bin/"
# --- 4. 安装 systemd unit ---
echo ""
echo "=== 步骤 4/5: 安装 systemd 单元 ==="
# 删除旧的(避免残留)
rm -f /etc/systemd/system/news-harvester@.service
rm -f /etc/systemd/system/news-harvester-digest.service /etc/systemd/system/news-harvester-digest.timer
install -m644 "$SCRIPT_DIR/systemd/news-harvester@.service" /etc/systemd/system/
# 安装栏目 timer:优先用备份里用户改过的版本,否则用项目自带示例
# 收集所有旧 timer 的栏目名
INSTALLED_TIMERS=()
for backup_timer in "$BACKUP_DIR/systemd"/news-harvester@*.timer; do
[ -f "$backup_timer" ] || continue
timer_name="$(basename "$backup_timer")"
# 用用户改过的版本(保留其 OnCalendar)
install -m644 "$backup_timer" "/etc/systemd/system/$timer_name"
info "恢复栏目 timer(保留用户配置): $timer_name"
INSTALLED_TIMERS+=("$timer_name")
done
# 如果备份里没有(全新安装),装示例 timer
if [ "${#INSTALLED_TIMERS[@]}" -eq 0 ]; then
for example_timer in "$SCRIPT_DIR/systemd"/news-harvester@*.timer; do
[ -f "$example_timer" ] || continue
timer_name="$(basename "$example_timer")"
install -m644 "$example_timer" "/etc/systemd/system/$timer_name"
info "安装示例 timer: $timer_name"
INSTALLED_TIMERS+=("$timer_name")
done
fi
if [ "$INSTALL_DIGEST" = "1" ]; then
install -m644 "$SCRIPT_DIR/systemd/news-harvester-digest.service" /etc/systemd/system/
install -m644 "$SCRIPT_DIR/systemd/news-harvester-digest.timer" /etc/systemd/system/
ok "digest service + timer 已安装"
fi
systemctl daemon-reload
ok "systemd 已重新加载"
# --- 5. 安装配置(不覆盖已有) ---
echo ""
echo "=== 步骤 5/5: 配置文件 ==="
install -d -m755 /etc/news-harvester
if [ ! -f /etc/news-harvester/default.conf ]; then
install -m644 "$SCRIPT_DIR/config/default.conf.example" /etc/news-harvester/default.conf
warn "已创建 /etc/news-harvester/default.conf(示例),请编辑填入真实 MAIL_TO"
else
ok "/etc/news-harvester/default.conf 已存在,保留不覆盖"
# 检查是否缺少新增的 digest 相关变量
if ! grep -q 'DIGEST_MAIL_ALWAYS' /etc/news-harvester/default.conf 2>/dev/null; then
warn "default.conf 缺少 digest 相关变量(DIGEST_MAIL_ALWAYS/DIGEST_RETAIN_DAYS/DIGEST_BUDGET_USD)"
info "可参考 config/default.conf.example 补充,否则用脚本默认值"
fi
fi
# --- 6. 启用 timer ---
echo ""
echo "=== 启用 timer ==="
for t in "${INSTALLED_TIMERS[@]}"; do
systemctl enable --now "$t" 2>/dev/null && ok "已启用 $t" || warn "启用 $t 失败"
done
if [ "$INSTALL_DIGEST" = "1" ]; then
systemctl enable --now news-harvester-digest.timer 2>/dev/null && ok "已启用 news-harvester-digest.timer" || warn "启用 digest timer 失败"
fi
# --- 完成 ---
echo ""
echo "========================================"
ok "迁移完成!"
echo "========================================"
echo ""
echo "后续操作:"
echo " 1. 编辑配置: sudo vim /etc/news-harvester/default.conf"
echo " 2. 配置 SMTP: install -m600 config/msmtprc.example ~${USER_NAME}/.msmtprc && vim ~${USER_NAME}/.msmtprc"
echo " 3. 查看定时器: systemctl list-timers 'news-harvester@*' news-harvester-digest.timer"
echo " 4. 手动测试: sudo systemctl start news-harvester@nikkei-world.service"
echo " 5. 查看日志: journalctl -u 'news-harvester@*' --since today -f"
echo " 6. 测试汇总: sudo systemctl start news-harvester-digest.service"
echo ""
echo "备份在: $BACKUP_DIR (如需回滚: sudo $0 --uninstall)"
echo ""
warn "重要:首次运行前请确认 default.conf 里的 MAIL_TO 已改为真实邮箱"
modify-source.sh 0 → 100755
View file @ c2d5873b
#!/bin/bash
# modify-source.sh — 修改已有栏目的定时运行时间(及可选配置)
#
# 用法:
# sudo ./modify-source.sh <source-name> [选项]
#
# 选项:
# --time HH:MM 新的运行时间,如 10:30(必填)
# --count N 更新抓取条数(写入/覆盖 <source>.conf)
# --mail-to ADDR 更新收件人
# --mail-on-success 0|1 更新成功是否发邮件
# --yes, -y 跳过确认
# --help, -h 显示帮助
#
# 示例:
# sudo ./modify-source.sh bloomberg --time 10:30
# sudo ./modify-source.sh nikkei-asia --time 14:00 --count 20 --mail-on-success 0
#
# 环境变量(可覆盖默认路径):
# SYSTEMD_DIR systemd unit 目录(默认 /etc/systemd/system)
# CONF_DIR 栏目配置目录(默认 /etc/news-harvester)
set -euo pipefail
SYSTEMD_DIR="${SYSTEMD_DIR:-/etc/systemd/system}"
CONF_DIR="${CONF_DIR:-/etc/news-harvester}"
SOURCE=""
TIME=""
COUNT=""
MAIL_TO=""
MAIL_ON_SUCCESS=""
ASSUME_NO=0
usage() {
sed -n '3,26p' "$0"
}
# --- 参数解析 ---
while [ $# -gt 0 ]; do
case "$1" in
--time) TIME="${2:?--time 需要值}"; shift 2;;
--count) COUNT="${2:?--count 需要值}"; shift 2;;
--mail-to) MAIL_TO="${2:?--mail-to 需要值}"; shift 2;;
--mail-on-success) MAIL_ON_SUCCESS="${2:?--mail-on-success 需要值}"; shift 2;;
--yes|-y) ASSUME_NO=1; shift;;
--help|-h) usage; exit 0;;
--) shift; break;;
-*) echo "未知选项: $1" >&2; usage >&2; exit 1;;
*)
if [ -z "$SOURCE" ]; then SOURCE="$1"
else echo "多余的参数: $1" >&2; exit 1; fi
shift;;
esac
done
# --- 辅助函数 ---
err() { echo "❌ $*" >&2; exit 1; }
ok() { echo "✅ $*"; }
warn() { echo "⚠️ $*"; }
info() { echo "ℹ️ $*"; }
confirm() {
[ "$ASSUME_NO" = "1" ] && return 0
local reply
read -rp "$1 [y/N] " reply
case "$reply" in y|Y|yes|YES) return 0 ;; *) return 1 ;; esac
}
# --- 校验 ---
[ -n "$SOURCE" ] || err "请指定栏目名:sudo $0 <source-name> --time HH:MM [选项]"
[[ "$SOURCE" =~ ^[a-zA-Z0-9_-]+$ ]] || err "栏目名只能含字母、数字、下划线、连字符: $SOURCE"
[ "$SOURCE" = "digest" ] && err "digest 是汇总服务,请用 sudo systemctl edit --full news-harvester-digest.timer 修改"
[ -n "$TIME" ] || err "必须指定 --time HH:MM"
[ "$(id -u)" -eq 0 ] || err "请用 sudo 运行"
validate_time() {
[[ "$1" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || err "时间格式应为 HH:MM(24小时制),如 10:30,得到: $1"
}
validate_count() {
[[ "$1" =~ ^[0-9]+$ ]] && [ "$1" -gt 0 ] || err "count 应为正整数: $1"
}
validate_mos() {
case "$1" in 0|1) ;; *) err "--mail-on-success 只能是 0 或 1: $1";; esac
}
validate_time "$TIME"
[ -n "$COUNT" ] && validate_count "$COUNT"
[ -n "$MAIL_ON_SUCCESS" ] && validate_mos "$MAIL_ON_SUCCESS"
# --- 检查 timer 是否存在 ---
TIMER_PATH="$SYSTEMD_DIR/news-harvester@${SOURCE}.timer"
CONF_PATH="$CONF_DIR/${SOURCE}.conf"
TIMER_NAME="news-harvester@${SOURCE}.timer"
DROPIN_DIR="$SYSTEMD_DIR/${TIMER_NAME}.d"
[ -f "$TIMER_PATH" ] || err "timer 不存在: $TIMER_PATH\n 可用栏目:list-timers 'news-harvester@*.timer'"
# --- 读取当前生效时间(优先 systemctl show,回退读文件) ---
CURRENT_TIME=""
# 方法1:从 systemctl show 读实际生效值(能反映 drop-in 覆盖)
local_show=$(systemctl show "$TIMER_NAME" -p OnCalendar 2>/dev/null || true)
if [ -n "$local_show" ] && [ "$local_show" != "OnCalendar=" ]; then
CURRENT_TIME=$(echo "$local_show" | grep -oE '[0-9]{2}:[0-9]{2}:[0-9]{2}' | head -1 || true)
fi
# 方法2:回退到直接读 timer 文件
if [ -z "$CURRENT_TIME" ]; then
CURRENT_TIME=$(grep '^OnCalendar=' "$TIMER_PATH" 2>/dev/null | head -1 | grep -oE '[0-9]{2}:[0-9]{2}:[0-9]{2}' | head -1 || true)
fi
[ -n "$CURRENT_TIME" ] && CURRENT_TIME="${CURRENT_TIME%:*}" # 取 HH:MM
[ -z "$CURRENT_TIME" ] && CURRENT_TIME="(未知)"
# --- 检查是否有 drop-in 覆盖(若有,改主文件不生效,需要先清除) ---
HAS_DROPIN=0
if [ -d "$DROPIN_DIR" ] && [ -n "$(ls -A "$DROPIN_DIR" 2>/dev/null)" ]; then
HAS_DROPIN=1
fi
# --- 读取当前 conf(若有) ---
read_conf_val() {
local key="$1" file="$CONF_PATH"
[ -f "$file" ] || { echo ""; return 0; }
grep -E "^${key}=" "$file" 2>/dev/null | head -1 | cut -d= -f2- || true
}
CUR_COUNT=$(read_conf_val COUNT)
CUR_MAIL_TO=$(read_conf_val MAIL_TO)
CUR_MOS=$(read_conf_val MAIL_ON_SUCCESS)
# --- 展示变更计划,确认 ---
echo "=== 修改栏目: $SOURCE ==="
echo " 定时时间: $CURRENT_TIME → $TIME"
[ -n "$COUNT" ] && { echo " 抓取条数: ${CUR_COUNT:-(继承default)} → $COUNT"; }
[ -n "$MAIL_TO" ] && { echo " 收件人: ${CUR_MAIL_TO:-(继承default)} → $MAIL_TO"; }
[ -n "$MAIL_ON_SUCCESS" ] && { echo " 成功发信: ${CUR_MOS:-(继承default)} → $MAIL_ON_SUCCESS"; }
echo " timer 文件: $TIMER_PATH"
if [ "$HAS_DROPIN" = "1" ]; then
warn "检测到 drop-in 覆盖目录: $DROPIN_DIR"
echo " 这会覆盖主文件的 OnCalendar,将被清除以确保新时间生效。"
fi
echo ""
confirm "确认修改?" || { echo "已取消。"; exit 0; }
# --- 改 timer 文件的 OnCalendar + Description ---
# 先清除 drop-in 覆盖目录(若有),否则会覆盖主文件的新 OnCalendar
if [ "$HAS_DROPIN" = "1" ]; then
rm -rf "$DROPIN_DIR"
warn "已清除 drop-in 目录: $DROPIN_DIR"
fi
# 用 sed 原地替换 OnCalendar 行
sed -i -E "s|^OnCalendar=.*|OnCalendar=*-*-* ${TIME}:00|" "$TIMER_PATH"
# 同步更新 Description 里的时间(若有)
sed -i -E "s|^(Description=.* at ).*|\1${TIME}|" "$TIMER_PATH"
ok "已更新 $TIMER_PATH (OnCalendar → ${TIME}:00)"
# --- 更新 conf(若有 conf 选项) ---
update_conf() {
local key="$1" val="$2"
if grep -qE "^${key}=" "$CONF_PATH" 2>/dev/null; then
# 已存在,替换
sed -i -E "s|^${key}=.*|${key}=${val}|" "$CONF_PATH"
else
# 不存在,追加
echo "${key}=${val}" >> "$CONF_PATH"
fi
}
if [ -n "$COUNT" ] || [ -n "$MAIL_TO" ] || [ -n "$MAIL_ON_SUCCESS" ]; then
mkdir -p "$CONF_DIR"
if [ ! -f "$CONF_PATH" ]; then
{
echo "# /etc/news-harvester/${SOURCE}.conf — 由 modify-source.sh 更新 $(date '+%Y-%m-%d %H:%M:%S')"
echo "# 仅写需要覆盖的字段,其余继承 default.conf"
echo ""
} > "$CONF_PATH"
fi
[ -n "$COUNT" ] && { update_conf COUNT "$COUNT"; }
[ -n "$MAIL_TO" ] && { update_conf MAIL_TO "$MAIL_TO"; }
[ -n "$MAIL_ON_SUCCESS" ] && { update_conf MAIL_ON_SUCCESS "$MAIL_ON_SUCCESS"; }
ok "已更新 $CONF_PATH"
fi
# --- daemon-reload + 重启 timer ---
# 先 stop,再 reload,再 start —— 某些 systemd 版本 restart 不会重新求值日历表达式
systemctl stop "$TIMER_NAME" 2>/dev/null || true
systemctl daemon-reload
ok "systemd 已重新加载"
systemctl start "$TIMER_NAME"
ok "已重启 $TIMER_NAME(新时间生效)"
echo ""
echo "========================================"
ok "栏目 $SOURCE 修改完成!"
echo "========================================"
echo ""
echo "下次触发时间:"
systemctl list-timers "news-harvester@${SOURCE}.timer" --no-pager 2>/dev/null || true
echo ""
echo "后续操作:"
echo " 查看定时器: systemctl list-timers news-harvester@${SOURCE}.timer"
echo " 手动测试: sudo systemctl start news-harvester@${SOURCE}.service"
echo " 查看运行结果: cat ~/Documents/News-Harvester/logs/${SOURCE}.last.log"
echo " 再次改时间: sudo $0 $SOURCE --time HH:MM"
systemd/news-harvester-digest.service 0 → 100644
View file @ c2d5873b
# 每日汇总服务 —— 扫描当天所有抓取记录,用 claude 无头模式生成汇总并报警
# 由 news-harvester-digest.timer 每天 22:00 拉起(等所有栏目跑完再汇总)。
[Unit]
Description=News Harvester — daily digest & alert
Documentation=file:/home/yichu/Documents/News-Harvester/bin/daily-digest.sh
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=yichu
Group=yichu
WorkingDirectory=/home/yichu/Documents/News-Harvester
# 只读 default.conf;digest 不需要每栏目配置
EnvironmentFile=-/etc/news-harvester/default.conf
ExecStart=/home/yichu/Documents/News-Harvester/bin/daily-digest.sh
TimeoutStartSec=600
Nice=10
Environment=HOME=/home/yichu
StandardOutput=journal
StandardError=journal
systemd/news-harvester-digest.timer 0 → 100644
View file @ c2d5873b
[Unit]
Description=Run News Harvester daily digest at 22:00
[Timer]
# 晚上 22:00 运行,确保当天所有栏目(通常上午 9 点档)都已跑完
OnCalendar=*-*-* 22:00:00
Persistent=true
AccuracySec=1min
Unit=news-harvester-digest.service
[Install]
WantedBy=timers.target
systemd/news-harvester-web.service 0 → 100644
View file @ c2d5873b
[Unit]
Description=News Harvester — Web Panel
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=root
WorkingDirectory=/home/yichu/Documents/News-Harvester
EnvironmentFile=-/etc/news-harvester/web.conf
ExecStart=/usr/bin/python3 /home/yichu/Documents/News-Harvester/web/app.py
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
systemd/news-harvester@.service 0 → 100644
View file @ c2d5873b
# 模板单元:实例化时 %i = 栏目名
# 例:news-harvester@nikkei-world.service → %i = nikkei-world
# 不需要 enable 此 service,由对应的 timer 拉起。
[Unit]
Description=News Harvester — fetch %i news into Obsidian
Documentation=file:/home/yichu/Documents/News-Harvester/bin/harvest.sh
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=yichu
Group=yichu
WorkingDirectory=/home/yichu/Documents/News-Harvester
# 配置加载顺序:default.conf 提供全局默认,%i.conf 覆盖/补充单栏目设置。
# 同名变量后者覆盖前者(systemd 规则)。文件不存在不报错(前导 -)。
# 可用变量:COUNT、MAIL_TO、MAIL_ON_SUCCESS、ALERT_COST_OVER、WORK_DIR、LOG_DIR
EnvironmentFile=-/etc/news-harvester/default.conf
EnvironmentFile=-/etc/news-harvester/%i.conf
# %i=栏目名, ${COUNT}=来自 conf 的条数(未设则为空,脚本兜底 20)
ExecStart=/home/yichu/Documents/News-Harvester/bin/harvest.sh %i ${COUNT}
TimeoutStartSec=1800
Nice=10
Environment=HOME=/home/yichu
StandardOutput=journal
StandardError=journal
systemd/news-harvester@nikkei-world.timer 0 → 100644
View file @ c2d5873b
[Unit]
Description=Run News Harvester (nikkei-world) daily at 09:00
[Timer]
OnCalendar=*-*-* 09:00:00
Persistent=true
AccuracySec=1min
Unit=news-harvester@nikkei-world.service
[Install]
WantedBy=timers.target
systemd/news-harvester@reuters.timer 0 → 100644
View file @ c2d5873b
[Unit]
Description=Run News Harvester (reuters) daily at 09:15
[Timer]
# 错开 15 分钟,避免与 nikkei-world 同时启动多个 claude 实例
OnCalendar=*-*-* 09:15:00
Persistent=true
AccuracySec=1min
Unit=news-harvester@reuters.service
[Install]
WantedBy=timers.target
web/app.py 0 → 100644
View file @ c2d5873b
#!/usr/bin/env python3
"""News Harvester Web Panel — Flask 单文件管理面板
路由:
GET / List 所有 source(timer 状态 + 最近运行摘要)
GET /source/<name> Source 详情(timer 信息 + 日志 + JSON 元数据)
GET /add 添加 source 表单
POST /add 执行 add-source.sh
GET /edit/<name> 编辑 source 表单(预填当前值)
POST /edit/<name> 执行 modify-source.sh
POST /run/<name> 手动触发抓取
POST /digest 手动触发每日汇总
POST /delete/<name> 删除 source(停用 + 删除 timer/conf)
"""
import functools
import glob
import json
import os
import re
import subprocess
import threading
from datetime import datetime
from pathlib import Path
from flask import (
Flask, abort, flash, redirect, render_template, request, url_for,
)
# ── 路径常量(与 bash 脚本一致) ──────────────────────────────
SYSTEMD_DIR = Path(os.environ.get("SYSTEMD_DIR", "/etc/systemd/system"))
CONF_DIR = Path(os.environ.get("CONF_DIR", "/etc/news-harvester"))
WORK_DIR = Path(os.environ.get("WORK_DIR", "/home/yichu/Documents/News-Harvester"))
LOG_DIR = Path(os.environ.get("LOG_DIR", str(WORK_DIR / "logs")))
PROJECT_DIR = Path(__file__).resolve().parent.parent
# 可被 deploy.sh 部署的 bin 脚本路径(服务器上)
ADD_SOURCE = str(WORK_DIR / "bin" / "add-source.sh")
MODIFY_SOURCE = str(WORK_DIR / "bin" / "modify-source.sh")
CONTINUE_SCRIPT = str(WORK_DIR / "bin" / "continue.sh")
DIGEST_SCRIPT = str(WORK_DIR / "bin" / "daily-digest.sh")
# ── Flask 应用 ─────────────────────────────────────────────
app = Flask(__name__)
app.secret_key = os.environ.get("WEB_SECRET", "news-harvester-web-change-me")
@app.context_processor
def inject_now():
return {"now": datetime.now().strftime("%Y-%m-%d %H:%M")}
# ── 后台任务跟踪(continue/repair) ──────────────────────────
# {source: {"mode": ..., "start": ..., "proc": Popen, "done": bool, "rc": int, "stdout": str, "stderr": str}}
_running_continues = {}
_running_lock = threading.Lock()
def _check_continue_status(source):
"""检查某 source 的后台 continue 任务状态,返回 dict 或 None。
如果任务刚完成(done=True),清理并返回结果(调用方负责 flash)。"""
with _running_lock:
if source not in _running_continues:
return None
info = _running_continues[source]
if not info.get("done"):
# 仍在运行
return {"running": True, "mode": info["mode"], "start": info["start"]}
# 已完成,取出结果并清理
result = {
"running": False,
"just_finished": True,
"mode": info["mode"],
"rc": info.get("rc", -1),
"stdout": info.get("stdout", ""),
"stderr": info.get("stderr", ""),
}
del _running_continues[source]
return result
def _is_continue_running(source):
"""快速检查某 source 是否有 continue 任务在运行"""
with _running_lock:
if source not in _running_continues:
return False
return not _running_continues[source].get("done", False)
# ── 后台任务跟踪(digest) ──────────────────────────────────
# 单槽:同时只允许一个汇总任务
_running_digest = {}
_digest_lock = threading.Lock()
def _check_digest_status():
"""检查后台 digest 任务状态。刚完成时取出结果并清理。"""
with _digest_lock:
if not _running_digest:
return None
if not _running_digest.get("done"):
return {"running": True, "start": _running_digest["start"]}
result = {
"running": False,
"just_finished": True,
"rc": _running_digest.get("rc", -1),
"stdout": _running_digest.get("stdout", ""),
"stderr": _running_digest.get("stderr", ""),
}
_running_digest.clear()
return result
def _is_digest_running():
with _digest_lock:
return bool(_running_digest) and not _running_digest.get("done", False)
# ── Basic Auth ─────────────────────────────────────────────
WEB_USER = os.environ.get("WEB_USER", "admin")
WEB_PASS = os.environ.get("WEB_PASS", "")
def require_auth(view):
"""HTTP Basic Auth 装饰器"""
@functools.wraps(view)
def wrapped(*args, **kwargs):
if WEB_PASS:
auth = request.authorization
if not auth or auth.username != WEB_USER or auth.password != WEB_PASS:
from flask import Response
return Response(
"需要认证", 401,
{"WWW-Authenticate": 'Basic realm="News Harvester"'},
)
return view(*args, **kwargs)
return wrapped
# ── 数据读取辅助函数 ───────────────────────────────────────
def get_timer_files():
"""扫描所有 news-harvester@*.timer 文件(不含 digest)"""
return sorted(SYSTEMD_DIR.glob("news-harvester@*.timer"))
def parse_timer_file(timer_path):
"""从 timer 文件解析 OnCalendar 时间"""
text = timer_path.read_text(encoding="utf-8", errors="replace")
m = re.search(r"^OnCalendar=.*?\d{2}:\d{2}:\d{2}", text, re.MULTILINE)
if m:
time_match = re.search(r"(\d{2}):(\d{2}):\d{2}", m.group(0))
if time_match:
return f"{time_match.group(1)}:{time_match.group(2)}"
return "-"
def read_oncalendar_from_systemd(timer_name):
"""从 systemctl show 读取实际生效的 OnCalendar(含 drop-in 覆盖)"""
try:
out = subprocess.run(
["systemctl", "show", timer_name, "-p", "OnCalendar"],
capture_output=True, text=True, timeout=5,
).stdout
m = re.search(r"(\d{2}):(\d{2}):\d{2}", out)
if m:
return f"{m.group(1)}:{m.group(2)}"
except Exception:
pass
return None
def get_systemd_props(timer_name):
"""获取 timer 的 systemd 属性:is-active / NextElapse / LastTrigger"""
props = {"active": "unknown", "next": "-", "last": "从未触发"}
try:
result = subprocess.run(
["systemctl", "is-active", timer_name],
capture_output=True, text=True, timeout=5,
)
props["active"] = result.stdout.strip() or "unknown"
except Exception:
pass
try:
out = subprocess.run(
["systemctl", "show", timer_name,
"-p", "NextElapseUSecRealtime", "-p", "LastTriggerUSec"],
capture_output=True, text=True, timeout=5,
).stdout
for line in out.splitlines():
if line.startswith("NextElapseUSecRealtime="):
val = line.split("=", 1)[1]
if val and val not in ("n/a", "0"):
props["next"] = val
elif line.startswith("LastTriggerUSec="):
val = line.split("=", 1)[1]
if val and val not in ("n/a", "0"):
props["last"] = val
except Exception:
pass
return props
def parse_last_log(source):
"""从 .last.log 提取运行摘要(状态/成本/Session/耗时/退出码/Token)"""
log_path = LOG_DIR / f"{source}.last.log"
if not log_path.exists():
return None
text = log_path.read_text(encoding="utf-8", errors="replace")
result = {}
for line in text.splitlines():
for key, label in [
("status", "状态:"), ("cost", "成本:"), ("session", "Session:"),
("duration", "耗时:"), ("exit_code", "退出码:"),
("turns", "Turns:"), ("token", "Token:"),
("denials", "拒绝:"), ("terminal", "Terminal:"),
]:
if line.startswith(label):
# 去掉 label 前缀(含可能的对齐空格)
val = line[len(label):].strip()
result[key] = val
# mtime
try:
mtime = datetime.fromtimestamp(log_path.stat().st_mtime)
result["mtime"] = mtime.strftime("%Y-%m-%d %H:%M")
except Exception:
result["mtime"] = "?"
result["raw"] = text
return result
def get_latest_json(source):
"""读取最新的 Claude JSON 输出文件"""
pattern = str(LOG_DIR / f"{source}.*.json")
files = sorted(glob.glob(pattern), reverse=True)
if not files:
return None
try:
with open(files[0], encoding="utf-8") as f:
data = json.load(f)
return {"path": files[0], "data": data}
except Exception:
return None
def read_conf(source):
"""读取 source 的 conf 文件,返回 dict"""
conf_path = CONF_DIR / f"{source}.conf"
if not conf_path.exists():
return {}
result = {}
for line in conf_path.read_text(encoding="utf-8", errors="replace").splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
if "=" in line:
k, v = line.split("=", 1)
result[k.strip()] = v.strip()
return result
def get_source_list():
"""获取所有 source 的完整信息列表"""
sources = []
for timer_path in get_timer_files():
# 从文件名提取 source 名
name = timer_path.stem # news-harvester@bloomberg-technology → 去掉 .timer
source = name.split("@", 1)[1] if "@" in name else name
timer_name = f"news-harvester@{source}.timer"
# 运行时间:优先 systemctl show,回退读文件
run_time = read_oncalendar_from_systemd(timer_name) or parse_timer_file(timer_path)
# systemd 属性
props = get_systemd_props(timer_name)
# 最近运行日志
log_info = parse_last_log(source)
# 状态判定
status_text = log_info.get("status", "") if log_info else ""
if "成功" in status_text:
status_class = "ok"
elif "失败" in status_text:
status_class = "fail"
else:
status_class = "unknown"
sources.append({
"source": source,
"timer_name": timer_name,
"run_time": run_time,
"active": props["active"],
"next": props["next"],
"last": props["last"],
"log": log_info,
"status_class": status_class,
})
# 按运行时间排序
sources.sort(key=lambda s: s["run_time"])
return sources
def run_script(cmd, timeout=30):
"""执行脚本,返回 (returncode, stdout, stderr)"""
try:
result = subprocess.run(
cmd, capture_output=True, text=True, timeout=timeout,
)
return result.returncode, result.stdout, result.stderr
except subprocess.TimeoutExpired:
return 124, "", "执行超时"
except Exception as e:
return 1, "", str(e)
# ── 路由 ───────────────────────────────────────────────────
@app.route("/")
@require_auth
def index():
sources = get_source_list()
# digest timer 状态
digest_props = get_systemd_props("news-harvester-digest.timer")
digest_run_time = read_oncalendar_from_systemd("news-harvester-digest.timer") or "10:00"
# 后台 digest 任务状态
digest_status = _check_digest_status()
auto_refresh = False
if digest_status:
if digest_status.get("running"):
auto_refresh = True
elif digest_status.get("just_finished"):
if digest_status["rc"] == 0:
flash(f"每日汇总完成\n{digest_status['stdout']}", "success")
else:
flash(f"每日汇总失败 (exit {digest_status['rc']})\n{digest_status['stdout']}\n{digest_status['stderr']}", "error")
return render_template("index.html", sources=sources,
digest_props=digest_props, digest_run_time=digest_run_time,
digest_status=digest_status, auto_refresh=auto_refresh)
@app.route("/source/<name>")
@require_auth
def source_detail(name):
if not re.match(r"^[a-zA-Z0-9_-]+$", name):
abort(404)
timer_path = SYSTEMD_DIR / f"news-harvester@{name}.timer"
if not timer_path.exists():
abort(404)
timer_name = f"news-harvester@{name}.timer"
run_time = read_oncalendar_from_systemd(timer_name) or parse_timer_file(timer_path)
props = get_systemd_props(timer_name)
log_info = parse_last_log(name)
json_info = get_latest_json(name)
conf = read_conf(name)
# 解析 JSON 中的元数据
json_meta = None
if json_info and json_info["data"]:
d = json_info["data"]
json_meta = {
"cost": d.get("total_cost_usd", "-"),
"turns": d.get("num_turns", "-"),
"duration_api": d.get("duration_api_ms", "-"),
"session_id": d.get("session_id", "-"),
"is_error": d.get("is_error", False),
"terminal": d.get("terminal_reason", "-"),
"usage": d.get("usage", {}),
"permission_denials": d.get("permission_denials", []),
}
# 检查后台 continue 任务状态
continue_status = _check_continue_status(name)
auto_refresh = False
if continue_status:
if continue_status.get("running"):
auto_refresh = True
elif continue_status.get("just_finished"):
mode_label = "续采" if continue_status["mode"] == "resume" else "修复"
if continue_status["rc"] == 0:
flash(f"{name} {mode_label}完成\n{continue_status['stdout']}", "success")
else:
flash(f"{name} {mode_label}失败 (exit {continue_status['rc']})\n{continue_status['stdout']}\n{continue_status['stderr']}", "error")
# 重新读日志(continue.sh 刚更新了 .last.log)
log_info = parse_last_log(name)
json_info = get_latest_json(name)
return render_template("detail.html", name=name, run_time=run_time,
props=props, log_info=log_info, json_info=json_info,
json_meta=json_meta, conf=conf,
continue_status=continue_status, auto_refresh=auto_refresh)
@app.route("/add", methods=["GET", "POST"])
@require_auth
def add_source():
if request.method == "GET":
return render_template("add.html")
# POST: 执行 add-source.sh
source = request.form.get("source", "").strip()
time = request.form.get("time", "").strip()
count = request.form.get("count", "").strip()
mail_to = request.form.get("mail_to", "").strip()
mail_on_success = request.form.get("mail_on_success", "").strip()
if not source or not re.match(r"^[a-zA-Z0-9_-]+$", source):
flash("栏目名只能含字母、数字、下划线、连字符", "error")
return render_template("add.html", form=request.form)
cmd = [ADD_SOURCE, source, "--yes"]
if time:
if not re.match(r"^([01]\d|2[0-3]):[0-5]\d$", time):
flash("时间格式应为 HH:MM(24小时制)", "error")
return render_template("add.html", form=request.form)
cmd += ["--time", time]
if count:
cmd += ["--count", count]
if mail_to:
cmd += ["--mail-to", mail_to]
if mail_on_success:
cmd += ["--mail-on-success", mail_on_success]
rc, stdout, stderr = run_script(cmd, timeout=30)
if rc == 0:
flash(f"栏目 {source} 添加成功\n{stdout}", "success")
return redirect(url_for("source_detail", name=source))
else:
flash(f"添加失败 (exit {rc})\n{stdout}\n{stderr}", "error")
return render_template("add.html", form=request.form)
@app.route("/edit/<name>", methods=["GET", "POST"])
@require_auth
def edit_source(name):
if not re.match(r"^[a-zA-Z0-9_-]+$", name):
abort(404)
timer_path = SYSTEMD_DIR / f"news-harvester@{name}.timer"
if not timer_path.exists():
abort(404)
if request.method == "GET":
timer_name = f"news-harvester@{name}.timer"
current_time = read_oncalendar_from_systemd(timer_name) or parse_timer_file(timer_path)
conf = read_conf(name)
return render_template("edit.html", name=name, current_time=current_time, conf=conf)
# POST: 执行 modify-source.sh
time = request.form.get("time", "").strip()
count = request.form.get("count", "").strip()
mail_to = request.form.get("mail_to", "").strip()
mail_on_success = request.form.get("mail_on_success", "").strip()
if not time or not re.match(r"^([01]\d|2[0-3]):[0-5]\d$", time):
flash("时间格式应为 HH:MM(24小时制)", "error")
timer_name = f"news-harvester@{name}.timer"
current_time = read_oncalendar_from_systemd(timer_name) or parse_timer_file(timer_path)
conf = read_conf(name)
return render_template("edit.html", name=name, current_time=current_time, conf=conf)
cmd = [MODIFY_SOURCE, name, "--time", time, "--yes"]
if count:
cmd += ["--count", count]
if mail_to:
cmd += ["--mail-to", mail_to]
if mail_on_success:
cmd += ["--mail-on-success", mail_on_success]
rc, stdout, stderr = run_script(cmd, timeout=30)
if rc == 0:
flash(f"栏目 {name} 修改成功\n{stdout}", "success")
else:
flash(f"修改失败 (exit {rc})\n{stdout}\n{stderr}", "error")
return redirect(url_for("source_detail", name=name))
@app.route("/run/<name>", methods=["POST"])
@require_auth
def run_source(name):
if not re.match(r"^[a-zA-Z0-9_-]+$", name):
abort(404)
rc, stdout, stderr = run_script(
["systemctl", "start", f"news-harvester@{name}.service"], timeout=10
)
if rc == 0:
flash(f"已触发 {name} 抓取", "success")
else:
flash(f"触发失败: {stderr}", "error")
return redirect(url_for("source_detail", name=name))
@app.route("/digest", methods=["POST"])
@require_auth
def run_digest():
if _is_digest_running():
flash("每日汇总任务已在运行中,请等待完成", "error")
return redirect(url_for("index"))
# 后台启动 daily-digest.sh(非阻塞)
# 以 yichu 用户执行,确保文件权限/邮件/PATH 全部正确
proc = subprocess.Popen(
["sudo", "-u", "yichu", DIGEST_SCRIPT],
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
)
def _wait_completion():
stdout, stderr = proc.communicate()
with _digest_lock:
_running_digest["done"] = True
_running_digest["rc"] = proc.returncode
_running_digest["stdout"] = stdout or ""
_running_digest["stderr"] = stderr or ""
with _digest_lock:
_running_digest.clear()
_running_digest["start"] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
_running_digest["done"] = False
t = threading.Thread(target=_wait_completion, daemon=True)
t.start()
flash("每日汇总任务已在后台启动,页面将自动刷新状态", "success")
return redirect(url_for("index"))
@app.route("/continue/<name>", methods=["POST"])
@require_auth
def continue_source(name):
if not re.match(r"^[a-zA-Z0-9_-]+$", name):
abort(404)
# 已有任务在运行,拒绝重复提交
if _is_continue_running(name):
flash(f"{name} 已有续采/修复任务在运行中,请等待完成", "error")
return redirect(url_for("source_detail", name=name))
mode = request.form.get("mode", "resume").strip()
if mode not in ("resume", "repair"):
mode = "resume"
debug_flag = request.form.get("debug", "").strip()
# 后台启动 continue.sh(非阻塞)
# 以 yichu 用户执行(通过 sudo -u),确保文件权限/邮件/PATH 全部正确
cmd = ["sudo", "-u", "yichu", CONTINUE_SCRIPT, name, "--mode", mode, "--yes"]
if debug_flag:
cmd.append("--debug")
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
)
def _wait_completion():
stdout, stderr = proc.communicate()
with _running_lock:
if name in _running_continues:
_running_continues[name]["done"] = True
_running_continues[name]["rc"] = proc.returncode
_running_continues[name]["stdout"] = stdout or ""
_running_continues[name]["stderr"] = stderr or ""
with _running_lock:
_running_continues[name] = {
"mode": mode,
"start": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"proc": proc,
"done": False,
}
# 后台线程等待完成,不阻塞 HTTP 请求
t = threading.Thread(target=_wait_completion, daemon=True)
t.start()
mode_label = "续采" if mode == "resume" else "修复"
flash(f"{name} {mode_label}任务已在后台启动,页面将自动刷新状态", "success")
return redirect(url_for("source_detail", name=name))
@app.route("/disable/<name>", methods=["POST"])
@require_auth
def disable_source(name):
"""停用栏目:stop + disable timer,清空 OnCalendar 使其不再触发。
不删除 timer/conf 文件,随时可以重新编辑启用。"""
if not re.match(r"^[a-zA-Z0-9_-]+$", name):
abort(404)
timer_name = f"news-harvester@{name}.timer"
timer_path = SYSTEMD_DIR / f"news-harvester@{name}.timer"
messages = []
# 停用 timer
run_script(["systemctl", "stop", timer_name], timeout=10)
run_script(["systemctl", "disable", timer_name], timeout=10)
messages.append(f"已停用 {timer_name}")
# 清空 OnCalendar(改为永不触发),保留文件
if timer_path.exists():
import re as _re
text = timer_path.read_text(encoding="utf-8", errors="replace")
text = _re.sub(r"^OnCalendar=.*", "OnCalendar=", text, flags=_re.MULTILINE)
timer_path.write_text(text, encoding="utf-8")
messages.append(f"已清空运行时间(OnCalendar 置空,不再触发)")
run_script(["systemctl", "daemon-reload"], timeout=10)
flash(f"栏目 {name} 已停用\n" + "\n".join(messages) + "\n\n文件保留,可通过编辑重新设置时间来启用", "success")
return redirect(url_for("source_detail", name=name))
if __name__ == "__main__":
app.run(host="127.0.0.1", port=8080, debug=False)
web/templates/add.html 0 → 100644
View file @ c2d5873b
{% extends "base.html" %}
{% block title %}添加栏目 — News Harvester{% endblock %}
{% block content %}
<h1>添加栏目</h1>
<form method="POST" action="{{ url_for('add_source') }}">
<div class="form-group">
<label for="source">栏目名称 *</label>
<input type="text" id="source" name="source" required
pattern="[a-zA-Z0-9_-]+"
placeholder="如 bloomberg-technology"
value="{{ form.source if form else '' }}">
<div class="hint">只能含字母、数字、下划线、连字符。不能用 digest。</div>
</div>
<div class="form-group">
<label for="time">运行时间 (HH:MM)</label>
<input type="text" id="time" name="time"
pattern="([01]\d|2[0-3]):[0-5]\d"
placeholder="留空自动错开(现有最晚 +15 分钟)"
value="{{ form.time if form else '' }}">
<div class="hint">24 小时制,如 09:30。留空则自动推算。</div>
</div>
<div class="form-group">
<label for="count">抓取条数</label>
<input type="number" id="count" name="count" min="1"
placeholder="留空继承 default.conf"
value="{{ form.count if form else '' }}">
</div>
<div class="form-group">
<label for="mail_to">收件人</label>
<input type="email" id="mail_to" name="mail_to"
placeholder="留空继承 default.conf(支持多个,逗号分隔)"
value="{{ form.mail_to if form else '' }}">
</div>
<div class="form-group">
<label for="mail_on_success">成功是否发邮件</label>
<select id="mail_on_success" name="mail_on_success">
<option value="">留空继承 default.conf</option>
<option value="1">是 — 每次都发</option>
<option value="0">否 — 仅失败时发</option>
</select>
</div>
<div class="actions">
<button type="submit" class="btn btn-primary">创建栏目</button>
<a class="btn" href="{{ url_for('index') }}">取消</a>
</div>
</form>
{% endblock %}
web/templates/base.html 0 → 100644
View file @ c2d5873b
<!DOCTYPE html>
<html lang="zh">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{% block title %}News Harvester{% endblock %}</title>
{% block extra_head %}{% endblock %}
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: -apple-system, "Segoe UI", Roboto, "Noto Sans SC", sans-serif;
background: #0f1117;
color: #d4d4d4;
line-height: 1.6;
padding: 0;
}
nav {
background: #1a1d27;
border-bottom: 1px solid #2a2d37;
padding: 14px 24px;
display: flex;
align-items: center;
gap: 24px;
}
nav .logo {
font-size: 18px;
font-weight: 700;
color: #569cd6;
text-decoration: none;
}
nav a {
color: #888;
text-decoration: none;
font-size: 14px;
transition: color 0.2s;
}
nav a:hover { color: #d4d4d4; }
.container {
max-width: 1600px;
margin: 0 auto;
padding: 24px;
}
h1 { font-size: 22px; margin-bottom: 16px; color: #e0e0e0; }
h2 { font-size: 18px; margin: 20px 0 10px; color: #c0c0c0; }
/* Flash messages */
.flash {
padding: 12px 16px;
border-radius: 6px;
margin-bottom: 16px;
font-size: 14px;
white-space: pre-wrap;
word-break: break-all;
}
.flash.success { background: #1a3a1a; border: 1px solid #2a5a2a; color: #6bbb6b; }
.flash.error { background: #3a1a1a; border: 1px solid #5a2a2a; color: #bb6b6b; }
/* Running banner */
.running-banner {
display: flex;
align-items: center;
gap: 10px;
padding: 14px 16px;
border-radius: 6px;
margin-bottom: 16px;
font-size: 14px;
background: #1a2333;
border: 1px solid #2a4a6a;
color: #569cd6;
}
.spinner {
width: 16px;
height: 16px;
border: 2px solid #2a4a6a;
border-top-color: #569cd6;
border-radius: 50%;
animation: spin 0.8s linear infinite;
flex-shrink: 0;
}
@keyframes spin { to { transform: rotate(360deg); } }
/* Table */
table {
width: 100%;
border-collapse: collapse;
margin-bottom: 20px;
}
th {
text-align: left;
padding: 10px 12px;
background: #1a1d27;
font-size: 12px;
text-transform: uppercase;
color: #888;
border-bottom: 1px solid #2a2d37;
letter-spacing: 0.5px;
}
td {
padding: 10px 12px;
border-bottom: 1px solid #1e2128;
font-size: 14px;
white-space: nowrap;
}
tr:hover { background: #161922; }
tr.ok td:first-child { border-left: 3px solid #4ec9b0; }
tr.fail td:first-child { border-left: 3px solid #f44747; }
tr.unknown td:first-child { border-left: 3px solid #888; }
/* Badges */
.badge {
display: inline-block;
padding: 2px 8px;
border-radius: 4px;
font-size: 12px;
font-weight: 600;
}
.badge.active { background: #1a3a1a; color: #6bbb6b; }
.badge.inactive { background: #3a1a1a; color: #bb6b6b; }
.badge.unknown { background: #2a2a2a; color: #888; }
.badge.ok { background: #1a3a1a; color: #6bbb6b; }
.badge.fail { background: #3a1a1a; color: #bb6b6b; }
/* Buttons */
.btn {
display: inline-block;
padding: 6px 14px;
border-radius: 4px;
border: 1px solid #3a3d47;
background: #252830;
color: #d4d4d4;
font-size: 13px;
text-decoration: none;
cursor: pointer;
transition: all 0.2s;
}
.btn:hover { background: #2d303a; border-color: #4a4d57; }
.btn-primary { background: #264f78; border-color: #3a6fa0; color: #fff; }
.btn-primary:hover { background: #2d5a8a; }
.btn-danger { background: #5a2a2a; border-color: #7a3a3a; color: #fff; }
.btn-danger:hover { background: #6a3333; }
.btn-sm { padding: 3px 10px; font-size: 12px; }
.btn:disabled { opacity: 0.5; cursor: not-allowed; }
/* Forms */
.form-group {
margin-bottom: 16px;
}
.form-group label {
display: block;
font-size: 13px;
color: #aaa;
margin-bottom: 4px;
}
.form-group input, .form-group select {
width: 100%;
max-width: 400px;
padding: 8px 12px;
background: #1a1d27;
border: 1px solid #2a2d37;
border-radius: 4px;
color: #d4d4d4;
font-size: 14px;
}
.form-group input:focus { border-color: #569cd6; outline: none; }
.form-group .hint { font-size: 12px; color: #666; margin-top: 3px; }
/* Pre / code */
pre {
background: #1a1d27;
border: 1px solid #2a2d37;
border-radius: 6px;
padding: 14px;
overflow-x: auto;
font-size: 13px;
line-height: 1.5;
font-family: "SF Mono", "Fira Code", "Noto Sans Mono", monospace;
color: #c8c8c8;
}
code { font-family: "SF Mono", "Fira Code", monospace; color: #ce9178; }
/* Detail page cards */
.cards { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 20px; }
.card {
background: #1a1d27;
border: 1px solid #2a2d37;
border-radius: 6px;
padding: 14px 18px;
min-width: 140px;
}
.card .label { font-size: 12px; color: #888; text-transform: uppercase; margin-bottom: 4px; }
.card .value { font-size: 18px; font-weight: 600; color: #e0e0e0; }
.card .value.green { color: #4ec9b0; }
.card .value.red { color: #f44747; }
.actions { display: flex; gap: 8px; margin: 16px 0; flex-wrap: wrap; }
.session-id { font-family: monospace; font-size: 12px; color: #569cd6; }
a.source-link { color: #569cd6; text-decoration: none; }
a.source-link:hover { text-decoration: underline; }
.footer { text-align: center; color: #555; font-size: 12px; margin-top: 30px; padding: 16px; }
</style>
</head>
<body>
<nav>
<a class="logo" href="{{ url_for('index') }}">📰 News Harvester</a>
<a href="{{ url_for('index') }}">栏目列表</a>
<a href="{{ url_for('add_source') }}">添加栏目</a>
</nav>
<div class="container">
{% with messages = get_flashed_messages(with_categories=true) %}
{% for category, message in messages %}
<div class="flash {{ category }}">{{ message }}</div>
{% endfor %}
{% endwith %}
{% block content %}{% endblock %}
</div>
<div class="footer">News Harvester Web Panel · {{ now }}</div>
</body>
</html>
web/templates/detail.html 0 → 100644
View file @ c2d5873b
{% extends "base.html" %}
{% block title %}{{ name }} — News Harvester{% endblock %}
{% block extra_head %}
{% if auto_refresh %}
<meta http-equiv="refresh" content="10">
{% endif %}
{% endblock %}
{% block content %}
<h1>{{ name }}</h1>
{% if continue_status and continue_status.running %}
<div class="running-banner">
<div class="spinner"></div>
<span>{{ continue_status.mode | replace('resume', '续采') | replace('repair', '修复') }}任务运行中… (启动于 {{ continue_status.start }})</span>
<span style="margin-left:auto;color:#666;font-size:12px">页面每 10 秒自动刷新</span>
</div>
{% endif %}
<div class="actions">
<a class="btn" href="{{ url_for('edit_source', name=name) }}">编辑配置</a>
<form method="POST" action="{{ url_for('run_source', name=name) }}" style="display:inline"
onsubmit="return confirm('确定手动触发 {{ name }} 抓取？')">
<button type="submit" class="btn btn-primary" {% if continue_status and continue_status.running %}disabled{% endif %}>手动触发</button>
</form>
<form method="POST" action="{{ url_for('continue_source', name=name) }}" style="display:inline"
onsubmit="return confirm('确定对 {{ name }} 执行继续采集？\n将恢复上次 Claude 对话,在已有基础上继续完成中断的采集。')">
<input type="hidden" name="mode" value="resume">
<button type="submit" class="btn" {% if continue_status and continue_status.running %}disabled{% endif %}>
{% if continue_status and continue_status.running and continue_status.mode == 'resume' %}续采中…{% else %}继续采集{% endif %}
</button>
</form>
<form method="POST" action="{{ url_for('continue_source', name=name) }}" style="display:inline"
onsubmit="return confirm('⚠️ 确定对 {{ name }} 执行采集修复？\n\n这将:\n1. 恢复上次 Claude 对话\n2. 清理之前归档的错误数据\n3. 重新完整采集并 force 推送\n\n此操作会删除之前本次采集的文章,无法撤销!')">
<input type="hidden" name="mode" value="repair">
<button type="submit" class="btn btn-danger" {% if continue_status and continue_status.running %}disabled{% endif %}>
{% if continue_status and continue_status.running and continue_status.mode == 'repair' %}修复中…{% else %}采集修复{% endif %}
</button>
</form>
<a class="btn" href="{{ url_for('index') }}">返回列表</a>
<form method="POST" action="{{ url_for('disable_source', name=name) }}" style="display:inline"
onsubmit="return confirm('确定停用栏目 {{ name }}？\n\n这将:\n1. 停止并 disable 定时器\n2. 清空运行时间(不再触发)\n\n文件保留,可随时通过编辑重新设置时间来启用。')">
<button type="submit" class="btn btn-danger" {% if continue_status and continue_status.running %}disabled{% endif %}>停用栏目</button>
</form>
</div>
<h2>Timer 信息</h2>
<div class="cards">
<div class="card">
<div class="label">运行时间</div>
<div class="value"><code>{{ run_time }}</code></div>
</div>
<div class="card">
<div class="label">状态</div>
<div class="value">
{% if props.active == 'active' %}
<span class="green">active</span>
{% else %}
<span class="red">{{ props.active }}</span>
{% endif %}
</div>
</div>
<div class="card">
<div class="label">上次触发</div>
<div class="value" style="font-size:14px">{{ props.last }}</div>
</div>
<div class="card">
<div class="label">下次触发</div>
<div class="value" style="font-size:14px">{{ props.next }}</div>
</div>
</div>
{% if conf %}
<h2>栏目配置 ({{ name }}.conf)</h2>
<table>
<tbody>
{% for k, v in conf.items() %}
<tr><td><code>{{ k }}</code></td><td>{{ v }}</td></tr>
{% endfor %}
</tbody>
</table>
{% else %}
<h2>栏目配置</h2>
<p style="color:#666">无独立配置,继承 <code>default.conf</code></p>
{% endif %}
{% if log_info %}
<h2>最近运行</h2>
<div class="cards">
<div class="card">
<div class="label">状态</div>
<div class="value
{% if '成功' in log_info.get('status', '') %}green
{% elif '失败' in log_info.get('status', '') %}red{% endif %}">
{{ log_info.get('status', '?') }}
</div>
</div>
<div class="card">
<div class="label">成本</div>
<div class="value">{{ log_info.get('cost', '-') }}</div>
</div>
<div class="card">
<div class="label">耗时</div>
<div class="value" style="font-size:14px">{{ log_info.get('duration', '-') }}</div>
</div>
<div class="card">
<div class="label">Turns</div>
<div class="value">{{ log_info.get('turns', '-') }}</div>
</div>
<div class="card">
<div class="label">修改时间</div>
<div class="value" style="font-size:14px">{{ log_info.get('mtime', '?') }}</div>
</div>
</div>
{% if log_info.get('session') %}
<div class="card" style="margin-bottom:16px">
<div class="label">Claude Session ID</div>
<div class="value" style="font-size:13px">
<span class="session-id">{{ log_info.session }}</span>
</div>
</div>
{% endif %}
<h2>运行日志 (.last.log)</h2>
<pre>{{ log_info.raw }}</pre>
{% else %}
<h2>最近运行</h2>
<p style="color:#666">暂无运行记录</p>
{% endif %}
{% if json_meta %}
<h2>Claude JSON 元数据</h2>
<div class="cards">
<div class="card">
<div class="label">总成本</div>
<div class="value">${{ json_meta.cost }}</div>
</div>
<div class="card">
<div class="label">Turns</div>
<div class="value">{{ json_meta.turns }}</div>
</div>
<div class="card">
<div class="label">API 耗时</div>
<div class="value" style="font-size:14px">{{ json_meta.duration_api }} ms</div>
</div>
<div class="card">
<div class="label">is_error</div>
<div class="value {% if not json_meta.is_error %}green{% else %}red{% endif %}">
{{ json_meta.is_error }}
</div>
</div>
<div class="card">
<div class="label">Terminal</div>
<div class="value" style="font-size:14px">{{ json_meta.terminal }}</div>
</div>
</div>
{% if json_meta.session_id and json_meta.session_id != '-' %}
<div class="card" style="margin-bottom:16px">
<div class="label">Session ID</div>
<div class="value" style="font-size:13px">
<span class="session-id">{{ json_meta.session_id }}</span>
</div>
</div>
{% endif %}
{% if json_meta.usage %}
<h3>Token 用量</h3>
<table>
<thead>
<tr><th>类型</th><th>Token 数</th></tr>
</thead>
<tbody>
{% for k, v in json_meta.usage.items() %}
<tr><td><code>{{ k }}</code></td><td>{{ v }}</td></tr>
{% endfor %}
</tbody>
</table>
{% endif %}
{% if json_meta.permission_denials %}
<h3 style="color:#f44747">权限拒绝 ({{ json_meta.permission_denials|length }} 次)</h3>
<pre>{{ json_meta.permission_denials | tojson(indent=2) }}</pre>
{% endif %}
<p style="color:#666;font-size:12px;margin-top:8px">来源: {{ json_info.path }}</p>
{% endif %}
{% endblock %}
web/templates/edit.html 0 → 100644
View file @ c2d5873b
{% extends "base.html" %}
{% block title %}编辑 {{ name }} — News Harvester{% endblock %}
{% block content %}
<h1>编辑栏目: {{ name }}</h1>
<form method="POST" action="{{ url_for('edit_source', name=name) }}">
<div class="form-group">
<label for="time">运行时间 (HH:MM) *</label>
<input type="text" id="time" name="time" required
pattern="([01]\d|2[0-3]):[0-5]\d"
value="{{ current_time }}">
<div class="hint">24 小时制,如 10:30。当前: {{ current_time }}</div>
</div>
<div class="form-group">
<label for="count">抓取条数</label>
<input type="number" id="count" name="count" min="1"
placeholder="留空不变"
value="{{ conf.get('COUNT', '') }}">
<div class="hint">当前: {{ conf.get('COUNT', '(继承 default.conf)') }}</div>
</div>
<div class="form-group">
<label for="mail_to">收件人</label>
<input type="text" id="mail_to" name="mail_to"
placeholder="留空不变"
value="{{ conf.get('MAIL_TO', '') }}">
<div class="hint">当前: {{ conf.get('MAIL_TO', '(继承 default.conf)') }}</div>
</div>
<div class="form-group">
<label for="mail_on_success">成功是否发邮件</label>
<select id="mail_on_success" name="mail_on_success">
<option value="">留空不变</option>
<option value="1" {% if conf.get('MAIL_ON_SUCCESS') == '1' %}selected{% endif %}>是 — 每次都发</option>
<option value="0" {% if conf.get('MAIL_ON_SUCCESS') == '0' %}selected{% endif %}>否 — 仅失败时发</option>
</select>
<div class="hint">当前: {{ conf.get('MAIL_ON_SUCCESS', '(继承 default.conf)') }}</div>
</div>
<div class="actions">
<button type="submit" class="btn btn-primary">保存修改</button>
<a class="btn" href="{{ url_for('source_detail', name=name) }}">取消</a>
</div>
</form>
{% endblock %}
web/templates/index.html 0 → 100644
View file @ c2d5873b
{% extends "base.html" %}
{% block title %}News Harvester — 栏目列表{% endblock %}
{% block extra_head %}
{% if auto_refresh %}
<meta http-equiv="refresh" content="10">
{% endif %}
{% endblock %}
{% block content %}
{% if digest_status and digest_status.running %}
<div class="running-banner">
<div class="spinner"></div>
<span>每日汇总运行中… (启动于 {{ digest_status.start }})</span>
<span style="margin-left:auto;color:#666;font-size:12px">页面每 10 秒自动刷新</span>
</div>
{% endif %}
<div class="actions">
<a class="btn btn-primary" href="{{ url_for('add_source') }}">+ 添加栏目</a>
<form method="POST" action="{{ url_for('run_digest') }}" style="display:inline"
onsubmit="return confirm('确定手动触发每日汇总？')">
<button type="submit" class="btn" {% if digest_status and digest_status.running %}disabled{% endif %}>
{% if digest_status and digest_status.running %}汇总中…{% else %}手动触发汇总{% endif %}
</button>
</form>
</div>
<h1>栏目列表 <span style="font-size:14px;color:#666">({{ sources|length }} 个)</span></h1>
<table>
<thead>
<tr>
<th>栏目</th>
<th>状态</th>
<th>运行时间</th>
<th>上次触发</th>
<th>下次触发</th>
<th>最近运行</th>
<th>成本</th>
<th>Session</th>
<th>操作</th>
</tr>
</thead>
<tbody>
{% for s in sources %}
<tr class="{{ s.status_class }}">
<td>
<a class="source-link" href="{{ url_for('source_detail', name=s.source) }}">{{ s.source }}</a>
</td>
<td>
{% if s.active == 'active' %}
<span class="badge active">active</span>
{% elif s.active == 'inactive' %}
<span class="badge inactive">inactive</span>
{% else %}
<span class="badge unknown">{{ s.active }}</span>
{% endif %}
</td>
<td><code>{{ s.run_time }}</code></td>
<td>{{ s.last }}</td>
<td>{{ s.next }}</td>
<td>
{% if s.log %}
{{ s.log.mtime }} ·
{% if '成功' in s.log.get('status', '') %}
<span class="badge ok">成功</span>
{% elif '失败' in s.log.get('status', '') %}
<span class="badge fail">失败</span>
{% else %}
{{ s.log.get('status', '?') }}
{% endif %}
{% else %}
<span style="color:#666">—</span>
{% endif %}
</td>
<td>
{% if s.log and s.log.get('cost') %}
{% set cost_num = s.log.cost | replace('$', '') | replace('成本:', '') | trim %}
{% set cost_str = cost_num | float %}
${{ "%.2f"|format(cost_str) }}
{% else %}
<span style="color:#666">—</span>
{% endif %}
</td>
<td>
{% if s.log and s.log.get('session') %}
<span class="session-id" title="{{ s.log.session }}">{{ s.log.session }}</span>
{% else %}
<span style="color:#666">—</span>
{% endif %}
</td>
<td>
<a class="btn btn-sm" href="{{ url_for('source_detail', name=s.source) }}">详情</a>
<a class="btn btn-sm" href="{{ url_for('edit_source', name=s.source) }}">编辑</a>
</td>
</tr>
{% endfor %}
</tbody>
</table>
<h2>每日汇总</h2>
<table>
<thead>
<tr>
<th>Timer</th>
<th>状态</th>
<th>运行时间</th>
<th>上次触发</th>
<th>下次触发</th>
<th>操作</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>news-harvester-digest.timer</code></td>
<td>
{% if digest_props.active == 'active' %}
<span class="badge active">active</span>
{% else %}
<span class="badge {{ digest_props.active }}">{{ digest_props.active }}</span>
{% endif %}
</td>
<td><code>{{ digest_run_time }}</code></td>
<td>{{ digest_props.last }}</td>
<td>{{ digest_props.next }}</td>
<td>
<form method="POST" action="{{ url_for('run_digest') }}" style="display:inline"
onsubmit="return confirm('确定手动触发每日汇总？')">
<button type="submit" class="btn btn-sm" {% if digest_status and digest_status.running %}disabled{% endif %}>
{% if digest_status and digest_status.running %}汇总中…{% else %}触发{% endif %}
</button>
</form>
</td>
</tr>
</tbody>
</table>
{% endblock %}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment