docs: 新机全流程部署手册(AI 执行版,含站点凭据与推送配置,仅限内网私仓)

3245cbf2 · 郭阔 · 7295d704 · 3245cbf2
Commit 3245cbf2 authored Oct 23, 2026 by 郭阔
Show whitespace changes
Inline Side-by-side

Showing with 247 additions and 0 deletions

新机部署手册.md docs/新机部署手册.md +247 -0

No files found.
--- a/docs/新机部署手册.md
+++ b/docs/新机部署手册.md
+# 新 Mac 全流程部署手册(AI 执行版)
+> **本文档写给 AI 看**:在一台全新的 Mac 上,按顺序执行本手册,把新闻采集系统完整跑起来——定时采集、自动登录、自动推送 News Hub、Web 管理面板。
+>
+> ⚠️ **本文档含明文账号密码与推送凭证**,仅限内网私有 GitLab(git.lab.bj.jiweinet.com)存放。若仓库将来外迁或开源,先删除《站点登录凭据》《推送配置》两节的敏感值。
+>
+> 执行原则:**每一步做完立即验证,验证失败先排查再前进;所有脚本幂等,失败可重跑。**
+---
+## 0. 系统架构(30 秒理解)
+```
+launchd 定时(com.newsharvester.<source>)
+  → harvest.sh <source>                 # bin/ 下,平台自适应
+      → 登录预检(站点凭据 + 配方,自动登录)
+      → claude CLI(headless,调 news-harvester 技能)
+      → 采到的文章 → Obsidian 库 ~/NewsHub/<站点>/<频道>/<日期>/*.md
+      → finalize → 自动推送 News Hub(192.168.1.7:8091)
+Web 面板(127.0.0.1:8080,常驻 launchd)= 管理/监控入口
+```
+关键模型:
+- **站点 = 源名第一段**:`nikkei-world/markets/tech` 同属 `nikkei` 站点,**一份凭据 + 一份登录配方覆盖全部频道**
+- **每个源独立浏览器实例**(独立 CDP 端口 9300+hash、独立 profile `~/.news-harvester/profiles/<source>/`),可并行采集互不干扰
+- **平台自适应**:所有脚本自动识别 macOS/Linux,无需适配代码
+---
+## 1. 人工前置(仅 2 件,先做完再继续)
+1. **拷贝 Claude 认证**(必须):从旧 Mac 拷 `~/.claude/settings.json` 到新机同路径。
+   内容为 GLM 中转配置(`https://open.bigmodel.cn/api/anthropic` + token + glm-4.7 模型映射)。
+   校验:`claude -p "回复 ok"` 能正常返回。
+2. **安装 Google Chrome**(必须):`brew install --cask google-chrome` 或官网下载。采集浏览器基于它。
+其他依赖(node/jq/gnu-sed/python3)由 setup.sh 自动检查,brew 可装的会征询后自动装。
+---
+## 2. 拉取两个仓库
+```bash
+mkdir -p ~/ZCodeProject && cd ~/ZCodeProject
+git clone https://git.lab.bj.jiweinet.com/guokuo/macos-runtime.git news-harvester-runtime
+# 技能仓会被 setup.sh 自动 clone 到 ~/news-hub-skills,也可手动:
+git clone https://git.lab.bj.jiweinet.com/guokuo/macos-hub-skills.git ~/news-hub-skills
+```
+git 认证:内网 GitLab 账号(首次 clone 会提示,输入一次后钥匙串记住)。
+## 3. 一键自举
+```bash
+cd ~/ZCodeProject/news-harvester-runtime
+./bin/setup.sh --skills-repo https://git.lab.bj.jiweinet.com/guokuo/macos-hub-skills.git --with-web --yes
+```
+setup.sh 自动完成(幂等可重跑):平台检测、目录结构(`~/NewsHarvester/{etc,logs,agents}`)、依赖检查、claude 认证检查、技能仓 clone + 三技能 `npm install` + 全量 bundle 导入、`default.conf`/`web.conf` 生成(**面板密码随机生成并打印一次,记下来**)、Web 面板常驻服务(launchd `com.newsharvester.web`)。
+**验证**:
+```bash
+bash bin/status.sh                 # 应列出源(初始可能为空)且无报错
+curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8080/   # 期望 401(面板存活且要求认证)
+```
+> ⚠️ 注意:WORK_DIR 是 `~/NewsHarvester`(不是 ~/Documents,规避 macOS TCC 对 launchd 的文件访问限制),**不要改**。
+## 4. 技能主配置(推送 + 14 个数据源)
+技能仓的 `config.json` 在 .gitignore 里(**新机 clone 后不存在,必须按本节创建**)。写到 `~/news-hub-skills/skills/news-harvester/config.json`:
+> `vaultPath` 用新机的家目录(把 `<HOME>` 替换为 `/Users/<新用户名>`)。
+```json
+{
+  "vaultPath": "<HOME>/NewsHub",
+  "registryWindowDays": 7,
+  "push": {
+    "enabled": true,
+    "endpoint": "http://192.168.1.7:8091/api/v1",
+    "appId": "app_harvester",
+    "appKeyId": "kh_harvester_default",
+    "appSecret": "Aijiwei@news_hub_harvest!",
+    "batchSize": 10,
+    "autoPushAfterFinalize": true,
+    "retry": { "maxAttempts": 5, "baseDelayMs": 1000 }
+  },
+  "chromium": {
+    "cdpPort": 9222,
+    "headless": false,
+    "executablePath": null,
+    "userDataDir": null,
+    "maxConcurrentTabs": 15
+  },
+  "sources": [
+    { "id": "the-conversation", "name": "The Conversation", "homepageUrl": "https://theconversation.com/us", "method": "direct", "articleUrlPattern": "theconversation\\.com/[\\w-]+-\\d+$", "vaultFolder": "The Conversation", "enabled": true },
+    { "id": "nikkei-world", "name": "日经亚洲世界", "homepageUrl": "https://asia.nikkei.com/location", "method": "browser", "vaultFolder": "日经亚洲/世界", "enabled": true, "concurrency": 2 },
+    { "id": "nikkei-markets", "name": "日经亚洲市场", "homepageUrl": "https://asia.nikkei.com/business/markets", "method": "browser", "vaultFolder": "日经亚洲/市场", "enabled": true, "concurrency": 2 },
+    { "id": "nikkei-tech", "name": "日经亚洲科技", "homepageUrl": "https://asia.nikkei.com/business/tech", "method": "browser", "vaultFolder": "日经亚洲/科技", "enabled": true, "concurrency": 2 },
+    { "id": "reuters-world", "name": "路透社世界", "homepageUrl": "https://www.reuters.com/world/", "method": "browser", "vaultFolder": "路透社/世界", "enabled": true, "concurrency": 4 },
+    { "id": "reuters-markets", "name": "路透社市场", "homepageUrl": "https://www.reuters.com/markets/", "method": "browser", "vaultFolder": "路透社/市场", "enabled": true, "concurrency": 2 },
+    { "id": "reuters-business", "name": "路透社商业", "homepageUrl": "https://www.reuters.com/business/", "method": "browser", "vaultFolder": "路透社/商业", "enabled": true, "concurrency": 1 },
+    { "id": "reuters-technology", "name": "路透社科技", "homepageUrl": "https://www.reuters.com/technology/", "method": "browser", "vaultFolder": "路透社/科技", "enabled": true, "concurrency": 1, "paginate": false },
+    { "id": "bloomberg-technology", "name": "彭博社科技", "homepageUrl": "https://www.bloomberg.com/technology", "method": "browser", "vaultFolder": "彭博社/科技", "enabled": true, "concurrency": 2 },
+    { "id": "digitimes-feed", "name": "电子时报·动态", "homepageUrl": "https://www.digitimes.com/calendar.php?d=7d", "method": "browser", "vaultFolder": "电子时报/动态", "enabled": true, "concurrency": 1, "paginate": false },
+    { "id": "digitimes-tech", "name": "电子时报·科技", "homepageUrl": "https://www.digitimes.com/tech/", "method": "browser", "vaultFolder": "电子时报/科技", "enabled": true, "concurrency": 2, "paginate": false },
+    { "id": "wccftech-news", "name": "Wccftech 新闻", "homepageUrl": "https://wccftech.com/category/news/", "method": "direct", "vaultFolder": "Wccftech/新闻", "enabled": true, "concurrency": 1, "paginate": false, "articleUrlPattern": "wccftech\\.com/(?!category/|topic/|tag/|review/|how-to/|videos/|roundup/|tip-us/|feed/|author/|page/|ethics-statement|how-we-test)[a-z0-9-]{20,}" },
+    { "id": "ft", "name": "金融时报", "homepageUrl": "https://www.ft.com/news-feed", "method": "browser", "vaultFolder": "金融时报", "enabled": true, "concurrency": 5, "paginate": true },
+    { "id": "wsj", "name": "华尔街日报", "homepageUrl": "https://www.wsj.com/news/latest-headlines?mod=wsjfooter", "method": "browser", "vaultFolder": "华尔街日报", "enabled": true, "concurrency": 3, "paginate": false }
+  ]
+}
+```
+**验证**:`node -e "console.log(require('/Users/' + process.env.USER + '/news-hub-skills/skills/news-harvester/config.json').sources.length)"` → 14
+## 5. 站点登录凭据(7 个平台)
+写入 `~/NewsHarvester/etc/news-harvester/logins/<站点>.json`,**权限必须 600**。每份格式 `{"url","user","password"}`:
+| 文件 | url | user | password |
+|---|---|---|---|
+| `ft.json` | https://www.ft.com/ | wangqy@lunion.com.cn | qijiwei@123.com |
+| `digitimes.json` | https://www.digitimes.com/ | admin@ijiwei.com | ijiwei@com9527 |
+| `nikkei.json` | https://asia.nikkei.com/Latestheadlines | wangqy@lunion.com.cn | ijiwei@com9527 |
+| `bloomberg.json` | https://www.bloomberg.com/technology | liuyi@lunion.com.cn | ijiwei@com9527 |
+| `wsj.json` | https://www.wsj.com/ | admin@ijiwei.com | ijiwei@com9527 |
+| `statista.json` | https://www.statista.com/ | wangqy@lunion.com.cn | ijiwei@com9527 |
+| `reuters.json` | https://www.reuters.com/ | admin@ijiwei.com | Aijiwei@123 |
+一键落盘(AI 直接执行):
+```bash
+cd ~/NewsHarvester/etc/news-harvester/logins   # 目录不存在则 mkdir -p
+python3 - <<'EOF'
+import json, pathlib
+creds = {
+ "ft":       ("https://www.ft.com/",                     "wangqy@lunion.com.cn", "qijiwei@123.com"),
+ "digitimes":("https://www.digitimes.com/",              "admin@ijiwei.com",     "ijiwei@com9527"),
+ "nikkei":   ("https://asia.nikkei.com/Latestheadlines", "wangqy@lunion.com.cn", "ijiwei@com9527"),
+ "bloomberg":("https://www.bloomberg.com/technology",    "liuyi@lunion.com.cn",  "ijiwei@com9527"),
+ "wsj":      ("https://www.wsj.com/",                    "admin@ijiwei.com",     "ijiwei@com9527"),
+ "statista": ("https://www.statista.com/",               "wangqy@lunion.com.cn", "ijiwei@com9527"),
+ "reuters":  ("https://www.reuters.com/",                "admin@ijiwei.com",     "Aijiwei@123"),
+}
+for site, (url, user, pw) in creds.items():
+    p = pathlib.Path(f"{site}.json"); p.write_text(json.dumps({"url": url, "user": user, "password": pw}, ensure_ascii=False, indent=2))
+    p.chmod(0o600)
+print("done")
+EOF
+```
+**登录配方不用配**:技能仓自带 `logins/{nikkei,digitimes,ft,statista,bloomberg}.json`(站点级,clone 即得)。
+## 6. 定时采集排程
+`cd ~/ZCodeProject/news-harvester-runtime` 后逐条执行(现网节奏,可按需调整):
+```bash
+./bin/add-source.sh digitimes-feed --time 09:00
+./bin/add-source.sh nikkei-world  --time 09:15
+./bin/add-source.sh wsj           --time 09:30
+./bin/add-source.sh ft            --time 09:45
+```
+每日汇总 22:00 定时已由 setup.sh 自动安装(`com.newsharvester.digest`),无需手动;`launchctl list | grep digest` 确认。邮件当前未配收件人(`default.conf` 的 `MAIL_TO` 为空),只落日志;需要发邮件时填 `MAIL_TO` 即可。
+**验证**:
+```bash
+systemctl list-timers 2>/dev/null || launchctl list | grep newsharvester
+bash bin/status.sh        # 每个源应有 next_fire 时间
+```
+## 7. 端到端验收(依次做,全部通过才算完成)
+```bash
+cd ~/ZCodeProject/news-harvester-runtime
+export SKILLS_DIR=~/news-hub-skills
+# ① 冒烟采集:direct 源,不需要浏览器,最快验证 claude/技能/归档/推送全链
+./bin/harvest.sh the-conversation
+#   期望:logs 下生成结果 JSON,status SUCCESS;
+#   ~/NewsHub/The Conversation/<今日>/ 出现 md 文件;
+#   日志含 finalize 后自动推送 News Hub 成功(created 条目)
+# ② 登录预检(会拉起各源专属浏览器实例)
+bash bin/platform.sh call nh_login_preflight ft        # 期望:已登录 或 自动登录成功
+bash bin/platform.sh call nh_login_preflight nikkei-world
+bash bin/platform.sh call nh_login_preflight digitimes-feed
+# ③ 浏览器源实采一次(验证登录态 + 付费正文)
+./bin/harvest.sh nikkei-world
+#   期望:19-20/20 成功,正文非付费墙截断
+# ④ News Hub 核对:人工打开 http://192.168.1.7:8091/articles 看新文章
+# ⑤ 面板
+open http://127.0.0.1:8080    # web.conf 里的账号密码登录
+#   检查:栏目列表(14 源)、站点登录页(7 站点)、运行历史
+```
+## 8. 已知边界(如实告知用户,不要反复重试)
+| 站点 | 状况 | 处理 |
+|---|---|---|
+| statista | 登录页 Cloudflare Turnstile 人机验证 | **人工**在弹出的浏览器里登录一次,验证 cookie 留在 profile,之后自动登录正常 |
+| bloomberg | 登录 iframe 吞合成点击,引擎暂不支持真实点击 | **人工**登录一次(密码 ijiwei@com9527);或等 trusted-click 引擎支持 |
+| wsj | DataDome 反爬 | 自动发现会如实报"需人工";采集照常(付费墙文章如实标记) |
+| reuters | 首页无登录入口 | 用户明确:暂不处理 |
+| digitimes | 账号有登录频率限制 | **不要反复测试登录**;预检失败等次日 |
+## 9. 日常运维速查
+```bash
+# 手动采一个源 / 续采修复
+./bin/harvest.sh <source>
+./bin/continue.sh <source>
+# 状态 / 日志 / 面板
+bash bin/status.sh
+ls ~/NewsHarvester/logs/            # <source>.<时间戳>.json/.log、launchd-*.log
+open http://127.0.0.1:8080
+# 改采集时间 / 停启用源(停启用走面板开关;命令行修改时间:)
+./bin/modify-source.sh <source> --time 10:00
+# 推送:补推某天 / 强制重推(幂等,按 slug 去重,后到变更新)
+cd ~/news-hub-skills/skills/news-harvester
+node scripts/push.js "<vault>/<站点>/<频道>/<YYYYMMDD>"
+node scripts/push.js "<dayDir>" --force --bump-processed-at
+# 数据位置
+~/NewsHub/<站点>/<频道>/<日期>/*.md    # Obsidian 库(原文+摘要)
+~/NewsHarvester/logs/                  # 运行记录
+```
+## 10. 故障排查入口
+- 定时没跑:`launchctl list | grep newsharvester` → 看 `~/NewsHarvester/logs/launchd-<source>.log`
+- 采集失败:结果 JSON 里 `failures[]` 有逐篇原因;`api_error` 类是模型服务问题,等几分钟 `continue.sh` 续采
+- 登录失败:面板「站点登录」→ 测试登录,看输出;对照第 8 节边界表
+- 推送失败:`node scripts/push.js <dayDir>` 直接跑看报错(401=凭证,网络=News Hub 不通)
+- 面板起不来:`launchctl kickstart -k gui/$(id -u)/com.newsharvester.web`,日志 `~/NewsHarvester/logs/launchd-web.log`
+---
+*部署完成后,本机与 192.168.1.7 老服务器共用同一 News Hub;同文章按 slug 幂等去重(后到的一方变版本更新),若要分工可在面板关掉重叠频道。*