Commit 97c2602b authored by xieyichu's avatar xieyichu

Remove Discord publishing capabilities

Discord publishing (per-article threads, weekly summary threads,
discordChannelId field, weeklyThreads config) has been removed from
the skill. This functionality will be handled by separate skills
that depend on the Agent's message tooling.

Changes:
- Delete scripts/save-weekly-thread.js
- harvest.js: remove isoWeekKey/weekDateRange helpers, Discord manifest
  fields (discordChannelId, weekKey, weekRange, weeklyThreadId), and
  publishing comment block
- source-manage.js: remove CHANNEL column from list, discordChannelId
  from add/edit/usage
- config-template.json: remove weeklyThreads and discordChannelId fields
- source-management.md: remove Discord fields and Weekly Thread section
- SKILL.md: remove Publishing to Discord section, Workflow step 7,
  discordThreadId from registry format, Discord from description/intro
- README.md: remove all Discord references from capability table,
  architecture diagram, prerequisites, and limitations
- .gitignore: add .zcode/ workspace dir
parent 3843445b
......@@ -11,3 +11,6 @@ node_modules/
# Logs / temp
*.log
/tmp/
# IDE / agent workspace files
.zcode/
# News Harvester Skill
一个 [Claude Code](https://docs.claude.com/en/docs/claude-code) Skill:从配置的新闻源(Reuters、The Conversation、The Guardian 等)采集最新文章,生成中文摘要 + 英文原文,归档到 Obsidian vault,并可选发布到 Discord 论坛频道。支持去重、分类、自动周报
一个 [Claude Code](https://docs.claude.com/en/docs/claude-code) Skill:从配置的新闻源(Reuters、The Conversation、The Guardian 等)采集最新文章,生成中文摘要 + 英文原文,归档到 Obsidian vault。支持去重和自动分类
> 适用于任何能加载 Claude Code Skill 的 Agent 环境(交互模式 / 无头模式 `-p`)。
......@@ -13,7 +13,6 @@
| 🗂 数据源管理 | 添加 / 列出 / 编辑 / 启用禁用新闻源 |
| 📡 文章采集 | 抓取首页链接 → 提取全文(标题/日期/作者/正文/配图) |
| 🗄 Obsidian 归档 | 每篇文章生成中文摘要 + 英文原文 + 当日索引 + 去重注册表 |
| 📢 Discord 发布 | 每篇文章一个论坛帖 + 每周汇总帖(可选) |
| 🔁 去重 | 滑动窗口(默认 7 天)的 registry,避免重复采集 |
| 🏷 自动分类 | 基于关键词映射到 7 个分类(地缘政治/科技AI/中美关系/财经/社会/国际冲突/国际) |
......@@ -26,18 +25,17 @@ config.json ──▶ harvest.js ──▶ fetcher-*.js ──▶ 文章数据(J
(数据源) (主调度器) (direct/browser) │
▲ │ ▼
│ dedup 去重 Agent(LLM)
│ (registry) 翻译/摘要/分类/发布
│ │ ▼
save-weekly-thread.js ◀──────────────────── Agent ┌─────────┐
(持久化周帖ID) ├─▶ Obsidian
└─▶ Discord
│ (registry) 翻译/摘要/分类
Obsidian vault
```
**脚本与 Agent 分工**
- **脚本**`scripts/`):确定性工作 —— HTTP 请求、HTML 解析、图片下载、去重、文件写入
- **Agent**(LLM):智能工作 —— 中英翻译、生成中文摘要、文章分类、Discord 发布
- **Agent**(LLM):智能工作 —— 中英翻译、生成中文摘要、文章分类
`harvest.js` 完成采集后输出 JSON manifest,提示 Agent 接管翻译与发布
`harvest.js` 完成采集后输出 JSON manifest,提示 Agent 接管翻译与文档生成
---
......@@ -57,8 +55,7 @@ news-harvester/
│ ├── harvest.js # 主入口:node harvest.js <source_id> <count>
│ ├── source-manage.js # 数据源增删改查
│ ├── fetcher-direct.js # 纯 HTTP 抓取(开放站点)
│ ├── fetcher-browser.js # Chromium CDP 抓取(JS 渲染站点)
│ └── save-weekly-thread.js # 保存 Discord 周帖 ID
│ └── fetcher-browser.js # Chromium CDP 抓取(JS 渲染站点)
└── data/ # ⚠️ 运行时去重数据,不入库
└── registry.json
```
......@@ -72,7 +69,6 @@ news-harvester/
- [Node.js](https://nodejs.org/) ≥ 18
- 一个可写的 Obsidian vault 目录
- (可选)Chromium —— 仅当使用 `browser` 方法的源(如 reuters)时需要
- (可选)Discord Bot —— 仅当需要发布到 Discord 时需要
### 1. 克隆并安装依赖
......@@ -106,7 +102,6 @@ ln -s "$(pwd)" ~/.claude/skills/news-harvester
{
"vaultPath": "/your/actual/obsidian/vault/path", // ← 必改
"registryWindowDays": 7,
"weeklyThreads": {},
"sources": [ ... ]
}
```
......@@ -153,7 +148,7 @@ node scripts/harvest.js the-conversation 5
```
`harvest.js` 会:抓取链接 → 去重 → 逐篇抓取全文 → 下载图片 → 输出 JSON manifest。
然后 Agent 接管:生成中文摘要 → 写入 Obsidian 文档 → 更新 index.md / registry.json → (可选)发布 Discord
然后 Agent 接管:生成中文摘要 → 写入 Obsidian 文档 → 更新 index.md / registry.json。
### 无头模式调用示例
......@@ -194,7 +189,6 @@ claude -p \
| `method` | `direct`(纯 HTTP)或 `browser`(需 Chromium) | `direct` |
| `articleUrlPattern` | 文章 URL 匹配正则 | `theconversation\\.com/[\\w-]+-\\d+$` |
| `vaultFolder` | vault 下的子文件夹名 | `The Conversation` |
| `discordChannelId` | Discord 论坛频道 ID(可空) | `1234567890123456789` |
| `enabled` | 是否启用 | `true` |
---
......@@ -227,7 +221,6 @@ YYYYMMDD/
--user-data-dir=/tmp/chromium-harvest --disable-gpu "about:blank" &>/dev/null &
```
- **图片下载无重试**:失败仅跳过
- **Discord 发布由 Agent 完成**:脚本不直接调 Discord API
---
......
---
name: news-harvester
description: Harvest, archive, and publish news articles from configured sources (Reuters, The Conversation, etc.) to Obsidian vault and Discord forum channels. Use this skill whenever the user wants to: fetch latest news from a source, add/list/edit news sources, run a harvest job, check what's been archived, avoid duplicate articles, or set up automated news collection. Triggers on phrases like "抓取新闻", "harvest news", "fetch articles", "add source", "run harvest", "news sources", or any request to collect/archive/publish articles from a website.
description: Harvest and archive news articles from configured sources (Reuters, The Conversation, etc.) to Obsidian vault with Chinese summaries + original text. Use this skill whenever the user wants to: fetch latest news from a source, add/list/edit news sources, run a harvest job, check what's been archived, avoid duplicate articles, or set up automated news collection. Triggers on phrases like "抓取新闻", "harvest news", "fetch articles", "add source", "run harvest", "news sources", or any request to collect/archive articles from a website.
---
# News Harvester Skill
Fetch latest articles from configured news sources, archive them to Obsidian vault with Chinese summaries + original text, and publish to Discord forum channels. Supports browser-based (Chromium CDP) and direct HTTP fetching. Deduplicates via per-day registry with a sliding window.
Fetch latest articles from configured news sources, archive them to Obsidian vault with Chinese summaries + original text. Supports browser-based (Chromium CDP) and direct HTTP fetching. Deduplicates via per-day registry with a sliding window.
## Quick Reference
......@@ -16,6 +16,8 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau
| List sources | `node source-manage.js list` |
| Edit source | `node source-manage.js edit <id>` |
> Scripts are invoked from the skill root directory.
Scripts live in: `scripts/` (relative to this skill's root directory)
Config file: `config.json` (auto-generated from `references/config-template.json` on first run — see Configuration below)
......@@ -23,7 +25,7 @@ Config file: `config.json` (auto-generated from `references/config-template.json
## Configuration
`config.json` is **not shipped** with this skill — it is machine-specific (contains local vault path, Discord channel IDs, and runtime thread IDs). On first run, any script auto-creates `config.json` by copying `references/config-template.json`.
`config.json` is **not shipped** with this skill — it is machine-specific (contains the local vault path). On first run, any script auto-creates `config.json` by copying `references/config-template.json`.
Before running a harvest, the agent MUST verify/edit these fields:
......@@ -31,8 +33,6 @@ Before running a harvest, the agent MUST verify/edit these fields:
|-------|-----------|-------------|
| `vaultPath` | ✅ Yes | Absolute path to the Obsidian vault root. Must exist and be writable. Replace the template placeholder `/path/to/your/obsidian/vault`. |
| `registryWindowDays` | Optional | Dedup sliding window in days (default `7`). |
| `sources[].discordChannelId` | Optional | Discord forum channel ID for publishing. Leave empty `""` to skip Discord. |
| `weeklyThreads` | Auto-managed | Populated at runtime by `scripts/save-weekly-thread.js`; never edit by hand. |
To point a source at a different vault subfolder, set `sources[].vaultFolder`. Articles are archived to `<vaultPath>/<vaultFolder>/YYYYMMDD/`.
......@@ -76,11 +76,7 @@ For each URL (in series, not parallel):
3. Generate Chinese summary doc → `<vaultPath>/<vaultFolder>/YYYYMMDD/<中文标题>.md`
4. Generate original doc → `<vaultPath>/<vaultFolder>/YYYYMMDD/<英文标题> - <SourceName>原文.md`
5. Update `YYYYMMDD/index.md`
6. Publish to Discord (see Publishing section)
7. Append to `YYYYMMDD/registry.json`
### 7. Post run summary to weekly thread
Find or create the weekly Discord thread (see Weekly Thread section), then post a summary reply.
6. Append to `YYYYMMDD/registry.json`
---
......@@ -217,7 +213,6 @@ File: `<vaultFolder>/YYYYMMDD/registry.json`
"titleZh": "中文标题",
"titleEn": "English Title",
"processedAt": "2026-02-28T14:30:00+08:00",
"discordThreadId": "1477xxx",
"category": "地缘政治",
"tags": ["tag1", "tag2"]
}
......@@ -227,38 +222,6 @@ File: `<vaultFolder>/YYYYMMDD/registry.json`
---
## Publishing to Discord
### Per-article thread
1. `thread-create` on `source.discordChannelId`:
- `threadName`: `<emoji> <中文标题>`**纯文字,不含任何 Markdown 符号(无 `**`、`#` 等)**,Discord 标题不渲染 Markdown
- `message`: formatted Chinese summary with sections (Markdown OK here)
- End with: `原文见下方回复 👇`
2. `send` reply to thread:
- Original title + `<url>` (wrapped in `<>`)
- Image URLs (plain, Discord auto-previews)
- Full English body (split if >1800 chars)
### Weekly thread
- Thread name format: `📰 <SourceName> 周归档 · YYYY-WNN(MM/DD–MM/DD)`
- Store thread ID in config under `weeklyThreads.<source_id>.<YYYY-WNN>`
- **ALWAYS check config for existing thread before creating a new one.** The manifest output from `harvest.js` includes `weeklyThreadId` — use it if non-null. If null, create the thread and immediately save the new ID:
```bash
node scripts/save-weekly-thread.js <source_id> <week_key> <new_thread_id>
```
- Each run appends one reply:
```
── YYYY-MM-DD HH:MM · 新增 N 篇 · 跳过 M 篇 ──
<emoji> <中文标题1>
<emoji> <中文标题2>
...
```
To find current week key: ISO week number, e.g. `2026-W09`.
---
## Source Management
Read `references/source-management.md` for detailed instructions on add/list/edit operations.
......
{
"vaultPath": "/path/to/your/obsidian/vault",
"registryWindowDays": 7,
"weeklyThreads": {},
"sources": [
{
"id": "the-conversation",
......@@ -10,7 +9,6 @@
"method": "direct",
"articleUrlPattern": "theconversation\\.com/[\\w-]+-\\d+$",
"vaultFolder": "The Conversation",
"discordChannelId": "",
"enabled": true
}
]
......
......@@ -5,9 +5,9 @@
Read `config.json` and display:
```
ID NAME METHOD CHANNEL ENABLED
reuters 路透社 browser 1234567890123456789
theconversation The Conversation direct (not set)
ID NAME METHOD ENABLED
reuters 路透社 browser ✅
theconversation The Conversation direct ❌
```
## Add Source
......@@ -22,7 +22,6 @@ Prompt the user for (or infer from context):
| `method` | `browser` (needs JS) or `direct` (plain HTTP) | `browser` |
| `articleUrlPattern` | Regex to match article URLs | `reuters\\.com/.+/\\d{4}-\\d{2}-\\d{2}/` |
| `vaultFolder` | Subfolder under vaultPath | `路透社` |
| `discordChannelId` | Forum channel ID | `1234567890123456789` |
| `enabled` | Start enabled? | `true` |
Auto-detect `articleUrlPattern` if not provided:
......@@ -33,7 +32,7 @@ Auto-detect `articleUrlPattern` if not provided:
After collecting all fields, append to `config.json` sources array and confirm:
```
✅ Added source "路透社" (id: reuters)
Method: browser | Channel: 1234567890123456789
Method: browser
Run: node harvest.js reuters 5
```
......@@ -43,7 +42,7 @@ Usage: `node source-manage.js edit <id>`
Show current values, prompt for field to change, update `config.json`.
Editable fields: `name`, `homepageUrl`, `method`, `articleUrlPattern`, `vaultFolder`, `discordChannelId`, `enabled`, `countPerRun` (if set as default).
Editable fields: `name`, `homepageUrl`, `method`, `articleUrlPattern`, `vaultFolder`, `enabled`, `countPerRun` (if set as default).
## Enable / Disable Source
......@@ -52,29 +51,3 @@ Toggle `enabled` field:
node source-manage.js enable reuters
node source-manage.js disable reuters
```
## Weekly Thread Management
Weekly thread IDs are stored in config under `weeklyThreads`:
```json
{
"weeklyThreads": {
"reuters": {
"2026-W09": "1234567890123456789"
}
}
}
```
To find current ISO week:
```python
from datetime import datetime, date
d = date.today()
week = d.isocalendar()[1]
year = d.isocalendar()[0]
key = f"{year}-W{week:02d}"
```
If no thread exists for current week, create one with `thread-create` and save the ID.
Thread name format: `📰 <SourceName> 周归档 · YYYY-WNN(MM/DD–MM/DD)`
Include week date range in the name (Monday–Sunday).
......@@ -36,22 +36,6 @@ function todayStr() {
return `${d.getFullYear()}${String(d.getMonth()+1).padStart(2,'0')}${String(d.getDate()).padStart(2,'0')}`;
}
function isoWeekKey(d = new Date()) {
const jan4 = new Date(d.getFullYear(), 0, 4);
const startOfWeek = new Date(jan4);
startOfWeek.setDate(jan4.getDate() - ((jan4.getDay() + 6) % 7));
const weekNum = Math.floor((d - startOfWeek) / (7 * 24 * 3600 * 1000)) + 1;
return `${d.getFullYear()}-W${String(weekNum).padStart(2,'0')}`;
}
function weekDateRange(d = new Date()) {
const day = d.getDay();
const mon = new Date(d); mon.setDate(d.getDate() - ((day + 6) % 7));
const sun = new Date(mon); sun.setDate(mon.getDate() + 6);
const fmt = (x) => `${String(x.getMonth()+1).padStart(2,'0')}/${String(x.getDate()).padStart(2,'0')}`;
return `${fmt(mon)}${fmt(sun)}`;
}
function getCategory(url, title) {
const u = url.toLowerCase(), t = title.toLowerCase();
if (/asia-pacific|pakistan|afghanistan/.test(u) || /巴基斯坦|阿富汗/.test(t)) return '地缘政治';
......@@ -274,10 +258,6 @@ ${catNav}
fs.writeFileSync(indexPath, content);
}
// ── Discord Publishing ────────────────────────────────────────────────────────
// Note: Discord publishing is done via the agent (message tool), not directly from this script.
// This script outputs a JSON manifest that the agent uses to publish.
// ── Main ──────────────────────────────────────────────────────────────────────
async function main() {
......@@ -413,10 +393,6 @@ async function main() {
sourceName: source.name,
today,
dayDir,
discordChannelId: source.discordChannelId,
weekKey: isoWeekKey(),
weekRange: weekDateRange(),
weeklyThreadId: (config.weeklyThreads && config.weeklyThreads[sourceId] && config.weeklyThreads[sourceId][isoWeekKey()]) || null,
processed,
skipped,
newCount: processed.length
......@@ -424,7 +400,7 @@ async function main() {
process.stdout.write(JSON.stringify(manifest, null, 2));
console.error(`\n Done: ${processed.length} processed, ${skipped} skipped`);
console.error(' Agent should now: generate Chinese summaries, create docs, publish to Discord, update registry');
console.error(' Agent should now: generate Chinese summaries, create docs, update registry');
}
main().catch(e => { console.error('Fatal:', e); process.exit(1); });
#!/usr/bin/env node
/**
* save-weekly-thread.js
* Save a weekly Discord thread ID to config.json so subsequent runs can reuse it.
* Usage: node save-weekly-thread.js <source_id> <week_key> <thread_id>
* Example: node save-weekly-thread.js the-conversation 2026-W09 1234567890123456789
*/
const fs = require('fs');
const path = require('path');
const SKILL_DIR = path.dirname(__dirname);
const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
const [sourceId, weekKey, threadId] = process.argv.slice(2);
if (!sourceId || !weekKey || !threadId) {
console.error('Usage: node save-weekly-thread.js <source_id> <week_key> <thread_id>');
console.error('Example: node save-weekly-thread.js reuters 2026-W09 1234567890123456789');
process.exit(1);
}
const config = JSON.parse(fs.readFileSync(CONFIG_PATH, 'utf8'));
if (!config.weeklyThreads) config.weeklyThreads = {};
if (!config.weeklyThreads[sourceId]) config.weeklyThreads[sourceId] = {};
config.weeklyThreads[sourceId][weekKey] = threadId;
fs.writeFileSync(CONFIG_PATH, JSON.stringify(config, null, 2));
console.log(`✅ Saved: ${sourceId} / ${weekKey}${threadId}`);
......@@ -50,15 +50,14 @@ const config = loadConfig();
switch (command) {
case 'list': {
console.log('\n📋 Configured Sources\n');
console.log('ID'.padEnd(20) + 'NAME'.padEnd(20) + 'METHOD'.padEnd(10) + 'CHANNEL'.padEnd(22) + 'STATUS');
console.log('─'.repeat(80));
console.log('ID'.padEnd(20) + 'NAME'.padEnd(20) + 'METHOD'.padEnd(10) + 'STATUS');
console.log('─'.repeat(60));
for (const s of config.sources) {
const status = s.enabled ? '✅ enabled' : '❌ disabled';
console.log(
s.id.padEnd(20) +
s.name.padEnd(20) +
s.method.padEnd(10) +
(s.discordChannelId || '(not set)').padEnd(22) +
status
);
}
......@@ -95,7 +94,6 @@ switch (command) {
method: args.method || 'direct',
articleUrlPattern: pattern || '',
vaultFolder: args.folder || args.name || id,
discordChannelId: args.channel || '',
enabled: args.enabled !== 'false'
};
......@@ -103,7 +101,7 @@ switch (command) {
saveConfig(config);
console.log(`\n✅ Added source "${newSource.name}" (id: ${id})`);
console.log(` Method: ${newSource.method} | Channel: ${newSource.discordChannelId || '(not set)'}`);
console.log(` Method: ${newSource.method}`);
console.log(` URL Pattern: ${newSource.articleUrlPattern}`);
console.log(` Vault Folder: ${newSource.vaultFolder}`);
console.log(`\n Run: node harvest.js ${id} 5\n`);
......@@ -115,7 +113,7 @@ switch (command) {
const source = config.sources.find(s => s.id === id);
if (!source) { console.error(`Source "${id}" not found`); process.exit(1); }
const editableFields = ['name', 'homepageUrl', 'method', 'articleUrlPattern', 'vaultFolder', 'discordChannelId', 'enabled'];
const editableFields = ['name', 'homepageUrl', 'method', 'articleUrlPattern', 'vaultFolder', 'enabled'];
let changed = false;
for (const field of editableFields) {
......@@ -153,7 +151,7 @@ switch (command) {
console.log(`
Usage:
node source-manage.js list
node source-manage.js add --id <id> --name <name> --url <homepage> --method browser|direct --channel <discord_id>
node source-manage.js add --id <id> --name <name> --url <homepage> --method browser|direct
node source-manage.js edit <id> --name "New Name" --channel 123456
node source-manage.js enable <id>
node source-manage.js disable <id>
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment