feat: complete skills.md

SKILL.md

@@ -20,6 +20,7 @@ Fetch latest articles from configured news sources, archive them to Obsidian vau
 | Edit source | `node scripts/source-manage.js edit <id>` |
 | Scaffold a source recipe | `node scripts/source-manage.js helper <id>` |
 | Inspect a source's page structure | `node scripts/inspect-source.js <url> --scroll 3 [--pattern '<regex>'] [--write <id>]` |
+| Run recipe regression tests | `node scripts/recipe-test.js <source_id>` |
 | Ensure Chromium (browser sources) | `node scripts/ensure-chromium.js` (or `--check`) |
 
 > **All commands assume `cd <skill-root>` first** (the directory containing this `SKILL.md`). Scripts live in the `scripts/` subfolder — **always prefix with `scripts/`**, e.g. `node scripts/harvest.js`, never `node harvest.js` (that fails with `MODULE_NOT_FOUND`).
@@ -127,9 +128,14 @@ Each source can have a **recipe** — a `helpers/<source-id>.md` file whose YAML
 
 > **Precedence**: when a recipe exists, its `urlPattern`/selectors are the source of truth and `config.json`'s `articleUrlPattern` is **ignored**. Sources with a recipe therefore don't need (and shouldn't carry) `articleUrlPattern` in config — it's a legacy-direct-source field only. `source-manage.js list` shows a `RECIPE` column so you can tell at a glance which sources are on which path.
 
-Recipes solve the "can't find high-quality data on new CDP sources" problem: instead of one hard-coded regex scanning the whole page (which mixes nav/sidebar/footer links with real articles), a recipe scopes link collection to a `listSelector` container, triggers lazy-load via `scrollSteps`, and validates URLs with a tightened `urlPattern`. The recipe is just markdown — edit it to adapt to a site redesign without touching code.
+Recipes solve the "can't find high-quality data on new CDP sources" problem: instead of one hard-coded regex scanning the whole page (which mixes nav/sidebar/footer links with real articles), a recipe scopes link collection to a `listSelector` container, triggers lazy-load via `scrollSteps`, and validates URLs with a `urlPattern`. Two layers of filtering keep the candidate pool clean:
 
-**Adding a new browser source** (3-step closed loop: add → inspect --write → verify):
+1. **Link collection** (`listSelector` + `urlPattern`): scope to the article-card containers and match a URL shape. This is the coarse filter — it runs once on the list page.
+2. **Article validation** (`validateArticle`, optional): when a candidate page is actually fetched, probe `og:type` + JSON-LD `@type` + body length and reject non-article pages (section/category pages that share an article's URL shape). This is the fine filter — it lets you use a *broad* `urlPattern` (no fragile "slug must have N hyphens" guesswork) and still keep section pages out of the archive.
+
+The recipe is just markdown — edit it to adapt to a site redesign without touching code.
+
+**Adding a new browser source** (4-step closed loop: add → inspect --write → verify → lock with regression tests):
 ```bash
 # 1. Add to config (browser sources should get a recipe — articleUrlPattern is legacy-only)
 node scripts/source-manage.js add --id mysrc --name "My Source" --url https://example.com/ --method browser --folder "My Source"
@@ -139,8 +145,38 @@ node scripts/inspect-source.js https://example.com/ --scroll 3 --write mysrc
 
 # 3. Verify (read-only)
 node scripts/harvest.js mysrc 3 --preview
+
+# 4. Lock against regressions (after the recipe is stable)
+node scripts/recipe-test.js mysrc   # needs helpers/mysrc.fixtures.json — see below
+```
+
+**Convergence check (step 3)**: preview passes when the fetched links are real articles (not nav/section pages) and `charCount > 500`. Diagnose failures against the candidate count `harvest.js` prints (`Collected N candidate links`):
+- **Too few candidates** → the `listSelector` is too narrow. Run `inspect` (without `--write`) and check whether articles live in *multiple* card-container classes. If so, use a **composite `listSelector`** (see pitfall below).
+- **Candidates are nav/section pages** → either tighten `urlPattern`, or (preferred for sites where article and section URLs share a shape) set `validateArticle: true` and let `fetchPage` reject section pages by `og:type`/body.
+- **Empty body** → fix `bodySelectors`.
+- **0 candidates, all `/location/`-style nav links** → `listSelector: null` (whole page) is almost always wrong; nav dropdowns flood the `maxLinks` cap before real articles are reached. Scope to card containers instead.
+Re-run preview after each edit. **Do not hand-edit the recipe blindly** — re-run `inspect` (without `--write`) to see the container groups if unsure.
+
+#### Pitfall: articles spread across multiple card containers
+`inspect-source.js` reports the *single largest* container group as `listSelector`, but on many sites (Nikkei, large news hubs) real articles are spread across several card-container classes of similar size, and no one container holds them all. Setting `listSelector` to just the biggest group silently drops the rest.
+
+Symptom: `inspect` shows ~30 article-like links on the page but `harvest --preview` collects only ~12. Fix: write `listSelector` as a **comma-separated composite** of every article-card class prefix, e.g. `[class*="StreamArticleCard"], [class*="SpotlightArticleCard"], [class*="SecondaryArticleCard"]`. Re-run `inspect` to enumerate which `*ArticleCard*` classes carry links if unsure.
+
+#### When to use `validateArticle` (and when not to)
+Set `validateArticle: true` when article and section/category URLs on the site **share a path shape** — i.e. you can't write a `urlPattern` that admits articles and rejects section pages purely from the URL. Nikkei is the canonical case: `/business/tech/semiconductors/<slug>` (article) and `/business/tech/semiconductors` (section) differ only by the trailing slug, and slug length is an unreliable signal. With `validateArticle`, `fetchPage` marks the section page `isArticle=false` (via `og:type=website` / `@type=Thing` / empty body) and `harvest.js` skips it.
+
+Don't set it when the URL alone cleanly separates articles from sections (e.g. Reuters' `/world/<slug>/<date>/` vs. `/world/`) — a tight `urlPattern` is cheaper than a per-candidate page fetch. `validateArticle` defaults to `false`, so existing recipes are unaffected.
+
+#### Locking a stable recipe with regression tests (step 4)
+Once a recipe converges, capture a fixture set so a future site redesign is caught instead of silently dropping articles. Create `helpers/<id>.fixtures.json`:
+```json
+{
+  "articles": ["https://.../real-article-1", "https://.../real-article-2"],
+  "sections": ["https://.../section-page-1", "https://.../section-page-2"],
+  "minLinks": 10
+}
 ```
-**Convergence check**: preview passes when the fetched links are real articles (not nav/section pages) and `charCount > 500`. If it fails — links are nav/section → tighten `urlPattern` in `helpers/mysrc.md`; too few links → raise `scrollSteps`; empty body → fix `bodySelectors`. Re-run preview after each edit. **Do not hand-edit the recipe blindly** — re-run `inspect` (without `--write`) to see the container groups if unsure.
+Then `node scripts/recipe-test.js <id>` asserts: every article URL fetches `isArticle=true` with `bodyChars ≥ bodyMinChars`; every section URL fetches `isArticle=false`; the homepage yields ≥ `minLinks` and zero `excludeUrlPattern` matches. Pick fixture URLs across the path depths the site uses (2-seg and 3-seg) so depth regressions surface. When the site archives a fixture article, swap in a current one from the same path shape.
 
 **Once verified, archive** by following Flow A below (harvest writes originals → you write `.summaries.json` → finalize). The harvest stderr prints the `Day dir` and the exact next command — follow that, don't parse the stdout manifest.
 

references/source-management.md

@@ -102,7 +102,8 @@ All fields optional except `urlPattern`; defaults reproduce legacy behaviour.
 | `linkSelector` | `a[href]` | Candidate links inside each container |
 | `urlPattern` | **required** | Regex source; an href must match to count as an article |
 | `excludeUrlPattern` | `null` | Regex source; drop matching hrefs (section pages, podcasts, etc.) |
-| `scrollSteps` | `0` | Bottom-scroll iterations to trigger lazy-loaded lists |
+| `scrollSteps` | `0` | Bottom-scroll iterations to trigger lazy-loaded lists (fixed-steps mode; used by `--preview` and when `maxScrollSteps` doesn't apply) |
+| `maxScrollSteps` | `12` | Scroll ceiling in target-count mode (archive harvest). When the harvest needs more candidates than a fixed pass yields, it keeps scrolling up to this many iterations, exiting early once enough candidates are collected or the stream goes stale (3 consecutive scrolls add nothing) |
 | `scrollWaitMs` | `1500` | Pause after each scroll |
 | `waitMs` | `12000` | Initial render wait after load |
 | `maxLinks` | `30` | Link cap |
@@ -113,6 +114,8 @@ All fields optional except `urlPattern`; defaults reproduce legacy behaviour.
 |-------|---------|---------|
 | `bodySelectors` | (6 selectors) | Tried in order; first yielding ≥ `bodyMinParagraphs` wins |
 | `bodyMinParagraphs` | `3` | Min paragraph count for a selector to qualify |
+| `validateArticle` | `false` | When `true`, `fetchPage` probes `og:type` + JSON-LD `@type` + body length and returns `isArticle`; `harvest.js` skips candidates where `isArticle === false`. Use with a *broad* `urlPattern` (so candidate URLs aren't pre-filtered by fragile slug heuristics) to reject section/category pages that slip through once the page is actually fetched. Off by default → existing recipes unchanged. See `helpers/nikkei-tech.md` for a worked example. |
+| `bodyMinChars` | `500` | Floor on body length for `isArticle` (only consulted when `validateArticle: true`) |
 | `dateSelector` | `time[datetime]` | Falls back to JSON-LD `datePublished` |
 | `authorSelector` | `[rel="author"], [class*="author"] a` | Falls back to JSON-LD `author.name` |
 | `imageSelector` | `img` | |

scripts/fetcher-helper.js

@@ -51,6 +51,14 @@ const DEFAULTS = {
     'article p'
   ],
   bodyMinParagraphs: 3,        // a selector must yield ≥ this many nodes to "win"
+  // Article validation: when validateArticle is true, fetchPage additionally
+  // probes og:type + JSON-LD @type + body length and returns `isArticle`. A
+  // recipe with a broad urlPattern (so candidate URLs aren't pre-filtered by
+  // fragile slug heuristics) sets this to reject section/category pages that
+  // slip through the URL filter once the page is actually fetched. Off by
+  // default → existing recipes keep their original behaviour.
+  validateArticle: false,
+  bodyMinChars: 500,
   dateSelector: 'time[datetime]',
   authorSelector: '[rel="author"], [class*="author"] a',
   imageSelector: 'img',
@@ -124,6 +132,11 @@ function buildExtractExpr(spec) {
   const noisePrefixes = JSON.stringify(spec.noisePrefixes);
   const titleStripRe = JSON.stringify(spec.titleStripSuffix);
   const maxBodyChars = spec.maxBodyChars;
+  // Article-validation knobs (see DEFAULTS). When validateArticle is false the
+  // isArticle signal is always true and the og:type/JSON-LD probes are skipped,
+  // so non-recipe / non-validating sources keep their original behaviour.
+  const validateArticle = !!spec.validateArticle;
+  const bodyMinChars = spec.bodyMinChars || 500;
   return `(function(){
     var title = document.title.replace(new RegExp(${titleStripRe}, 'g'), '').trim();
     // JSON-LD fallback: many sites (Nikkei, Reuters) embed datePublished/author
@@ -163,23 +176,48 @@ function buildExtractExpr(spec) {
         && !${excludeImage}.some(function(x){ return i.src.toLowerCase().indexOf(x) !== -1; });
     }).slice(0, 6);
     var body = '';
+    var bodyParagraphs = 0;
     var sels = ${bodySelectors};
     for (var s = 0; s < sels.length; s++) {
       var els = document.querySelectorAll(sels[s]);
       if (els.length >= ${bodyMinParagraphs}) {
+        bodyParagraphs = els.length;
         body = Array.from(els).map(function(e){ return e.innerText.trim(); })
           .filter(function(t){ return t.length > 40; }).join('\\n\\n');
         break;
       }
     }
     if (!body) {
-      body = Array.from(document.querySelectorAll('p'))
+      var fallback = Array.from(document.querySelectorAll('p'))
         .map(function(e){ return e.innerText.trim(); })
         .filter(function(t){ return t.length > 60
-          && !${noisePrefixes}.some(function(n){ return t.indexOf(n) === 0; }); })
-        .join('\\n\\n');
+          && !${noisePrefixes}.some(function(n){ return t.indexOf(n) === 0; }); });
+      bodyParagraphs = fallback.length;
+      body = fallback.join('\\n\\n');
     }
-    return JSON.stringify({ title: title, date: date, authors: authors, imgs: imgs, body: body.substring(0, ${maxBodyChars}) });
+    // Article-validation signal: og:type + JSON-LD @type + body length. A
+    // section/category page typically has og:type=website, @type=Thing, and
+    // zero body paragraphs, so this cleanly separates real articles from the
+    // candidate URLs a widened urlPattern lets through on list pages. When
+    // validateArticle is off the probe cost is avoided and isArticle is true.
+    var ogType = '', ldType = '', isArticle = true;
+    if (${validateArticle ? 'true' : 'false'}) {
+      var ogEl = document.querySelector('meta[property="og:type"]');
+      if (ogEl) ogType = ogEl.content || '';
+      if (ld) {
+        var t = ldGet(ld, '@type');
+        ldType = Array.isArray(t) ? t.join(',') : (t || '');
+      }
+      isArticle = (ogType === 'article' || /NewsArticle|Article/.test(ldType))
+        && body.length >= ${bodyMinChars}
+        && bodyParagraphs >= ${bodyMinParagraphs};
+    }
+    return JSON.stringify({
+      title: title, date: date, authors: authors, imgs: imgs,
+      body: body.substring(0, ${maxBodyChars}),
+      isArticle: isArticle, ogType: ogType, ldType: ldType,
+      bodyChars: body.length, bodyParagraphs: bodyParagraphs
+    });
   })()`;
 }
 

scripts/harvest.js

@@ -472,6 +472,11 @@ Examples:
         continue;
       }
       if (!articleData || !articleData.title) { console.error('  ✗ No title, skip'); continue; }
+      // Recipe article-validation: a broad urlPattern may let section/category
+      // pages through as candidate links; fetchPage marks them isArticle=false
+      // (via og:type / JSON-LD / body-length probes). Non-validating recipes
+      // always return isArticle=true, so this is a no-op for them.
+      if (articleData.isArticle === false) { console.error('  ✗ Not an article (og:type/body), skip'); continue; }
       const category = getCategory(url, articleData.title);
       const entry = {
         title: articleData.title,
@@ -527,7 +532,12 @@ Examples:
       continue;
     }
 
-    if (!articleData || !articleData.title || (articleData.body || '').length < 100) {
+    if (!articleData || !articleData.title) { console.error('  ✗ No title, skip'); continue; }
+    // Recipe article-validation: reject section/category pages that slipped
+    // through a broad urlPattern (fetchPage set isArticle=false via og:type /
+    // JSON-LD / body-length probes). No-op for non-validating recipes.
+    if (articleData.isArticle === false) { console.error('  ✗ Not an article (og:type/body), skip'); continue; }
+    if ((articleData.body || '').length < 100) {
       console.error(`  ✗ Insufficient content (body: ${(articleData?.body||'').length} chars)`);
       continue;
     }

scripts/recipe-test.js

+#!/usr/bin/env node
+/**
+ * recipe-test.js — Regression tests for a source's helper.md recipe.
+ *
+ * Guards against two classes of breakage when a site redesigns:
+ *   1. Article URLs that used to fetch as real articles (og:type=article +
+ *      real body) start getting rejected, or section pages start passing —
+ *      i.e. the article/section boundary drifted.
+ *   2. The list page stops yielding enough candidate links, or lets through
+ *      URLs the excludeUrlPattern is supposed to drop.
+ *
+ * A fixture set lives in helpers/<id>.fixtures.json (articles[], sections[],
+ * minLinks). Each article URL must fetch with isArticle=true and bodyChars ≥
+ * a floor; each section URL must fetch with isArticle=false; the homepage
+ * must yield ≥ minLinks and zero excludeUrlPattern matches.
+ *
+ * Usage:
+ *   node scripts/recipe-test.js <helperId>
+ *
+ * Exit 0 if all assertions pass, 1 otherwise.
+ */
+
+const fs = require('fs');
+const path = require('path');
+
+const SKILL_DIR = path.dirname(__dirname);
+const CONFIG_PATH = path.join(SKILL_DIR, 'config.json');
+const TEMPLATE_PATH = path.join(SKILL_DIR, 'references', 'config-template.json');
+
+function loadConfig() {
+  const cfgPath = fs.existsSync(CONFIG_PATH) ? CONFIG_PATH : TEMPLATE_PATH;
+  return JSON.parse(fs.readFileSync(cfgPath, 'utf8'));
+}
+
+const { loadSpec, fetchPage, fetchHomeLinks } = require('./fetcher-helper');
+
+// ── Assertions ───────────────────────────────────────────────────────────────
+
+let passed = 0, failed = 0;
+function assert(cond, label, detail) {
+  if (cond) {
+    passed++;
+    console.log(`  ✓ ${label}`);
+  } else {
+    failed++;
+    console.log(`  ✗ ${label}${detail ? ' — ' + detail : ''}`);
+  }
+}
+
+// ── Main ─────────────────────────────────────────────────────────────────────
+
+async function main() {
+  const helperId = process.argv[2];
+  if (!helperId) {
+    console.error('Usage: node scripts/recipe-test.js <helperId>');
+    console.error('Example: node scripts/recipe-test.js nikkei-tech');
+    process.exit(1);
+  }
+
+  const helperPath = path.join(SKILL_DIR, 'helpers', `${helperId}.md`);
+  const fixturesPath = path.join(SKILL_DIR, 'helpers', `${helperId}.fixtures.json`);
+  if (!fs.existsSync(helperPath)) {
+    console.error(`✗ No recipe found: helpers/${helperId}.md`);
+    process.exit(1);
+  }
+  if (!fs.existsSync(fixturesPath)) {
+    console.error(`✗ No fixtures found: helpers/${helperId}.fixtures.json`);
+    console.error(`  Create one with: { "articles": [...], "sections": [...], "minLinks": N }`);
+    process.exit(1);
+  }
+
+  const config = loadConfig();
+  const source = config.sources.find(s => s.id === helperId);
+  if (!source) {
+    console.error(`✗ Source "${helperId}" not in config.json`);
+    process.exit(1);
+  }
+
+  const spec = loadSpec(helperPath);
+  const fixtures = JSON.parse(fs.readFileSync(fixturesPath, 'utf8'));
+
+  console.error(`🧪 recipe-test: ${helperId} (${source.homepageUrl})`);
+  console.error(`   validateArticle=${spec.validateArticle}, urlPattern=${spec.urlPattern}`);
+  console.error(`   Ensuring Chromium (CDP)...`);
+
+  const { ensureChromium } = require('./ensure-chromium');
+  try {
+    const r = await ensureChromium(config);
+    console.error(`   CDP ready on :${r.port}${r.alreadyRunning ? ' (already running)' : ` (pid ${r.pid})`}`);
+  } catch (e) {
+    console.error(`   Chromium startup failed:\n${e.message}`);
+    process.exit(1);
+  }
+
+  // ── 1. Article URLs must validate as articles ─────────────────────────────
+  console.error(`\n📄 Article fixtures (must be isArticle=true, body ≥ ${spec.bodyMinChars || 500} chars):`);
+  for (const url of (fixtures.articles || [])) {
+    try {
+      const data = await fetchPage(url, spec);
+      if (!data) { assert(false, `article: ${url}`, 'fetchPage returned null'); continue; }
+      assert(data.isArticle !== false, `article: ${data.title ? data.title.slice(0,60) : url}`,
+        `isArticle=${data.isArticle} og=${data.ogType} ld=${data.ldType} body=${data.bodyChars}c/${data.bodyParagraphs}p`);
+    } catch (e) {
+      assert(false, `article: ${url}`, e.message);
+    }
+  }
+
+  // ── 2. Section/category URLs must NOT validate as articles ────────────────
+  // Only meaningful when the recipe uses validateArticle; otherwise this group
+  // is skipped (no isArticle signal to assert against).
+  if (spec.validateArticle) {
+    console.error(`\n🗂️  Section fixtures (must be isArticle=false):`);
+    for (const url of (fixtures.sections || [])) {
+      try {
+        const data = await fetchPage(url, spec);
+        if (!data) { assert(false, `section: ${url}`, 'fetchPage returned null'); continue; }
+        assert(data.isArticle === false, `section: ${url}`,
+          `isArticle=${data.isArticle} og=${data.ogType} ld=${data.ldType} body=${data.bodyChars}c/${data.bodyParagraphs}p`);
+      } catch (e) {
+        assert(false, `section: ${url}`, e.message);
+      }
+    }
+  }
+
+  // ── 3. Homepage link collection: count + no excluded URLs ─────────────────
+  console.error(`\n🔗 Homepage links (≥ ${fixtures.minLinks || 10}, none matching excludeUrlPattern):`);
+  try {
+    const links = await fetchHomeLinks(source.homepageUrl, spec);
+    assert(links.length >= (fixtures.minLinks || 10),
+      `homepage yields ≥ ${fixtures.minLinks || 10} links`, `got ${links.length}`);
+    if (spec.excludeUrlPattern) {
+      const exRe = new RegExp(spec.excludeUrlPattern);
+      const leaked = links.filter(u => exRe.test(u));
+      assert(leaked.length === 0, `no links match excludeUrlPattern (${spec.excludeUrlPattern})`,
+        leaked.length ? leaked.slice(0,3).join(' ; ') : '');
+    }
+    // Also verify every link matches urlPattern (sanity: the recipe isn't
+    // letting unrelated hrefs through).
+    const urlRe = new RegExp(spec.urlPattern);
+    const offPattern = links.filter(u => !urlRe.test(u));
+    assert(offPattern.length === 0, 'all links match urlPattern',
+      offPattern.length ? offPattern.slice(0,3).join(' ; ') : '');
+  } catch (e) {
+    assert(false, 'homepage link collection', e.message);
+  }
+
+  // ── Summary ───────────────────────────────────────────────────────────────
+  console.error(`\n${failed === 0 ? '✅' : '❌'} ${passed} passed, ${failed} failed`);
+  process.exit(failed === 0 ? 0 : 1);
+}
+
+main().catch(e => { console.error('Fatal:', e.message); process.exit(1); });