• 郭阔's avatar
    feat(news-harvester): sitemap candidate discovery + dateline body fallback · 90a72794
    郭阔 authored
    Reuters /technology/ renders a fixed ~20-article list with no lazy-load
    or load-more — on high-volume days the window fully churns between two
    harvest runs, making gap articles structurally un-capturable from the
    DOM. Two recipe-gated additions to fetcher-helper:
    
    - sitemap discovery (sitemapUrl/sitemapFromStep/sitemapPages/sitemap-
      Include): archive-mode link collection reads Google-News sitemap pages
      via same-origin fetch() inside the site session (DataDome cookies
      apply), newest-first, section-prefixed so a site-wide sitemap cannot
      swallow sibling channels' articles. Any failure falls back to the DOM
      scan. Enabled for all four reuters channel recipes (machine-local).
    - datelineFallback: wire-service bodies (Reuters) render in wildly
      varying DOM structures, so selector lists go stale and articles look
      paywalled while the full text sits in innerText. When selectors
      under-produce (< bodyMinChars), slice innerText from the dateline
      lead ("AMSTERDAM, Nov 12 (Reuters) -") through the byline
      ("Reporting by"), stripping boilerplate lines — immune to markup
      churn. Also adds cdpEvalAsync (awaitPromise) for in-page fetch().
    90a72794
Name
Last commit
Last update
..
news-harvester Loading commit data...
news-hub-feed Loading commit data...
news-source-analyzer Loading commit data...