-
郭阔 authored
Reuters /technology/ renders a fixed ~20-article list with no lazy-load or load-more — on high-volume days the window fully churns between two harvest runs, making gap articles structurally un-capturable from the DOM. Two recipe-gated additions to fetcher-helper: - sitemap discovery (sitemapUrl/sitemapFromStep/sitemapPages/sitemap- Include): archive-mode link collection reads Google-News sitemap pages via same-origin fetch() inside the site session (DataDome cookies apply), newest-first, section-prefixed so a site-wide sitemap cannot swallow sibling channels' articles. Any failure falls back to the DOM scan. Enabled for all four reuters channel recipes (machine-local). - datelineFallback: wire-service bodies (Reuters) render in wildly varying DOM structures, so selector lists go stale and articles look paywalled while the full text sits in innerText. When selectors under-produce (< bodyMinChars), slice innerText from the dateline lead ("AMSTERDAM, Nov 12 (Reuters) -") through the byline ("Reporting by"), stripping boilerplate lines — immune to markup churn. Also adds cdpEvalAsync (awaitPromise) for in-page fetch().90a72794
| Name |
Last commit
|
Last update |
|---|---|---|
| skills | ||
| .gitignore | ||
| README.md |