- 13 Nov, 2026 1 commit
-
-
郭阔 authored
- refetch.js: fetch one article via its source recipe on the site-shared browser instance; refresh 原文.md in place for already-archived articles (registry ownership-checked, index rebuilt from registry) and re-push via push.js --url; new URLs archive to today's dir pending summaries. Busy site locks are respected; only self-launched instances are recycled. - push.js: --url <articleUrl> re-pushes a single article (query/hash normalized match, ledger ignored for it); recordResult now appends a ledger entry when the article postdates the ledger instead of silently dropping a successful push result (refetch-into-finalized-day case). - harvest.js: export downloadImage/normalizeUrl for refetch.
-
- 28 Oct, 2026 1 commit
-
-
郭阔 authored
-
- 27 Oct, 2026 1 commit
-
-
郭阔 authored
- login-tools.js: 无 steps 的检测型配方(reuters/wsj)在 login 命令下不再抛 'recipe.steps is not iterable',改为提示走面板人工登录(exit 6) - keepalive.js: 用站点共享实例访问一次首页刷新反爬(DataDome)cookie, 维持常驻访客画像;实例离线则拉起并在结束后回收(有采集锁时不回收); 检测撞墙写状态文件(NH_KEEPALIVE_STATUS),撞墙 exit 3
-
- 23 Oct, 2026 3 commits
-
-
郭阔 authored
Polls site login state for the panel's 人工登录 flow: opens the login page (PUT /json/new with Page.navigate fallback — this Chromium build ignores the ?url= param), probes via existing recipe or heuristics (visible sign-in elements absent + account-testid element present), and on first successful login can auto-generate a detection-only recipe (--write, unique data-testid candidates only). JSON line output; exit 0 logged-in, 3 timeout.
-
郭阔 authored
Detection-only (checkUrl reuters.com + loggedInSelector [data-testid=AccountButton], measured on a live logged-in session; no steps — reuters sign-in lives on account.reuters.com, manual login in the site instance window persists in the site-shared profile)
-
郭阔 authored
- logins/wsj.json: detection-only recipe (checkUrl + loggedOutSelector sso.accounts.wsj.com link, no steps — DataDome blocks auto-login); preflight now reports real wsj login state instead of 无登录配方 - scripts/instance-quit.js: idempotent browser instance recycle (close all tabs → Browser.close fallback); called by harvest.sh EXIT trap
-
- 22 Oct, 2026 4 commits
-
-
郭阔 authored
- recognize Auth0-style type=text username fields (not just type=email) - fill via native value setter so React controlled inputs register the value (Auth0 'email required' despite filled field) - bootstrap checkUrl tab when no site tab open before running steps - entry picker prefers exact 'Sign In'/'Log In' over long matches (was clicking 'Bloomberg Anywhere Remote Login') - iframe context matching skips the main frame first (same-host login iframes like bloomberg.com/account/iframe/signin were executing in the main frame and 'seeing' no form) - new site recipes: statista.json (Auth0 two-step, verified live), bloomberg.json (two-step; auto-login still needs trusted-click engine support, signin iframe swallows synthetic DOM clicks)
-
郭阔 authored
- login-tools: loadRecipe falls back to site recipe (logins/<site>.json); discover writes site-level recipe so nikkei-* channels share one recipe - rename recipes to site names: nikkei.json, digitimes.json (ft.json already)
-
郭阔 authored
- discover <source>: open homepage → find login entry by visible text (fallback to common login paths) → click/open → detect main-frame vs iframe form (captcha/challenge iframes excluded) → enumerate fields → generate recipe (single-page / two-step / password-only) with remember-me checkbox ticking → write logins/<source>.json → optional live login verification (--no-login to skip) - loggedOutText: login-state check by visible button/link text (regex), for sites whose login control has no stable class - honest boundaries: CAPTCHA/OTP/perimeter protection exits with a clear human-required message and writes nothing (validated against wsj's DataDome layer); validated green path against nikkei Piano ID
-
郭阔 authored
- engine: step.frame executes fill/submit in the matching iframe's execution context (cross-origin auth widgets, e.g. Piano ID) - engine: step.click {text} clicks a main-frame button by visible text (opens login modals); loggedOutSelector = logged-in when absent - logins/nikkei-world.json: Piano two-step form (email -> password in auth.asia.nikkei.com iframe), validated with a real login
-
- 21 Oct, 2026 4 commits
-
-
郭阔 authored
The only checkbox on join.asp is SaveID (remember email); extend the tick pattern with save|. Session TTL on this site is short and there is no keep-me-signed-in option, so preflight-before-harvest is the primary mechanism; form login frequency stays at most once per harvest.
-
郭阔 authored
- submit button now resolved within the filled field's form first (document-wide query hit the header search form's identical button[type=submit] — clicked Search instead of Next) - fill/checks/submit merged into one page-side step; checkbox ticking with label/name/id pattern filter (keep|remember|signed|记住) - tab tracking by target id (open→use→close), immune to stale/concurrent same-site tabs; verification opens its own tab and closes it - digitimes recipe: explicit Next button selector + keep-me-signed-in checkbox for long-lived session cookie
-
郭阔 authored
- check <source>: open checkUrl, assert loggedInSelector (exit 3 when logged out); login <source>: recipe-driven multi-step form flow with $USER/$PASS substitution from NH_LOGIN_* env, tab-relocation hardening against concurrent harvests, post-login verification - recipes live in logins/<source>.json; ships digitimes-feed.json - honors CDP_PORT_OVERRIDE / CDP_PROFILE_OVERRIDE (per-source instances)
-
郭阔 authored
Pair with CDP_PORT_OVERRIDE so parallel harvests can each drive a dedicated Chromium instance (own port + own user-data-dir).
-
- 11 Sep, 2026 1 commit
-
-
谢宇轩 authored
-
- 09 Sep, 2026 1 commit
-
-
谢宇轩 authored
-
- 08 Sep, 2026 2 commits
- 28 Aug, 2026 1 commit
-
-
谢宇轩 authored
- pack.js: --pattern (articleUrlPattern) for direct sources; recipe no longer required (and forbidden) for --method direct, since a recipe forces the browser path at harvest time - source-manage.js install: carry bundle articleUrlPattern into config - fetcher-direct.js: decode HTML entities in title/img alt - new wccftech-news direct bundle (server-rendered WordPress, no login, no Chromium); verified end-to-end via harvest --preview - docs: SKILL.md, recipe-guide.md, source-management.md updated
-
- 27 Aug, 2026 1 commit
-
-
谢宇轩 authored
-
- 21 Aug, 2026 2 commits
- 14 Aug, 2026 2 commits
- 13 Aug, 2026 5 commits
- 12 Aug, 2026 2 commits