--- a/scraper/README.md +++ b/scraper/README.md @@ -86,8 +86,11 @@ ## Failure protection and silent-blocking detection -A run is published only if every check passes. Otherwise its files go to `runs///` for diagnosis and -`latest//products.json` is not touched. +A run is published only if every check passes and it is a full run. Otherwise its files go to `runs///` +for diagnosis (a passed limited run also to `samples///`) and the published delivery is not touched. +v2.2.18: products, coupons, report and the store's observation history are published together by one writer +(`marianos_scraper/delivery.py`): a new release folder first, then one conditional write of +`latest//CURRENT.json`; a failure at any write leaves the previous delivery whole. | Check | Catches | |---|---| --- a/scraper/config/settings.json +++ b/scraper/config/settings.json @@ -65,6 +65,9 @@ "~", "\\bca\\.?\\s" ], + "collection": { + "deadline_seconds": 18000 + }, "site_source": { "cdp_url": "http://127.0.0.1:9444", "assortment_key": "22322f30-e01a-41fd-bebe-d1ccbcdb1692", @@ -77,6 +80,8 @@ "max_requests": null, "max_seconds": null, "max_requests_per_run": null, + "request_timeout_seconds": 90, + "request_reservation_block": 25, "oxylabs_geo_location": "United States", "oxylabs_sticky_ip": true, "oxylabs_timeout_seconds": 180, --- a/scraper/docs/AZURE_DEPLOY.md +++ b/scraper/docs/AZURE_DEPLOY.md @@ -84,8 +84,10 @@ ## Where the data is ``` -latest//products.json last run that passed validation (point the pipeline here) -latest//report.json its counts, completeness, validation result +latest//CURRENT.json pointer to the last FULL run that passed validation (point the pipeline here): + its release folder, the SHA-256 of each file, the release it replaced, the history +releases//// that delivery: products.json, coupons.json, report.json, catalog.json (never rewritten) +samples/// passed runs limited to some categories / a partial coupon lookup (never latest/) runs/// every run, passed or held back: products, report, availability, excluded, raw/, requests work//current.json state of the current/unfinished run state//catalog.json every product ever published with its last observation (carry-forward) @@ -122,4 +124,4 @@ | Run stopped with exit 5 | The run's request budget (`--max-requests-per-run`) is spent; nothing more was sent. Read the run log, then either start the run again with a higher budget (it resumes) or start over with `-FreshFirst` | | Oxylabs errors (exit 2) | 3 Oxylabs-side failures in a row, or the API user / payload rejected: see `oxylabs_failure` lines in the run log; not a site block | | Stuck lock after a crash | Lease older than 6 h is broken automatically; or delete the `work//run.lock` blob | -| Need an older dataset | Blob versioning: restore a previous version of `latest//products.json` | +| Need an older dataset | every earlier delivery stays in `releases//` (listed in the `history` of `latest//CURRENT.json`); Blob versioning keeps older pointer versions too | --- /dev/null +++ b/scraper/docs/CHANGES-2.2.18.md @@ -0,0 +1,165 @@ +# v2.2.18: corrections from the review of 2026-11-19 + +Offline only. No request was sent to marianos.com, Kroger or Oxylabs to build or test this version. The VM and the +schedules stayed off, and the remaining allowance was not used. The operator tools r3 and the v2.2.17 package stay +unchanged, and their published hashes still hold. v2.2.18 is a separate package. + +Tests: 164 pass (the 132 earlier tests plus 32 new ones in `tests/test_v2218.py`; same Python, no network). +I also ran a mutation check. Each v2.2.17 behaviour was put back one at a time, 18 in all, and in every case the test +for that finding failed. The script is `tools/mutants_2218.py`. + +## Finding, change, test + +### F1. A five-record Dairy run replaced a 145-record full delivery + +**Cause.** `--categories` only skipped the yield comparison. A partial run that passed its own checks was written to +`latest/` like a full run. + +**Change.** +- Only a full run can replace the delivery. A run counts as full only when it has no category subset, uses the full + coupon lookup and is not a fallback run. +- Any other run that passes is released as a sample under `samples///`, and `latest/` stays untouched. + The report shows this as `status: sample`, `delivery.kind: sample` and `delivery.limited_by`. +- The scope is read from the run's own recorded binding. A refinish without `--categories` therefore cannot turn a + sample into a full delivery. +- The API runner follows the same rule. +- Exit code 0 means published, or a passed sample. + +**Tests (`TestF1PartialRunNeverReplacesFull`).** +- `test_five_record_dairy_run_is_a_sample_and_the_145_record_delivery_stays` is your case: a 145-record full run, + then a 5-record Dairy run. The pointer and all four delivery files are byte-identical before and after. +- `test_refinish_judges_a_run_by_its_own_scope` +- `test_api_runner_category_subset_is_a_sample_too` + +### F2. Missing totals and overlapping category counts could look complete + +**Cause.** +- When the site omitted `totalCount`, `search()` used the number of items on the page, so the unit looked complete. +- Category coverage added up the per-unit counts. A product listed by two units of the same category (a node and its + child, or two aisles) was counted twice. +- Coupons were not checked against the site's totals. + +**Change.** +- A missing total is recorded as missing (`api_total: null`, `site_total_reported: false`). A missing root or node + count stops planning. A missing unit count fails the new check `site_count_reported_for_every_unit`. +- Category coverage counts distinct product ids. The report shows both `ids_collected` (distinct) and + `ids_listed_incl_overlap`. +- Coupons have two new checks: + - `coupons_complete_vs_site_totals`: for each coupon type (standard, cash back), the distinct coupons collected must + reach the site's own `totalCount`. The paged list re-sorts between pages, so it is read up to three times and the + passes are combined by coupon id. A missing total fails the check. + - `coupon_lookup_covered_every_product`: every distinct product must have been asked for its coupons. + +**Tests (`TestF2CompletenessIsProven`).** +- `test_a_listing_without_the_site_total_is_not_complete` +- `test_overlapping_units_are_counted_once`: two aisles, each listing the same 5 products, with a site count of 10. + Coverage is 0.5, so the run is held back. Before this change the sum gave 10/10. +- `test_coupons_short_of_the_site_total_are_read_again_then_held_back` +- `test_coupons_without_a_site_total_are_not_complete` +- `test_every_product_must_have_been_asked_for_its_coupons` + +### F3. A checkpoint could resume under another store + +**Cause.** The run state, the plan and the unit checkpoints were keyed only by the store key. An API run, a changed +store configuration or a different transport would resume them. + +**Change.** +- Every run records a binding: source, transport, store key, location id, facility id, ZIP, chain, scope and coupon + lookup scope. The binding, or its digest, is stored in `current.json`, `plan.json` and every unit checkpoint. +- A run resumes only under exactly the same binding, and a checkpoint is used only if its digest matches. Otherwise + `CheckpointMismatch` is raised: nothing is sent, the state is untouched, and the exit code is 6. The wrappers do + not retry exit 6; a person resumes with the same settings or starts with `--fresh`. +- On resume, the store the site answers for (location id, facility, ZIP) must also equal the store recorded at the + first start. +- Runs started by v2.2.17 have no binding, so they are not resumed. Start them over with `--fresh`. + +**Tests (`TestF3CheckpointBinding`).** +- `test_an_unfinished_run_never_resumes_under_another_store`: nothing is sent and the state is byte-identical. +- `test_a_store_that_answers_as_another_store_on_resume_is_refused` +- `test_another_source_never_resumes_a_website_run` +- `test_a_foreign_checkpoint_file_is_not_used` + +### F4. A failed coupon write left new products with the old report + +**Cause.** `latest/products.json`, `coupons.json` and `report.json`, and `state/catalog.json`, were four separate +writes. + +**Change.** There is now one writer, `marianos_scraper/delivery.py`. +1. Products, coupons, report and catalog (the store's observation history) are written to a new folder, + `releases////`, which is never reused. Each file is read back. +2. Only then is the single pointer `latest//CURRENT.json` replaced, in one conditional write. Locally the + condition is the SHA-256 of the pointer that was read; on Blob it is If-Match on its ETag. The pointer names the + folder, the SHA-256 of each file, the release it replaced and the release history. + +A failure at any step leaves the pointer on the previous release, whose files were never touched. Readers go through +the pointer and check each hash, so a mixed delivery cannot be read. A failed publish reports `status: publish_failed` +and exits with code 1. + +Stores written by v2.2.17 are read from their old files as the baseline until the first v2.2.18 publish. The old +files are never written again. + +**Tests (`TestF4OneWriterAtomicDelivery`).** +- `test_a_failed_write_leaves_the_previous_delivery_whole`: your case, a coupon write that fails, plus the same with + `report.json` and `catalog.json`. The pointer and all four files are byte-identical to the previous delivery. +- `test_products_coupons_report_and_history_move_together` +- `test_a_second_writer_cannot_overwrite_the_pointer` +- `test_a_changed_release_file_is_never_read` +- `test_publish_refuses_a_partial_file_set` + +### F5. Requests are now reserved durably before they are sent + +**Cause.** The run's request count was saved only after each finished unit. A killed process lost the count of the +requests in the unit it was working on. + +**Change.** +- The transport calls a reservation hook immediately before it counts and sends each attempt (`SiteSession.reserve`). + The run reserves requests in blocks in `current.json` (`site_requests_reserved`, `request_reservation_block`, + default 25) and reads the write back. A reservation never goes beyond `--max-requests-per-run`. If the reservation + cannot be written or read back, nothing is sent and the run stops resumably. +- After a clean stop the exact count replaces the reservation. After a crash the whole reservation counts as sent + (`requests_counted_from_reservation`). +- `--transport oxylabs` collection now needs `--ledger`: the jobs are reserved in the same shared allowance ledger + the probe uses, with the same lease, If-Match and read-back, before the first job is sent. Without `--ledger` + nothing is built or sent (exit 2). The wrappers have new options: `-Ledger` and `MARIANOS_LEDGER`. + +**Tests (`TestF5DurableReservation`).** +- `test_every_request_is_reserved_in_the_run_state_before_it_is_sent` +- `test_a_killed_process_is_counted_at_its_full_reservation_and_the_budget_holds` +- `test_no_request_is_sent_when_the_reservation_cannot_be_written` +- `test_the_real_transport_reserves_before_it_counts_and_sends` +- `test_oxylabs_collection_without_the_allowance_ledger_sends_nothing` +- `test_oxylabs_collection_reserves_jobs_in_the_ledger_before_the_first_one` + +### F6. Normal collection now has a deadline, not only the probe + +**Cause.** +- Without `--max-seconds`, a collection process had no runtime cap. +- The browser `fetch()` was sent with no time limit. +- The retry pause between passes was a fixed 30 s sleep. + +**Change.** +- Every collection process gets `collection.deadline_seconds` (default 18,000 s = 5 h; `--max-seconds` can only + shorten it). The value must stay below the 6 h stale-lock age, so a live run's lock never looks stale. +- Every request has its own timeout (`request_timeout_seconds`, default 90 s), whether or not a cap is set. +- The deadline is checked before every unit, and the retry pause stops at it. Either way the run stops resumably + (exit 4). +- A watchdog stops a process that is still running 300 s after its deadline (exit 4; the stuck request is already + inside a reservation). + +**Tests (`TestF6CollectionDeadline`).** +- `test_normal_collection_gets_the_deadline_without_any_option` +- `test_every_request_has_a_timeout_even_without_a_runtime_cap` +- `test_a_site_run_stops_resumably_at_the_deadline`: also resumes the same run afterwards. +- `test_an_api_run_stops_at_the_deadline_between_units` +- `test_the_retry_wait_stops_at_the_deadline` +- `test_a_stuck_call_is_ended_by_the_watchdog` +- `test_the_deadline_stays_below_the_stale_lock_age` + +## Limits stated plainly + +- All of this is tested offline with doubles of the site, storage and ledger. Live behaviour will only show on an + approved run. +- A live run whose coupon list never reaches the site's `totalCount` after three passes is now held back, by design. +- An API fallback run is always a sample now. The setting `publish_fallback_runs` can no longer put one into + `latest/`. +- The probe and the operator tools are unchanged. --- a/scraper/docs/OPERATIONS_MANUAL.md +++ b/scraper/docs/OPERATIONS_MANUAL.md @@ -8,9 +8,13 @@ `python -m marianos_scraper run` collects, per configured store, the 22-field records from marianos.com's own JSON endpoints, twice a week. A run verifies the store from the site's locator, checks canary products, works through -units (one retailer taxonomy node each, checkpointed), verifies the store again, validates, and only then replaces -`latest//products.json`. A run that stops for any reason resumes from its checkpoints on the next start (within -48 hours). Requests leave either through a desktop Chrome session (`--transport site`, the default) or through the +units (one retailer taxonomy node each, checkpointed), verifies the store again, validates, and only then publishes +the delivery: products, coupons, report and the store's observation history are written to a new release folder and +`latest//CURRENT.json` is switched to it in one conditional write (v2.2.18). A run limited to some categories +or to a partial coupon lookup is only a sample release (`samples///`) and never replaces the full +delivery. A run that stops for any reason resumes from its checkpoints on the next start (within 48 hours), and only +under the same source, transport, store and scope. Every collection process has a deadline +(`collection.deadline_seconds`, 5 h by default; every request also has its own timeout). Requests leave either through a desktop Chrome session (`--transport site`, the default) or through the Oxylabs Web Scraper API (`--transport oxylabs`). Everything after the transport is the same code. ## 2. Setup @@ -91,11 +95,12 @@ | What | Where | |---|---| -| Current data for the pipeline | `latest//products.json` (only runs that passed every check) | +| Current data for the pipeline | `latest//CURRENT.json` names the release folder and the SHA-256 of `products.json`, `coupons.json`, `report.json`, `catalog.json` in it (only full runs that passed every check). Read the files from that folder and check the hashes | +| Sample (limited) runs | `samples///` (never latest/) | | Why a run was or was not published | `runs///report.json` -> `status`, `validation.checks` | | Every request of a run | `runs///requests.jsonl` | | Unavailable, carried-forward, excluded products | `availability.jsonl`, `excluded.jsonl` in the run folder | -| State of an unfinished run | `work//current.json` (`site_requests_used` = requests so far, across resumes) | +| State of an unfinished run | `work//current.json` (`binding` = source/transport/store/scope; `site_requests_used` = requests so far, across resumes; `site_requests_reserved` = requests reserved before they were sent) | | Attempt history on the VM | `logs\scheduler.log`, `logs\run-*.log` (also uploaded to Blob `logs/`) | | Log Analytics | `Event | where Source == "ShifullahScraper"` (one event per attempt: id 1001 published, 1002 held back, 1003 aborted, 1005 stopped/resumable, 1006 budget spent) | @@ -110,12 +115,13 @@ | Code | Meaning | Retried by the wrappers? | |---|---|---| -| 0 | published | no | -| 1 | finished but held back by validation; `latest/` unchanged | no: a person reads `report.json` | +| 0 | published, or a passed run released as a sample | no | +| 1 | finished but held back by validation, or the publish failed; `latest/` unchanged | no: a person reads `report.json` | | 2 | aborted (store check failed, source error, Oxylabs credentials or payload rejected) | yes, up to the attempt limit | | 3 | another run of the same store is active (lock) | no | -| 4 | stopped and resumable (site refused, request/runtime cap of that process reached, stop requested) | yes | -| 5 | the run's request budget (`--max-requests-per-run`) is spent; nothing more was sent | no: a person decides | +| 4 | stopped and resumable (site refused, request/runtime cap or deadline of that process reached, stop requested) | yes | +| 5 | the run's request budget (`--max-requests-per-run`) or the shared Oxylabs allowance is spent, or the ledger refused; nothing more was sent | no: a person decides | +| 6 | the unfinished run (or a checkpoint) belongs to another source, transport, store or scope; nothing was sent | no: resume with the same settings or start over with `--fresh` | ### 3.4 Run now / stop / start over @@ -141,7 +147,7 @@ | Run budget spent (exit 5) | nothing more was sent | read the run log; resume with a higher `-MaxRequestsPerRun`, or `-FreshFirst` | | Credentials rotated | | `az keyvault secret set --vault-name -n --value `; the next run reads it | | Stuck lock after a crash | a lease older than 6 h is broken automatically | or delete `work//run.lock` | -| Need an older dataset | Blob versioning keeps every version | restore a previous version of `latest//products.json` | +| Need an older dataset | every earlier delivery stays in `releases//` | pick it from the `history` of `latest//CURRENT.json` | ## 5. Changes --- a/scraper/infra/azure_vm_run.ps1 +++ b/scraper/infra/azure_vm_run.ps1 @@ -24,7 +24,8 @@ [int] $MaxAttempts = 12, [int] $WaitMinutes = 20, [ValidateSet("", "site", "oxylabs")] [string] $Transport = "", - [int] $MaxRequestsPerRun = 0 + [int] $MaxRequestsPerRun = 0, + [string] $Ledger = "" # v2.2.18: required with -Transport oxylabs (blob: of the shared allowance ledger) ) $ErrorActionPreference = "Stop" $root = Split-Path -Parent $PSScriptRoot @@ -93,6 +94,7 @@ if ($StopAfterUnits -gt 0 -and $i -eq 1) { $extra += @("--stop-after-units", "$StopAfterUnits") } if ($Transport) { $extra += @("--transport", $Transport) } if ($MaxRequestsPerRun -gt 0) { $extra += @("--max-requests-per-run", "$MaxRequestsPerRun") } + if ($Ledger) { $extra += @("--ledger", $Ledger) } python -m marianos_scraper run --all @extra *> $log $code = $LASTEXITCODE $blob = Save-LogToBlob $log --- a/scraper/infra/run_until_done.ps1 +++ b/scraper/infra/run_until_done.ps1 @@ -6,11 +6,15 @@ -Transport oxylabs sends the requests through the Oxylabs Web Scraper API (OXYLABS_USERNAME / OXYLABS_PASSWORD must be in this process's environment; Chrome is not needed). -MaxRequestsPerRun N caps the whole run across all attempts. - Exit codes of `python -m marianos_scraper run`: 0 published, 1 held back, 2 aborted, 3 another run active, 4 stopped (resumable), - 5 run request budget spent (not retried). + -Ledger is required with -Transport oxylabs (v2.2.18): every Oxylabs job is reserved in the shared + allowance ledger before it is sent. + + Exit codes of `python -m marianos_scraper run`: 0 published (or a passed sample release), 1 held back / publish failed, + 2 aborted, 3 another run active, 4 stopped or deadline reached (resumable), 5 budget/allowance spent (not retried), + 6 checkpoint belongs to another source/store/scope (not retried). #> param([int] $MaxAttempts = 12, [int] $WaitMinutes = 20, [switch] $FreshFirst, [ValidateSet("", "site", "oxylabs")] [string] $Transport = "", - [int] $MaxRequestsPerRun = 0) + [int] $MaxRequestsPerRun = 0, [string] $Ledger = "") $root = Split-Path -Parent $PSScriptRoot Set-Location $root New-Item -ItemType Directory -Force (Join-Path $root "logs") | Out-Null @@ -29,6 +33,7 @@ $extra = @(); if ($FreshFirst -and $i -eq 1) { $extra = @("--fresh") } # start a new run instead of resuming if ($Transport) { $extra += @("--transport", $Transport) } if ($MaxRequestsPerRun -gt 0) { $extra += @("--max-requests-per-run", "$MaxRequestsPerRun") } + if ($Ledger) { $extra += @("--ledger", $Ledger) } python -m marianos_scraper run --all @extra *> $log $code = $LASTEXITCODE Add-Content (Join-Path $root "logs\scheduler.log") "$(Get-Date -Format s) attempt $i exit $code log $log" --- a/scraper/infra/vm/run_until_done.sh +++ b/scraper/infra/vm/run_until_done.sh @@ -1,15 +1,17 @@ #!/bin/bash # Linux twin of infra/run_until_done.ps1, started by marianos-run.service (systemd timer, Monday and Thursday). # Exit codes of the scraper: 0 published, 1 held back, 2 aborted (store check / source failure), 3 another run active, -# 4 stopped and resumable (site refused), 5 run request budget spent. Only 2 and 4 are retried after a pause. +# 4 stopped and resumable (site refused, deadline), 5 budget/allowance spent, 6 checkpoint of another source/store/scope. +# Only 2 and 4 are retried after a pause. # run_until_done.sh [--fresh-first] [max_attempts] [wait_minutes] # Optional environment (e.g. in the systemd unit): MARIANOS_TRANSPORT=oxylabs sends requests through the Oxylabs Web # Scraper API (API user from the Key Vault secrets oxylabs-username / oxylabs-password, no Chrome needed); -# MARIANOS_MAX_REQUESTS_PER_RUN=N caps the whole run across all attempts. +# MARIANOS_MAX_REQUESTS_PER_RUN=N caps the whole run across all attempts; MARIANOS_LEDGER=blob: (required with +# oxylabs, v2.2.18) reserves every Oxylabs job in the shared allowance ledger before it is sent. set -uo pipefail FRESH=""; if [ "${1:-}" = "--fresh-first" ]; then FRESH="--fresh"; shift; fi MAX_ATTEMPTS="${1:-12}"; WAIT_MINUTES="${2:-20}" -TRANSPORT="${MARIANOS_TRANSPORT:-}"; RUN_BUDGET="${MARIANOS_MAX_REQUESTS_PER_RUN:-}" +TRANSPORT="${MARIANOS_TRANSPORT:-}"; RUN_BUDGET="${MARIANOS_MAX_REQUESTS_PER_RUN:-}"; LEDGER="${MARIANOS_LEDGER:-}" APP=/var/lib/marianos/app/scraper; VENV=/var/lib/marianos/venv cd "$APP" @@ -40,6 +42,7 @@ args=(run --all); if [ "$i" = 1 ] && [ -n "$FRESH" ]; then args+=("$FRESH"); fi if [ -n "$TRANSPORT" ]; then args+=(--transport "$TRANSPORT"); fi if [ -n "$RUN_BUDGET" ]; then args+=(--max-requests-per-run "$RUN_BUDGET"); fi + if [ -n "$LEDGER" ]; then args+=(--ledger "$LEDGER"); fi "$VENV/bin/python" -m marianos_scraper "${args[@]}" rc=$? logger -t marianos-scraper "attempt $i exit $rc" --- a/scraper/marianos_scraper/__init__.py +++ b/scraper/marianos_scraper/__init__.py @@ -1,2 +1,2 @@ """Mariano's location-level product, price, PPM and deal collector (marianos.com, Kroger Public API fallback), built for scheduled runs on Azure.""" -__version__ = "2.2.17" +__version__ = "2.2.18" --- a/scraper/marianos_scraper/__main__.py +++ b/scraper/marianos_scraper/__main__.py @@ -9,6 +9,11 @@ python -m marianos_scraper run ... --transport oxylabs same run, requests via Oxylabs Web Scraper API python -m marianos_scraper run ... --max-requests 40 --max-seconds 1800 stop (resumable) when a budget is spent python -m marianos_scraper run ... --max-requests-per-run 2500 budget of the whole run across resumes (exit 5 when spent) + python -m marianos_scraper run ... --transport oxylabs --ledger blob: Oxylabs collection: jobs reserved in the + shared allowance ledger before any is sent (required) + Exit codes of run: 0 published (or a passed sample release), 1 held back / publish failed, 2 aborted (resumable), + 3 another run active, 4 interrupted or deadline reached (resumable), 5 budget/allowance spent or ledger refused, + 6 checkpoint belongs to another source/store/scope (not resumed; same settings or --fresh) python -m marianos_scraper probe --location ... --transport oxylabs --upcs A,B --roles ppm-volume,unavailable --max-requests 15 --max-seconds 1200 --out DIR capped store+product+coupon test, evidence to DIR; exit 0 accepted, 1 completed but not accepted, 2/4 stopped @@ -17,12 +22,15 @@ import argparse import json +import os import sys +import threading +import time from .config import load_config from .kroger_api import KrogerApi, SourceError from .log import log -from .runner import Runner +from .runner import CheckpointMismatch, Runner, deadline_seconds from .site_runner import SiteRunner from .site_source import RunBudgetSpent, SiteSession, SiteUnavailable from .storage import LockHeld, open_storage @@ -40,6 +48,47 @@ from .oxylabs_source import OxylabsSession return OxylabsSession(settings) return SiteSession(settings["cdp_url"], settings) + + +WATCHDOG_GRACE_SECONDS = 300 + + +def start_watchdog(seconds: float, exit=os._exit) -> threading.Timer: + """Hard stop of a collection process still running `seconds` after it started: a request or browser call that + never returns cannot hold the run, its lock or its reservation past the deadline. Exit 4 (resumable): finished + units are checkpointed and the stuck request is already counted in the run's durable reservation.""" + def fire(): + def say(): + try: + sys.stderr.write(json.dumps({"level": "error", "event": "deadline_watchdog", "after_seconds": seconds, + "note": "process stopped hard; the next start resumes the run"}) + "\n") + sys.stderr.flush() + except Exception: # noqa: BLE001 + pass + t = threading.Thread(target=say, daemon=True) # a blocked stderr cannot keep the process alive + t.start() + t.join(0.5) + exit(4) + timer = threading.Timer(seconds, fire) + timer.daemon = True + timer.start() + return timer + + +def reserve_allowance(args, loc, site): + """Oxylabs collection: reserve this process's jobs in the shared ledger before anything is sent; cap the + transport at the reservation. Returns the Allowance (finalized by the caller).""" + from . import __version__ + from .allowance import Allowance, open_store + allowance = Allowance(open_store(args.ledger), ceiling=args.ledger_ceiling) + stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime()) + wanted = args.max_requests if args.max_requests is not None else site.max_requests + info = allowance.reserve(f"collect/{loc.key}/{stamp}", f"v{__version__}", wanted) + log("info", "allowance_reserved", **{k: v for k, v in info.items() if k != "ledger"}) + if info["cap_this_run"] <= 0: + raise RunBudgetSpent(f"the shared allowance has no jobs left ({info['counted_before']} of {info['ceiling']} counted)") + site.max_requests = info["cap_this_run"] + return allowance def resumable_site_run(cfg, loc) -> bool: @@ -68,6 +117,8 @@ cmd.add_argument("--max-seconds", type=float, help="stop (resumable) once this process has run this long") cmd.add_argument("--max-requests-per-run", type=int, help="request budget of the whole run, counted across resumes; when spent: exit 5, not retried") + cmd.add_argument("--ledger", help="shared allowance ledger (blob: or a local file); required with --transport oxylabs") + cmd.add_argument("--ledger-ceiling", type=int, default=15, help="total Oxylabs jobs allowed across all executions") probe = sub.add_parser("probe", help="capped store + product + coupon test of one transport; writes an evidence folder") probe.add_argument("--location", required=True) probe.add_argument("--transport", choices=["site", "oxylabs"], default="site") @@ -141,15 +192,26 @@ cats = [c.strip() for c in args.categories.split(",")] if args.categories else None source = args.source or cfg.settings.get("source", "auto") transport = args.transport or cfg.settings.get("transport", "site") + if transport == "oxylabs" and source != "api" and not args.ledger: + log("error", "run_not_started", error="--transport oxylabs needs --ledger: every Oxylabs job is reserved in the " + "shared allowance ledger before it is sent; nothing was sent") + return 2 + deadline = deadline_seconds(cfg.settings) + if args.max_seconds is not None: + deadline = min(deadline, args.max_seconds) exit_code = 0 for loc in locations: opts = dict(fresh=args.fresh, only_categories=cats, stop_after_units=args.stop_after_units) + watchdog = start_watchdog(deadline + WATCHDOG_GRACE_SECONDS) + allowance, site = None, None try: if source == "api": report = Runner(cfg, loc, **opts).run() else: try: - site = build_site(cfg, transport, args.max_requests, args.max_seconds) + site = build_site(cfg, transport, args.max_requests, args.max_seconds if args.max_seconds is not None else deadline) + if transport == "oxylabs": + allowance = reserve_allowance(args, loc, site) report = SiteRunner(cfg, loc, site=site, max_requests_per_run=args.max_requests_per_run, **opts).run() except SiteUnavailable as exc: if source == "site": @@ -163,6 +225,10 @@ # unless settings allow publishing fallback runs. log("error", "site_unavailable_fallback", location=loc.key, error=str(exc)) report = Runner(cfg, loc, fallback_reason=str(exc), **{**opts, "fresh": True}).run() + except CheckpointMismatch as exc: + log("error", "checkpoint_mismatch", location=loc.key, error=str(exc)) + exit_code = max(exit_code, 6) # not retried: a person resumes with the same settings or starts --fresh + continue except LockHeld as exc: log("warning", "skipped_overlapping_run", location=loc.key, detail=str(exc)) exit_code = max(exit_code, 3) @@ -175,10 +241,25 @@ log("error", "run_aborted", location=loc.key, error=str(exc)) exit_code = max(exit_code, 2) continue + except Exception as exc: # noqa: BLE001 - a ledger that cannot be read/written/verified: a person decides + from .allowance import LedgerError + if not isinstance(exc, LedgerError): + raise + log("error", "allowance_refused", location=loc.key, error=str(exc), note="nothing more is sent") + exit_code = max(exit_code, 5) + continue + finally: + watchdog.cancel() + if allowance is not None and allowance.reservation_id: + try: + allowance.finalize(site.stats.requests, exit_code, time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())) + log("info", "allowance_finalized", jobs=site.stats.requests) + except Exception as exc: # noqa: BLE001 - the full reservation stays counted + log("error", "allowance_not_finalized", error=str(exc), note="the full reservation stays counted") status = report.get("status") if status in ("blocked", "interrupted"): exit_code = max(exit_code, 4) # resumable: the next start continues the same run - elif status != "published": + elif status not in ("published", "sample"): exit_code = max(exit_code, 1) log("info", "location_finished", location=loc.key, status=status, records=report.get("unique_records")) return exit_code --- /dev/null +++ b/scraper/marianos_scraper/delivery.py @@ -0,0 +1,139 @@ +"""The one writer of every delivery (v2.2.18). + +A delivery is products.json, coupons.json, report.json and catalog.json (the store's observation history) of one run. +They are published together or not at all: + + 1. every file is written to a new, never-reused folder releases//// and read back; + 2. only then is the single pointer latest//CURRENT.json replaced, with one conditional write (local: the + SHA-256 of the pointer read; Blob: If-Match on its ETag). The pointer names the release folder, the SHA-256 of + every file in it, the release it replaces and the release history of the store. + +Until step 2 succeeds the pointer still names the previous release, whose files were never touched, so a failure at +any write (a coupon file that cannot be written, a lost connection, a killed process) leaves the previous delivery +whole. Readers go through the pointer and check each file's SHA-256, so a mixed delivery cannot be read. + +A run that is not a full delivery (only some categories, a limited coupon lookup, a fallback source) is written as a +sample release under samples/// and never touches the pointer. + +Stores written by v2.2.17 and earlier have no pointer; their latest//*.json and state//catalog.json +are read as the baseline until the first v2.2.18 publish, and never written again. +""" +from __future__ import annotations + +import hashlib +import json +import secrets +from datetime import datetime, timezone + +from .storage import PreconditionFailed + +DELIVERY_FILES = ("products.json", "coupons.json", "report.json", "catalog.json") +_LEGACY = {"products.json": "latest/{loc}/products.json", "coupons.json": "latest/{loc}/coupons.json", + "report.json": "latest/{loc}/report.json", "catalog.json": "state/{loc}/catalog.json"} + + +class PublishFailed(Exception): + """The delivery was not published; the previous delivery is unchanged.""" + + +class DeliveryDamaged(Exception): + """A release file does not match the SHA-256 its pointer names: it is not read.""" + + +def sha256_text(text: str) -> str: + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def _utc() -> str: + return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +class Delivery: + def __init__(self, storage, loc_key: str): + self.storage, self.loc = storage, loc_key + self.pointer_key = f"latest/{loc_key}/CURRENT.json" + + # --- reading ------------------------------------------------------------------------------------------- + def current(self) -> dict | None: + text, _ = self.storage.read_with_etag(self.pointer_key) + if text is None: + return None + ptr = json.loads(text) + if not isinstance(ptr, dict) or not ptr.get("run_id") or not isinstance(ptr.get("files"), dict): + raise DeliveryDamaged(f"{self.pointer_key} is not a valid delivery pointer") + return ptr + + def baseline(self, run_id: str) -> dict | None: + """The newest published delivery that is not run_id itself (a refinish of the published run compares with + the release it replaced). {"legacy": True} for a store last written by v2.2.17; None when there is none.""" + ptr = self.current() + if ptr is None: + if self.storage.exists(_LEGACY["products.json"].format(loc=self.loc)): + return {"legacy": True, "run_id": None} + return None + if ptr["run_id"] != run_id: + return ptr + return ptr.get("previous") + + def read(self, release: dict | None, name: str) -> str | None: + """One file of a release (as returned by current()/baseline()), checked against its SHA-256.""" + if not release: + return None + if release.get("legacy"): + key = _LEGACY[name].format(loc=self.loc) + return self.storage.read_text(key) if self.storage.exists(key) else None + want = release["files"].get(name) + if want is None: + return None + text = self.storage.read_text(release["release"] + name) + if sha256_text(text) != want: + raise DeliveryDamaged(f"{release['release']}{name} does not match the SHA-256 in its pointer") + return text + + # --- writing ------------------------------------------------------------------------------------------- + def _write_verified(self, prefix: str, files: dict[str, str]) -> dict[str, str]: + hashes = {} + for name, text in files.items(): + try: + self.storage.write_text(prefix + name, text) + back = self.storage.read_text(prefix + name) + except Exception as exc: # noqa: BLE001 - any storage failure means: not published + raise PublishFailed(f"writing {prefix}{name} failed: {type(exc).__name__}: {exc}") from exc + if back != text: + raise PublishFailed(f"{prefix}{name} read back differently") + hashes[name] = sha256_text(text) + return hashes + + def publish(self, run_id: str, files: dict[str, str], summary: dict) -> dict: + missing = [n for n in DELIVERY_FILES if n not in files] + if missing: + raise PublishFailed(f"a delivery needs {', '.join(DELIVERY_FILES)}; missing {missing}") + prefix = f"releases/{self.loc}/{run_id}/{datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ')}-{secrets.token_hex(3)}/" + hashes = self._write_verified(prefix, files) + try: + old_text, etag = self.storage.read_with_etag(self.pointer_key) + except Exception as exc: # noqa: BLE001 + raise PublishFailed(f"reading {self.pointer_key} failed: {exc}") from exc + old = json.loads(old_text) if old_text else None + entry = {"run_id": run_id, "release": prefix, "published_utc": _utc(), **summary} + pointer = {"run_id": run_id, "release": prefix, "files": hashes, "published_utc": entry["published_utc"], + "summary": summary, + "previous": ({k: old[k] for k in ("run_id", "release", "files", "published_utc")} if old else None), + "history": ((old or {}).get("history") or []) + [entry]} + text = json.dumps(pointer, indent=1, ensure_ascii=False) + try: + self.storage.write_if_match(self.pointer_key, text, etag) + except PreconditionFailed as exc: + raise PublishFailed(f"{self.pointer_key} changed during the publish (another writer?): {exc}") from exc + except Exception as exc: # noqa: BLE001 + raise PublishFailed(f"writing {self.pointer_key} failed: {type(exc).__name__}: {exc}") from exc + back, _ = self.storage.read_with_etag(self.pointer_key) + if back != text: + raise PublishFailed(f"{self.pointer_key} read back differently") + return pointer + + def release_sample(self, run_id: str, files: dict[str, str]) -> str: + """A run that is not a full delivery: its files under samples/, the pointer untouched.""" + prefix = f"samples/{self.loc}/{run_id}/" + self._write_verified(prefix, files) + return prefix --- a/scraper/marianos_scraper/oxylabs_source.py +++ b/scraper/marianos_scraper/oxylabs_source.py @@ -77,6 +77,8 @@ class OxylabsSession(SiteSession): + transport_name = "oxylabs" + def __init__(self, settings: dict, *, username: str | None = None, password: str | None = None, http: requests.Session | None = None): super().__init__(cdp_url="", settings=settings) --- a/scraper/marianos_scraper/runner.py +++ b/scraper/marianos_scraper/runner.py @@ -4,7 +4,9 @@ work//current.json run state; an unfinished run is resumed by the next start work///units/.json one finished category/subcategory/term unit (the checkpoint) runs///... every run's output, report, request log and run log (good or failed) - latest//products.json last run that PASSED validation; only ever replaced by another pass + latest//CURRENT.json pointer to the last FULL run that passed validation (v2.2.18: delivery.py) + releases//// that delivery: products, coupons, report, catalog, published together + samples/// a passed run that is not a full delivery (subset, fallback): never latest/ state//canaries.json canary products seeded from the first good run """ from __future__ import annotations @@ -20,6 +22,7 @@ from pathlib import Path from .config import Config, Location +from .delivery import Delivery, PublishFailed from .kroger_api import KrogerApi, SourceBlocked, SourceError from .site_source import RequestCapReached from .log import add_sink, log, remove_sink, set_context @@ -29,6 +32,7 @@ RESUME_MAX_AGE_HOURS = 48 LOCK_STALE_SECONDS = 6 * 3600 +DEFAULT_DEADLINE_SECONDS = 5 * 3600 # one collection process; below LOCK_STALE_SECONDS, so a live lock never looks stale CANARY_COUNT = 20 @@ -84,6 +88,23 @@ class Interrupted(Exception): pass + + +class CheckpointMismatch(SourceError): + """An unfinished run (or one of its checkpoints) belongs to another source, store or scope: never resumed or + used, nothing is sent (exit 6, not retried; resume it with the same settings or start over with --fresh).""" + + +def deadline_seconds(settings: dict) -> float: + """Runtime deadline of one collection process (settings collection.deadline_seconds), always finite.""" + value = float((settings.get("collection") or {}).get("deadline_seconds") or DEFAULT_DEADLINE_SECONDS) + if not 0 < value < LOCK_STALE_SECONDS: + raise ValueError(f"collection.deadline_seconds must be between 0 and {LOCK_STALE_SECONDS}, got {value}") + return value + + +def digest(data) -> str: + return hashlib.sha256(json.dumps(data, sort_keys=True).encode("utf-8")).hexdigest()[:16] class Runner: @@ -106,6 +127,74 @@ self._interrupted = False self.matchers: dict[str, list] = {} self.excluders: dict[str, re.Pattern | None] = {} + self.delivery = Delivery(self.storage, location.key) + self.deadline = deadline_seconds(cfg.settings) + self._t0 = time.monotonic() + + # --- what this run is (v2.2.18) --------------------------------------------------------------------- + def binding(self) -> dict: + """Source, store and scope of this run. Stored with the run state, the plan and every unit checkpoint; a run + is resumed, and a checkpoint used, only under exactly the same binding.""" + return {"source": "kroger_public_api" + (" (fallback)" if self.fallback_reason else ""), + "location_key": self.loc.key, "location_id": self.loc.location_id, "zipcode": self.loc.zipcode, + "chain": self.loc.expected_chain, "scope": sorted(self.only) if self.only else "full"} + + def limited_reasons(self) -> list[str]: + """Why this run is not a full delivery (empty = full). A limited run is released as a sample only. Read from + the run's own binding, so a refinish judges the run by how it was collected, not by today's options.""" + b = (getattr(self, "_state", None) or {}).get("binding") or self.binding() + out = [] + if b.get("scope") != "full": + out.append("categories limited to: " + ", ".join(b.get("scope") or ["(unknown)"])) + if "fallback" in str(b.get("source", "")): + out.append("fallback source: " + (self.fallback_reason or "Kroger Public API")) + if b.get("coupon_link_scope", "all") != "all": + out.append(f"coupon lookup limited (coupon_link_scope={b['coupon_link_scope']})") + return out + + def time_left(self) -> float: + return self.deadline - (time.monotonic() - self._t0) + + def _wait(self, seconds: float) -> None: + """A wait between units that stops at the deadline instead of running past it.""" + if seconds >= self.time_left(): + raise Interrupted("cap") + time.sleep(seconds) + + def _check_unit(self, state: dict, unit_id: str, result: dict) -> None: + if result.get("binding_digest") != state.get("binding_digest"): + raise CheckpointMismatch(f"checkpoint {unit_id} of run {state['run_id']} was written for binding " + f"{result.get('binding_digest') or 'none (before v2.2.18)'}, the run is " + f"{state.get('binding_digest')}: not used") + + def _baseline_report(self, run_id: str) -> dict | None: + text = self.delivery.read(self.delivery.baseline(run_id), "report.json") + return json.loads(text) if text else None + + def _deliver(self, state: dict, report: dict, files: dict) -> None: + """A passed run: a full delivery through the one writer, or a sample release. On any failure the previous + delivery stays as it was.""" + run_id, out = state["run_id"], f"runs/{self.loc.key}/{state['run_id']}/" + try: + if report["delivery"]["kind"] == "sample": + report["delivery"]["release"] = self.delivery.release_sample(run_id, files) + state["status"] = "sample" + log("info", "run_released_as_sample", release=report["delivery"]["release"], + limited_by=report["delivery"]["limited_by"], note="latest/ untouched") + else: + ptr = self.delivery.publish(run_id, files, {"records": report["unique_records"], + "products_sha256": report["evidence"]["products_sha256"]}) + report["delivery"]["release"] = ptr["release"] + report["delivery"]["replaced"] = (ptr.get("previous") or {}).get("run_id") + state["status"] = "published" + log("info", "run_published", records=report["unique_records"], release=ptr["release"], + runtime_seconds=report["runtime_seconds"]) + except PublishFailed as exc: + report["status"] = "publish_failed" + report["delivery"]["error"] = str(exc) + state["status"] = "publish_failed" + log("error", "publish_failed", error=str(exc), note="the previous delivery is unchanged") + self.storage.write_text(out + "report.json", json.dumps(report, indent=1, ensure_ascii=False)) def classify(self, unit: dict, title: str) -> tuple[str, str]: """Pick the subcategory whose words appear in the title, preferring the one being searched. @@ -244,26 +333,37 @@ state = json.loads(self.storage.read_text(key)) age_h = (time.time() - state["started_epoch"]) / 3600 if state["status"] in ("running", "incomplete", "interrupted") and age_h < RESUME_MAX_AGE_HOURS: + if state.get("binding") != self.binding(): + raise CheckpointMismatch( + f"unfinished run {state['run_id']} was started for " + f"{state.get('binding') or 'an unrecorded source/store/scope (before v2.2.18)'}; this start is " + f"for {self.binding()}: not resumed, nothing sent; resume it with the same settings or start " + f"over with --fresh") state["resumed"] = state.get("resumed", 0) + 1 + state["_resumed_after_crash"] = False if state["segments"][-1]["end"] is None: # previous process died without a clean stop seg = state["segments"][-1] seg["end"] = max(seg["start"], state.get("last_progress", seg["start"])) state["crashed_segments"] = state.get("crashed_segments", 0) + 1 + state["_resumed_after_crash"] = True state["segments"].append({"start": time.time(), "end": None}) state["_event"] = ("resume_run", {"age_hours": round(age_h, 1), "resumed": state["resumed"]}) return state # Timestamp for humans + random suffix so two starts in the same second never share checkpoints. run_id = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + "-" + secrets.token_hex(2) state = {"run_id": run_id, "started": utc_now(), "started_epoch": time.time(), "status": "running", - "resumed": 0, "segments": [{"start": time.time(), "end": None}], "only": sorted(self.only or [])} + "resumed": 0, "segments": [{"start": time.time(), "end": None}], "only": sorted(self.only or []), + "binding": self.binding(), "binding_digest": digest(self.binding()), "_resumed_after_crash": False} state["_event"] = ("start_run", {}) return state def _save_state(self, state: dict) -> None: - self.storage.write_text(f"work/{self.work_key}/current.json", json.dumps(state, indent=1)) + self.storage.write_text(f"work/{self.work_key}/current.json", + json.dumps({k: v for k, v in state.items() if not k.startswith("_")}, indent=1)) def _run_locked(self) -> dict: state = self._load_or_start_state() + self._state = state run_id = state["run_id"] set_context(run_id=run_id, location=self.loc.key) log_path = Path(self.cfg.storage_root) / "_runlogs" / f"{self.loc.key}-{run_id}.log" @@ -281,6 +381,12 @@ state["status"] = "failed_location" self._save_state(state) raise SourceError(f"Store verification failed: {before['detail']}") + prior = state.get("location") + if prior and any(str(prior.get(k, "")) != str(location.get(k, "")) for k in ("location_id", "facility_id", "zipcode")): + raise CheckpointMismatch(f"run {run_id} was collected for store {prior.get('location_id')} facility " + f"{prior.get('facility_id')} zip {prior.get('zipcode')}; the store answering now " + f"is {location.get('location_id')} / {location.get('facility_id')} / " + f"{location.get('zipcode')}: not resumed") state["location"] = location canaries = self.check_canaries() @@ -303,6 +409,9 @@ for unit in todo: if self._interrupted or (self.stop_after_units is not None and processed >= self.stop_after_units): raise Interrupted() + if self.time_left() <= 0: + log("warning", "deadline_reached", deadline_seconds=self.deadline, unit=unit["id"]) + raise Interrupted("cap") try: result = self.collect_unit(unit, location) except SourceBlocked as exc: @@ -316,6 +425,7 @@ log("error", "unit_failed", unit=unit["id"], attempt=attempt, error=str(exc)) retry_later.append(unit) continue + result["binding_digest"] = state["binding_digest"] self.storage.write_text(unit_prefix + unit["id"] + ".json", json.dumps(result)) processed += 1 state["last_progress"] = time.time() @@ -325,7 +435,7 @@ todo = retry_later if not todo: break - time.sleep(30) + self._wait(30) failed = [u["id"] for u in todo] after, _ = self.verify_location() @@ -367,6 +477,7 @@ if unit["id"] not in by_id: continue result = json.loads(self.storage.read_text(by_id[unit["id"]])) + self._check_unit(state, unit["id"], result) excluded.update(result["excluded"]) labelled.update(result.get("labelled", {})) products_seen += result["products_seen"] @@ -391,9 +502,8 @@ counts_cat = Counter(r["category"] for r in records) counts_sub = Counter(f"{r['category']} > {r['sub_category']}" for r in records) supported = [c.name for c in self.cfg.categories if c.supported and (not self.only or c.name.casefold() in self.only)] - previous = None - if self.storage.exists(f"latest/{self.loc.key}/report.json") and not self.only: - previous = json.loads(self.storage.read_text(f"latest/{self.loc.key}/report.json")) + limited = self.limited_reasons() + previous = None if limited else self._baseline_report(run_id) priced_share = products_priced / products_seen if products_seen else 0.0 extra = [] @@ -408,7 +518,9 @@ stats = self.api.stats products_json = json.dumps(records, ensure_ascii=False, indent=1) report = { - "run_id": run_id, "status": "published" if validation["passed"] else "held_back", + "run_id": run_id, "status": ("sample" if limited else "published") if validation["passed"] else "held_back", + "delivery": {"kind": "sample" if limited else "full", "limited_by": limited, + "rule": "only a full run that passes every check replaces latest/; a limited run is a sample release"}, "source": "kroger_public_api" + (" (fallback)" if self.fallback_reason else ""), "location": state["location"], "started": state["started"], "finished": utc_now(), "runtime_seconds": round(runtime, 1), "resumed_times": state.get("resumed", 0), @@ -438,13 +550,16 @@ earlier = [self.storage.read_text(k) for k in self.storage.list(f"work/{self.work_key}/{run_id}") if "/requests-" in k] self.storage.write_text(out + "requests.jsonl", "\n".join(earlier + [json.dumps(r) for r in stats.request_log])) if validation["passed"]: - # Publish: only now is latest/ touched. A failed run above never reaches this line. - self.storage.write_text(f"latest/{self.loc.key}/products.json", products_json) - self.storage.write_text(f"latest/{self.loc.key}/report.json", json.dumps(report, indent=1, ensure_ascii=False)) - if not self.storage.exists(f"state/{self.loc.key}/canaries.json") and not self.only: + # Publish: only now is latest/ touched, and only through the one writer. A failed run never gets here. + base = None if limited else self.delivery.baseline(run_id) + files = {"products.json": products_json, + "coupons.json": json.dumps({"standard": [], "cash_back": [], "note": "the Kroger Public API has no coupons"}), + "report.json": json.dumps(report, indent=1, ensure_ascii=False), + # the API has no availability data: the store's observation history is carried over unchanged + "catalog.json": self.delivery.read(base, "catalog.json") or json.dumps({"run_id": None, "products": {}})} + self._deliver(state, report, files) + if state["status"] == "published" and not self.storage.exists(f"state/{self.loc.key}/canaries.json"): self._seed_canaries(records) - state["status"] = "published" - log("info", "run_published", records=len(records), runtime_seconds=report["runtime_seconds"]) else: state["status"] = "incomplete" if failed else "held_back" failed_checks = [c for c in validation["checks"] if not c["passed"]] --- a/scraper/marianos_scraper/site_runner.py +++ b/scraper/marianos_scraper/site_runner.py @@ -24,8 +24,9 @@ from .kroger_api import KrogerApi, SourceError from .log import log from .mapper import FIELD_ORDER, SiteMapper -from .runner import Runner, code_fingerprint, slug, utc_now -from .site_source import SEARCH_MAX_OFFSET, PAGE_SIZE, RunBudgetSpent, SiteSession +from .delivery import Delivery +from .runner import CheckpointMismatch, Runner, code_fingerprint, slug, utc_now +from .site_source import SEARCH_MAX_OFFSET, PAGE_SIZE, RequestCapReached, RunBudgetSpent, SiteSession from .validate import completeness, record_key, validate_run @@ -59,6 +60,17 @@ self.taxonomy = json.loads((Path(cfg.config_dir) / "site_taxonomy.json").read_text(encoding="utf-8"))["categories"] self.run_calls: list[dict] = [] self._run_id = "" + # v2.2.18: normal collection always has a runtime deadline (settings collection.deadline_seconds unless + # --max-seconds sets a shorter one); every request and wait of the transport stops at it + if getattr(self.site, "max_seconds", "absent") is None: + self.site.max_seconds = self.deadline + + def binding(self) -> dict: + b = super().binding() + b.update(source="marianos.com", transport=getattr(self.site, "transport_name", type(self.site).__name__), + facility_id=self.loc.facility_id, + coupon_link_scope=self.cfg.settings["site_source"].get("coupon_link_scope", "all")) + return b # --- run-level request accounting -------------------------------------------------------------------- def _drain_calls(self) -> list[dict]: @@ -76,9 +88,19 @@ def _load_or_start_state(self) -> dict: state = super()._load_or_start_state() self._run_id = state["run_id"] - # requests this run already used in earlier segments (kept in the run state at every save) - state["site_requests_base"] = state.get("site_requests_used", 0) + # requests this run already used in earlier segments. v2.2.18: requests are reserved in the run state before + # they are sent; when the previous process died, everything it had reserved counts as sent + used = state.get("site_requests_used", 0) + reserved = state.get("site_requests_reserved", used) + if state.get("_resumed_after_crash") and reserved > used: + state["requests_counted_from_reservation"] = state.get("requests_counted_from_reservation", 0) + reserved - used + log("warning", "reservation_counted_in_full", run_id=state["run_id"], recorded_used=used, reserved=reserved, + note="the previous process stopped without a clean save; its whole reservation counts as sent") + used = reserved + state["site_requests_base"] = state["site_requests_used"] = state["site_requests_reserved"] = used self.site.stats.requests = 0 + self._state = state + self.site.reserve = self._reserve_request if self.max_requests_per_run is not None: left = self.max_requests_per_run - state["site_requests_base"] if left <= 0: @@ -94,7 +116,42 @@ def _save_state(self, state: dict) -> None: state["site_requests_used"] = state.get("site_requests_base", 0) + self.site.stats.requests + if state.get("segments") and state["segments"][-1].get("end") is not None: + state["site_requests_reserved"] = state["site_requests_used"] # segment closed: the exact count is known + else: + state["site_requests_reserved"] = max(state.get("site_requests_reserved", 0), state["site_requests_used"]) super()._save_state(state) + + def _reserve_request(self, path: str) -> None: + """Called by the transport immediately before it counts and sends a request: makes sure the request lies + inside a reservation already written to the run state (and read back). Reservations are taken in blocks + (site_source.request_reservation_block), never beyond --max-requests-per-run. If the reservation cannot be + written, nothing is sent and the run stops resumably.""" + st = self._state + need = st["site_requests_base"] + self.site.stats.requests + 1 + if need <= st.get("site_requests_reserved", 0): + return + block = max(1, int(self.cfg.settings["site_source"].get("request_reservation_block", 25))) + new = need - 1 + block + if self.max_requests_per_run is not None: + new = min(new, self.max_requests_per_run) + if new < need: + raise RequestCapReached(f"run request budget {self.max_requests_per_run} fully reserved before {path[:80]}") + before = st.get("site_requests_reserved", 0) + st["site_requests_reserved"], st["site_requests_used"] = new, need - 1 + key = f"work/{self.work_key}/current.json" + text = json.dumps({k: v for k, v in st.items() if not k.startswith("_")}, indent=1) + try: + self.storage.write_text(key, text) + ok = self.storage.read_text(key) == text + except Exception as exc: # noqa: BLE001 - any storage failure: the request is not sent + st["site_requests_reserved"] = before + raise RequestCapReached(f"request reservation could not be written ({type(exc).__name__}: {exc}); " + f"nothing sent for {path[:80]}") from exc + if not ok: + st["site_requests_reserved"] = before + raise RequestCapReached(f"request reservation read back differently; nothing sent for {path[:80]}") + log("info", "requests_reserved", reserved_up_to=new, block=new - before) # --- store --------------------------------------------------------------------------------------------- def verify_location(self) -> tuple[dict, dict]: @@ -142,6 +199,9 @@ plan_key = f"work/{self.work_key}/{self._run_id}/plan.json" if self.storage.exists(plan_key): plan = json.loads(self.storage.read_text(plan_key)) + if plan.get("binding_digest") != self._state.get("binding_digest"): + raise CheckpointMismatch(f"plan of run {self._run_id} belongs to binding {plan.get('binding_digest')}, " + f"the run is {self._state.get('binding_digest')}: not used") self.expected = plan["expected"] plan["units"] = list({u["id"]: u for u in plan["units"]}.values()) if not any(u.get("kind") == "coupon_links" for u in plan["units"]): @@ -161,6 +221,9 @@ cat_expected = 0 for root in spec["roots"]: _, total, facets = self.site.search(self.loc.location_id, root, size=1, facets=True) + if total is None: + raise SourceError(f"the site gave no product count for taxonomy root {root}: completeness " + f"cannot be proven, nothing planned") node = _find_node(facets, root) or {"id": root, "count": total, "children": []} cat_expected += total - sum((_find_node(facets, e) or {}).get("count", 0) for e in excl if e.startswith(root)) level2 = [node] if len(root) >= 5 else [c for c in node.get("children", []) if c["id"] not in excl] @@ -178,12 +241,17 @@ units.append(COUPON_LINK_UNIT) units = list({u["id"]: u for u in units}.values()) # the site's facet tree can list a node twice self.expected = expected - self.storage.write_text(plan_key, json.dumps({"units": units, "expected": expected, "planned_at": utc_now()}, indent=1)) + self.storage.write_text(plan_key, json.dumps({"units": units, "expected": expected, "planned_at": utc_now(), + "binding": self._state.get("binding"), + "binding_digest": self._state.get("binding_digest")}, indent=1)) self.run_calls += self._drain_calls() return units def _split(self, node: dict) -> list[dict]: - if node.get("count", 0) <= self.max_node: + if node.get("count") is None: + raise SourceError(f"the site gave no product count for taxonomy node {node.get('id')}: completeness " + f"cannot be proven, nothing planned") + if node["count"] <= self.max_node: return [node] children = node.get("children") if not children: @@ -204,7 +272,8 @@ def _brand_split(self, node: dict) -> list[dict]: """A node larger than the 350-id search window, split by the site's brand facet (the brand counts add up to it).""" _, total, brands = self.site.search(self.loc.location_id, node["id"], size=1, brand_facet=True) - if brands and sum(b.get("count", 0) for b in brands) >= total: + if brands and total is not None and all(b.get("count") is not None for b in brands) \ + and sum(b["count"] for b in brands) >= total: return [{"id": node["id"], "taxonomy": node["id"], "name": node["name"], "brand": b["name"], "count": b["count"]} for b in brands] return [{**node, "children": []}] # cannot be split further; collected up to the window and reported as capped @@ -217,17 +286,31 @@ if unit.get("kind") == "coupon_links": return self._link_coupons(unit, t0) if unit.get("kind") == "coupons": - standard, meta = self.site.all_coupons("standard") - cashback, _ = self.site.all_coupons("cash_back") + # v2.2.18: the site's own total (meta totalCount) of each coupon type is kept; the paged list re-sorts + # between pages, so it is read again (at most three passes, union by coupon id) until every coupon the + # site counts has been seen. A missing total is recorded as missing and holds the run back. + got, totals, entries, counts = {}, {}, {}, {} + for kind in ("standard", "cash_back"): + seen, listed, total, meta = {}, 0, None, {} + for _ in range(3): + batch, meta = self.site.all_coupons(kind) + listed += len(batch) + total = int(meta["totalCount"]) if meta.get("totalCount") is not None else None + for c in batch: + seen.setdefault(str(c.get("id")), c) + if total is None or len(seen) >= total: + break + got[kind], totals[kind], entries[kind] = list(seen.values()), total, listed + counts.update(meta.get("userSavingsInfoByType") or {}) raw = [c.get("raw") for c in self.site.calls if c.get("raw") is not None] self._write_raw(unit, raw) calls = self._drain_calls() - return {"unit": unit, "records": [], "coupons": {"standard": standard, "cash_back": cashback}, - "site_counts": (meta.get("userSavingsInfoByType") or {}), "excluded": {}, "pages": len(calls), - "api_total": len(standard) + len(cashback), "products_seen": 0, "products_priced": 0, + return {"unit": unit, "records": [], "coupons": got, "site_totals": totals, "coupons_page_entries": entries, + "site_counts": counts, "excluded": {}, "pages": len(calls), + "api_total": len(got["standard"]) + len(got["cash_back"]), "products_seen": 0, "products_priced": 0, "requests": calls, "seconds": round(time.time() - t0, 2), "finished_at": utc_now()} - upcs, total, pages = [], None, 0 + upcs, total, pages, total_missing = [], None, 0, False for _ in range(3): offset = 0 while True: @@ -236,6 +319,9 @@ for it in items: if it.get("upc") and it["upc"] not in upcs: upcs.append(it["upc"]) + if total is None: + total_missing = True # v2.2.18: no site count = completeness unknown, never "complete" + break last_offset, offset = offset, offset + PAGE_SIZE if not items or offset >= total: break @@ -247,7 +333,7 @@ offset = SEARCH_MAX_OFFSET # the listing order can shift between pages (one id shown twice, another skipped): when fewer ids than the # site's count came back, read the listing again and keep the union (at most three passes) - if not total or len(upcs) >= min(total, SEARCH_MAX_OFFSET + PAGE_SIZE): + if total_missing or not total or len(upcs) >= min(total, SEARCH_MAX_OFFSET + PAGE_SIZE): break products = self.site.products(upcs) if upcs else [] returned = {p.get("id") for p in products} @@ -298,7 +384,8 @@ calls = self._drain_calls() return {"unit": unit, "records": records, "meta": metas, "excluded": {k: v for k, v in excluded.items() if v}, "excluded_items": excluded_items, - "pages": pages, "api_total": total or 0, "found_upcs": len(upcs), "products_seen": len(products), + "pages": pages, "api_total": total, "site_total_reported": not total_missing, "listed_upcs": upcs, + "found_upcs": len(upcs), "products_seen": len(products), "products_priced": priced, "store_mismatch": mismatched, "price_status": dict(status), "coupons_linked": sum(1 for r in records if r["digital_deals_description"]), "requests": calls, "seconds": round(time.time() - t0, 2), "finished_at": utc_now()} @@ -331,7 +418,8 @@ raw = [c.get("raw") for c in self.site.calls if c.get("raw") is not None] self._write_raw(unit, raw) calls = self._drain_calls() - return {"unit": unit, "records": [], "links": links, "candidates": len(candidates), "coupons_answered": len(found), + return {"unit": unit, "records": [], "links": links, "candidates": len(candidates), "candidate_upcs": candidates, + "coupons_answered": len(found), "excluded": {}, "pages": len(calls), "api_total": len(candidates), "products_seen": 0, "products_priced": 0, "requests": calls, "seconds": round(time.time() - t0, 2), "finished_at": utc_now()} @@ -367,6 +455,10 @@ state = json.loads(self.storage.read_text(f"work/{self.work_key}/current.json")) if state["run_id"] != run_id: raise SourceError(f"current run is {state['run_id']}, not {run_id}") + if not state.get("binding"): + raise CheckpointMismatch(f"run {run_id} has no source/store/scope binding (written before v2.2.18): not refinished") + self._state = state + self.only = set(state.get("only") or []) or None # the run's own scope, whatever this command line says old = json.loads(self.storage.read_text(f"runs/{self.loc.key}/{run_id}/report.json")) if "finished_epoch" not in state: state["finished_epoch"] = datetime.strptime(old["finished"], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc).timestamp() @@ -381,7 +473,15 @@ def _load_catalog(self, state: dict) -> tuple[dict, str | None]: """Every product of the store ever published, with its last observation, as it stood BEFORE this run - (a refinish of an already published run uses the copy saved before that publish). Bootstrapped from latest/ once.""" + (a refinish of an already published run uses the release that publish replaced). v2.2.18: read from the + published delivery; a store last written by v2.2.17 is read from its legacy files once.""" + base = self.delivery.baseline(state["run_id"]) + if base and not base.get("legacy"): + text = self.delivery.read(base, "catalog.json") + if text: + data = json.loads(text) + return data["products"], data.get("run_id") or base["run_id"] + return self._catalog_from_run(base["run_id"], base["release"] + "products.json") for key in (f"state/{self.loc.key}/catalog.json", f"state/{self.loc.key}/catalog_previous.json"): if self.storage.exists(key): data = json.loads(self.storage.read_text(key)) @@ -408,6 +508,16 @@ products = json.loads(self.storage.read_text(products_key)) return {r["upc"]: {"record": r, "last_observed_run": run, "meta": ev.get(r["upc"], {})} for r in products}, run + def _catalog_of(self, loc_key: str) -> dict | None: + """Another configured store's published catalog (identity fill only), or None.""" + d = Delivery(self.storage, loc_key) + ptr = d.current() + if ptr is not None: + text = d.read(ptr, "catalog.json") + return json.loads(text)["products"] if text else None + key = f"state/{loc_key}/catalog.json" + return json.loads(self.storage.read_text(key))["products"] if self.storage.exists(key) else None + def _carry_api_stats(self, state: dict) -> None: state["run_calls_carry"] = state.get("run_calls_carry", []) + self.run_calls + self._drain_calls() self.run_calls = [] @@ -432,7 +542,8 @@ excluded, excluded_items = Counter(), {} coupons = {"standard": [], "cash_back": []} coupon_site_counts, listed_entries = {}, {} - cat_found = Counter() + cat_ids, cat_listed = defaultdict(set), Counter() # v2.2.18: DISTINCT ids per category + coupon_site_totals, coupon_candidate_upcs = {}, None unit_requests, unit_seconds, pages, capped = [], 0.0, 0, [] products_seen = products_priced = store_mismatch = 0 unit_cover, site_drift = [], Counter() @@ -440,19 +551,22 @@ if unit["id"] not in by_id: continue res = json.loads(self.storage.read_text(by_id[unit["id"]])) + self._check_unit(state, unit["id"], res) unit_requests += res.get("requests", []) unit_seconds += res.get("seconds", 0) pages += res.get("pages", 0) if unit.get("kind") == "coupons": coupons = res["coupons"] coupon_site_counts = res.get("site_counts", {}) - listed_entries = {t: len(v) for t, v in coupons.items()} + coupon_site_totals = res.get("site_totals", {}) + listed_entries = res.get("coupons_page_entries") or {t: len(v) for t, v in coupons.items()} # the paged list can return one coupon twice (it re-sorts between pages): one entry per coupon id coupons = {t: list({str(c.get("id")): c for c in reversed(v)}.values())[::-1] for t, v in coupons.items()} continue if unit.get("kind") == "coupon_links": coupon_links = res.get("links", {}) coupon_candidates = res.get("candidates", 0) + coupon_candidate_upcs = set(res.get("candidate_upcs") or []) continue excluded.update(res.get("excluded", {})) for x in res.get("excluded_items", []): @@ -460,7 +574,8 @@ products_seen += res.get("products_seen", 0) products_priced += res.get("products_priced", 0) store_mismatch += res.get("store_mismatch", 0) - cat_found[unit["category"]] += res.get("found_upcs", 0) + cat_ids[unit["category"]].update(res.get("listed_upcs") or []) + cat_listed[unit["category"]] += res.get("found_upcs", 0) unit_cover.append((unit["id"], res.get("found_upcs", 0), res.get("api_total", 0))) # the site's own count for this node can fall between planning and collecting (products leave the listing); # when every id the site listed at collection time was collected, that drop is the site's change, not a gap @@ -495,7 +610,8 @@ # Availability rule: nothing is dropped for being unavailable. A product observed in an earlier published run # but not listed by the site this time is carried forward with its last observation (its crawl_date stays the # date it was last seen), and availability.jsonl marks it carried_forward, so an old price never reads as new. - catalog, prev_run = ({}, None) if self.only else self._load_catalog(state) + limited = self.limited_reasons() + catalog, prev_run = ({}, None) if limited else self._load_catalog(state) availability, changes = [], [] for upc, m in output_meta.items(): before = catalog.get(upc, {}).get("meta", {}) @@ -521,9 +637,9 @@ if others is None: others = {} for loc in self.cfg.locations: - key = f"state/{loc.key}/catalog.json" - if loc.key != self.loc.key and self.storage.exists(key): - others[loc.key] = json.loads(self.storage.read_text(key))["products"] + other = self._catalog_of(loc.key) if loc.key != self.loc.key else None + if other is not None: + others[loc.key] = other source = next(((c[r["upc"]], k) for k, c in others.items() if r["upc"] in c and c[r["upc"]]["record"].get("product_title")), (None, None)) if source[0] is None: @@ -575,8 +691,9 @@ site_total = exp["site_total"] drift = min(site_drift.get(cat, 0), site_total) current = site_total - drift - found = cat_found.get(cat, 0) + found = len(cat_ids.get(cat, ())) completeness_by_cat[cat] = {"site_count": site_total, "ids_collected": found, + "ids_listed_incl_overlap": cat_listed.get(cat, 0), "records_written": counts_cat.get(cat, 0), "site_count_drop_during_run": drift, "coverage_vs_planned_count": round(min(found, site_total) / site_total, 4) if site_total else 1.0, @@ -604,12 +721,33 @@ found_only[kind] += 1 # no percentage may hide a short node: every unit must hold every id the site listed for it when it was read short_units = [f"{uid} {found}/{total}" for uid, found, total in unit_cover if total is not None and found < total] + no_total = [uid for uid, _, total in unit_cover if total is None] + coupon_short = [] + if coupon_planned: + for kind in ("standard", "cash_back"): + site_total, have = coupon_site_totals.get(kind), len(coupons.get(kind, [])) + if site_total is None or have < site_total: + coupon_short.append(f"{kind}: {have} distinct, site total " + f"{site_total if site_total is not None else 'not reported'}") + fresh_upcs = {r["upc"] for r in fresh_records} + link_scope_all = self.cfg.settings["site_source"].get("coupon_link_scope", "all") == "all" + not_looked_up = sorted(fresh_upcs - (coupon_candidate_upcs or set())) if coupon_planned and link_scope_all else [] extra = [ + {"check": "site_count_reported_for_every_unit", "passed": not no_total, + "detail": "the site reported its own product count for every taxonomy unit" if not no_total + else f"{len(no_total)} units without a site count (completeness unknown): {no_total[:20]}"}, + {"check": "coupons_complete_vs_site_totals", "passed": not coupon_short, + "detail": "every coupon type: distinct coupons collected >= the site's own total" if not coupon_short + else f"incomplete: {coupon_short}"}, + {"check": "coupon_lookup_covered_every_product", "passed": not not_looked_up, + "detail": f"all {len(fresh_upcs)} distinct products were asked for their coupons" if not not_looked_up + else f"{len(not_looked_up)} products never asked for their coupons: {not_looked_up[:10]}"}, {"check": "every_unit_paginated_completely", "passed": not short_units, "detail": "every taxonomy unit collected every id the site listed for it" if not short_units else f"{len(short_units)} units short of the site's own count: {short_units[:20]}"}, {"check": "category_completeness_vs_site_counts", "passed": not low, - "detail": f"every category collected at least {v['min_category_coverage']:.0%} of the product count the site reports" + "detail": f"every category collected at least {v['min_category_coverage']:.0%} of the product count the site reports " + f"(distinct products; an id listed by several units of a category counts once)" if not low else f"below {v['min_category_coverage']:.0%}: {low}"}, {"check": "store_on_every_price", "passed": not foreign, "detail": f"every price in the output came back with sourceLocationId {self.loc.location_id}; " @@ -622,11 +760,12 @@ ] priced_share = products_priced / products_seen if products_seen else 0.0 previous = None - prev_key = (f"runs/{self.loc.key}/{state['compared_with']}/report.json" if state.get("compared_with") - else f"latest/{self.loc.key}/report.json") - if self.storage.exists(prev_key) and not self.only: - prev = json.loads(self.storage.read_text(prev_key)) - previous = prev if prev.get("source", "").startswith("marianos.com") and prev.get("run_id") != run_id else None + if not limited: + prev_key = f"runs/{self.loc.key}/{state['compared_with']}/report.json" if state.get("compared_with") else None + prev = (json.loads(self.storage.read_text(prev_key)) if prev_key and self.storage.exists(prev_key) + else self._baseline_report(run_id)) + if prev and prev.get("source", "").startswith("marianos.com") and prev.get("run_id") != run_id: + previous = prev if previous: state["compared_with"] = previous["run_id"] # kept, so a later refinish compares with the same run supported = [c for c in completeness_by_cat] @@ -647,8 +786,10 @@ "no_deal": [r for r in records if not (r["sale_price"] or r["volume_deals_description"] or r["digital_deals_description"])][:3], } report = { - "run_id": run_id, "status": "published" if validation["passed"] else "held_back", - "source": "marianos.com (real Chrome session, site JSON endpoints)", + "run_id": run_id, "status": ("sample" if limited else "published") if validation["passed"] else "held_back", + "delivery": {"kind": "sample" if limited else "full", "limited_by": limited, + "rule": "only a full run that passes every check replaces latest/; a limited run is a sample release"}, + "source": "marianos.com (site JSON endpoints, transport %s)" % (state.get("binding") or {}).get("transport", "unknown"), "location": state["location"], "started": state["started"], "finished": datetime.fromtimestamp(state["finished_epoch"], timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), "runtime": {"active_seconds": round(active, 1), "elapsed_seconds": round(elapsed, 1), @@ -659,6 +800,9 @@ "between an interruption and the resume"}, "runtime_seconds": round(active, 1), "resumed_times": state.get("resumed", 0), "scope": "partial (categories: %s)" % ", ".join(sorted(self.only)) if self.only else "full catalog", + "binding": state.get("binding"), + "requests_reserved": {"counted_from_reservation_after_a_crash": state.get("requests_counted_from_reservation", 0), + "run_total_counted": state.get("site_requests_base", 0) + self.site.stats.requests}, "unique_records": len(records), "duplicates_removed": duplicates_removed, "placed_by_retailer_taxonomy": moved, "availability": {"fresh_records": len(fresh_records), "carried_forward_records": len(records) - len(fresh_records), "carried_forward_from_run": prev_run, "price_status": dict(price_status), @@ -691,7 +835,7 @@ "coupons": {"digital_standard": len(coupons["standard"]), "cash_back": len(coupons["cash_back"]), "coupons_page_entries": listed_entries, "coupons_page_unique": listed_unique, "found_only_via_product_lookup": dict(found_only), - "site_reported": coupon_site_counts, + "site_reported": coupon_site_counts, "site_totals": coupon_site_totals, "digital_coupon_upcs": len(coupon_upcs), "of_which_in_output": len(coupon_upcs & output_upcs), "records_with_digital_deal": sum(1 for r in records if r["digital_deals_description"]), "link_candidates_checked": coupon_candidates, "products_linked_to_coupons": len(coupon_links)}, @@ -758,29 +902,25 @@ self.storage.write_text(out + "availability_changes.json", json.dumps(changes, indent=1)) self.storage.write_text(out + "excluded.jsonl", "\n".join(json.dumps(x, ensure_ascii=False) for x in sorted(excluded_items.values(), key=lambda x: (x["reason"], x["upc"])))) + coupons_json = json.dumps(coupons, indent=1, ensure_ascii=False) self.storage.write_text(out + "products.json", products_json) self.storage.write_text(out + "placements.json", json.dumps(placements, indent=1)) - self.storage.write_text(out + "coupons.json", json.dumps(coupons, indent=1, ensure_ascii=False)) + self.storage.write_text(out + "coupons.json", coupons_json) self.storage.write_text(out + "report.json", json.dumps(report, indent=1, ensure_ascii=False)) self.storage.write_text(out + "requests.jsonl", "\n".join(json.dumps(r) for r in all_requests)) if validation["passed"]: - self.storage.write_text(f"latest/{self.loc.key}/products.json", products_json) - self.storage.write_text(f"latest/{self.loc.key}/coupons.json", json.dumps(coupons, indent=1, ensure_ascii=False)) - self.storage.write_text(f"latest/{self.loc.key}/report.json", json.dumps(report, indent=1, ensure_ascii=False)) - if not self.storage.exists(f"state/{self.loc.key}/canaries.json") and not self.only: + # v2.2.18: products, coupons, report and the store's observation history (catalog) are published together + # by the one writer (delivery.py); a limited run becomes a sample release and never touches latest/ + new_catalog = dict(catalog) + for r in fresh_records: + new_catalog[r["upc"]] = {"record": r, "last_observed_run": run_id, "meta": output_meta[r["upc"]]} + for upc in excluded_items: + new_catalog.pop(upc, None) + self._deliver(state, report, {"products.json": products_json, "coupons.json": coupons_json, + "report.json": json.dumps(report, indent=1, ensure_ascii=False), + "catalog.json": json.dumps({"run_id": run_id, "products": new_catalog})}) + if state["status"] == "published" and not self.storage.exists(f"state/{self.loc.key}/canaries.json"): self._seed_canaries(fresh_records) - if not self.only: - if self.storage.exists(f"state/{self.loc.key}/catalog.json"): - old_catalog = self.storage.read_text(f"state/{self.loc.key}/catalog.json") - if json.loads(old_catalog).get("run_id") != run_id: - self.storage.write_text(f"state/{self.loc.key}/catalog_previous.json", old_catalog) - for r in fresh_records: - catalog[r["upc"]] = {"record": r, "last_observed_run": run_id, "meta": output_meta[r["upc"]]} - for upc in excluded_items: - catalog.pop(upc, None) - self.storage.write_text(f"state/{self.loc.key}/catalog.json", json.dumps({"run_id": run_id, "products": catalog})) - state["status"] = "published" - log("info", "run_published", records=len(records), runtime_seconds=report["runtime_seconds"]) else: state["status"] = "incomplete" if failed else "held_back" log("error", "run_held_back", reason="validation failed; last good dataset left untouched", --- a/scraper/marianos_scraper/site_source.py +++ b/scraper/marianos_scraper/site_source.py @@ -63,6 +63,8 @@ class SiteSession: + transport_name = "chrome-cdp" # part of a run's checkpoint binding: a run never resumes on another transport + def __init__(self, cdp_url: str, settings: dict): self.cdp_url = cdp_url self.min_interval = settings.get("min_interval_seconds", 3.0) @@ -73,6 +75,11 @@ # N requests leave this machine. None = unlimited (the production default). self.max_requests: int | None = settings.get("max_requests") self.max_seconds: float | None = settings.get("max_seconds") + # Longest one request may take even without a runtime cap (v2.2.18: normal collection never waits unbounded). + self.request_timeout: float = float(settings.get("request_timeout_seconds", 90)) + # Called immediately before an attempt is counted and sent; the run sets it to reserve the request durably in + # its state (v2.2.18). It raises to stop the request; None = no reservation (probe, tests). + self.reserve = None self.started = time.time() # wall clock, for evidence timestamps only self._t0 = time.monotonic() # the runtime cap is measured on this clock (immune to clock changes) # Evidence capture (the probe turns it on): every attempt with its exact answer, kept before any parse or raise. @@ -116,6 +123,8 @@ """The last step before a request leaves: both caps checked again, then the attempt is counted. A cap of N therefore means at most N dispatched attempts, whatever happens to them (answer, error, timeout, lost page).""" self.check_caps(path) + if self.reserve is not None: + self.reserve(path) # durable reservation first; nothing is sent if it cannot be written self.stats.requests += 1 rec = {"seq": self.stats.requests, "t": round(time.time(), 3), "elapsed_s": round(self.elapsed(), 3), "path": path[:500], "attempt": attempt + 1} @@ -205,7 +214,7 @@ self._sleep(wait, path) att = self._begin_attempt(path, attempt) # rechecked and counted immediately before dispatch self._last = time.monotonic() - limit = self._request_limit(None) + limit = self._request_limit(self.request_timeout) try: res = self._page.evaluate(_FETCH_JS, [path, headers, int(limit * 1000) if limit else 0]) except Exception as exc: @@ -276,11 +285,15 @@ if brand_facet: body = self.get(q + "&option.facets=BRAND", expect="productsSearch") meta = (body.get("meta") or {}).get("productsSearch") or {} - return [], int(meta.get("totalCount", 0)), (meta.get("facets") or {}).get("brands", []) + total = meta.get("totalCount") + return [], (int(total) if total is not None else None), (meta.get("facets") or {}).get("brands", []) body = self.get(q, expect="productsSearch") meta = (body.get("meta") or {}).get("productsSearch") or {} items = body["data"]["productsSearch"] - return items, int(meta.get("totalCount", len(items))), (meta.get("facets") or {}).get("category", []) + # v2.2.18: a missing count is reported as None, never replaced by the number of items on the page, so a + # listing without the site's own total can never look complete + total = meta.get("totalCount") + return items, (int(total) if total is not None else None), (meta.get("facets") or {}).get("category", []) def products(self, upcs: list[str], verified: bool = True) -> list[dict]: out = [] --- a/scraper/marianos_scraper/storage.py +++ b/scraper/marianos_scraper/storage.py @@ -15,6 +15,15 @@ class LockHeld(Exception): pass + + +class PreconditionFailed(Exception): + """A conditional write found the object changed since it was read (or present when it had to be absent).""" + + +def _etag_of(data: bytes) -> str: + import hashlib + return hashlib.sha256(data).hexdigest() class LocalStorage: @@ -47,6 +56,23 @@ def delete(self, key: str) -> None: self._p(key).unlink(missing_ok=True) + + def read_with_etag(self, key: str) -> tuple[str | None, str | None]: + """Text and ETag (SHA-256 of the bytes) of an object, or (None, None) when it does not exist.""" + path = self._p(key) + if not path.is_file(): + return None, None + data = path.read_bytes() + # text exactly as read_text() returns it (text mode: Windows line endings folded back), ETag over the bytes + return data.decode("utf-8").replace("\r\n", "\n"), _etag_of(data) + + def write_if_match(self, key: str, text: str, etag: str | None) -> None: + """Replace the object only if it is still the one read (etag None: only if it does not exist yet). The run + lock makes the run the only writer; this check also refuses a write over anything changed meanwhile.""" + _, current = self.read_with_etag(key) + if current != etag: + raise PreconditionFailed(f"{key} changed since it was read") + self.write_text(key, text) def list(self, prefix: str) -> list[str]: base = self._p(prefix) @@ -109,6 +135,31 @@ def list(self, prefix: str) -> list[str]: return sorted(b.name for b in self.container.list_blobs(name_starts_with=prefix.rstrip("/") + "/")) + def read_with_etag(self, key: str) -> tuple[str | None, str | None]: + from azure.core.exceptions import ResourceNotFoundError + try: + d = self.container.download_blob(key) + except ResourceNotFoundError: + return None, None + return d.readall().decode("utf-8"), d.properties.etag + + def write_if_match(self, key: str, text: str, etag: str | None) -> None: + """One PUT with If-Match on the ETag read (etag None: If-None-Match *, i.e. create only).""" + from azure.core import MatchConditions + from azure.core.exceptions import HttpResponseError, ResourceExistsError + blob = self.container.get_blob_client(key) + try: + if etag is None: + blob.upload_blob(text.encode("utf-8"), overwrite=False) + else: + blob.upload_blob(text.encode("utf-8"), overwrite=True, etag=etag, match_condition=MatchConditions.IfNotModified) + except ResourceExistsError as exc: + raise PreconditionFailed(f"{key} exists already") from exc + except HttpResponseError as exc: + if getattr(exc, "status_code", None) == 412: + raise PreconditionFailed(f"{key} changed since it was read") from exc + raise + def acquire_lock(self, key: str, stale_after_seconds: int) -> None: """Infinite blob lease = cluster-wide mutex. A lease older than stale_after_seconds is broken (crashed run).""" from azure.core.exceptions import HttpResponseError, ResourceExistsError --- a/scraper/tests/test_scraper.py +++ b/scraper/tests/test_scraper.py @@ -61,10 +61,21 @@ def make_runner(tmp_path, api, **kw): + # v2.2.18: a run limited to some categories is only a sample release, so these API-runner tests use a + # configuration whose whole catalogue is Dairy: a full run of that configuration + import dataclasses loc = CFG.location("marianos-lakeshore-east") CFG.storage_root = str(tmp_path) - runner = Runner(CFG, loc, api=api, storage=LocalStorage(str(tmp_path)), only_categories=["Dairy"], **kw) + cfg = dataclasses.replace(CFG, categories=[c for c in CFG.categories if c.name == "Dairy"]) + runner = Runner(cfg, loc, api=api, storage=LocalStorage(str(tmp_path)), **kw) return runner + + +def latest_text(tmp_path, name="products.json"): + """A file of the published delivery, read through latest//CURRENT.json (None when nothing is published).""" + from marianos_scraper.delivery import Delivery + d = Delivery(LocalStorage(str(tmp_path)), "marianos-lakeshore-east") + return d.read(d.current(), name) @pytest.fixture(autouse=True) @@ -113,7 +124,7 @@ def test_good_run_publishes_and_dedupes(tmp_path): report = make_runner(tmp_path, FakeApi()).run() assert report["status"] == "published" - latest = json.loads((tmp_path / "latest/marianos-lakeshore-east/products.json").read_text(encoding="utf-8")) + latest = json.loads(latest_text(tmp_path)) assert len(latest) == 40 # every Dairy term returns the same 40 products -> deduped assert report["duplicates_removed"] > 0 assert report["validation"]["duplicates_in_output"] == 0 @@ -121,16 +132,16 @@ def test_failed_run_never_replaces_last_good_data(tmp_path): make_runner(tmp_path, FakeApi()).run() - good = (tmp_path / "latest/marianos-lakeshore-east/products.json").read_text(encoding="utf-8") + good = latest_text(tmp_path) report = make_runner(tmp_path, FakeApi(fail_terms={"milk", "cheese", "yogurt"}), fresh=True).run() assert report["status"] == "held_back" - assert (tmp_path / "latest/marianos-lakeshore-east/products.json").read_text(encoding="utf-8") == good + assert latest_text(tmp_path) == good def test_lost_store_context_is_blocked(tmp_path): report = make_runner(tmp_path, FakeApi(unpriced=True)).run() assert report["status"] == "held_back" - assert not (tmp_path / "latest/marianos-lakeshore-east/products.json").exists() + assert latest_text(tmp_path) is None def test_wrong_store_aborts_before_collection(tmp_path): @@ -264,7 +275,7 @@ fallback_reason="site refused", fresh=True).run() assert report["status"] == "held_back" and report["source"].endswith("(fallback)") assert json.loads(storage.read_text("work/marianos-lakeshore-east/current.json"))["run_id"] == "SITE-RUN" - assert not storage.exists("latest/marianos-lakeshore-east/products.json") + assert latest_text(tmp_path) is None def test_refinish_compares_with_the_catalog_from_before_the_run(tmp_path): @@ -436,7 +447,8 @@ return out def all_coupons(self, kind): - return ([COUPON] if kind == "standard" else []), {} + found = [COUPON] if kind == "standard" else [] + return found, {"totalCount": len(found)} # the site states its own total, as the real one does def coupons_for(self, upcs): return [] @@ -475,7 +487,7 @@ out = f"runs/marianos-lakeshore-east/{report['run_id']}/" products = {r["upc"]: r for r in json.loads(storage.read_text(out + "products.json"))} avail = {a["upc"]: a for a in map(json.loads, storage.read_text(out + "availability.jsonl").splitlines())} - latest = {r["upc"] for r in json.loads(storage.read_text("latest/marianos-lakeshore-east/products.json"))} + latest = {r["upc"] for r in json.loads(latest_text(tmp_path))} assert set(products) == latest return report["run_id"], products, avail --- /dev/null +++ b/scraper/tests/test_v2218.py @@ -0,0 +1,501 @@ +"""v2.2.18: one test (or a small group) per finding of the 2026-11-19 review, each reproducing the reviewer's case +offline (LocalStorage, the ChangingStore website double from test_scraper.py, no network, no Azure, no Oxylabs). + + F1 a five-record Dairy run replaced a 145-record full delivery -> TestF1PartialRunNeverReplacesFull + F2 missing totals and overlapping category counts could look complete -> TestF2CompletenessIsProven + F3 a checkpoint could resume under another store -> TestF3CheckpointBinding + F4 a failed coupon write left new products with the old report -> TestF4OneWriterAtomicDelivery + F5 requests were not reserved durably before dispatch -> TestF5DurableReservation + F6 normal collection had no deadline (only the probe had one) -> TestF6CollectionDeadline +""" +from __future__ import annotations + +import dataclasses +import json +import threading +import time + +import pytest + +from test_scraper import CFG, COUPON, ChangingStore, FakeApi, latest_text # noqa: E402 (same tests folder) + +from marianos_scraper.delivery import Delivery, PublishFailed +from marianos_scraper.runner import CheckpointMismatch, Runner, deadline_seconds +from marianos_scraper.site_runner import SiteRunner +from marianos_scraper.storage import LocalStorage, PreconditionFailed + +LOC_KEY = "marianos-lakeshore-east" + + +@pytest.fixture(autouse=True) +def site_thresholds(monkeypatch, tmp_path): + """The thresholds the ChangingStore catalogue (five products per taxonomy root) can meet.""" + for k, val in {"min_records_per_category": 1, "min_ppm_share": 0.0, "min_total_records": 1}.items(): + monkeypatch.setitem(CFG.settings["validation"], k, val) + monkeypatch.setitem(CFG.settings["site_source"], "api_cross_check_sample", 0) + CFG.storage_root = str(tmp_path) + + +def site_runner(storage, site, loc=None, **kw): + return SiteRunner(CFG, loc or CFG.location(LOC_KEY), site=site, api=FakeApi(), storage=storage, **kw) + + +def delivered(storage): + d = Delivery(storage, LOC_KEY) + ptr = d.current() + return ptr, {n: d.read(ptr, n) for n in ("products.json", "coupons.json", "report.json", "catalog.json")} + + +def failed_checks(report): + return {c["check"] for c in report["validation"]["checks"] if not c["passed"]} + + +# --- F1 --------------------------------------------------------------------------------------------------------- +class TestF1PartialRunNeverReplacesFull: + def test_five_record_dairy_run_is_a_sample_and_the_145_record_delivery_stays(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + full = site_runner(storage, ChangingStore()).run() + assert full["status"] == "published" and full["unique_records"] == 145 # the reviewer's full delivery + ptr_before, files_before = delivered(storage) + + dairy = site_runner(storage, ChangingStore(), only_categories=["Dairy"], fresh=True).run() + assert dairy["unique_records"] == 5 # the reviewer's Dairy run + assert dairy["status"] == "sample" and dairy["delivery"]["kind"] == "sample" + assert dairy["delivery"]["release"] == f"samples/{LOC_KEY}/{dairy['run_id']}/" + assert len(json.loads(storage.read_text(dairy["delivery"]["release"] + "products.json"))) == 5 + + ptr_after, files_after = delivered(storage) + assert ptr_after == ptr_before and files_after == files_before # latest/ untouched + assert len(json.loads(files_after["products.json"])) == 145 + assert json.loads(files_after["report.json"])["run_id"] == full["run_id"] + + def test_refinish_judges_a_run_by_its_own_scope(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + site_runner(storage, ChangingStore()).run() + ptr_before, _ = delivered(storage) + dairy = site_runner(storage, ChangingStore(), only_categories=["Dairy"], fresh=True).run() + # refinish without --categories must not turn the Dairy sample into a full delivery + again = site_runner(storage, ChangingStore()).refinish(dairy["run_id"]) + assert again["status"] == "sample" and delivered(storage)[0] == ptr_before + + def test_api_runner_category_subset_is_a_sample_too(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + report = Runner(CFG, CFG.location(LOC_KEY), api=FakeApi(), storage=storage, only_categories=["Dairy"]).run() + assert report["status"] == "sample" and latest_text(tmp_path) is None + + +# --- F2 --------------------------------------------------------------------------------------------------------- +class NoTotalStore(ChangingStore): + """The site answers one Dairy listing without its totalCount.""" + def search(self, location_id, taxonomy, offset=0, size=100, brand="", facets=False, brand_facet=False): + items, total, f = super().search(location_id, taxonomy, offset, size, brand, facets, brand_facet) + if not facets and taxonomy == self.leaf("02"): + return items, None, f + return items, total, f + + +class OverlapStore(ChangingStore): + """Dairy root '02' has two aisles of five products each (site count 10), but both aisles list the SAME five + products: summed per unit that is 10/10 'complete', distinct it is 5 of 10.""" + def search(self, location_id, taxonomy, offset=0, size=100, brand="", facets=False, brand_facet=False): + if taxonomy == "02" and facets: + return [], 10, [{"id": "02", "name": "Dairy", "count": 10, "children": [ + {"id": "02001", "name": "Aisle A", "count": 5, "children": []}, + {"id": "02002", "name": "Aisle B", "count": 5, "children": []}]}] + if taxonomy in ("02001", "02002") and not facets: + ids = self.upcs_of("02001") + return [{"upc": u} for u in ids[offset:offset + size]], 5, {} + return super().search(location_id, taxonomy, offset, size, brand, facets, brand_facet) + + +class ShortCouponStore(ChangingStore): + """The site says 3 digital coupons; the paged list keeps returning only one of them.""" + def __init__(self, total=3): + super().__init__() + self.total, self.passes = total, 0 + + def all_coupons(self, kind): + if kind == "standard": + self.passes += 1 + return [COUPON], ({"totalCount": self.total} if self.total is not None else {}) + return [], {"totalCount": 0} + + +class TestF2CompletenessIsProven: + def test_a_listing_without_the_site_total_is_not_complete(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + report = site_runner(storage, NoTotalStore()).run() + assert report["status"] == "held_back" + assert "site_count_reported_for_every_unit" in failed_checks(report) + assert latest_text(tmp_path) is None + + def test_overlapping_units_are_counted_once(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + report = site_runner(storage, OverlapStore()).run() + dairy = report["category_completeness"]["Dairy"] + assert dairy["ids_listed_incl_overlap"] == 10 and dairy["ids_collected"] == 5 and dairy["coverage"] == 0.5 + assert report["status"] == "held_back" and "category_completeness_vs_site_counts" in failed_checks(report) + + def test_coupons_short_of_the_site_total_are_read_again_then_held_back(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + site = ShortCouponStore(total=3) + report = site_runner(storage, site).run() + assert site.passes == 3 # three passes over the re-sorting list + assert report["coupons"]["site_totals"]["standard"] == 3 + assert report["status"] == "held_back" and "coupons_complete_vs_site_totals" in failed_checks(report) + + def test_coupons_without_a_site_total_are_not_complete(self, tmp_path): + report = site_runner(LocalStorage(str(tmp_path)), ShortCouponStore(total=None)).run() + assert report["status"] == "held_back" and "coupons_complete_vs_site_totals" in failed_checks(report) + + def test_every_product_must_have_been_asked_for_its_coupons(self, tmp_path): + class SkipsLookup(ChangingStore): + def coupons_for(self, upcs): + return [] + storage = LocalStorage(str(tmp_path)) + runner = site_runner(storage, SkipsLookup()) + real = runner._link_coupons + + def lookup_half(unit, t0): # as if the lookup had covered only part of the catalogue + res = real(unit, t0) + res["candidate_upcs"] = res["candidate_upcs"][: len(res["candidate_upcs"]) // 2] + return res + runner._link_coupons = lookup_half + report = runner.run() + assert report["status"] == "held_back" and "coupon_lookup_covered_every_product" in failed_checks(report) + + +# --- F3 --------------------------------------------------------------------------------------------------------- +class OtherStore(ChangingStore): + STORE = {**ChangingStore.STORE, "locationId": "53100502", "facilityId": "15092", "storeNumber": "00502", + "locale": {"address": {"name": "Vernon Hills", "addressLines": ["275 Hawthorn Village Commons"], + "cityTown": "VERNON HILLS", "stateProvince": "IL", "postalCode": "60061"}}} + + +class CountingStore(ChangingStore): + def __init__(self): + super().__init__() + self.sent = 0 + + def store(self, zipcode, location_id): + self.sent += 1 + return super().store(zipcode, location_id) + + def search(self, *a, **kw): + self.sent += 1 + return super().search(*a, **kw) + + +class TestF3CheckpointBinding: + def interrupted(self, storage): + first = site_runner(storage, ChangingStore(), stop_after_units=3).run() + assert first["status"] == "interrupted" + return first, storage.read_text(f"work/{LOC_KEY}/current.json") + + def test_an_unfinished_run_never_resumes_under_another_store(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + first, state_before = self.interrupted(storage) + other = dataclasses.replace(CFG.location(LOC_KEY), location_id="53100502", zipcode="60061", facility_id="15092") + site = CountingStore() + with pytest.raises(CheckpointMismatch, match="not resumed"): + site_runner(storage, site, loc=other).run() + assert site.sent == 0 # nothing sent + assert storage.read_text(f"work/{LOC_KEY}/current.json") == state_before # run state untouched + + def test_a_store_that_answers_as_another_store_on_resume_is_refused(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + self.interrupted(storage) + loose = dataclasses.replace(CFG.location(LOC_KEY), facility_id="") # facility not pinned in config + # binding differs (facility) -> refused before anything is sent + with pytest.raises(CheckpointMismatch): + site_runner(storage, OtherStore(), loc=loose).run() + + def test_another_source_never_resumes_a_website_run(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + self.interrupted(storage) + api = FakeApi() + with pytest.raises(CheckpointMismatch): + Runner(CFG, CFG.location(LOC_KEY), api=api, storage=storage).run() + assert api.calls == 0 + + def test_a_foreign_checkpoint_file_is_not_used(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + first, _ = self.interrupted(storage) + units = f"work/{LOC_KEY}/{first['run_id']}/units/" + key = sorted(k for k in storage.list(units))[1] + res = json.loads(storage.read_text(key)) + storage.write_text(key, json.dumps({**res, "binding_digest": "0123456789abcdef"})) # from another binding + with pytest.raises(CheckpointMismatch, match="not used"): + site_runner(storage, ChangingStore()).run() + assert latest_text(tmp_path) is None + + +# --- F4 --------------------------------------------------------------------------------------------------------- +class FailingStorage(LocalStorage): + """Fails every write of one file name (e.g. the coupon file of a new delivery).""" + def __init__(self, root, fail_name): + super().__init__(root) + self.fail_name = fail_name + + def write_text(self, key, text): + if key.startswith("releases/") and key.endswith(self.fail_name): + raise OSError(f"simulated write failure: {key}") + super().write_text(key, text) + + +class TestF4OneWriterAtomicDelivery: + @pytest.mark.parametrize("fail_name", ["coupons.json", "report.json", "catalog.json"]) + def test_a_failed_write_leaves_the_previous_delivery_whole(self, tmp_path, fail_name): + good = site_runner(LocalStorage(str(tmp_path)), ChangingStore()).run() + ptr_before, before = delivered(LocalStorage(str(tmp_path))) + + site = ChangingStore() + site.absent = {site.upcs_of(site.leaf("02"))[0]} # the new run differs from the old delivery + new = site_runner(FailingStorage(str(tmp_path), fail_name), site, fresh=True).run() + assert new["status"] == "publish_failed" and fail_name in new["delivery"]["error"] + + ptr_after, after = delivered(LocalStorage(str(tmp_path))) + assert ptr_after == ptr_before and after == before # same release, same bytes: nothing mixed + report = json.loads(after["report.json"]) + assert report["run_id"] == good["run_id"] == json.loads(after["catalog.json"])["run_id"] + from marianos_scraper.delivery import sha256_text + assert report["evidence"]["products_sha256"] == sha256_text(after["products.json"]) + assert json.loads(LocalStorage(str(tmp_path)).read_text(f"runs/{LOC_KEY}/{new['run_id']}/report.json"))["status"] \ + == "publish_failed" + + def test_products_coupons_report_and_history_move_together(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + a = site_runner(storage, ChangingStore()).run() + b = site_runner(storage, ChangingStore(), fresh=True).run() + ptr, files = delivered(storage) + assert ptr["run_id"] == b["run_id"] and ptr["previous"]["run_id"] == a["run_id"] + assert [h["run_id"] for h in ptr["history"]] == [a["run_id"], b["run_id"]] + assert json.loads(files["report.json"])["run_id"] == json.loads(files["catalog.json"])["run_id"] == b["run_id"] + assert set(ptr["files"]) == {"products.json", "coupons.json", "report.json", "catalog.json"} + + def test_a_second_writer_cannot_overwrite_the_pointer(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + d = Delivery(storage, LOC_KEY) + files = {n: "{}" for n in ("products.json", "coupons.json", "report.json", "catalog.json")} + d.publish("RUN-A", files, {}) + _, etag = storage.read_with_etag(d.pointer_key) + d.publish("RUN-B", files, {}) # another writer got in between + with pytest.raises(PreconditionFailed): + storage.write_if_match(d.pointer_key, "{}", etag) + assert d.current()["run_id"] == "RUN-B" + + def test_a_changed_release_file_is_never_read(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + site_runner(storage, ChangingStore()).run() + ptr, _ = delivered(storage) + storage.write_text(ptr["release"] + "coupons.json", "[]") + from marianos_scraper.delivery import DeliveryDamaged + with pytest.raises(DeliveryDamaged): + Delivery(storage, LOC_KEY).read(ptr, "coupons.json") + + def test_publish_refuses_a_partial_file_set(self, tmp_path): + with pytest.raises(PublishFailed): + Delivery(LocalStorage(str(tmp_path)), LOC_KEY).publish("R", {"products.json": "[]"}, {}) + + +# --- F5 --------------------------------------------------------------------------------------------------------- +class Killed(BaseException): + """The process dies (power loss, kill -9): nothing below run() gets to save anything.""" + + +class ReservingStore(ChangingStore): + """Every request goes through the transport's reserve hook first, exactly as SiteSession._begin_attempt does; + at each dispatch the test reads the DURABLE run state and checks the request is already reserved there.""" + def __init__(self, storage, kill_after=None): + super().__init__() + self.storage, self.kill_after, self.reserve, self.dispatched = storage, kill_after, None, 0 + self.unreserved_dispatches = [] + + def _send(self, what): + if self.kill_after is not None and self.dispatched >= self.kill_after: + raise Killed() + if self.reserve is not None: + self.reserve(what) + self.stats.requests += 1 + self.dispatched += 1 + state = json.loads(self.storage.read_text(f"work/{LOC_KEY}/current.json")) + if state["site_requests_reserved"] < state["site_requests_base"] + self.stats.requests: + self.unreserved_dispatches.append(what) + + def store(self, zipcode, location_id): + self._send("store") + return super().store(zipcode, location_id) + + def search(self, *a, **kw): + self._send("search") + return super().search(*a, **kw) + + def products(self, upcs, verified=True): + self._send("products") + return super().products(upcs, verified) + + +class TestF5DurableReservation: + def test_every_request_is_reserved_in_the_run_state_before_it_is_sent(self, tmp_path): + storage = LocalStorage(str(tmp_path)) + site = ReservingStore(storage) + report = site_runner(storage, site).run() + assert report["status"] == "published" + assert site.dispatched > 0 and site.unreserved_dispatches == [] + state = json.loads(storage.read_text(f"work/{LOC_KEY}/current.json")) + assert state["site_requests_reserved"] == state["site_requests_used"] == site.dispatched # exact at the end + + def test_a_killed_process_is_counted_at_its_full_reservation_and_the_budget_holds(self, tmp_path, monkeypatch): + monkeypatch.setitem(CFG.settings["site_source"], "request_reservation_block", 10) + storage = LocalStorage(str(tmp_path)) + budget = 60 + first = ReservingStore(storage, kill_after=23) + with pytest.raises(Killed): + site_runner(storage, first, max_requests_per_run=budget).run() + storage.release_lock(f"work/{LOC_KEY}/run.lock") # a dead process's lock is stale + on_disk = json.loads(storage.read_text(f"work/{LOC_KEY}/current.json")) + assert on_disk["site_requests_used"] < first.dispatched <= on_disk["site_requests_reserved"] + + second = ReservingStore(storage) + report = site_runner(storage, second, max_requests_per_run=budget).run() + state = json.loads(storage.read_text(f"work/{LOC_KEY}/current.json")) + assert state["requests_counted_from_reservation"] == on_disk["site_requests_reserved"] - on_disk["site_requests_used"] + assert state["site_requests_used"] >= first.dispatched + second.dispatched # never under-counted + assert first.dispatched + second.dispatched <= budget # the run budget held + assert report["status"] in ("interrupted", "published") + + def test_no_request_is_sent_when_the_reservation_cannot_be_written(self, tmp_path): + class NoStateWrites(LocalStorage): + armed = False + + def write_text(self, key, text): + if self.armed and key.endswith("current.json"): + raise OSError("simulated storage outage") + super().write_text(key, text) + storage = NoStateWrites(str(tmp_path)) + site = ReservingStore(storage) + runner = site_runner(storage, site) + runner._load_or_start_state() + runner._save_state(runner._state) + storage.armed = True + from marianos_scraper.site_source import RequestCapReached + with pytest.raises(RequestCapReached, match="nothing sent"): + site.store("60601", "53100503") + assert site.dispatched == 0 + + def test_the_real_transport_reserves_before_it_counts_and_sends(self): + from marianos_scraper.site_source import SiteSession + order = [] + s = SiteSession("", {**CFG.settings["site_source"], "min_interval_seconds": 0}) + s.reserve = lambda path: order.append(("reserve", s.stats.requests)) + + class Page: + def evaluate(self, js, args): + order.append(("send", s.stats.requests)) + return {"status": 200, "type": "application/json", "body": json.dumps({"data": {"stores": []}}), "ms": 1} + s._page = Page() + s.get("/atlas/v1/stores/v2/locator?filter.query=60601", expect="stores") + assert order == [("reserve", 0), ("send", 1)] + + def test_oxylabs_collection_without_the_allowance_ledger_sends_nothing(self, monkeypatch, tmp_path): + from marianos_scraper import __main__ as cli + monkeypatch.setenv("SCRAPER_DATA_DIR", str(tmp_path)) + monkeypatch.setattr(cli, "build_site", lambda *a, **k: pytest.fail("no transport may be built")) + assert cli.main(["run", "--location", LOC_KEY, "--transport", "oxylabs", "--source", "site"]) == 2 + + def test_oxylabs_collection_reserves_jobs_in_the_ledger_before_the_first_one(self, monkeypatch, tmp_path): + from marianos_scraper import __main__ as cli + ledger = tmp_path / "ledger.json" + seen = {} + + class Ledgered(ChangingStore): + transport_name = "oxylabs" + + def store(self, zipcode, location_id): + if "at_first_job" not in seen: + seen["at_first_job"] = json.loads(ledger.read_text(encoding="utf-8")) + self.stats.requests += 1 + return super().store(zipcode, location_id) + site = Ledgered() + monkeypatch.setenv("SCRAPER_DATA_DIR", str(tmp_path / "data")) + monkeypatch.setenv("KROGER_CLIENT_ID", "") + monkeypatch.setattr(cli, "build_site", lambda *a, **k: site) + code = cli.main(["run", "--location", LOC_KEY, "--transport", "oxylabs", "--source", "site", + "--max-requests", "7", "--ledger", str(ledger), "--ledger-ceiling", "15", + "--categories", "Dairy"]) + entry = seen["at_first_job"]["executions"][0] + assert entry["status"] == "reserved" and entry["reserved_jobs"] == 7 and site.max_requests == 7 + final = json.loads(ledger.read_text(encoding="utf-8"))["executions"][0] + assert final["status"] == "final" and final["oxylabs_jobs_submitted"] == site.stats.requests + assert code in (0, 1, 4) + + +# --- F6 --------------------------------------------------------------------------------------------------------- +class SlowStore(ChangingStore): + def search(self, location_id, taxonomy, offset=0, size=100, brand="", facets=False, brand_facet=False): + if not facets: + time.sleep(0.05) # each unit's listing is slow; planning is not + return super().search(location_id, taxonomy, offset, size, brand, facets, brand_facet) + + +class TestF6CollectionDeadline: + def test_normal_collection_gets_the_deadline_without_any_option(self): + from marianos_scraper.site_source import SiteSession + s = SiteSession("", dict(CFG.settings["site_source"])) + assert s.max_seconds is None # nothing on the command line + runner = SiteRunner(CFG, CFG.location(LOC_KEY), site=s, api=FakeApi(), storage=object()) + assert s.max_seconds == runner.deadline == deadline_seconds(CFG.settings) == 18000 + + def test_every_request_has_a_timeout_even_without_a_runtime_cap(self): + from marianos_scraper.site_source import SiteSession + s = SiteSession("", {**CFG.settings["site_source"], "min_interval_seconds": 0}) + sent = [] + + class Page: + def evaluate(self, js, args): + sent.append(args[2]) + return {"status": 200, "type": "application/json", "body": json.dumps({"data": {"stores": []}}), "ms": 1} + s._page = Page() + s.get("/atlas/v1/stores/v2/locator?filter.query=60601", expect="stores") + assert sent == [90000] # v2.2.17 sent 0 = no limit when no --max-seconds was given + + def test_a_site_run_stops_resumably_at_the_deadline(self, tmp_path, monkeypatch): + monkeypatch.setitem(CFG.settings, "collection", {"deadline_seconds": 0.5}) + storage = LocalStorage(str(tmp_path)) + report = site_runner(storage, SlowStore()).run() + assert report["status"] == "interrupted" + done = storage.list(f"work/{LOC_KEY}/{report['run_id']}/units/") + assert 0 < len(done) < 30 and latest_text(tmp_path) is None + monkeypatch.setitem(CFG.settings, "collection", {"deadline_seconds": 600}) + resumed = site_runner(storage, SlowStore()).run() # same transport: the run is resumed + assert resumed["run_id"] == report["run_id"] and resumed["status"] == "published" + + def test_an_api_run_stops_at_the_deadline_between_units(self, tmp_path, monkeypatch): + import dataclasses as dc + + class SlowApi(FakeApi): + def search_page(self, term, location_id, start): + time.sleep(0.05) + return super().search_page(term, location_id, start) + monkeypatch.setitem(CFG.settings, "collection", {"deadline_seconds": 0.2}) + cfg = dc.replace(CFG, categories=[c for c in CFG.categories if c.name == "Dairy"]) + report = Runner(cfg, CFG.location(LOC_KEY), api=SlowApi(), storage=LocalStorage(str(tmp_path))).run() + assert report["status"] == "interrupted" + + def test_the_retry_wait_stops_at_the_deadline(self, monkeypatch): + monkeypatch.setitem(CFG.settings, "collection", {"deadline_seconds": 10}) + runner = Runner(CFG, CFG.location(LOC_KEY), api=FakeApi(), storage=object()) + from marianos_scraper.runner import Interrupted + with pytest.raises(Interrupted): + runner._wait(30) # v2.2.17 slept 30 s regardless + + def test_a_stuck_call_is_ended_by_the_watchdog(self): + from marianos_scraper.__main__ import start_watchdog + fired = threading.Event() + codes = [] + start_watchdog(0.2, exit=lambda code: (codes.append(code), fired.set())) + assert fired.wait(3) and codes == [4] + + def test_the_deadline_stays_below_the_stale_lock_age(self): + with pytest.raises(ValueError): + deadline_seconds({"collection": {"deadline_seconds": 6 * 3600}}) --- /dev/null +++ b/scraper/tools/mutants_2218.py @@ -0,0 +1,87 @@ +"""Put each v2.2.17 behaviour back, one at a time, in a copy of the package and check that the test class for that +finding FAILS (a test that still passes with the bug back proves nothing). + python tools/mutants_2218.py (from the scraper folder; offline, works on a temporary copy)""" +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +SRC = Path(__file__).resolve().parents[1] +MUTANTS = [ + ("F1 limited run treated as full", "marianos_scraper/site_runner.py", + " limited = self.limited_reasons()\n catalog, prev_run", " limited = []\n catalog, prev_run", + "TestF1PartialRunNeverReplacesFull and not api_runner"), + ("F1 api subset treated as full", "marianos_scraper/runner.py", + " limited = self.limited_reasons()\n previous = None if limited", " limited = []\n previous = None if limited", + "test_api_runner_category_subset_is_a_sample_too"), + ("F2 missing total read as page length", "marianos_scraper/site_runner.py", + " if total is None:\n total_missing = True", " if total is None:\n total = len(items)", + "test_a_listing_without_the_site_total_is_not_complete"), + ("F2 category counts summed with overlap", "marianos_scraper/site_runner.py", + " found = len(cat_ids.get(cat, ()))", " found = cat_listed.get(cat, 0)", + "test_overlapping_units_are_counted_once"), + ("F2 coupon totals not checked", "marianos_scraper/site_runner.py", + " if site_total is None or have < site_total:", " if False:", + "coupons_short or coupons_without"), + ("F2 coupon lookup coverage not checked", "marianos_scraper/site_runner.py", + " not_looked_up = sorted(", " not_looked_up = [] and sorted(", + "test_every_product_must_have_been_asked_for_its_coupons"), + ("F3 binding not checked on resume", "marianos_scraper/runner.py", + " if state.get(\"binding\") != self.binding():", " if False:", + "TestF3CheckpointBinding and not foreign"), + ("F3 unit checkpoint binding not checked", "marianos_scraper/runner.py", + " if result.get(\"binding_digest\") != state.get(\"binding_digest\"):", " if False:", + "test_a_foreign_checkpoint_file_is_not_used"), + ("F4 pointer written before the files", "marianos_scraper/delivery.py", + " hashes = self._write_verified(prefix, files)\n", " hashes = {n: sha256_text(t) for n, t in files.items()}\n", + "test_a_failed_write_leaves_the_previous_delivery_whole"), + ("F4 pointer write not conditional", "marianos_scraper/storage.py", + " if current != etag:\n raise PreconditionFailed(f\"{key} changed since it was read\")\n self.write_text(key, text)", + " self.write_text(key, text)", "test_a_second_writer_cannot_overwrite_the_pointer"), + ("F5 no reservation before dispatch", "marianos_scraper/site_source.py", + " self.reserve(path) # durable", " pass # self.reserve(path) # durable", + "test_the_real_transport_reserves_before_it_counts_and_sends"), + ("F5 runner does not reserve", "marianos_scraper/site_runner.py", + " self.site.reserve = self._reserve_request\n", "\n", + "test_every_request_is_reserved or killed_process"), + ("F5 crash not counted at the reservation", "marianos_scraper/site_runner.py", + " used = reserved\n", "\n", "killed_process"), + ("F5 oxylabs run without a ledger", "marianos_scraper/__main__.py", + " if transport == \"oxylabs\" and source != \"api\" and not args.ledger:", " if False:", + "without_the_allowance_ledger"), + ("F6 no default deadline", "marianos_scraper/site_runner.py", + " self.site.max_seconds = self.deadline", " pass", + "test_normal_collection_gets_the_deadline_without_any_option"), + ("F6 request without timeout", "marianos_scraper/site_source.py", + " limit = self._request_limit(self.request_timeout)", " limit = self._request_limit(None)", + "test_every_request_has_a_timeout_even_without_a_runtime_cap"), + ("F6 no deadline between units", "marianos_scraper/runner.py", + " if self.time_left() <= 0:", " if False:", + "stops_at_the_deadline"), + ("F6 retry wait ignores the deadline", "marianos_scraper/runner.py", + " if seconds >= self.time_left():\n raise Interrupted(\"cap\")", " pass", + "test_the_retry_wait_stops_at_the_deadline"), +] + +bad = 0 +for name, rel, old, new, select in MUTANTS: + dst = Path(tempfile.mkdtemp(prefix="mut2218-")) / "scraper" + shutil.copytree(SRC, dst, ignore=shutil.ignore_patterns("__pycache__", ".pytest_cache", "data", "logs")) + p = dst / rel + raw = p.read_bytes().decode("utf-8") + crlf = "\r\n" in raw + s = raw.replace("\r\n", "\n") + if s.count(old) != 1: + print(f"!! {name}: pattern found {s.count(old)} times"); bad += 1; continue + s = s.replace(old, new) + p.write_bytes((s.replace("\n", "\r\n") if crlf else s).encode("utf-8")) + r = subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", "tests/test_v2218.py", "-k", select], + cwd=dst, capture_output=True, text=True) + tail = [l for l in r.stdout.splitlines() if " passed" in l or " failed" in l or "error" in l.lower()][-1:] + caught = r.returncode == 1 and "failed" in (tail[0] if tail else "") + bad += not caught + print(f"{'CAUGHT ' if caught else 'MISSED!'} {name}: {tail[0] if tail else r.stdout[-300:]}") + shutil.rmtree(dst.parent, ignore_errors=True) +print("ALL MUTANTS CAUGHT" if not bad else f"{bad} NOT CAUGHT") +sys.exit(1 if bad else 0)