# v2.2.18: corrections from the review of 2026-11-19 Offline only. No request was sent to marianos.com, Kroger or Oxylabs to build or test this version. The VM and the schedules stayed off, and the remaining allowance was not used. The operator tools r3 and the v2.2.17 package stay unchanged, and their published hashes still hold. v2.2.18 is a separate package. Tests: 164 pass (the 132 earlier tests plus 32 new ones in `tests/test_v2218.py`; same Python, no network). I also ran a mutation check. Each v2.2.17 behaviour was put back one at a time, 18 in all, and in every case the test for that finding failed. The script is `tools/mutants_2218.py`. ## Finding, change, test ### F1. A five-record Dairy run replaced a 145-record full delivery **Cause.** `--categories` only skipped the yield comparison. A partial run that passed its own checks was written to `latest/` like a full run. **Change.** - Only a full run can replace the delivery. A run counts as full only when it has no category subset, uses the full coupon lookup and is not a fallback run. - Any other run that passes is released as a sample under `samples///`, and `latest/` stays untouched. The report shows this as `status: sample`, `delivery.kind: sample` and `delivery.limited_by`. - The scope is read from the run's own recorded binding. A refinish without `--categories` therefore cannot turn a sample into a full delivery. - The API runner follows the same rule. - Exit code 0 means published, or a passed sample. **Tests (`TestF1PartialRunNeverReplacesFull`).** - `test_five_record_dairy_run_is_a_sample_and_the_145_record_delivery_stays` is your case: a 145-record full run, then a 5-record Dairy run. The pointer and all four delivery files are byte-identical before and after. - `test_refinish_judges_a_run_by_its_own_scope` - `test_api_runner_category_subset_is_a_sample_too` ### F2. Missing totals and overlapping category counts could look complete **Cause.** - When the site omitted `totalCount`, `search()` used the number of items on the page, so the unit looked complete. - Category coverage added up the per-unit counts. A product listed by two units of the same category (a node and its child, or two aisles) was counted twice. - Coupons were not checked against the site's totals. **Change.** - A missing total is recorded as missing (`api_total: null`, `site_total_reported: false`). A missing root or node count stops planning. A missing unit count fails the new check `site_count_reported_for_every_unit`. - Category coverage counts distinct product ids. The report shows both `ids_collected` (distinct) and `ids_listed_incl_overlap`. - Coupons have two new checks: - `coupons_complete_vs_site_totals`: for each coupon type (standard, cash back), the distinct coupons collected must reach the site's own `totalCount`. The paged list re-sorts between pages, so it is read up to three times and the passes are combined by coupon id. A missing total fails the check. - `coupon_lookup_covered_every_product`: every distinct product must have been asked for its coupons. **Tests (`TestF2CompletenessIsProven`).** - `test_a_listing_without_the_site_total_is_not_complete` - `test_overlapping_units_are_counted_once`: two aisles, each listing the same 5 products, with a site count of 10. Coverage is 0.5, so the run is held back. Before this change the sum gave 10/10. - `test_coupons_short_of_the_site_total_are_read_again_then_held_back` - `test_coupons_without_a_site_total_are_not_complete` - `test_every_product_must_have_been_asked_for_its_coupons` ### F3. A checkpoint could resume under another store **Cause.** The run state, the plan and the unit checkpoints were keyed only by the store key. An API run, a changed store configuration or a different transport would resume them. **Change.** - Every run records a binding: source, transport, store key, location id, facility id, ZIP, chain, scope and coupon lookup scope. The binding, or its digest, is stored in `current.json`, `plan.json` and every unit checkpoint. - A run resumes only under exactly the same binding, and a checkpoint is used only if its digest matches. Otherwise `CheckpointMismatch` is raised: nothing is sent, the state is untouched, and the exit code is 6. The wrappers do not retry exit 6; a person resumes with the same settings or starts with `--fresh`. - On resume, the store the site answers for (location id, facility, ZIP) must also equal the store recorded at the first start. - Runs started by v2.2.17 have no binding, so they are not resumed. Start them over with `--fresh`. **Tests (`TestF3CheckpointBinding`).** - `test_an_unfinished_run_never_resumes_under_another_store`: nothing is sent and the state is byte-identical. - `test_a_store_that_answers_as_another_store_on_resume_is_refused` - `test_another_source_never_resumes_a_website_run` - `test_a_foreign_checkpoint_file_is_not_used` ### F4. A failed coupon write left new products with the old report **Cause.** `latest/products.json`, `coupons.json` and `report.json`, and `state/catalog.json`, were four separate writes. **Change.** There is now one writer, `marianos_scraper/delivery.py`. 1. Products, coupons, report and catalog (the store's observation history) are written to a new folder, `releases////`, which is never reused. Each file is read back. 2. Only then is the single pointer `latest//CURRENT.json` replaced, in one conditional write. Locally the condition is the SHA-256 of the pointer that was read; on Blob it is If-Match on its ETag. The pointer names the folder, the SHA-256 of each file, the release it replaced and the release history. A failure at any step leaves the pointer on the previous release, whose files were never touched. Readers go through the pointer and check each hash, so a mixed delivery cannot be read. A failed publish reports `status: publish_failed` and exits with code 1. Stores written by v2.2.17 are read from their old files as the baseline until the first v2.2.18 publish. The old files are never written again. **Tests (`TestF4OneWriterAtomicDelivery`).** - `test_a_failed_write_leaves_the_previous_delivery_whole`: your case, a coupon write that fails, plus the same with `report.json` and `catalog.json`. The pointer and all four files are byte-identical to the previous delivery. - `test_products_coupons_report_and_history_move_together` - `test_a_second_writer_cannot_overwrite_the_pointer` - `test_a_changed_release_file_is_never_read` - `test_publish_refuses_a_partial_file_set` ### F5. Requests are now reserved durably before they are sent **Cause.** The run's request count was saved only after each finished unit. A killed process lost the count of the requests in the unit it was working on. **Change.** - The transport calls a reservation hook immediately before it counts and sends each attempt (`SiteSession.reserve`). The run reserves requests in blocks in `current.json` (`site_requests_reserved`, `request_reservation_block`, default 25) and reads the write back. A reservation never goes beyond `--max-requests-per-run`. If the reservation cannot be written or read back, nothing is sent and the run stops resumably. - After a clean stop the exact count replaces the reservation. After a crash the whole reservation counts as sent (`requests_counted_from_reservation`). - `--transport oxylabs` collection now needs `--ledger`: the jobs are reserved in the same shared allowance ledger the probe uses, with the same lease, If-Match and read-back, before the first job is sent. Without `--ledger` nothing is built or sent (exit 2). The wrappers have new options: `-Ledger` and `MARIANOS_LEDGER`. **Tests (`TestF5DurableReservation`).** - `test_every_request_is_reserved_in_the_run_state_before_it_is_sent` - `test_a_killed_process_is_counted_at_its_full_reservation_and_the_budget_holds` - `test_no_request_is_sent_when_the_reservation_cannot_be_written` - `test_the_real_transport_reserves_before_it_counts_and_sends` - `test_oxylabs_collection_without_the_allowance_ledger_sends_nothing` - `test_oxylabs_collection_reserves_jobs_in_the_ledger_before_the_first_one` ### F6. Normal collection now has a deadline, not only the probe **Cause.** - Without `--max-seconds`, a collection process had no runtime cap. - The browser `fetch()` was sent with no time limit. - The retry pause between passes was a fixed 30 s sleep. **Change.** - Every collection process gets `collection.deadline_seconds` (default 18,000 s = 5 h; `--max-seconds` can only shorten it). The value must stay below the 6 h stale-lock age, so a live run's lock never looks stale. - Every request has its own timeout (`request_timeout_seconds`, default 90 s), whether or not a cap is set. - The deadline is checked before every unit, and the retry pause stops at it. Either way the run stops resumably (exit 4). - A watchdog stops a process that is still running 300 s after its deadline (exit 4; the stuck request is already inside a reservation). **Tests (`TestF6CollectionDeadline`).** - `test_normal_collection_gets_the_deadline_without_any_option` - `test_every_request_has_a_timeout_even_without_a_runtime_cap` - `test_a_site_run_stops_resumably_at_the_deadline`: also resumes the same run afterwards. - `test_an_api_run_stops_at_the_deadline_between_units` - `test_the_retry_wait_stops_at_the_deadline` - `test_a_stuck_call_is_ended_by_the_watchdog` - `test_the_deadline_stays_below_the_stale_lock_age` ## Limits stated plainly - All of this is tested offline with doubles of the site, storage and ledger. Live behaviour will only show on an approved run. - A live run whose coupon list never reaches the site's `totalCount` after three passes is now held back, by design. - An API fallback run is always a sample now. The setting `publish_fallback_runs` can no longer put one into `latest/`. - The probe and the operator tools are unchanged.