Recent runs
The most recent real runs, newest first: runs recorded in the Nook repo, plus one direct Best Buy tool run with no model. Each one is a real quote or read on a real store. Nothing was bought in any of them. Steps are condensed from the recorded evidence and scrubbed for this public page.
| Store | The shopper said | Result | Price | Model | Steps | When |
|---|---|---|---|---|---|---|
| Amazon | “USB-C to USB-C charging cable, 6 ft, quantity 1, under $15” | QuotedReached order review | $5.38 | typesafe-ai/jev | 4 | Oct 26, 4:12 PM |
| Best Buy | “USB-C to USB-C charging cable, 6 ft, quantity 1, under $15” | Product page onlyFound product, no cart or checkout | $14.95 before tax | No assistant (direct tool calls) | 4 | Oct 26, 4:45 PM |
| Amazon | “Re-run of the last DJI request from the run below, on the exact kit the person had chosen” | QuotedReady quote | $87.10 | No assistant (direct engine check) | 7 | Oct 25, 9:12 PM |
| Amazon | “can you get me the dji mic mini thing from amazon” | Blockedproduct_mismatch ×5 | $35.00 and $79.00 listings | DeepSeek V4.1 Flash | 7 | Oct 25, 8:21 PM |
| Instacart | “Regular Coca-Cola (exact message not in the repo record)” | Needs the shopperproduct_choice_requires_user | No total | DeepSeek V4.1 Flash | 5 | Oct 25 |
| Amazon | “Crest toothpaste (exact message not in the repo record)” | QuotedReady quote | $4.33 | DeepSeek V4.1 Flash | 6 | Oct 25, 7:55 PM |
| Uber Eats | “Status of an earlier Uber Eats order (read-only continuation; exact message not in the repo record)” | Status readDelivered (read) | n/a (read only) | DeepSeek V4.1 Flash | 5 | Oct 25, 7:57 PM |
Sources: the Best Buy run is our own run record; the others come from the Nook repo's changelog, its lab integration notes and its Amazon identity log at main 8ca92315. The raw per-event trace files are kept in the lab's private run history and are not published; IDs, accounts, addresses and card details are removed.
Charts
Every chart here is built from real records: aggregate counts over the Nook quote lab's 458 retained runs (596 assistant runs, 527 quotes, Sep 26 to Oct 25, 2026 PDT), Nook's changelog and the store list. Only counts, medians and ranges are shown; no run IDs, prompts, accounts or addresses.
Lab runs per day
Quote tests started in the Nook quote lab, by day.
Run outcomes
Final verdict for each assistant run.
- Quote ready16828%
- Quote blocked24040%
- Asked the person9516%
- No quote6210%
- Other315%
Quotes by store
Ready versus blocked quotes at each of the six quoting stores.
- Amazon
101 ready 101/188 - Uber Eats
34 ready 34/131 - Instacart
23 ready 23/102 - Walmart
13 ready 13/56 - Chewy
10 ready 10/27 - Target
5 ready 5/24
Why quotes were blocked
Top ten blocker codes. Most are Nook stopping to ask, not errors.
- product_choice_requires_user140
- product_mismatch19
- checkout_link_missing12
- item_dialog_missing12
- below_store_minimum12
- store_not_found12
- item_choices_required11
- quote_failed10
- over_budget10
- checkout_not_ready9
Time to a quote
Seconds from quote request to a finished quote, by store.
- Walmart n=5434.3s · 74.5s
- Amazon n=18834.6s · 60.7s
- Instacart n=10139.3s · 49.6s
- Uber Eats n=12946.2s · 117.8s
- Target n=2356.8s · 115.7s
- Chewy n=2763.1s · 90.3s
Where the time goes
Median seconds into a quote when each step was reached.
- 8.2sBrowser ready
- 16.2sSigned in
- 23.3sSearch results
- 26.4sProduct chosen
- 39.7sCheckout read
- 43.9sQuote checked
- 49.2sCart restored
Medians over the quotes that reached each stage (n from 196 to 548), so later stages are measured on fewer quotes.
Trace events per run
How many events one assistant run records (requests, tool calls, model turns, VM steps).
Assistant models
Runs, trace size, time and model cost by assistant model.
| Assistant model | Runs | Median trace events | Median time | Median model cost per run |
|---|---|---|---|---|
| GPT-5.6 Sol | 381 | 77 | 47.2s | $0.0639 |
| DeepSeek V4.1 Flash | 129 | 95 | 64s | $0.0068 |
| GPT-5.6 Luna | 51 | 105 | 78.9s | $0.0047 |
| Claude Opus 4.8 | 10 | 85.5 | 64.4s | $0.0924 |
| Grok 4.7 | 10 | 82 | 56s | $0.0116 |
| Kimi K3 | 9 | 84 | 58.4s | $0.0295 |
Models with 8 or more runs. Model cost is the provider usage reported per run; it excludes the cloud browser.
Quoted totals
Ready quote totals at each store, cents included (tax, fees and tip as checkout showed).
- Amazon n=101$11.00
- Uber Eats n=34$26.80
- Instacart n=23$26.80
- Walmart n=13$13.10
- Chewy n=10$9.30
- Target n=5$40.50
Listing price vs quoted total
Two documented runs from Oct 25.
- DJI Mic Mini (2 TX + 1 RX) Oct 25, 9:12 PM$79.00 listing+$8.10$87.10 quoted · +10.3%
- Crest Pro-Health toothpaste 4.3 oz Oct 25, ~7:59 PM$3.93 listing+$0.40$4.33 quoted · +10.2%
The listing price is what the product page showed; the quoted total is what checkout showed, including tax. In both runs the difference is tax only.
Store coverage
Which stores connect, and which also quote and buy.
- AmazonConnect · quote · buy
- Uber EatsConnect · quote · buy
- InstacartConnect · quote · buy
- WalmartConnect · quote · buy
- TargetConnect · quote · buy
- ChewyConnect · quote · buy
- DoorDashConnect only
- ResyConnect only
- DETOUREConnect · quote
- DeltaConnect only
- AppleConnect only
- Urban OutfittersConnect only
Plus 28 sites Nook set up from their web address, connect only (Skyscanner also offers flight search and price alerts).
Test suites passing
Tests passing in each Nook suite at the last full check.
- Cloud436/436
- Lab170/170
- Service140/140
- Onboarding18/18
Changelog activity
Entries per day in Nook's changelog (BeUI heat-calendar style).
- 29Sep 17
- 18Sep 18
- 14Sep 22
- 38Sep 23
- 3Sep 25
- 28Sep 26
- 42Sep 27
- 28Sep 28
- 13Sep 29
- 24Oct 2
- 8Oct 25
245 changelog entries on 11 days with any entry. Days with none are left out.
Amazon
Reached order reviewReal · Oct 26, 2026, 4:12 PM PDT (ran 41.2 s)Signed-in run that found the right cable and reached Amazon's order review page at $5.38, under the $15 cap. It stopped there; nothing was bought. Score 1.0 (4 of 4 checks passed).
“USB-C to USB-C charging cable, 6 ft, quantity 1, under $15”
- start0 s
Signed-in cloud Chrome session
- searchno time in record
Right item found: Amazon Basics USB-C to USB-C cable, 6 ft
- checkno time in record
Reached order review: $5.38 total vs $15 cap
- done41.2 s
Stopped before payment; nothing bought
A time is shown only where the record has one (seconds are from the start of that run).
Reached order reviewSigned-in run that found the right cable and reached Amazon's order review page at $5.38, under the $15 cap. It stopped there; nothing was bought. Score 1.0 (4 of 4 checks passed).
Best Buy
Found product, no cart or checkoutReal · Oct 26, 2026, 4:45 PM PDT (ran 35.9 s)Stopped at the product page. Never reached the cart or order review, unlike the Amazon run. Not signed in, no model.
Signed-out run that found a 6 ft USB-C to USB-C cable listed at $14.95 before tax, under the $15 cap, with free shipping by Oct 30. The Pomerado tool has no cart step, so nothing was added to cart, no order total was seen, and nothing was bought. The checks below cover only what a product page can show.
“USB-C to USB-C charging cable, 6 ft, quantity 1, under $15”
- start0 s
Pomerado Best Buy tool, signed out, shipping ZIP set
- search20.0 s
20 listings; right item found: NOCO XGrid XC2 240W USB-C to USB-C cable, 6 ft, $14.95
- read35.9 s
Product details: SKU 10555185, $14.95, Shipping FREE Fri, Oct 30; pickup not listed
- done35.9 s
Stopped at product quote; no cart step in this tool; nothing bought
A time is shown only where the record has one (seconds are from the start of that run).
Found product, no cart or checkoutStopped at the product page. Never reached the cart or order review, unlike the Amazon run. Not signed in, no model.
Signed-out run that found a 6 ft USB-C to USB-C cable listed at $14.95 before tax, under the $15 cap, with free shipping by Oct 30. The Pomerado tool has no cart step, so nothing was added to cart, no order total was seen, and nothing was bought. The checks below cover only what a product page can show.
Amazon
Ready quoteReal · Oct 25, 2026, 9:12 PM PDTThe same DJI Mic Mini (2 TX + 1 RX) that production had blocked as product_mismatch, re-quoted with the PR #158 title-alias fix. Quote-only; nothing bought.
“Re-run of the last DJI request from the run below, on the exact kit the person had chosen”
- start0 s
Fresh cloud VM; the lab's managed Amazon session restored
- readno time in record
Product page read: full document title extends the short visible heading; page address, canonical address and Buy/Add form agree on the chosen kit, so one title alias is recorded
- cartno time in record
Exact chosen kit, quantity 1
- checkno time in record
Checkout title matched the alias; quantity, seller, charges, requested address and requested card all matched
- quote38.8 s
Quote published: DJI Mic Mini (2 TX + 1 RX) $79.00 + tax $8.10 = $87.10
- cleanup47.4 s
Original empty cart restored; bought: 0
- done49.1 s
Session saved, VM closed
A time is shown only where the record has one (seconds are from the start of that run).
Ready quoteThe same DJI Mic Mini (2 TX + 1 RX) that production had blocked as product_mismatch, re-quoted with the PR #158 title-alias fix. Quote-only; nothing bought.
Amazon
product_mismatch ×5Real · Oct 25, 2026, 8:21 PM PDT (ran 27 min 7 s; traced 9:03 PM PDT)384 trace events. The assistant asked Nook for quotes on two DJI Mic Mini kits. All five quotes were blocked as product_mismatch: the checkout showed a longer product title than the product page, and no product ID at checkout. Nothing bought.
“can you get me the dji mic mini thing from amazon”
- requestno time in record
Person: “can you get me the dji mic mini thing from amazon”
- listingno time in record
Kit seen: DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for Smartphone, $35.00
- listingno time in record
Kit seen: DJI Mic Mini (2 TX + 1 RX), Ultralight, Detail-Rich Audio, $79.00
- checkno time in record
Checkout (2 TX + 1 RX): same wording plus extra descriptive text, 1 unit, $79 + $8.10 tax; no product ID readable
- blockno time in record
Five quotes across the two kits: product_mismatch (exact-title rule)
- cleanupno time in record
All five cart restorations recorded
- replyno time in record
Assistant explained it as a same-product-ID issue and a transient error worth retrying; the record supports neither
A time is shown only where the record has one (seconds are from the start of that run).
product_mismatch ×5384 trace events. The assistant asked Nook for quotes on two DJI Mic Mini kits. All five quotes were blocked as product_mismatch: the checkout showed a longer product title than the product page, and no product ID at checkout. Nothing bought.
Instacart
product_choice_requires_userReal · Oct 25, 2026 (start time not recorded, before 8:04 PM PDT)Reached real sign-in and search, then stopped and asked the person to choose: the observed listing was a 24-pack and Nook could not tell it was the requested regular variant. No cart add, no purchase.
“Regular Coca-Cola (exact message not in the repo record)”
- requestno time in record
Request included the saved address and default card
- sessionno time in record
Real Instacart sign-in restored
- searchno time in record
Search returned “Coca-Cola Soda (24 x 12 fl oz)”
- blockno time in record
Quote blocked: product_choice_requires_user (fit to the requested regular variant unclear)
- replyno time in record
Assistant said the title did not confirm the amount, and read preferences.mode:none as “no saved address or card”; both went beyond the evidence
A time is shown only where the record has one (seconds are from the start of that run).
product_choice_requires_userReached real sign-in and search, then stopped and asked the person to choose: the observed listing was a 24-pack and Nook could not tell it was the requested regular variant. No cart add, no purchase.
Amazon
Ready quoteReal · Oct 25, 2026, 7:55 PM PDT (ran 1 min 9 s)Quote-only production run through the lab's new CLI/API. Complete quote with the account's selected address and card, cart cleaned up, no warnings.
“Crest toothpaste (exact message not in the repo record)”
- startno time in record
Test started from the lab CLI/API: production, quote only
- quoteno time in record
Checkout read: Crest Pro-Health Whitening Gel Toothpaste (4.3 oz) × 1
- checkno time in record
Address and card: the account's selected ones, as requested
- quoteno time in record
$3.93 item + $0.00 shipping + $0.40 tax = $4.33
- replyno time in record
Assistant reported the card the checkout actually showed instead of switching to its remembered preference
- cleanupno time in record
cleanup: done, no warnings; assistant VM stopped
A time is shown only where the record has one (seconds are from the start of that run).
Ready quoteQuote-only production run through the lab's new CLI/API. Complete quote with the account's selected address and card, cart cleaned up, no warnings.
Uber Eats
Delivered (read)Real · Oct 25, 2026, 7:57 PM PDT (ran 47 s)Read-only check of an Oct 2 order. One status request; the order showed as delivered. No new quote or purchase.
“Status of an earlier Uber Eats order (read-only continuation; exact message not in the repo record)”
- startno time in record
Read-only continuation of an earlier purchase
- statusno time in record
One order-status request sent to Nook
- read7:58:09 PM
Order status: delivered (read from Uber's concluded-order view; scope: delivery status; receipt not re-checked)
- replyno time in record
Assistant kept the Oct 2 receipt and ETA separate from the fresh delivery reading and reported no current ETA
- doneno time in record
Assistant VM stopped; zero quote or purchase requests
A time is shown only where the record has one (seconds are from the start of that run).
Delivered (read)Read-only check of an Oct 2 order. One status request; the order showed as delivered. No new quote or purchase.
Test flight
One exact-prompt campaign run on Oct 25, 2026 against real store accounts. Each case was run once with DeepSeek V4.1 Flash and once with GPT-5.6 Luna. One sample per case, shared real accounts, not a model ranking.
Each cell groups the reviewer's outcome label into one of five plain buckets. The reviewer's own label is shown next to it on the run page. "Quoted correctly" counts quotes the reviewer did not fault; the record shows 11 runs reaching a ready quote, some with caveats the reviewer flagged.
- Prompts and expected outcomes were frozen in c3d38e2 before any run; all 30 prompt hashes match.
- Each case shows the real prompt, scrubbed: tip amounts, saved-address labels, addresses, store locations, card brands and names are removed and marked inline.
- Runs went one after another on shared real store accounts. One sample per case and model: not a model ranking or an independent-account benchmark.
- Outcome labels and notes were written by a reviewer reading each run's record. They are not graded by code.
- Prices and availability are what the run saw at the time, not current offers.
- All assistant VMs stopped; every run with cart activity has cart-restore steps; zero unresolved quotes.
Source: the campaign results file in the Nook lab (regressions, 2026-10-25 exact prompts), read only. Run IDs, replay links, store locations, saved-address labels, card brands and tip amounts are left out; removals are marked inline in the prompts. Catalog product numbers (ASINs, Instacart and Walmart IDs) are published.
A1 · Amazon
DeepSeek V4.1 Flash
partialReal · Oct 25, 2026$4.33 proposal; usual tube unverified
ugh we're out of toothpaste AGAIN. can u grab the crest one we always get, the white tube not the kids one. just 1
- Crest Pro-Health Whitening Gel Toothpaste (4.3oz)ASIN
B0BQ1M98FB· quoted
Adult Crest gel, quantity one, $4.33 with reconciled charges and the saved destination and payment matched. The assistant presented it as a proposed gel and asked before going further, but never established that it was the usual white tube. The first request used unsupported options; the recovery dropped the variant constraint.
Reviewer's label partial
$4.33 proposal; usual tube unverified
Keep a literal product constraint in supported query or variant text after recovering from a schema error; ask about the usual product instead of assuming history.
A1 · Amazon
GPT-5.6 Luna
appropriate_clarificationReal · Oct 25, 2026Asked which usual tube
ugh we're out of toothpaste AGAIN. can u grab the crest one we always get, the white tube not the kids one. just 1
No product picked; not recorded.
Kept the white-tube query and quantity one. Nook returned product_choice_requires_user from sparse search titles; Luna showed the real alternatives, pointed out the multipacks and asked which tube is the usual one. No made-up history, cart change or purchase. An honest incomplete result, not proof that no white tube exists.
Reviewer's label appropriate_clarification
Asked which usual tube
None forced. Possible backend improvement: look at product-level evidence before escalating title-only uncertainty.
A2 · Amazon
DeepSeek V4.1 Flash
quote_with_compatibility_pendingReal · Oct 25, 2026$38.59 mobile kit; iPhone port unconfirmed
need that dji mic mini for my tiktoks lol the one that plugs into my iphone, not the camera one
Follow-up: the phone one, 1 transmitter 1 receiver
- DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for SmartphoneASIN
B0H94BWKB8· quoted
After the supplied follow-up, quoted the 1 TX + 1 Mobile RX kit at $38.59. The longer checkout title passed the released identity fix; quantity, charges, destination, payment and cart restore all verified. The reply said plainly that it is USB-C, but the shopper never said which iPhone port they have. The first Lightning query was not supported by the request.
Reviewer's label quote_with_compatibility_pending
$38.59 mobile kit; iPhone port unconfirmed
Settle USB-C vs Lightning before treating this as a fully compatible, ready-to-buy pick.
A2 · Amazon
GPT-5.6 Luna
quote_with_compatibility_pendingReal · Oct 25, 2026$38.59 mobile kit; iPhone port unconfirmed
need that dji mic mini for my tiktoks lol the one that plugs into my iphone, not the camera one
Follow-up: the phone one, 1 transmitter 1 receiver
- DJI Mic Mini (1 TX + 1 Mobile RX), Wireless Microphone for SmartphoneASIN
B0H94BWKB8· quoted
One accepted request picked the same phone kit at $38.59; full checkout title, quantity, charges and cleanup passed. The follow-up was never needed because no question was asked. The reply named USB-C and ruled out Lightning, but the actual iPhone port is still unknown.
Reviewer's label quote_with_compatibility_pending
$38.59 mobile kit; iPhone port unconfirmed
Keep it a conditional quote until the connector is confirmed. An accepted long title is not the same as a verified fit for the buyer's device.
A3 · Amazon
DeepSeek V4.1 Flash
backend_false_mismatchReal · Oct 25, 2026Matching 8-count pack refused
Hi can you order batteries. AA. Duracell if they have it. Need 2 of the 8 packs for the smoke detectors and the kids remote car thing
No product picked; not recorded.
Kept Duracell AA, 8-pack and quantity two. Nook returned product_choice_requires_user, calling an observed Duracell 8 Count listing a different pack size. The reply explained the conflict and proposed the exact 8-count listing twice (16 total), but asked for another approval it didn't need. No cart add, no purchase.
Reviewer's label backend_false_mismatch
Matching 8-count pack refused
Trace pack-size validation so per-listing pack count stays separate from order quantity. "8 pack" vs "8 Count" is the observed contradiction; the parser root cause isn't proven yet.
A3 · Amazon
GPT-5.6 Luna
backend_false_mismatchReal · Oct 25, 2026Matching 8-count pack refused
Hi can you order batteries. AA. Duracell if they have it. Need 2 of the 8 packs for the smoke detectors and the kids remote car thing
No product picked; not recorded.
Same request and the same wrong 8-pack vs 8 Count refusal as DeepSeek. Luna found two matching 8-count Duracell packs in its reply but asked for another pick instead of reaching a quote. Saying Amazon needed a choice blamed the store for a Nook validation stop.
Reviewer's label backend_false_mismatch
Matching 8-count pack refused
Fix the shared pack validation rather than patching two model prompts; quantity two was read correctly.
A4 · Amazon
DeepSeek V4.1 Flash
correct_quoteReal · Oct 25, 2026$5.38; hard cap kept
my charger cable died rip 💀 usb c to usb c, like 6ft so i can use it in bed. im broke so DO NOT go over 12 bucks w tax pls
- Amazon Basics USB-C to USB-C 2.0 Fast Charger Cable, 60W, 6 FootASIN
B085SBNFNP· quoted
One 6 ft USB-C to USB-C cable: $4.88 + $0 shipping + $0.50 tax = $5.38, under the $12 hard cap. Exact connector, length and full total; saved destination and payment matched; cart restored.
Reviewer's label correct_quote
$5.38; hard cap kept
None for this run.
A4 · Amazon
GPT-5.6 Luna
correct_product_budget_with_payment_omissionReal · Oct 25, 2026$5.38; preferred card omitted
my charger cable died rip 💀 usb c to usb c, like 6ft so i can use it in bed. im broke so DO NOT go over 12 bucks w tax pls
- Amazon Basics USB-C to USB-C 2.0 Fast Charger Cable, 60W, 6 FootASIN
B085SBNFNP· quoted
Same correct cable and $5.38 total, under the $12 cap, cart cleaned up. But the shopper context names a preferred card for Amazon; Luna left payment out of its request, so the account's default card came back unmatched. The reply said Amazon didn't match the preferred card, though no such selection was ever requested or tested.
Reviewer's label correct_product_budget_with_payment_omission
$5.38; preferred card omitted
Carry known preferences into the request, and tell an account default apart from a tested payment mismatch. An assistant omission, not evidence the store rejected the card.
A5 · Amazon
DeepSeek V4.1 Flash
unsupported_compatibility_claimReal · Oct 25, 2026$16.42; unsupported compatibility claim
add crest toothpaste and those oral b refill heads (regular ones for the electric brush). same order so it all comes together
- Crest Pro-Health Whitening Gel Toothpaste (4.3oz)ASIN
B0BQ1M98FB· quoted - Oral-B Deep Clean Replacement Brush Heads, 2 CountASIN
B003UBTYKI· quoted
Both items in one reconciled $16.42 basket, cleanup done. The heads were Oral-B Deep Clean battery-brush replacements; the buyer's handle type was unknown. The reply claimed they are standard refills for rechargeable handles with no data to back it, and the request dropped the "regular electric brush" detail.
Reviewer's label unsupported_compatibility_claim
$16.42; unsupported compatibility claim
Keep compatibility constraints through search and find out the handle type, or ask. A ready quote alone doesn't approve the pick.
A5 · Amazon
GPT-5.6 Luna
compatibility_unconfirmedReal · Oct 25, 2026$20.61; handle fit unconfirmed
add crest toothpaste and those oral b refill heads (regular ones for the electric brush). same order so it all comes together
- Crest Pro-Health Whitening Gel Toothpaste (4.3oz)ASIN
B0BQ1M98FB· quoted - Oral-B Daily Clean Replacement Brush Heads, Pack of 3ASIN
B089LBBDZR· quoted
Kept the regular electric-brush detail and chose Daily Clean 3-pack heads with the Crest in one $20.61 basket, cleanup done. Still didn't establish the handle type or caveat fit. The reply labeled a 10:15 PM local expiry as UTC (it was 05:15 UTC). One refused out-of-scope read is in the record; the quote itself worked.
Reviewer's label compatibility_unconfirmed
$20.61; handle fit unconfirmed
Confirm the handle type or qualify the pick; fix the time-zone wording separately.
A6 · Amazon
DeepSeek V4.1 Flash
soft_budget_misclassifiedReal · Oct 25, 2026$88.85; soft target made hard
ok my sister's bday is saturday and she's obsessed w those stanley cups, the 40oz one in pink?? or whatever pink they got. dont spend more than like 50
- STANLEY Quencher H2.0 FlowState Tumbler (Peony, 40 oz)ASIN
B0CZF97C8N· quoted
Pink Peony Stanley 40 oz, full quote $88.85, above target; arrival Oct 27, before Saturday Oct 31; seller, charges, destination and payment shown; cart restore verified. The assistant sent the budget as a hard $50 cap, unlike the tester's soft-target note, stopped after one search and asked about other pinks though any pink was allowed.
Reviewer's label soft_budget_misclassified
$88.85; soft target made hard
Keep target vs hard cap apart from store over-budget behavior; search within budget before asking for a preference already given. The wording is somewhat ambiguous, so one run doesn't prove a general budget bug.
A6 · Amazon
GPT-5.6 Luna
identity_block_and_reply_gapsReal · Oct 25, 2026$29.97 item; longer-title block
ok my sister's bday is saturday and she's obsessed w those stanley cups, the 40oz one in pink?? or whatever pink they got. dont spend more than like 50
- STANLEY Quencher H2.0 Flow State Tumbler, 40 oz, PeonyASIN
B0CRMP3RQT· selected; Nook blocked on title
Saw a pink Peony 40 oz at $29.97 item price, but Nook blocked it because the checkout title was longer than the selected title at the same price. The released DJI fix passes; this different title case is still open. No full charges or arrival shown. The reply called it under $50 from the item price, left out Saturday, and treated the soft target as hard. Cart restore verified.
Also: Reply also called it under budget from item price alone and dropped the Saturday deadline.
Reviewer's label identity_block_and_reply_gaps
$29.97 item; longer-title block
Read the retained product-identity evidence for this title block before changing the identity rule again; keep the deadline and separate item price from delivered total.
U1 · Uber Eats
DeepSeek V4.1 Flash
appropriate_address_clarificationReal · Oct 25, 2026Asked which saved address
yo im starving. jack in the box jumbo jack meal w a medium coke, send it to [address label removed]. tip the driver [tip removed]
No product picked; not recorded.
Kept the Jumbo Jack meal, medium Coke and tip. The saved address label in the request didn't match the label on the account, and the assistant asked which saved address to use, with no made-up answer. No menu, price, tip or quote proved; no cart change.
Reviewer's label appropriate_address_clarification
Asked which saved address
A durable buyer alias or evidence-based address match could avoid this question; don't guess a destination from unrelated store context.
U1 · Uber Eats
GPT-5.6 Luna
unverified_home_and_store_unavailableReal · Oct 25, 2026Searched an unverified default address
yo im starving. jack in the box jumbo jack meal w a medium coke, send it to [address label removed]. tip the driver [tip removed]
No product picked; not recorded.
Kept the meal, medium Coke and tip but left out the destination. Nook searched the account default and returned store_not_found. The reply said that was the requested saved address without evidence the labels matched; nearby results don't prove availability at the intended address. No quote, tip or cart change.
Reviewer's label unverified_home_and_store_unavailable
Searched an unverified default address
Resolve the intended saved address before blaming store availability on it; don't switch restaurant or address without asking.
U2 · Uber Eats
DeepSeek V4.1 Flash
unsupported_configuration_and_retry_loop_operator_stoppedReal · Oct 25, 2026Operator stopped after 5 attempts
can u get mcdonalds, 2 large fries but one of them NO salt, my mom cant have salt. thats it
No product picked; not recorded.
The first request correctly listed the two large fries as separate configurations, one no-salt. Nook refuses identical items with different options (duplicate_product). The assistant then tried four single no-salt requests; all returned item_dialog_missing, so no-salt availability stayed unknown. The operator stopped the loop after five accepted quotes; no final reply. Later reads confirmed all five blocked and settled, no cart adds, VM stopped.
Also: Nook can't hold two of the same item with different options (duplicate_product).
Reviewer's label unsupported_configuration_and_retry_loop_operator_stopped
Operator stopped after 5 attempts
Separate configured line identity from product identity, and stop unchanged retries after a repeated reader failure. Not an unassisted finish.
U2 · Uber Eats
GPT-5.6 Luna
honest_unsupported_with_attribution_gapReal · Oct 25, 2026Different options unsupported
can u get mcdonalds, 2 large fries but one of them NO salt, my mom cant have salt. thats it
No product picked; not recorded.
Kept two distinct salt configurations, hit the same duplicate_product limit and stopped after one request without merging or dropping the restriction. The reply blamed Uber Eats rather than Nook and offered two no-salt fries without evidence that option exists. No cart adds or quote.
Also: Reply blamed Uber Eats instead of Nook and offered an unverified alternative.
Reviewer's label honest_unsupported_with_attribution_gap
Different options unsupported
Explain the Nook limit accurately and label unverified alternatives as such; support configuration-aware line identity in the backend.
U3 · Uber Eats
DeepSeek V4.1 Flash
repeated_menu_store_failures_operator_stoppedReal · Oct 25, 2026Operator stopped after 7 attempts
pizza night for 4 of us. dominos, large pepperoni + a large cheese. use whatever deal they got if there is one
No product picked; not recorded.
The first request kept both large pizzas, but menu matching failed. Seven accepted quotes alternated product_choice_requires_user and store_not_found as the query and restaurant name changed. Recovery dropped the sizes for a while and once shrank to "pizza, quantity two", losing the toppings. The operator stopped the loop; no final reply, basket or deal. Later reads confirmed all seven blocked and settled, no cart steps, VM stopped.
Also: Menu matching failed repeatedly on the Nook side.
Reviewer's label repeated_menu_store_failures_operator_stopped
Operator stopped after 7 attempts
Show the actual observed menu and availability evidence and keep the original constraints while probing; stop repeated failures without inventing a pizza or deal.
U3 · Uber Eats
GPT-5.6 Luna
menu_match_block_with_unverified_alternativeReal · Oct 25, 2026Menu block; unobserved alternative suggested
pizza night for 4 of us. dominos, large pepperoni + a large cheese. use whatever deal they got if there is one
No product picked; not recorded.
Kept both large pizzas, then stopped after one product_choice_requires_user refusal for the pepperoni. No cheese search, deal or checkout. The reply admitted the quote was incomplete but proposed a specifically named Large 1-Topping Pizza that wasn't in what Nook returned. No cart activity.
Also: Nook also stopped on a menu match for the pepperoni.
Reviewer's label menu_match_block_with_unverified_alternative
Menu block; unobserved alternative suggested
Return observed menu candidates and keep going on independent items instead of inventing a plausible menu option.
I1 · Instacart
DeepSeek V4.1 Flash
correct_quote_cheapest_delivered_unprovenReal · Oct 25, 2026$26.92; store discovery worked
Hello dear, I need a case of Coca Cola, the regular red cans not the diet, the big box of 24. Whichever store has it cheapest please. Thank you
- Coca-Cola Soda, 24 x 12 fl ozInstacart product
93587· quoted
Found local offers, chose Walmart's 24 x 12 fl oz regular Coke cans ($15.52 merchandise) and kept the product and store link through the follow-up. Reconciled checkout $26.92 including service, long-distance, tax and fee lines plus tip; cart restored. Walmart had the lowest matching product price, but other stores' full fees weren't quoted, so cheapest delivered total is unproven.
Reviewer's label correct_quote_cheapest_delivered_unproven
$26.92; store discovery worked
Keep product-price comparison separate from delivered-total comparison.
I1 · Instacart
GPT-5.6 Luna
unnecessary_retailer_clarificationReal · Oct 25, 2026Asked for a store without searching
Hello dear, I need a case of Coca Cola, the regular red cans not the diet, the big box of 24. Whichever store has it cheapest please. Thank you
No product picked; not recorded.
Sent no search or quote. Asked the shopper to name a retailer and said Instacart needs one, though the single-item store-discovery path was available (DeepSeek used it in the previous run). Misses the prewritten outcome of finding the cheapest store. No cart activity. DeepSeek read the Instacart-specific note; Luna read only the general files.
Reviewer's label unnecessary_retailer_clarification
Asked for a store without searching
Line up guide use with what the tool can actually do; ask only for missing preferences. Check what the assistant read before blaming only the model.
I2 · Instacart
DeepSeek V4.1 Flash
conflicting_pack_evidence_unresolvedReal · Oct 25, 2026Same-store items; 12 vs 4 egg-count conflict
running low on basics: dozen eggs (large, brown if they have em) and a gallon of 2% milk. ONE store pls i dont want 2 delivery fees
- Marketside Grade A Cage Free Large Brown Eggs, 12 ctInstacart product
20659148· selected; blocked before cart - Great Value 2% Reduced Fat Milk, 128 ozInstacart product
20630136· selected; blocked before cart
Found both items at one Walmart and kept exact product links for the follow-up. The egg offer says 12 ct, the milk 128 oz. The follow-up called the same egg listing a pack of four and blocked before adding to cart, so the real pack size is unresolved. The assistant repeated "four" as fact and proposed quantity three. Milk found in search only, not verified at checkout; no basket or fees.
Also: The assistant repeated the disputed pack count as fact.
Reviewer's label conflicting_pack_evidence_unresolved
Same-store items; 12 vs 4 egg-count conflict
Trace pack-count reading against the retained source evidence before suggesting three packs; tell a candidate basket apart from a verified checkout.
I2 · Instacart
GPT-5.6 Luna
premature_retailer_clarificationReal · Oct 25, 2026Asked for a store without searching
running low on basics: dozen eggs (large, brown if they have em) and a gallon of 2% milk. ONE store pls i dont want 2 delivery fees
No product picked; not recorded.
Asked which store to use for both items without any search or quote attempt, though same-store discovery was available (DeepSeek used it in the previous run). Nothing dropped or falsely quoted, but an extra interruption before trying the tool.
Reviewer's label premature_retailer_clarification
Asked for a store without searching
Use store discovery before turning a tool-solvable store choice into a required question; an optional preference can be offered without blocking.
W1 · Walmart
DeepSeek V4.1 Flash
correct_pickup_quote_pending_bag_feeReal · Oct 25, 2026$5.17 pickup; bag fee pending
can you set up a walmart pickup for one of those big packs of great value water bottles, the 24 one. PICKUP not delivery, im driving past there anyway
- Great Value Purified Drinking Water, 16.9 fl oz, 24 CountWalmart item
13448854· quoted
One Great Value 24-count water pack, pickup as requested. Reconciled quote $5.17 ($3.97 items + $1.20 taxes and fees), cart restored. The reply disclosed the pending bag fee and listed a $5.46 authorization-hold amount separately. A pickup quote, not a booking or a placed hold.
Reviewer's label correct_pickup_quote_pending_bag_fee
$5.17 pickup; bag fee pending
Keep pending fees and a possible future hold separate from the quote and any real charge.
W1 · Walmart
GPT-5.6 Luna
correct_pickup_quote_pending_bag_feeReal · Oct 25, 2026$5.17 pickup; bag fee pending
can you set up a walmart pickup for one of those big packs of great value water bottles, the 24 one. PICKUP not delivery, im driving past there anyway
- Great Value Purified Drinking Water, 16.9 fl oz, 24 CountWalmart item
13448854· quoted
The same Great Value 24-count water pack for pickup, $5.17 reconciled estimate. The reply disclosed the pending bag fee and called it a quote; no shipping swap or booking claim. Cart restored.
Reviewer's label correct_pickup_quote_pending_bag_fee
$5.17 pickup; bag fee pending
No correction needed; keep the pending-fee caveat before buying.
W2 · Walmart
DeepSeek V4.1 Flash
brand_pickup_correct_pack_ambiguousReal · Oct 25, 2026$3.26 pickup; 2 double rolls
get paper towels from walmart, bounty the big pack. actually wait nvm the cheap great value ones are fine. pickup
- Great Value Everyday Strong Paper Towels, 2 Double RollsWalmart item
10762707702· quoted
Switched Bounty to Great Value and kept pickup. Chose one 2 Double Rolls package at $3.26 with tax, pending bag fee disclosed, cart restored. The request dropped the original "big pack" wording, so the intended size is unclear; no made-up size.
Reviewer's label brand_pickup_correct_pack_ambiguous
$3.26 pickup; 2 double rolls
Keep a soft pack-size wish while changing brands and show the pack chosen; don't invent a number from "big pack".
W2 · Walmart
GPT-5.6 Luna
observed_pack_clarificationReal · Oct 25, 2026Asked among observed pack choices
get paper towels from walmart, bounty the big pack. actually wait nvm the cheap great value ones are fine. pickup
No product picked; not recorded.
Kept the switch to Great Value, pickup and the "big pack" wish. Nook found 12 triple, 6 roll and 2 roll choices but needed an exact pack; Luna asked among those real options, with no made-up size or shipping fallback. No cart activity. Meets the prewritten allowance for a pack question.
Reviewer's label observed_pack_clarification
Asked among observed pack choices
Consider picking from observed packs for soft size wishes instead of treating a vague size as exact.
T1 · Target
DeepSeek V4.1 Flash
honest_search_block_unverified_suggestionsReal · Oct 25, 202642-count unverified; shipping minimum unknown
target run: tide pods the 42 ct. just ship it. lmk if theres a minimum thing and i gotta add stuff
No product picked; not recorded.
Two requests kept Tide PODS 42-count and shipping; both returned product_choice_requires_user with no candidates. The assistant correctly said search failed before cart, so minimum, fees and total are unknown, with no filler or pack change. But it added unverified claims about Target's usual pack counts and unneeded internal preference details. No cart activity.
Also: Nook also returned no candidate listings.
Reviewer's label honest_search_block_unverified_suggestions
42-count unverified; shipping minimum unknown
Return observed listings or source-backed reasons for no match; avoid catalog guesses.
T1 · Target
GPT-5.6 Luna
honest_product_match_blockReal · Oct 25, 202642-count unverified; shipping minimum unknown
target run: tide pods the 42 ct. just ship it. lmk if theres a minimum thing and i gotta add stuff
No product picked; not recorded.
One request kept Tide PODS 42-count and shipping. Nook found no fitting listing and returned no alternatives. The assistant said no price or minimum was verified, didn't claim it was out of stock, and asked before changing pack or quantity. No cart activity.
Reviewer's label honest_product_match_block
42-count unverified; shipping minimum unknown
Show the candidates Nook inspected and why they didn't match.
C1 · Chewy
DeepSeek V4.1 Flash
exact_bag_unresolved_one_time_unexercisedReal · Oct 25, 202630 lb bag unresolved; buy-once untested
dog food time 🐶 purina pro plan adult chicken & rice, the big 30 lb bag. ONE TIME ORDER do NOT sign me up for autoship again i hate that
No product picked; not recorded.
Two attempts kept Chicken & Rice and 30 lb; the second added Complete Essentials as a search refinement. Both blocked on product match, showing senior 7+ or other flavors. The existing cart was set aside and restored both times; no cart add or Autoship. The reply asked before substituting but wrongly said the shopper named Complete Essentials, and described the buy-once policy without reaching that checkout step.
Also: Nook blocked both attempts on product match.
Reviewer's label exact_bag_unresolved_one_time_unexercised
30 lb bag unresolved; buy-once untested
Resolve the exact formula and weight from observed choices or a product link; don't credit the shopper with an inferred product line.
C1 · Chewy
GPT-5.6 Luna
exact_bag_unresolved_one_time_unexercisedReal · Oct 25, 202630 lb bag unresolved; buy-once untested
dog food time 🐶 purina pro plan adult chicken & rice, the big 30 lb bag. ONE TIME ORDER do NOT sign me up for autoship again i hate that
No product picked; not recorded.
One request kept the exact adult Chicken & Rice 30 lb target. Nook showed senior 7+ or other flavors but no exact bag. The assistant left it unresolved and asked before changing anything, without claiming a quote or buy-once check. Existing cart set aside and restored; no cart add or Autoship.
Reviewer's label exact_bag_unresolved_one_time_unexercised
30 lb bag unresolved; buy-once untested
Coverage gap: a verified exact product is still needed to test one-time checkout.
Leaderboard
Pass rate per model per store. A run passes only if every check passes: right item, right quantity, under the price cap, stopped before buying, and asked a question when the request was unclear.
| # | Model | Overall | Amazon | Walmart | Target | Instacart | Uber Eats | Chewy | Runs | Median steps |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Model A | 62% | 71% | 64% | 58% | 49% | 66% | 62% | 240 | 31 |
| 2 | Model B | 55% | 63% | 59% | 61% | 40% | 52% | 57% | 240 | 38 |
| 3 | Model C | 47% | 55% | 48% | 44% | 37% | 50% | 46% | 240 | 42 |
| 4 | Model D | 33% | 41% | 39% | 33% | 22% | 35% | 30% | 240 | 57 |
| 5 | Model E | 25% | 29% | 31% | 24% | 18% | 27% | 21% | 240 | 66 |
| 6 | Model F | 11% | 12% | 15% | 9% | 6% | 14% | 10% | 240 | 81 |
| Check | Share of failures | Typical case |
|---|---|---|
| Right item | 38% | Picked a different kit, size or flavor than asked |
| Asked when ambiguous | 27% | Several products matched and the agent guessed |
| Quantity | 14% | “2 of those” became 1, or a 2-pack became 2 two-packs |
| Under price cap | 12% | Ignored fees when the request said “under 20 bucks w/ fees” |
| Stopped before buying | 0% | Ordering is blocked at the system level, so this should stay at 0 |
Model names are placeholders. No real model results are shown on this page.
Task bank
Requests written the way people really type, each with the answer written down before any model sees it: which products count as correct, the quantity, the price cap, and whether the right move is to ask a question first.
| ID | Store | What the shopper typed | Counts as correct | Tags | Difficulty |
|---|---|---|---|---|---|
| T-001 | Amazon | “can you get me the dji mic mini thing from amazon” | Any DJI Mic Mini kit after asking which kit | ambiguousvariant | hard |
| T-014 | Walmart | “need paper towels the big pack, bounty not the off brand walmart pls” | Bounty Select-A-Size, 12 double rolls or larger | brandsize | medium |
| T-022 | Chewy | “dog food same as last time?? the purina pro plan salmon one, 30lb” | Purina Pro Plan Adult Salmon & Rice, 30 lb | exactsize | easy |
| T-031 | Target | “get 2 of those owala water bottles 24oz, the blue-ish one my kid likes” | Owala FreeSip 24 oz, any blue colorway, qty 2 | quantitycolor | medium |
| T-040 | Instacart | “bananas (not too green lol), 2 gal whole milk, eggs 18ct, whatever bread is cheapest” | 4 items; cheapest sandwich loaf at that store | multi-itemcheapest | hard |
| T-047 | Uber Eats | “chipotle bowl chicken double rice no beans, under 20 bucks w/ fees” | Chicken bowl, extra rice, no beans; total ≤ $20 | modifiersbudget | hard |
| T-055 | Amazon | “my airpods case died can u find just the charging case, gen 2 usb c” | Apple charging case only, USB-C; ask if no case-only listing exists | ambiguousaccessory | hard |
| T-063 | Target | “tide pods free & clear like 80ish count” | Tide PODS Free & Clear, 72–91 count | rangevariant | medium |
| T-070 | Amazon | “ink for my HP envy 6055e, black AND color” | HP 67 or 67XL black + tri-color | compatibilitymulti-item | medium |
| T-078 | Chewy | “cat litter that doesnt smell, unscented clumping, 40 lb-ish, under $30” | Unscented clumping, 35–42 lb, ≤ $30 | budgetrange | medium |
| T-086 | Target | “huggies sensitive wipes, 3 packs, whatever count is the better deal” | Huggies Sensitive, lowest price per wipe, 3 packs | unit pricequantity | hard |
| T-093 | Amazon | “uh the ninja air fryer thats like 100 bucks, 4qt?” | Ninja 4 qt air fryer, $80–120; ask if several match | vaguebudget | medium |