Visa’s 2026 shopping test found that AI agents can still buy before they fully understand what a customer needs. The result points to gaps in learning customer requirements and making sound decisions, not just finding products. Those gaps matter when an AI assistant can act on someone’s behalf.
For consumers, a mistaken choice can affect more than a product match: it can lead to a payment the shopper did not intend. Trust in AI shopping depends on an agent knowing when information is incomplete and when a no-purchase decision is the better outcome. This report explains what the test shows and where its limits lie.
Key Takeaways
- Reliable shopping requires understanding needs, not just searching products.
- An agent may buy even when it has not met the test’s standard.
- Product selection and completed payment are different steps.
- Human review remains useful before an AI-led purchase.
- The test offers a way to see AI shopping attempts firsthand.
The key question is whether an agent can reliably learn when to buy and when to stop.
How the Visa AI Shopping Benchmark 2026 Works
Visa’s benchmark tested whether an AI could ask the right questions before choosing a product or declining to buy. Pass rate means the share of test runs that meet the benchmark’s requirements for discovering customer needs and making the correct purchase or no-purchase decision. The first round tested 22 models from 12 labs across 15 shopping challenges.
Launched on November 5, 2026, the Visa AI shopping benchmark 2026 gave each model three attempts at every challenge. The visa agentic commerce test used a simulated customer whose private requirements emerged through conversation, when the AI asked relevant questions or took the right action. The controlled market used the same real product listings, prices, seller policies, and reviews for every model.
A simulated shopper reveals requirements through conversation
The 15 shopping scenarios covered televisions, laptops, and personal care appliances. Ten episodes involved televisions; one episode was public, while 14 remained private. In a television challenge, the AI needed to learn the customer’s budget, format compatibility, HDR10+ support for family videos, and gaming needs.
A plausible product match is not enough when a task includes a hidden need.
The Visa Research benchmark paper describes how agents research offers and compare seller terms in a controlled market. Each episode ended when the simulated customer selected an offer or declined to buy. Checkout, payment, and order placement were outside the test.
A pass requires the right decision, not a lucky guess
To pass, a model had to uncover every customer requirement defined by the task and then make the correct buy-or-no-buy decision. A lucky guess did not count. The test assessed whether the agent had enough information to support its choice, rather than whether its choice happened to match the right product.
A model had to uncover every requirement assigned to the task before its decision could count as a pass.
What the 42.8% Failure Rate Reveals
The models passed 57.2% of runs overall, leaving a 42.8% failure rate. The 990 scored runs included four distinct outcomes, from meeting the standard to ending without a choice.
The five models with the highest pass rates reached 82.2% of attempts, while the lowest-performing models passed 28.9%. The table shows what happened across all scored runs.
| Run outcome | Runs | What happened |
|---|---|---|
| Passed | 566 | Met the requirement-discovery and decision standard |
| Purchase failed standard | 229 | Bought without meeting the pass standard |
| No purchase despite suitable offer | 114 | Declined when a suitable offer existed |
| No decision | 81 | Ended without a choice |
Pass rates varied, and repeat performance was weak
Only 34.2% of scenarios passed all three attempts. Repeating a successful scenario did not reliably reproduce success. More models passed when they uncovered more hidden buying factors: the five models that uncovered the most averaged a 70.2% pass rate, compared with 44.4% for the five that uncovered the fewest.
Run cost did not guarantee a better result. The five models with the highest average cost per run passed 67.1% of runs, compared with 50.2% for the five least expensive. Asking about customer requirements had a clearer relationship with success than spending more on a run.
Many failed runs still ended in a purchase
54% of failed runs still ended in a purchase, so a buying decision did not necessarily show that the agent had a sound basis. Some models passed 82.2% of attempts, but a purchase could still fail the benchmark’s standard.
In the two episodes where no product met every requirement, 46 of 132 runs still ended with a purchase decision.
What the Results Mean for Shoppers and AI Commerce
AI assistant use is far more common than trust in letting AI handle a payment. Visa’s September 2026 Trust Index found that 72% of U.S. consumers had used an AI assistant, while 23% trusted AI with payment transactions. Product selection is not the same as completed payment, and this benchmark stopped before checkout.
Visa plans to add shopping scenarios and product categories, then extend testing to checkout, payment, and order placement. The public benchmark includes an interactive experience and replays of AI attempts. This research complements Visa Intelligent Commerce; it is not directly part of that program.
A benchmark result speaks to the task tested, not every step that may follow.
Before delegating a purchase, shoppers can use three practical checks:
- Confirm the agent has captured the shopper’s requirements.
- Review the exact item the agent proposes.
- Require human approval before payment.
Visa Agentic Commerce research gives consumers a way to inspect how models handle shopping decisions. Before delegating a purchase, keep a human approval step between the agent’s recommendation and payment.
Conclusion
Visa’s results show that asking, deciding, and stopping reliably are all part of competent AI shopping. The benchmark measures a meaningful early step in agentic commerce: whether a model can learn a customer’s requirements and make a supported choice, including choosing not to buy.
Its current scope does not establish that an agent can safely complete a payment. A successful product choice is only one part of a transaction, and shoppers still have reason to keep control of spending. To see how different attempts play out, readers can try the interactive benchmark and watch the attempt replays on Visa Payments Frontier.