Visa Benchmark: AI Shopping Agents Fail 42.8% of Test Runs

On November 5, 2026, Visa released the Visa Agentic Commerce Benchmark, an open test of how well AI shopping agents buy on a person’s behalf. Across 22 AI models and 15 shopping scenarios, 42.8% of runs failed. More than half of those failed runs still ended in a purchase.

The findings matter to anyone who sells online. AI shopping agents are assistants that search, compare and then buy products for a user, and they are starting to appear in chat apps, browsers and online stores. Visa’s test shows where these agents still fall short, and those weak spots are closely tied to how stores present their product data.

How Visa tested AI shopping agents

According to Visa’s announcement, issued from San Francisco, the first round covered 22 AI models from 12 labs. Each model took on 15 shopping challenges, such as buying a TV, a laptop or a personal care appliance. Every model attempted each scenario three times.

The challenges were built to resemble real customers. The shopper in each scenario does not state every requirement at the start, so the agent has to draw them out through conversation. The agent must also gather enough information to justify each choice, which rules out lucky guesses. When no product fits, the correct outcome is for the agent to decline to buy.

A run passes only if the agent uncovers every requirement defined in the scenario and makes the right call on whether to buy. Visa described a TV scenario as an example. The shopper mentions a brand a relative recommended and a membership with a third-party retailer, and nothing else. The agent has to find out the budget and the technical needs on its own, including format compatibility, HDR10+ support for watching family videos and gaming features.

The benchmark, replays of the agents’ attempts and a whitepaper are published on Visa’s Payments Frontier site. Consumers can take the same test and compare their choices with the AI’s.

Results and the top-scoring models

The overall pass rate was 57.2%. The best model passed 82.2% of its attempts, while the weakest passed only 28.9%. A gap that wide shows that the choice of model has a large effect on the quality of a shopping agent.

Visa did not name the models in its statement. Per-model rankings were reported by Fortune and cited by The Next Web: agents from Google and Meta shared the top score of 82%, a Claude agent placed third at 76% and an OpenAI agent placed sixth at 64%. Because these figures do not appear in Visa’s official release, they should be treated as media reporting.

The most useful findings concern how the agents behaved. The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate. The five that uncovered the fewest passed 44.4% of the time. Asking questions before choosing a product had a large effect on the outcome.

Cost played a role too, though a smaller one. The five models with the highest average cost per run passed 67.1% of runs, compared with 50.2% for the five cheapest. That 16.9-point gap is smaller than the 25.8-point gap between models that probe for needs and models that do not.

Consistency is still low

Consistency is another problem. Across all models and scenarios, all three attempts passed in only 34.2% of cases. Models that passed a scenario once often failed when they repeated it. For a store or a bank considering an AI agent, a single successful trial is therefore weak evidence.

Purchases that happen when the agent fails

About 54% of failed runs still ended with the AI making a purchase. In those cases the agent bought a product that did not meet the shopper’s needs, or bought before it had learned all of them. For the shopper, the result is the wrong item. For the store, it means returns, complaints and the cost of shipping a replacement.

Rubail Birwadker, global head of growth at Visa, said in the release, “AI agents have the potential to power the next era of commerce, but trust is key to making it work.”

Visa’s consumer data points the same way. Its Trust Index survey from September 2026 found that 72% of U.S. consumers had used an AI assistant, but only 23% trusted AI to make payments on their behalf. Visa also said the results do not mean AI agents are unready for commerce. In its view, they show that consistency and reliability remain open challenges.

What Visa plans next

Visa plans to add more scenarios and product categories and to extend testing to checkout, payment and order placement. It has also shared the benchmark with model providers for feedback. According to The Next Web, Visa had already brought its services into ChatGPT and joined Mastercard and Ant International on identity checks for AI agents, known as “know your agent”.

What this means for online stores

In Indonesia, shopping through AI agents is not yet common, but the direction is becoming visible. Large retailers abroad have started letting agents in, as with Walmart’s Agent Gateway, which caps what agents can spend. Online stores, electronics distributors and brand owners can use Visa’s findings as a preparation checklist.

1. Complete your product data. An agent can only match needs against the information it can find. Key specifications such as screen size, HDR10+ support, ports, power draw or warranty should sit in structured fields on product pages and in catalog feeds. Details that appear only in images or free-form descriptions are harder for an agent to read. Clean data also helps search engines and price comparison sites.
2. Teach your shopping assistant to ask. If your store builds its own assistant, design it to ask about budget, intended use and constraints before it recommends anything. It also needs permission to say that nothing in stock fits. Teams working on AI solution development usually encode rules like these in system instructions and test scenarios.
3. Test repeatedly. Run the same scenario several times and record the results. Visa’s 34.2% figure shows that one passing run does not guarantee the next.
4. Require approval before payment. As long as agents can still buy the wrong item, a confirmation step for the shopper and a clear returns policy protect both sides.

For banks and payment providers, the findings add weight to transaction limits and agent identity checks. For stores, product data and the ordering flow are part of a retail management system worth reviewing now, before more orders start arriving from agents.

Frequently asked questions

What is the Visa Agentic Commerce Benchmark?

It is an open test created by Visa to measure how well AI models shop on behalf of users. Models are scored on whether they uncover a shopper’s needs through conversation and make the right decision to buy or not to buy.

Which AI models scored highest?

Visa did not name the models. According to Fortune, as cited by The Next Web, agents from Google and Meta shared the top score of 82%, followed by a Claude agent at 76%. In Visa’s official data, the best model passed 82.2% of its attempts.

Is it safe to let an AI agent pay for purchases?

The results show that agents can still buy items that do not match what the shopper wanted. Until reliability improves, users should turn on approval before payment or a spending limit, and review the order summary before confirming.

Conclusion

Visa’s benchmark gives a clear picture of where AI shopping agents stand today. They often pick correctly, but their results are inconsistent and some still buy without asking. For online stores, the best preparation is complete product data, an assistant that asks before it acts and a clear approval step. If your store is designing a shopping assistant or tidying its catalog for AI agents, the Merakit team can help you plan the next steps.

Sources

• Visa: Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail (November 5, 2026)
• The Next Web: Visa tested 22 AI models as shoppers, and they failed 42.8% of the time (November 5, 2026)
• PYMNTS: Visa Finds AI Models Fail 43% of Shopping Test Runs (November 5, 2026)