Guide

We asked ChatGPT, Claude and Grok to fix three Shopify problems

We gave ChatGPT, Claude and Grok the same three Shopify problems and checked every step against Shopify's help pages. What they got right, what they got wrong, and what all three missed.

Published 8 min read
We asked 3 AIs to fix Shopify problems. Results: low-stock email, all 3 work; card-testing attack, mostly right; match a payout, all 3 work; the 28 October Shopify Protect change, all 3 missed.

Plenty of store owners now ask an AI assistant before they open the Shopify Help Center. So we gave ChatGPT, Claude and Grok the same three everyday Shopify problems and checked every step they gave against Shopify's own help pages. All three would have got a store owner most of the way there. None of them was perfect, the mistakes were small but real, and all three missed a Shopify change from the week before.

We build automations with Claude, and Claude helped us draft this article. That is a reason to read the Claude results with care, so we scored every answer against Shopify's pages, not against each other, and we list each AI's mistakes in full below.

Scorecard for three Shopify questions asked to ChatGPT, Claude and Grok on 2 November 2026. Low-stock email: Grok works but alerts at 5, not below 5; ChatGPT works with no errors found; Claude works but has the trigger's old and new names reversed. Card-testing attack: Grok good with two claims not in Shopify's pages; ChatGPT most complete with one path we could not verify; Claude good with the best cancel detail but no Blocked view. Matching payouts: Grok good but no Pending check; ChatGPT good with no errors found; Claude most precise with no errors found. All three missed the 28 October Shopify Protect change.
One run each, 2 November 2026. Every step checked against Shopify's help pages.

How did we test them?

We wrote three questions the way a store owner would, with the details that matter. These are the exact prompts:

  1. Low-stock email. "I run a Shopify store on the Grow plan. I want to get an email when any product variant drops below 5 units in stock, without paying for an app. Give me exact step-by-step instructions in the Shopify admin, using the real menu and template names."
  2. A card-testing attack. "Last night my Shopify store got about 40 small orders in an hour, most with failed card payments and a few that went through. I use Shopify Payments. What exactly should I do in the Shopify admin today? Give step-by-step instructions with the real menu paths."
  3. A short payout. "My Shopify Payments payout this week was less than my sales. How do I match each Shopify payout to the deposit in my bank account, step by step, in the Shopify admin?"

Each question went into a fresh chat with no history or memory, on 2 November 2026, using each product's default settings: Grok's default Fast model in a private chat, ChatGPT's default model in a temporary chat with medium thinking effort, and Claude's default Opus 5.5 model at medium effort in an incognito chat. All three cited web sources, and Grok and Claude showed the searches they ran. Claude asked permission once to read a Shopify help page, and we allowed it. We ran each question once, so treat this as a snapshot, not a ranking. Ask again tomorrow and the wording will change.

To score, we opened the Shopify help pages behind each answer and checked every menu path, setting name and claim. We sorted problems into three kinds: wrong (Shopify's page says otherwise), not in Shopify's pages (we could not confirm it), and missing (a step that matters was left out). Our own guides for the same three problems are linked at the end of each section.

Question 1: can they set up a low-stock email?

Yes, all three. Each one sent us to Shopify Flow, which is free, and to the right template by its exact name, Get notified by email when product variant inventory is low (Shopify Help Center, Product variant inventory quantity changed). All three also added the step most quick guides skip: a second condition on the previous stock level, so you get one email when stock crosses the line instead of one after every sale. Shopify warns that if you check only the current amount, your reorder email is sent each time the product is ordered until you restock (Shopify Help Center, examples of workflows).

  • ChatGPT: no errors found. It set the condition to less than 5 with a previous quantity of 5 or more, named the trigger correctly as Product variant inventory quantity changed, previously Inventory quantity changed, and added a table of which stock changes would send an email.
  • Claude: one error. The setup was right, but it said Shopify's docs call the trigger Inventory quantity changed and older guides use the longer name. It's the other way round: Shopify's page names the trigger Product variant inventory quantity changed and says it was previously called Inventory quantity changed. Harmless if you're following along in Flow, confusing if you're searching for it.
  • Grok: one slip. It set the condition to 5 or fewer, so the email arrives at 5 units, not below 5 as we asked. It did mention "less than" as an option. It was also the only one to point out that the trigger runs even when inventory tracking is off, which Shopify confirms.

Our step-by-step version: get a low-stock email on Shopify without an app.

Question 2: do they know what to do after a card-testing attack?

Mostly. All three recognised card testing straight away and gave the same core plan: check Shopify's fraud analysis on each order that went through, don't ship them, cancel and refund, turn on Decline charges that fail CVV verification under Settings, Payments, Shopify Payments, Manage, then switch to manual payment capture for a while. We checked those paths and setting names against Shopify's pages and they are right (Shopify Help Center, configuring Shopify Payments and payment authorization and capture).

  • ChatGPT: the most complete answer, in eight steps. It named the Order risk evaluation section correctly, included the optional Send a warning 1 day before an authorization expires setting, the Blocked view in abandoned checkouts, Fraud Control rules and the chargeback filter on the Orders page. One step we could not confirm: the exact route it gave for reaching a human in Shopify's Help Center.
  • Claude: the best detail on cancelling. It warned that cancelling in bulk can refund orders that were already refunded, which Shopify's cancelling page states, and that card fees aren't returned on refunds. It also gave the 7-day authorization window for Shopify Payments, which is right. It left out the Blocked view in abandoned checkouts.
  • Grok: a good plan with the Blocked view included, but two claims we could not find in Shopify's pages: that reporting the attack helps your account standing, and its advice to unpublish or reprice the cheap product being targeted. The second may be sensible, it just isn't Shopify's guidance.

What all three missed: on 28 October 2026, five days before our test, Shopify said Shopify Protect is on for every business on Shopify Payments, declines payments it is near-certain are fraud before they reach the bank, and gives each order a plain-language explanation of its risk (Shopify blog, Introducing Shopify Protect). None of the three named Shopify Protect or its new risk explanations, though each said in general terms that Shopify already blocks some attempts. Their advice still works. It just doesn't tell you what changed that week. None of them named the free Flow template Capture payment if order is not high fraud risk either, which pairs with manual capture so you only check the risky orders by hand.

Our step-by-step version: how to catch card-testing orders on Shopify.

Question 3: can they match a payout to a bank deposit?

Yes, and this was the strongest round. All three gave the right route, Finance, then View payouts in the Payouts section, said to open each payout and use its breakdown of charges, refunds, fees, adjustments and reserves, and said to export the payout transactions as a CSV. All three said deposits typically take 1 to 3 business days to arrive, which matches Shopify's page (Shopify Help Center, payout details).

  • Claude: the most precise, because it read Shopify's payout page before answering. It covered the older Money page for admins that still have it, the bank reference number to give your bank, retried payouts, and the Pending section for sales not yet paid out. No errors found.
  • ChatGPT: no errors found. It filtered by payout date, checked for Deposited status and Pending funds, and pointed to the Shopify Payments activity report if weekly totals still differ.
  • Grok: correct, and the only one to mention the Bank Reference column in payout exports, which Shopify's changelog shows was added in January 2026 (Shopify changelog). It called the bank reference a "Trace ID", which isn't Shopify's term, guessed that a bank might take a small fee, and didn't mention Pending, which is often why a week's payouts look short.

Our step-by-step version: match Shopify payouts to bank deposits.

So which one should a store owner use?

On three questions and one run each, we would not crown a winner. ChatGPT had the fewest errors, Claude was the most exact when it read the source page, and Grok was quick and mostly right with a couple of loose claims. Their mistakes were small. What they all missed was a Shopify change from the week before.

So use the steps they give you, then check them before you change anything in your store.

Four checks before you follow an AI's Shopify steps. One, open the source it cites and find the step on the Shopify page. Two, match the exact names of menus, templates and settings. Three, test on one product or one order before changing everything. Four, check Shopify's changelog for anything from the last few weeks.
Four checks that would have caught most problems in this test.
  1. Open the source. All three cited Shopify pages. Click one and find the step there. That alone catches a reversed name or an off-by-one condition.
  2. Match the exact names. Menus, templates and settings in your admin should match the answer word for word. If they don't, search the Help Center for the name you can see.
  3. Test small first. Lower one product's stock, or cancel one order, before you change a setting for the whole store.
  4. Check what changed recently. Shopify's changelog and its announcements cover the last few weeks, which is exactly where AI answers are weakest.

What are the limits of this test?

It was small on purpose: three questions, three assistants, one run each, on one day. Answers change between runs and between model versions, and paid tiers or other settings could do better or worse. We only scored claims we could check against Shopify's own pages. Anything we could not confirm is marked as such rather than called wrong.

Axioprax builds this for Shopify stores

  • Fraud Hold$349Available

    Hold risky orders for a person to check before they ship, and keep a dispute evidence pack ready.

Frequently asked questions

Can ChatGPT, Claude or Grok give accurate Shopify instructions?

Mostly, in our test. On three everyday Shopify problems all three gave steps that would work, with small errors such as a reversed trigger name, an alert that fired at 5 instead of below 5, and a few claims we could not find in Shopify's pages.

Which AI was best for Shopify questions?

On three questions and one run each, we would not name a winner. ChatGPT had the fewest errors, Claude was the most exact when it read Shopify's page first, and Grok was quick and mostly right with a couple of loose claims.

Do AI assistants know about recent Shopify changes?

Not reliably. On 2 November 2026 none of the three mentioned Shopify's 28 October Shopify Protect change when asked about a card-testing attack, even though each one cited web sources.

How do I check an AI's Shopify instructions?

Open the Shopify page it cites and find the step there, match menu, template and setting names word for word, test on one product or order first, and check Shopify's changelog for recent changes.

Your next step

Is this happening in your business?

Tell us what it looks like on your side, in a sentence or two. A person reads it with this guide in mind and replies with questions or next steps.

Nothing is sent until you review it on the next page.