Shopify A/B Testing: How It Works in 2026 (Native + PageFly)

For years, running an A/B test on Shopify meant the same routine: install a third-party app, inject a script into your theme, split your traffic, and wait weeks for enough visitors to call a winner. Stores without much traffic often couldn't test at all.

In 2026 that changed. Shopify shipped native A/B testing in its Winter '26 Edition — plus a way to test a change against AI shoppers before a single real customer sees it. A/B testing (also called split testing) is the practice of showing two versions of a page or theme to different visitors at the same time, then keeping the version that converts better. What's new is where that testing now lives, and how early you can start.

This guide covers the three layers of Shopify A/B testing in 2026 — native Rollouts, SimGym's AI simulation, and page-level testing with PageFly — and, more importantly, which one to reach for depending on what you're changing. It's written for ecommerce managers, CRO specialists, and founder-operators who'd rather ship a tested change than guess.

Test variations of any page — layouts, visuals, or CTAs — directly in the editor.

What Shopify A/B testing is — and what changed in 2026

The mechanics haven't changed. You still split visitors into a control group (version A, your current page) and a variant group (version B, your change), run both at once, and measure a metric like add-to-cart rate or conversion rate to decide which wins.

What changed is the tooling. Until this year, testing was something you bolted on. Now it's built into Shopify itself, and a new AI tool lets you pressure-test an idea before you spend real traffic on it. The practical effect. Testing is no longer reserved for high-traffic Plus stores with a CRO budget — smaller merchants can validate a change in minutes instead of waiting weeks for significance.

That matters because untested changes are expensive in both directions. Ship a worse layout and you quietly lose sales until someone notices. Sit on a better one and you leave conversions on the table the whole time it's in review.

Shopify's native A/B testing: Rollouts

Rollouts is Shopify's native A/B testing and staged-deployment tool, introduced in the Winter '26 Edition. There's no third-party app to install, no external script injected into your storefront, and no extra monthly fee — it runs inside the Shopify admin.

Its core idea is the staged rollout. Instead of flipping a theme change live for everyone, you release it to a slice of your traffic first — typically 10%, then 25%, then 50%, then 100% — while you watch how each version performs. If the new version underperforms, you roll it back before it reaches most of your customers.

Rollouts is built for theme- and template-level changes: a new theme, a redesigned homepage template, a different product-page layout. That's its strength and its limit. It compares whole versions of a template, so it's the right tool for big, structural swaps — and a blunt instrument for testing one headline or one button. As of launch it's rolling out to Shopify Plus and Advanced plans in early access, so availability depends on your plan.

Test before you launch: SimGym's AI shoppers

The bigger shift in 2026 is SimGym — a free Shopify app, built with NVIDIA, that runs a simulated A/B test using AI shoppers instead of real visitors. Shopify's engineering team describes it as sending hundreds of "robots," each in its own cloud browser with a persona, a budget, and a shopping intent, to browse your store and decide what to do — click, scroll, add to cart, or leave.

A twin set of shoppers does the same on the alternate version. The result is a simulated A/B test in minutes instead of weeks, even for stores with zero real traffic. Shopify says the system now runs hundreds of thousands of these shopping sessions a day.

Why it's a big deal. SimGym solves the oldest problem in conversion testing: not having enough traffic to reach significance. It can't replace a real test against real buyers — shopper simulations are a prediction, not a guarantee — but it lets you kill a weak idea early and only spend real traffic on changes that already look promising. Together, SimGym and Rollouts form a two-stage pipeline: simulate the change with AI first, then confirm it with real traffic.

Where native testing stops: page, section, and element tests

Native Rollouts is excellent for theme-wide decisions. But most conversion wins don't come from swapping an entire theme — they come from the small, high-intent details on a single page: the order of sections on a landing page, the hero image on a product page, the wording and color of one call-to-action.

Those are page- and element-level tests, and they're a different job. This is where a page builder with built-in testing fits in. PageFly A/B Experiments is a built-in feature of the PageFly page builder that compares a Control and a Variant of the same page and uses Bayesian statistics — the same approach behind tools like VWO and Optimizely — to call a winner. Because it's tied to the editor, you test the exact design change you just made, down to a single section or button, rather than a whole template. It's included in PageFly's paid plans and works even with limited traffic, so it isn't gated behind a Plus subscription.

It also answers the question most merchants get stuck on — what should I even test? PageFly's AI Page Checkup scans a page across SEO, copy, layout, and conversion signals, scores its health, and prescribes prioritized fixes. That turns a blank-page "I don't know what to change" into a ranked list of hypotheses you can run as A/B tests.

How to run an A/B test in PageFly (step by step)

Here's the full workflow, matching PageFly's A/B testing documentation:

Start a test from the CRO Center — or from any page in the PageFly editor.

1. Create the test. Open your page in the PageFly editor and click A/B testing, or go to CRO Center > Manage A/B testing > Create A/B test. PageFly duplicates your current page into a Variant.
2. Name it and set it Active. An Active test publishes the Control and the Variant together when you hit publish; you can also schedule it for a launch or sale date.
3. Set traffic allocation. Choose how much of your page's visitors join the test, then the Control/Variant split. A 50/50 split collects a verdict fastest.
4. Pick a goal metric and confidence. Measure add-to-cart rate, product view rate, or a PageFly conversion event, and set a desired win probability — 95% is the industry standard.
5. Design the Variant. Switch to version B and make your change — a new hero, reordered sections, different CTA copy.
6. Publish and start. PageFly shows "Gathering data" until it has enough visitors and conversions to judge.
7. Publish the winner. When a version wins at your confidence level, publish it to 100% of traffic — then start your next test.

The A/B testing panel: set the test Active, split traffic, schedule it, and let Bayesian testing call the winner.

Track each test's status and elapsed time in Analytics, then publish the winner.

For a sense of how long to run it: PageFly recommends at least 7 days with a minimum of 500 visitors and 10 conversions per version, and 2–4 weeks for more reliable reads. The more traffic the test gets, the faster the result becomes trustworthy.

Native vs PageFly vs dedicated apps: which should you use?

These tools overlap less than they look — each is built for a different unit of change. The honest answer is that many serious stores use more than one.

A simple way to choose: simulate every meaningful change with SimGym first. Reach for Rollouts when you're swapping a whole theme or template. Reach for PageFly when the change lives on one page — a landing page, a product page, a single section or CTA. Reach for a dedicated app when you specifically need price or shipping testing, which the page- and theme-level tools don't cover. PageFly also integrates with Shoplift if you want theme-section testing alongside your page builds.

A/B testing mistakes that quietly waste traffic

Testing too many things at once. If version B changes the headline, the image, and the button, a win tells you nothing about why it won. Change one variable per test so the result is actionable.

Calling it early. A test that "looks like it's winning" after two days usually isn't. Wait for your confidence threshold (95%) and a real sample — for most stores that's at least a week, often two.

Testing with too little traffic and no plan. If a page gets a handful of visitors a week, a live test may never reach significance. That's exactly the gap SimGym fills — simulate first — and a reason to test on your highest-traffic pages where a win compounds.

Not shipping the winner. A test only pays off when you publish the winning version and move on to the next hypothesis. Treat A/B testing as a loop: test, learn, ship, repeat. Pair it with broader conversion rate optimization habits and apply what you learn to your next Shopify landing page.

Shopify made A/B testing native in 2026, and AI made it possible without a traffic firehose. The merchants who win won't be the ones with the most tools — they'll be the ones who actually test the changes that decide whether a visitor adds to cart.