Incrementality Test Plan Template

Most teams that run A/B tests on their site think they are measuring incrementality. They are not. A conversion rate test tells you which landing page wins among visitors who were already coming. An incrementality test tells you whether running the campaign caused more purchases than would have happened without it. Those are different questions, and confusing them is expensive.

Why an incrementality test is not a CRO test

A/B tests on your site split traffic between two on-page experiences. The test population is visitors who showed up regardless of your ads. Incrementality testing - whether through geo holdout, ghost ads, or on-off scheduling - splits exposure to the campaign itself. The control group is people who would have converted anyway. The question is not which ad performs better. It is whether the ad causes any purchase at all beyond organic baseline.

Paid social platforms are especially prone to claiming credit for organic demand. If someone was going to buy regardless of your retargeting ad, the platform still counts that conversion. An incrementality test is the only practical way to see how much of that reported attribution is real. Marketing incrementality testing covers the strategic case in full and explains how different businesses approach holdout sizing. This template is the execution layer: a worksheet you complete before you launch the test, not a framework you piece together after the results come in.

The three methods

Pick one method before filling in anything else. Geo holdout: ads run in test regions and are suppressed in matched control regions. Ghost ads: the control group sees the same budget served to non-converting placements, holding impression volume constant. On-off: the campaign alternates live and paused in matched time windows, and sales lift is compared across periods. Geo holdout is the most rigorous. Ghost ads require explicit platform support. On-off is the easiest to run but is sensitive to seasonality. Choose based on what your ad platform and sales volume actually support, not what sounds cleanest in a slide.

The worksheet

How to read the result

Calculate iROAS: incremental revenue in the test group minus the control group, divided by the spend that drove the test group. If iROAS is materially lower than your platform-reported ROAS, your attribution is overstating real contribution. How large that gap needs to be before you act on it depends on your margin and spend level. Run the numbers through the A/B test significance calculator to confirm the result clears your minimum threshold before changing anything. If it does not clear the threshold, resist the urge to declare a trend.

A non-significant result is not proof the channel is working. It is an underpowered test. Go back to the duration and MDE section of the worksheet, extend the window, or narrow the scope of the hypothesis. A test that cannot detect a meaningful lift tells you nothing useful about incrementality.

Frequently asked questions

How is this different from a standard A/B test?

A standard A/B test on your site splits visitors between two page variations. An incrementality test splits exposure to the campaign itself. You are not measuring which version converts better - you are measuring whether running the campaign causes any lift at all compared to no campaign. The control group never sees the ads.

Which method should I start with?

Geo holdout is the most reliable if you have enough regional sales volume to detect a signal. If volume is too thin to split geographically, an on-off time-based test is the easiest to run. Avoid ghost ads unless your platform explicitly supports the feature - synthetic control placements are easy to configure incorrectly. The best A/B testing tools list includes platforms with built-in holdout functionality worth checking before you build a manual setup.

How long does an incrementality test need to run?

Long enough to hit your minimum detectable effect at your chosen confidence level. For most paid social campaigns that means at least two to four weeks. Shorter windows almost always underpower the test. If your sales volume is low, extend the duration before you reduce the MDE - a test sized to detect a trivially small lift is not answering the question you care about.

The worksheet is not a formality. The forcing function is filling in every section before the test launches, especially the decision rule. A result means nothing if the team did not agree in advance on what they would do with it.