Solution Finder
What is your customer trying to achieve? Pick the problem closest to what they actually said — this translates it into the right methodology, data requirements, and Lab.
You said: “Understand whether advertising caused incremental conversions”
Incrementality Testing
MeasurementWhy this fits
Incrementality Testing uses a holdout group to separate conversions caused by advertising from conversions that would have happened anyway.
What it answers
Of the conversions a platform attributes to advertising, how many actually happened because of the advertising, versus how many would have happened regardless?
How it works
A treatment group is exposed to advertising and a control group is deliberately withheld from it (a holdout); comparing conversion rates between the two groups reveals the incremental effect the platform-attributed number does not.
Data required
- A genuine, sufficiently sized holdout/control group
- Randomised or matched assignment to treatment and control
- Conversion tracking consistent across both groups
- Platform-attributed conversion figures for comparison
Discovery questions
Can a true holdout group be created, or only a geographic/time-based proxy?
A true randomised holdout gives the cleanest causal estimate; proxy designs (geo-based, time-based) introduce more assumptions and potential bias.
How large does the holdout need to be to detect a meaningful effect?
Underpowered tests can fail to detect real incrementality, leading to false conclusions that a channel doesn't work.
How will stakeholders react to incremental ROAS being lower than platform-reported ROAS?
This is usually the most contentious moment in an incrementality engagement, and needs to be set up as an expectation early.
Implementation approaches
Risks & limitations
Risks
- A poorly designed or contaminated holdout produces a misleading incrementality estimate
- Organisational resistance when incremental results contradict platform-reported performance
Limitations
- Incrementality tests are usually channel-specific and don't automatically generalise across all campaigns
- Requires enough conversion volume to reach statistical significance within a reasonable test window