Statistical Significance for Facebook Ad Creative Tests: Advanced Facebook Ad Creative Testing

Quick answer

For Facebook creative tests, do not use frequentist p-values. Use a Bayesian probability-to-be-best (PTBB) calculation. Declare a winner only when one variant has 95% PTBB on CPA AND has accumulated at least 50 conversions or 50,000 impressions, whichever comes first. Anything weaker is theatre.

Why p-values mislead on Meta

Frequentist tests assume fixed sample sizes and binary outcomes. Meta delivery is non-stationary (CPMs drift hourly), variants get unequal exposure (CBO reallocates), and you peek at the data daily. All three break frequentist assumptions. Bayesian PTBB handles peeking, unequal samples, and continuous metrics naturally.

The framework, step by step

1. For each variant, log impressions, link clicks, and conversions daily.
2. Compute the posterior distribution of conversion rate using a Beta(1,1) prior. Most ad-test calculators do this for you.
3. Run 10,000 Monte Carlo draws across all variants. The PTBB for each variant is the share of draws where it has the best CPA.
4. Declare a winner when one variant has PTBB >= 95% AND has accumulated >= 50 conversions OR >= 50,000 impressions.
5. If no variant hits the threshold by day 7, declare "no significant winner" and ship the lowest-CPA candidate as a default — but flag the campaign as inconclusive.
6. Re-run the failed test with bigger creative swings. Iso-variants of an exhausted concept will never produce significance.

Example test matrix

Variant	Conversions	Spend	CPA	PTBB
A	28	£820	£29.3	12%
B	41	£810	£19.8	78%
C	33	£830	£25.2	9%
D	22	£790	£35.9	1%

In this case, B is the lowest-CPA candidate but has not yet hit 95% PTBB. Run 2 more days, then re-evaluate.

Pitfalls to avoid

• Declaring winners on 5 conversions. Below 30, the posterior is wide enough to drive a bus through.
• Using CTR instead of CPA for significance. CTR is a leading indicator, not the goal.
• Mixing tests across CBO and ABO ad sets. The delivery distortions break the calculation.
• Stopping at "first to 95%" without the volume floor. Small samples spike PTBB on noise.
• Comparing to a moving baseline. Lock the test cohort and only compare within it.

5 FAQs

Can I just use Meta's significance flag? No. Meta's "winner" label uses thresholds that are too loose for £1k+ accounts.

What if I have only one conversion event per ad? Use add-to-cart or initiate checkout as a proxy and validate against purchases weekly.

Is 95% the right threshold? Yes for purchases. For top-of-funnel events you can drop to 90%.

Should I use Bayesian for CTR-only tests? Yes. Beta-Binomial is exactly the right model for CTR.

Where can I run the calculation? Any Bayesian A/B calculator (e.g. Evan Miller's, Convertize, custom Python with PyMC). Set prior to Beta(1,1).

Test fewer, learn more

Pix-Vu gives you the iso-variant volume needed to reach significance — five strong executions per concept, no filler. https://pix-vu.com.