What is the minimum detectable effect in SEO testing, and how do you size a test?

A minimum detectable effect (MDE) is the smallest real change that a test would catch with a stated probability, conventionally 80%. For an SEO test, the MDE is a percentage lift versus what pages you left alone predict. You size a test by checking that its MDE is no larger than the smallest lift that would change your decision.

• Power is the probability that a test finds an effect that exists, and 80% is a common convention for adequate power, not a formal standard (Wikipedia, article on statistical power, read 30 October 2026).
• In Penlid's simulations with six comparison pages and no drift between pages, a +50% lift on a page earning 4 clicks a day was found 61.23% of the time and a +80% lift 92.01% of the time (Penlid, methodology, read 30 October 2026).
• In the same simulations with 15% drift between pages, a +80% lift was found 36.95%, 59.65%, 79.55% and 82.20% of the time at 1.4, 4, 40 and 400 clicks a day (Penlid, methodology, read 30 October 2026).

What does minimum detectable effect mean?

The minimum detectable effect is the smallest true change a test would catch often enough to rely on. Wikipedia's article on statistical power states the idea: If resources and thus sample sizes are fixed, power analyses can also be used to calculate the minimum effect size that is likely to be detected. Power is the probability of finding an effect that is really there, so every MDE carries a power level. The same article adds that there are no formal standards for power; 80% is only a convention.

Our reading: a test has one MDE for each pair of significance level and power. This guide uses the 5% level Penlid applies (a p-value below 0.05, covered in what significant means for an SEO change) and 80% power.

Why do generic A/B calculators not fit an SEO test?

Generic A/B calculators do not fit an SEO test because, as Penlid's calculator page puts it, they assume visitors are split at random between two versions, which a page edit is not. An SEO test changes one page and compares it with pages you leave alone, which we call comparison pages.

Counting noise shrinks as clicks grow: if clicks are Poisson counts, as Penlid's simulations assume, the variance equals the mean (Wikipedia, article on the Poisson distribution), so the relative noise of n clicks is about 1 ÷ √n, which is 9.4% at 112 clicks and 0.94% at 11,200 (our arithmetic). Drift is pages moving apart on their own, and the calculator page says its size, tau, is the noise a real change has to stand out from, and it does not shrink as a page gets bigger.

In the methodology's run, 15% drift cut the chance of finding a +80% lift on a 4-clicks-a-day page from 92.01% to 59.65%, while ten times the traffic, from 40 to 400 clicks a day, added only 2.65 points (our subtraction). Our reading: past a few dozen clicks a day, a bigger page buys little. For the traffic half of sizing, see how much traffic a page needs.

What are the two thresholds in Penlid's SEO test calculator?

Penlid's SEO test calculator reports a significance threshold and, separately, the change that gets an 80% chance of a verdict. It simulates Penlid's estimator thousands of times on pages shaped like yours, so both numbers come from simulation, not from a formula.

• The significance threshold is the median, over simulated tests where nothing changed, of the smallest estimated lift that would have been called significant. A real lift of that size is caught only about half the time in the tests that have enough comparison pages.
• The change that gets 80% is the real change the estimator finds in 80% of simulated tests. It is searched for by bisection in 7 steps, and the printed figure is where a curve fitted to the simulated points crosses 80%.

Our reading: the second is the minimum detectable effect in the power sense, and the first is only where an estimate starts to count. The methodology also says a "we cannot tell" result states the smallest lift the measurement could have detected, without saying whether that is the first figure or the second.

How do you size an SEO test, step by step?

Size an SEO test by running the plan through the calculator before you touch the page, then comparing its change for an 80% chance with the smallest lift that would change your decision.

1. Write down the smallest lift worth acting on before you see the calculator's answer.
2. Take the page's clicks for its last 28 days from Search Console. The calculator treats that count as the page's normal level; with no clicks the page is new and cannot be measured, and with 1 to 9 it earns a direction and no size.
3. Choose the comparison pages before the change. Penlid needs at least 3 pages you will not touch for the 28 days after the change, and uses at most 8.
4. Pick tau: 0.05, 0.15 (Penlid's suggestion if you are unsure) or 0.35, or estimate it from a page-by-day file of at least 84 days. Reading tau too low flatters every chance.
5. Compare. Set the chance of a verdict at each size of change, and the change that gets 80%, against your first step. The answer arrives 38 days after the change goes live (28 + 7 + 3, Penlid's own constants), and the SEO significance calculator then reads it.

What can you do when the minimum detectable effect is too big?

When the minimum detectable effect is too big, change the design of the test, because Penlid's tools fix both windows at 28 days. Pick comparison pages that move with the page, since drift costs a lot. Add comparison pages up to 8: with no drift, a +50% lift on a 4-clicks-a-day page was found 61.23% of the time with six pages and 70.02% with eight. Make the change bolder, because Wikipedia says effects that are large relative to the variability of the data tend to provide more power. If no lever closes the gap, "we cannot tell" is still an answer; see inconclusive SEO tests.