What does "statistically significant" mean for an SEO change, and when does a lift count?

A statistically significant SEO lift is a gain that would be rare if the change did nothing, measured against pages you left alone. The usual cutoff is p below 0.05. A lift counts when the windows and comparison pages were fixed first, the interval's low end still matters, and the page was not picked for declining.

• Researchers fix the significance level before collecting data, typically at 5% or much lower (Wikipedia, Statistical significance, read 30 October 2026).
• Statistical significance measures neither the size of an effect nor the importance of a result (American Statistical Association, ASA statement on p-values, read 30 October 2026).
• Run alone, Penlid's estimator called an unchanged page significant 3.26% of the time on average across 675 simulated scenarios (Penlid, methodology, read 30 October 2026).
• Pages picked because they were declining, then left unchanged, came back "improved" 5.50% to 9.20% of the time in Penlid's simulations (Penlid, methodology, read 30 October 2026).
• Google says it can be hard to tell whether a page change caused a gain, because other events also contribute (Google, Search Console Help, read 30 October 2026).

What does "statistically significant" mean for an SEO change?

A change is statistically significant when a gap at least this large would be very infrequent if the null hypothesis, here that the change did nothing, were true. For a page, the question Penlid's methodology asks is did this page do better than the pages we left alone would have predicted?; our plain-words reading of the do-nothing case is that the answer is no.

The p-value turns that question into a number. The American Statistical Association defines it, informally, as the chance, assuming a specified statistical model holds, of seeing a summary of the data at least as extreme as the one observed.

The cutoff is set before you look. Penlid's SEO significance calculator tests at 0.05, so it calls a change improved or declined only when p falls below that line. It reads the 95% interval and the p-value from one distribution, so an interval that excludes zero and p below 0.05 are the same statement; the guide to confidence intervals in SEO reporting covers reading one.

Does a significant result mean the change worked?

No: a significant result says only that a gap this large would be rare if the change did nothing. The American Statistical Association is direct about what that leaves out: P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. So p = 0.01 is not a 99% chance that the edit worked.

Nor does it measure size: A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. With enough clicks, a tiny lift can earn a small p-value.

And it does not establish cause. Google says It can be difficult to determine if a specific change you made to a page directly caused an improvement in Search performance, and Penlid calls its own output timing evidence, not proof.

How often does a page that did nothing look significant?

In Penlid's simulations, the estimator on its own called an unchanged page significant 3.26% of the time on average, where a 95% test allows 5%. These are results on simulated sites where the true answer was known, from the 675 scenarios in the methodology; they are not field results, and any single figure can be a point or so off. Significant here counts both improved and declined.

The worst scenario, 5.85%, runs above the nominal 5%. The last two rows are the trap: pages chosen for a refresh because their last 28 days sat at least one standard deviation below their own average, then changed in no way at all. A page that has just had a bad four weeks tends to have a better next four whatever you do to it, and no comparison group can subtract that. The pre-trend warning appeared on only 10.9% to 22.7% of those pages, so it is a flag and not a safeguard.

Many tests add false wins. If 20 changes did nothing and each is tested at 5%, we expect 20 × 0.05 = 1.00 false win, and the chance of at least one is 1 − 0.95^20 = 64.2% (our arithmetic, assuming independent tests; see false wins in SEO testing).

When does a lift count?

A lift counts when it passes five checks.

1. Fixed windows. Penlid compares the 28 days before a change with the 28 days after, skips the day of the change and uses only windows that have ended. Peeking and stopping on a good number breaks the p-value: a paper on A/B tests calls the usual p-values and intervals unreliable when sample sizes are chosen by continuously monitoring a test.
2. Comparison pages chosen first. Use three to eight pages you did not change, from the same kind of query and not tiny, and write them down before you read their numbers; the guide to testing without page groups covers the choice.
3. Enough clicks. With 10 or more clicks in the 28 days before the change you get a size and an interval, with 1 to 9 a direction only, and with none nothing to measure; see how much traffic a page needs.
4. A passed cutoff and a worthwhile low end. The p-value is below 0.05, the same as an interval that excludes zero. Then read the interval's low end as the cautious case and ask whether even that is worth having (our advice, not a rule of the estimator).
5. Not picked for declining. If you did pick on decline, an "improved" may include a recovery that was coming anyway. Choosing pages without looking at their trend (every page in a set, or alternate pages) reduces it; see regression to the mean.

Timing matters too. Penlid reads a change 38 days after it goes live: the 28-day after window, 7 days it allows for the newest days to settle, and about 3 days of reporting lag, as the SEO test calculator sets out. The 7 days is Penlid's own allowance, not a Google figure.