Twenty page changes that did nothing should produce about one false verdict, win or loss, at the 5% level, and at least one 64% of the time if the tests are independent. Penlid's simulations put false wins alone at 3.10% at most (about 0.6 in 20), unless the pages were picked because they were declining: then 5.50% to 9.20%.
- For 100 independent tests at the 5% level where every null hypothesis is true, 5 false positives are expected and at least one is about 99.4% likely (Wikipedia, "Multiple comparisons problem", read 30 October 2026).
- Across 675 simulated scenarios with nothing done to the page, Penlid's estimator called a change significant 3.26% of the time on average and "improved" falsely at worst 3.10% (Penlid, methodology, read 30 October 2026).
- Pages picked because they were declining and then left unchanged came back "improved" 5.50% to 9.20% of the time in Penlid's simulations (Penlid, methodology, read 30 October 2026).
How many false wins should you expect from 20 changes that did nothing?
About one false verdict, and fewer false wins: if each of 20 changes did nothing and each is tested at the 5% level, you expect 20 × 0.05 = 1.00 false verdicts. A false verdict is an "improved" or "declined" result for a change that did nothing, and a false win is the "improved" kind. Penlid's SEO test calculator counts false alarms the same way and says they should be at or under about 5% at the 95% level, so that allowance is shared between wins and losses.
The chance of at least one false verdict is 1 − 0.95^20 = 1 − 0.358 = 64.2%, if the tests are independent. Most batches have none (35.8%) or one (37.7%), and two or more turn up 26.4% of the time (our arithmetic). Wikipedia's article on the multiple comparisons problem shows the same pattern at 100 tests.
Only changes that did nothing can produce false wins, which is our reasoning from the definition: if 4 of your 20 truly worked, count the other 16, and 16 × 0.05 = 0.80 false verdicts. For significance in general, see the guide to statistical significance for an SEO change.
What false-win rate did Penlid measure when nothing was done to the page?
Penlid's simulations put false wins at 3.10% at most, and false verdicts of either kind at 3.26% on average, across 675 scenarios in which nothing was done to the page (methodology). The worst single scenario was 5.85% either way. For "improved" alone the methodology gives only worst cases, 3.10% and 3.05%, so we treat them as ceilings, not forecasts.
| Rate per test | What it counts | Expected in 20 | Chance of at least one |
|---|---|---|---|
| 5.00% | Either direction; what a 95% test allows | 1.00 | 64.2% |
| 3.26% | Either direction; average of 675 scenarios | 0.65 | 48.5% |
| 5.85% | Either direction; worst scenario | 1.17 | 70.0% |
| 3.10% | "Improved" only; worst case | 0.62 | 46.7% |
| 5.50% to 5.75% | "Improved" only; declining pages, no drift | 1.10 to 1.15 | 67.7% to 69.4% |
| 9.15% to 9.20% | "Improved" only; declining pages, 15% drift | 1.83 to 1.84 | 85.3% to 85.5% |
At the 3.10% ceiling the count is 20 × 0.031 = 0.62 false wins, below the textbook 1.00, but that holds only for pages not picked because they were declining.
Why do pages picked because they were declining give more false wins?
Because a page that has just had a bad stretch is likely to recover on its own, and untouched pages cannot subtract a recovery they were not picked for. The methodology says A page that has just had a bad four weeks is likely to have a better next four whatever you do to it
, and calls this the largest remaining source of false "improved" results.
We measured it. Pages whose last 28 days sat at least one standard deviation below their own average, then left completely alone, came back "improved" 5.50% of the time at 4 clicks a day and 5.75% at 40 clicks a day with no drift between pages, and 9.15% and 9.20% with 15% drift. All four are false wins alone and exceed the 5% a 95% test allows for wins and losses together. The pre-trend warning appeared on only 10.9% to 22.7% of these pages, so it is a flag, not a safeguard.
Pick pages to change without looking at their trend, such as every page in a set or alternate pages. If you did pick on decline, read "improved" as at least as good as the pages left alone, and possibly some recovery that was coming anyway
. See regression to the mean and choosing pages to change without bias.
Do choices made after you look add false wins?
Yes: in our reasoning the 5% describes one test run exactly as planned, so each choice made after seeing results adds chances for a win that the 5% does not cover. The American Statistical Association says that running several tests and reporting only those with certain p-values, typically those passing a significance threshold, renders the reported p-values essentially uninterpretable
. In an SEO batch that means reporting only the winners of 20. By our reasoning it also means checking clicks, impressions and position and keeping whichever passes, since each metric, window or page group is another test.
Comparison pages are another place. The SEO significance calculator asks you to write down which pages you will compare against before you look at their clicks after the change, because if you pick pages that make the answer look the way you hoped, the interval and p-value are for a question nobody asked
. Penlid's methodology reports that qualifying comparison pages on the "before" days instead of a separate earlier stretch added 1.08 percentage points of false "improved" results in the worst simulated case, against −0.08 for the separate stretch. The guides on control pages and measuring without page groups cover the choice.
How do you keep false wins down?
Decide the tests before you see results, count them, report all of them, and hold a large batch to a stricter bar.
- Count and report every test. Write down how many changes, metrics and windows you will measure, and show that count beside each win. The SEO estimator tests each change separately at 0.05, and our pages describe no adjustment across changes.
- Choose before you look. Fix the pages to change, the comparison pages and the two windows first. The SEO significance calculator prints the lift, a 95% interval and a p-value for one change and stores nothing, so the record of all 20 runs is yours to keep.
- Raise the bar for a large batch. Wikipedia describes making tests stricter to counter multiple testing:
The best known such adjustment is the Bonferroni correction
. It divides 0.05 by the number of tests, here 0.05 ÷ 20 = 0.0025, which by our arithmetic cuts the chance of any false verdict across 20 independent do-nothing changes from 64.2% to 1 − 0.9975^20 = 4.9%. Wikipedia says the classical Bonferroni method should not be used because it is overly conservative, and points to the more modern Holm-Bonferroni method and Benjamini-Hochberg procedure, so treat 0.0025 as a conservative bar. Applying any such bar by hand to the printed p-value is our suggestion, not a Penlid setting. The cost, in our reasoning, is that real changes need stronger evidence too, so more come back "cannot tell"; see inconclusive SEO tests and the minimum detectable effect.
Example (illustrative numbers)
These numbers are invented to show the arithmetic, except two rates borrowed from Penlid's simulations. A team changes 20 pages, measures each against its own untouched pages, and sees three "improved". Wikipedia loosely defines the false discovery rate, the share of wins that are false, as the expected proportion of false positives among all significant tests
; it depends on how many changes really worked. Unless a row says otherwise, a real change is found 80% of the time and a change that did nothing returns a win 5% of the time.
| Assumption | True wins | False wins | All wins | Share that is false |
|---|---|---|---|---|
| 4 of the 20 really worked (base case) | 4 × 0.80 = 3.20 | 16 × 0.05 = 0.80 | 4.00 | 0.80 ÷ 4.00 = 20.0% |
| Only 2 of the 20 really worked | 2 × 0.80 = 1.60 | 18 × 0.05 = 0.90 | 2.50 | 0.90 ÷ 2.50 = 36.0% |
| 4 worked, but only half of real changes are found | 4 × 0.50 = 2.00 | 16 × 0.05 = 0.80 | 2.80 | 0.80 ÷ 2.80 = 28.6% |
| Base case, false wins at Penlid's 3.10% ceiling | 4 × 0.80 = 3.20 | 16 × 0.031 = 0.496 | 3.696 | 0.496 ÷ 3.696 = 13.4% |
| Base case, all 20 pages picked for decline (9.20%) | 4 × 0.80 = 3.20 | 16 × 0.092 = 1.472 | 4.672 | 1.472 ÷ 4.672 = 31.5% |
In the base case a fifth of all wins are false, so of the three "improved" results about 3 × 0.20 = 0.6 are expected to be false; with only 2 real changes it is 3 × 0.36 = 1.1, and with pages picked for decline 3 × 0.315 = 0.9. Expected shares describe a batch, and nothing in the table says which wins are the false ones.
What this cannot tell you
- The textbook figures assume independence. They are for 20 independent tests at exactly 5%. Tests that share comparison pages or dates are not independent; in our reasoning the average then stays at 1.00 but the chance of at least one moves away from 64.2%.
- Penlid's rates come from simulated sites. Each scenario ran 2,000 times, so any single figure can be a point or so off. Real daily counts are somewhat more variable than the simulated ones; in the acceptance grid's extra-variance check the false alarms were at most 6.05%, inside that check's 8% limit. The grid stops at 400 clicks a day, 35% drift between pages and eight comparison pages.
- A count cannot say which wins are false. A win is timing evidence, not proof: the untouched pages remove what hit every page, not what hit only yours. Penlid's estimator also reads clicks only, so it cannot show what happened to rankings.
- The share of false wins depends on numbers you do not have. How many changes really worked is unknown; in the example that one number moves the share from 20.0% to 36.0%. A tool that tests only for improvement puts the whole 5% on wins, while Penlid's splits it between wins and losses (our reasoning).
Questions people also ask
What is a false win in SEO testing?
A false win is a result that reports a gain after a change that did nothing, where chance alone produced the gap between the page and the pages left alone. Every test run at a 5% level allows some, and the rate climbs when pages are picked after a decline, when comparison pages are chosen after their results are visible, and when many tests are run and only the successful ones are reported.
Why does SEO not produce instant results?
Google says some changes can take effect in a few days while others could take several months, suggests waiting a few weeks before reviewing Search Console again, and gives no guarantee of a noticeable effect. Reading a result too early mixes real movement with noise, which in our reasoning is one way false wins arise. A fixed pair of 28-day windows, as the SEO significance calculator uses, builds the wait into the method.
What are the common mistakes that create false wins?
Four choices inflate them: changing pages because their traffic fell, choosing comparison pages after seeing how they did, checking several metrics or date ranges and keeping the best, and reporting only the winners of a batch. Each adds chances for a win that the stated 5% never counted. Penlid's simulations single out the first as the largest remaining source of false "improved" results.
How many SEO tests does it take to get a false win?
At 5% per test, with every change doing nothing and the tests independent, 14 tests give even odds of at least one false verdict: 1 − 0.95^14 = 51.2%, against 48.7% for 13 (our arithmetic). That counts wins and losses together. For wins alone at Penlid's worst-case rate of 3.10%, it takes 23 tests to pass even odds: 1 − 0.969^23 = 51.5%.
Sources
- Multiple comparisons problem, Wikipedia, read 30 October 2026.
The best known such adjustment is the Bonferroni correction, but other methods have been developed.
The FDR, loosely defined as the expected proportion of false positives among all significant tests
- The ASA Statement on p-Values: Context, Process, and Purpose, American Statistical Association, read 30 October 2026.
Conducting multiple analyses of the data and reporting only those with certain p-values (typically those passing a significance threshold) renders the reported p-values essentially uninterpretable.
- Debugging drops in Google Search traffic, Google, Search Central, read 30 October 2026.
- Methodology: how we measure, and what we can't, Penlid, read 30 October 2026.
A page that has just had a bad four weeks is likely to have a better next four whatever you do to it
at least as good as the pages left alone, and possibly some recovery that was coming anyway
- SEO significance calculator, Penlid, read 30 October 2026.
If you pick the pages that make the answer look the way you hoped, the interval and p-value are for a question nobody asked.
- SEO test calculator, Penlid, read 30 October 2026.