A confidence interval on an SEO result is a range around the measured lift whose width shows how noisy the comparison is. Get one by comparing the changed page with untouched pages over the same dates, then report the lift, its 95% interval and the p-value. If the range includes zero, the result is too close to call.
- A 95% confidence level reflects the long-run performance of the method that produced the interval, not a 95% chance for the one interval you hold (Wikipedia, "Confidence interval", read 30 October 2026).
- Penlid reads the 95% interval and the p-value from one distribution, so an interval that excludes zero and a p-value below 0.05 are the same statement (Penlid, methodology, read 30 October 2026).
- With three to eight untouched pages the amount of drift between pages is itself uncertain, so few comparison pages means a wide interval (Penlid, methodology, read 30 October 2026).
- On simulated sites where the true lift was known, Penlid's 95% interval contained it at least 94.70% of the time for pages with 1.4 clicks a day or more (Penlid, methodology, read 30 October 2026).
What does a 95% confidence interval on an SEO result mean?
A 95% confidence interval on an SEO result describes how well the method performs over many repeats, not the odds that your one range is right. Wikipedia says A 95% confidence level does not imply a 95% probability that the true parameter lies within a particular CI calculated from a sample
.
In practice (our reading), about 95 in 100 intervals built this way should contain the true lift. The Penlid methodology tests that on simulated sites: its 95% interval contained the true lift at least 94.70% of the time for pages with 1.4 clicks a day or more. Each scenario ran 2,000 times, so a single figure can be a point or so off, and these are simulation results, not field results.
Wikipedia notes that an interval can be expressed in terms of statistical significance; the guide to what statistically significant means for an SEO change covers the test side.
How do you put a confidence interval on an SEO result?
You put a confidence interval on an SEO result by comparing the changed page with pages you left alone and taking the interval of that comparison. The free SEO significance calculator does this from typed clicks: the changed page's before and after, and the same two figures for three to eight untouched pages. It returns the lift with its 95% interval and p-value, or a plain statement that the numbers cannot tell. Its interval covers every level of between-page drift the untouched pages allow.
- Choose the untouched pages first. Write them down before you read their after-change clicks; if you pick pages that make the answer look the way you hoped,
the interval and p-value are for a question nobody asked
. See testing without page groups. - Fix the windows. Use the 28 days before the change and the 28 days from the day after it, over the same calendar days for every page, and only windows that have fully ended, because Google says the newest data can be preliminary.
- Report the working with the lift. Give the p-value, the interval and the number of untouched pages beside a percentage versus what the untouched pages predict, not percentage points.
The changed page needs 10 or more clicks in the 28 days before the change to get an interval; with 1 to 9 it gets a direction only, and with none there is nothing to measure.
How do you read a confidence interval in a report?
Read a confidence interval in a report by checking, in this order, whether it includes zero, how wide it is, and whether its low end would still matter to you. That order is our advice.
| The interval | How to read it |
|---|---|
| Entirely above zero | Improved: more clicks than the untouched pages predicted |
| Includes zero | Too close to call: chance could explain the gap, which is not evidence that nothing happened |
| Entirely below zero | Declined: fewer clicks than the untouched pages predicted |
| Very wide | A noisy comparison that cannot pin down a size |
Width is the second check. The methodology says Few comparison pages means a wide interval.
An interval also widens when the untouched pages disagree with each other. More traffic helps only with counting noise: the SEO test calculator says the drift between untouched pages does not shrink as a page gets bigger
(see how much traffic a page needs).
The low end is the third check. An interval above zero says the lift is distinguishable from zero, not that it is large: the American Statistical Association says A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
Report both ends, not just the midpoint.
Do overlapping confidence intervals mean there is no difference?
No: overlapping confidence intervals can still hide a real difference, because the overlap check is conservative. A paper in The American Statistician states that failure to reject by the method of examining overlap does not imply failure to reject by the standard method
.
Penlid applies the overlap rule when its dashboard compares AI engines; the SEO significance calculator instead reads one interval, for the lift, against zero.
Many intervals bring another risk. In Wikipedia's example of 100 intervals with 95% coverage each, the expected number of non-covering intervals is 5
, so by the same arithmetic (ours) a report with 20 intervals should expect about one to miss. The guide to false wins in SEO testing covers 20 page changes.
How do you put a confidence interval on a rate, such as a citation rate?
For a rate counted as k successes out of n tries, such as the share of AI answers that cite you, put a Wilson score interval at 95% on it. Penlid uses it rather than the textbook normal approximation, which behaves badly at small n and near 0% or 100%; Wikipedia says Wilson can be safely employed with small samples and skewed observations
.
| Observed | Samples | 95% interval |
|---|---|---|
| 3 of 10 | 10 | 10.8% to 60.3% |
| 30 of 100 | 100 | 21.9% to 39.6% |
| 300 of 1,000 | 1,000 | 27.2% to 32.9% |
All three read 30%, yet they are not the same measurement: each tenfold increase in samples narrows the interval about threefold (widths of 49.5, 17.7 and 5.7 points, our arithmetic from the table).
Example (illustrative numbers)
These counts are invented to show the arithmetic; they are not data from any site. A rewritten page is compared with two sets of five untouched pages that have the same total clicks.
| Page | Before | After, set A | After, set B |
|---|---|---|---|
| Changed page | 1,184 | 1,468 | 1,468 |
| Untouched 1 | 1,040 | 1,126 | 1,290 |
| Untouched 2 | 1,085 | 1,224 | 1,034 |
| Untouched 3 | 1,012 | 1,100 | 1,235 |
| Untouched 4 | 1,060 | 1,175 | 1,006 |
| Untouched 5 | 1,103 | 1,208 | 1,268 |
| Untouched total | 5,300 | 5,833 | 5,833 |
Step 1, the lift. The changed page went from 1,184 to 1,468 clicks: 1,468 ÷ 1,184 = 1.2399, or +24.0%. Both sets total 5,300 before and 5,833 after, so 5,833 ÷ 5,300 = 1.1006, and they predict +10.1%. The page did 1.2399 ÷ 1.1006 = 1.1266 times as well, which is +12.7% versus what the untouched pages predict.
Step 2, a floor for the interval (our own arithmetic, not Penlid's estimator). Treat clicks as Poisson counts, as Penlid's simulations do; Wikipedia gives the variance of a Poisson count as equal to its mean. The log of a ratio of counts a and b then has a standard error of about √(1/a + 1/b), and for the lift all four totals count: √(1/1,184 + 1/1,468 + 1/5,300 + 1/5,833) = √0.001886 = 0.0434. The log of 1.1266 is 0.1192, and 0.1192 ± 1.96 × 0.0434 gives 0.0341 to 0.2043, which is +3.5% to +22.7% after exp(x) − 1. That counts only the noise in the clicks.
Step 3, the calculator. The same counts typed into the SEO significance calculator give this table.
| Comparison set | Lift | 95% interval | p | Reading |
|---|---|---|---|---|
| A: five pages that move together | +12.7% | +3.3% to +22.8% | 0.008 | Improved |
| B: same totals, pages disagree | +13.2% | −24.0% to +68.7% | 0.44 | Too close to call |
| C: first three pages of A | +12.7% | +0.1% to +27.0% | below 0.05 | Improved |
In set A the pages agree, so the interval lands close to the floor and stays above zero. Set B has the same totals, but its pages range from −5.1% to +24.0%, so the estimator allows for a lot of drift; the interval spans zero, and the calculator adds that the smallest lift it could have detected is about +49.0%. Its lift for B moves a little because it compares log ratios page by page, not pooled totals. Set C drops two pages, so the interval widens until its low end is +0.1%, barely above zero. Only the untouched pages differ between A and B.
What this cannot tell you
- An interval measures chance, not bias. A link, a ranking move, a news event or an unrecorded edit that hit only the changed page sits outside the range (our reasoning). The calculator calls its answer timing evidence, not proof.
- It depends on the pages you compare. A different set of untouched pages can give a different answer, as sets A and B show.
- The 95% was measured on simulated sites. The SEO test calculator says real daily counts are somewhat more variable than the Poisson counts it simulates.
- Pages picked because they were declining read "improved" too often. In Penlid's simulations such pages, left unchanged, came back "improved" 5.50% to 9.20% of the time, and no comparison group can subtract that.
- Peeking spoils an interval. An arXiv paper on A/B testing warns that intervals and p-values become wholly unreliable when sample sizes are chosen by continuously monitoring the test. Read an SEO interval only after its windows have ended (our reasoning).
Questions people also ask
How do you report a 95% confidence interval?
State the estimate, both ends of the range and the confidence level together, then add the p-value, the number of untouched pages and the dates. In the worked example, set A reads: +12.7% versus what five untouched pages predict, 95% interval +3.3% to +22.8%, p = 0.008, with 28-day windows. For a sized result, Penlid's estimator prints the p-value, the interval and the number of untouched pages with the lift, so none of them travels without the others.
How do you interpret confidence interval results?
Start with where the range sits: wholly above zero means the page beat the untouched pages, wholly below zero means it fell behind, and a range that spans zero means the numbers cannot separate the change from chance. Then judge the width, which grows when comparison pages are few or disagree, and the low end, which shows the smallest gain still consistent with the data.
What does it mean if a confidence interval includes 0?
A lift of zero sits inside the range, so the result is too close to call and chance could explain the gap. In Penlid's estimator that is the same as a p-value at or above 0.05. It is not proof of no effect: a wide range can hide a real lift, which is why the calculator also states the smallest lift its numbers could have detected.
Do overlapping confidence intervals mean two results are the same?
Not necessarily. Two intervals that do not overlap indicate a real difference, but two that overlap can still belong to results that differ, because the overlap check is the more conservative of the two tests. Penlid labels an overlapping pair of AI engines as not shown to differ, which is weaker than shown to be the same.
How do you calculate a confidence interval for a rate?
Count the successes k out of n samples and use a Wilson score interval. With p = k ÷ n and z = 1.96, the center is (p + z²/(2n)) ÷ (1 + z²/n) and the half-width is z × √(p(1 − p)/n + z²/(4n²)) ÷ (1 + z²/n); the interval is the center plus and minus the half-width. For 3 of 10 that gives 10.8% to 60.3%, matching Penlid's table. This is the standard formula, shown as our arithmetic. A Search Console click-through rate is also a ratio, clicks divided by impressions, but a binomial interval assumes a fixed number of independent trials with the same chance of success each, as Wikipedia describes it, and whether impressions meet that is an assumption to state, not a fact to rely on.
Sources
- Confidence interval, Wikipedia, read 30 October 2026.
A 95% confidence level does not imply a 95% probability that the true parameter lies within a particular CI calculated from a sample
- Methodology: how we measure, and what we can't, Penlid, read 30 October 2026.
Few comparison pages means a wide interval.
- SEO significance calculator, Penlid, read 30 October 2026.
the interval and p-value are for a question nobody asked
- Performance report (Search results): Overview and basic setup, Google, Search Console Help, read 30 October 2026.
- SEO test calculator, Penlid, read 30 October 2026.
does not shrink as a page gets bigger
- The ASA Statement on p-Values: Context, Process, and Purpose, American Statistical Association, The American Statistician, read 30 October 2026.
A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
- On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals, American Statistical Association, The American Statistician, read 30 October 2026.
failure to reject by the method of examining overlap does not imply failure to reject by the standard method
- Multiple comparisons problem, Wikipedia, read 30 October 2026.
the expected number of non-covering intervals is 5
- Binomial proportion confidence interval, Wikipedia, read 30 October 2026.
It can be safely employed with small samples and skewed observations.
- Poisson distribution, Wikipedia, read 30 October 2026.
- Always Valid Inference: Bringing Sequential Analysis to A/B Testing, arXiv, read 30 October 2026.