Difference-in-differences compares a changed page's change over time with the change in comparable untouched pages over the same dates. It beats a plain before-and-after comparison because the untouched pages absorb the seasons, updates and site-wide shifts that move every page. What is left is timing evidence, not proof.
- A difference-in-differences estimate is the treated group's change over time minus a control group's change over the same period (Wikipedia, Difference in differences, read 30 October 2026).
- The method assumes parallel trends, and pre-trend tests of that assumption often have low power, so passing one does not necessarily show it holds (arXiv, a survey of difference-in-differences research, read 30 October 2026).
- Google says it can be difficult to tell whether a specific page change directly caused an improvement, because other events also move performance (Google, Search Console Help, read 30 October 2026).
- On simulated sites, Penlid's estimator found a real +80% lift on a 4-click-a-day page, with six untouched pages and no drift, in 92.01% of 200,000 experiments, against 90.79% for an exact t test on the same data (Penlid, methodology, read 30 October 2026).
What is difference-in-differences?
Difference-in-differences is a quasi-experimental technique that subtracts the change in a control group from the change in a treated group over the same period. In SEO the treated group is the page you changed, the control group is pages you left alone, and the two periods are a window before the change and a window after it.
The first difference is each group's own change between the two windows. The second difference is the gap between those two changes, and that gap is the estimate. If your page rose 20% and the untouched pages rose 20% too, the second difference is zero: the page did only what everything did. Both groups must be measured over the same calendar days, so that a season, an update or a holiday moves them alike and the subtraction removes it. Penlid's methodology puts the test as a question: did this page do better than the pages we left alone would have predicted?
Why does a before-and-after comparison mislead for SEO?
A before-and-after comparison misleads because it credits your edit with everything else that changed between the two periods. Penlid's free tool opens with the classic case: You rewrote a page. Traffic went up. So did everyone's, because it was September.
Google's help page on measuring page changes agrees: other events, such as news events or improvements to a competing site, also move performance.
Wikipedia's article on internal validity names the rival explanations; the table sets four of them against untouched pages.
| Rival explanation | In SEO | Do untouched pages help? |
|---|---|---|
| History | A holiday, season or Google update lands in the same weeks | Yes, when it also hits the untouched pages on the same dates |
| Maturation | The page was already climbing or decaying | Partly: only if the untouched pages were moving at a similar pace |
| Regression toward the mean | The page was picked because it had a bad month | No: the untouched pages were not picked for a bad month |
| Selection | You edited your strongest pages and left the weaker ones alone | Only if the untouched pages resemble the changed page in size and trend |
A plain before-and-after comparison has no group to set against any of the four (our reasoning). It still pins down timing: for core updates Google recommends waiting at least a full week after a core update completes
and then comparing with the week before the update started, which helps pinpoint what changed. For the timing side, see measuring what a Google update did to your site.
How does Penlid turn the two differences into one number?
Penlid compares each page's clicks after the change with its clicks before as a log ratio, then sets the changed page's ratio against what the untouched pages' ratios predict. Ratios remove the difference in size: a 20% rise is the same number on a 100-click page and on a 10,000-click page. The result is stated as "X% versus what the untouched pages predict", not as a difference in percentage points, and it cannot fall below −100%, because a ratio cannot.
Pages also drift apart on their own: a season hits one more than another, a ranking wobbles, a link lands on one page and not its neighbor. When the untouched pages spread further apart than counting noise explains, Penlid estimates the surplus as between-page drift (the Paule-Mandel method) and assumes the changed page can drift by the same amount, so the bar for a result is higher on a site whose pages wander. The 95% interval and the p-value are read from one distribution, so "the interval excludes zero" and "p below 0.05" are the same statement.
With fewer than three comparison pages the result is "inconclusive" or "not measurable" and says why: We do not fall back to a bare before-and-after number and let it imply a cause.
The methodology sets out every step.
What does difference-in-differences assume?
Difference-in-differences assumes parallel trends: without the change, the changed page and the untouched pages would have moved alike over time. The assumption fails when something other than your change hits only one group, such as a new link to your page or a ranking move on it. Nobody can observe the version of events in which you made no change, so the assumption can be doubted but never proven (our reasoning).
The usual probe is the stretch before the change. Penlid repeats its calculation one stretch earlier, treating days −56 to −29 as "before" and days −28 to −1 as "after", and attaches a warning when the page was already moving differently (p below 0.05). It warns instead of blocking because refreshing a page that is already decaying is the main use of the product.
A clean pre-trend check is weaker evidence than it sounds. A survey of difference-in-differences research on arXiv cautions that the absence of a significant pre-trend does not necessarily imply that parallel trends holds, and that these tests often have low power. Our numbers agree: on simulated pages that were picked because they were declining and then left unchanged, the warning appeared on only 10.9% to 22.7% of them.
How well does the estimator work, and what can it not fix?
On simulated sites where the true answer was known, Penlid's estimator found a real +80% lift 92.01% of the time at 4 clicks a day with six untouched pages and rarely raised a false alarm when nothing was done, but it could not remove the error from pages picked because they were declining.
| Situation | Result |
|---|---|
| Real +80% lift, 4 clicks a day, six untouched pages, no drift | Found 92.01% of the time; an exact t test on the same data found 90.79% |
| Nothing done to the page, 675 scenarios | Called significant 3.26% of the time on average with the estimator alone (worst scenario 5.85%), where a 95% test allows 5% |
| Nothing done, but the page was picked because it was declining | Called "improved" 5.50% to 9.20% of the time, against at most 3.10% when not picked that way |
The last row is the one a comparison group cannot fix. Wikipedia's lead warns that, depending on how the treated group is chosen, the method can still suffer from biases such as mean regression. A page picked after a bad four weeks tends to have a better next four whatever you do, and the untouched pages were not picked that way, so they cannot subtract it. Where you did pick on decline, read "improved" as at least as good as the pages left alone, and possibly some recovery that was coming anyway
. See regression to the mean and choosing pages to change without bias.
How do you run a difference-in-differences comparison on your own pages?
Run it in three steps: choose the untouched pages before you read their numbers, measure identical dates for every page, and run the free tool, then read the interval and any warning with the percentage.
- Choose the untouched pages first. Pick at least three pages that earn clicks from the same kind of query, are not tiny and were not changed in either window, and write them down before you look at their results; the guide to control pages covers the choice.
- Use identical dates. Penlid's windows are 28 days before and 28 days after the day the change went live, which is left out.
- Run the free did-it-work tool. It takes a Search Console export made with Compare or, better, a day-by-day table with Date and Page columns, and your file never leaves your browser. The day-by-day file lets it pick comparison pages from days −56 to −29 and run the pre-trend check; a Compare export cannot, so the tool uses every other page with at least 10 clicks across the two periods and says so above the number.
Wait until both windows have fully ended; see how long to wait after a content update for the timing.
Example (illustrative numbers)
These counts are invented to show the arithmetic; they are not data from any site. All five pages were measured over the same 28 days before and the same 28 days after the rewrite.
| Page | Before | After | Change |
|---|---|---|---|
| Changed page | 740 | 958 | +29.5% |
| Untouched page A | 3,420 | 3,690 | +7.9% |
| Untouched page B | 2,090 | 2,486 | +18.9% |
| Untouched page C | 1,715 | 1,941 | +13.2% |
| Untouched page D | 2,265 | 2,661 | +17.5% |
| Four untouched pages together | 9,490 | 10,778 | +13.6% |
The two differences. The changed page moved 958 ÷ 740 = 1.2946, which is +29.5%, and the four untouched pages together moved 10,778 ÷ 9,490 = 1.1357, which is +13.6%. A before-and-after reading stops at the first number. Had the page moved like the untouched pages, it would have gone from 740 to 740 × 1.1357 = 840.4 clicks. It got 958, which is 958 ÷ 840.4 = 1.1399 times the prediction: +14.0% versus what the untouched pages predict, less than half of the +29.5%. On the log scale the same step is a subtraction: ln 1.2946 − ln 1.1357 = 0.25820 − 0.12725 = 0.13095, and e raised to 0.13095 is 1.1399.
Why ratios, not clicks. Subtracting raw clicks instead, the changed page gained 958 − 740 = 218 and the average untouched page gained (10,778 − 9,490) ÷ 4 = 322, so the gap is 218 − 322 = −104 clicks, a false loss, because the changed page is smaller (740 clicks against an average of 2,372.5) and cannot add as many clicks even when it grows faster. In our reasoning, subtracting raw clicks only makes sense for pages of similar size, which is why Penlid works with ratios.
Is it more than chance? Typed into the SEO significance calculator, the same counts give +13.5% versus what the untouched pages predict (it works page by page with a random-effects model, so it differs a little from our pooled +14.0%), with a 95% interval of −4.7% to +34.8% and p = 0.12. The interval includes zero and p is above 0.05, which are the same statement, so the answer is too close to call; the smallest lift it could have detected is about +19.1%. The changed page has only 740 clicks and the untouched pages disagree with one another, which widens the interval.
The opposite season. If the page instead fell from 740 to 709 clicks (709 ÷ 740 = 0.9581, so −4.2%) while the four untouched pages fell from 9,490 to 8,160 (8,160 ÷ 9,490 = 0.8599, so −14.0%), a before-and-after reading calls the rewrite a loss. Against the prediction of 740 × 0.8599 = 636.3 clicks, 709 ÷ 636.3 = 1.1143 is +11.4% versus the untouched pages, though chance could still explain that gap.
What this cannot tell you
- Timing evidence, not proof. Untouched pages remove what hit every page, not what hit only yours: a link, a ranking move, a news event or an edit nobody recorded.
- Parallel trends stay an assumption. A pre-trend check can raise a warning but cannot confirm it, and Penlid's comparison pages are matched, not randomly assigned, and matching cannot remove all confounding.
- Picking pages because they were declining inflates "improved". The comparison cannot subtract that recovery, and the pre-trend warning catches only a minority of such cases.
- Few untouched pages mean a wide interval. With three to eight, the drift between pages is itself uncertain, so on a small site the honest answer is often "we cannot tell", which is not "it did nothing". See inconclusive SEO tests, or measuring without page groups when no untouched pages exist.
- Small pages get a direction, not a size. A page with 1 to 9 clicks in the 28 days before the change gets a direction and no size, and one with none gets no measurement.
Questions people also ask
What is a difference-in-differences approach?
A difference-in-differences approach estimates the effect of a change by measuring how far the changed group moved and subtracting how far a comparison group moved over the same dates. The comparison group stands in for what would have happened anyway, provided the two groups would have moved alike without the change. For SEO, the changed group is usually one page and the comparison group is pages you left alone.
What is the difference-in-differences equation?
In words, the estimate is the changed group's change minus the comparison group's change. Written out, it is (changed after − changed before) − (untouched after − untouched before). Clicks differ enormously from page to page, so Penlid divides instead of subtracting: for each page it divides the clicks after the change by the clicks before (each with half a click added), takes the natural log, and sets the changed page's value against what the untouched pages predict, converted back to a percentage.
Does difference-in-differences account for seasonality?
It removes the part of a seasonal swing that hits the changed page and the untouched pages alike, because both are measured over the same calendar days. It cannot remove a season that hits your page differently. Penlid treats that unevenness as between-page drift, which widens the interval, and its decay check applies the same idea by subtracting the site-wide move over its window, so a January dip that hit everything does not make every page look like it is decaying.
What is a difference-in-differences example for SEO?
Suppose a rewritten page rose from 500 to 575 clicks, which is +15%, while pages you left alone rose from 8,000 to 8,600, which is +7.5%, over the same 28 days. The page did 1.15 ÷ 1.075 = 1.0698 times as well as the prediction, so +7.0% versus the untouched pages, not +15%. Whether chance explains a gap that size depends on the interval, which the free tools report.
Sources
- Difference in differences, Wikipedia, read 30 October 2026.
- What's Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literature, arXiv, read 30 October 2026.
- Internal validity, Wikipedia, read 30 October 2026.
- Performance report (Search results): Common tasks and use cases, Google, Search Console Help, read 30 October 2026.
- Google Search's core updates and your website, Google, Search Central, read 30 October 2026.
waiting at least a full week after a core update completes
- Methodology: how we measure, and what we can't, Penlid, read 30 October 2026.
did this page do better than the pages we left alone would have predicted?
We do not fall back to a bare before-and-after number and let it imply a cause.
at least as good as the pages left alone, and possibly some recovery that was coming anyway
- Did that content change actually work?, Penlid, read 30 October 2026.
You rewrote a page. Traffic went up. So did everyone's, because it was September.
- SEO significance calculator, Penlid, read 30 October 2026.