A content update worked if the updated page beat comparable untouched pages over the same dates by more than chance explains. Rising traffic alone proves little, because seasons and Google updates move many pages at once. Compare the page with pages you left alone, over identical calendar days, and read the interval, not only the percentage.
- Google says it can be hard to tell whether a page change caused a gain, and suggests checking that gains line up in time with your changes against similar earlier periods (Google, Search Console Help, read 30 October 2026).
- Penlid reads a page change 38 days after it goes live: a 28-day window, a 7-day allowance for the newest days to settle and about 3 days of reporting lag (Penlid, SEO test calculator, read 30 October 2026).
- In 675 simulated scenarios with nothing done to the page, Penlid's estimator called a change significant 3.26% of the time on average, where a 95% test allows 5% (Penlid, methodology, read 30 October 2026).
- Pages picked because they were declining, then left unchanged, came back "improved" 5.50% to 9.20% of the time in Penlid's simulations, against at most 3.10% otherwise (Penlid, methodology, read 30 October 2026).
Why is a plain before-and-after comparison not enough?
A plain before-and-after comparison cannot separate your edit from everything else that changed in the same weeks. Such rival explanations are called threats to internal validity, and four of them apply to page measurements:
- History. A season, a holiday or a Google update lands in the same weeks as your edit.
- Maturation. A page's clicks climb or decay on their own.
- Regression toward the mean. A page picked for a bad stretch tends to look better in the next one.
- Selection. The pages you changed differ from the pages you left alone.
Google's help page on measuring page changes warns that other events can also contribute to changes in performance, and suggests a timing check: do gains line up with your changes? That is evidence about when something happened, not why. A difference-in-differences comparison adds the missing piece: pages you left alone (comparison or control pages), measured over the same dates. The guide to control pages covers choosing them.
What should a usable answer look like?
A usable answer states the lift versus what the untouched pages predict, with its 95% interval and p-value, and says plainly when the evidence cannot support a size. The Penlid methodology asks did this page do better than the pages we left alone would have predicted?
A result of +30% means 30% more clicks than the untouched pages predicted.
| Situation | What the answer contains |
|---|---|
| No clicks before the change | "New page": no baseline, so no lift to measure |
| 1 to 9 clicks before the change | A direction (improved, declined or too close to call) and no size |
| 10 or more clicks before the change | A size, its 95% interval and p-value, or "we cannot tell" plus the smallest lift it could have detected |
| Fewer than 3 usable untouched pages | "Not measurable" or inconclusive, with the reason |
How do you run the comparison, step by step?
Run the comparison in five steps: record the change, choose the untouched pages before you look, fix the windows, export identical dates for every page, and read the interval with its warnings.
- Record the change. Write down the page, the date it went live and what changed. Make one change at a time and leave the page alone until the after window ends.
- Choose the untouched pages first. Pick pages that earn clicks from the same kind of query, are not tiny and were not changed from 56 days before your change to 28 days after it, and write them down before you read their numbers. Penlid's SEO significance calculator needs at least three and accepts at most eight; if more qualify, choose by something other than their results, such as closeness in size. Which pages you left alone is the one fact you cannot rebuild afterwards.
- Fix the windows. Use 28 days before the change and 28 days after, with the day of the change left out, and only windows that have fully ended, because Google says the newest data can be preliminary. The table below gives the days.
- Export the same dates for every page. Use identical calendar days for the changed page and each untouched page; Search Console labels daily data in Pacific Time. The free did-it-work tool takes a Search Console export made with Compare or, better, a day-by-day table with Date and Page columns, and your file never leaves your browser.
- Read the interval, the p-value and the warnings together. The tool measures the changed page against what the untouched pages predict and prints the lift, a 95% interval and a p-value. On a day-by-day file it also warns you if the page was already moving differently before the change.
| Window | Days | What it is for |
|---|---|---|
| Earlier window | −56 to −29 | Choosing untouched pages, and checking whether the page was already moving differently |
| Before window | −28 to −1 | The baseline |
| Day of the change | 0 | Left out |
| After window | 1 to 28 | The measured result |
How long should you wait before reading the result?
Wait until the 28-day after window has ended and the newest days have settled, which Penlid puts at 38 days after the change goes live (28 + 7 + 3, as set out on the SEO test calculator). The 7 days is our own allowance, not a Google figure. Google says collected data is usually available in 2–3 days, and it gives no fixed time for a content change to move rankings. The guide to how long to wait after a content update covers the cases.
What can still fool a comparison against untouched pages?
Three things can still fool a comparison against untouched pages: changing pages because they were declining, an event that hits only the changed page, and a Google update inside your windows. The first is the largest remaining source of false "improved" results, and we measured it on simulated sites where the true answer was known.
| Situation | Result |
|---|---|
| Real +80% lift, 4 clicks a day, 6 untouched pages, no drift | Found 92.01% of the time |
| Nothing done to the page, 675 scenarios | Called significant 3.26% of the time on average; worst scenario 5.85% |
| Nothing done, page picked because it was declining, no drift | Called "improved" 5.50% (4 clicks a day) and 5.75% (40 clicks a day) |
| The same, with 15% drift between pages | Called "improved" 9.15% and 9.20% |
| Nothing done, page not picked that way | Called "improved" falsely at most 3.10% |
No comparison group can remove the first error, because the untouched pages were not chosen for having had a bad four weeks, and the pre-trend warning appeared on only 10.9% to 22.7% of those declining pages. What reduces it is choosing which pages to change without looking at how they were trending: every page in a set, or alternate pages. Where you did pick on decline, read an "improved" as at least as good as the pages left alone, and possibly some recovery that was coming anyway
. See regression to the mean and choosing pages to change without bias.
The other two are timing problems. A link, a ranking move or a news event can hit only the changed page, and the untouched pages cannot subtract it. A Google update inside your windows is a second event on the same dates. For a core update Google recommends waiting at least a full week after a core update completes
before comparing; see how to measure what a Google update did to your site.
Example (illustrative numbers)
These counts are invented to show the arithmetic; they are not data from any site.
| Page | Before | After | Change |
|---|---|---|---|
| Changed page | 1,240 | 1,612 | +30.0% |
| Untouched page A | 2,050 | 2,460 | +20.0% |
| Untouched page B | 880 | 1,030 | +17.0% |
| Untouched page C | 1,530 | 1,860 | +21.6% |
| Untouched page D | 640 | 770 | +20.3% |
| Four untouched pages together | 5,100 | 6,120 | +20.0% |
A plain before-and-after reads 1,612 ÷ 1,240 = 1.30, so +30%. The four untouched pages together went from 5,100 to 6,120 clicks, and 6,120 ÷ 5,100 = 1.20, so they predict +20%. Against that prediction the page did 1.30 ÷ 1.20 = 1.083 times as well, which is +8.3% versus what the untouched pages predict. Subtracting would give 10 points, not 8.3%.
Typed into the SEO significance calculator, the same counts give +8.3% with a 95% interval of −0.6% to +18.1% and p = 0.07, and the smallest lift the measurement could have detected is about +9.0%. The interval includes zero and p is above 0.05, which are the same statement, so the answer is too close to call: chance could explain the gap.
Now change only the month. If the four pages had gone from 5,100 to 5,355 clicks (2,150, 925, 1,607 and 673 after), they would have risen 5.0%, and the same +30% reads 1.30 ÷ 1.05 = 1.238, so +23.8% versus the prediction, with a 95% interval of +13.9% to +34.6% and p below 0.001: improved. The changed page is identical in both months; only the pages left alone changed the verdict.
What this cannot tell you
- Timing evidence, not proof. The untouched pages remove what hit every page, not what hit only yours: a link, a ranking move, a news event, or an edit nobody recorded.
- It depends on the pages you compare. Comparison pages are matched, not randomly assigned. Matching on earlier behavior removes a lot of confounding but not all of it, and a different set can give a different answer.
- "We cannot tell" is not "it did nothing". A small change on a small page often cannot be told from chance. The result then names the smallest lift the measurement could have detected, and a smaller real lift may sit below it. The guide to inconclusive SEO tests covers what to do next.
- It measures clicks, not what they are for. It says nothing about leads or revenue and promises nothing about rankings.
- Old changes may be out of reach. Search Console retains the most recent 16 months of data, so an old change's before window may be gone.
Questions people also ask
How do you know if SEO is working?
SEO is working when the pages you worked on gain more than pages you left alone over the same dates, by more than chance explains. A single site-wide trend cannot show that, because a season moves many pages at once. When debugging a traffic drop, Google suggests setting the date filter to the last 16 months to see yearly patterns and comparing against a similar earlier period; the guide to seasonality in SEO data shows how.
How long does it take to see SEO changes?
Google gives no fixed time. It describes some changes acting within days and others needing several months. Penlid reads a single page change 38 days after it goes live; that is our own rule, not a Google figure.
Does updating content help SEO?
Google does not promise that it will. It says no change is guaranteed to have a noticeable effect, and that its results are dynamic, with constant change that can bring both gains and drops, so an updated page can rise, fall or stay level. Measuring the page against pages you left alone gives timing evidence from your own site, not proof.
How do you measure an SEO content update?
Record the change date, choose the untouched pages before you look, then compare the 28 days before with the 28 days after for every page over identical dates. The free did-it-work tool does the arithmetic from a Search Console Compare export, and its page explains how to set the two date ranges.
Sources
- Performance report (Search results): Common tasks and use cases, Google, Search Console Help, read 30 October 2026.
- Connect Search Console to Google Analytics, Google, Analytics Help, read 30 October 2026.
- Performance report (Search results): Troubleshooting data discrepancies, Google, Search Console Help, read 30 October 2026.
- Performance report (Search results): About the data, Google, Search Console Help, read 30 October 2026.
- Google Search's core updates and your website, Google, Search Central, read 30 October 2026.
waiting at least a full week after a core update completes
- Debugging drops in Google Search traffic, Google, Search Central, read 30 October 2026.
- Internal validity, Wikipedia, read 30 October 2026.
- Methodology: how we measure, and what we can't, Penlid, read 30 October 2026.
did this page do better than the pages we left alone would have predicted?
at least as good as the pages left alone, and possibly some recovery that was coming anyway
- Did that content change actually work?, Penlid, read 30 October 2026.
- SEO significance calculator, Penlid, read 30 October 2026.
- SEO test calculator, Penlid, read 30 October 2026.