Measuring content and search changes, honestly
Each guide answers one question: a direct answer, a worked example with illustrative numbers, what the method cannot tell you, and the primary sources. Penlid builds the free tools most of them use.
Measuring changes
- How do you tell if a content update actually worked?
A content update worked if the updated page beat comparable untouched pages over the same dates by more than chance explains.
- What are control pages in SEO measurement, and how do you choose them before you look?
A control page is an untouched page measured over the same dates as the page you changed, so that anything affecting every page shows up in it.
- What is difference-in-differences, and why does it beat a before-and-after comparison for SEO?
Difference-in-differences compares a changed page's change over time with the change in comparable untouched pages over the same dates.
- How long should you wait before judging a content update?
A content update can be judged once its 28-day after window has ended and Search Console's newest days have settled, which Penlid puts at 38 days.
- How do you separate seasonality from a real SEO change?
To separate seasonality from a real SEO change, measure a page against pages you left alone over the same calendar days and read the gap.
- Why do pages that dropped often recover on their own, and how much does that fool a measurement?
Regression to the mean is one reason pages that dropped often recover on their own: part of a low month is bad luck, and next month brings new luck.
- Which pages should you change if you want a clean measurement?
Change pages chosen by a rule fixed before you see results, such as alternate pages, not pages picked because they were declining.
- How do you measure what a Google update did to your site, and why is it timing evidence rather than proof?
Compare whole weeks of clicks before a Google update began with whole weeks after its rollout, then judge the change against your site's ordinary swings.
- What should you do when an SEO test comes back inconclusive?
An inconclusive SEO test means the evidence cannot show whether your change did anything, which is different from showing that it did nothing.
Statistics for SEO
- What does "statistically significant" mean for an SEO change, and when does a lift count?
A statistically significant SEO lift is a gain that would be rare if the change did nothing, measured against pages you left alone.
- How much traffic does a page need before you can measure a change to it?
A page needs 10 or more clicks in the 28 days before a change for Penlid to put a size on that change.
- How do you put a confidence interval on an SEO result, and how do you read one in a report?
A confidence interval on an SEO result is a range around the measured lift whose width shows how noisy the comparison is.
- What is the minimum detectable effect in SEO testing, and how do you size a test?
A minimum detectable effect (MDE) is the smallest real change that a test would catch with a stated probability, conventionally 80%.
- Can you measure an SEO change without splitting pages into test and control groups?
You can measure an SEO change without splitting pages into test and control groups, by comparing the changed page with pages you left alone.
- If you measure 20 page changes, how many false wins should you expect?
Twenty page changes that did nothing give about one false verdict at the 5% level, and at least one 64% of the time if the tests are independent.
Search Console analysis
- Why does a Search Console export stop at 1,000 rows, and how do you get the rest?
A Search Console export stops at 1,000 rows because Google truncates it to 1,000 rows of representative examples, the same cap the report's table has.
- How do you keep Search Console data older than 16 months?
Search Console data older than 16 months can be kept only by saving it while it is still inside the window Search Console keeps.
- Was that traffic drop real, or a Search Console data anomaly?
A Search Console traffic drop is more likely real when clicks fell, because Google says its documented impressions logging error did not affect clicks.
- Why do the last few days in Search Console change, and when is a day final?
The last few days in Search Console change because the newest data can be preliminary; no Google page we read gives a figure for when a day is final.
AI search
- How do you read Search Console's Generative AI performance report, and what does it not show?
Search Console's Generative AI performance report counts impressions only: how often links to your site were shown in Google's AI Overviews and AI Mode.