Can you measure an SEO change without splitting pages into test and control groups?

You can measure an SEO change without splitting pages into test and control groups, by comparing the changed page with pages you left alone over the same dates. That matched comparison is observational, not randomized, so it gives timing evidence rather than proof, and the comparison pages must be chosen before you see their results.

• Google says it can be difficult to tell whether a page change caused a gain, and suggests checking that gains line up in time with your changes against similar earlier periods (Google, Search Console Help, read 30 October 2026).
• In Penlid's worst simulated case, qualifying comparison pages on the before days added 1.08 percentage points of false "improved" results, while a separate earlier stretch changed the figure by −0.08 (Penlid, methodology, read 30 October 2026).
• Pages picked because they were declining, then left unchanged, came back "improved" 5.50% to 9.20% of the time in Penlid's simulations, against at most 3.10% otherwise (Penlid, methodology, read 30 October 2026).

What does measuring without test and control groups look like?

Measuring without test and control groups means you make the change you want, then compare the changed page with pages you left alone, over the same calendar days. By a split we mean the other design: sorting similar pages into test and control groups before anything changes, ideally at random. Without one, you match the comparison pages by hand instead of assigning them by chance.

You still need the comparison. A page's own history cannot separate your edit from everything else that moved in the same weeks, and Google names news events and a competing site's improvements among the other causes. Its own suggestion is to confirm that an improvement maps closely in timing to your site changes, compared to similar time periods in the past. Pages left alone add a second yardstick, since a season, an update or a holiday moves them on the same days. Outside SEO, this kind of design is known as difference in differences.

What has to be true for the comparison to be fair?

In plain terms, the changed page would have kept moving like its comparison pages had you done nothing. Wikipedia's article on difference in differences (DiD) says DiD requires a parallel trend assumption, and names a shortcoming: something other than the treatment changes in one group but not the other at the same time as the treatment.

The pre-trend check in Penlid's methodology tests this on the past: it repeats the calculation one window earlier, with days −56 to −29 as before and days −28 to −1 as after, and warns if the page was already moving differently from its comparison pages. The free calculator runs it when you type the optional earlier window for every page. The warning is a flag, not a gate or a safeguard: on pages picked because they were declining, it appeared on only 10.9% to 22.7% of them.

How do you choose comparison pages when nobody assigned them?

Choose comparison pages you did not change in either window, that draw clicks from similar queries and are not tiny, and write the list down before you read their numbers, because trying sets until one passes manufactures false wins. The calculator's page warns that when pages are picked to flatter the answer, the interval and p-value are for a question nobody asked.

The paid product reads which pages qualify from days −56 to −29, outside the two measured windows; in the worst simulated case, qualifying them on the before days instead added 1.08 percentage points of false "improved" results, while the separate stretch changed it by −0.08. Penlid lists two exceptions: the size check reads the changed page's baseline, and pages that qualify are ordered by a trend that includes measured days. For a hand-built list, our reasoning is to pick pages from what you knew before the change, such as size, never from the clicks after it.

If you can choose which pages to change, should you split them after all?

Yes, where you control the choice: picking pages blind is cleaner than comparing pages picked for a reason. Penlid's methodology advises, when a result matters, to choose which pages to change without looking at how they were trending, for example every page in a set or alternate pages. The trap is picking by decline: a page that has just had a bad four weeks tends to have a better next four whatever you do, and no comparison group can subtract that (regression to the mean). In Penlid's simulations, pages picked that way and then left unchanged came back "improved" 5.50% to 9.20% of the time, against at most 3.10% otherwise. See also choosing pages to change without bias.

Our reading: alternate pages is a split in all but name, and it earns its keep because it takes the choice out of your hands. A no-split comparison remains the tool for a change already made, or for a single page.

How do you run the comparison with the free calculator?

Fix the pages and windows first, take every figure the same way, and read the answer once the windows end.

1. Write the list first. Pick three to eight pages; if more qualify, decide by something other than their results, such as closeness in size.
2. Fix the windows. Before is days −28 to −1 and after is days 1 to 28, with the change day (day 0) left out, on the same calendar days for every page.
3. Take every figure the same way. In Search Console's Performance report, set the date range to one window at a time and read each page's clicks from the Pages tab. Google labels daily data in Pacific Time and says the newest data can be preliminary, so use windows that have fully ended.
4. Read the interval with the p-value. They come from one distribution, so an interval that excludes zero and p below 0.05 are the same statement. Statistical significance for an SEO change explains p.
5. Wait 38 days. Penlid reads a change 38 days after it goes live: the 28-day window, a 7-day allowance for the newest days to settle and about 3 days of reporting lag. The 7 days is Penlid's constant, not Google's.

On a day-by-day Search Console export, the did-it-work tool chooses comparison pages for you from days −56 to −29.

What if there are too few untouched pages?

With fewer than three untouched pages left, this method cannot give a number: Penlid's result is "inconclusive" or "not measurable", and it does not fall back to a bare before-and-after figure. Our reasoning on when that happens: a change to every page on a site, such as a template, leaves nothing to compare with. Another approach, implemented in the CausalImpact package, predicts what would have happened had no intervention taken place; its authors discuss it in those settings where a randomised experiment is unavailable. Penlid's estimator is a different method.