Change pages chosen by a rule you fix before looking at their results, such as every page in a set or alternate pages, not pages picked because they were declining. Pages picked after a bad stretch tend to improve on their own, and comparison pages chosen differently cannot subtract that. Keep at least three similar pages untouched.
- Unchanged pages picked because their last 28 days sat at least one standard deviation below their own average were called "improved" 5.50% to 9.20% of the time in Penlid's simulations, against at most 3.10% for pages not picked that way (Penlid, methodology, read 30 October 2026).
- With fewer than 10 clicks in the 28 days before a change, Penlid gives a direction and no size, and with none it measures no lift (Penlid, SEO test calculator, read 30 October 2026).
- Penlid refuses to measure a page changed twice within 56 days, or a change that moves clicks between addresses, such as a redirect, a prune or a canonical change (Penlid, SEO test calculator, read 30 October 2026).
Why does picking pages by their results spoil a measurement?
Picking pages by their recent results spoils a measurement because the pick itself tilts the answer before you change anything. Wikipedia calls the general problem selection bias, a bias in a statistical analysis from non-random selection, and notes that in the general case statistical analysis of existing data alone cannot overcome it. The form we measured is picking pages because they were declining: Wikipedia's article on regression toward the mean says the reading after an unusually high or low one tends to land nearer the average, so a page that just had a bad four weeks is likely to do better next whatever you do to it. Penlid's methodology names this regression to the mean and says no comparison group can subtract it, because the comparison pages were not chosen for having had a bad four weeks
. We measured the problem on simulated sites where the true answer was known; the guide to regression to the mean explains the mechanism.
| How the page was picked | Called "improved" |
|---|---|
| Last 28 days at least one standard deviation below its own average; no drift between pages | 5.50% at 4 clicks a day, 5.75% at 40 |
| The same, with 15% drift between pages | 9.15% and 9.20% |
| Not picked that way | At most 3.10% |
We call refreshing pages because they were declining the largest remaining source of false "improved" results, and we cannot remove it. Our pre-trend warning appeared on only 10.9% to 22.7% of those pages.
How do you choose which pages to change without looking at results?
Choose the pages with a rule that cannot see results: change every page in a set, change alternate pages down a list ordered in advance, or draw the pages by lot. Penlid's methodology says that when a result matters you should choose which pages to change without looking at how they were trending
, by changing every page in a set, or alternate pages
. Wikipedia describes random assignment as placing participants in groups by chance, so that group differences are not systematic; if chance or a fixed rule picks the pages, their recent history cannot.
- List the candidates: pages of one kind that earn clicks from the same kind of query and are not tiny.
- Screen every candidate by the same fixed checks, such as those in the table below, before you split. Our reasoning, not a Penlid measurement: a screen applied to all candidates before the split touches both groups alike.
- Split by a rule that ignores results. Sort by URL or publish date and take every other page, or sort by a spreadsheet's random number and take the top half.
- Write the split down before you look. Penlid freezes its pool of untouched pages when it records a change, because
the one fact that cannot be rebuilt afterwards is which pages you left alone
.
What if the pages you most want to change are the declining ones?
Penlid still measures them: its pre-trend check is a warning, not a gate. Read any "improved" as at least as good as the pages left alone, and possibly some recovery that was coming anyway
. Our reasoning, not a Penlid result: when more pages are declining than you can refresh, draw the ones to change by lot and leave the rest alone, so both groups were picked the same way and share whatever recovery is coming. The SEO significance calculator lets you type three to eight comparison pages yourself, so you can use untouched pages from your own split; it takes at most eight, so with more candidates pick the eight by something other than their results, such as closest in size.
Which pages cannot be measured cleanly, however you pick them?
Pages with no clicks or very few, pages changed twice inside 56 days and changes that move clicks to another address cannot be measured cleanly, however you pick them.
| Situation | What Penlid does |
|---|---|
| No clicks in the 28 days before the change | Calls it a new page: no lift can be measured |
| 1 to 9 clicks in those 28 days | Gives a direction and no size |
| 10 or more clicks in those 28 days | Gives a size, its 95% interval and p-value |
| Changed twice within 56 days, or the change moves clicks between addresses (a redirect, a prune, a canonical change) | Refuses to measure it |
| Fewer than 3 untouched pages remain | Gives no verdict ("not measurable") |
Penlid leaves out any page that was itself changed from 56 days before a change to 28 days after it, and the calculator's limits say The comparison pages must also stay untouched for the whole period.
So every page you change drops out of the comparison for any other change inside that window.
Penlid needs at least 3 untouched pages and uses at most 8. Our arithmetic: of 12 similar pages, changing 6 leaves 6 untouched, inside the range, while changing 10 leaves 2, below the minimum. Keep a few more than 3 in reserve, because candidates that fail Penlid's click and size checks are dropped; the guide to control pages covers choosing them.
This guide covers edits made in place. For tests that serve variant URLs, Google recommends rel=canonical on the variants rather than noindex, says to use a 302 redirect rather than a 301 when a test redirects users to a variant, and says cloaking is against its spam policies.
How do you check that a page is worth testing before you change it?
Run the SEO test calculator on each candidate before you change anything: it runs Penlid's own estimator thousands of times on pages shaped like yours and reports how often it would catch a real change of each size. You type the page's clicks over its last 28 days, the number and size of the untouched pages and how differently your pages move from one another. It returns the chance of a clear verdict for lifts and drops of several sizes, the change that gets an 80% chance, and when the answer arrives: 38 days after the change goes live (28 days of after window, plus Penlid's own allowances of 7 days for the newest days to settle and 3 days of reporting lag).
Use it to set pages aside by a rule you fix first, for example every candidate where the change that gets an 80% chance is bigger than any lift you would expect. After the change, read the result with the SEO significance calculator or the did-it-work tool, and see how long to wait after a content update and what to do when a test is inconclusive.
Example (illustrative numbers)
These counts are invented to show the arithmetic; they are not data from any site. Nobody changes anything on these eight similar pages, so the right answer for any grouping is 0%.
| Page | Days −56 to −29 | Days −28 to −1 | Days 1 to 28 |
|---|---|---|---|
| Page 1 | 4,135 | 4,052 | 4,096 |
| Page 2 | 3,545 | 3,013 | 3,527 |
| Page 3 | 4,775 | 4,011 | 4,702 |
| Page 4 | 2,950 | 3,038 | 2,996 |
| Page 5 | 3,879 | 3,724 | 3,810 |
| Page 6 | 4,406 | 3,789 | 4,358 |
| Page 7 | 3,378 | 2,894 | 3,309 |
| Page 8 | 2,634 | 2,581 | 2,664 |
Pick by decline. Pages 2, 3, 6 and 7 fell the most from the first window to the second, by 14% to 16% (Page 3: 4,011 ÷ 4,775 = 0.840). "Refresh" them and leave Pages 1, 4, 5 and 8 alone. The picked pages went from 3,013 + 4,011 + 3,789 + 2,894 = 13,707 clicks to 3,527 + 4,702 + 4,358 + 3,309 = 15,896, so 15,896 ÷ 13,707 = 1.160. The other four went from 13,395 to 13,566, so 13,566 ÷ 13,395 = 1.013. The picked pages did 1.160 ÷ 1.013 = 1.145 times as well: +14.5% for an edit nobody made.
Pick by alternating. Change Pages 1, 3, 5 and 7 and leave 2, 4, 6 and 8 alone. The changed half went from 14,681 to 15,917 clicks, so 15,917 ÷ 14,681 = 1.084; the untouched half went from 12,421 to 13,545, so 13,545 ÷ 12,421 = 1.090. Then 1.084 ÷ 1.090 = 0.994, or −0.6%: no effect, which is the right answer. Both halves rose 8% to 9% because each held two of the dipped pages, and a recovery both sides share cancels out.
Our estimator works page by page on log ratios, so its exact figures would differ a little, and with four pages a side luck can still tilt a split. These counts show direction only.
What this cannot tell you
- A clean pick removes one bias, not all of them. A link, a ranking move or a news event that hits only one changed page still lands in its result, and Penlid's comparison pages are matched, not randomly assigned. The answer is timing evidence, not proof.
- The published rates describe simulated sites. The calculator simulates neither seasonality nor regression to the mean, and the rates for pages picked on decline use comparison pages that were not picked that way. Penlid's published figures do not include a lottery inside a declining set.
- Edits nobody recorded look untouched. A comparison page someone edited outside Penlid looks untouched to it, so keep a change log for the whole site.
Questions people also ask
How do you select pages for an SEO split test?
List pages of one kind that you could change, remove any that fail a screen you set in advance, then divide the rest with a rule that ignores their results: alternate down a list ordered by URL, or sort by a random number. Record the split before the change goes live. Penlid reads one changed page against three to eight untouched pages, so keep enough back.
Should you test on your worst-performing pages?
Not if you want a clean measurement. A page that has just had a poor month is likely to look better next month with or without an edit, and comparison pages chosen another way cannot take that recovery out. In Penlid's simulations, unchanged pages picked this way were called "improved" 5.50% to 9.20% of the time, where pages not picked that way stayed at or under 3.10%.
How many pages do you need for an SEO test?
To get a size rather than only a direction, Penlid's estimator needs the changed page to have at least 10 clicks in the 28 days before the change, plus between 3 and 8 untouched pages. More untouched pages help up to the cap: in Penlid's published simulations, a real +50% lift on a page with 4 clicks a day was found 61.23% of the time with six untouched pages and 70.02% with eight, with no drift between pages.
Can you change several pages at once and still measure each one?
Yes, but every edit takes pages out of the others' comparison. Penlid drops from a page's comparison any page edited between 56 days before that page's change and 28 days after it, so a batch of edits in one month shrinks the untouched pool for the whole batch, and a pool under 3 returns "not measurable". Limit the batch and keep a reserve of similar pages untouched.
Sources
- Minimize A/B testing impact in Google Search, Google, Search Central, read 30 October 2026.
- Random assignment, Wikipedia, read 30 October 2026.
- Selection bias, Wikipedia, read 30 October 2026.
- Regression toward the mean, Wikipedia, read 30 October 2026.
- Methodology: how we measure, and what we can't, Penlid, read 30 October 2026.
choose which pages to change without looking at how they were trending
every page in a set, or alternate pages
no comparison group can subtract it, because the comparison pages were not chosen for having had a bad four weeks
the one fact that cannot be rebuilt afterwards is which pages you left alone
at least as good as the pages left alone, and possibly some recovery that was coming anyway
- SEO test calculator, Penlid, read 30 October 2026.
The comparison pages must also stay untouched for the whole period.
- SEO significance calculator, Penlid, read 30 October 2026.
- Did that content change actually work?, Penlid, read 30 October 2026.