Why did your AI visibility change when you changed nothing? Model versions and sampling noise

AI visibility can change when you changed nothing, because the engine's model can be updated under the same name and because answers vary from run to run. We treat a rate measured across a model update as two measurements, not one, and a small sample can move on chance alone.

• OpenAI's ChatGPT release notes have 15 entries whose titles begin with GPT- between 10 February and 26 October 2026, including updates to GPT-5.5 Instant on 28 May and 24 June (OpenAI, ChatGPT release notes, read 30 October 2026).
• Bing's AI Performance help page lists model updates among the reasons citation volume changes (Microsoft, Bing Webmaster Tools Help, read 30 October 2026).
• A 2026 preprint on AI search says that identical queries submitted at different times can produce different responses and cite different sources (arXiv preprint, read 30 October 2026).
• Penlid's methodology table reads 3 of 10 samples as 10.8% to 60.3% and 300 of 1,000 as 27.2% to 32.9% at 95%, though both are 30% (Penlid, methodology page, read 30 October 2026).

Can an AI engine's model change without its name changing?

Yes: OpenAI's ChatGPT release notes show one model name, GPT-5.5 Instant, rolled out on 5 May 2026 and updated on 28 May and 24 June.

That is a rollout and two updates under one name in 50 days (our count: 26 days from 5 to 31 May, plus 24 in June). So two readings taken either side of 24 June can be readings of two versions of one named model, although nothing in those sentences says what changed for any particular prompt. A 3 March 2026 entry for GPT-5.3 Instant describes richer and better-contextualized results when searching the web. That clause says nothing about which sites get cited, so we read it only as a sign that web search is something an update can touch (our reasoning).

The pattern is older than 2026: a 2023 preprint on ChatGPT found that the behavior of the "same" LLM service can change substantially in a relatively short amount of time. Microsoft's help page for Bing's AI Performance report also lists model updates among the reasons citation volume changes.

Why does the same question get different answers when the model has not changed?

Because a language model's answer is not deterministic: each run is one draw from a spread of possible answers, and a new draw can differ from the last (our reading). Our methodology says Answers from language models are stochastic: ask the same question twice and you can get two different sets of recommended vendors. A 2026 preprint on measuring AI search says that identical queries submitted at different times can produce different responses and cite different sources. Our guide to checking whether ChatGPT recommends your brand explains why one answer is not a measurement.

How large the noise is depends on the number of samples; the table shows the 95% Wilson interval our methodology uses at four sample sizes.

As of 30 October 2026, our free AI visibility check asks one engine, Claude with live web search, four of the questions your buyers actually type, and its page says four samples is four samples and we won't pretend otherwise. Suppose one run finds 1 of 4 and a later run finds 3 of 4: the intervals, 4.6% to 69.9% and 30.1% to 95.4%, overlap, so under our rule that is "no significant change". And if your true rate stayed at 30%, a run reads 0 of 4 with probability 0.7 × 0.7 × 0.7 × 0.7 = 0.24, about one run in four (our arithmetic, assuming independent samples).

How can you tell a model update from sampling noise?

Record the model version with every sample, split the data at each recorded change, and put an interval on each part.

1. Record the version. Our methodology says we record the model version returned with every sample, and that the dashboard names the version behind each engine's rate and flags, with the date, a change inside the window you are reading. As of 30 October 2026 there is no citation chart to annotate yet, so the note sits in words under the per-engine rates.
2. Split at the recorded date. A rate that spans a model change is two measurements, not one. Pick the split from the version record before you look at the numbers, because a split placed where the line drops can make noise look like a step (our reasoning).
3. Put an interval on each part. Our rule is Two figures are reported as significantly different only when their intervals do not overlap. It is conservative and has no correction for many comparisons, so each extra comparison makes a spurious difference more likely.
4. Hold the model still where you can. OpenAI's API documentation says different snapshots in a model family could produce different results and recommends pinning production applications to specific snapshots. Anthropic says the underlying model remains constant for the lifetime of that ID for a Claude model ID, while the convenience aliases it accepts for some earlier models point to the most recent dated snapshot for that minor version, so in our reading an alias can move. It adds that infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed, so pinning holds the weights, not everything around them (our reading). Both pages describe API use, and our methodology warns that an API can differ from the consumer app.

Even then, a change that lines up with a version date is timing evidence, not proof. Our guide to attributing AI visibility to a content change covers what can and cannot be measured.

Does a confidence interval account for a model update?

No: a Wilson interval is computed from the number of hits and the number of samples alone, so it measures sampling noise for the prompts you asked and cannot see which model version answered (our reasoning from its formula). A tight interval around a rate that spans a model change is therefore a tight interval around an average of two different things.