Why AI answers change from day to day
Sampling, model updates, source churn and your own chat history all move AI answers day to day. Why one snapshot misleads, and how to read the change.
- A language model generates text token by token with randomness in the mix, so the same prompt reaches different wording and sometimes different brands.
- Four things move an answer between days: sampling, model updates, changes in the retrieved sources, and the history in your own session.
- One screenshot is an anecdote. A daily run schedule turns the same prompt into a sample you can average and compare.
- Judge change by comparing equal length windows and reading the confidence interval, then confirm across more than one prompt before reporting it.
Why the same prompt gives a different answer each time
A language model does not look up one stored reply. It writes text one token at a time with randomness in the sampling, so the same prompt can take different paths and land on different wording, different sources and sometimes a different list of brands. This is normal behaviour, nothing wrong with your account.
The API knob for this is called temperature, and OpenAI's API reference describes it as the sampling temperature to use, where higher values make output more random and lower values make it more focused and deterministic. Lowering it reduces variation without removing it. A 2025 engineering writeup from Thinking Machines traced the residual variation to how inference servers batch requests behind the scenes: even a temperature of zero can diverge between runs because the hardware path differs.
The question keeps showing up in public forums. One OpenAI community thread from December 2025 asks running the same prompt multiple times sometimes produces noticeably different responses, is this expected, and the answer from other developers is yes, the models are not deterministic by default. Another thread asks why the answer varies for the same question asked multiple times. You are not holding it wrong.
Four things that move an answer between days
Sampling changes the wording, model updates change the writer, source churn changes the material, and your own history changes the question the model thinks it was asked.
- Sampling. Within one model version, randomness in token choice shifts phrasing, order and which of several defensible options the answer picks. This is the day to day wobble.
- Model updates. Vendors ship new versions on their own schedule. A new writer can produce a different shortlist from identical sources, which looks exactly like your GEO work succeeding or failing, and is neither.
- Retrieved-source churn. Answers are assembled at question time. Pages get edited, reviews publish, forum threads rise and fall, a comparison roundup quietly adds a product. The shortlist the engine pulls today is not the one it pulled last Tuesday.
- Personalisation and history. Prior messages, account settings, location and experiments all change the input. The same sentence typed into a fresh window and into a long thread are two different requests wearing the same words.
Why two people get different answers to the same question
Because they are not sending the same request. Chat history, account settings and model choice all change what the model reads before it answers, and a fresh window sees none of your context.
This is the confusion behind most screenshots shared internally: someone in a long project thread asks the question and gets a different list from a colleague who opened a clean tab. On top of that sit sampling, which shortlist version they are on, and any A/B test the vendor is running that week. A single conversation proves what one person saw once, in their own context, at that moment.
The measurement implication is straightforward. Collect the prompts from a clean, standardised session on a schedule, so the only thing varying from day to day is the world the engine is reading, and not who happened to ask.
What this means for tracking AI visibility
One screenshot is an anecdote. A visibility number only means something next to the count of answers behind it, so tracking means running the same prompts on a schedule and reading windows instead of moments.
Daily repetition is what converts variance into a sample. Ask the same prompt enough times and the share of answers that mention you settles into a number you can compare with last week, while any single answer stays useless for decisions. That is also why every figure should say how many answers it rests on: “40% based on 90 answers” and “40% based on 3 answers” are different claims wearing the same percentage.
In advisible this is the product loop: each prompt runs daily on ChatGPT, Perplexity, Gemini, Grok and Claude, every original answer is kept unedited in the answer archive, and every metric prints its sample size. Run now exists for the moments you just changed a page and want a fresh sample without waiting for tomorrow. Scores built from few answers stay hidden until there are enough of them to read, which is a slower dashboard and a truer one.
A decision rule for telling signal from noise
Compare equal length windows, require the change to clear its confidence interval, and confirm it across more than one prompt before you tell anyone a number moved.
- Collect for at least two weeks before judging anything. A month is better. Anything shorter is weather, not climate.
- Compare equal windows. This seven days against the previous seven. Never today against a monthly average, because the windows have different noise in them.
- Read the 95% confidence interval, which is the range the true value most likely falls in. If last week's interval and this week's overlap, the correct reading is flat.
- Look for agreement. One prompt flipping is normal. The same direction across several prompts and more than one engine is a pattern.
- Open the answers before announcing a win. There is usually a visible cause: a new citation, a competitor page that dropped out, or a model version change you had not heard about.
How many answers you need before a difference means anything depends on how big the difference is and how wobbly your prompt set is. That calculation has its own page: how many answers before you trust a number.
Common questions
Is this expected behaviour?
Yes. Models are not deterministic by default, so the same prompt with the same settings can branch into different outputs. Engine builders acknowledge the same thing in their own forums.
Does ChatGPT give the same answer to everyone?
No. Chat history, account settings, model choice and sampling all change what the model reads before it answers, so two people sending the same words are rarely sending the same request.
Can I force deterministic answers?
Not from a chat window. API users can lower a sampling setting called temperature, which reduces variation, and even the lowest setting is not perfectly repeatable on a production serving system. You cannot control this from outside, so the workable response is to collect many answers instead of one.
How often should I run each prompt?
Daily is the practical default: it grows a comparable sample at a cost you can predict. Running the same prompt twenty times in one hour tells you about that hour only, and the sample keeps growing with days, not with minutes.
Do model updates wipe out my progress?
They move the baseline. A new model version can change mention rates overnight regardless of anything you published, so note the date when you suspect a release and read the weeks on either side of it as separate series.
How many answers before you can trust a number?
What a sample line like based on 18 answers licenses you to say, our rule of thumb for ignoring noise, and how a confidence interval reads without the math.
What is AI search visibility?
AI search visibility is how often your brand shows up when buyers ask ChatGPT, Perplexity or Gemini about your category. What it means and how to measure it.
How ChatGPT, Perplexity and Gemini choose which brands to mention
No engine publishes how it picks brands. The inputs are observable: what the sources say, query fit, consistency and prominence. What you can influence.
Visibility, average position, share of voice and sentiment explained
What each AI visibility metric counts, worked examples with invented numbers, what each one cannot tell you, and which to put in front of your boss.
Track this for your own brand
advisible runs your prompts on ChatGPT, Perplexity, Gemini, Grok and Claude every day, and every number shows how many answers it is based on.