How many answers before you can trust a number?

What “based on 18 answers” actually says

Eighteen answers is a small sample from a system that varies run to run, so the figure beside it is a snapshot with visible wobble. It licenses “in this window, roughly 4 in 10 answers mentioned us” and nothing sharper.

The number is not wrong, it is young. Each run is a fresh draw: the same prompt set, untouched on your side, produces different answers on different days because the models sample differently, the sources rotate, and the wording shifts. Small samples swing hardest, which is why the most dramatic percentages in any dashboard are almost always the ones based on the fewest answers.

So the sample line sets the grammar of the claim. Under about ten answers you may say “two of the three answers mentioned us”, in full, with the count. Past that you may give a percentage with its window attached. A claim about direction, that something improved, needs two comparable windows, which is the next section.

Our rule of thumb before reacting to a move

Equal windows on both sides, a threshold you pick before you look, and no charting of single-digit samples. That is the whole rule, and it prevents most of the bad decisions a noisy metric invites.

1. Same window, both sides. The last 30 days against the 30 before it. Never 30 days against 7, and never the day you happened to fix something against the day before it.
2. Pick your threshold first. Ours is 5 points of Visibility across a 30-day window: below that, we write nothing down. If both windows show a 95% confidence interval and the ranges overlap, treat it the same way, no matter what the point estimates say.
3. Single-digit samples are anecdotes. Worth reading the underlying answers, not worth charting, quoting or defending.
4. One change at a time. If you published four pages and changed the pricing copy in the same window, the number cannot tell you which one did it.

A threshold is a choice, not a law. A category with heavy seasonal swings may want 8 points; a very stable prompt set on one engine may read 3. The point is writing the number down before seeing the data, because a threshold chosen after the fact will always justify whatever the dashboard happened to show.

What a 95% confidence interval means for you

A 95% confidence interval is the range the true value most likely falls in, given the answers in hand. Wide range, small sample. If two windows show overlapping ranges, treat the difference as noise.

The maths stays in the tool. What you need is the shape of the thing: a point estimate is a single mark, the interval is how far that mark is allowed to travel before you should stop believing it. Read the range, not the headline.

The range narrows slowly. You need roughly four times as many answers to cut a width in half, which is why the correct way to tighten a number is a longer window or a steadier prompt set, and the easy way is to drop the interval from the chart.

Why a hidden score beats a confident 0 or 100

A score computed from three answers looks precise and is worthless: 0% claims you are invisible when the truth is that you have not asked enough. Hiding the figure until the sample can carry it keeps a slow week from looking like a crisis.

Small samples fail in both directions. One answer that omits you and a brand-new workspace reads 0% for a day; one that names you twice reads 100%. Both are true statements about a handful of draws and false statements about your visibility, and the screenshot outlives the caveat.

The cost of hiding is a blank tile for the first days, which is a fair trade against the cost of a confident 0% appearing in a board deck. Precision you have not earned is worse than a gap that admits what it does not yet know, and the gap invites the question that actually matters: how many more days until this number means something.

How advisible handles small samples

Every figure carries its sample line, perception scores carry a 95% confidence interval, and scores stay hidden until there are enough answers to trust them.

• Sample lines everywhere: “72 / 100, based on 18 answers”, and where a metric is computed over mentions rather than answers the unit stays real, “based on 18 mentions”.
• Sentiment and other perception scores show their interval, so a two-point dip announces itself as overlapping ranges instead of a finding.
• Scores stay hidden below the threshold. A workspace in its first days sees no number rather than a confident one, and the tile says what will appear there once the answers accumulate.
• Daily runs on ChatGPT, Perplexity, Gemini, Grok and Claude through each workspace's own provider key, which is what makes “equal windows” possible: every prompt gets the same number of chances each day.
• Run now for an immediate check after you publish, with the same caveat as before: one run is one answer, and it is a prompt to read, not a metric.

None of this makes the noise go away. It keeps the noise in the interval where it belongs, so the number in front of you is one you can defend in a room that includes someone who wants to disagree with it. The companion pieces are what each metric means and which prompts are feeding them.