Visibility jumped eight points this week. Someone in the channel says it's the article published on Tuesday. Before agreeing, rule out the boring explanations — they're more common than the interesting ones.
#1. Variance
AI answers are not deterministic. The same prompt, collected twice, can return different brands in a different order. With a small panel, a handful of answers flipping can move a percentage by several points.
Check: how many runs does each number rest on? If you only collected once per prompt, you can't separate signal from noise. Collect repeatedly and look at the range. A change inside the range isn't a change.
#2. The panel changed
Adding, removing or rewording prompts changes what you're measuring. Even a "small" edit — a new competitor name in a comparison prompt — shifts the mix.
Check: compare the same fixed prompts across both periods. Report newly added prompts separately.
#3. Collection changed
A provider model update, a new default setting, or a change in country or language can move answers without anything changing about you.
Check: is the engine, model version, location and language identical? Storing these conditions with each answer makes this a lookup, not a debate.
#4. Collection failed
If failed runs are counted as "not mentioned", an outage looks like a visibility crash. If they're dropped silently, the denominator shrinks and rates swing.
Check: report valid, failed and excluded runs separately.
#5. The competitive set moved
Share of voice is relative. If a competitor was mentioned more, yours can fall without a single answer about you changing.
Check: look at mention counts, not just shares.
#6. Then, maybe, a real cause
Once variance, panel, collection and denominators are ruled out, look at causes: a crawler-access change, a page edit, new coverage on a source that engines cite. Even then, be careful with language. Observational data can show that a change preceded a shift; it rarely proves the change caused it. Record what changed and when, keep conditions comparable, and describe the result as evidence rather than proof.
#A short checklist
- Each number rests on repeated runs
- The prompt panel is unchanged (or changes are isolated)
- Engine, model, location and language are unchanged
- Failures are reported separately
- Counts, not only shares, were compared
- The proposed cause has a date and a mechanism
The goal isn't to be a killjoy about wins. It's to make the wins you report the ones that survive a sceptical colleague.