Every AI visibility tool, ours included, answers the same question: what do AI assistants tell people about this brand? None of them can show you exactly what a particular person saw. It helps to know why, and what each way of collecting answers gets right and wrong.
#Two ways to collect an answer
1. Ask the provider's API. The tool sends the prompt to OpenAI, Anthropic, Google or Perplexity through their developer platforms, with web search switched on so the answer can cite live pages. This is how Citeroot collects answers.
2. Capture the consumer product. The tool drives the chat app in a browser, or buys results from a data provider that does, and records what the interface shows. For Google's AI Overviews and AI Mode, which have no public API, this means reading Google's results page.
#Where API answers differ from the app
- No personalisation or memory. An API call carries no chat history, saved preferences or account signals. The app often does.
- Different instructions and tools. Consumer apps add their own system instructions, formatting and features that the API does not.
- Model choice. The app may route a question to a different model than the one a tool calls, and that routing changes over time.
- Location is approximate. APIs accept a country or region hint; they are not a browser in that city.
#Where app captures differ from what people see
- Usually signed out. Captures typically run without a real person's account, so they miss personalisation too.
- Terms of service. Most assistants restrict automated access to their consumer apps. Anthropic's consumer terms prohibit accessing Claude "through automated or non-human means" except with an API key, and OpenAI's Terms of Use list "automatically or programmatically extract data or Output" among the things users may not do. Check how a tool collects before you depend on it.
- Brittle. Interfaces change often, and a capture that breaks quietly produces gaps that look like lost visibility.
#What neither method fixes: variation
Ask the same question twice and you often get a different answer. In a study by SparkToro and Gumshoe, the chance of an assistant returning the same list of brands twice was under 1 in 100 (Search Engine Land). Whatever the collection method, one answer is an anecdote. The useful number is how often you appear across many runs, with a range around it.
#How Citeroot handles it
- APIs with web search, recorded as collected. Each answer is stored with its date, engine, the exact model that produced it and the sources it cited. If a provider changes models, the new name appears on new answers, so a model change can't hide inside a trend.
- Repeated runs and ranges. Rates come with 95% intervals, and a change inside normal variation is labelled as such. How we handle variance.
- Every answer says where it came from. Each one is labelled API or Search page (Google AI Mode), with whether the engine actually searched the web and the country used. Sources are never added together into one number.
- App check measures the gap for your own questions. Paste what ChatGPT, Gemini, Perplexity, Claude or Google AI Mode showed you, and Citeroot compares it with its own answers to the same question: how often each named you, and whether the brands differ by more than normal run-to-run variation, with a 95% interval and the number of questions behind it. How App check works.
- Failures are not absences. If a provider errors, the answer is marked failed and left out of the rates instead of counting as "not mentioned".
- We say what we don't cover. Google AI Overviews, Microsoft Copilot, Meta AI and Grok are not tracked today.
#Check any tool's numbers in ten minutes
- Pick three prompts the tool reports on.
- Ask each one in the consumer app, signed out or in a private window, three times.
- Compare which brands appear and which sources are cited with what the tool recorded.
You won't get identical text. You should see the same brands turning up at roughly the rates the tool reports, and many of the same cited domains. If you don't, ask the vendor how they collect.
In Citeroot, App check does this comparison for you: paste each answer and it lines them up with Citeroot's own answers to the same question. It also suggests a few random questions each week, so you don't only check the ones where you expect a difference.