Why you should track AI visibility over time, not once

Plenty of business owners have done it: open ChatGPT, ask who the best installer in the area is, and check whether their name shows up. It's understandable, but it tells you very little. AI models don't give the same answer to the same question every time. In this article, we look at what research says about that variation and what it means for tracking AI visibility in a way that's actually useful.

Same prompt, different list every time

In early 2026, Rand Fishkin (SparkToro) and Patrick O'Donnell (Gumshoe) published a study on how consistent AI recommendations are. Hundreds of volunteers ran 12 prompts asking for recommendations, covering chef's knives, headphones, cancer care hospitals, digital marketing consultants and more, through ChatGPT, Claude and Google (AI Overviews or AI Mode). Each prompt ran 60 to 100 times per platform, 2,961 times in total¹.

The result: the chance that ChatGPT or Google's AI returned the same list on a repeat was less than 1 in 100. The same list in the same order came up less than 0.1% of the time¹. Nearly every answer differed in three ways: which names were on it, in what order and how many. Claude produced the same list slightly more often, but rarely in the same order¹.

At the same time, the researchers saw a stable pattern. When 142 participants each wrote their own prompt about headphones, the prompts barely resembled each other. Yet the same four big players showed up in 55 to 77% of the 994 responses¹. The researchers' takeaway: how often you're named, measured across many prompts, is far more consistent than your position on the list¹.

Caveats: the study wasn't peer reviewed, participants used their own settings, and Gumshoe sells tracking software. The researchers disclose this themselves.

Why AI models vary

Variation is built in. A language model picks each next word based on probabilities, and in normal use there's deliberate randomness in that. But even without it, answers aren't fixed. Thinking Machines Lab sent the same prompt 1,000 times to a large open model at temperature 0, the setting that in theory always picks the most likely word. It got 80 different completions². According to the researchers, the main cause is how servers process requests in varying batches².

On top of that, AI assistants often search the web first. Which pages they find depends on the searches they write themselves and the index they use. Ahrefs found Perplexity cites from Google's top 10 28.6% of the time, versus around 8% for ChatGPT, Gemini and Copilot⁵. Even two Google products, AI Mode and AI Overviews, cited the same URLs only 13.7% of the time in an Ahrefs analysis, though they usually gave substantively the same answer³.

Sources change too

Beyond run-to-run variation, there's drift over weeks and months. Semrush tracked more than 230,000 prompts for 13 weeks. ChatGPT cited Reddit in close to 60% of responses in early August 2025; by mid-September, it was around 10%⁴. A shift like that directly affects businesses that were mainly mentioned through that source. If you'd only checked in July, you'd have had the wrong picture by October.

The GEO study also shows that optimization effects differ by domain⁶. A change that works in one industry may do little in another. The only way to know is to measure before and after.

Why a one-off check misleads

It follows that a single manual check can go wrong in two ways. You don't see yourself and conclude you're invisible, while you're actually named in half the answers. Or you do see yourself and assume you're fine, while that was a lucky run. On top of that, your own ChatGPT account can be shaped by your past conversations, and customers phrase their questions differently than you do. Your location and language settings can shift the answer as well.

That makes ranking position a weak metric for AI answers. What works is counting across many runs: in what percentage of answers you're named (win rate), and what share of all mentions in your category is yours (share of voice). Those numbers get more reliable the more runs and weeks you have.

How many measurements do you need?

A simple example shows why repetition matters. This is statistics, not a finding from the studies above. Say an AI model actually names your business in 40% of answers to a prompt. With a single check, you'll see yourself 4 times out of 10 and miss yourself 6 times out of 10: the result is basically a coin flip. With 10 runs, the measured rate usually lands somewhere between about 25% and 55%. With 30 runs, that margin is already almost half as wide. The more runs, prompts and weeks you combine, the smaller the noise and the sooner you can tell a real change from chance.

It works the other way too. If your win rate jumps from 30% to 45% week over week based on a handful of answers, that could be luck. Only when a rise holds across several measurements, and doesn't also show up for your competitors, is it reasonable to conclude something changed.

Measuring the effect of an action

Most businesses don't just want to know how visible they are. They want to know what helps. That takes the same discipline as any experiment: a baseline, a clear change and a follow-up measurement with the same prompts. Seer Interactive showed that businesses cited in AI Overviews get 35% more organic clicks than those that aren't, but also stresses that cause and effect are hard to separate⁷. That's true of any after-the-fact analysis. By measuring before and after an action, and using competitors' trends as a comparison, you get closer to an honest answer on whether your change made the difference.

Allow for lag. A new page has to be found and indexed first, and a mention on another website has to be picked up by the search systems AI models use. A window of several weeks is more realistic than a few days.

What this means for your business

• Look at percentages, not positions. "Named in 4 out of 10 answers" tells you more than "#2 on one try".
• Track each AI model. ChatGPT, Perplexity, Gemini and Claude use different sources. An average hides where you're strong or weak.
• Follow trends over weeks. Sources and models change. Only a series of measurements shows whether your visibility is rising, falling or stable.
• Measure the effect of what you do. A new page, a mention on a comparison site or better reviews take time to get picked up by AI models. Measure before and a few weeks after an action, with the same prompts.
• Compare with competitors. If everyone in your industry gets named less in a given week, that's the AI model, not you. Share of voice corrects for that.
• Use your customers' questions. The SparkToro study showed people phrase the same need in very different ways. Track a set of prompts that together cover how customers talk about your service, not one favorite keyword.

In short: treat AI visibility as a probability, not a position. You measure how big that probability is, how it develops and what increases it.

How GeoReady measures this

GeoReady is built around this variation. Your prompts run 3× per week on ChatGPT, Perplexity, Gemini and Claude through their official APIs, with repeated runs per prompt, so one random answer doesn't decide the outcome. From that, GeoReady calculates your win rate and share of voice per AI model and over time, sentiment with the sentence it's based on, and which URLs get cited. Every action you take gets a baseline; after 30 days you see the measured effect next to your competitors' trends. That makes your GeoReady score a trend over weeks, not a snapshot.