How LLM Tracking Tools Are Degrading Google and Bing Data

For nearly two decades, Google Search Console and Bing Webmaster Tools were the closest thing SEOs had to ground truth. Direct data from the top two search engines is helpful and should form the basis of your analytics and strategy, but we all know to take it with a pinch of salt. 

Google and Bing try to provide enough data to keep the tool valuable, but they avoid giving you too much so you can't rework their systems. 

This is where we leaned on third-party tracking tools; in SEO, the big names are Moz, Ahrefs, and SEMrush, but plenty more are available, each focusing on a niche or using a unique method for sourcing and modelling SERP and keyword data.

• Third-party rank trackers estimated.
• Analytics platforms sampled.

And they also contextualise a lot of what we see across SERPs, which is helpful.

But Search Console pulled first-party data straight from the search engine itself: these are the queries you appeared for, this is where you ranked, this is what people clicked.

That reliability is eroding fast.

Between AI Mode folding conversational prompts into the query report, a yearlong impressions logging bug, the combination of search and LLM queries for ranking average position, the removal of the &num=100 parameter, it's not been a great 12 - 24 months for our trusted tool, Search Console.

And now, with the demand to optimise for LLMs, a rapidly growing industry of AI visibility tools hammering search engines with synthetic prompts, the data most marketers still treat as gospel has quietly become a mixture of human demand, machine probing, and statistical noise. 

The Rise of LLM Tracking and FanOut Query Tools

As AI Overviews, AI Mode, ChatGPT, Perplexity, Copilot and Gemini started intercepting queries that used to end in a click, a new tooling category appeared almost overnight.

You only have to search for the term "AI Search Tracking Tool" to find a dozen of these new tools popping up. 

AI visibility trackers, prompt monitoring platforms, GEO dashboards, "share of model" scorecards. Ahrefs, Semrush, Similarweb, Profound, Peec, and dozens of smaller entrants now offer some version of the same promise:

Type in your brand and your category, and we'll tell you how often AI systems mention you.

Most of these tools work on the same underlying mechanic: query fanout simulation.

What Are Fan-out Queries And How Do They Work?

When a user asks an AI assistant a complex question, the system doesn't run one search. It decomposes the prompt into multiple related subqueries comparisons, definitions, alternatives, constraints and retrieves sources for each one before synthesising an answer.

That decomposition is the fanout.

Whoever gets cited inside those subqueries influences the final answer.

So tracking tools try to reverse-engineer it.

They generate a list of plausible prompts for your category, expand each one into the subqueries an LLM would likely produce, run them repeatedly against the AI surfaces, and record whether your brand appears.

Run that on a schedule daily, sometimes hourly across hundreds of prompts and dozens of clients, and you have an AI visibility dashboard.

It looks rigorous. It mostly isn't.

Why These Tools Are Fundamentally Limited

The core problem is that none of the LLM platforms gives us real query data. There is no AI equivalent of the Search Console query report. No prompt volume.

• No impressions by prompt.
• No competitive share by actual user intent.

That absence forces every tracking tool into the same compromise, and it creates four compounding limitations:

1. The prompts are invented, not observed. With Search Console, however degraded, you start from queries real people actually used. With prompt tracking, you start with a list somebody made up a marketer, an account manager, or an LLM asked to imagine what customers might type. You're measuring visibility against a hypothetical demand curve. If the prompt list is wrong, the dashboard is confidently wrong.

2. Fanout expansion is a guess about a black box. Google has never published how AI Mode decomposes a query. Neither has OpenAI or Anthropic. Tools infer the fanout pattern from observed behaviour and their own LLM-generated approximations. Those approximations resemble reality; they aren't reality.

3. LLM outputs are nondeterministic. Ask the same model the same question three times, and you can get three different sets of cited sources. Answers vary by personalisation, session context, geography, model version, and routing. A "visibility score" derived from a handful of runs against a stochastic system carries an error bar that almost no dashboard displays.

4. There's no volume weighting. Even if a tool correctly reports you appear in 40% of tracked prompts, it can't tell you whether those prompts represent 40% of actual demand or a rounding error. Without prompt volume data, share-of-voice metrics are unweighted averages of an arbitrary sample.

In summation, these tools measure a reality, not the reality. They're directionally useful for spotting whether you exist in a topic space at all. They are not a measurement system in the way Search Console was.

How Automated FanOut Checks Are Polluting Search Console

While these tools can be helpful, they come with a trade-off, and now that demand is high enough, we're seeing it in our default datasets from Search Console. 

If you've noticed, AI search queries are skyrocketing, not only from organic use, but there are a few things juicing their metrics. 

Google's AI Mode processes every message in a conversation as a search. Initial prompts, follow-up prompts, even one-word replies like "yes" or "tell me more" get logged as queries in the Search Console performance report for whichever sites were retrieved.

John Mueller has publicly confirmed that these prompt-like strings appearing in query reports are genuine AI Mode follow-ups.

Now, stack on top of that, the tracking industry that I mentioned earlier.

Every AI visibility platform running scheduled fanout checks is firing synthetic prompts into these same systems. Those prompts trigger retrieval. That retrieval generates impressions. Those impressions land in the Search Console of every site that got surfaced attributed to a query no human ever typed.

The result is that your query report increasingly contains three distinct classes of data blended into one table:

• Real human search queries the traditional keyword data Search Console was built for.
• Real conversational fragments, genuine AI Mode prompts, and follow-ups from actual users.
• Machine probe traffic tracker bots, agent harnesses, and automated fanout checks from AI visibility tools testing your topic space on a loop.

Nothing in the interface separates them. There's no "exclude automated probes" filter. And because these checks are scheduled and static the same prompt list, run at the same cadence, week after week they produce a persistent, artificial impression baseline that never converts and never behaves like human demand.

Practitioners who've dug into their own data have started documenting query classes that didn't exist eighteen months ago, then appeared suddenly and ran at a steady, machine-like cadence from that point forward. 

Bing Has the Same Problem With Better Transparency

Bing Webmaster Tools is in a similar position, with one meaningful difference: Microsoft actually surfaces grounding queries the subqueries Copilot generated to answer a user's prompt as a distinct reporting concept, rather than silently folding them into the general query table.

That transparency is genuinely better than Google's approach. But it doesn't solve the pollution problem. Grounding queries triggered by a tracking tool's automated probe look identical to grounding queries triggered by a real user, because mechanically they are identical. Bing labels the surface better; it still has no way to label the intent behind the request.

And because Copilot sits inside Bing's index, and a large share of AI visibility tools now monitor Copilot alongside ChatGPT and AI Mode, Bing's smaller absolute query volumes mean synthetic probe traffic can distort a site's reporting proportionally more than it does on Google.

The Wider Credibility Problem

This isn't happening in isolation. Search Console has taken several credibility hits in a short window:

• Google confirmed a logging error that overreported impressions for roughly a year from May 2025, and chose not to reconstruct the historical data.
• The removal of the &num=100 parameter in September 2025 revealed how much of the industry's "rising impressions" narrative had been scraper-driven all along.
• AI Mode activity was blended into Web results totals with no native filter, meaning year-over-year comparisons now span surfaces that didn't previously exist.
• The Generative AI features report launched in June 2026 reports impressions but not clicks, positions, or query visibility without diagnosis.

Layer automated fanout probes on top of that and you get the pattern SEOs have been calling the "crocodile mouth".

Impressions climbing steadily while clicks flatline or fall.

For a long time the industry read that as AI Overviews satisfying intent before the click.

Some of it was.

A meaningful portion was bots, bugs, and now tracking tools measuring each other's shadows.

Common Complaints From the Industry

• "My impressions doubled, and my clicks didn't move." The single most common observation, and the one most often misdiagnosed as an AI Overview problem when it's partly instrumentation.
• "My query report is full of prompts nobody would type." Long, oddly formal, comparison-shaped strings that read as if an LLM wrote them because one did.
• "Position data has become meaningless." Average position inside an AI answer block isn't the same construct as a blue-link ranking, but Search Console reports them in the same column.
• "Queries behind AI clicks are anonymised." A large share of AI Mode activity shows up with no query attached at all, so the visible fragments are a biased sample of a bigger, hidden set.
• "We're paying tools to generate the noise we then pay to filter." The circularity complaint: agencies running visibility trackers on clients whose Search Console data is being degraded by the same category of tool.
• "There's no way to exclude bot probes." No filter, no user agent breakdown, no official guidance from either engine on identifying automated AI traffic.

The Current State of Search Tracking

The industry isn't going to change overnight; this is the new landscape we have to work with, and more noise means more misoptimisation. 

The short answer is we're cooked!

But the good news for SEO's is we're all cooked together. 

Since it takes a keen eye and skill to clean up this noise, it gives you a distinct advantage over everyone who isn't aware of it or ignores it. 

For those who think they can run their own SEO with Claude or GPT's latest models, results will only decay over time as the models accept slop in and push slop out. 

On top of that, the SynthID Scarlett letter rolling out in search will only widen the gap between programmatic SEO and artisanal human lead SEO.

• Invasion of Synthetic/Bot Traffic: Automated LLM tracking tools, verification bots, and query "fan-out" expansions execute thousands of conversational queries to check AI visibility. These machine-generated runs inflate GSC metrics with unnaturally long, full-sentence queries that human searchers rarely type.
• High Impressions with Zero Clicks: Webmasters are seeing a massive surge in long-tail query impressions that yield zero click-throughs and zero real human page views, skewing click-through rates (CTR) and performance analytics.
• Misallocated Resources: SEOs risk wasting time and money optimising content for long, complex queries that appear popular in GSC but are actually generated entirely by AI crawlers rather than genuine user interest.
• Lack of Transparency & Filtering: Google offers no built-in way to distinguish between organic human searches, cited AI answers, and automated bot scraping, forcing marketers to rely on third-party workarounds or pay for ad verification to isolate real traffic.

What to Actually Do About It

Practical adjustments while the measurement layer sorts itself out:

1. Demote impressions to a directional signal. Clicks were unaffected by the logging bug and are far harder for probe traffic to fabricate. Report on clicks, conversions, and revenue per landing page.
2. Segment prompt-shaped queries deliberately. Use regex filters in Search Console to isolate long conversational strings and single-word follow-ups, then analyse them as a separate class rather than letting them sit inside your keyword data.
3. Treat tracker-generated prompts as a signal about tooling, not demand. If a query class appears suddenly and runs at a perfectly regular cadence, it's a machine. Exclude it from editorial prioritisation.
4. Validate prompt lists against real fragments. The conversational fragments genuinely leaking into your query report are a small but real sample of how people phrase things at your content. Use them to sanity-check the invented prompt list your tracking tool is running.
5. Measure entity strength, not dashboard position. Getting recommended by an AI system is a function of what the model knows about your brand citations, consistent entity data, and third-party corroboration, not of climbing a prompt-tracking leaderboard.
6. Build your own baseline. Search Console API into BigQuery gives you far more rows than the interface, plus the ability to classify and exclude probe traffic yourself.

Learn To Filter Out The Noise

The AI visibility tooling industry emerged to solve a real problem that could be solved if LLMs provided direct tooling; there would be no need for this if ChatGPT, Claude and the like had a similar tool to Search Console.

It's crazy that they haven't; it's an obvious revenue generator. When SEOs and site owners are already paying for 3rd-party tools, LLMs could cut out that middleman and give you data directly from their models for a fraction of the cost.  

But since LLM platforms give us no query data, the market has come up with a solution to approximate it. But the approximation method, mass automated fanout probing, is now actively degrading the one dataset that was grounded in real user behaviour.

Search Console is becoming a blended log of humans, AI conversations, bots and the AI measurement industry's own machinery, with no way to tell them apart.

Until the engines ship proper segmentation, or the LLM platforms open up genuine prompt data, the only defensible approach is to anchor reporting on the metrics that are hardest to fake.

Clicks, conversions, and revenue.

Since these metrics remain heavily human-generated for now, until agentic workflows, skills.md and MCPs start to erode those user actions and outsource them to bots, but that's a story for another time.

Need Help With Your AI strategy?

Are you looking to promote your business?

Business owners can create their free business listings on nichemarket. The more information you provide about your business, the easier it will be for your customers to find you online. Registering with nichemarket is easy; just head to our sign-up form and follow the instructions.

If you need a more detailed guide on creating your profile or listing, we recommend checking out the following articles. 

• How to create a nichemarketer profile 
• How to create a nichemarket business listing 

Recommended reading

If you enjoyed this post and have a little extra time to dive deeper down the rabbit hole, why not check out the following posts on SEO.

• Beware Of The DMCA link Building Scam
• A Comprehensive Guide To Link Building Scams
• How To Build Links From Journalists
• How To Use The Forgotten Art of Dead Link Building To Boost SEO
• How To Build Local Links To Your Website

Tags: LLMs, AI Search, GEO, SEO