I’ve been running a small monthly study: take ten buyer-intent questions from one industry, ask four AI assistants the identical question, and record every source each one actually read.
The analysis is the easy bit. Set overlap, count the domains, done.
Getting the citations out of four different APIs in a shape you can compare — that’s where I lost a weekend. Writing it up because I couldn’t find this anywhere when I needed it.
The problem
All four providers will tell you which URLs they used. None of them agree on where to put that, what to call it, or even what a “URL” is.
Here’s roughly what each one gives you. (Shapes as of mid-2026 — they move, check the docs before you copy this.)
OpenAI, Responses API with the web search tool. Citations ride along as annotations on the text output:
const urls = [];
for (const item of resp.output ?? []) {
for (const block of item.content ?? []) {
for (const a of block.annotations ?? []) {
if (a.type === "url_citation") urls.push(a.url);
}
}
}
Enter fullscreen mode Exit fullscreen mode
Anthropic, Messages API with the web search tool. Search results come back as their own content block type, separate from the text:
const urls = [];
for (const block of msg.content ?? []) {
if (block.type === "web_search_tool_result") {
for (const r of block.content ?? []) {
if (r.url) urls.push(r.url);
}
}
}
Enter fullscreen mode Exit fullscreen mode
Gemini, with Google Search grounding. It’s on the candidate, under grounding metadata:
const urls = (resp.candidates?.[0]?.groundingMetadata?.groundingChunks ?? [])
.map(c => c.web?.uri)
.filter(Boolean);
Enter fullscreen mode Exit fullscreen mode
Perplexity is the friendly one — a flat array on the response:
const urls = resp.citations ?? resp.search_results?.map(r => r.url) ?? [];
Enter fullscreen mode Exit fullscreen mode
Four different nesting depths, three different words for the same concept (“annotation”, “search result”, “grounding chunk”), and one of them isn’t giving you the URL at all. More on that in a second.
So: one adapter per provider, normalise to { engine, question, urls[] }, and never let provider-shaped data past that boundary. Obvious in hindsight. I did not do this first and regretted it.
Gotcha 1: Gemini’s grounding URIs aren’t the source
This is the one that would have quietly wrecked my numbers.
Gemini’s groundingChunks[].web.uri isn’t the publisher’s URL. It’s a redirect through Google’s own grounding endpoint. If you take it at face value, every single Gemini citation looks like it came from the same domain, and your “which sites does Gemini read” analysis returns exactly one site.
You have to resolve it:
async function resolveFinal(url) {
try {
const r = await fetch(url, { method: "HEAD", redirect: "follow" });
return r.url;
} catch {
return url; // keep the original, flag it, move on
}
}
Enter fullscreen mode Exit fullscreen mode
Two notes from doing this at volume. HEAD is enough and much cheaper than GET — you only want the final URL. And cache aggressively, because the same redirect will come up over and over and you don’t want to hammer anything.
I noticed this because Gemini’s numbers looked absurd on the first run. Worth building a sanity check that screams when one engine’s domain diversity collapses to near-zero — that’s almost always an extraction bug, not a finding.
Gotcha 2: the same page in five costumes
Once you have real URLs, they still won’t match each other. All of these are the same page:
https://www.example.com/clinic/
http://example.com/clinic
https://example.com/clinic?utm_source=chatgpt.com
https://example.com/clinic#reviews
https://example.com/Clinic
Enter fullscreen mode Exit fullscreen mode
If you’re comparing sets, five spellings of one page means five “different” sources, and your overlap number comes out lower than reality. Which, if your whole finding is “they overlap surprisingly little”, is the exact direction you don’t want an unforced error pointing.
The canonicaliser I settled on:
const TRACKING = /^(utm_|fbclid|gclid|msclkid|ref|source$)/i;
function canonical(raw) {
const u = new URL(raw);
u.protocol = "https:";
u.hostname = u.hostname.replace(/^www\./, "").toLowerCase();
u.hash = "";
for (const k of [...u.searchParams.keys()]) {
if (TRACKING.test(k)) u.searchParams.delete(k);
}
u.pathname = u.pathname.replace(/\/+$/, "") || "/";
return u.toString();
}
Enter fullscreen mode Exit fullscreen mode
Deliberately not lowercasing the path — paths are case-sensitive on plenty of servers and I’d rather over-count than merge two genuinely different pages.
One that made me laugh: several assistants append ?utm_source=chatgpt.com (or their own equivalent) to the URLs they hand back. The tools are tagging their own referral traffic. Strip it, or you’ll have engine-specific duplicates of the same page.
Then the analysis, which really is boring
Set overlap per question, pairwise across engines:
const jaccard = (a, b) => {
const A = new Set(a), B = new Set(b);
const inter = [...A].filter(x => B.has(x)).length;
const union = new Set([...A, ...B]).size;
return union ? inter / union : 0;
};
Enter fullscreen mode Exit fullscreen mode
Jaccard has a real flaw here: it treats a source read once exactly the same as one read by every engine on every question. I’ve gone back and forth on weighting it and haven’t convinced myself either way. If you’ve got a better metric for “did these two systems consult the same evidence”, I’d genuinely like to hear it.
What the numbers looked like
This month’s run was cosmetic and aesthetic clinics. Ten questions, nine cities, four engines.
- 214 different websites read across just ten questions
- 75% of those were read by one engine and no other
- Overlap between any two engines on the same question: 7%
- Exactly 1 question out of ten produced a website all four read
- About 13% of what they read was a directory; the rest was businesses’ own sites
That last one flipped between industries, which I didn’t expect — in the law firms run last month, directory domains dominated the top of the list.
The objection I had immediately, and you probably do too: LLMs are non-deterministic, so of course two calls differ — this could be noise.
So I re-ran a subset three times per engine and compared each engine against itself:
- Same engine, asked again: 51% overlap
- Different engines, same run: 7%
About 7.5× more consistent with itself than with its competitors. There’s real run-to-run variance, but it’s much smaller than the gap between engines. The divergence is structural, not sampling.
If you want to poke at that from the other direction, I wrote up whether a single assistant can even see a given site separately — same problem, one engine at a time.
What I can’t claim
Ten questions in one vertical is a sample, not a census. The variance control was two questions re-run three times, not the whole set. “Read” only means the URL appeared in that response’s citations — I can’t see weighting, or how much of the page actually mattered. And all of it is a July 2026 snapshot; these systems change under you.
The takeaway for anyone building on this
If you’re writing anything that compares retrieval across providers, budget most of your time for extraction and normalisation, not analysis. My rough split ended up 80/20 in favour of the boring part.
And build the sanity checks early. Both of my real bugs — the redirect one and the tracking-param one — were invisible in the output. The code ran fine. The numbers were just wrong, in a direction that happened to flatter my hypothesis. That’s the kind of bug worth being paranoid about.
The wider point, if you take one thing away: treating “AI visibility” as a single channel doesn’t survive contact with the data. Four engines built almost entirely separate reading lists from the same question, so it’s four problems, not one.
Full write-up of this month’s run, with the question-by-question breakdown, is here. Happy to share the raw JSONL if you want to check my counts — I’d rather be corrected than cited.
(Disclosure: I build a tool in this space. The study isn’t gated and there’s no signup on it.)
답글 남기기