Merchants are told two opposite things about AI crawlers: block them, because
they take your content and give nothing back; and admit them, because that is
how a store gets into an AI shopping answer. Both assume a decision is being
made. We wanted to know how many stores have made one, and which way.
Method
Every store in a corpus of 10,099 known Shopify storefronts has its
/robots.txt read as part of a scan. The file is parsed the way the major
crawlers document parsing it: most specific User-agent group wins, * is
the fallback, longest matching path rule wins, Allow beats Disallow on a
tie. Each of twelve crawler names is asked one question: may it fetch
/products/? A store counts as blocking a crawler when the answer is no.
A second reading looks at the product page for a meta name="robots" tag
carrying noai or noimageai.
Readings were taken between 29 August and 2 September 2026. Every store that
blocked at least one crawler, or carried the tag, is one row in the CSV at
the end. The rest of the corpus blocked nothing and is the denominator.
What this cannot see. A robots.txt is a request. A store can also block a
crawler at the edge, with a bot-management rule or a firewall, and that block
is invisible here because the scanner is not the crawler being blocked. Every
count below is a floor.
Result
133 of 10,099 stores block at least one crawler: 1.32%.
Crawler What it feeds Fetches at answer time Stores blocking CCBot Common Crawl no 81 GPTBot OpenAI training & retrieval no 77 Bytespider TikTok / Doubao no 72 Amazonbot Alexa+ / Rufus no 60 Google-Extended AI Overviews & AI Mode grounding no 58 ClaudeBot Claude retrieval & citations no 53 Applebot-Extended Apple Intelligence no 48 meta-externalagent Meta AI no 47 ChatGPT-User Live fetches during a ChatGPT chat yes 11 PerplexityBot Perplexity search & shopping yes 10 OAI-SearchBot ChatGPT search & shopping results yes 6 Perplexity-User Live fetches when a Perplexity user asks yes 3“Fetches at answer time” marks the four crawlers that read a page, or index
for a search result, at the moment a person is asking. The other eight crawl
ahead of time, to train or ground a model. The split follows each operator’s
published description of the name. The twelve are the names merchants’
block lists actually carry, not every answer-time agent that exists;
Anthropic’s and Google’s live-fetch agents are not among them.
How they block
- 130 of 133 block with
Disallow: /. The whole site, not the product pages. 3 block/products/alone. - 54 block exactly one crawler.
- 28 block the same eight, verbatim: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, meta-externalagent. 4 more block those eight and others. Identical lists do not arise independently; this one reads like a copied snippet, and it contains none of the four answer-time crawlers.
- 1 store blocks all twelve.
Training or answering
Of the 133 stores that block anything, 120 block only training and grounding
crawlers. 13 block at least one crawler that fetches at answer time; 1 of
those blocks only answer-time crawlers, 12 block both kinds.
The clearest pair is OpenAI’s. GPTBot trains; OAI-SearchBot is what
ChatGPT’s search and shopping results are built from. 77 stores block the
first. 6 block the second, and every one of those 6 also blocks the first.
The reverse, keep the shopping crawler out and let the training crawler in,
happens on 0 stores. No store in the corpus has decided to stay out of AI shopping answers.
The stores that are out of them by robots.txt are there because a copied
training opt-out happened to include the name.
The noai tag
8 stores carry a noai or noimageai meta robots tag on their product page.
6 of the 8 block nothing at all in robots.txt. By their names, six of the eight are
musicians’ merchandise stores, which again looks like one template rather
than eight decisions. The tag is a separate mechanism and this study makes no
claim about which crawlers honour it.
What it does and does not mean
Blocking AI crawlers is rare on Shopify. 98.7% of stores block none of
the twelve by robots.txt. Where it happens it is mostly a whole-site rule
aimed at training crawlers, and mostly one list.
The shopping crawlers are almost never blocked. 13 stores in 10,099 block
any of the four answer-time crawlers we checked. For the rest, whatever stands between
the store and an AI shopping answer, it is not robots.txt.
It is not a recommendation. A merchant who does not want their catalogue
in a training set has a reason, and a copied list that does the job is a
fine way to act on it. The finding is narrower: the list most of them copied
was written before the answer-time crawlers existed, so a store using it has
made a decision about training and, without noticing, none about shopping.
It is a floor. Edge blocks are real and this cannot count them.
Reproduce it
curl -s https://EXAMPLE.com/robots.txt
Enter fullscreen mode Exit fullscreen mode
Find the User-agent: group that names the crawler; if none does, the *
group applies. Under that group, the longest rule matching /products/
decides, and Disallow: / matches everything.
All 139 rows, one per store, are in
ai-crawler-blocking.csv:
host, grade, date read, one column per crawler, whether the noai tag was
present, and whether the rules were whole-site or product pages only. Every
row names a domain whose robots.txt you can read yourself.
Originally published at
shelfglance.com/research/ai-crawler-blocking,
where the corpus of 10,099 scanned storefronts lives.