I was about to start a monthly column about AI search visibility — how to get your pages cited by ChatGPT, Perplexity, Google’s AI Mode. Medium was the obvious venue. Big audience, publications that distribute for you, no infrastructure to run.
Before writing the first paragraph I ran the check I run on every client site:
curl -s https://medium.com/robots.txt
Enter fullscreen mode Exit fullscreen mode
Medium disallows the crawlers I was writing about.
What’s actually in the file
Checked 2026-09-17, HTTP 200. Alongside the usual * rules there’s a second group:
User-Agent: Amazonbot
User-Agent: Applebot-Extended
User-Agent: Bytespider
User-Agent: ClaudeBot
User-Agent: FacebookBot
User-Agent: GoogleOther
User-Agent: GPTBot
User-Agent: meta-externalagent
Disallow: /
Allow: /about
Allow: /business
Allow: /earn
Allow: /gift
Allow: /membership
Allow: /partner-program
Allow: /verified-authors
Enter fullscreen mode Exit fullscreen mode
Eight user-agents, Disallow: /, and an allow-list containing exactly the pages that sell Medium. Your post is not on that list.
That’s a defensible business decision — Medium licenses its archive and doesn’t want it taken for free. It’s just the opposite of what I needed from the platform.
Blocked isn’t the same as invisible
This is where most takes on this get sloppy, so it’s worth being precise. “AI crawler” covers two jobs, and Medium only closes one of them.
Training and bulk collection — closed. GPTBot (OpenAI), ClaudeBot (Anthropic), Applebot-Extended (Apple Intelligence), meta-externalagent (Meta), Bytespider (ByteDance), Amazonbot.
Search-grounded retrieval — open. These are not in the blocked group, so the * rules apply and they’re allowed:
Googlebot
Google’s index → AI Overviews, AI Mode
Bingbot
Bing’s index → Copilot
PerplexityBot
Perplexity’s own index
OAI-SearchBot
ChatGPT’s search index
ChatGPT-User
user-triggered fetch when ChatGPT opens a link
Google-Extended
Gemini / Vertex training use of Google-crawled pages
CCBot
Common Crawl
So a Medium post can absolutely turn up as a cited source in Perplexity or an AI Overview. It reaches those engines through a search index, and those indexes are open.
Two of those rows deserve a second look.
Google-Extended isn’t blocked. It doesn’t crawl anything itself — it’s the token that controls whether content Google already crawled may be used for Gemini training. Left unblocked, that path stays open while OpenAI’s and Anthropic’s equivalents are shut.
CCBot isn’t blocked either. Common Crawl is a public archive that a lot of training corpora are built from. Blocking the labs’ own crawlers while leaving the archive they can download open is a narrower measure than it first looks.
I don’t read this as a loophole anyone left on purpose. I read it as the normal state of a robots.txt: a list of names, maintained by hand, always slightly behind a user-agent landscape that adds a new one every few months.
Six platforms, same check
Same command, same day:
platform AI crawler rules in the* group
medium.com
separate group blocking 8 AI user-agents
dev.to
none — only /*/actions_panel* disallowed
hashnode.com
none — Allow: /
substack.com
none — only /action/ disallowed
news.ycombinator.com
none — open, Crawl-delay: 30
reddit.com
User-agent: * + Disallow: / — everything blocked
Reddit is the interesting one. It doesn’t single out AI crawlers; it disallows all of them, then licenses the archive to Google and OpenAI directly. Reddit threads do reach those two models, but through a contract, not through a crawl. If you were counting a Reddit comment as third-party evidence any model could go verify, that’s not what it is.
Why I cared enough to check
A model reading your own domain treats what it finds as a self-description. The same sentence on a domain you don’t control reads differently — and that difference is the entire reason to publish somewhere other than your own site.
I have a concrete case. A comparison prompt to Gemini described my tool as a “Local SEO & AI Indexing” product “built on top of Scrapeless.” There is no such product underneath it. The acronym did the damage: a lot of indexed repositories use GEO and “geographic” interchangeably, so a model reading a bare GEO with no nearby context can resolve it to the geographic sense and attach whatever vendor sits in that space.
The fix was boring. Put the expansion next to the name where a model reads first — hero paragraph, meta description, the WebSite and WebApplication nodes in the JSON-LD graph. On the built HTML, the entity-disambiguation check went from 1/3 to 3/3 and the citability score from 60 to 62.
But that fix only works on pages I own. The correction that matters more is the one on a domain that isn’t mine — which is why the platform’s robots.txt is a prerequisite, not a detail.
The rule
Before you commit a column to a platform, read the platform’s robots.txt. It’s one command, and it tells you which of the two channels you’re buying.
Then be honest about which you needed:
- Human readers and distribution → Medium is fine. It has the audience and the publications.
- Corroboration a model can crawl → pick a platform that doesn’t disallow the crawlers.
I wanted the second, so this column is on dev.to.
Three caveats worth stating plainly. robots.txt is a request, not enforcement — it tells you a platform’s stated policy, not what every crawler does. The lists change; mine is a reading from one day, and yours should be your own. And a publication on a custom subdomain can serve a different file than its parent, so check the host you’ll actually publish under.
If you want the same check across the AI user-agents on your own domain, that’s the first thing the CLI I maintain does — geo access against your robots.txt. But you don’t need it for this. You need curl and two minutes, on the platform you were about to hand your writing to.