The robots.txt mistake that hides your whole site from Google

One line can hide your whole site from Google

Disallow: / in your robots.txt tells every crawler to ignore your entire
site. It’s shockingly common — usually left over from a staging config that got
copied to production. And nothing tells you: your pages just quietly stop
ranking.

I kept getting bitten by robots.txt and sitemap.xml issues, so I added free
checks for them to OGCheck.

Check your robots.txt in one request

curl "https://ogcheck-app.azurewebsites.net/robots?url=https://your-site.com"

Enter fullscreen mode Exit fullscreen mode

It confirms the file is reachable, parseable, flags a site-wide block,
and tells you whether it points at your sitemap.

The robots.txt mistakes that quietly cost you traffic

1. Disallow: / — blocks everything. The classic staging-to-prod accident.
2. Missing file — crawlers assume everything is allowed, which may expose pages you didn’t mean to (and you lose the chance to point at your sitemap).
3. No Sitemap: line — search engines crawl your site less efficiently.
4. Blocking your CSS/JS — Google renders pages; blocking assets can hurt how it sees them.

While you’re at it, check your sitemap too

A broken or empty sitemap.xml hurts crawl efficiency the same quiet way:

curl "https://ogcheck-app.azurewebsites.net/sitemap?url=https://your-site.com"

Enter fullscreen mode Exit fullscreen mode

Both checks are free, zero-dependency (pure Python stdlib), and open source:
https://github.com/rahulatrkm/ogcheck. There’s also a social-preview check
(/check?url=…) that confirms your og:image actually loads.

Feedback welcome — what else silently breaks your site’s visibility that a
one-request check could catch?