SaaS Robots.txt Leaderboard
How the web's top SaaS companies & web apps compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
1%
of 211 ranked sites block GPTBot in robots.txt
211
sites graded, last checked Nov 11, 2026
99/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 204 sites
- B
- 4 sites
- C
- 2 sites
- D
- 1 site
- F
- 0 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 51 |
Constant Contact
constantcontact.com
|
A | 100 | 0/11 | Allows all | US |
| 52 |
Conta Azul
contaazul.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 53 |
Contentful
contentful.com
|
A | 100 | 0/11 | Allows all | EU |
| 54 |
Contentsquare
contentsquare.com
|
A | 100 | 0/11 | Allows all | EU |
| 55 |
CrowdStrike
crowdstrike.com
|
A | 100 | 0/11 | Allows all | US |
| 56 |
Culture Amp
cultureamp.com
|
A | 100 | 0/11 | Allows all | Oceania |
| 57 |
Cursor
cursor.com
|
A | 100 | 0/11 | Allows all | US |
| 58 |
Darktrace
darktrace.com
|
A | 100 | 0/11 | Allows all | UK |
| 59 |
Darwinbox
darwinbox.com
|
A | 100 | 0/11 | Allows all | Asia |
| 60 |
Databricks
databricks.com
|
A | 100 | 0/11 | Allows all | US |
| 61 |
Datadog
datadoghq.com
|
A | 100 | 0/11 | Allows all | US |
| 62 |
Deel
deel.com
|
A | 100 | 0/11 | Allows all | US |
| 63 |
DeepL
deepl.com
|
A | 100 | 0/11 | Allows all | EU |
| 64 |
Dext
dext.com
|
A | 100 | 0/11 | Allows all | UK |
| 65 |
DigitalOcean
digitalocean.com
|
A | 100 | 0/11 | Allows all | US |
| 66 |
DingTalk
dingtalk.com
|
A | 100 | 0/11 | Allows all | Asia |
| 67 |
dLocal
dlocal.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 68 |
Docker
docker.com
|
A | 100 | 0/11 | Allows all | US |
| 69 |
Docusign
docusign.com
|
A | 100 | 0/11 | Allows all | US |
| 70 |
Dropbox
dropbox.com
|
A | 100 | 0/11 | Allows all | US |
| 71 |
Elementor
elementor.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 72 |
Expensify
expensify.com
|
A | 100 | 0/11 | Allows all | US |
| 73 |
Factorial
factorialhr.com
|
A | 100 | 0/11 | Allows all | EU |
| 74 |
Fastly
fastly.com
|
A | 100 | 0/11 | Allows all | US |
| 75 |
Fivetran
fivetran.com
|
A | 100 | 0/11 | Allows all | US |
| 76 |
Framer
framer.com
|
A | 100 | 0/11 | Allows all | EU |
| 77 |
FreeAgent
freeagent.com
|
A | 100 | 0/11 | Allows all | UK |
| 78 |
freee
freee.co.jp
|
A | 100 | 0/11 | Allows all | Asia |
| 79 |
Freshworks
freshworks.com
|
A | 100 | 0/11 | Allows all | US |
| 80 |
Front
front.com
|
A | 100 | 0/11 | Allows all | US |
| 81 |
GetResponse
getresponse.com
|
A | 100 | 0/11 | Allows all | EU |
| 82 |
GitLab
gitlab.com
|
A | 100 | 0/11 | Allows all | US |
| 83 |
Glean
glean.com
|
A | 100 | 0/11 | Allows all | US |
| 84 |
GoCardless
gocardless.com
|
A | 100 | 0/11 | Allows all | UK |
| 85 |
GoDaddy
godaddy.com
|
A | 100 | 0/11 | Allows all | US |
| 86 |
Gong
gong.io
|
A | 100 | 0/11 | Allows all | US |
| 87 |
Gorgias
gorgias.com
|
A | 100 | 0/11 | Allows all | US |
| 88 |
Grafana Labs
grafana.com
|
A | 100 | 0/11 | Allows all | US |
| 89 |
Grammarly
grammarly.com
|
A | 100 | 0/11 | Allows all | US |
| 90 |
Gusto
gusto.com
|
A | 100 | 0/11 | Allows all | US |
| 91 |
Harvey
harvey.ai
|
A | 100 | 0/11 | Allows all | US |
| 92 |
Help Scout
helpscout.com
|
A | 100 | 0/11 | Allows all | US |
| 93 |
Heroku
heroku.com
|
A | 100 | 0/11 | Allows all | US |
| 94 |
Hetzner
hetzner.com
|
A | 100 | 0/11 | Allows all | EU |
| 95 |
Hootsuite
hootsuite.com
|
A | 100 | 0/11 | Allows all | CA |
| 96 |
Hotjar
hotjar.com
|
A | 100 | 0/11 | Allows all | EU |
| 97 |
HubSpot
hubspot.com
|
A | 100 | 0/11 | Allows all | US |
| 98 |
Hugging Face
huggingface.co
|
A | 100 | 0/11 | Allows all | US |
| 99 |
Intercom
intercom.com
|
A | 100 | 0/11 | Allows all | US |
| 100 |
Intuit QuickBooks
intuit.com
|
A | 100 | 0/11 | Allows all | US |
All 211 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!