Free public ranking

SaaS Robots.txt Leaderboard

How the web's top SaaS companies & web apps compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

1%

of 211 ranked sites block GPTBot in robots.txt

211

sites graded, last checked Nov 11, 2026

99/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
204 sites
B
4 sites
C
2 sites
D
1 site
F
0 sites
# AI crawlers Region
51
Constant Contact
constantcontact.com
A 100 0/11 Allows all US
52
Conta Azul
contaazul.com
A 100 0/11 Allows all LatAm
53
Contentful
contentful.com
A 100 0/11 Allows all EU
54
Contentsquare
contentsquare.com
A 100 0/11 Allows all EU
55
CrowdStrike
crowdstrike.com
A 100 0/11 Allows all US
56
Culture Amp
cultureamp.com
A 100 0/11 Allows all Oceania
57
Cursor
cursor.com
A 100 0/11 Allows all US
58
Darktrace
darktrace.com
A 100 0/11 Allows all UK
59
Darwinbox
darwinbox.com
A 100 0/11 Allows all Asia
60
Databricks
databricks.com
A 100 0/11 Allows all US
61
Datadog
datadoghq.com
A 100 0/11 Allows all US
62
Deel
deel.com
A 100 0/11 Allows all US
63
DeepL
deepl.com
A 100 0/11 Allows all EU
64
Dext
dext.com
A 100 0/11 Allows all UK
65
DigitalOcean
digitalocean.com
A 100 0/11 Allows all US
66
DingTalk
dingtalk.com
A 100 0/11 Allows all Asia
67
dLocal
dlocal.com
A 100 0/11 Allows all LatAm
68
Docker
docker.com
A 100 0/11 Allows all US
69
Docusign
docusign.com
A 100 0/11 Allows all US
70
Dropbox
dropbox.com
A 100 0/11 Allows all US
71
Elementor
elementor.com
A 100 0/11 Allows all MENA-Africa
72
Expensify
expensify.com
A 100 0/11 Allows all US
73
Factorial
factorialhr.com
A 100 0/11 Allows all EU
74
Fastly
fastly.com
A 100 0/11 Allows all US
75
Fivetran
fivetran.com
A 100 0/11 Allows all US
76
Framer
framer.com
A 100 0/11 Allows all EU
77
FreeAgent
freeagent.com
A 100 0/11 Allows all UK
78
freee
freee.co.jp
A 100 0/11 Allows all Asia
79
Freshworks
freshworks.com
A 100 0/11 Allows all US
80
Front
front.com
A 100 0/11 Allows all US
81
GetResponse
getresponse.com
A 100 0/11 Allows all EU
82
GitLab
gitlab.com
A 100 0/11 Allows all US
83
Glean
glean.com
A 100 0/11 Allows all US
84
GoCardless
gocardless.com
A 100 0/11 Allows all UK
85
GoDaddy
godaddy.com
A 100 0/11 Allows all US
86
Gong
gong.io
A 100 0/11 Allows all US
87
Gorgias
gorgias.com
A 100 0/11 Allows all US
88
Grafana Labs
grafana.com
A 100 0/11 Allows all US
89
Grammarly
grammarly.com
A 100 0/11 Allows all US
90
Gusto
gusto.com
A 100 0/11 Allows all US
91
Harvey
harvey.ai
A 100 0/11 Allows all US
92
Help Scout
helpscout.com
A 100 0/11 Allows all US
93
Heroku
heroku.com
A 100 0/11 Allows all US
94
Hetzner
hetzner.com
A 100 0/11 Allows all EU
95
Hootsuite
hootsuite.com
A 100 0/11 Allows all CA
96
Hotjar
hotjar.com
A 100 0/11 Allows all EU
97
HubSpot
hubspot.com
A 100 0/11 Allows all US
98
Hugging Face
huggingface.co
A 100 0/11 Allows all US
99
Intercom
intercom.com
A 100 0/11 Allows all US
100
Intuit QuickBooks
intuit.com
A 100 0/11 Allows all US
Showing 51 to 100 of 211 results
Download full leaderboard (CSV)

All 211 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required