Free tool
Is your store blocking the AI crawlers?
Buyers now ask an assistant which product to get. The assistant can only name a site it was allowed to read. One line in your robots.txt decides that, and most owners have never looked at theirs.
Enter your address. We read the same public file the crawlers read and show you, one crawler at a time, exactly what it says.
What we check
16 crawlers, 10 operators.
Each one has a different job. Some build the index an assistant answers from, some fetch your page while a person is waiting, and some only collect text for training. Blocking them is not one decision.
Questions
What people ask first.
- Is my site blocking AI crawlers?
- Enter your address above and you'll have the answer in a few seconds. We read your robots.txt — the same public file the crawlers read — and report, for each AI crawler, whether it is allowed to fetch your site root, plus the exact line that decides it.
- What is robots.txt and why does it decide this?
- It's a plain text file at the root of your domain that tells crawlers which parts of the site they may fetch. The AI crawlers honor it the same way search engines do. If it says a crawler can't read you, that crawler doesn't read you — and an assistant can't cite a page it was never allowed to open.
- Does blocking GPTBot keep me out of ChatGPT?
- Not on its own, and this is where most people get it wrong. OpenAI runs several crawlers with different jobs: GPTBot gathers training data, OAI-SearchBot builds the search index ChatGPT answers from, and ChatGPT-User fetches a page while someone is waiting on an answer. Blocking GPTBot keeps you out of training. Blocking the other two keeps you out of answers. They are separate rules and most robots.txt files treat them as one.
- Should I block AI crawlers?
- It depends on which one and what you sell. If you're a store that wants to be recommended, blocking answer-time crawlers works directly against you. If you're a publisher whose product is the text itself, blocking training crawlers while allowing answer-time ones is a coherent position. The mistake is doing either by accident.
- My robots.txt looks fine but the check says my edge blocked you. What does that mean?
- Your CDN or firewall turned our request away before it reached the file. That's worth knowing: robots.txt is meant to be readable by anything, and a rule aggressive enough to stop us will usually stop a crawler arriving the same way. The block is upstream of your robots.txt, so fixing the file won't help until the edge rule is fixed.
- I don't have a robots.txt at all. Is that bad?
- No. With no file, nothing is disallowed — every crawler is free to read you. A missing robots.txt is the most permissive state there is, not a gap.
After access
Being readable is the floor, not the finish.
A crawler that can reach you still has to decide you’re worth quoting. The free AI-visibility check asks the next question: when a buyer asks an assistant about your category, does it name you — and if not, who does it name instead?