robots.txt Tester — Check if a URL Is Crawlable
Paste your robots.txt and any URL to see whether Googlebot, Bingbot, GPTBot, or any user-agent is allowed to crawl it. Implements the Google longest-match rule.
Runs 100% in your browser — nothing is uploaded.
Disallow: /admin/ · Group: User-agent: *Frequently asked
The Google-style spec: longest matching path wins, `*` matches any sequence, `$` anchors to end-of-URL. When Allow and Disallow tie on length, Allow wins.
No — robots.txt is a voluntary convention. Google, Bing, and most major crawlers respect it. Malicious scrapers ignore it entirely. Use authentication or rate-limiting for anything you actually need to protect.
The tester picks the most specific group that matches your user-agent (case-insensitive exact match), then falls back to the `User-agent: *` group. If neither exists, the URL is treated as allowed.
Related tools
- URL ParserBreak any URL into its protocol, host, port, path, query parameters, and hash. U
- Meta Tag GeneratorGenerate SEO, Open Graph, and Twitter Card meta tags from your title, descriptio
- UTM URL BuilderBuild UTM-tagged campaign URLs for Google Analytics, Adobe Analytics, and other
- QR Code GeneratorGenerate QR codes for URLs, text, WiFi credentials, and more. Customize colors a
- User-Agent ParserParse any User-Agent string to identify the browser, engine, OS, and device. Aut