Robots.txt Tester
Read a site's robots.txt, see the rules that apply to each crawler, and test whether a specific URL would be allowed or blocked — the file where an accidental line can remove a whole site from search.
- Limits No file upload · Up to 1,000 runs a day
- Where it runs Runs on our server; no file is uploaded.
- Cost Free, with the limits above. No account needed.
Where to find it
Every address below opens the same tool. The first address is the one to bookmark and share.
- robots.openvibe.toolsPrimary
People also call this
What it's for: Fetch and analyse any site's robots.txt, check which URLs are blocked, and catch rules that hide your site from search engines.
- robots.txt tester
- robots txt checker
- crawl rules
- disallow checker
- seo crawl test
Use it from code
POST https://openvibe.tools/api/v1/tools/robots/run- No account needed: anonymous calls run on the lowest quota tier; a token with
tools.tool.runraises it. - It answers with the result.
curl -X POST https://openvibe.tools/api/v1/tools/robots/run \
-H 'Content-Type: application/json' \
-d '{"input":{"target":"example.com"}}'Inputs: JSON Schema · OpenAPI · from JavaScript: tools.run("robots", …) in openvibe-sdk · How the API works
Similar tools
All Network Tools
Domain & IP LookupLook up everything about a domain or IP in one go: DNSlookup.openvibe.tools
What Is My IPSee your public IP address, ISP, approximate location andmyip.openvibe.tools
IP Address LookupLook up any IP address to see its owner, network,ip.openvibe.tools
IPv4 Lookup & CIDR CalculatorLook up IPv4 addresses and calculate CIDR subnets: networkipv4.openvibe.tools
IPv6 LookupLook up IPv6 addresses, expand or compress notation, andipv6.openvibe.tools
IP GeolocationFind the country, region, city, timezone and coordinatesgeoip.openvibe.tools
One account works on every site. Open source and community-run.