04 / Forge / SEO / Server-side
robots.txt validator
Fetch and parse a site's robots.txt: groups, rules, sitemaps, and mistakes. Fetches the target through the engine's validated fetcher: public http and https addresses only, size and time limits, rate-limited.
What this robots.txt validator does
Enter a domain and the tool fetches its robots.txt, splits it into user-agent groups, and lists the allow and disallow rules for each. It reports the sitemap directives, counts the lines, and flags problems: rules that appear before any user-agent line, paths that do not start with a slash, unknown directives, sitemap lines that are not absolute URLs, and a missing sitemap directive.
It also raises the one mistake that matters most: a wildcard group that disallows the root, which tells every crawler to stay out of the whole site.
Why check robots.txt
A single misplaced line can remove a site from search results, and the file is easy to break during a migration or a staging copy going live. Checking it after every deploy takes seconds and prevents the most expensive SEO accident there is.
Questions
What does "Disallow: /" mean?
It blocks the entire site for the user-agents in that group. Under a wildcard user-agent it blocks all crawlers, which is appropriate for staging and catastrophic for production.
Does robots.txt remove pages from Google?
No. It stops crawling, not indexing. A blocked page can still appear in results without a snippet. To remove a page, allow crawling and use a noindex meta tag.
What if the site has no robots.txt?
The tool reports the status it received. A 404 is fine; crawlers treat it as "crawl everything". A 500 or a timeout is worth investigating because some crawlers back off when the file errors.
Have a project in mind?