Robots.txt Tester

How to use the Robots.txt Tester

1. Paste your robots.txt

   Fetch it from yoursite.com/robots.txt and paste the whole file, comments included. Test the live file rather than what you think you deployed — the two disagree more often than you would expect.

2. Enter the URL you want to check

   A full URL or just the path. Test the ones that matter: your money pages, your blog index, and anything that has recently dropped out of search results.

3. Pick the crawler

   Googlebot for search, Googlebot-Image for image search, Bingbot for Bing. Rules under a specific user-agent override the wildcard group entirely, which surprises people constantly.

4. Read which rule matched

   The result shows the exact line that decided the outcome. Where Allow and Disallow conflict, the longer, more specific path wins — not the one that appears first.

5. Fix anything flagged, then re-test

   Correct the file, redeploy it to the site root, and run the check again against the live URL before you trust it.

What this tool does

What robots.txt actually controls

A robots.txt file controls crawling, not security and not direct indexing removal. It gives compliant search-engine crawlers instructions about which paths they may or may not request. Understanding that distinction is important because many robots.txt problems come from expecting it to do something it was never designed to do.

It does not hide a page. The robots.txt file itself is publicly accessible at a predictable URL, and the paths listed inside it can reveal areas of a website. Never use robots.txt to protect an admin panel, staging environment or private directory. Use authentication and access controls for anything that should remain private.

It also does not reliably remove a URL from Google Search. Google can still discover blocked URLs from links or other sources, and can show the URL in search results without crawling its content If you want Google to remove a page from Search using noindex, Google needs to be able to crawl the page and read the noindex directive.

That is why robots.txt and noindex should not be treated as interchangeable controls. Robots.txt is primarily about crawl access, while noindex is an indexing directive that Google can process when it can access the page.

The rules, briefly

The file is a set of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow directives.

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/public-info.html

Sitemap: https://www.example.com/sitemap.xml

Three things decide the outcome:

• Only one group applies. A crawler uses the most specific group that names it, and ignores every other group entirely. If you have a User-agent: Googlebot section, Googlebot reads that one and completely ignores User-agent: * — including any Disallow lines you assumed were global.
• Longest match wins. Where an Allow and a Disallow both match a URL, the rule with the longer path takes precedence, regardless of order in the file.
• Paths are prefixes. Disallow: /blog blocks /blog, /blog/ and /blogging-tips. Add the trailing slash unless you mean the prefix.

Google and other major search engines support limited wildcards in robots.txt. The wildcard can match any sequence of characters, while $ indicates the end of the URL path. For example, Disallow: /.pdf$ can target paths ending in .pdf, while a URL such as /report.pdf?download=1 does not match that end-of-path pattern.

The mistakes that take sites offline

1. Disallow: / —  this tells compliant crawlers not to crawl any path on the host. It is especially dangerous when a staging robots.txt file is accidentally deployed to the live website. If important pages suddenly stop being crawled, check the live robots.txt file early in the investigation.
2. Blocking CSS and JavaScript. Google renders pages to evaluate them. Block your/assets/ or /wp-includes/ folder and Googlebot sees a broken, unstyled page, which hurts mobile-usability assessment. Modern sites should leave CSS and JS crawlable.
3. Blocking a page you also want deindexed. If Google cannot crawl a page because robots.txt blocks it, Google cannot read a noindex robots meta tag or X-Robots-Tag header on that page. A blocked URL may still be known to Google and can appear in Search without its page content.
4. A specific user-agent group that silently disables the wildcard rules. Adding a User-agent: Googlebot group does not make Googlebot follow the rules from the wildcard group as well. Check the complete applicable group whenever you add crawler-specific rules. *.
5. Wrong location. The file must be at the domain root— example.com/robots.txt. At example.com/files/robots.txt it does nothing at all. Subdomains need their own file; blog.example.com does not inherit from example.com.
6. Missing sitemap line. This is not an error ; the Sitemap: field is optional, but adding the absolute sitemap URL can help crawlers discover the sitemap. The sitemap field is independent of a particular user-agent group.

A sane starting file

For most brochure and business sites, less is more:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?s=
Allow: /

Sitemap: https://www.example.com/sitemap.xml

That blocks the admin area, the transactional pages that have no business in search, and internal search result pages which generate near-infinite low-quality URLs if left crawlable. Everything else is open, including assets.

For WordPress specifically, resist the enormous robots.txt files that circulate on forums.  Blocking /wp-content/ breaks rendering. Blocking /wp-admin/ is fine, but add Allow: /wp-admin/admin-ajax.php, since some themes and plugins depend on it for front-end functionality.

Crawl budget — and who it applies to

Managing crawl budget” is often given as a reason for adding aggressive robots.txt rules. For most small websites, that is not a good reason to block large sections of the site.

Crawl-budget management becomes more relevant for very large websites, frequently updated sites, or websites that generate large numbers of crawlable URLs, such as e-commerce sites with faceted navigation and extensive URL parameters. Google’s current crawl-budget guidance also emphasizes managing unnecessary URLs rather than using robots.txt simply to redistribute crawl activity.

If a website has only a few hundred useful pages, blocking important sections just to “save crawl budget” can create unnecessary SEO problems. Make sure crawlers can reach the pages that matter and use robots.txt where there is a genuine reason to prevent crawling of particular paths or resources.

After you change the file

Deploy the robots.txt file to the correct root URL, then open /robots.txt in a browser and make sure the live file contains the rules you wanted. After this, test the required URLs again with your Robots.txt Tester. Check both allowed and blocked URLs, especially those pages that are affected by new Allow, Disallow, wildcard or user-agent rules. If you are fixing visibility issues related to appearing in Google Search, use Google Search Console and URL Inspection to check the affected URLs. Also remember that Google may cache robots.txt for some time, so changes made to the live file may not appear immediately in every crawl. Google says it typically caches robots.txt for up to 24 hours, although in some situations caching can last longer.

One final precaution: if you use a staging environment, make the robots.txt file environment-aware (changing according to the environment) instead of a static file in your repository. A one-line conditional command that shows < Disallow: /'on staging and the real file in production prevents the biggest and most expensive mistake that can happen on this page.

Frequently asked questions