> ## Content Index > Fetch the complete content index at: https://www.hyperank.ai/blog/llms.txt > Use this file to discover other available public pages before exploring further. # robots.txt Disallow All vs Allow All: How to Block or Open Your Whole Site - URL: https://www.hyperank.ai/blog/robots-txt-disallow-all/ - Published: 2026-10-28T19:30:05.000Z - Updated: 2026-10-28T19:30:05.000Z - Description: Learn how to effectively use robots.txt disallow all and allow all to control search engine access to your entire site in this step-by-step guide. - Author: Hyperank AI *robots.txt disallow all | Updated October 2026 | hyperank Editorial Team | 10-20 minutes | Beginner* ## What You'll Learn **To disallow all crawlers from your whole site, put these two lines in robots.txt: `User-agent: *` and `Disallow: /`. To allow everything, use `User-agent: *` with an empty `Disallow:` (Google ignores a rule with no path), or `Allow: /`, or simply have no rules at all. Disallow tells compliant crawlers not to fetch matching URLs; it doesn't remove pages from Google, and a disallowed URL can still be indexed if other pages link to it. To keep pages out of search results, use noindex instead and let crawlers see it.** This guide, robots.txt Disallow All vs Allow All: How to Block or Open Your Whole Site, shows the exact rules, how to block one crawler or one folder, when blocking everything is a mistake, and the difference between "do not crawl" and "do not index". - **Exact syntax:** Copy-ready rules to block or allow everything. - **Targeted blocks:** How to block a single crawler, a folder or a file type. - **Common traps:** Staging rules going live and blocking pages that are already indexed. - **Crawl vs index:** Why Disallow doesn't remove pages from Google. **Prerequisites:** Access to edit your robots.txt, through your server, CMS or SEO plugin. --- ## Why robots.txt Disallow Rules Matter in 2026 robots.txt is a small file with a big blast radius. A single line, `Disallow: /`, tells every compliant crawler to stay out of your entire site. That's exactly what you want on a staging server, and exactly what you don't want on your live site. The stakes have grown with AI. Besides Googlebot and Bingbot, AI companies run their own crawlers, such as OpenAI's OAI-SearchBot and GPTBot and PerplexityBot, and many site owners now want different rules for each. Google's [robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec?ref=hyperank.ai) explains how crawlers pick which group of rules to follow, which matters once you start writing rules for specific bots. It's also widely misunderstood. Google's [introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro?ref=hyperank.ai) stresses that robots.txt is for managing crawler traffic, not for keeping a page out of Google. --- ## The Process at a Glance | Step | Action | Time | Outcome | | ---- | ------------------------------------ | ----- | ------------------------------- | | 1 | Write the two-line disallow all rule | 2 min | Whole site blocked to crawlers | | 2 | Write an allow all rule instead | 2 min | Whole site open to crawlers | | 3 | Understand what Disallow means | 3 min | Know what rules actually do | | 4 | Block specific crawlers only | 5 min | One bot blocked, others allowed | | 5 | Block a folder or file type | 5 min | Targeted paths blocked | | 6 | Avoid disallow all mistakes | 5 min | Live site stays crawlable | | 7 | Use noindex for removal instead | 5 min | Pages actually leave Google | **Total time:** About 10 to 20 minutes, plus testing. --- ## Step 1: Disallow All With Two Lines ### What You're Doing You're telling every compliant crawler not to crawl any URL on your site. ### How to Do It 1. Open or create robots.txt at the root of your site, for example https://staging.example.com/robots.txt. 2. `User-agent: *` means the rules apply to all crawlers without a more specific group of their own. `Disallow: /` matches the root and every URL below it. 3. Save it as a UTF-8 plain text file named exactly robots.txt and upload it to the top-level directory. 4. Load the file in a browser to confirm it's live. Add exactly these two lines: ``` User-agent: * Disallow: / ``` ### Common Mistakes - **Expecting it to stop everyone:** robots.txt is a request, not a lock. Reputable crawlers follow it; malicious bots can ignore it. Use passwords or IP restrictions for truly private content. - **Using it on your live site by accident:** See Step 6. ### What Done Looks Like Your robots.txt shows the two lines, and a robots.txt tester reports any URL on the site as blocked. --- ## Step 2: Allow All: Empty Disallow vs Allow: / ### What You're Doing The opposite case is just as common: you want every crawler to access everything, and you want the file to say so clearly. ![Google Search Central page showing robots.txt documentation and specifications link](https://amhvrlzbdnkjotzniuhu.supabase.co/storage/v1/object/public/hyperank/screenshots/guide-a1f831f3-e978-4742-826b-896b220083d1.webp) ### How to Do It Any of these three options allows all crawling: | Option | robots.txt content | Why it works | | ------------------- | -------------------------- | --------------------------------------------------------------------------------------------------- | | Empty Disallow | User-agent: \*Disallow: | Google's spec says "when no path is specified, the rule is ignored" | | Allow root | User-agent: \*Allow: / | Explicitly allows the root and all URLs below it | | No rules or no file | Empty file, or none at all | Google treats a 4xx robots.txt response (except 429) as if no file existed, meaning no restrictions | 1. Pick one option; the empty Disallow is the classic convention and widely understood. 2. Add your Sitemap line at the end, for example `Sitemap: https://www.example.com/sitemap.xml`. Our guide to [adding your sitemap to robots.txt](https://www.hyperank.ai/blog/sitemap-in-robots-txt/) covers the details. ### Common Mistakes - **Confusing "Disallow:" with "Disallow: /":** One character separates allowing everything from blocking everything. Double-check the slash. ### What Done Looks Like Your file clearly allows all crawling, and tests show key URLs as allowed. --- ## Step 3: Understand What robots.txt Disallow Means ### What You're Doing Blocking or opening everything is the easy part; most real sites need something in between. Before writing targeted rules, you're learning exactly how Disallow is read, so your rules do what you expect. ### How to Do It - **Disallow means "don't fetch":** It asks crawlers not to request URLs whose path starts with the value. `Disallow: /admin` blocks /admin, /admin/ and /admin-login. - **Paths are case-sensitive:** `/Private/` and `/private/` are different. - **Wildcards are supported:** Google's spec says `*` matches zero or more characters and `$` marks the end of the URL. - **The most specific rule wins:** When Allow and Disallow both match a URL, Google uses the rule with the longest matching path; if they're equal, it uses the less restrictive one. - **One group per crawler:** A crawler follows only the most specific user-agent group that matches it. Google's spec notes that user-agent-specific groups and the global `*` group are not combined. - **Caching:** Google generally caches robots.txt for up to 24 hours, so changes aren't instant. ### Example | Rule | Blocks | Doesn't block | | ------------------- | ---------------------------------------- | -------------------------------------------------- | | Disallow: /private/ | /private/, /private/report.html | /private-events/ | | Disallow: /\*.pdf$ | /files/guide.pdf | /files/guide.pdf?download=1 | | Disallow: /search | /search, /search?q=shoes, /searchresults | /products/search-tips (doesn't start with /search) | Test any URL against your rules with our free [robots.txt checker](https://www.hyperank.ai/tools/technical-seo/robots-txt-checker?ref=hyperank.ai), which applies Google's longest-match logic and shows whether it's allowed or blocked. ### What Done Looks Like You can predict whether a given URL is blocked by reading your rules. --- ## Step 4: Block Specific Crawlers Only (for Example a Single AI Crawler) ### What You're Doing With the matching rules clear, you can get selective. Often you don't want to block everyone, just one bot, so you're writing a group for that crawler while leaving the rest open. ![Google Search Central robots.txt introduction page explaining how robots.txt works](https://amhvrlzbdnkjotzniuhu.supabase.co/storage/v1/object/public/hyperank/screenshots/guide-3e9a523b-3948-4bc9-9231-2469c44a3a75.webp) ### How to Do It 1. Find the crawler's user-agent token in its owner's documentation. 2. Decide deliberately which AI bots to block. Blocking a training crawler is different from blocking a search crawler. Add a group for it with `Disallow: /`, and keep a separate open group for everyone else: ``` User-agent: GPTBot Disallow: / User-agent: * Disallow: ``` | Token | Owner | Used for (per owner's docs) | Effect of blocking | | --------------- | ---------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | GPTBot | OpenAI | Crawling content that may be used in training | Opts out of training use | | OAI-SearchBot | OpenAI | Surfacing websites in ChatGPT's search features | Can keep you out of ChatGPT search results | | PerplexityBot | Perplexity | Surfacing and linking websites in Perplexity search | Can keep you out of Perplexity answers | | Google-Extended | Google | Controlling whether crawled content may be used to train future Gemini models | Opts out of that use; Google says it's a control token, not a separate crawler | Sources: [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots?ref=hyperank.ai), [Perplexity's bot documentation](https://docs.perplexity.ai/guides/bots?ref=hyperank.ai) and [Google's list of common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers?ref=hyperank.ai). Check which bots can currently reach you with our free [AI bot crawler checker](https://www.hyperank.ai/tools/ai-visibility/ai-bot-crawler-checker?ref=hyperank.ai), which tests GPTBot, ClaudeBot, PerplexityBot and 11 other AI bots by robots.txt rule and by live request. ### What Done Looks Like The chosen crawler is blocked while Googlebot, Bingbot and other bots you want remain allowed. --- ## Step 5: Block a Folder or File Type ### What You're Doing The same logic works for parts of your own site. Most sites only need to block a few areas, such as admin, cart or internal search, so you're writing targeted rules. ### How to Do It 1. Test each rule with the robots.txt checker before saving. Re-allow something inside a blocked folder with a longer Allow rule: ``` User-agent: * Disallow: /media/ Allow: /media/press-kit/ ``` Block URL parameters, such as internal search or sort options: ``` User-agent: * Disallow: /*?sort= Disallow: /search ``` Block a file type with a wildcard and end anchor: ``` User-agent: * Disallow: /*.pdf$ ``` Block a folder with a trailing slash: ``` User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ ``` ### Best Practices - **Don't block CSS or JavaScript** that pages need to render; Google needs them to understand your pages. - **Keep it short:** Fewer, clearer rules are easier to maintain. Google enforces a 500 KiB file size limit. ### What Done Looks Like Only the intended folders, files or parameters are blocked, and important pages test as allowed. --- ## Step 6: Know When Disallow All Is a Mistake ### What You're Doing Now for the cautionary part. The disallow all rule causes more SEO accidents than almost anything else, so you're learning the situations where it backfires. ### How to Do It - **Staging sites going live:** A staging robots.txt with `Disallow: /` gets copied to production during launch, and the whole site stops being crawled. Add a robots.txt check to every launch checklist. It's usually spotted a week later, when someone asks why traffic fell off a cliff. - **CMS "discourage search engines" settings:** In WordPress, the Settings > Reading option to discourage search engines is often left on after launch. Our WordPress robots.txt guide explains where to check. - **Trying to remove indexed pages:** Blocking pages that are already in Google stops crawlers from seeing a noindex tag, so they can stay indexed. See Step 7. - **Protecting private data:** robots.txt is public, so listing secret folders advertises them. Use authentication instead. - **Blocking AI search crawlers by accident:** A broad block can keep you out of ChatGPT or Perplexity answers you'd actually want to appear in. | Situation | Use disallow all? | Better approach | | ---------------------------- | ----------------------------- | ---------------------------------------------- | | Private staging or test site | Yes, plus password protection | Password or IP restriction | | Live site you want in search | No | Allow all, block only specific paths | | Removing pages from Google | No | noindex, or the Removals tool for urgent cases | | Opting out of AI training | No, not for all bots | Block only the relevant training tokens | ### What Done Looks Like Your live site has no blanket Disallow, and your launch checklist includes a robots.txt check. --- ## Step 7: Understand Do Not Crawl vs Do Not Index ### What You're Doing This is the distinction most people get wrong. You're using the right tool to keep a page out of search results. ### How to Do It 1. **Know the rule:** Disallow controls crawling, not indexing. Google's robots.txt introduction explains that a page blocked by robots.txt can still appear in search results if other pages link to it, usually without a description. 2. **Use noindex to keep pages out:** Add `` or an X-Robots-Tag HTTP header. Check pages with our noindex checker. 3. **Don't block noindexed pages in robots.txt:** If Google can't crawl the page, it never sees the noindex tag. 4. **Don't put noindex in robots.txt:** Google stopped supporting noindex rules in robots.txt in 2019, as its [note on unsupported rules](https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt?ref=hyperank.ai) explains. 5. **For urgent removals:** Use the Removals tool in Search Console as a temporary measure while noindex takes effect; Google's [guide to removing information](https://developers.google.com/search/docs/crawling-indexing/remove-information?ref=hyperank.ai) covers the options. | Goal | robots.txt Disallow | noindex | | ---------------------------- | ------------------------ | ------------------ | | Stop crawling | ✓ | ✗ | | Keep out of search results | ✗ (may still be indexed) | ✓ | | Crawler must access the page | No | Yes | | Works for non-HTML files | ✓ | ✓ via X-Robots-Tag | ### What Done Looks Like Pages you want out of Google carry noindex and are crawlable; robots.txt only blocks what you don't need crawled. --- ## What to Do After Changing robots.txt - **Phase 1, same day, test:** Check key URLs with a robots.txt tester and confirm the live file loads. - **Phase 2, first week, monitor:** Watch Search Console's robots.txt and Page indexing reports for "Blocked by robots.txt" changes. - **Phase 3, ongoing, protect:** Re-check robots.txt after every launch, migration or CMS change; our technical SEO audit checklist includes it. Getting crawler access right is also the first step to appearing in AI answers. To see what AI engines actually say about your brand once they can reach you, hyperank asks six major AI engines your buyers' questions every day and shows what they say, what they get wrong and which competitors they recommend. --- ## Resources You'll Need | Resource | Role | Label | Price | | --------------------------------------------------------------------------------- | ------------------------------------- | ----------- | ----- | | hyperank robots.txt checker | Test URLs against your rules | Required | Free | | hyperank AI bot crawler checker | See which AI bots can crawl your site | Recommended | Free | | hyperank noindex checker | Confirm noindex tags | Recommended | Free | | [Google Search Console](https://search.google.com/search-console?ref=hyperank.ai) | robots.txt report and indexing status | Required | Free | | Google's robots.txt specification | Official syntax and matching rules | Optional | Free | --- ## Troubleshooting Common Issues ### Pages show "Indexed, though blocked by robots.txt" **Likely cause:** The URL is disallowed but linked from elsewhere. **Fix:** If it should be out of Google, remove the Disallow and add noindex; if it should rank, remove the Disallow. ### Your whole site disappeared from Google **Likely cause:** A `Disallow: /` went live. **Fix:** Replace it with an allow-all file, then request recrawling of robots.txt in Search Console. ### A specific bot ignores your block **Likely cause:** Wrong user-agent token, or a bot that doesn't follow robots.txt. **Fix:** Check the token in the owner's docs; block non-compliant bots at the server or firewall. ### Changes aren't taking effect **Likely cause:** Google caches robots.txt for up to 24 hours. **Fix:** Wait, or request a recrawl in Search Console's robots.txt report. --- ## Conclusion The robots.txt disallow all rule is two lines, `User-agent: *` and `Disallow: /`, and allow all is the same with an empty Disallow. Use disallow all only on private staging sites, block specific bots or folders with targeted groups, and use noindex, not robots.txt, to keep pages out of Google. ### Key Takeaways - **Outcome:** You can block or open your whole site, a folder, a file type or a single crawler. - **Key insight:** Disallow stops crawling, not indexing; noindex needs the page to be crawlable. - **Next action:** Open your live robots.txt now and make sure it doesn't contain `Disallow: /` by mistake. --- ## FAQ ### How do you disallow all or allow all in robots.txt to block or open your whole site? robots.txt Disallow All vs Allow All: How to Block or Open Your Whole Site comes down to two short rules. To block every compliant crawler from the whole site, use `User-agent: *` followed by `Disallow: /`. To allow everything, use `User-agent: *` with an empty `Disallow:`, which Google ignores because it has no path, or `Allow: /`. Remember that Disallow only stops crawling: blocked URLs can still be indexed if linked elsewhere, so use a noindex tag on crawlable pages to keep them out of Google. ### What does robots.txt Disallow mean? It asks compliant crawlers not to fetch URLs whose path starts with the given value. It controls crawling, not indexing. ### Does "Disallow:" with nothing after it block my site? No. An empty Disallow has no path, and Google's spec says such rules are ignored, so it allows everything. ### How do I block only one AI crawler? Add a group with that bot's user-agent token and `Disallow: /`, for example `User-agent: GPTBot`, and keep a separate open group for all other crawlers. ### Will disallow all remove my site from Google? Not reliably. It stops crawling, but already-known URLs can remain indexed, usually without descriptions. Use noindex or the Removals tool to take pages out. ### What robots.txt rules block crawlers from a folder? Use `Disallow: /folder-name/` under the relevant User-agent line. Add a longer Allow rule to re-open a subfolder. ### Do all bots follow robots.txt? Reputable crawlers like Googlebot, Bingbot and the documented AI crawlers do. Malicious bots may ignore it, so use server-level blocks for those. --- *Methodology: Syntax, matching, caching and error-handling rules come from Google's robots.txt specification and introduction, Google's 2019 note on unsupported rules and its list of common crawlers; AI bot purposes come from OpenAI's and Perplexity's own documentation. Behaviour of other search engines may differ slightly.*