What Is robots.txt? How It Works, Where It Goes, and Whether You Need One
Discover what is robots.txt, how it functions, where to place it, and whether your website truly needs one for effective search engine management.
what is robots.txt | Updated October 2026 | 12 min read | hyperank Editorial Team
What is robots.txt? It's a plain text file at the root of a website that tells crawlers, such as Googlebot or GPTBot, which parts of the site they may or may not crawl. It's a voluntary standard: well-behaved bots follow it, but it doesn't hide pages, secure them or force any bot to obey. Most sites benefit from a simple robots.txt that blocks low-value areas and points to the XML sitemap. This guide, What Is robots.txt? How It Works, Where It Goes, and Whether You Need One, covers how the file works, where it must live, what it can't do, its legal status, how it applies to AI crawlers, and when to use noindex instead.
robots.txt is a request, not a lock. It tells polite crawlers where not to go; it doesn't stop anyone who chooses to ignore it.
Getting robots.txt right matters more than ever, because the same file now controls access for search engines and for AI crawlers that power assistants like ChatGPT and Perplexity. A single misplaced line can remove your site from both.
What Is robots.txt?
robots.txt is a text file, placed at the root of a domain (for example, https://example.com/robots.txt), that gives crawlers rules about which URLs they may request. It follows the Robots Exclusion Protocol, formally standardized as RFC 9309 in 2022.
- Purpose: Manage crawler access, mainly to avoid wasting crawl activity on unimportant or duplicate URLs.
- Format: Plain UTF-8 text with simple directives like User-agent, Disallow and Allow.
- Audience: Automated crawlers, not human visitors.
- Visibility: Public; anyone can read your robots.txt by visiting /robots.txt.
A minimal example
A simple file might look like this:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlThis tells all crawlers not to crawl the admin and cart areas, and points them to the sitemap. Everything else is allowed by default.
What Does robots.txt Do, and How Does It Work?
robots.txt works through groups of rules. Each group starts with one or more User-agent lines naming the crawler, followed by Allow and Disallow rules for URL paths. A crawler finds the group that matches its name most specifically and follows those rules, ignoring the others.
| Directive | What it does | Example |
|---|---|---|
| User-agent | Names the crawler the group applies to; * means all | User-agent: Googlebot |
| Disallow | Blocks crawling of a path | Disallow: /private/ |
| Allow | Permits a path, often inside a disallowed folder | Allow: /private/press/ |
| Sitemap | Gives the location of an XML sitemap | Sitemap: https://example.com/sitemap.xml |
How crawlers apply the rules
- Fetch first: A compliant crawler requests /robots.txt before crawling other URLs on that host.
- Match the group: It uses the most specific User-agent group that matches it; if none matches, it uses the * group.
- Longest match wins: According to Google's robots.txt documentation, when Allow and Disallow rules conflict, Google uses the most specific (longest) matching rule, and Allow wins a tie.
- Wildcards: Google supports * for any sequence of characters and $ for the end of a URL.
- Caching: Google generally caches robots.txt for up to 24 hours, so changes may take a day to apply.
A worked example of rule matching
Suppose your file contains these rules:
User-agent: *
Disallow: /shop/
Allow: /shop/sale/
User-agent: Googlebot
Disallow: /shop/filters/Googlebot matches its own group, so it ignores the * group entirely and only avoids /shop/filters/. Every other compliant crawler uses the * group: it skips /shop/ except /shop/sale/, because the longer Allow rule is more specific. This is why it's common to repeat important rules in each named group rather than assuming the * group applies to everyone.
What happens if robots.txt is missing
If there's no robots.txt (the server returns a 404), crawlers assume they can crawl everything. Google notes that server errors are treated differently: persistent 5xx errors can cause Google to pause crawling the site. For a deeper look at how crawlers behave, see what a web crawler is.
Where Should robots.txt Be Located?
robots.txt must be at the root of each host, at exactly /robots.txt. A file at example.com/folder/robots.txt is ignored. Google's documentation also notes that rules apply only to the host, protocol and port where the file is served.
| Situation | Where the file goes | Notes |
|---|---|---|
| Main domain | https://example.com/robots.txt | Applies only to example.com |
| Subdomain | https://blog.example.com/robots.txt | Each subdomain needs its own file |
| HTTP vs HTTPS | Separate files per protocol | Usually only the HTTPS version matters after redirects |
| WordPress | Virtual file at the root, editable via SEO plugins | See our WordPress guide |
| Shopify | Generated at /robots.txt; customizable with robots.txt.liquid | Edit through the theme code |
Does robots.txt apply to subdomains?
No. Each subdomain is a separate host, so shop.example.com and blog.example.com each need their own robots.txt if you want to set rules for them.
Platform notes
- WordPress: WordPress generates a virtual robots.txt; plugins like Yoast or Rank Math let you edit it. See our WordPress robots.txt guide.
- Shopify: Shopify generates a default file, and you can customize it by adding a robots.txt.liquid template, as explained in Shopify's developer documentation.
- Size: Google processes up to 500 KiB of a robots.txt file and ignores anything beyond that.
Do I Need a robots.txt File?
You don't strictly need a robots.txt file; without one, crawlers can access everything. But most sites benefit from a simple one that keeps crawlers out of low-value areas and lists the sitemap.
- Small sites: A basic file with a sitemap line is enough; many small sites need no Disallow rules at all.
- Ecommerce sites: Useful for blocking cart, checkout, account and endless filtered URLs.
- Large sites: Important for managing crawl activity across faceted navigation and internal search pages.
- Staging sites: Often blocked, though password protection is safer.
Is robots.txt mandatory?
No. It isn't required by search engines, and a missing file won't cause a penalty. It's a tool to use when you have crawling you want to manage.
Key Takeaway: If you don't need to block anything, a robots.txt with just a sitemap line is a clean, useful default. You can create one with the sitemap and robots.txt generator.
A sensible default for most sites
If you're unsure where to start, a common pattern is to allow everything, block only clearly private or duplicate areas such as admin, cart, checkout and internal search results, and list your sitemap. Review the file whenever you launch a new section, change platforms or move to a new domain, since those are the moments rules most often break.
What robots.txt Cannot Do
robots.txt controls crawling, not indexing or security. It doesn't hide pages, protect private content, or guarantee that every bot stays out.
| Myth | Reality |
|---|---|
| Blocking a page removes it from Google | A blocked URL can still be indexed (without content) if other pages link to it |
| robots.txt hides private pages | The file is public and can reveal the paths you list |
| All bots obey it | Compliance is voluntary; malicious scrapers often ignore it |
| It's a security control | Use authentication and access controls for sensitive content |
| Noindex works in robots.txt | Google stopped supporting noindex in robots.txt in 2019 |
Is robots.txt a vulnerability?
Not by itself, but it can point people to sensitive paths if you list them. Don't rely on robots.txt to hide admin panels or private files; protect them with logins instead, and avoid listing secret URLs.
Can robots.txt stop crawlers?
It stops crawlers that choose to follow it, which includes major search engines and the main AI companies' documented crawlers. It can't stop bots that ignore the rules; that requires server-side blocking, rate limiting or a firewall.
If something must stay private, put it behind a login. robots.txt only works on bots that agree to listen.
Common mistakes that block too much
- A stray slash: "Disallow: /" blocks the whole site; it's often left over from a staging environment.
- Blocking CSS and JavaScript: Search engines need these files to render pages properly.
- Blocking pages you want deindexed: Crawlers can't see a noindex tag on a page they're not allowed to fetch.
- Forgetting subdomains: Rules on the main domain don't cover blog or shop subdomains.
Is robots.txt Legally Binding?
robots.txt is a voluntary technical convention, not a law or contract in itself. RFC 9309 states that the rules are not a form of access authorization. Whether ignoring robots.txt has legal consequences depends on other factors, such as a site's terms of service, copyright and computer misuse laws, and courts' views vary by jurisdiction. This isn't legal advice; consult a lawyer for a specific situation.
- Voluntary standard: Compliance relies on crawler operators choosing to respect it.
- Industry practice: Reputable search engines and AI companies publish their crawler names and state that they honor robots.txt.
- Can you ignore robots.txt? Technically a bot can, but doing so goes against an established web convention and may breach a site's terms. Responsible crawlers and scrapers should respect it.
Is robots.txt Still Used, and Does It Work With AI Crawlers?
Yes. robots.txt is still the main way to manage crawler access, and it now matters for AI crawlers too. Major AI companies publish user agents that respect robots.txt, so you can allow or block them individually.
| Crawler token | Operator | Main purpose |
|---|---|---|
| GPTBot | OpenAI | Collecting content for model training |
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search |
| ClaudeBot | Anthropic | Collecting content for model training |
| PerplexityBot | Perplexity | Indexing sites for Perplexity answers |
| Google-Extended | Control token for Gemini training use; doesn't affect Google Search |
- Blocking is a trade-off: Blocking AI search crawlers can keep your content out of AI answers that might mention or cite your brand.
- Separate training from search: Some operators use different user agents for training and for search, so you can treat them differently.
- Check what you allow: The AI bot crawler checker shows which AI crawlers your robots.txt allows or blocks.
hyperank tracks what AI engines such as ChatGPT, Perplexity and Gemini actually say about your brand, which helps you see whether crawler access decisions are affecting how you show up. For a related file aimed at AI tools, see our llms.txt guide.
When to Use robots.txt, and When to Use Noindex Instead
Use robots.txt when you want to stop crawlers from fetching URLs. Use a noindex meta tag or X-Robots-Tag header when you want a page kept out of search results. The two don't mix well: if robots.txt blocks a page, search engines can't crawl it to see its noindex tag.
| Goal | Use | Why |
|---|---|---|
| Save crawl activity on filters or internal search | robots.txt Disallow | Stops crawlers fetching low-value URLs |
| Keep a page out of Google results | noindex tag or header | Search engines must crawl the page to see it |
| Protect private content | Password protection | Neither robots.txt nor noindex secures content |
| Block an entire staging site | Password protection, plus robots.txt | Prevents both crawling and accidental indexing |
| Block or allow AI crawlers | robots.txt user-agent groups | Major AI crawlers honor these rules |
- Removing pages: Add noindex, let Google recrawl, and only then consider blocking in robots.txt if needed.
- Check noindex: The noindex checker shows whether a page carries a noindex directive.
- Blocking everything: See our guide to robots.txt Disallow all vs Allow all before using a site-wide block.
- Sitemaps: Learn how to add your sitemap to robots.txt.
robots.txt FAQ: Case Sensitivity, Visibility and Editing
Three practical questions come up often: whether robots.txt is case sensitive, whether it should be visible, and how to edit it.
- Case sensitivity: The file name must be lowercase robots.txt, and URL paths in rules are case sensitive, so Disallow: /Folder/ doesn't block /folder/. Directive names like "disallow" are not case sensitive.
- Visibility: robots.txt should be publicly accessible; crawlers need to fetch it without logging in, and it should return a 200 status.
- Editing: Edit the file on your server, through your CMS or SEO plugin, or through a theme template on platforms like Shopify. Then test it with the robots.txt checker.
A quick check routine
After any change, open yoursite.com/robots.txt in a browser, run it through a checker, confirm important pages and your sitemap aren't blocked, and review the robots.txt report in Google Search Console a day later.
Conclusion
robots.txt is a simple, public file at the root of each host that tells crawlers which URLs they may request. It manages crawling for search engines and AI crawlers, but it doesn't hide pages, secure content or guarantee compliance, so use noindex for search removal and authentication for privacy.
Key Takeaways
- What it is: A voluntary standard (RFC 9309) for crawler access rules.
- Where it goes: At /robots.txt on every host, including each subdomain.
- Do you need one: Not required, but a simple file with a sitemap line is a good default.
- Limits: Not a security tool and not a way to deindex pages.
- AI crawlers: Major AI crawlers honor robots.txt, so your choices affect AI visibility.
Check what your robots.txt allows with the free robots.txt checker, and see what AI says about your brand with hyperank.
FAQ
What is robots.txt, how does it work, where does it go, and do you need one?
robots.txt is a public text file at the root of a host that tells crawlers which URLs they may crawl, using User-agent groups with Allow and Disallow rules. It must be at /robots.txt on each host and subdomain. It isn't mandatory, but most sites benefit from a simple file that blocks low-value areas and lists the sitemap.
Is robots.txt case sensitive?
The file name must be lowercase, and URL paths in rules are case sensitive. Directive names are not.
Should robots.txt be visible?
Yes. It must be publicly accessible so crawlers can read it, which also means anyone can see it.
How do I edit robots.txt?
Edit the file on your server, use your CMS or SEO plugin, or on Shopify add a robots.txt.liquid template. Test the result afterwards.
Does robots.txt still work?
Yes. Major search engines and AI companies' documented crawlers still follow robots.txt.
Is robots.txt legally binding?
It's a voluntary convention, not a law. Legal consequences of ignoring it depend on other factors and jurisdictions; seek legal advice for specific cases.
Can robots.txt block AI crawlers?
Yes, for crawlers that honor it, such as GPTBot, ClaudeBot and PerplexityBot. Blocking them may reduce how AI tools can use or cite your content.
Methodology: This article draws on RFC 9309 (the Robots Exclusion Protocol), Google Search Central's robots.txt documentation and Shopify's developer documentation. Crawler names reflect each operator's published documentation as of October 2026. This article is not legal advice.
See where your brand stands
How six AI engines describe you, next to your competitors. About two minutes.