Noindex in robots.txt: Why It No Longer Works and What to Use Instead
Discover why the robots txt no index directive is outdated and learn effective alternatives in our comprehensive step-by-step guide.
robots txt no index | Updated October 2026 | hyperank Editorial Team | 20 minutes | Easy
You can't use noindex in robots.txt any more: Google stopped supporting it on September 1, 2019, so a "Noindex:" line in robots.txt is simply ignored. To keep a page out of search results, use a meta robots noindex tag, an X-Robots-Tag HTTP header for PDFs and other files, or password protection or removal for pages that shouldn't exist publicly. Don't combine Disallow with noindex, because a crawler that can't fetch the page can't see the tag. This guide, Noindex in robots.txt: Why It No Longer Works and What to Use Instead, walks through each option step by step and shows how to check that it worked.
robots.txt decides whether a crawler may fetch a page. noindex decides whether a page may appear in results. Mixing them up is one of the most common reasons "blocked" pages still show in Google.
What You'll Learn
Here's the short version. Remove any Noindex lines from robots.txt. For HTML pages, add <meta name="robots" content="noindex"> and make sure robots.txt allows crawling. For PDFs, images and other files, send an X-Robots-Tag: noindex header. For anything private, use a login or delete the page and return a 404 or 410. Then check the result with a noindex checker and Google Search Console.
- Why it changed: What Google announced in 2019 and why.
- The trap: Why Disallow plus noindex fails.
- Three fixes: Meta robots, X-Robots-Tag, and protection or removal.
- Verification: How to confirm a page's noindex status.
The Process at a Glance
| Step | Action | Time | Outcome |
|---|---|---|---|
| 1 | Confirm Google ignores noindex in robots.txt | 2 minutes | Old rules removed |
| 2 | Understand why Disallow doesn't deindex | 3 minutes | Right tool chosen |
| 3 | Fix the Disallow plus noindex mistake | 5 minutes | Crawlers can see your noindex |
| 4 | Add a meta robots noindex tag | 5 minutes per template | HTML pages kept out of results |
| 5 | Use an X-Robots-Tag header | 10 minutes | PDFs and files kept out of results |
| 6 | Protect or remove the page | Varies | Private content secured |
| 7 | Check noindex status | 5 minutes | Confirmed result |
| 8 | Get quick answers | 3 minutes | Edge cases covered |
Step 1: Know That Google Doesn't Support Noindex in robots.txt
Google does not support noindex in robots.txt. In a July 2, 2019 post, A note on unsupported rules in robots.txt, Google said it would retire code handling unsupported and unpublished rules, including noindex, nofollow and crawl-delay, effective September 1, 2019.
- Why Google dropped it: The rule was never part of the documented standard, and Google found it was rarely used and usually contradicted by other rules.
- What happens now: A line like "Noindex: /old-page/" in robots.txt has no effect on Google.
- The standard: The Robots Exclusion Protocol, published as RFC 9309, defines User-agent, Allow and Disallow, not noindex.
How to find old noindex rules
Open yoursite.com/robots.txt in a browser and search for "noindex", "nofollow" and "crawl-delay". These lines often date from an older SEO setup or plugin. Note which URLs each one was meant to cover, because those pages may now need a meta tag or header instead.
Alternatives Google listed
In the same announcement, Google pointed site owners to noindex in robots meta tags, 404 and 410 status codes, password protection, Disallow in robots.txt (for crawling, not indexing), and the Remove URLs tool in Search Console.
Key Takeaway: Delete any Noindex lines from your robots.txt so they don't give anyone on your team a false sense of security. Test the file with the robots.txt checker.
Step 2: Understand Why Disallow Alone Doesn't Keep a Page Out of Search
Disallow stops crawling, not indexing. If other sites or pages link to a URL blocked in robots.txt, Google can still index the URL, usually showing it without a description, because it knows the page exists but can't read it.
| Method | Stops crawling? | Keeps page out of results? |
|---|---|---|
| robots.txt Disallow | Yes, for compliant crawlers | No, the URL can still be indexed |
| Meta robots noindex | No | Yes, once crawled |
| X-Robots-Tag noindex | No | Yes, once crawled |
| Password protection | Yes | Yes |
| 404 or 410 status | Not applicable | Yes, after recrawl |
- Google's guidance: Google's robots.txt introduction notes that robots.txt isn't a mechanism for keeping a page out of Google.
- Where Disallow helps: Saving crawl activity on filters, internal search or duplicate URLs.
For more on blocking whole sites, see robots.txt Disallow all vs Allow all.
Step 3: Fix the Common Mistake of Disallow Plus a Noindex Tag
The most common mistake is adding a noindex tag to a page and also blocking it in robots.txt. Google can't crawl the blocked page, so it never sees the noindex tag, and the URL can stay indexed.
How to fix it
- Remove the Disallow rule: Let crawlers fetch the page so they can read the noindex.
- Keep the noindex tag: Leave it on the page.
- Wait for recrawl: Google needs to revisit the page; you can request this in Search Console's URL Inspection tool.
- Optionally block later: Once the page has dropped out of results, you may add a Disallow rule if you need to save crawl activity, knowing new links could resurface the URL.
Pages where this mistake shows up most
- Thank-you and confirmation pages: Often blocked and noindexed at the same time.
- Internal search results: Blocked to save crawling, then noindexed when they appear in Google.
- Staging copies: Blocked in robots.txt but linked from somewhere public.
- Old campaign landing pages: Hidden from navigation and blocked, but still linked from emails or ads.
Example of the wrong and right setup
| Setup | robots.txt | Page | Result |
|---|---|---|---|
| Wrong | Disallow: /thank-you/ | noindex tag | Tag unseen; URL may stay indexed |
| Right | No rule for /thank-you/ | noindex tag | Page crawled and dropped from results |
If you want Google to forget a page, you have to let Google visit it one more time.
Step 4: Option 1, Add a Meta Robots Noindex Tag
For HTML pages, the standard way to keep a page out of search is a robots meta tag in the page's head. According to Google's guide to blocking indexing, Google drops a page from results after it crawls the page and sees the noindex rule.
<meta name="robots" content="noindex">- All crawlers: name="robots" applies to every compliant search engine.
- Google only: name="googlebot" targets just Google's crawler.
- Combine rules: content="noindex, nofollow" also asks crawlers not to follow links on the page.
- Placement: The tag must be in the <head> of the HTML.
Meta robots values you might need
| Value | What it does | Typical use |
|---|---|---|
| noindex | Keeps the page out of results | Thank-you pages, internal search results |
| nofollow | Asks crawlers not to follow the page's links | Pages with untrusted user links |
| noindex, follow | Keeps the page out but lets links be followed | Tag or archive pages |
| none | Equivalent to noindex, nofollow for Google | Pages you want fully excluded |
Adding it on common platforms
| Platform | How to add noindex |
|---|---|
| WordPress | Use an SEO plugin's per-page "no index" setting, such as in Yoast or Rank Math |
| Shopify | Add a conditional robots meta tag in the theme's theme.liquid head |
| Custom sites | Add the tag to the page template's head |
Step 5: Option 2, Use the X-Robots-Tag HTTP Header
For PDFs, images, videos and other non-HTML files, you can't add a meta tag, so use the X-Robots-Tag HTTP header instead. Google's robots meta tag and X-Robots-Tag documentation explains that the header supports the same rules as the meta tag.
X-Robots-Tag: noindexServer examples
- Apache (.htaccess): Use a FilesMatch block for PDFs, for example <FilesMatch "\.pdf$"> Header set X-Robots-Tag "noindex" </FilesMatch>.
- Nginx: Inside a location block for PDFs, add: add_header X-Robots-Tag "noindex";
- CDNs and hosts: Many let you set response headers by path in their dashboard.
How to confirm the header is sent
Run curl -I https://example.com/file.pdf in a terminal and look for an X-Robots-Tag: noindex line in the response headers. Browser developer tools also show response headers under the Network tab. If the header is missing, check that your rule matches the file path and that a CDN isn't stripping it.
When to use the header for HTML too
The header also works for HTML pages, which is useful when you can't edit templates but can set server rules, or when you want to apply noindex to a whole directory at once.
Step 6: Option 3, Use Password Protection or Remove the Page
If content is private or shouldn't exist, noindex isn't enough. Put it behind a login, or delete it and return a 404 or 410 status so search engines drop it.
| Situation | Best option | Why |
|---|---|---|
| Staging or test site | Password protection | Stops crawling and accidental exposure |
| Customer or internal documents | Authentication | noindex doesn't stop people opening a known URL |
| Outdated page with no replacement | 404 or 410 | Search engines remove it after recrawling |
| Outdated page with a replacement | 301 redirect | Passes visitors and signals to the new URL |
| Urgent removal from Google | Search Console Removals tool | Temporarily hides the URL while you fix it properly |
- Temporary only: The Removals tool hides URLs for a limited time; pair it with noindex, deletion or protection.
- Security first: Neither robots.txt nor noindex secures content.
Choosing between 404 and 410
Both tell search engines the page is gone. A 410 says the removal is intentional and permanent, while a 404 simply means not found. In practice Google treats them similarly over time, so use whichever your platform supports, and make sure the page isn't redirecting to the homepage instead.
Step 7: Check a Page's Noindex Status
After making changes, confirm the noindex is in place and that Google has seen it. Check both the page itself and how Google currently treats it.
- Check the page: The free noindex checker shows whether a URL carries a noindex in its meta tags or X-Robots-Tag header.
- Check robots.txt: Make sure the page isn't disallowed, using the robots.txt checker.
- Check Google's index: The Google index checker shows whether the URL still appears.
- Use URL Inspection: In Search Console, inspect the URL to see whether Google found the noindex and request a recrawl.
Reading the results
| What you see | What it means | Next step |
|---|---|---|
| noindex found, page allowed in robots.txt | Correct setup | Wait for Google to recrawl |
| noindex found, page disallowed | Google can't see the tag | Remove the Disallow rule |
| No noindex found | Tag or header missing or misplaced | Check the head or server config |
| Still indexed after weeks | Not yet recrawled | Request indexing via URL Inspection |
Noindex also affects AI search features that rely on the search index, so hyperank users often check that pages they want AI engines to cite aren't accidentally noindexed. hyperank shows how ChatGPT, Perplexity and Gemini describe your brand, which helps you spot when key pages aren't coming through.
Audit the whole site, not just one URL
Noindex mistakes usually come from templates, so one wrong setting can affect hundreds of pages. Crawl the site with an SEO crawler, filter for pages with noindex, and check the list against pages you actually want hidden. In Search Console, the Pages report lists URLs "Excluded by 'noindex' tag" and "Indexed, though blocked by robots.txt," which is a quick way to spot both problems.
Step 8: Get Quick Answers
These short answers cover the questions we hear most about robots.txt and noindex.
- What about Bing? Bing documents meta robots and X-Robots-Tag for noindex, so use those for Bing too.
- Can I noindex a whole folder? Yes, with an X-Robots-Tag header rule for that path, or a template-level meta tag.
- How long does deindexing take? It depends on how often Google recrawls the page, from days to weeks.
What to Do After You Fix It
- Phase 1, today: Remove Noindex lines from robots.txt and unblock any pages that carry a noindex tag.
- Phase 2, this week: Add meta robots or X-Robots-Tag noindex where needed and request recrawls in Search Console.
- Phase 3, ongoing: Review robots.txt and noindex settings after every site launch, migration or template change.
Resources You'll Need
| Resource | What it's for | Label |
|---|---|---|
| hyperank noindex checker | Confirming noindex on a URL | Recommended |
| hyperank robots.txt checker | Making sure pages aren't disallowed | Recommended |
| Google Search Console | URL Inspection and Removals | Required |
| Access to page templates or server config | Adding tags or headers | Required |
| hyperank | Seeing how AI engines describe your brand | Optional |
Troubleshooting Common Issues
| Problem | Cause | Fix |
|---|---|---|
| Blocked page still in Google | Disallow prevents Google seeing noindex | Allow crawling and keep the noindex tag |
| "Indexed, though blocked by robots.txt" | Links point to a disallowed URL | Use noindex instead of Disallow |
| PDF still appears in results | No way to add a meta tag | Send an X-Robots-Tag: noindex header |
| Noindex added by mistake | Plugin or template setting | Remove it and request indexing |
Conclusion
Noindex in robots.txt hasn't worked on Google since September 1, 2019, and Disallow alone doesn't keep pages out of search. Use a meta robots noindex tag for HTML, an X-Robots-Tag header for files, and password protection or removal for private or obsolete pages, then check the result.
- No robots.txt noindex: Google ignores it.
- Disallow isn't deindexing: Blocked URLs can still be indexed.
- Don't combine: Disallow hides your noindex tag from Google.
- Right tools: Meta robots, X-Robots-Tag, authentication or 404/410.
- Verify: Use a noindex checker and URL Inspection.
Check any page's indexing directives with the free noindex checker.
FAQ
Does noindex in robots.txt still work, and what should you use instead?
No. Google stopped supporting noindex in robots.txt on September 1, 2019. Use a meta robots noindex tag for HTML pages, an X-Robots-Tag header for PDFs and other files, or password protection or a 404/410 for private or removed content, and don't block those pages in robots.txt.
Can I use noindex in robots.txt?
You can write it, but Google ignores it. Use a meta tag or HTTP header instead.
Does Disallow remove a page from Google?
No. Disallow stops crawling, but the URL can still be indexed if other pages link to it.
What's the difference between noindex and Disallow?
Disallow controls whether crawlers fetch a URL; noindex controls whether a page appears in results.
How do I noindex a PDF?
Send an X-Robots-Tag: noindex HTTP header for the file, set in your server or CDN configuration.
How do I check if a page is noindexed?
Use a noindex checker to read the page's meta tags and headers, and Search Console's URL Inspection to see what Google detected.
Methodology: This guide draws on Google Search Central's 2019 announcement on unsupported robots.txt rules, Google's documentation on blocking indexing, robots meta tags and the X-Robots-Tag header, and RFC 9309. Server configuration examples are illustrative; confirm syntax for your server version.
See where your brand stands
How six AI engines describe you, next to your competitors. About two minutes.