WordPress robots.txt: Where It Is, How to Edit It, and What to Put in It

robots.txt wordpress | Updated October 2026 | hyperank Editorial Team | 15-30 minutes | Beginner

What You'll Learn

Your robots.txt in WordPress lives at yoursite.com/robots.txt. By default WordPress generates it on the fly as a "virtual" file, so you won't find it in your folders until you or a plugin creates a real one. You can edit it with Yoast SEO (Tools, then File editor), Rank Math (General Settings, then Edit robots.txt) or by uploading a file via FTP. This guide shows each method, a safe file to copy, and how to test it.

• Find your robots.txt and tell whether it's virtual or a real file
• Edit it safely with Yoast, Rank Math or FTP
• Copy a conservative robots.txt for WordPress, including AI crawler rules
• Avoid the blocks that break how Google renders your site

Prerequisites: admin access to WordPress, and FTP or hosting file manager access if you don't use an SEO plugin.

Why Your WordPress robots.txt Matters in 2026

robots.txt is a plain-text file that tells crawlers which parts of your site they may request. It doesn't hide pages from Google's index on its own, but a wrong line can stop search engines and AI crawlers from reading pages you want found. Google explains the rules in its robots.txt introduction.

The file matters more now because AI companies run their own crawlers, such as OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot and PerplexityBot. Your robots.txt is where you decide which of them can read your content.

Key Takeaway: A correct robots.txt protects your admin area and decides which search and AI crawlers can read your site; a wrong one can quietly hide you.

The Process at a Glance

Total time: 15-30 minutes.

Step 1: Find Where robots.txt Is in WordPress

What You're Doing

Most people go looking for robots.txt in their folders and find nothing. That's normal: first, check whether your site serves a virtual file generated by WordPress or a real file in the root folder, because that decides how you edit it.

How to Do It

1. Visit yourdomain.com/robots.txt in a browser to see what crawlers see.
2. Open your hosting file manager or FTP and look in the site's root folder (usually public_html) for a file named robots.txt.
3. If the URL shows content but no file exists, WordPress (or an SEO plugin) is generating a virtual file.

What Done Looks Like

You know whether your robots.txt is virtual or a physical file.

Step 2: Understand the WordPress Default robots.txt

What You're Doing

Now that you know where the file comes from, read what's already in it, so you don't delete something important.

How to Do It

WordPress's do_robots() function outputs this by default (paths depend on your install):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Since WordPress 5.5, the built-in sitemap feature usually adds a Sitemap: line too. And since 5.3, ticking "Discourage search engines" no longer adds Disallow: /; WordPress uses a noindex robots meta tag instead.

What Done Looks Like

You can explain each line in your current file.

Step 3: Edit robots.txt in Yoast SEO

What You're Doing

If Yoast is installed, you can edit the file without ever touching FTP. Its built-in file editor is the easiest route.

How to Do It

1. In WordPress, go to Yoast SEO, then Tools, then File editor, as described in Yoast's help guide.
2. Click Create robots.txt file if none exists, or edit the existing one.
3. Make your changes and save. Then open yourdomain.com/robots.txt in a private window to see the live version.

Common Mistakes

Not seeing the File editor at all. Yoast hides it when file editing is disabled in WordPress or the server won't allow writing the file; use FTP instead (Step 5).

What Done Looks Like

Your changes appear at yourdomain.com/robots.txt.

Step 4: Edit robots.txt With Other SEO Plugins

What You're Doing

Not on Yoast? Rank Math and similar plugins manage their own virtual file, with one catch worth knowing.

How to Do It

1. In Rank Math, switch to Advanced Mode (top right of the Rank Math dashboard).
2. Go to General Settings, then Edit robots.txt, per Rank Math's guide.
3. If a physical robots.txt exists in your root folder, delete it first; a real file overrides the plugin's virtual one.

What Done Looks Like

Your plugin edits show up live, with no physical file overriding them.

Step 5: Edit robots.txt via FTP or Your Host's File Manager

What You're Doing

No SEO plugin, or a plugin that won't cooperate? A physical file works on any WordPress site.

How to Do It

1. Create a plain-text file named exactly robots.txt (lowercase).
2. Upload it to your site's root folder with FTP (for example FileZilla) or your host's file manager.
3. Make sure it's at the root, so it loads at yourdomain.com/robots.txt, not in a subfolder.

What Done Looks Like

The physical file loads at the root URL and replaces the virtual one.

Step 6: Use the Best robots.txt for WordPress

What You're Doing

With editing sorted, here's what to actually put in the file: a conservative version that keeps the defaults, points to your sitemap and states your AI crawler choices.

How to Do It

Copy this and change the sitemap URL to yours:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# AI search crawlers: allowed so you can be cited in AI answers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yourdomain.com/wp-sitemap.xml

If you want to opt out of AI model training while staying visible in AI search, add separate blocks for training crawlers such as GPTBot, ClaudeBot and Google-Extended with Disallow: /. OpenAI documents its crawlers on its bots page. If you use Yoast or Rank Math sitemaps, use their sitemap URL instead, such as /sitemap_index.xml.

hyperank's free sitemap and robots.txt generator can build a file for you.

What Done Looks Like

Your robots.txt keeps the defaults, lists your sitemap and reflects a deliberate AI crawler decision.

Step 7: Know What Not to Block in WordPress

What You're Doing

Just as important is what you leave out. A few well-meant blocks can stop Google from rendering your pages properly.

How to Do It

• Don't block /wp-content/: It holds your theme's CSS and JavaScript and your uploads. Google needs them to render pages and index images.
• Don't block /wp-includes/: Core scripts live there.
• Don't use robots.txt to hide pages: A blocked page can still appear in results. Use a noindex tag instead.

What Done Looks Like

Your file only blocks the admin area and anything you've deliberately chosen to exclude.

Step 8: Test Your WordPress robots.txt

What You're Doing

One typo can block a whole site, so don't skip this. Confirm the live file is valid and doesn't block anything you need crawled.

How to Do It

1. Paste your URL into hyperank's free robots.txt checker to catch syntax errors and risky rules.
2. Run the AI bot crawler checker to see which AI crawlers can reach your site.
3. In Google Search Console, open the robots.txt report under Settings to confirm Google fetched the latest version.

Being crawlable is the first step to being cited, and hyperank helps with what comes next. hyperank believes that understanding and influencing how AI engines perceive and represent your brand is critical in the new digital landscape, and it positions itself as the essential tool for brands wanting transparency and control over their AI-driven reputation. hyperank helps brands 'see what AI says about your brand' by aggregating and displaying daily insights from six major AI engines, capturing qualitative answers and trends, not just rankings.

What Done Looks Like

Your robots.txt passes the checks, and Search Console shows the current version.

What to Do After You Fix robots.txt

• Phase 1 (today): Submit your sitemap. Add it in Search Console and make sure the robots.txt Sitemap line matches.
• Phase 2 (this month): Watch crawl stats. Check Search Console's crawl stats for errors after any change.
• Phase 3 (quarterly): Review AI crawler rules. New AI crawlers appear regularly; revisit your choices.

Resources You'll Need

Troubleshooting Common Issues

My edits don't show up at /robots.txt

Likely cause: A physical file is overriding your plugin's virtual one, or a cache is serving the old version. Fix: Delete or update the physical file and clear your site and CDN cache.

Yoast's File editor is missing

Likely cause: File editing is disabled in WordPress. Fix: Edit via FTP or your host's file manager instead.

Pages show "Indexed, though blocked by robots.txt"

Likely cause: You blocked pages that other sites link to. Fix: Remove the block and add a noindex tag if you want them out of results.

Google can't render my pages properly

Likely cause: CSS or JavaScript in /wp-content/ is blocked. Fix: Remove the block and retest with URL Inspection.

Conclusion

Key Takeaways

• Your robots.txt for WordPress is at /robots.txt and is virtual until you or a plugin create a real file.
• Keep the defaults, add your sitemap, decide on AI crawlers, and never block /wp-content/.
• Test the live file now with the free robots.txt checker.

FAQ

Where is robots.txt in WordPress, how do you edit it, and what should you put in it?

WordPress serves robots.txt at yourdomain.com/robots.txt, usually as a virtual file it generates itself. Edit it with Yoast SEO (Tools, then File editor), Rank Math (General Settings, then Edit robots.txt, in Advanced Mode) or by uploading a real file to your root folder via FTP. Keep the default rules for /wp-admin/, add your sitemap, decide which AI crawlers to allow, and don't block /wp-content/.

What does this guide cover?

It covers finding the file, the WordPress default, editing with Yoast, Rank Math or FTP, a recommended file with AI crawler rules, what not to block, and testing.

Where is the robots.txt file in WordPress?

There's often no physical file: WordPress generates it virtually. If one exists, it's in your site's root folder, usually public_html.

Where is robots.txt in Yoast?

In WordPress, go to Yoast SEO, then Tools, then File editor. If you don't see it, file editing is disabled on your site.

Should I block AI crawlers in robots.txt?

Only if you mean to. Blocking AI search crawlers such as OAI-SearchBot and PerplexityBot can remove you from AI answers; blocking training crawlers like GPTBot affects model training only.

Can robots.txt remove a page from Google?

No. It stops crawling, not indexing. Use a noindex tag to keep a page out of search results.

Why does my robots.txt show old rules?

A physical file or a cache is usually overriding your edits. Update or delete the file and clear caches.

Methodology: Steps were checked against WordPress developer documentation, Yoast and Rank Math help pages, and Google's robots.txt documentation in October 2026. Plugin menus change between versions; check your plugin's current docs if a menu looks different.