Free tool
Sitemap to Robots.txt Generator
Turn a sitemap into the robots.txt that points crawlers at it.
How it works
What is the Sitemap to Robots.txt Generator?
The Sitemap to Robots.txt Generator is a free tool that writes a robots.txt file declaring your sitemaps with Sitemap: lines. You can add Allow and Disallow paths and a Crawl-delay for one user agent. The result is the finished file, ready to copy and upload to the root of your site.
How to use the Sitemap to Robots.txt Generator
- Paste your sitemap URLs into Sitemap URLs, one per line. Use full URLs, such as https://example.com/sitemap.xml, because a
Sitemap:line needs the complete address. - Leave User agent as
*to write rules for every crawler, or enter one crawler's name, such as Googlebot. - Add any paths to Allow or Disallow, one per line, and set Crawl delay in seconds if you need it. Zero leaves the line out.
- Press Generate robots.txt, then press Copy and save the text as robots.txt at the root of your domain.
How do I add a sitemap to robots.txt?
Add a line that starts Sitemap: followed by the sitemap's full URL, such as Sitemap: https://example.com/sitemap.xml. The line can sit anywhere in the file, applies to every crawler whatever user-agent group is near it, and can be repeated once per sitemap.
A relative path such as Sitemap: /sitemap.xml does not work for Google. Use the full URL, including https://.
Does Disallow in robots.txt stop a page from being indexed?
No. Disallow stops a crawler fetching a URL, but the URL can still be indexed if other pages link to it. It then appears in results as a bare address with no description.
To keep a page out of search, allow crawling and add a noindex tag. Blocking the page as well defeats the noindex, because a crawler that cannot fetch the page never sees it.
Does Google support Crawl-delay?
No. Google ignores Crawl-delay. Bing supports it. The generator leaves the line out when you set zero, which is right for most sites.
Each run writes the group for one user agent. A crawler follows only the most specific group that matches its name, so a group for Googlebot replaces the * group for Googlebot rather than adding to it. To set rules for several crawlers, generate each group and join them in one file.
Questions
- Should I add my sitemap to robots.txt?
- Yes. Adding your sitemap to robots.txt lets any crawler that reads the file find it, including search engines where you have no webmaster account. It costs one line and works alongside a Search Console submission.
- Can robots.txt list multiple sitemaps?
- Yes. robots.txt can list as many sitemaps as you need, with one
Sitemap:line each. You can also list a single sitemap index that points at the rest. - Can the sitemap URL in robots.txt be relative?
- No. Google requires a full URL in a
Sitemap:line, such ashttps://example.com/sitemap.xml. A relative path such as /sitemap.xml may be ignored. - Can robots.txt point to a sitemap on another domain?
- Yes. A
Sitemap:line in robots.txt can point to a sitemap hosted on another domain, and search engines take it as the site owner's approval. This is how cross-site sitemap submission works. - What is the difference between robots.txt and a sitemap?
- robots.txt tells crawlers which URLs they may not fetch. A sitemap lists the URLs you want them to find and index. They work together: the
Sitemap:line in robots.txt tells crawlers where the sitemap is. - Can I block AI crawlers with robots.txt?
- Yes. GPTBot, ClaudeBot and PerplexityBot each use a named user agent, and their operators say they follow robots.txt. Enter one as the User agent here with
/under Disallow to block it. Blocking also keeps your pages out of the answers those tools give.
Sitemap Tools
More in Sitemap Tools
Sitemap Index Generator
Combine several sitemaps into the one index file search engines expect.
Generate indexReady when you are
Before your next buyer asks AI about you, ask AI about yourself
One check, four engines, about twenty seconds. We show you the answers before we ask for anything.