Sitemap & Robots.txt Generator

Generate XML sitemaps and robots.txt files for your website. Build crawl rules, block specific AI/LLM bots while keeping search engines allowed, combine sitemaps into an index, audit a live sitemap, and export files ready for Google Search Console, instantly.

Built and maintained by Shehzad Munir ·

Needs the internet: the sitemap is fetched on our server. Nothing is stored after the request.

Sitemap & Robots.txt Generator
Loading Sitemap & Robots Engine
  • SEO Tools
  • Free · No sign-up
  • Uses the internet
  • 9 features
Share
Cite or link to this tool

Using this tool in an article, lesson or report? Copy a ready-made link or citation.

HTML link

<a href="https://www.toolnova.org/seo-tools/sitemap-robots-engine">Sitemap &amp; Robots.txt Generator</a> by ToolNova

APA

ToolNova. (2026, September 5). Sitemap & Robots.txt Generator [Online tool]. https://www.toolnova.org/seo-tools/sitemap-robots-engine

MLA

“Sitemap & Robots.txt Generator.” ToolNova, 5 Sept. 2026, www.toolnova.org/seo-tools/sitemap-robots-engine.

Generate clean sitemaps and robots.txt

A three-in-one crawl-control toolkit: build a valid XML sitemap (or a sitemap index) from a URL list, craft a matching robots.txt with allow/disallow rules and per-bot AI blocking, and audit any live sitemap, all in your browser, free and with no signup. Presets for WordPress, Shopify and static sites, plus built-in validation warnings, keep your crawl directives consistent and Search Console-ready.

This engine builds a valid XML sitemap and a matching robots.txt so search engines can discover and crawl your pages efficiently. Correct crawl directives are one of the most overlooked ranking foundations, a single misplaced Disallow can hide an entire section of your site from Google. The sitemap builder adds changefreq, priority and today's lastmod automatically, and can even weight priority by URL depth so your homepage and top sections outrank deep pages.

Features

  • XML sitemap generator with changefreq, priority & auto lastmod
  • Sitemap index mode for large sites (past the 50,000-URL limit)
  • Robots.txt builder with allow/disallow rules & crawl-delay
  • Presets: allow all, block all, WordPress, Shopify, static / Next.js, block AI
  • Per-bot AI blocking, tick GPTBot, ClaudeBot, Google-Extended & more
  • 'Keep search engines allowed' to permit Google & Bing while blocking the rest
  • Live validation warnings for common crawl mistakes
  • Live sitemap audit with sitemap-index support
  • Search Console-ready sitemap.xml & robots.txt exports

How to use Sitemap & Robots.txt Generator

  1. 1

    Build your robots.txt

    Start from a preset (WordPress, Shopify, static) or set your own allow/disallow paths, then tick the AI bots to block and optionally keep search engines allowed.

  2. 2

    Generate your sitemap

    Paste your page URLs to build a sitemap.xml with changefreq, priority and lastmod, or switch to index mode to combine several child sitemaps.

  3. 3

    Check the warnings

    Review the built-in validation notes to catch a de-indexing Disallow, an ignored crawl-delay, or a missing sitemap line before you ship.

  4. 4

    Audit it live

    Paste a live sitemap URL into the Audit tab to see entry counts, HTTPS status, and drill into a sitemap index.

  5. 5

    Export & submit

    Download robots.txt and sitemap.xml, upload them to your domain root, and submit the sitemap in Google Search Console.

Common use cases

  • Create a sitemap.xml from a list of page URLs
  • Block ChatGPT, Claude and other AI bots while keeping Google allowed
  • Generate a robots.txt for WordPress, Shopify or a Next.js site
  • Combine multiple sitemaps into one sitemap index for a large site
  • Audit and validate a live sitemap before submitting to Search Console
  • Fix 'Discovered: not indexed' by keeping robots, sitemap and host consistent

Detailed guide

Precise control over AI crawlers and search engines

AI training and scraping bots, GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider and more, are listed individually so you can tick exactly which to block instead of an all-or-nothing switch. Turn on 'Keep search engines allowed' to explicitly permit Googlebot, Bingbot and social crawlers even while you Disallow: / for everyone else, the classic 'block everything except search' recipe. One-click presets cover WordPress, Shopify and static/Next.js projects.

Build a sitemap index and avoid the 50,000-URL wall

A single sitemap file may hold at most 50,000 URLs, so large sites split their pages across several files and tie them together with a sitemap index. Switch the builder to index mode to generate that `<sitemapindex>` from your child-sitemap URLs, and the tool warns you the moment a URL list crosses the limit.

Avoid the crawl mistakes that block indexing

Common failures include declaring the wrong canonical host, blocking assets that pages need to render, or listing URLs in the sitemap that redirect elsewhere. Live validation warnings flag a full-site Disallow, a Crawl-delay that Googlebot ignores, a missing sitemap line, or non-absolute sitemap URLs, exactly the mistakes that cause 'Discovered, not indexed' and 'Page with redirect' issues. The live audit tab then parses any sitemap or sitemap index so you can confirm the final result.

Frequently asked questions

How do I create a sitemap.xml for my website for free?

Enter your website's URLs (one per line) or provide your domain. The tool generates a properly formatted XML sitemap file you can download and upload to your server at yourdomain.com/sitemap.xml.

What is an XML sitemap and why do I need one?

An XML sitemap is a file that lists all your website's important pages, helping search engines find and index them faster. Websites with sitemaps submitted to Google Search Console typically get crawled more thoroughly.

How do I submit my sitemap to Google?

After generating your sitemap, upload it to your root domain (yourdomain.com/sitemap.xml). Then go to Google Search Console → Sitemaps → Enter your sitemap URL → Submit.

What should I put in my robots.txt file?

Block pages you don't want indexed: admin panels (/admin/), login pages, API endpoints (/api/), duplicate content, and search result pages. Always allow: /, /sitemap.xml, and all public content pages.

How often should I update my sitemap?

Update your sitemap whenever you add or remove pages. For active blogs or ecommerce sites, set lastmod dates and regenerate monthly. Many CMS platforms (WordPress, Shopify) do this automatically.

What is crawl budget and how does a sitemap improve it?

Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. Submitting a clean, accurate sitemap signals which pages are most important, helping Google prioritize crawling your best content instead of wasting resources on low-value or duplicate pages.

How do I block ChatGPT, Claude and other AI bots but keep Google?

Turn on 'Block AI / LLM training bots' and tick exactly which crawlers to block, GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, Bytespider and more. Because those rules target named user-agents, Googlebot and Bingbot stay allowed under your main 'User-agent: *' block. You can also switch on 'Keep search engines allowed' to permit them explicitly, even alongside a full Disallow: /.

My site has more than 50,000 URLs, what do I do?

A single sitemap file is capped at 50,000 URLs (and 50MB uncompressed). Split your pages across several sitemap files, then use the Sitemap index mode to generate a sitemapindex that lists each child sitemap. Submit only the index to Google Search Console and it will fetch the rest.

How do I create a sitemap.xml and robots.txt for my site?

Enter your URLs (or site details) to generate a valid sitemap.xml, and configure allow/disallow rules to produce a robots.txt that references your sitemap. Copy both files to your site root.

How do I allow or block AI crawlers in robots.txt?

Add user-agent rules for bots like GPTBot, ClaudeBot, PerplexityBot and Google-Extended, then Allow or Disallow as you prefer. The generator can include these AI-crawler directives alongside standard rules.

What should a robots.txt file include?

At minimum, a User-agent line, any Disallow/Allow rules, and a Sitemap reference pointing to your sitemap.xml. The generator builds a valid file with these and lets you add custom rules.

What is crawl budget?

It's how many pages a search engine will crawl on your site in a given period. A clean sitemap and robots.txt help it spend that budget on your important pages.

From the journal

All articles →
How to Remove EXIF Metadata From Your PhotosTool Guides

How to Remove EXIF Metadata From Your Photos

Photos carry hidden data like GPS location and camera details. Here's why that matters for privacy and how to strip it before sharing.

SM
Shehzad Munir
Aug 19, 2026 · 3 min
What Is CPS? How to Test and Improve Your Click SpeedTool Guides

What Is CPS? How to Test and Improve Your Click Speed

CPS means clicks per second. Here's how it's measured, what a good score looks like, and how to test your own click speed for free.

SM
Shehzad Munir
Aug 17, 2026 · 3 min
How to Merge PDF Files for Free (Without Uploading Them)Tool Guides

How to Merge PDF Files for Free (Without Uploading Them)

Combine several PDFs into one document, reorder the pages, and keep everything private — no software and no server upload.

SM
Shehzad Munir
Aug 14, 2026 · 2 min