How do I create a sitemap.xml for my website for free?+
Enter your website's URLs (one per line) or provide your domain. The tool generates a properly formatted XML sitemap file you can download and upload to your server at yourdomain.com/sitemap.xml.
What is an XML sitemap and why do I need one?+
An XML sitemap is a file that lists all your website's important pages, helping search engines find and index them faster. Websites with sitemaps submitted to Google Search Console typically get crawled more thoroughly.
How do I submit my sitemap to Google?+
After generating your sitemap, upload it to your root domain (yourdomain.com/sitemap.xml). Then go to Google Search Console → Sitemaps → Enter your sitemap URL → Submit.
What should I put in my robots.txt file?+
Block pages you don't want indexed: admin panels (/admin/), login pages, API endpoints (/api/), duplicate content, and search result pages. Always allow: /, /sitemap.xml, and all public content pages.
How often should I update my sitemap?+
Update your sitemap whenever you add or remove pages. For active blogs or ecommerce sites, set lastmod dates and regenerate monthly. Many CMS platforms (WordPress, Shopify) do this automatically.
What is crawl budget and how does a sitemap improve it?+
Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. Submitting a clean, accurate sitemap signals which pages are most important, helping Google prioritize crawling your best content instead of wasting resources on low-value or duplicate pages.
How do I block ChatGPT, Claude and other AI bots but keep Google?+
Turn on 'Block AI / LLM training bots' and tick exactly which crawlers to block, GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, Bytespider and more. Because those rules target named user-agents, Googlebot and Bingbot stay allowed under your main 'User-agent: *' block. You can also switch on 'Keep search engines allowed' to permit them explicitly, even alongside a full Disallow: /.
My site has more than 50,000 URLs, what do I do?+
A single sitemap file is capped at 50,000 URLs (and 50MB uncompressed). Split your pages across several sitemap files, then use the Sitemap index mode to generate a sitemapindex that lists each child sitemap. Submit only the index to Google Search Console and it will fetch the rest.
How do I create a sitemap.xml and robots.txt for my site?+
Enter your URLs (or site details) to generate a valid sitemap.xml, and configure allow/disallow rules to produce a robots.txt that references your sitemap. Copy both files to your site root.
How do I allow or block AI crawlers in robots.txt?+
Add user-agent rules for bots like GPTBot, ClaudeBot, PerplexityBot and Google-Extended, then Allow or Disallow as you prefer. The generator can include these AI-crawler directives alongside standard rules.
What should a robots.txt file include?+
At minimum, a User-agent line, any Disallow/Allow rules, and a Sitemap reference pointing to your sitemap.xml. The generator builds a valid file with these and lets you add custom rules.
What is crawl budget?+
It's how many pages a search engine will crawl on your site in a given period. A clean sitemap and robots.txt help it spend that budget on your important pages.
