ProgrammersOSProgrammersOS
Sign In

robots.txt Generator

Build a valid robots.txt with per-crawler rules, allow and disallow paths, crawl delay and sitemap links.

Use * for every crawler, or a name such as Googlebot.

Ignored by Google; honoured by some other crawlers.

User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

How to use

1

Enter the crawler name for the first group, or * for every crawler.

2

List the paths to block under Disallow and any exceptions under Allow, one per line.

3

Add more groups for other crawlers, then list your sitemap URLs.

4

Copy the result and save it as robots.txt at the root of your site.

Example

Block admin and API paths for everyone

Each path becomes its own Disallow line; the sitemap goes at the end of the file.

Input
User agent: *
Disallow: /admin/, /api/
Sitemap: https://example.com/sitemap.xml
Output
User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Frequently Asked Questions

No. It asks well-behaved crawlers not to fetch a path, but the page stays reachable and its address stays visible in the file. Use authentication to protect anything sensitive.

Capabilities

  • Several user-agent groups
  • Allow and Disallow paths
  • Crawl-delay between 0 and 120 seconds
  • Sitemap lines checked as absolute URLs
  • Copy the finished file

Related Tools

Open Graph & Social Meta Tag Generator

Generate title, description, Open Graph and Twitter meta tags with a search preview.

URL Parser

Break a URL into protocol, host, path, query parameters and hash.

Content Security Policy (CSP) Builder

Configure Content-Security-Policy headers protecting against XSS and injection attacks.

HTTP Status Code Reference & Debugger

Complete RFC-compliant catalog of HTTP status codes (1xx-5xx) with client/server debug tips.

Explore

Connected career roadmaps, algorithmic problems, and real-world curricula related to robots.txt Generator.

Career & Skill Roadmaps

Clear, milestone-driven paths from first fundamentals to your target engineering role.

🗺️ Career Tracks
Frontend Developer Roadmap

Frontend Engineer / Senior UI Engineer · 6-9 months

Career Path
Full-Stack Developer Roadmap

Full-Stack Software Engineer · 8-12 months

Career Path
Flutter & Mobile Developer Roadmap

Flutter Mobile Engineer · 6-9 months

Career Path