Cloudflare robots.txt Generator

Generate robots.txt for Cloudflare Pages sites, custom domains, pages.dev staging blocking, sitemap URLs, and private path rules.

Output:A ready-to-use configuration file for Cloudflare robots.txt with best practices applied.

What framework are you using?

ProductionStaging (Block All)

Production Domain

Sitemap URL

Disallow Rules

Disallow API paths (/api/)Disallow Private paths (/admin/, /dashboard/)Block AI Crawlers (GPTBot, CCBot)

Custom paths to disallow (comma separated)

File Location

Where to place your robots.txt

Source Repository

public/robots.txt

After Build

out/robots.txt

•Next.js server-side redirects in next.config.ts do not work for static export on Cloudflare Pages.
•Use output: 'export' in your Next.js config.
•Place robots.txt in the public directory before building.
•Cloudflare Pages should deploy the 'out' directory.

Cloudflare pages.dev domainsA robots.txt on your custom domain does not apply to your pages.dev domain. To fix duplicate content on pages.dev, you must either deploy a robots.txt specifically for that domain, or redirect the domain. See our Staging Domain Blocker.

User-agent: *
Allow: /

Quick Summary

Use the Cloudflare robots.txt Generator to create a file that tells search engines like Google which parts of your Cloudflare Pages site to crawl, where your sitemap is, and what private paths to ignore.

What is this tool?

A robots.txt file is a standard way to communicate with web crawlers. When deploying to Cloudflare Pages, ensuring this file is properly formatted and placed in the root of your domain is critical for technical SEO.

How to Use This Tool

Select your framework (Next.js, Astro, React, etc.).
Enter your production domain and sitemap URL.
Add any specific paths you want to disallow (like /api/ or /admin/).
Optionally block aggressive AI crawlers (GPTBot, CCBot).
Copy the output and place it in the recommended file location.

What This Tool Generates

robots.txt — A fully compliant robots exclusion protocol file.

Best Practices

Always include the absolute URL to your sitemap.xml at the bottom of the file.
Never use robots.txt to hide sensitive information. Malicious bots will ignore robots.txt; use authentication instead.
Keep the file simple. Start with 'User-agent: *' and 'Allow: /' before adding restrictions.

Common Mistakes

Assuming robots.txt prevents pages from being indexed. It prevents crawling; use 'noindex' to prevent indexing.
Blocking JS or CSS folders. Googlebot needs to render your page to understand it. Do not block static assets.
Using a single robots.txt to try and block the pages.dev domain from the custom domain. They are separate hosts.

Security Notes

Do not disallow paths that you want to keep completely secret (like /secret-admin-login-123), because anyone can read your robots.txt and find the path.

Frequently Asked Questions

How do I add robots.txt to Cloudflare Pages?

You simply create a robots.txt file in your framework's designated public/static directory (like public/robots.txt for Next.js). During the build, it is copied to the root of your deployment.

How do I block pages.dev from Google?

A robots.txt on your custom domain will NOT block Google from crawling your pages.dev domain. The pages.dev domain needs its own robots.txt, or you should use Cloudflare Bulk Redirects to redirect pages.dev to your custom domain.

Should staging be noindex?

Yes, staging environments should use both an X-Robots-Tag: noindex header in your _headers file AND ideally be protected by Cloudflare Access to guarantee they don't cause duplicate content.

Does robots.txt remove indexed pages?

No. Disallowing a URL in robots.txt only stops crawling. If a page is already indexed, you need to allow crawling, add a 'noindex' tag to the page, wait for Google to recrawl it, and THEN disallow it in robots.txt if desired.

Where do I put robots.txt in Next.js static export?

Place it in the public/ folder. When you run 'next build', it will automatically be copied to your 'out' directory.

How We Keep Your Configs Safe & Valid

Built-in Error Checking

Every file is checked against official rules. We catch missing fields and bad syntax. YAML indentation errors are flagged right away. Kubernetes, Terraform, and Docker specs are all covered. API versions and labels are verified too. You get valid output every time you generate.

100% Private & Local

All tools run in your browser only. Your API keys never leave your machine. We do not use any tracking scripts. No data is sent to any server. Passwords and secrets stay on your device. Crypto operations use the Web Crypto API. Your privacy is fully protected at all times.

Secure Settings by Default

Configs use safe defaults out of the box. Containers run as non-root users. Root filesystems are set to read-only. Dangerous Linux capabilities are dropped. Network policies limit pod-to-pod traffic. TLS 1.3 is enabled for web servers. Security headers are added where needed.

Ready for CI/CD & Git

Output files are ready for your Git repo. Use them with ArgoCD, Flux, or GitHub Actions. Files use clear formatting and comments. Code review is easy for your team. Indentation and key order are consistent. Test in staging before going to production. Every file is clean and well-structured.

Infrastructure as Code

Store configs in Git alongside your code. Terraform modules include typed variables. Backend configs support remote state locking. Outputs work across multiple modules. Ansible playbooks use clear task steps. Chef and Puppet configs are also supported. Every file works with version control tools.

Monitoring & Tracing

Set up Prometheus with auto-discovery rules. Create Grafana dashboards with template variables. Add alerting rules with severity labels. Use OpenTelemetry for trace collection. Forward logs to Loki or Elasticsearch. Connect to Jaeger or Tempo for tracing. Monitor metrics, logs, and traces together.

Container & Docker Safety

Dockerfiles use multi-stage builds for small images. Base images are pinned to exact versions. Dev files are excluded from final images. Health checks are added for orchestrator use. Containers switch to non-root users. Docker Compose uses named volumes and networks. Resource limits are set in deploy configs.

Multiple Output Formats

Export as YAML, JSON, HCL, or TOML. Kubernetes uses YAML with proper separators. Terraform uses HCL with correct escaping. JSON output has consistent indentation. Copy to clipboard with one click. Preview output with syntax highlighting. Line numbers help you review quickly.