How to Create a Robots.txt File: Complete Step-by-Step Guide

What Is a Robots.txt File?

A robots.txt file is a plain text file at the root of your website that tells search engine crawlers which pages to access and which to skip. It is the first file Googlebot reads when it visits your site.

Why Robots.txt Matters for SEO

Search engines have a crawl budget. If crawlers waste time on low-value pages, your important pages may not get crawled as often.

A proper robots.txt file prevents crawl budget waste, protects sensitive areas, and guides crawlers toward your highest-priority pages.

Robots.txt Syntax Explained

User-agent: Which crawler the rules apply to. Use * for all crawlers.

Disallow: URLs crawlers cannot access. Disallow: /admin/ blocks everything under /admin/.

Allow: Overrides a Disallow rule for a specific path.

Sitemap: Points crawlers to your XML sitemap.

Common Robots.txt Mistakes

Mistake 1: Blocking CSS and JavaScript. Google needs to render your pages to understand them.

Mistake 2: Using robots.txt to remove pages from indexing. Robots.txt prevents crawling, not indexing. Use noindex meta tags instead.

Mistake 3: Forgetting the Sitemap directive. Always include Sitemap in your robots.txt.

Step-by-Step: Create Your Robots.txt File

Step 1: Plan Your Rules. Decide which areas should be crawled and which should not.

Step 2: Write the File. Start with User-agent, add Disallow and Allow directives, end with Sitemap.

Step 3: Test Your Rules. Use Google Search Console Robots.txt Tester to verify.

Step 4: Upload to Root Directory. Place at yourdomain.com/robots.txt.

Step 5: Submit to Search Engines. Submit your robots.txt URL in Google Search Console.

Generate Your Robots.txt File

Our free Robots.txt Generator creates a complete robots.txt file in seconds.