When a new WordPress site starts getting indexed, one of the first issues I regularly see is unnecessary pages appearing in search results. Tag archives, admin URLs, search result pages, thin category pages — search engines crawl all of them unless you tell them not to. A robots.txt file is the primary way to give that instruction.
It’s a small plain text file that sits at the root of your website and tells crawlers which parts of the site they’re allowed to access. It doesn’t block pages from appearing in search results on its own — that’s what noindex does — but it does control where search engines spend their crawl budget. For most WordPress sites, a simple, clean robots.txt is all you need.
Quick Answer
WordPress generates a default robots.txt file automatically. For most sites, the default is fine with two additions: a directive blocking the /wp-admin/ directory, and a Sitemap line pointing to your XML sitemap. You can edit the robots.txt directly through an SEO plugin, or for a plugin-free setup, create a physical file in your root directory via your hosting file manager. Don’t block /wp-content/ — that prevents images and scripts from being indexed.
Why Robots.txt Matters for WordPress
Search engines allocate a crawl budget to each site — a limit on how many pages they’ll crawl in a given period. On a small site this rarely matters, but as a site grows, letting crawlers spend time on admin pages, login pages, and internal search results means they have less time for your actual content.
A correctly configured robots.txt keeps crawlers focused on the pages that matter. It also prevents some internal pages from being accidentally indexed — though for pages you genuinely need to hide from search results, a noindex meta tag is more reliable than a robots.txt disallow rule. Google’s documentation on how robots.txt works explains the distinction clearly.
Step-by-Step Instructions
Step 1: Check Your Current Robots.txt
Before making any changes, check what’s already there. Visit https://yourdomain.com/robots.txt in your browser. If WordPress is installed and running, you’ll see a default file that looks something like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThis is WordPress’s virtual robots.txt — it’s generated dynamically and not a physical file on the server. It’s a reasonable starting point, but most sites benefit from a few additions.
Step 2: Decide What to Block
The standard areas to disallow on a WordPress site are:
- /wp-admin/ — The admin dashboard. Already in the WordPress default, with the admin-ajax.php exception (which needs to remain accessible for plugins that use it on the frontend).
- /search/ — Internal WordPress search result pages. These are thin, duplicate-content pages with no SEO value.
- /?s= — The query string version of search results, if your site uses that format.
What not to block:
- /wp-content/ — This directory contains your images, theme files, and plugin assets. Blocking it prevents Google from indexing your images and may affect how your pages render in search.
- /wp-includes/ — Core WordPress files. No need to block, and doing so can cause rendering issues.
- Your actual content pages, categories, or tags — unless you have a specific reason to block them, which is better handled with noindex.
If your site has pagination, filtered archive pages, or category/tag archives generating thin content, those are better handled with noindex on the page level rather than a blanket robots.txt disallow. The guide on organising WordPress categories and tags for SEO covers how to manage archive page indexing sensibly.
Step 3: Add Your Sitemap Reference
Add a Sitemap line at the bottom of the robots.txt pointing to your XML sitemap. This helps search engines find and process your sitemap even if they haven’t visited your site through Search Console. The line looks like this:
Sitemap: https://yourdomain.com/wp-sitemap.xmlWordPress generates a default sitemap at /wp-sitemap.xml. If you’ve submitted your sitemap to Google Search Console, you should be pointing to the same URL here. The guide on how to create an XML sitemap in WordPress covers the sitemap setup in full.
Step 4: Edit the Robots.txt File
There are two ways to edit your robots.txt in WordPress:
Option 1 — Via an SEO plugin: If you use an SEO plugin that includes a robots.txt editor (Rank Math and Yoast both have one), use that. The plugin creates a physical file that overrides the WordPress virtual file. Changes take effect immediately.
Option 2 — Physical file via hosting: Log into your hosting control panel and open the File Manager. Navigate to the root of your WordPress installation (usually public_html or www). If a robots.txt file already exists there, edit it. If not, create a new file named robots.txt and paste in your content.
A clean, complete robots.txt for most WordPress sites looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /search/
Disallow: /?s=
Sitemap: https://yourdomain.com/wp-sitemap.xmlReplace yourdomain.com with your actual domain. Save the file and verify it’s accessible at https://yourdomain.com/robots.txt.
Step 5: Test It in Google Search Console
Google Search Console includes a robots.txt tester. Log into Search Console, go to the Settings section, and find the robots.txt report. It shows your current file and lets you test whether specific URLs are allowed or blocked. If you haven’t set up Search Console yet, the guide on setting up Google Search Console for WordPress covers the process.
After testing, give Google a few days to re-crawl the file. If you’ve made significant changes — particularly if you’ve removed a disallow rule that was blocking important content — submit a recrawl request via the URL inspection tool in Search Console.
Practical Tips
Keep the file simple. A robots.txt with dozens of rules is harder to maintain and easier to misconfigure. Block only what genuinely needs blocking and handle the rest at the page level with noindex.
Never block your CSS and JavaScript files. Some older SEO advice recommended this, but Google now needs to render your pages to evaluate them properly. Blocking render resources can negatively affect how your pages are assessed.
Check the file whenever you make significant structural changes to the site. A plugin you’ve installed may add new URL patterns worth reviewing. It’s worth opening /robots.txt after any major site change to confirm everything still looks correct.
Common Mistakes
Blocking /wp-content/. This is the most common robots.txt mistake on WordPress sites. It prevents Google from accessing your images, stylesheets, and plugin files — which affects both image search visibility and how well Google can render and evaluate your pages.
Using robots.txt to hide sensitive content. Robots.txt is a public file — anyone can read it. Disallowing a URL doesn’t prevent people from visiting it directly, and it doesn’t guarantee search engines won’t index it if other sites link to it. Use password protection or authentication for genuinely private pages.
Accidentally blocking the whole site. A rule like Disallow: / blocks all crawlers from the entire site. This is occasionally added by mistake during development and left in place after launch. Always verify your live robots.txt after any deployment.
Conclusion
For most WordPress sites, a minimal robots.txt — blocking wp-admin and search results, including a sitemap line, and leaving everything else open — is all you need. Set it up early, verify it in Search Console, and revisit it only when your site structure changes significantly. It’s one of the smaller SEO tasks, but getting it wrong can cause real crawlability problems that are easy to avoid. For the broader sequence of steps involved in building and setting up a WordPress website, the essential steps to build a website covers each stage in order.
Etienne Basson works with website systems, SEO-driven site architecture, and technical implementation. He writes practical guides on building, structuring, and optimizing websites for long-term growth.