When a new WordPress site starts getting crawled, one of the first issues I regularly see is unnecessary pages showing up in that crawl — tag archives, admin URLs, internal search-result pages, thin category listings. Search engines will crawl all of them unless told otherwise, and a robots.txt file is the primary way to give that instruction.
It’s a small plain-text file at the root of your site telling crawlers which parts they’re allowed to access. It doesn’t remove pages from search results on its own — that’s what noindex does — but it does control where a crawler spends its time. For most WordPress sites, a short, clean robots.txt is all that’s needed, and getting it wrong is easier than it looks.
Quick Answer
WordPress generates a default robots.txt automatically. For most sites the default is fine with two additions: a Disallow rule for internal search-result pages, and a Sitemap line pointing to your XML sitemap. Edit it through your SEO plugin’s robots.txt editor if you use one, or create a physical file via your hosting file manager for a plugin-free setup. Never block /wp-content/ — that hides your images and scripts from being indexed and rendered properly.
Crawl Budget: Who Actually Needs to Worry About It
Google’s own crawl-budget documentation, rewritten as recently as July 2026, is direct about this: crawl budget is a real constraint mainly for sites with over a million pages, or sites in the 10,000+ page range that publish or change content daily — large news publishers and big ecommerce catalogues, essentially. If your site has fewer than 10,000 pages and new content gets indexed within a day or two of publishing, crawl budget isn’t limiting you, and no amount of robots.txt tuning will meaningfully speed anything up.
The July 2026 update also added a detail worth knowing: every site now starts with the same conservative default crawl-capacity limit, which Google raises automatically as demand grows and your server keeps responding quickly. That capacity is shared across all of Google’s crawlers — including its AI-training crawlers — so heavy AI-crawler traffic against a site can, in principle, leave less headroom for Googlebot itself. None of this changes what a small site should actually do with robots.txt, but it explains why the advice below focuses on removing genuinely low-value URLs rather than chasing marginal crawl-efficiency gains that mostly matter at a much larger scale.
Step-by-Step Instructions
Step 1: Check Your Current Robots.txt
Visit https://yourdomain.com/robots.txt in a browser before changing anything. A fresh WordPress install shows a default file that looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThis is WordPress’s virtual robots.txt, generated dynamically rather than sitting as a physical file on the server. It’s a reasonable starting point, but most sites benefit from the two additions below.
Step 2: Decide What to Block — and What Not To
The standard areas worth disallowing on a WordPress site:
- /wp-admin/ — already in WordPress’s default, with the admin-ajax.php exception left open since plugins rely on it from the front end.
- /search/ and /?s= — internal WordPress search-result pages, thin and duplicate with no SEO value of their own.
What not to block:
- /wp-content/ — your images, theme files, and plugin assets live here. Blocking it stops Google from indexing images and can distort how it renders and evaluates your pages, since Google needs to load your CSS and scripts to judge a page fairly.
- /wp-includes/ — core WordPress files. Blocking these has caused real rendering issues on some sites for no upside.
- Actual content pages, categories, or tags — handle these with noindex on the page itself if you genuinely need them out of search, not a blanket robots.txt rule.
If pagination or filtered archive pages are generating thin content, noindex at the page level is the right tool, not a robots.txt disallow — the guide on organising categories and tags for SEO covers managing that properly.
Step 3: Add Your Sitemap Reference
Add a Sitemap line at the bottom pointing to your XML sitemap, which helps crawlers find and process it even outside a Search Console visit:
Sitemap: https://yourdomain.com/wp-sitemap.xmlWordPress core generates a sitemap at /wp-sitemap.xml by default since WordPress 5.5. If you’ve submitted a sitemap to Search Console, make sure this line points to that same URL — see creating an XML sitemap in WordPress for the full setup.
Step 4: Edit the File
Via an SEO plugin: if your plugin includes a robots.txt editor, it creates a physical file overriding WordPress’s virtual one, and changes apply immediately. Via hosting file manager: log into your hosting control panel, navigate to the root of your install (usually public_html), and create or edit robots.txt directly.
A clean, complete file for most WordPress sites:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /search/
Disallow: /?s=
Sitemap: https://yourdomain.com/wp-sitemap.xmlReplace yourdomain.com, save, and confirm it loads at https://yourdomain.com/robots.txt.
Step 5: Check It in Search Console — Correctly This Time
Here’s a distinction most guides get wrong: Google retired its old standalone robots.txt Tester tool in December 2023. What replaced it is a robots.txt report under Search Console’s Settings section, and the two do different jobs. The report shows the last time Google fetched your file, plus any fetch errors or warnings across up to 20 hosts — it does not let you test whether one specific URL is blocked. For that, use the URL Inspection tool instead, which shows exactly how Google’s crawler currently sees a given page, including whether robots.txt is blocking it.
After editing your file, give Google a few days to re-fetch it, and if you’ve removed a rule that was previously blocking something important, submit a recrawl via URL Inspection rather than waiting. If you haven’t set up Search Console yet, setting up Google Search Console for WordPress covers the process.
A Worked Example: Diagnosing a Blocked WooCommerce Page
Here’s a scenario I see often enough to walk through in full: a WooCommerce store owner notices a product category page isn’t appearing in search, opens Search Console’s Page Indexing report, and finds it listed under “Blocked by robots.txt.” The instinct is to immediately edit robots.txt — but the right first step is Search Console’s URL Inspection tool, entering the exact URL to see precisely which rule is matching it and why, since a plugin (often an SEO or caching plugin) may have quietly added its own Disallow line for a URL pattern that happens to match this category’s structure.
A common culprit in this scenario is a caching plugin’s own robots.txt addition blocking a pattern like /*?orderby=, added to stop crawlers indexing every sort-order variation of a shop page. That’s a reasonable idea on its own, since orderby parameters do generate near-duplicate URLs — but a wildcard written broadly enough to match “any query string” rather than specifically the sort parameter can catch the plain category URL too, the moment an unrelated tracking parameter gets appended to it by an ad platform or an email link. The fix in that case isn’t removing the rule — the underlying goal was sound — it’s narrowing the pattern to match only the specific parameter it was meant to catch. This is the actual value of checking URL Inspection before editing blindly: two different problems (a legitimately unhelpful URL and an accidentally caught one) can produce the identical “Blocked by robots.txt” message, and only inspecting the specific rule tells you which one you’re dealing with.
Practical Tips
Keep the file short. A robots.txt with dozens of rules is harder to maintain and easier to misconfigure than one with five. Handle anything genuinely tricky at the page level with noindex instead of stacking more Disallow lines.
Never block your CSS and JavaScript files. Older SEO advice recommended this; Google now needs to render pages fully to evaluate them, so blocking render resources actively works against you.
Open /robots.txt again after any major structural change to the site — a new plugin can introduce URL patterns worth reviewing, and it’s a five-second check.
Common Mistakes
Blocking /wp-content/. The single most common robots.txt mistake on WordPress sites, and it directly hurts image search visibility and rendering evaluation.
Using robots.txt to hide sensitive content. It’s a public file anyone can read. A Disallow rule doesn’t stop direct visits, and doesn’t guarantee search engines skip indexing a URL if other sites link to it. Use password protection or authentication for anything genuinely private.
Accidentally blocking the whole site. A stray Disallow: / left over from a staging setup blocks every crawler from everything. Always check the live file right after any deployment.
Conclusion
For most WordPress sites, a minimal robots.txt — blocking wp-admin and internal search, referencing your sitemap, leaving everything else open — is genuinely all that’s needed. It’s one of the smaller SEO tasks on a site, but a misconfigured one can cause real crawlability problems that are entirely avoidable. For the fuller sequence of building and launching a site, see the essential steps to build a WordPress website.
Etienne Basson works with website systems, SEO-driven site architecture, and technical implementation. He writes practical guides on building, structuring, and optimizing websites for long-term growth.