WordPress robots.txt: the default, what to change, what to leave

Google describes the file's job narrowly: it "is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." On most WordPress blogs the default file is already close to right. The damage usually comes from rules added on top of it, copied from an old checklist.

The default WordPress robots.txt

Visit /robots.txt on a fresh WordPress install and you get this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

The function do_robots() builds it when the URL is requested and passes it through the robots_txt filter, so plugins can append their own rules. The paths follow the site's admin URL, so WordPress installed in a subdirectory gets that subdirectory in both lines. The Allow line matters because front-end features of themes and plugins send requests to admin-ajax.php, and blocking the rest of /wp-admin/ should not break them. While WordPress's own sitemaps are active on a public site, WordPress 5.5 and later also add a Sitemap: line pointing to /wp-sitemap.xml.

Two details trip people up. First, the file is virtual: Yoast's documentation puts it plainly, "WordPress generates a virtual robots.txt file if the site root does not contain a physical file." You will not find it over FTP until you create one. Second, the "Discourage search engines from indexing this site" setting no longer touches robots.txt. Since WordPress 5.3 it no longer adds Disallow: /; WordPress uses a robots meta tag on each page instead.

How to edit robots.txt in WordPress

There are three ways, and they do not mix well. Pick one.

Through your SEO plugin

Yoast SEO has a file editor under Yoast SEO, Tools, File editor, with a "Create robots.txt file" button. The file it creates replaces the WordPress default with a block that has an empty Disallow: and a Sitemap: line pointing to the Yoast sitemap index. In Yoast's words, "These directives allow all search engines to crawl your site." If the menu is missing, file editing is disabled on the install and you need FTP or your host's file manager instead. Our roundup of SEO plugins for WordPress covers the other main plugins.

With a physical file

Create a plain-text file named robots.txt, write your rules, and upload it to the web root, the folder that holds wp-config.php on most installs. A physical file wins over the virtual one completely. The web server answers the request without running WordPress, so nothing plugins add through the filter appears any more, including the core sitemap line. If you go this route, you maintain every line yourself.

With the robots_txt filter

Developers can keep the file virtual and add rules in code, in a small plugin or a child theme. This works only while no physical file exists:

add_filter( 'robots_txt', function ( $output, $public ) {
	if ( $public ) {
		$output .= "\nUser-agent: GPTBot\nDisallow: /\n";
	}
	return $output;
}, 10, 2 );

The $public flag is WordPress's own "is this site public" setting, so the rule disappears automatically on a staging copy set to discourage search engines.

What not to block

Theme files, scripts and uploads. Old WordPress guides told you to disallow /wp-includes/ or /wp-content/. Both hold CSS, JavaScript and images that pages need. Google's rule for resource files: if their absence makes a page "harder for Google's crawler to understand," don't block them. Blocking /wp-content/uploads/ also keeps your images out of Google's image results, because robots.txt is the documented way to "prevent image, video, and audio files from appearing in Google Search results."

Pages you want out of search. A disallowed URL can still be indexed. Google: "we might still find and index a disallowed URL if it is linked from other places on the web," and such a result appears without a description. Use noindex for that job.

Leftovers you forgot about. Old landing pages, test pages and posts nobody links to are better handled at the source. If a page is unlinked and unwanted, redirect or delete it; if it is unlinked and wanted, it is an orphan page that needs links, not a Disallow line.

robots.txt or noindex?

	robots.txt Disallow	noindex
Controls	Whether crawlers fetch the URL	Whether the page appears in search results
Can the URL still show up in Google?	Yes, if linked from elsewhere, without a description	No, once Google crawls the page and sees the tag
Set it in WordPress through	robots.txt (plugin editor, physical file or filter)	Your SEO plugin's per-post setting, or the site-wide visibility setting
Use it for	Crawl load: internal search results, endless filter URLs	Pages that stay live but out of search, such as thank-you pages or thin archives

Never use both on the same URL. Google cannot crawl a page you disallowed, so it never sees the noindex tag on it, and Google's guide warns that "combining multiple crawling and indexing rules might cause some rules to counteract other rules." The same logic applies to near-identical URLs, such as tracking parameters or print versions. Blocking them stops Google from reading the canonical tag that names the original, which is why duplicate content is usually fixed with canonicals and redirects rather than robots.txt.

Rules for AI crawlers

AI companies publish their own robots.txt user-agent tokens, and the file is where you set them. Two common examples:

• OpenAI runs GPTBot, which crawls content "that may be used in training our generative AI foundation models," and OAI-SearchBot, which surfaces sites in ChatGPT search. The settings are independent: a site can allow OAI-SearchBot "in order to appear in search results while disallowing GPTBot." A third agent, ChatGPT-User, visits pages when a person asks ChatGPT to, and OpenAI notes that "robots.txt rules may not apply" to it.
• Google-Extended is not a separate crawler but a token that controls whether content Google crawls may be used for training Gemini models and for grounding in Gemini Apps. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

A file that opts out of model training but stays visible in ChatGPT search looks like this:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Remember what robots.txt cannot do. Google is blunt that "it's up to the crawler to obey" the rules; well-behaved crawlers do, others may not. And it is a different thing from llms.txt, a proposed Markdown file that points AI systems at your best pages without granting or refusing access.

A sensible robots.txt for a WordPress blog

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap_index.xml

The first three lines are the WordPress default. The two search lines keep crawlers out of internal search results, an example of the "unimportant or similar pages" Google says robots.txt can keep crawlers away from. The last line names whichever sitemap index you actually submit to Search Console: sitemap_index.xml for Yoast SEO or Rank Math, wp-sitemap.xml for core. Add AI-crawler groups only if you have decided you want them.

robots.txt on a headless site

On a headless setup the public front end serves robots.txt, so it lives in the front-end project, not in WordPress. The CMS host, such as cms.example.com, needs more than a Disallow line, because a disallowed URL can still be indexed from links. Keep it out of search with noindex or password protection, as Google advises for anything that must not appear. The headless CMS SEO checklist covers the rest of what the front end owns.

How Structura handles this

Structura never writes or edits robots.txt. The plugin adds nothing to WordPress's robots_txt filter, so the file your site serves comes from WordPress, your SEO plugin or you. On headless sites, robots.txt belongs to your front end.

The plugin does serve two other root files, and neither changes your crawl rules. When the IndexNow channel is connected, it answers requests for the IndexNow key file, with a noindex header so the key never shows up in search. And when no SEO plugin is active, it serves a fallback /llms.txt. Before you connect IndexNow, the install panel warns you if the site is set to discourage search engines, since pinging Bing about a site that tells search engines to stay away does nothing useful.

Where a robots.txt mistake would show up is indexing. With Google Search Console connected on a paid plan, Structura shows whether Google has indexed each post, so a post that a bad rule keeps out of Google stands out instead of silently earning nothing. The blog SEO software page covers what each post ships with and what the Search Console checks report.

FAQ