Your WordPress site has a robots.txt at /robots.txt and no robots.txt file anywhere on disk. WordPress generates it on request in do_robots(), which is why FTP into the root directory shows you nothing to edit.
The second thing worth knowing before you touch it: since WordPress 5.3, switching off Search engine visibility no longer puts Disallow: / in that output. Core moved that job to a noindex meta tag.
Plenty of staging and sandbox advice has not caught up.
What WordPress actually outputs, line by line
Here is the whole generating function, from wp-includes/functions.php in WordPress 7.1:
function do_robots() {
if ( ! headers_sent() ) {
header( 'Content-Type: text/plain; charset=utf-8' );
}
do_action( 'do_robotstxt' );
$output = "User-agent: *\n";
$public = (bool) get_option( 'blog_public' );
$output .= 'Disallow: ' . wp_parse_url( admin_url(), PHP_URL_PATH ) . "\n";
$output .= 'Allow: ' . wp_parse_url( admin_url( 'admin-ajax.php' ), PHP_URL_PATH ) . "\n";
echo apply_filters( 'robots_txt', $output, $public );
}
That is four lines of output on a stock install, and three facts most guides get wrong.
The paths are derived, not hard-coded. Both Disallow and Allow come from admin_url(), so a WordPress installed in a subdirectory emits Disallow: /blog/wp-admin/, not Disallow: /wp-admin/. If you have ever copied a "recommended WordPress robots.txt" off a blog post onto a subdirectory install, the paths in it were wrong.
$public is read but not used. get_option( 'blog_public' ) is assigned and then only passed to the filter. The default output is identical whether the site is public or not.
Disallow: /wp-includes/ is not in there. It has not been in core output for years, yet it is still the first line of nearly every "ideal robots.txt" people paste around.
Where the Sitemap: line comes from
If your robots.txt has a Sitemap: line and you never added one, core added it. WP_Sitemaps::add_robots(), added in WordPress 5.5, hooks the robots_txt filter:
public function add_robots( $output, $is_public ) {
if ( $is_public ) {
$output .= "\nSitemap: " . esc_url( $this->index->get_index_url() ) . "\n";
}
return $output;
}
This is the one place blog_public still changes the file. Turn Search engine visibility off and the Sitemap: line disappears, because $is_public is false.
Nothing else in the output moves.
So the full practical answer to "what does discouraging search engines do to my robots.txt" is: it removes one line, the sitemap pointer. It does not add a blanket disallow.
What replaced Disallow: / in WordPress 5.3
Core's own changelog for do_robots() records the 5.3 change as removing the Disallow: / output "in favor of robots meta HTML tag via wp_robots_no_robots() filter callback".
The mechanism now lives in wp_robots_noindex(), added in WordPress 5.7:
function wp_robots_noindex( array $robots ) {
if ( ! get_option( 'blog_public' ) ) {
return wp_robots_no_robots( $robots );
}
return $robots;
}
That puts noindex into the <meta name="robots"> tag in the page head rather than a directive in robots.txt.
The swap was a correctness fix, not a regression. A Disallow tells a crawler not to fetch a URL, which means it also cannot read a noindex on that URL.
Google states the consequence plainly: "A page that's disallowed in robots.txt can still be indexed if linked to from other sites" (Google Search Central, robots.txt introduction, fetched 2026-10-15). Its advice for keeping a URL out of results is the noindex tag or response header, not a disallow.
A site blocked only by robots.txt can end up in the index as a bare URL with no description. A site serving noindex gets crawled and then dropped.
The second is what you want for a site that should not be found.
Editing it: the filter, or a real file
There are exactly two ways to change the output, and they do not mix.
The robots_txt filter (since WordPress 3.0) takes the generated string and the $public boolean:
add_filter( 'robots_txt', function ( $output, $public ) {
if ( ! $public ) {
return "User-agent: *\nDisallow: /\n";
}
$output .= "Disallow: /?s=\n";
return $output;
}, 10, 2 );
Put that in an mu-plugin if you want it to survive theme switches and plugin deactivation. That example also restores the pre-5.3 behaviour for non-public sites, which is a reasonable belt-and-braces addition as long as you understand it is belt and braces, not the actual block.
A physical robots.txt in the web root wins outright, and the filter then never runs.
The reason is the standard WordPress rewrite rules: they send a request to index.php only when no real file matches, so a file on disk is served by the web server and PHP is never reached. If you add a real file and then wonder why your filter stopped working, this is why.
The same !-f condition is in the default WordPress .htaccess.
What there is no supported way to do is edit it from the WordPress admin. There is no core screen for robots.txt.
The SEO plugins that offer an editor are writing a real file, or filtering robots_txt, which means you inherit whichever of the two behaviours above they chose.
Multisite and subdirectory installs
Both cases trip people up for the same reason: robots.txt is a per-host file.
Google's spec is explicit: "The robots.txt file must be located at the root of the site host to which it applies," and it "cannot be placed in a subdirectory" (Google Search Central, create a robots.txt file, fetched 2026-10-15).
For a subdirectory install at example.com/blog/, crawlers read example.com/robots.txt and ignore example.com/blog/robots.txt. WordPress happily serves the second one, with the correct derived /blog/wp-admin/ paths in it, and no crawler will ever look at it.
You need the rules at the domain root, which is outside WordPress.
For subdomain multisite, each site is its own host, so each gets its own robots.txt and its own blog_public option. For subdirectory multisite, only the main site's robots.txt is at a host root, so the sub-site ones are decorative.
That asymmetry is one more thing to weigh when you are choosing between subdomain and subdirectory.
Checking what you are really serving
Request it rather than guessing, because a cache plugin, a CDN, or a stale physical file can all be in between.
curl -sS https://example.com/robots.txt
Then confirm whether it came from PHP or from disk:
# Is there a real file?
ls -la robots.txt
# What is blog_public set to?
wp option get blog_public
# What does the filtered output look like from inside WordPress?
wp eval 'echo apply_filters( "robots_txt", "", (bool) get_option( "blog_public" ) );'
wp option get blog_public returning 1 means public, 0 means discouraged. If ls finds a file, the filter is irrelevant and that file is your answer.
On SandyWP every sandbox has WP-CLI preinstalled and SSH access, so all three of those run directly in the sandbox with no plugin and no local setup.
The honest limitation: robots.txt is not access control
Both of the top-ranking guides on this phrase describe robots.txt as a way to keep sensitive areas private. It is not, and this is the most consequential thing to get right.
A robots.txt is a public document at a predictable URL that lists the paths you would rather people not visit. Compliance is voluntary.
Well-behaved crawlers honour it, and anything scraping content or probing for vulnerabilities does not.
If a URL genuinely must not be reached, the control has to be at the server: HTTP authentication, an IP allowlist, or a login wall. Use robots.txt for crawl budget and for keeping junk URLs like internal search results out of the crawl, and use noindex for keeping pages out of the index.
Neither is a permission system.
What this means for a staging or sandbox site
This is where the 5.3 change actually bites, because "tick Discourage search engines" is the standard advice for a copy of a live site, and people assume it writes a disallow.
It does not. A site with blog_public = 0 serves the same four robots.txt lines as a public one, minus the sitemap pointer, plus a noindex meta tag on every page.
The noindex is the part doing the work, and it only works if crawlers can fetch the page to see it.
Which means the common belt-and-braces setup, discourage search engines and a hand-written Disallow: /, is worse than either alone. The disallow stops the crawl, the crawler never sees the noindex, and the URL can still surface from an inbound link.
Two things that do work on a staging copy, in order of strength:
- HTTP authentication at the server. Nothing is crawled, nothing is indexed, nothing is readable. This is the only one that is actually a control rather than a request.
noindexalone, with no disallow. Let crawlers in, let them read the tag, let them drop the URL.
On SandyWP this is set for you. Every sandbox gets blog_public set to 0 at provisioning time, in wordpress-scripts.ts, on both a fresh install and an imported copy of a live site, so the noindex is in place before you ever open the dashboard.
If you are importing a production site, that matters: a clone carries the original's blog_public = 1 otherwise, and you get a second indexable copy of a site you already rank for.
If the site in question is a long-lived staging environment that real people need to reach, a sandbox is the wrong tool and HTTP auth on your staging host is the right one. What a sandbox is good for is the throwaway copy you make to test a plugin update or reproduce a bug, where the correct outcome is that search engines never form an opinion about it at all.
The short version
- WordPress serves robots.txt from
do_robots(), not from a file. There is nothing on disk to find. - Default output is four lines, and the
wp-adminpaths are derived fromadmin_url(), so they differ on a subdirectory install. - Since WordPress 5.3, discouraging search engines does not add
Disallow: /. It adds anoindexmeta tag instead, and removes theSitemap:line. - Change the output with the
robots_txtfilter, or with a real file, which silently disables the filter. - robots.txt only works at a host root, so subdirectory installs and subdirectory multisite sub-sites cannot have their own.
- It controls crawling, never access. For a copy of a live site, HTTP auth beats everything, and
noindexwithout a disallow beats both of them combined.
