Search engines are the primary source of traffic for many websites. However, there are many reasons you don’t want the search engines to crawl and index your site. Good thing is that WordPress allows you to hide your site using different methods depending on your needs. In this article, we will explain all possible ways to hide WordPress site from search engines like Google and Bing.
Why Hide Your WordPress Site from Search Engines?
Here are some common reasons you might want to prevent your site from showing up in search engine results:
- Development phase: You are still designing or building the site.
- Private content: You are creating content for a closed group of users or team members.
- Duplicate content: You are staging or cloning a live site and don’t want to hurt your SEO.
- Membership or client-only content: You want content to be accessible only to certain users.
- Internal tools: Your site is used for documentation, internal portals, or tools that shouldn’t be public.
For example, you might have launched a new site with dummy content and need some time to cleanup the setup. Let us explore 6 different methods in detail in the following sections.
1. Using Default WordPress Settings
WordPress has a setting to hide your site from search engines.
- Login to your admin panel and navigate to “Settings > Reading” section.
- Scroll down to the section labeled “Search Engine Visibility”.
- Enable the option that says, “Discourage search engines from indexing this site”.
- Click “Save Changes” button.
This will automatically update robots.txt file and block all search engines trying to crawl your site.
2. Using Robots.txt File
Instead of using default settings, you can also manually edit robots.txt file to block search engines. You can use FTP account or use File Manager app from hosting account or plugins to edit the file. For example, if you are using Yoast SEO plugin, go to “Yoast SEO > Tools” menu. Click on “File editor” option and paste the below code under robots.txt text box.
User-agent: *
Disallow: /If you want to hide a specific page, just use the relative URL of the page like below.
User-agent: *
Disallow: /page-url/3. Using Meta Robots Tag
Instead of robots.txt file you can use the meta robots tags both at site level as well as on individual page level. You can add the below meta tag in your header.php file to hide the entire site.
<meta name="robots" content="noindex, nofollow"/>Similarly, you can use the same code on individual page’s header section to hide that particular page. Many SEO plugins allow you to setup meta robots index.
For example, follow the below steps in the popular Yoast SEO plugin to index a single post.
- Go to the post editor and then go to “SEO” tab in the plugin’s meta box.
- Expand the “SEO Analysis > Advanced” sections.
- Click on the dropdown and select “No” for “Allow search engines to show this content in search results?”
- This will remove the post from XML Sitemap thus hiding it from search engine crawlers.
Follow the below steps for site level settings:
- Go to “Yoast SEO > Settings” menu.
- Expand “Content Types” and select “Posts” option.
- Turn off “Show posts in search results?” option.
- You can setup the same for each content type like Pages and other custom post types on your site.
Unfortunately, robots.txt file and meta robots tags will not help in completely hiding the site or page. Search engines like Google or Bing will follow these instructions. However, you can still see the page in search results with a message like below:
4. Use Password Protection or Private Status
WordPress also allows you to password protect individual pages that you can use an alternate option for hiding from search engines. When you are in the Gutenberg post editor, go to right sidebar settings of the “Post” and click on the “Status” link.
Select “Password protected” to add a password for the post. WordPress will prompt you to enter the password when opening the URL. Only users entering the correct password can unlock the content for reading.
When you password protect a page, it will be still part of the XML Sitemap. You can use the “Private” status to hide it from XML Sitemap. In this case, the post is not published and only can be seen by site admins and editors.
Unfortunately, there are no options in WordPress.org to set password for the entire site. However, you can use the password protection option from your hosting account to setup an authorization for accessing your WordPress installation folder. Login to your hosting account and setup a strong password for /public_html folder if you have installed WordPress in the server’s root. Otherwise, choose the subdirectory or subdomain where you have installed WordPress and setup a password.
Note: WordPress.com offers an option to keep your entire site in “Private” mode and invite specific users to view its content.
5. Coming Soon or Maintenance Mode
Sometimes, you may stuck with the broken site’s layout after updating a plugin or theme. In this case, the first option for you is to install a maintenance mode plugin and hide the content from users till the time you fix the problem. Though search engines will show the pages in the results, all users will land on the maintenance mode page.
In most cases, you need plenty of time to build content after installing WordPress on your domain. Unfortunately, simply hiding the site with robots.txt file is not sufficient as users can view the under construction pages. The best option in this case is to use a coming soon plugin and release the site after finishing. You can also use this option to hide the product or event launch pages till specific time.
6. Content Blocking
Content blocking is an option to hide the content from users and show them only after they complete certain actions. There are many plugins available to show content to users only after social sharing or subscribing to newsletter. You can also use multi-purpose plugins like Shortcodes Ultimate to block partial content of a page from users and search engines.
In addition, subscription based membership plugins also hide the premium content from search engines. Only premium users who paid for the content can login and view based on their assigned role.
Final Words
Popular search engines like Google and Bing will follow meta robots tag and robots.txt file instructions. However, it will not guarantee to hide the content as users can directly access the URL from the browser. Therefore, you can use password protection or other content locking options.
