How to Hide WordPress Site from Search Engines?

Search engines are the primary source of traffic for many websites. However, there are many reasons you need to hide your site from search engines. WordPress allows you to hide your site in different ways depending upon the need. In this article we will explain all possible ways to hide WordPress site from search engines like Google.

1. Use Default Setting to Edit Robots.txt File

WordPress has a setting to hide your site from search engines. Login to your admin panel and navigate to “Settings > Reading” section. Enable the option that shows “Discourage search engines from indexing this site”.  

This will automatically update robots.txt file and block all search engines try to crawl your site.

Instead of using default setting, you can also manually edit robots.txt file to block search engines. You can use FTP account or use File Manager app from hosting account or plugins to edit the file. For example, if you are using Yoast SEO plugin, go to “SEO > Tools” menu. Click on  “File editor” option and paste the below code under robots.txt text box.

User-agent: *
Disallow: /

If you want to hide a specific page, just use the relative URL of the page like below.

User-agent: *
Disallow: /page-url/

2. Use Meta Robots Tag

Instead of robots.txt file you can use meta robots tags both at site level as well as on individual page level. You can add the below meta tag in your header.php file to hide the entire site.

<meta name="robots" content="noindex, nofollow"/>

Similarly, you can use the same code on individual page’s header section to hide that particular page. Many SEO plugins allow you to setup meta robots index. For example, if you are using the popular Yoast SEO plugin, go to the post editor and then go to “SEO > Advanced” section in the plugin’s meta box. Click on the dropdown and select “No” for “Allow search engines to show this Post in search results?”. This will remove the post from XML Sitemap thus the hiding from search engine crawling bots.

For site level settings, you can go to “SEO > Search Appearance” menu and go to “Content Types” section. Select “No” for “Show Posts in search results?” option. You can setup the same for each content type like pages and other custom post types on your site.

3. Password Protection

WordPress also allows you to password protect individual pages that you can use an alternate option for hiding from search engines. When you are in the post editor, click on the “Public” link against “Visibility” option. You can see this under “Document” section on the right sidebar.

Select “Password Protected” to add a password for the page. WordPress will prompt you to enter the password when opening the URL. Only users entering the correct password can unlock the content for reading.

When you password protect a page, it will be still part of the XML Sitemap. You can use the “Private” status to hide it from XML Sitemap. In this case, the post is not published and only can be seen by site admins and editors.

Unfortunately, there are no options in WordPress to set password for the entire site. However, you can use the password protection option from your hosting account to setup an authorization for accessing your WordPress installation folder. Login to your hosting account and setup a strong password for /public_html folder if you have installed WordPress in the server’s root. Otherwise, choose the subdirectory or subdomain where you have installed WordPress and setup a password.

4. Coming Soon or Maintenance Mode

Sometimes, you may struck with the broken site’s layout after updating a plugin or theme. In this case, the first option for you is to install a maintenance mode plugin and hide the content from users till the time you fix the problem. Though search engines will show the pages in the results, all users will land on the maintenance mode page.

In most cases, you need plenty of time to build content after installing WordPress on your domain. Unfortunately, simply hiding the site with robots.txt file is not sufficient as users can view the under construction pages. The best option in this case is to use a coming soon plugin and release the site after finishing. You can also use this option to hide the product or event launch pages till specific time.

5. Content Blocking

Content blocking is an option to hide the content from users and show them only after they complete certain actions. There are many plugins available to show content to users only after social sharing or subscribing to newsletter. You can also use multi-purpose plugins like Shortcodes Ultimate to block partial content of a page from users and search engines.

In addition, subscription based membership plugins also hide the premium content from search engines. Only premium users who paid for the content can login and view based on their assigned role.