Stop Search Engines from Crawling a WordPress Site
How to Stop Search Engines from Crawling a WordPress Site?
Search engine crawlers are the best source of traffic for any website. So then why would someone want to stop the search engine crawlers from crawling a WordPress site? It seems unrealistic but there are many reasons.
Stopping the search engine from crawling a WordPress site
Search engines can find your website even if you do not have any links pointing to your domain. You cannot control a lot of things happening on the web. However, there are certain things still under your control like blocking the search engines from crawling a WordPress site.
There is an inbuilt feature in WordPress where you can instruct the search engines not to index your website. You can give this instruction by going into the ‘Reading’ section of admin settings. Here just check the box that says Search Engine Visibility. Sometimes, even after doing all this, the search engines ignore the request and index your page. To make sure that the search engines do not index or crawl your website, you need to password protect your entire WordPress website at the server level.
You need to access your cPanel and click on ‘Password Protect Directories’ under the Security section. If you do not have access to cPanel, then you can also use plugins like SeedProd and Password Protected.
Why would someone want to stop the crawler from crawling a WordPress site?
A lot of people start developing their website on a publicly accessible domain name. So, they don’t want Google to index their page which is under construction or maintenance.
Many people also use WordPress to create private blogs. Obviously, they don’t want these blogs indexed. WordPress is also used for project management or Intranet by a lot of people. Search engine crawling and indexing would make their internal documents publicly accessible.
If you are in one of the above three situations, you will definitely want t the search engine crawler to stop crawling a WordPress site. Now you know how to do that.
