Best robots.txt of a WordPress Website

You are here : Home / Best robots.txt of a WordPress Website

A text file is known as robots.txt that you put on your website to tell the crawlers that which pages you would like them to visit and not to visit. Search engines obey the things that are asked not to do, but robots.txt is not mandatory for them. Robot.txt is not any type of firewall or password protection for search engines. It is also not preventing search engine from crawling your website. If you really have any data which you do not want to show in search results then you never have to trust on robots.txt to keep it from being indexed and displayed in search results.

robots.txt must be resides in the main directory. Search engines are only capable to discover it in the main directory, if it resides in any other place then search engines do not search it in the whole website  and unable to find it. Search engines firstly look it into the main directory and if it doesn’t exists then search engines assume that robots.txt file does not exist in the website. So, if robots.txt is not placed in the correct place then search engine displays everything they find.

Syntax of a robots.txt File

There are many search engines are present and many different files that you want to disallow. The syntax of robots.txt file is as follows:-

Search engine’s crawlers are written in the user-agent and the list of directories and files which you do not want to display or crawl are written in front of the Disallow.

You can also add a comment line by using hash (#) sign at the beginning of the line.

For Example:-

The above example shows that User-agent: * means it includes all the search engine’s crawlers and Disallow: /temp/ means that it disallow the file name temp to display.

Important Things for Best robots.txt of a WordPress Website

If you are dealing with wordpress then you want to display your pages and posts by the search engines but you do not want the search engines to crawl your core wordpress files and directories and also trackbacks and feeds. The contents of robots.txt file varies from site to site differently. You must have to create robots.txt file in the root directory of your website. There isn’t a standardized robots.txt file for WordPress but the following points give you a clear idea about the best robots.txt file of a wordpress website.

1. Things you should always block

There are some of the files and directories in the wordpress site which should be blocked every time. The directories which you should disallow in the robot.txt file are “cgi-bin” directory and the standard WP directories.Some servers don’t allow to access “cgi-bin” directory but you have to include it in your disallow directive in the robot.txt file and it won’t be harmful if you do that.

The standard wordpress directories that you should block are wp-admin, wp-content, wp-includes. These directories do not have any data that are initially useful for the search engines, but the exception is there i.e. a subdirectory named as “uploads” exists in the wp-content directory. This sub directory should be allowed in the robot.txt because it includes everything you upload using WP media upload feature. So, you must have to make it unblocked.

The directives used for above are given below:-

2. Things to block depending on your WP configuration

You must have to know about your wordpress site uses tags or categories to structure the content or uses both categories and tags or uses none of them. If you are using categories then you must have to block the tag archives from search engines and vice-versa. Firstly check the base, just go to Admin panel > Settings > Permalinks.

By default the base is tag, if the field is blank. You have to disallow tag in the robot.txt as given below:

If you are using category then you have to block category in the the robot.txt as given below:

If you are using both categories and tags then don’t have to do anything in the robot.txt file.

If you are using neither tags nor categories then block both of them in the the robot.txt as given below:

3. Files to block separately

Different files are used in wordpress to display the content. These all files need not to available for the search engines. So you have to block them also. The different files mostly used for display the content are PHP files, JS files, INC files, CSS files.

You have to block them in the robot.txt as given below:

The “$” character matches the end of an URL string.

Keep in mind that it is recommended not to block the files that are not in uploads directory.

4. Things not to block

There are many things which you do not want to block its depending upon your choice. Here I don’t want to block images from Google image search, so I have to allow it in the robot.txt file as given below:

You can add the things that you do not want to block as written in the above example.

 Best robots.txt of a Wordpress Website

Flipper Code

We have been building WordPress Plugins at flipper code since 2008. We follow wordpress coding standard that ensures we deliver the excellent wordpress plugins and services.

More Posts - Website

Follow Me:
twitter Best robots.txt of a Wordpress Websitefacebook Best robots.txt of a Wordpress Websitegoogleplus Best robots.txt of a Wordpress Website

Customize Product according to your requirements?

We will be glad to discuss any of your needs related to our wordpress plugins . if any of functionality you'd like to see in products, you can reach us by via email hello@flippercode.com.