How to write a proper Robots.txt

robots txt

Greetings to all my site visitors. Today’s article will be devoted to such a great file as robots.txt. I would like to share how to generate this file, as well as tell you how this file affects the promotion of the site as a whole.

To date, many novice professionals are interested in this file. Even more, not only interested, but also try to set it up on their own. Why such close attention to robots? There is an opinion that the correct setting depends on how in the future the site will move in extradition. To be honest, this statement, in my opinion, has nothing to do with the truth, although the file is really important, but not as described by “gurus”.

What then is the role of robots at all? In fact, robots.txt serves to hide pages from search engine eyes. This is its main function. This is especially true when you use a CMS. It is engines create a ton of duplicates, which can safely close in the robots.txt file. By the way, I’d like to recommend you an article: “Does the CMS of the site on the promotion and positions in general”. Relevant information, and I mentioned in this article about CMS.

Let’s go back to robots.txt. Next, I’ll show you what each entry in the file means and give you the ideal robots structure for Google. But before we go into detail about the parameters, let’s find out what robots are.

Robots.txt is a text file located in a folder with web sites (in hosting). This file specifies the instructions that need to be done on the site. As a rule, almost all instructions are related to the indexing of pages and site as a whole.

Basic commands in robots.txt

Disallow: / – blocks access to the entire site
Disallow: /admin – blocks access to pages beginning with admin

Allow: / – allows access to the whole site
Allow: /admin – allows access to pages beginning with admin

“*” – is a special symbol which means any set of symbols. Usually applies when using Allow and Disallow directives.

Sitemap: – directive shows search robot where the sitemap is located. Remembering the address of the sitemap once, in the future search robot refers to this address. Example of use: Sitemap: http://

Host: – directive indicating what mirror of your site is main. Of course used if your site has mirrors.

These are the main parameters that can be found in the file robots.txt. Below, ideally composed robots.txt for Google.

Example of a robot for Google:

User-agent: Googlebot

Disallow: /admin


There is another option that occurs most often. This option, as a rule, is not written for different search engines, on the contrary, it combines them all.


User-agent: *

Disallow: /admin


Conclusion: I use the latter option. I like him as something – more appealing. You can also choose the robots that is exactly what you like. All of them correct and help you to promote your site.

About me:

SEO and PPC specialist with more than 10 years of experience. Founder of the school SEOpractic.