I promised you a guide to successfully create the robots.txt file and here it is:
1) First of all you have to know what a robots.txt file: it is a file that is read by the spider search engines and tells them what areas of the site can access and what not, also indicates the 'where are located the address of your sitemap.
2) To create one, first create a plain text file called robots.txt .
3) Open the file you created and write the first line " User-agent: " after this field you must enter the name of the spider that you want enforce your rules (the names you can find them here ), although I suggest you enter "* " instead of names so the rules will apply to all existing and spiders.
4) Go over and write " Disallow: , this field must contain all addresses that do not want to be inspected by the spider (write only the second part of 'that is the address after the' extension of domain, eg. / admin), you must write a "Disallow:" for each URL you want to hide.
5) Finally go over and write " Sitemap: , this field must contain the url, this time complete of your sitemap (eg www.miosito.com / sitemap.xml).
6) If you want to post comments used # and write your comment will not be read by the spider.
7) Finally upload your file in the root folder of your site (eg www.miosito.com / robots.txt), will be found automatically by the spider.
last thing: if you want to completely deny the 'access to your site in the " Disallow: " Enter only / .
0 comments:
Post a Comment