Hi all, Just want to find out what others think of robots.txt. the question is " Is robots.txt helpful or not? " I know we all talk about it and use it but I read posts here and elsewhere where people say yes it works and no it does not work. So what is it? if anyone has any input, please let me know. Thanks
robots.txt is essential to my site... I've got areas of the site I don't want spidered, and robots.txt is the only way to prevent 'good' spiders (like googlebot) from indexing those areas, while still allowing them to index the rest of the material.
robots.txt DOES work, IF:
1. You have all of your statements formatted correctly. Yesterday I had a spider plow through an area I *thought* was blocked, but since I had the line blocking that area written incorrectly, it didn't work.
After emailing the spider's owner (antarcti.ca), determining the problem and fixing it, the terrifically nice folks at antarcti.ca's tech dept. sent their spider through again, and my robots.txt worked like a charm.
2. The robot in question follows robots.txt conventions. All of the major search engines and important/good spiders DO follow robots.txt instructions...
Any robot I find that doesn't request a robots.txt file, or ignores *properly formatted* directions therein, is banned form my site via htaccess, and loud complaints are sent to its owner.
Thanks mivox for the input. Now how do I go by exactly trying to write a robots.txt that I know will work. Also if any of ya'll have a web resource that you think is very discriptive please post it so that I may take a look at it.