joined:Mar 16, 2005
I have never used a robots.txt file. My site is 5 years old and I've have had no problems. However in the last month or so I am seeing some strange things. Yahoo is hitting my site 5X as much as google and I'm wondering if it would be any benefit for me to add a robots.txt?
Also In my errors log it shows google is unable to crawl some pages do to a robots.txt file of which I do not have one....