To use or not to use a robots.txt file

[con771]

I have never used a robots.txt file. My site is 5 years old and I've have had no problems. However in the last month or so I am seeing some strange things. Yahoo is hitting my site 5X as much as google and I'm wondering if it would be any benefit for me to add a robots.txt?

Also In my errors log it shows google is unable to crawl some pages do to a robots.txt file of which I do not have one....

Any ideas

[bilalseo]

well using robots.txt file is sometimes better and sometimes it reflects bad impact if we can see a pure crawling. the basic purpose of robots.txt file is blocking site's restricted areas otherthan no benefit you can get from it. You must re-study your pages for which google felt busy while crawling, might there is some script you are using that reflects out nature in respect of crawling. One more thing search engines doesn't cares if you are using robots.txt file, they jump over it and pass through without knowing what you are seeking for.
Now a days google agent with Mozila 5.0 with a dispute, that most of the people saying that it revels in respect of spamming.

thanks,

bilal

[bilalseo]

I'm still looking Goodroi comments....