Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

To use or not to use a robots.txt file

12:11 pm on May 20, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 16, 2005
votes: 0

I have never used a robots.txt file. My site is 5 years old and I've have had no problems. However in the last month or so I am seeing some strange things. Yahoo is hitting my site 5X as much as google and I'm wondering if it would be any benefit for me to add a robots.txt?

Also In my errors log it shows google is unable to crawl some pages do to a robots.txt file of which I do not have one....

Any ideas

6:01 pm on May 20, 2008 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
votes: 0

well using robots.txt file is sometimes better and sometimes it reflects bad impact if we can see a pure crawling. the basic purpose of robots.txt file is blocking site's restricted areas otherthan no benefit you can get from it. You must re-study your pages for which google felt busy while crawling, might there is some script you are using that reflects out nature in respect of crawling. One more thing search engines doesn't cares if you are using robots.txt file, they jump over it and pass through without knowing what you are seeking for.
Now a days google agent with Mozila 5.0 with a dispute, that most of the people saying that it revels in respect of spamming.



6:03 pm on May 20, 2008 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
posts: 303
votes: 0

I'm still looking Goodroi comments....

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members