Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt - Why?

1:53 pm on Apr 20, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 6, 2003
votes: 0

Other than keeping SE bots out of areas that you don't want indexed (which I don't see as being absolutely vital for us either), is robots.txt really worth worrying about for the average site?

IOW, aren't the "bad" bots going to do bad things even though you have a file that tells them not to?

I'm sure that there are "bad" bots out there that will respect a robots.txt file, but I would think the vast majority of "bad" bots won't.

I've got my hands full trying to manage SEO stuff, I'm tempted to stop worrying about robots.txt and just remove it from our site or put an allow everything.

1:59 pm on Apr 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 3, 2003
votes: 0

Whether it is worthwhile to prevent the "good" bots from spidering areas of your site is your own call. If it is worthwhile, then robots.txt is an easy way to do it. If it's not worthwhile, then don't bother.
2:08 pm on Apr 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0

A few reasons:

1) Prevent 404-Not Found entries in log from robots looking for robots.txt
2) Control bandwidth costs by limiting spidering.
3) Control presentation in various SERPs by controlling spidering.
4) Identify good vs. bad bots:

  • Disallow a 'page' in robots.txt
  • Rewrite that 'page' URI to a script.
  • Use that script to ban any bad-bot that fetches the Disallowed 'page' URL.



    Join The Conversation

    Moderators and Top Contributors

    Hot Threads This Week

    Featured Threads

    Free SEO Tools

    Hire Expert Members