Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt - Why?



1:53 pm on Apr 20, 2004 (gmt 0)

10+ Year Member

Other than keeping SE bots out of areas that you don't want indexed (which I don't see as being absolutely vital for us either), is robots.txt really worth worrying about for the average site?

IOW, aren't the "bad" bots going to do bad things even though you have a file that tells them not to?

I'm sure that there are "bad" bots out there that will respect a robots.txt file, but I would think the vast majority of "bad" bots won't.

I've got my hands full trying to manage SEO stuff, I'm tempted to stop worrying about robots.txt and just remove it from our site or put an allow everything.


1:59 pm on Apr 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Whether it is worthwhile to prevent the "good" bots from spidering areas of your site is your own call. If it is worthwhile, then robots.txt is an easy way to do it. If it's not worthwhile, then don't bother.


2:08 pm on Apr 20, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

A few reasons:

1) Prevent 404-Not Found entries in log from robots looking for robots.txt
2) Control bandwidth costs by limiting spidering.
3) Control presentation in various SERPs by controlling spidering.
4) Identify good vs. bad bots:

  • Disallow a 'page' in robots.txt
  • Rewrite that 'page' URI to a script.
  • Use that script to ban any bad-bot that fetches the Disallowed 'page' URL.



    Featured Threads

    Hot Threads This Week

    Hot Threads This Month