homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robots.txt - Why?

 1:53 pm on Apr 20, 2004 (gmt 0)

Other than keeping SE bots out of areas that you don't want indexed (which I don't see as being absolutely vital for us either), is robots.txt really worth worrying about for the average site?

IOW, aren't the "bad" bots going to do bad things even though you have a file that tells them not to?

I'm sure that there are "bad" bots out there that will respect a robots.txt file, but I would think the vast majority of "bad" bots won't.

I've got my hands full trying to manage SEO stuff, I'm tempted to stop worrying about robots.txt and just remove it from our site or put an allow everything.



 1:59 pm on Apr 20, 2004 (gmt 0)

Whether it is worthwhile to prevent the "good" bots from spidering areas of your site is your own call. If it is worthwhile, then robots.txt is an easy way to do it. If it's not worthwhile, then don't bother.


 2:08 pm on Apr 20, 2004 (gmt 0)

A few reasons:

1) Prevent 404-Not Found entries in log from robots looking for robots.txt
2) Control bandwidth costs by limiting spidering.
3) Control presentation in various SERPs by controlling spidering.
4) Identify good vs. bad bots:

  • Disallow a 'page' in robots.txt
  • Rewrite that 'page' URI to a script.
  • Use that script to ban any bad-bot that fetches the Disallowed 'page' URL.


  • Global Options:
     top home search open messages active posts  

    Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved