homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

spiders list

5+ Year Member

Msg#: 3617813 posted 6:24 am on Apr 3, 2008 (gmt 0)

So which bots do you stop? Also why and how have you done it?




WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3617813 posted 12:56 pm on Apr 9, 2008 (gmt 0)

hi hugh,

i prefer to allow most bots on my sites. it may cost me some bandwidth but i prefer to focus my time on making money and not limiting access to bots. when i do want to block a bot i use robots.txt (if they are a nice bot) and also i use htaccss (if they are a bad bot).

please remember that robots.txt is a voluntary protocol. robots.txt will not stop bots that are broken or intentionally programmed to ignore robots.txt. if you want to protect data you should use htaccess or a similar alternative.


5+ Year Member

Msg#: 3617813 posted 12:49 pm on Jun 6, 2008 (gmt 0)

I was just checking into haidu or is it haldu (must change fonts on browser) and gigabot. google led me here. got the registration page and figured I should at least identify myself as a friendly real human.

I might fire up my website again, had no success with it for a year and a whole lot of work deleting messages to my Blog, all of them from a .info yadda yadda you prob'ly know the rest.

The same ones which invaded yahoo groups for the past couple of years.

Of course this is one of my areas of interest, so I'll probably be checking back. Unless my new site keeps me too busy making cash, that is. Thanks for putting up with me.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved