homepage Welcome to WebmasterWorld Guest from 54.227.171.163
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Why bother
Is it worth blocking spambots with robots.txt
WyrdoManx



 
Msg#: 189 posted 12:48 pm on Nov 6, 2003 (gmt 0)

Hiya,

It seems some spam bots have known User Agents. I've seen sample robots.txt that block spam bots.

But really, is it worthwhile? Do these bots actually announce themselves? Do they obey robots.txt? It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?

Am I wrong?

 

Yidaki

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 189 posted 12:52 pm on Nov 6, 2003 (gmt 0)

>Is it worth blocking spambots with robots.txt

Not really - htaccess is your friend.

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 189 posted 1:06 pm on Nov 6, 2003 (gmt 0)

Hi WyrdoManx,

But really, is it worthwhile?

No

Do these bots actually announce themselves?

Yes

Do they obey robots.txt?


No

It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?

Yidaki is right. I guess you will have better results blocking bad bots with .htaccess. Do a site search [webmasterworld.com] on " perfect ban list " (without quotes). robots.txt is for good bots only.

Am I wrong?

You are 100 % righ.

hafnius

10+ Year Member



 
Msg#: 189 posted 11:52 pm on Nov 8, 2003 (gmt 0)

Hi all

>Is it worth blocking spambots with robots.txt

I can see one scenario where it might be a good idea. When it comes to bots (and humans) that spams guestbooks. putting your guestbook dir. in the robots.txt will remove it from the SE and it will allow you to make a bot trap on the guestbook page for the bots that make it to the page anyway.

Just a thougt :)

Kind Regards
Hafnius

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved