Welcome to WebmasterWorld Guest from 54.196.244.186

Forum Moderators: goodroi

Message Too Old, No Replies

Why bother

Is it worth blocking spambots with robots.txt

     
12:48 pm on Nov 6, 2003 (gmt 0)

New User

joined:Oct 20, 2003
posts:18
votes: 0


Hiya,

It seems some spam bots have known User Agents. I've seen sample robots.txt that block spam bots.

But really, is it worthwhile? Do these bots actually announce themselves? Do they obey robots.txt? It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?

Am I wrong?

12:52 pm on Nov 6, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 8, 2002
posts:2012
votes: 0


>Is it worth blocking spambots with robots.txt

Not really - htaccess is your friend.

1:06 pm on Nov 6, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 30, 2000
posts:3300
votes: 0


Hi WyrdoManx,

But really, is it worthwhile?

No

Do these bots actually announce themselves?

Yes

Do they obey robots.txt?


No

It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?

Yidaki is right. I guess you will have better results blocking bad bots with .htaccess. Do a site search [webmasterworld.com] on " perfect ban list " (without quotes). robots.txt is for good bots only.

Am I wrong?

You are 100 % righ.

11:52 pm on Nov 8, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 21, 2003
posts:108
votes: 0


Hi all

>Is it worth blocking spambots with robots.txt

I can see one scenario where it might be a good idea. When it comes to bots (and humans) that spams guestbooks. putting your guestbook dir. in the robots.txt will remove it from the SE and it will allow you to make a bot trap on the guestbook page for the bots that make it to the page anyway.

Just a thougt :)

Kind Regards
Hafnius