Welcome to WebmasterWorld Guest from 50.19.156.19

Forum Moderators: goodroi

Message Too Old, No Replies

Why bother

Is it worth blocking spambots with robots.txt

     

WyrdoManx

12:48 pm on Nov 6, 2003 (gmt 0)



Hiya,

It seems some spam bots have known User Agents. I've seen sample robots.txt that block spam bots.

But really, is it worthwhile? Do these bots actually announce themselves? Do they obey robots.txt? It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?

Am I wrong?

Yidaki

12:52 pm on Nov 6, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>Is it worth blocking spambots with robots.txt

Not really - htaccess is your friend.

Macguru

1:06 pm on Nov 6, 2003 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Hi WyrdoManx,

But really, is it worthwhile?

No

Do these bots actually announce themselves?

Yes

Do they obey robots.txt?


No

It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?

Yidaki is right. I guess you will have better results blocking bad bots with .htaccess. Do a site search [webmasterworld.com] on " perfect ban list " (without quotes). robots.txt is for good bots only.

Am I wrong?

You are 100 % righ.

hafnius

11:52 pm on Nov 8, 2003 (gmt 0)

10+ Year Member



Hi all

>Is it worth blocking spambots with robots.txt

I can see one scenario where it might be a good idea. When it comes to bots (and humans) that spams guestbooks. putting your guestbook dir. in the robots.txt will remove it from the SE and it will allow you to make a bot trap on the guestbook page for the bots that make it to the page anyway.

Just a thougt :)

Kind Regards
Hafnius