Forum Moderators: goodroi
It seems some spam bots have known User Agents. I've seen sample robots.txt that block spam bots.
But really, is it worthwhile? Do these bots actually announce themselves? Do they obey robots.txt? It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?
Am I wrong?
Not really - htaccess is your friend.
But really, is it worthwhile?
No
Do these bots actually announce themselves?
Yes
Do they obey robots.txt?
It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?
Yidaki is right. I guess you will have better results blocking bad bots with .htaccess. Do a site search [webmasterworld.com] on " perfect ban list " (without quotes). robots.txt is for good bots only.
Am I wrong?
You are 100 % righ.
>Is it worth blocking spambots with robots.txt
I can see one scenario where it might be a good idea. When it comes to bots (and humans) that spams guestbooks. putting your guestbook dir. in the robots.txt will remove it from the SE and it will allow you to make a bot trap on the guestbook page for the bots that make it to the page anyway.
Just a thougt :)
Kind Regards
Hafnius
