homepage Welcome to WebmasterWorld Guest from 54.227.12.219
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Why bother
Is it worth blocking spambots with robots.txt
WyrdoManx




msg:1527699
 12:48 pm on Nov 6, 2003 (gmt 0)

Hiya,

It seems some spam bots have known User Agents. I've seen sample robots.txt that block spam bots.

But really, is it worthwhile? Do these bots actually announce themselves? Do they obey robots.txt? It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?

Am I wrong?

 

Yidaki




msg:1527700
 12:52 pm on Nov 6, 2003 (gmt 0)

>Is it worth blocking spambots with robots.txt

Not really - htaccess is your friend.

Macguru




msg:1527701
 1:06 pm on Nov 6, 2003 (gmt 0)

Hi WyrdoManx,

But really, is it worthwhile?

No

Do these bots actually announce themselves?

Yes

Do they obey robots.txt?


No

It seems unlikely - so why should I risk sticking my head above the parapet and try to block them?

Yidaki is right. I guess you will have better results blocking bad bots with .htaccess. Do a site search [webmasterworld.com] on " perfect ban list " (without quotes). robots.txt is for good bots only.

Am I wrong?

You are 100 % righ.

hafnius




msg:1527702
 11:52 pm on Nov 8, 2003 (gmt 0)

Hi all

>Is it worth blocking spambots with robots.txt

I can see one scenario where it might be a good idea. When it comes to bots (and humans) that spams guestbooks. putting your guestbook dir. in the robots.txt will remove it from the SE and it will allow you to make a bot trap on the guestbook page for the bots that make it to the page anyway.

Just a thougt :)

Kind Regards
Hafnius

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved