Forum Moderators: goodroi

Message Too Old, No Replies

can put two robots at one line at robot.txt to ban two spiders?

robots.txt, ban google, yahoo bots

         

johnlim9988

1:58 pm on Feb 6, 2007 (gmt 0)

10+ Year Member



Hi,
is the following correct robots.txt to ban googlebot and yahoo slurp, can put two robots at one line? As this duplciated site still have some MSN traffic so we decide to ban google and yahoo.

*****************************************
User-agent: googlebot, Slurp
Disallow: /
*****************************************

Thanks,

jdMorgan

2:49 pm on Feb 6, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Not recommended...

You can use this, but only with major search engines:


User-agent: googlebot
User-agent: Slurp
Disallow: /


Jim

johnlim9988

9:49 pm on Feb 6, 2007 (gmt 0)

10+ Year Member



Is this better?

User-agent: googlebot
Disallow: /
User-agent: Slurp
Disallow: /

phranque

3:54 am on Feb 7, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



according to the robots.txt standard [robotstxt.org] this is better:
User-agent: googlebot
Disallow: /

User-agent: Slurp
Disallow: /

with a space between records...