Forum Moderators: open

Message Too Old, No Replies

Ban msnbot and slurp

Correct format for the disallow robots.txt

         

jdhuk

10:29 pm on Jan 4, 2005 (gmt 0)

10+ Year Member



Unknowingly i purchased an expired domain thats doing well in Yahoo and MSN. However after waiting for a re-inclusion request at Google i gave up and purchased a new domain thats now in Google with the exactly the same content as the expired domain.

So to avoid a duplicate penalty in Yahoo and MSN i think the obvious solution is to ban both msnbot and yahoo slurp from crawling the new site in Google.

Please could someone give me the correct robots.txt for banning MSN and Yahoo bots from crawling.

At the moment i have ......

User-agent: *
Disallow:

The Contractor

11:05 pm on Jan 4, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



User-agent: Slurp
Disallow: /

User-agent: msnbot
Disallow: /

fiestagirl

11:09 pm on Jan 4, 2005 (gmt 0)

10+ Year Member



User-agent: slurp
Disallow: /

User-agent: msnbot
Disallow: /

Complete instruction: [robotstxt.org...]

Good thread: [webmasterworld.com...]