Forum Moderators: goodroi

Message Too Old, No Replies

Naming da' bots

Same place, different face

         

Reno

11:58 am on Oct 7, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I noticed recently that occasionally I'll get a bot from MS named "msnbot-media". And from Yahoo I'll get "YahooSeeker".

Question: Will my robots.txt file recognize these two, if I already have formats for msnbot and slurp? Or do I need to add them, as in:

User-agent: msnbot
Disallow: /cgi-bin/
Disallow: /*.cgi$

User-agent: msnbot-media
Disallow: /cgi-bin/
Disallow: /*.cgi$

User-agent: slurp
Disallow: /cgi-bin/
Disallow: /*.cgi$

User-agent: YahooSeeker
Disallow: /cgi-bin/
Disallow: /*.cgi$

..................................