Forum Moderators: goodroi
Question: Will my robots.txt file recognize these two, if I already have formats for msnbot and slurp? Or do I need to add them, as in:
User-agent: msnbot
Disallow: /cgi-bin/
Disallow: /*.cgi$
User-agent: msnbot-media
Disallow: /cgi-bin/
Disallow: /*.cgi$
User-agent: slurp
Disallow: /cgi-bin/
Disallow: /*.cgi$
User-agent: YahooSeeker
Disallow: /cgi-bin/
Disallow: /*.cgi$
..................................