Forum Moderators: goodroi
User-agent: aipbot
Disallow: /
User-agent: aipbot/2-beta (aipbot dev; [aipbot.com;...] aipbot@aipbot.com)
Disallow: /
User-agent: aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)
Disallow: /
or can I just use:
User-agent: aipbot
Disallow: /
and that will cover the others?
Also, do I need to list the data in the parentheses, such as "(aipbot; [aipbot.com;...] aipbot@aipbot.com)"?
Thanks in advance
User-agent: FAST
User-agent: FAST Enterprise Crawler
User-agent: FAST-WebCrawler
User-agent: FAST MetaWeb Crawler
(Note: I found that none of the relentless, FAST-related robots ever heeded robots.txt or even backed off after 403 Forbiddens, so I ended up systematically nuking all originating IPs via firewall.)
Also, here are a few more bots you might think are the same because of big brother Googlebot, but they're different and may behave differently:
User-agent: Mediapartners-Google*
User-agent: Mediapartners-Google/2.1
User-agent: Googlebot-Image
And as Dijkgraaf mentioned, checking each bot's site (via URL info in the UA, or via major search engines) will provide you with their 'preferred' ID. So will checking your logs.
Finally, here are still more related, but not identical, siblings:
User-agent: Lycos
User-agent: Lycos_Spider_(T-Rex)
User-agent: Lycos_Spider_(modspider)
So many bots, so little time!:)