Forum Moderators: goodroi

Message Too Old, No Replies

Are different User-agent entries needed for versions of similar robot

         

Ceyx

7:06 am on Jan 31, 2006 (gmt 0)

10+ Year Member



Do in need to have a separate User-agent entry for multiple versions of the same bot? As in:

User-agent: aipbot
Disallow: /

User-agent: aipbot/2-beta (aipbot dev; [aipbot.com;...] aipbot@aipbot.com)
Disallow: /

User-agent: aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)
Disallow: /

or can I just use:

User-agent: aipbot
Disallow: /

and that will cover the others?

Also, do I need to list the data in the parentheses, such as "(aipbot; [aipbot.com;...] aipbot@aipbot.com)"?

Thanks in advance

Dijkgraaf

8:36 am on Jan 31, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



According to their website "user agent string aipbot"
so "User-agent: aipbot" is correct thing to have, you don't need to worry about the version or full UA string.

Pfui

2:29 am on Feb 1, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Sometimes what appear to be versions of the same bot are not. For example, these look similar, but they're different critters:

User-agent: FAST

User-agent: FAST Enterprise Crawler

User-agent: FAST-WebCrawler

User-agent: FAST MetaWeb Crawler

(Note: I found that none of the relentless, FAST-related robots ever heeded robots.txt or even backed off after 403 Forbiddens, so I ended up systematically nuking all originating IPs via firewall.)

Also, here are a few more bots you might think are the same because of big brother Googlebot, but they're different and may behave differently:

User-agent: Mediapartners-Google*

User-agent: Mediapartners-Google/2.1

User-agent: Googlebot-Image

And as Dijkgraaf mentioned, checking each bot's site (via URL info in the UA, or via major search engines) will provide you with their 'preferred' ID. So will checking your logs.

Finally, here are still more related, but not identical, siblings:

User-agent: Lycos

User-agent: Lycos_Spider_(T-Rex)

User-agent: Lycos_Spider_(modspider)

So many bots, so little time!:)