Marketing related robots?

[JorgeV]

Hello,

I am wondering if there is a list of "legitimate" robots, which are used for marketing purpose, or by ad networks? In other words, a list of robots you shouldn't block, if you are displaying ads on your site?

I only know very few:

Adsense :
User agent: Mediapartners-Google
[support.google.com...]
[support.google.com...]

Ahrefs:
User agent: Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)
[ahrefs.com...]

Comscore/Proximic
User agent: Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)
[comscore.com...]

There are certainly more?

[lammert]

This question came up before in Marketing / Ads related bots [webmasterworld.com] and the answer of the then-moderator of this forum was "there is no list for obvious reasons", whatever those obvious reasons may be.

[Dimitri]

The problem is, there are new robots coming all the time, new companies appearing and disappearing.

My tips :

All clients claiming to be a robot, have to have visited the robots.txt file first. So log requests to robots.txt to identify IP / UA which are potentially "good" bots.

From this list, check each robot, they must have a page explaining who they are and why they crawl web sites. The page, has to feature a given number of information beside its use. It has to provide the UA, as well as a way to authenticate the robot, this can be IP rages, reverse IP, etc...

Then, make your mind to see if this robot can "help" or "hurt" your site.

Now, some companies are lazy, I say, too bad for them, come back when you'll show your professionalism with a robot well coded.