Media.net bot ?

[Travis]

Since there are experts of bots identification, I am wondering if anyone know about a "media.net" bot (like there is an adsense bot).

A couple of times, I wrote to media.net to ask how they were fetching the pages of a site, to select ads, and if they had a "robot" doing this. I asked this to be sure to whistle it. They claimed they had no "bot", but they didn't tell me how they were identifying the content of a page (industrial secrets?).

But they must have a way, to crawl pages, no? Or may be they manually review each site, and set the themes of the site globally ?

ps: I meant "media.net" in the title, if someone can fix it.

[keyplyr]

I keep a diligent eye on most all UAs and have not identified anything calling itself "media.net" bot.

There are thousands and thousands of generically named scraping agents [webmasterworld.com] so they are likely using one of those non-identified UAs.

For example:

http-client

[Travis]

I don't have the skill and experience of keyplyr, but I have a "feeling" that "may be", "eventually", media.net has something running from xlhost.com .

Since , I started the topic, I made a little test. I created a page, without incoming links, with a media.net code and without any bot/ip filtering. I visited the page myself, and in the same second, there was a hit from an IP located at xlhost.com , with a referrer field being set to the page itself , and with a user agent, which was "mine" (I tried changing the user agent string to check). So I guess it means media.net is running a bot that way. (and may be from other networks, but I tried several times, and each time it was different IPs from xlhost.com).

[keyplyr]

Yeah, many bots masquerade as human browsers.

[Travis]

I am not going to bother much longer, but just in case, someone, one day is investigating on the subject... , EACH time I visit my test page, there is a hit from an IP at xlhost, and the ad starts loading only after the hit from xlhost occurred, which as a result is making ads slow to show.

[keyplyr]

EACH time I visit my test page, there is a hit from an IP at xlhost

Because you have a tracking cookie on your hard drive set through your browser.

[Travis]

I didn't understand.