Allowing in the Good Bots While Kicking Out the Bad Bots

Which bots inform Adwords Advertisers in a way that profits publishers?

[Webwork]

I've been doing my best to kick to the curb as many bots that "don't make me money" as possible. You know, all the bots that are about collecting data for back-link analyzers / SEO tools, competitive analysis bots, etc.

My question is, beyond Googlebot, Bingbot or Slurp which bots actually do publishers any good? Some of the bots in the "ad placement analytics" space are relentless, hitting page after page, again and again. This can slow a website or server AND, as we have been told, speed is a factor in ranking and ranking affects traffic which affects income . . . soooooooo . . bot . . show me the money or I'll show you the door.

Which bots do you "let in" and which do you kick to the curb?

Has anyone "played around with this" and have any insight to share? Did you ban bots x,y & z (remember, I'm NOT talking about G/B/Y bots) and did that affect your income?

Wouldn't it be nice if the geniuses running these bots actually shared some of their data with publishers to demonstrate how they are actually doing "us publishers" some good. "Share data"? Yeah, as in "We identified Site X. Based upon our data we recommended to our client C that they target Site X for ad placements / targeting. Site X reaped the following benefit." Yeah, like <that.

[not2easy]

I think the most thorough general discussion was this one (2014): [webmasterworld.com...]

Not a copy/paste resource it does require some thoughtful analysis, but there is a lot of info in that longish discussion.

[Webwork]

n2e - Thanks. That thread primarily speaks to blocking non-North American traffic and is quite a good resource for that topic. However, I'm looking at a somewhat different issue: bots created and deployed, by agencies, which bots collect data that is used to advise agency clients (online advertisers) regarding "which websites the advertiser should target". Such bots, regardless of IP, could be beneficial to website operators / publishers IF the data collected supports targeting the publisher's website.

For example, one bot I see A LOT of is named "get intent". I looked up the bot and it appears to be a bot whose purpose is to collect data helpful to advertisers. (I could be wrong about this.) For the moment, give their bot's voracious appetite (daily multi-page or full site hits), I have them blocked.

[lucy24]

Minimalist question: Do they honor robots.txt? If they don't, they would have to be performing a very, very important service in order to merit an exemption. So that wipes out a lot of robots right away.

I assume you're not asking about AdSense's internal robots, since they would presumably tell you if they tried to get in and couldn't.

[tangor]

Like lucy24, a good bot respects robots.txt ...

What I would rather see ... than guess ... is a list of bots KNOWN to provide publisher positioning to advertisers. Sans that, in my opinion, there are no good bots, other than my whitelist. :)

[keyplyr]

What is a good bot? Only you can determine that. If you sell ad space or publish ads (example: Adsense) you may want to allow the marketing & advertising companies (and their many contributors) access. If you don't offer ads on your site, these are often considered the bad guys.

I remember when social media took off. Many members here trashed all the new bots crawling their sites. With a bit of research, these resources can be a huge traffic stream.