Welcome to WebmasterWorld Guest from 18.208.159.25

Forum Moderators: martinibuster

Message Too Old, No Replies

Allowing in the Good Bots While Kicking Out the Bad Bots

Which bots inform Adwords Advertisers in a way that profits publishers?

     
6:54 pm on Jun 18, 2016 (gmt 0)

Moderator

WebmasterWorld Administrator webwork is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 2, 2003
posts:8055
votes: 101


I've been doing my best to kick to the curb as many bots that "don't make me money" as possible. You know, all the bots that are about collecting data for back-link analyzers / SEO tools, competitive analysis bots, etc.

My question is, beyond Googlebot, Bingbot or Slurp which bots actually do publishers any good? Some of the bots in the "ad placement analytics" space are relentless, hitting page after page, again and again. This can slow a website or server AND, as we have been told, speed is a factor in ranking and ranking affects traffic which affects income . . . soooooooo . . bot . . show me the money or I'll show you the door.

Which bots do you "let in" and which do you kick to the curb?

Has anyone "played around with this" and have any insight to share? Did you ban bots x,y & z (remember, I'm NOT talking about G/B/Y bots) and did that affect your income?

Wouldn't it be nice if the geniuses running these bots actually shared some of their data with publishers to demonstrate how they are actually doing "us publishers" some good. "Share data"? Yeah, as in "We identified Site X. Based upon our data we recommended to our client C that they target Site X for ad placements / targeting. Site X reaped the following benefit." Yeah, like <that.
7:19 pm on June 18, 2016 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4527
votes: 350


I think the most thorough general discussion was this one (2014): [webmasterworld.com...]

Not a copy/paste resource it does require some thoughtful analysis, but there is a lot of info in that longish discussion.
10:40 pm on June 18, 2016 (gmt 0)

Moderator

WebmasterWorld Administrator webwork is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 2, 2003
posts:8055
votes: 101


n2e - Thanks. That thread primarily speaks to blocking non-North American traffic and is quite a good resource for that topic. However, I'm looking at a somewhat different issue: bots created and deployed, by agencies, which bots collect data that is used to advise agency clients (online advertisers) regarding "which websites the advertiser should target". Such bots, regardless of IP, could be beneficial to website operators / publishers IF the data collected supports targeting the publisher's website.

For example, one bot I see A LOT of is named "get intent". I looked up the bot and it appears to be a bot whose purpose is to collect data helpful to advertisers. (I could be wrong about this.) For the moment, give their bot's voracious appetite (daily multi-page or full site hits), I have them blocked.
3:08 am on June 19, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15897
votes: 876


Minimalist question: Do they honor robots.txt? If they don't, they would have to be performing a very, very important service in order to merit an exemption. So that wipes out a lot of robots right away.

I assume you're not asking about AdSense's internal robots, since they would presumably tell you if they tried to get in and couldn't.
4:09 am on June 19, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10472
votes: 1099


Like lucy24, a good bot respects robots.txt ...

What I would rather see ... than guess ... is a list of bots KNOWN to provide publisher positioning to advertisers. Sans that, in my opinion, there are no good bots, other than my whitelist. :)
12:55 pm on June 19, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


What is a good bot? Only you can determine that. If you sell ad space or publish ads (example: Adsense) you may want to allow the marketing & advertising companies (and their many contributors) access. If you don't offer ads on your site, these are often considered the bad guys.

I remember when social media took off. Many members here trashed all the new bots crawling their sites. With a bit of research, these resources can be a huge traffic stream.