Forum Moderators: martinibuster

Message Too Old, No Replies

Bots and IP Ranges to Not Block, Those Helping Advertisers Target Ads

Besides Gbot, Bbot, Slurp any other bots or IP ranges it pays to not block?

         

Webwork

1:32 pm on Sep 7, 2016 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I admin my own VPS. I've tweaked Apache to block many IP ranges, mostly blocking traffic from global regions that generate traffic bent on doing more harm than good. (Login hacks, injection attacks, etc.)

I've also blocked a number of bots that rip my sites . . err . . harvest data . . for the benefit of the bot operator's clients (rarely for my benefit). These are bots used to study links, evaluate ranking factors, evaluate content for competitive analysis, copyright infringement, etc. (I feel like putting out a sign "I'll give you access to my data/content IF you give me access to your data.")

I understand there are bots used to gather data for agencies that use/sell access to that data, intending to assist clients in targeting sites as "advertising opportunities", that is, placing ads on those sites. I'm not clear which bots are the good-guys-gals and what their IP ranges are. I don't wish to block the bots (potentially) doing me some direct good.

Please list any bots / bot IP ranges that you know are used, specifically, to assist advertisers in choosing which sites to target with their AdWords Content Network advertising dollars/inventory.

Which bots are beneficial to those deploying AdSense? Which bots (besides G's bots) may help publishers make money via Adsense targeting?

Webwork

3:37 pm on Sep 8, 2016 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



FWIW, since I may be misleading myself or beating a dead horse, here's an example of a digital marketing intelligence bot - "Clickagy Intelligence Bot v2" - that routinely shows up in my visitor data. Daily visitor to any and all sites. Not sure I want them using up CPU cycles or bandwidth.

Another example: GetIntent Crawler (http://getintent.com/bot.html)

Y'all know the likes of mj12/majestic and Moz's rogerbot. I don't see either bot doing me much direct good when the service the bot's operator offers is ~competitive intelligence. "Yeah, I want you're bot to pick me apart so potential competitors can use that data - your ripping of my site - to inform their actions".

Am I making ad-sense? Am I a knucklehead? (FWIW, I look at this as a sort of "set and forget" proposition, not something I work on every day. However, the damned bots visit every day, so . . )

Who do you block and which bot(s) - besides Googlebot, Bingbot and Slurp (and other relevant search engine bots) - do AdSensers any real good?

netmeg

5:52 pm on Sep 8, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



It's definitely a good question, but one I don't have an answer for, since I've mostly been concentrating on the bad guys.

topr8

8:15 pm on Sep 9, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



no, there are good bots - in this context - that analyse pages and sell the data ... except they are selling the data to companies that are advertising using the adsense or other networks.

here's a few with notes:

proximic, Analyses content for ads.
GrapeshotCrawler, Used to analyse pages that are serving ads, for a number of clients of this company.
NetSeer, analyses pages for premium publishers.
GetintentCrawler, seems to be analysis software for running ad campaigns.
WeSEE, seems to analyse images to classify the websites for adversiters.
adbeat_bot, analyses pages as targets for contextual advertisers.

but remember these bots should not be allowed without other tests like comparing the request ip to the known ip range of the bot.
as once they are mentioned publically of course, bot runners might thenuse those bot names for their nefarious bots!