| This 46 message thread spans 2 pages: < < 46 ( 1  ) || |
|Bots Artificially Inflate AdSense Impressions|
Ask, Snap, Alexa and Others Trigger AdSense Ads
| 6:28 pm on Jul 16, 2007 (gmt 0)|
That's no longer true.
Both Ask and Snap make screen shots of various pages in your site so it's not just a single home page image we're talking about. Snap has attempted to make upwards of 40K screen shorts on my site alone before I stopped counting. Ask seems to be a little more conservative in the quantity of their screen shots, but that could be growing pains as it remains to be seen if they'll screen shot entire sites and/or all sites they index.
What's unknown at this time is whether or not Google discounts the impressions being made by these screen shot bots at Ask or Snap, or whether those screen shots contribute to your lower CTRs and whatever penalties (smart pricing?) that may bring.
Imagine the impact it would have if a low traffic site with maybe 1,000 pages suddenly had 1,000 screen shots made all at once.
You would see a traffic surge with very low CTR, it would look abysmal.
Would the AdSense algorithms respond in this situation?
Worse yet, now that the screen shot genie is out of the bag and everyone wants them on their sites for a variety of purposes, more and more screen shot providers are now taking more and more screen shots to fill this need. Before long a percentage of your daily ad impressions will probably be nothing more than screen shots.
It would be nice if the AdSense Advisor could chime in and let us know what impact, if any, these screen shots are having on our AdSense stats.
| 6:36 pm on Jul 17, 2007 (gmt 0)|
|just don't show the ads to the bots. |
Not so easy.
In the case of Snap it ID's itself as Firefox on Linux, not as a bot, because that's the browser they use to make the screen shots. As a matter of fact most screen shot tools use a browser so that's what you would see instead of a bot name.
|If there are ads shown in those screenshots then the services creating them are flouting the robots.txt guidelines |
Technically robots.txt doesn't apply to a browser as it's a browser taking the screen shot.
| 6:55 pm on Jul 17, 2007 (gmt 0)|
|Technically robots.txt doesn't apply to a browser as it's a browser taking the screen shot. |
That may not be completely accurate. While it's a browser taking the screen shot, it is a robot running the browser NOT a human, so one would think the robots.txt would apply (obviously it's not, but that doesn't mean it shouldn't).
From the robotstxt.org site...
|The Robots Exclusion Protocol is a method that allows Web site administrators to indicate to visiting robots which parts of their site should not be visited by the robot. |
| 7:06 pm on Jul 17, 2007 (gmt 0)|
|While it's a browser taking the screen shot, it is a robot running the browser NOT a human |
Technically the robot crawler behaved properly and didn't access areas it shouldn't.
Rendering an image of a page may require elements not crawled, but those elements still aren't being indexed by the crawler.
For instance, I don't let them index my page graphics in the image search so I block "\style\images" from the crawl but it's certainly reasonable for the screen shot to include those images.
It's a fuzzy gray area to say the least.
Back to the main topic:
I wonder how many AdWords advertisers with CPM ads are paying money just to appear in screen shots?
More importantly, now that this issue has been brought to their attention, will Google audit CPM ads shown on identifiable IP ranges for screen shot tools for Ask, Snap, etc and issue refunds to those advertisers?
[edited by: incrediBILL at 7:12 pm (utc) on July 17, 2007]
| 7:17 pm on Jul 17, 2007 (gmt 0)|
BTW, here's a link to that thread I mentioned in which this phenomenon was discussed a few months ago.
| 7:38 pm on Jul 17, 2007 (gmt 0)|
|BTW, here's a link to that thread I mentioned in which this phenomenon was discussed a few months ago |
Good catch as I wasn't paying that much attention to Exalead, I block them :)
I also block Snap now, specifically nuked them when they went on a screen shot making rampage on my server. I wouldn't have minded so much but they cleared the browser cache on Firefox between every screen shot so it kept downloading all the supplemental files and images over and over which resulted in a nice bandwidth spike so I had to shut 'em down.
However, when I saw the screen shots on Ask, a major player, and AdSense was showing there as well, I knew it was time to start reviewing other screen shot sites to see the extent of the issue and find out what the impact was.
| 7:39 pm on Jul 17, 2007 (gmt 0)|
In regards to the robots.txt issue, visiting = crawling = downloading.
Robots.txt isn't about indexing or archiving (although it'd be sweet if they could add that in), it's about downloading in the first place.
And, technically, they aren't doing that rendering with a browser per se. It's not like there's a computer somewhere with a new IE window popping open every second and a lego robot tapping the screenshot key. :)
| 7:52 pm on Jul 17, 2007 (gmt 0)|
|And, technically, they aren't doing that rendering with a browser per se. It's not like there's a computer somewhere with a new IE window popping open every second and a lego robot tapping the screenshot key. |
Not all work that way, but I have a batch screen shot utility I run and that's exactly what it does. It opens MSIE in a hidden window and generates a copy of screen when the page load is complete and it's fully automated.
| 9:11 pm on Jul 17, 2007 (gmt 0)|
True. I'd be pretty surprised if Ask or Snap were firing up an actual browser to do the screenshots. But, hey, I've heard some crazy stories about enterprise programming. :)
However, whether it's an actual browser instance or just the rendering library doesn't really affect the issue at hand. After all, most crawlers pass off their downloaded files for post-processing by an indexer. Fact is a bot post-processing files through additional programs does not make it any less of a bot.
| 1:08 am on Jul 18, 2007 (gmt 0)|
|True. I'd be pretty surprised if Ask or Snap were firing up an actual browser to do the screenshots. But, hey, I've heard some crazy stories about enterprise programming. :) |
Actually I would be surprised if they didn't use actual browser - their objective (presumably) is to capture how page looks like when normal user (human) visits it (ok I am not talking text browsers, special circumstances, etc. - just plain old most common visit).
I don't like nor support what, and how, 'big boys' are doing for snapshoting (spl?) a page, especially with regards to AdSense and I senceiraly hope that Google addresses the issue on therr end. BTW, Alexa offers for sale image snapshots of our pages...
[edited by: Tastatura at 1:11 am (utc) on July 18, 2007]
| 2:32 pm on Jul 19, 2007 (gmt 0)|
I'm not too worried about this, because it's something that Google can track and ignore. After all, the impressions will all be coming from a single IP address, or a small block of IP addresses, and will be spread across a large number of sites. Surely this kind of information will trip one of the fraud filters at some point, if it hasn't already, and then they can simply remove all impressions coming from those addresses.
| 1:59 am on Jul 20, 2007 (gmt 0)|
That's it Eric.
| 4:32 am on Jul 23, 2007 (gmt 0)|
I've conferred with our tech specialists and engineers and they assure me that our system is already aware of the bots from sites like Ask.com and Snap.com, including those that take screenshots. We're also constantly on the lookout for new sites and services that might inflate AdSense reports, so rest assured that your reports are safe and accurate.
| 5:47 am on Jul 23, 2007 (gmt 0)|
Cool. Thanks ASA.
| 6:03 am on Jul 23, 2007 (gmt 0)|
That is good to hear from Google, but I think a lot of the responsibility lies with the bots owners too. If a company wants to run scripting on a page then they need to take things like this into consideration. Adsense is not the only thing which will become inflated. What about traffic stats from services like analytics and other ad networks, not to mention the hundreds of thousands of in house solutions being screwed up...
| 4:06 am on Jul 24, 2007 (gmt 0)|
Guess I'm confused why there aren't PSA's showing in the screen shots if AdSense is aware it's a screen shot bot from Ask or Snap, oh well.
| 5:02 am on Jul 24, 2007 (gmt 0)|
Yah, it does make it appear as if it were an impression. However the AdSense team has built enough goodwill and credibility that if they say an impression gets filtered out so it doesn't count, I'm going to believe them.
| This 46 message thread spans 2 pages: < < 46 ( 1  ) |