homepage Welcome to WebmasterWorld Guest from 54.205.144.54
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google AdSense
Forum Library, Charter, Moderators: incrediBILL & jatar k & martinibuster

Google AdSense Forum

This 46 message thread spans 2 pages: < < 46 ( 1 [2]     
Bots Artificially Inflate AdSense Impressions
Ask, Snap, Alexa and Others Trigger AdSense Ads
incrediBILL




msg:3396087
 6:28 pm on Jul 16, 2007 (gmt 0)

Conventional wisdom has been that most web crawlers don't run javascript so therefore the spiders won't artificially inflate your AdSense ads.

That's no longer true.

The new trend is to augment the SERPs with screen shots in places like Ask, Snap and others, and the screen shots are made with javascript enabled so that javascript page elements (menus, etc.) are visible. That means your AdSense ads are also being included in those screen shots and possibly skewing your statistics.

Both Ask and Snap make screen shots of various pages in your site so it's not just a single home page image we're talking about. Snap has attempted to make upwards of 40K screen shorts on my site alone before I stopped counting. Ask seems to be a little more conservative in the quantity of their screen shots, but that could be growing pains as it remains to be seen if they'll screen shot entire sites and/or all sites they index.

What's unknown at this time is whether or not Google discounts the impressions being made by these screen shot bots at Ask or Snap, or whether those screen shots contribute to your lower CTRs and whatever penalties (smart pricing?) that may bring.

Imagine the impact it would have if a low traffic site with maybe 1,000 pages suddenly had 1,000 screen shots made all at once.

You would see a traffic surge with very low CTR, it would look abysmal.

Would the AdSense algorithms respond in this situation?

Worse yet, now that the screen shot genie is out of the bag and everyone wants them on their sites for a variety of purposes, more and more screen shot providers are now taking more and more screen shots to fill this need. Before long a percentage of your daily ad impressions will probably be nothing more than screen shots.

It would be nice if the AdSense Advisor could chime in and let us know what impact, if any, these screen shots are having on our AdSense stats.

 

incrediBILL




msg:3397316
 6:36 pm on Jul 17, 2007 (gmt 0)

just don't show the ads to the bots.

Not so easy.

In the case of Snap it ID's itself as Firefox on Linux, not as a bot, because that's the browser they use to make the screen shots. As a matter of fact most screen shot tools use a browser so that's what you would see instead of a bot name.

If there are ads shown in those screenshots then the services creating them are flouting the robots.txt guidelines

Technically robots.txt doesn't apply to a browser as it's a browser taking the screen shot.

Philosopher




msg:3397329
 6:55 pm on Jul 17, 2007 (gmt 0)

Technically robots.txt doesn't apply to a browser as it's a browser taking the screen shot.

That may not be completely accurate. While it's a browser taking the screen shot, it is a robot running the browser NOT a human, so one would think the robots.txt would apply (obviously it's not, but that doesn't mean it shouldn't).

From the robotstxt.org site...

The Robots Exclusion Protocol is a method that allows Web site administrators to indicate to visiting robots which parts of their site should not be visited by the robot.

incrediBILL




msg:3397346
 7:06 pm on Jul 17, 2007 (gmt 0)

While it's a browser taking the screen shot, it is a robot running the browser NOT a human

Technically the robot crawler behaved properly and didn't access areas it shouldn't.

Rendering an image of a page may require elements not crawled, but those elements still aren't being indexed by the crawler.

For instance, I don't let them index my page graphics in the image search so I block "\style\images" from the crawl but it's certainly reasonable for the screen shot to include those images.

It's a fuzzy gray area to say the least.

Back to the main topic:

I wonder how many AdWords advertisers with CPM ads are paying money just to appear in screen shots?

More importantly, now that this issue has been brought to their attention, will Google audit CPM ads shown on identifiable IP ranges for screen shot tools for Ask, Snap, etc and issue refunds to those advertisers?

[edited by: incrediBILL at 7:12 pm (utc) on July 17, 2007]

jomaxx




msg:3397365
 7:17 pm on Jul 17, 2007 (gmt 0)

BTW, here's a link to that thread I mentioned in which this phenomenon was discussed a few months ago.

[webmasterworld.com ]

incrediBILL




msg:3397384
 7:38 pm on Jul 17, 2007 (gmt 0)

BTW, here's a link to that thread I mentioned in which this phenomenon was discussed a few months ago

Good catch as I wasn't paying that much attention to Exalead, I block them :)

I also block Snap now, specifically nuked them when they went on a screen shot making rampage on my server. I wouldn't have minded so much but they cleared the browser cache on Firefox between every screen shot so it kept downloading all the supplemental files and images over and over which resulted in a nice bandwidth spike so I had to shut 'em down.

However, when I saw the screen shots on Ask, a major player, and AdSense was showing there as well, I knew it was time to start reviewing other screen shot sites to see the extent of the issue and find out what the impact was.

whoisgregg




msg:3397385
 7:39 pm on Jul 17, 2007 (gmt 0)

In regards to the robots.txt issue, visiting = crawling = downloading.

The robot that is rendering the page for a screenshot and downloading javascripts listed in robots.txt in order to perform that rendering is most assuredly flouting robots.txt.

Robots.txt isn't about indexing or archiving (although it'd be sweet if they could add that in), it's about downloading in the first place.

And, technically, they aren't doing that rendering with a browser per se. It's not like there's a computer somewhere with a new IE window popping open every second and a lego robot tapping the screenshot key. :)

incrediBILL




msg:3397400
 7:52 pm on Jul 17, 2007 (gmt 0)

And, technically, they aren't doing that rendering with a browser per se. It's not like there's a computer somewhere with a new IE window popping open every second and a lego robot tapping the screenshot key.

Not all work that way, but I have a batch screen shot utility I run and that's exactly what it does. It opens MSIE in a hidden window and generates a copy of screen when the page load is complete and it's fully automated.

whoisgregg




msg:3397464
 9:11 pm on Jul 17, 2007 (gmt 0)

True. I'd be pretty surprised if Ask or Snap were firing up an actual browser to do the screenshots. But, hey, I've heard some crazy stories about enterprise programming. :)

However, whether it's an actual browser instance or just the rendering library doesn't really affect the issue at hand. After all, most crawlers pass off their downloaded files for post-processing by an indexer. Fact is a bot post-processing files through additional programs does not make it any less of a bot.

Tastatura




msg:3397618
 1:08 am on Jul 18, 2007 (gmt 0)

True. I'd be pretty surprised if Ask or Snap were firing up an actual browser to do the screenshots. But, hey, I've heard some crazy stories about enterprise programming. :)

Actually I would be surprised if they didn't use actual browser - their objective (presumably) is to capture how page looks like when normal user (human) visits it (ok I am not talking text browsers, special circumstances, etc. - just plain old most common visit).

If you use 'ordinary browser (IE, FF,...) you already saved a lot of development time ( read money) with regards to fetching, rendering, etc., and it will produce desired result with respect how page looks like (meet the objective). Actually I did this while back - I used it on my site for research purposes - and it took me all of couple of hours to code bot like that. What I did, on Win box, is made bunch of calls to Internet Explorer's API (all documented by MS), and wholaaa - I have perfect fetcher, renderer, javascript executor, iframe viewer, etc. (and through API call you can set variable so that browser never displays and takes up screen real estate...)

I don't like nor support what, and how, 'big boys' are doing for snapshoting (spl?) a page, especially with regards to AdSense and I senceiraly hope that Google addresses the issue on therr end. BTW, Alexa offers for sale image snapshots of our pages...

[edited by: Tastatura at 1:11 am (utc) on July 18, 2007]

EricGiguere




msg:3399082
 2:32 pm on Jul 19, 2007 (gmt 0)

I'm not too worried about this, because it's something that Google can track and ignore. After all, the impressions will all be coming from a single IP address, or a small block of IP addresses, and will be spread across a large number of sites. Surely this kind of information will trip one of the fraud filters at some point, if it hasn't already, and then they can simply remove all impressions coming from those addresses.

Alioc




msg:3399655
 1:59 am on Jul 20, 2007 (gmt 0)

That's it Eric.

AdSenseAdvisor




msg:3401663
 4:32 am on Jul 23, 2007 (gmt 0)

I've conferred with our tech specialists and engineers and they assure me that our system is already aware of the bots from sites like Ask.com and Snap.com, including those that take screenshots. We're also constantly on the lookout for new sites and services that might inflate AdSense reports, so rest assured that your reports are safe and accurate.

-ASA

JohnDoealias




msg:3401694
 5:47 am on Jul 23, 2007 (gmt 0)

Cool. Thanks ASA.

Drag_Racer




msg:3401702
 6:03 am on Jul 23, 2007 (gmt 0)

That is good to hear from Google, but I think a lot of the responsibility lies with the bots owners too. If a company wants to run scripting on a page then they need to take things like this into consideration. Adsense is not the only thing which will become inflated. What about traffic stats from services like analytics and other ad networks, not to mention the hundreds of thousands of in house solutions being screwed up...

incrediBILL




msg:3402715
 4:06 am on Jul 24, 2007 (gmt 0)

Guess I'm confused why there aren't PSA's showing in the screen shots if AdSense is aware it's a screen shot bot from Ask or Snap, oh well.

martinibuster




msg:3402758
 5:02 am on Jul 24, 2007 (gmt 0)

Yah, it does make it appear as if it were an impression. However the AdSense team has built enough goodwill and credibility that if they say an impression gets filtered out so it doesn't count, I'm going to believe them.

This 46 message thread spans 2 pages: < < 46 ( 1 [2]
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google AdSense
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved