homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google AdSense
Forum Library, Charter, Moderators: incrediBILL & jatar k & martinibuster

Google AdSense Forum

Yahoo Slurp parses Adsense Code
CTR tanked

 11:15 pm on Nov 25, 2012 (gmt 0)

Just investigated why my Adsense CTR suddenly tanked (not earnings) and found out I unblocked some IPs for Slurp a few days ago. They really seem to parse HTML including Javascript. This means in your Adsense stats this counts as a Page View.

You can even check in Analytics with Advanced Segment: Direct Traffic and you will find Slurp counts as direct traffic or direct visitor.

Look at this: - - [25/Nov/2012:13:26:06 -0500] "GET /texas/widget.html HTTP/1.1" 200 5978 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; [help.yahoo.com...]

and every time Slurp appears in my logs it is followed by the Mediapartners bot: - - [25/Nov/2012:13:26:07 -0500] "GET /texas/widget.html HTTP/1.1" 200 5978 "-" "Mediapartners-Google"



 11:51 pm on Nov 25, 2012 (gmt 0)

Since Slurp has been showing up in Statcounter for at least a year or more, I figured it was executing javascript.


 2:52 am on Nov 26, 2012 (gmt 0)

fwiw: I can't be bothered to keep track, so I just block 'em by name:

BrowserMatch Yahoo keep_out

pretty well covers it ;)

This probably dates back to last January when I looked closely at robotic behavior and ended up tossing several into the "because I don't like your face" bin.


 4:58 am on Nov 26, 2012 (gmt 0)

Bing and Yahoo are still my main constant and reliable best converting traffic sources (Bing/Yahoo 93%, Goog 5%).

Blocking them would not be wise therefore I unblocked some of their IPs. I don't understand why Slurp is still alive, they have Bingbot which is really easy to handle with the crawl control in their WMT. It seems Bingbot only fetches html files and Slurp the rest.

But at least Goog should block Slurp from Analytics (no, this is not direct traffic LOL) and Adsense, it should be really easy for them.


 7:01 am on Nov 26, 2012 (gmt 0)

They fetch complete packages. In my case this means that every request is followed by a fetch of errorstyles.css. Every single time. Which in turn means they actually go to the 403 page, rather than swallowing the numbers and moving on. Very un-robotlike. Rarely they also get the favicon.

Random hopping through raw logs brought up only one request for robots.txt, and that was in October. Of 2011. (I later found a day in March where all they asked for was robots.txt. Two redirects, four successful pickups. I guess they freeze them for later.)

Along the way I was staggered to discover that they've been steadily asking for the same handful of pages over and over again, several times a day. Maybe they lost their shopping list and these are the only titles they can remember.

I have also now remembered that it was YahooCacheSystem that originally offended me. Slurp is just along for the ride. Makes no difference to searches, since they don't do their own crawling.


 12:59 pm on Nov 26, 2012 (gmt 0)

I can't be bothered to keep track, so I just block 'em

I stopped bothering about bots long time ago too.

There's just so much humanly possible to keep a track of in this increasingly 'social' web climate, especially with huge traffic turnover sites.

And Slurp is one of the well-behaved ones. There are JS-executing bad bots who don't actually announce themselves openly.

Not to mention armies of human scrapers employed specifically to spam by... ahem... "marketing" outfits. Try beating them.

The more you investigate, the deeper the rabbit hole goes and you end up doing 'just' that and not web development.

So in the end, I have cautious confidence in AdSense to be better equipped to filter them out, or else it's not worth the hype it deserves.


 9:04 pm on Nov 28, 2012 (gmt 0)

Yes, AdSense impressions by direct visits as Yahoo Slurp from Bethesda, Maryland with a 100% bounce rate - it's ridiculous and getting old...

Resolution: 1024x768
Operating System: (Not Set)
Screen Colors: 16 bit
Bounce Rate: 100%
Location: Bethesda, Maryland (according GA)
Location: California (according to IP address designated as Yahoo)


 12:26 am on Nov 30, 2012 (gmt 0)

Exactly is is Bethesda, Maryland.
I had to: SetEnvIf User-Agent Slurp get_out

Even the Slurp help page which you see in your logs is not available any more. It is just completely pointless.

When I first checked adsense analytics based on screen size I thought, what's wrong with 1024x768, they almost never click, they always bounce, it was a mystery, now I know it's Yahoo.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google AdSense
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved