homepage Welcome to WebmasterWorld Guest from 54.205.59.78
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Yahoo Mobile access with Safari
Every sign of being a bot not a proxy
dstiles




msg:4464546
 7:22 pm on Jun 12, 2012 (gmt 0)

I've been seeing a LOT of accesses from a yahoo IP in the range 98.139.241.nnn (resolves to ycar2.mobile.bf1.yahoo.com).

I recently spent some time removing this range from an auto-ban based on its header, working on the idea that it was a proxy for mobile devices and as such I was killing customers.

I am now seeing this IP range access sites that range from often-visited to almost never. Action is always the same: load default home page, load favicon (the real one, held in a subdirectory and only known via a meta tag in page headers).

Frequency has now built up to over 1000 hits per day across a dozen or so domains (one server, several IPs).

In each case the access is through a double proxy that resolves to (typically) inktomi and a broadband IP from one of the UK ISPs. In a few cases the combination is Yahoo SG and an SG broadband IP. There are probably other combinations but too many to analyze in a short time. There are no other hits from either the broadband proxy or the yahoo IPs (at least, not on the seldom-accessed sites).

The UA is always (at least, this month):
Mozilla/5.0 (iPhone; CPU iPhone OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3

I am looking at this now as some kind of cache rather than a real person. Something along the lines of G's web preview. The hits are far too frequent on one of our seldom-visited sites - almost the whole log for such a site today consisted of these hits.

I am thinking of pushing the IP range back into the "bad bots" section with a 403 or similar. Any comments on this ploy? Is anyone seeing real traffic on the back of such yahoo hits?

 

santapaws




msg:4469962
 7:53 am on Jun 27, 2012 (gmt 0)

just wondering if this ended up in your ban list or not? I am seeing the same pattern.

dstiles




msg:4470245
 8:54 pm on Jun 27, 2012 (gmt 0)

Yes, I blocked the range a few hours later, there being no report in here to counter my findings. Still getting loads of hits through the proxied range. There still seems no logical reason for it.

wilderness




msg:4470325
 3:04 am on Jun 28, 2012 (gmt 0)

There's another thread or two on "YahooCacheSystem"

I saved this the other day because the BlackBerry requests were unique to me.

98.139.241.245 - - [24/Jun/2012:17:01:51 +0100] "GET / HTTP/1.1" 403 559 "-" "YahooCacheSystem"
98.139.241.241 - - [24/Jun/2012:22:57:22 +0100] "GET / HTTP/1.1" 403 559 "-" "YahooCacheSystem"
98.139.241.245 - - [25/Jun/2012:02:49:01 +0100] "GET / HTTP/1.1" 200 6010 "-" "Mozilla/5.0 (BlackBerry; U; BlackBerry 9650; en-US) AppleWebKit/534.8+ (KHTML, like Gecko) Version/6.0.0.524 Mobile Safari/534.8+"
98.139.241.245 - - [25/Jun/2012:02:49:02 +0100] "GET /favicon.ico HTTP/1.1" 200 318 "-" "Mozilla/5.0 (BlackBerry; U; BlackBerry 9650; en-US) AppleWebKit/534.8+ (KHTML, like Gecko) Version/6.0.0.524 Mobile Safari/534.8+"
98.139.241.243 - - [25/Jun/2012:04:22:21 +0100] "GET / HTTP/1.1" 200 6010 "-" "Mozilla/5.0 (BlackBerry; U; BlackBerry 9670; en-US) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.407 Mobile Safari/534.1+"
98.139.241.243 - - [25/Jun/2012:04:22:22 +0100] "GET /favicon.ico HTTP/1.1" 200 318 "-" "Mozilla/5.0 (BlackBerry; U; BlackBerry 9670; en-US) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.407 Mobile Safari/534.1+"

dstiles




msg:4470655
 8:42 pm on Jun 28, 2012 (gmt 0)

Did it make sense to you at the time?

Had you blocked 98.139.241/24 would you have seen the 403 or would your BB still have loaded the proper requested page? And if the latter, from a previous cache or from your site?

wilderness




msg:4470685
 9:23 pm on Jun 28, 2012 (gmt 0)

Did it make sense to you at the time?


No it did not.

Don't believe I've the Ip range denied, rather the UA YahooCacheSystem

or perhaps even portions of same UA.

dstiles




msg:4471075
 8:50 pm on Jun 29, 2012 (gmt 0)

Thanks. I was wondering what effect my blocking the IP range had on what people saw on mobile devices. Not being able to check that kind of thing is about the only drawback of not having a mobile device. :)

wilderness




msg:4506698
 1:14 am on Oct 11, 2012 (gmt 0)

FWIW

98.139.241.241 - - [11/Oct/2012:00:23:03 +0100] "GET / HTTP/1.1" 403 559 "-" "YahooCacheSystem"
98.139.241.248 - - [11/Oct/2012:01:25:05 +0100] "GET / HTTP/1.1" 403 559 "-" "Mozilla/5.0 (Linux; U; Android 4.0.4; en-ca; SGH-I747M Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30"

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved