homepage Welcome to WebmasterWorld Guest from 54.167.174.90
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
New Yahoo China bot range
dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4237913 posted 4:16 pm on Dec 2, 2010 (gmt 0)

Not seen anything serious from this IP range before but today there was a flurry of yahoo slurp china from it.

UA is: Yahoo! Slurp China

Which is not the usual Yahoo bot so it may be faked, although if true why so many IPs? Haven't check if it's true bot behaviour, robots.txt etc.

There may be more IPs within the Alisoft range running this bot but this is all I've seen so far. No Yahoo rDNS: all I've checked have been UNKNOWN-(ip)-aliyun.com. There may also be non-bot IPs within the given range - I haven't made substantial checks.

"Yahoo slurp" range (approx): 110.75.171.0 - 110.75.176.255

Host rangeAlisoft Singapore: 110.75.160.0 - 110.75.191.255

Part of China CNNIC range.

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4237913 posted 5:56 pm on Dec 2, 2010 (gmt 0)

In the past there has not been any penalty by the major SE's if your website was North American-based and access was denied to same SE's from non-NA IP's.

I'm curious to know if the reverse is true?

If your website is either RIPE or APNIC based and you denied access to North American-based-major-SE's while allowing their bots to spider from their perspective RIPE or APNIC ranges?
Anybody have any insight on this?

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4237913 posted 10:26 pm on Dec 2, 2010 (gmt 0)

Our server is UK based with mostly either UK or COM domains.

I block Chinese bots such as baidu and yahoo whilst letting in their Japanese versions because one of my clients has Japanese customers and one of our own sites features well in Japan.

Can't read the Japanese baidu site (eg) but his site comes up in it, although ours seems not to. Both come up in Japanese yahoo. The sites do not feature in the Chinese baidu or yahoo SEs.

I think (although not proven) that blocking (eg) bing or google here would be a big mistake. :)

Mokita

5+ Year Member



 
Msg#: 4237913 posted 1:35 am on Dec 3, 2010 (gmt 0)

wilderness wrote:
If your website is either RIPE or APNIC based and you denied access to North American-based-major-SE's while allowing their bots to spider from their perspective RIPE or APNIC ranges?


Most of our sites are APNIC based. I don't think I have ever seen a verifiable, googlebot or bingbot/msnbot crawling from anything other than ARIN based IPs. Blocking those would be incredibly self-defeating (unless you don't want any traffic).

Yahoo only crawls from ARIN based IPs or Chinese - and I have been blocking Slurp China (APNIC based) for a very long time, with no apparent effect on our listings in Yahoo.

HTH.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4237913 posted 9:55 pm on Dec 3, 2010 (gmt 0)

Looking longer at the IP ranges and at the UA, I'm now inclined to think this may have been a bad bot, not really Yahoo after all. But that's only my opinion. If it was I think it was probably from a compromised server.

Alisoft is a subsidiary of Alibaba and is apparently involved in a Chinese character input software project. but there seems to be several other sites in the group - can't say more 'cause I never learnt Chinese. :(

MxAngel



 
Msg#: 4237913 posted 1:53 pm on Dec 14, 2010 (gmt 0)

I had them for the first time in my logs today.

Host: 110.75.172.109
/
Http Code: 200 Date: Dec 14 05:41:44 Http Version: HTTP/1.1 Size in Bytes: 56260
Referer: -
Agent: Yahoo! Slurp China

I block all Chinese spiders so my intention was to apply it to this one too.

When looking up the IP I noticed that they are located in a Chinese Cloud. I already block any bot from the Amazon Cloud so I wont certainly allow a Chinese Cloud access to my website.

[linkedin.com...]

From: [en.wikipedia.org...]

China Yahoo
In October 2005, Alibaba Group formed a strategic partnership with Yahoo! Inc and acquired China Yahoo! (www.yahoo.com.cn), which is a Chinese portal offering search, email, and an enhanced focus on entertainment content.

# Alibaba Cloud Computing
deny from 110.75.160.0/19

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved