homepage Welcome to WebmasterWorld Guest from 54.211.95.201
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
TweetedTimes Now on Yandex?
incrediBILL




msg:4475660
 5:33 pm on Jul 14, 2012 (gmt 0)

I noticed this before but was too lazy to post it:

100.43.81.9
Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)

IP Range: 100.43.64.0 - 100.43.95.255
ORG: Yandex Inc

They used to crawl from comcastbusiness not too long ago per Pfui:
[webmasterworld.com...]

 

GaryK




msg:4475668
 6:14 pm on Jul 14, 2012 (gmt 0)

Weren't they bought by Yandex about a year ago?

dstiles




msg:4475700
 10:17 pm on Jul 14, 2012 (gmt 0)

Thanks for the heads-up, Bill.

I had a few IPs in the range 7 - 11 hit by something in June - only minor hits but your posting brought them to my notice. I've now blocked the range 100.43.81/24 - can't block the whole IP range because there are real (US) yandex bots on 100.43.83.129 - 100.43.83.161.

incrediBILL




msg:4475733
 1:11 am on Jul 15, 2012 (gmt 0)

can't block the whole IP range because there are real (US) yandex bots on 100.43.83.129 - 100.43.83.161.


Sure you can.

You block the whole range then punch holes in the firewall only if it's a Yandex bot.

I do the same thing with Google, I block the entire Google range and open holes in the firewall just for the stuff I want so a ton of stuff from their ranges goes bouncing off my servers on a regular basis.

I wouldn't have to do that if the crawlers wouldn't mix the usage of their IPs and/or properly label the user agents for other IP activities or set the rDNS, but they don't so BLOCK.

keyplyr




msg:4475741
 1:26 am on Jul 15, 2012 (gmt 0)


Unlike Google, Yandex leases subnets to business who may then release some for private use.

incrediBILL




msg:4475748
 1:56 am on Jul 15, 2012 (gmt 0)

Yandex leases subnets to business who may then release some for private use.


How sad for them :)

dstiles




msg:4475896
 8:01 pm on Jul 15, 2012 (gmt 0)

Bill, yes, I know, that's what I do (although I do not currently 403 the parent range; I have a special by-pass within it for bots). I was being literately lazy. Sorry. :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved