homepage Welcome to WebmasterWorld Guest from 54.227.171.163
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Blocked IP Appends Data on URL
Angonasec

10+ Year Member



 
Msg#: 4690994 posted 6:17 am on Jul 27, 2014 (gmt 0)

I now block Exxon 158.25.0.0/16

Because a blocked Polish IP appends junk to file names, and when it receives a 403, immediately (ie. the next second) uses an RU IP to do the same.

188.116.4.204 - - [26/Jul/2014] "GET /example.htm/RK=0/RS=oMTszqk.FkrwF.7aIXBrmeN8RXk- HTTP/1.1" 403 294 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36"
158.255.0.157 - - [26/Jul/2014] "GET /example.htm/RK=0/RS=oMTszqk.FkrwF.7aIXBrmeN8RXk- HTTP/1.1" 404 959 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0"

PS. I'm curious if anybody is able to interpret the append? RK=0/RS=o

 

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4690994 posted 1:29 pm on Jul 27, 2014 (gmt 0)

PS. I'm curious if anybody is able to interpret the append? RK=0/RS=o
Apparently this is a problem caused by a (lazy? not very bright?) scraper who used Yahoo search results to scrape and just grabbed the URLs for their "Directories" from the results, as is without editing out Yahoo's internal link structure. There was some discussion about that around here. If you were to search here for "RK=0" you can find the entire answer.
Angonasec

10+ Year Member



 
Msg#: 4690994 posted 11:09 pm on Jul 27, 2014 (gmt 0)

Thank you iBill for fixing this thread (I mistakenly lumped the innocent Exxon IP cidr 158.25.0.0/16 with the guilty 158.255.0.0/24 nasties. I no longer block Exxon, of course. My apologies.)

Thank you not2easy, yes, well spotted. I do see that same RK=0/RS=o code in my logs used by legitimate Yahoo! search requests.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved