homepage Welcome to WebmasterWorld Guest from 23.20.63.27
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Asia and Pacific Region
Forum Library, Charter, Moderators: bill

Asia and Pacific Region Forum

    
Shanghainet / http://www.putclub.net / Referer Spam
Some Criterias to block idiots/spiders
thermoman




msg:798295
 4:37 pm on Jan 21, 2004 (gmt 0)

Hi and hello,

today i discovered a spider run over my websites from a dialup account in SHANGHAINET (211.99.210.0 - 211.99.217.255).

The first hits looked like this:

211.99.213.19 - - [21/Jan/2004:16:23:27 +0100] "GET / HTTP/1.1" 200 7813 "http://www.example.com" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98"
211.99.213.19 - - [21/Jan/2004:16:23:42 +0100] "GET /cgi-bin/stats HTTP/1.1" 404 292 "http://www.example.com" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98"
211.99.213.19 - - [21/Jan/2004:16:23:43 +0100] "GET /start HTTP/1.1" 200 7828 "http://www.example.com" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98"

As i see this it was clear to me that this was going to be a referer spam spider run (to promote [example.com...] . So I decided first to block all requests with this string in referer.

Some minutes later i realized hits from IP 211.99.213.20 - this time with realistic looking referer but still the same broken user-agent string (closing bracket is left). So i decided to block this too to get rid of these idiots:

snipped from .htaccess:

SetEnvIf Remote_Addr ^211\.99\.21[0-7]\. client_is_bad
SetEnvIf User-Agent "^Mozilla/4\.0\ \(compatible;\ MSIE\ 5\.00;\ Windows\ 98$" client_is_bad

Order Allow,Deny
Deny from env=client_is_bad
Allow from all

Since my websites doesn't address to asian people i simply block the whole SHANGHAINET.

Perhaps it's useful to you too.

Greetings,
thermoman

[edited by: Woz at 11:43 pm (utc) on Jan. 21, 2004]
[edit reason] No URLs please [/edit]

 

Romeo




msg:798296
 4:33 pm on Jan 23, 2004 (gmt 0)

Hi Thermoman,

yes, I have also seen this one, on 2003-12-01 coming from the same SHANGHAI address 211.99.213.19, using fake referers and the same bogus UA string you observed with the unmatched right bracket, which makes it easy to identify though. The bot's purpose is unknown to me, and it got my attention by running straight into a bot trap for ignoring the /robots.txt ...

Regards,
R.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Asia and Pacific Region
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved