Forum Moderators: open
It didn't read robots.txt. It crawled one of my sites at a very leisurely pace over three days so my abusive spider trap didn't catch it. On the fourth day it fell into a different type of spider trap which put a stop to it.
The IP Address does not seem to belong to Wise but since most of the WHOIS page was question marks it was hard to tell for sure.
I found it somewhat ironic that while most of the content at the URL in the user agent was unreadable there was an English link to the robotstxt.org website.
Hanaro Telecom (HANANET)
211.108.62.0 - 211.108.62.255
211.172.0.0 - 211.199.255.255
211.200.176.0 - 211.200.179.255
211.211.134.0 - 211.211.134.255
Korea Network Information Center (KRNIC)
211.32.0.0 to 211.63.255.255
211.46.0.0 - 211.49.255.255
211.54.0.0 - 211.59.255.255
211.104.0.0 - 211.119.255.255
211.232.0.0 - 211.255.255.255
KOREA TELECOM (KORNET)
211.105.188.192 - 211.105.188.255
And those don't include the .kr troublemakers. Sheesh, I wish Korean telcos would clean up their acts!
Asking rev1.kornet.net. for 188.140.39.211.in-addr.arpa PTR record: Reports that no PTR records exist [from 211.216.50.170].
Note that all Internet accessible hosts are expected to have a reverse DNS entry (per RFC1912 [faqs.org] 2.1),
and many mailservers (such as AOL) will likely block E-mail from mailservers with no reverse DNS entry.
Jim
It likley would't hurt to deny the entire class (I did for a long time).
Banning an entire A class seems extreme to me. What made you try it and what made you change your mind? I'm not questioning your judgment, just trying to decide if I should try the same thing.
Gary,
When I first started with htaccess, it was prior to my coming to this fourm, rather with the tools and examples that I could find by searching google.
Those examples offered were very limited.
It was my desire to allow access for the Oceanic countries, however I was not aware of a method at that time. Nor was I even aware of Rewrite.
Instead I used only "deny from" and "SetEnvIf".
At one time I had 202, 203, 209, 210 and 211 denied.
I still have other Class A's denied today and I'm not alone.
After a time I had request to find a solution for access for the Oceanic countries while still denying the remainder of the APNIC ranges.
Jim was kind enough to provide me with one.
BTW, the first country that I desired to deny access to was Korea. At the time the spam and spidering was rampant. I referring to either late 1999 or 2000.
Later I added the UK.
Today I have most all non-North American ranges.
Occassionally somebody slips though.
I will and DO make adjustments when I'm contacted for an access solution. Fortuantely these contacts do not come through the website, rather my participation in the "widget field."
Nothing however, upsets me worse than to "make" the time to allow access and then have some phony reply how marvelous and extensive my websites are with content after I have seen in my visitor logs that the person only viewed a
page or two ;)
These choices that are make are assuredly NOT for everbody.
My content is quite specific and caters to a very small and specialized group of a specific horse breed.
Course that opens the doors for the plagues of gamblers that fail to comprehend that the gambling is not the beginning and end of interest. Most of these sites and even referrals from many gambling sites are denied also (the criteria being if they uses off-shore wagering sponsored links which are detrimental to the livlihood of North American racing.)
Don
edited by wilderness.
I should add that until recently I was not even aware of the possibility of using those sub-net limited ranges outside a class range in deny from.
Ex: 209.21. /15 of something similar. I still don't understand the conversion of the numbers today. Nor will I have time to search before this edit option expires ;)
edited again.
CIDR ranges.
Just a random number for example:
209.21.55.0/25
My servers run IIS so I don't have htaccess available to me. I do use ISAPI_Rewrite which is kind of similar, but it's not very good for this sort of thing. So mainly I use my firewall to ban by IP Address. It's cumbersome because I have to enter everything manually. I wish the firewall had a XML or other type of human-readable data file that I could use for adding IP ranges via scripting. A new vendor just bought the firewall I use (KSF) and that's one of the feature requests I've made of them.
Is it possible your website content is so good that one only needs to view a few pages to see how great it is?
The problem with most firewalls is they maintain a flat file linear list of IP addresses and traverse the entire list each time looking for IPs to be banned so the bigger the list, the slower your server gets.
It's easy to write a simple script that runs before any pages are shown, well at least it is in Linux, and look up the IPs quickly from a database instead of overloading the firewall. Maybe you can do this with ASP or PHP on your box, could be a quick day project. I'll bet you can find some code already on the net that even does this for IIS.
Bill, in ASP there's an event that fires at the start of a session. I use that event to automatically login my members if they have the properly encrypted login cookies available.
I'm sure I can write something to check for IP Addresses. The thing I like about the firewall option is these naughty user agents can't even see my sites. But you're right about it slowing down the server. However even if I use the event I mentioned I'd still have to either scan a list of addresses or do a db query.
I'll see what I can come up with. Right now I'm struggling to finish a project that is due by the end of June. As usual it all works fine until I try to use the DOM which means I have to write separate code for IE and in some cases I can't make IE do things that are easily doable in Fx, Opera and the other more standards compliant browsers.
Gary,
There never has been much support for IIS in Forum 11 (SESI). It has been primarily an Apache htaccess forum.
Course Jim took a lot of folks with him over to fourm92 (Apache Web server). Perhaps these folks were never interested in creepy crawlers anyway. ;)
The only saved link I have which provides any hint is:
[webmasterworld.com...]
korkus2000 was a regular participant in SESI before the forum went down (Nov 2003 thru June 2004).
I'm not sure if he's still around Webmaster World or not.
He was most knowlegable in IIS issues.
Don
