Forum Moderators: phranque
These addresses are part of Innove*t Sy*tems or SavvisNet or Servers of same. Some of these probably don't need to be blocked and may be redundant. Here are steps I've taken:
Deny From 165.193.0.0/16
#The following is the IP returned from Innovestsy*tems
Deny From 165.193.43.64
#Continuing
Deny From 204.70.0.0/15
Deny From 204.194.8.0/21
Deny From 208.65.144.0/21
Deny From 209.16.192.0/19
Deny From 209.83.128.0/24
Deny From 216.88.0.0/20
and
RewriteCond %{HTTP_REFERER} host177\.innovestsy*tems\.com [NC,OR]
RewriteCond %{HTTP_REFERER} host178\.innovestsy*tems\.com [NC]
(the usual Rewrite string here)
The latest log string entry:
host178.innovestsy*tems.com--[23/Dec/2009:02:02:05-0500]GET /domi/domi.htm HTTP/1.1 -200- 1828 - Mozilla/4.0 (compatible;)
Response Code: 200
Got Hints?
[edited by: jdMorgan at 6:11 pm (utc) on Dec. 24, 2009]
[edit reason] Disabled smilies and obscured domain [/edit]
Mozilla/4.0 (compatible)
If the UA is exactly as you've provided?
It's simple
# begins and ends with
SetEnvIf User-Agent ^Mozilla/4\.0$ keep_out
or
SetEnvIf User-Agent ^Mozilla/4\.0\ \(compatible\)$ keep_out
(please note; in lieu of "keep_out" (term of choice) many use other terms in their file)
I'm afraid that won't work as there are many requests using the same UA that should be allowed.
than, there are simply being used to cache images or pages and "generally speaking" should NOT be allowed.
You may add exceptions to specific IP ranges, by utilizing multiple conditions on a RewriteCond (rather than using the mod_SetEnvIf).
On the other hand, if the UA that you provided is just the beginning of the actual UA, than there is more terminology to focus upon
[edited by: jdMorgan at 6:12 pm (utc) on Dec. 24, 2009]
[edit reason] Disabled smilies. [/edit]
Deny from 165.193.43.64[b]/26[/b]
...
RewriteCond %{[b]REMOTE_HOST[/b]} [b]in[/b]novestsy.tems\.com [NC]
I will change the RewriteCond from RewriteCond %{HTTP_REFERER} to RewriteCond %{REMOTE_HOST} and give that a try. I seem to have a problem figuring out which to use; Referrer or Host when this subject arises.
You posted both "Deny From" and "RewriteCond" code. Does one need both of them? Currently I have 23 CDIRs that correspond to this domain we are discussing. Do I need to take them out of my .htaccess file?
Also, what is the purpose of editing the domain I posted? Is there a legal constraint or is it just good netiquette? Or Written/Unwritten rules of the house? It seems like anyone with the slightest bit of knowledge, anyone using this site of course, could find out what the domain is by the URL(s) posted.
As always, I appreciate the advice you share with this group. You make a great instructor. Merry Christmas.
Jim
I see by your edit remarks that you removed "smilies", etc. I don't know where the "smilies" came from as I did not put them in my original or subsequent postings.
When you use a standard reply (rather than a Quick Reply) there is an option below the reply box to disable smiles/graphics.
When not disable the forum software sometimes groups of characters into graphics.
I will change the RewriteCond from RewriteCond %{HTTP_REFERER} to RewriteCond %{REMOTE_HOST} and give that a try.
You should be aware that if you have a significant volume of HOST request that they could become server intensive.
Also, what is the purpose of editing the domain I posted?
Jim did not edit the domain name, rather he eliminated the leading server numbers (not neccessary) and changed a * (asterik) leading the sub-folder to a period.
Don
My name is Ray Umerley and I'm the Chief Security Officer for Innovest Systems, LLC. innovestsy*tems.com is our domain name and the IP range specified above is allocated to us by Savvis Communications.
According to your logs above the source of the scrapes is coming from host177.innovestsy*tems.com and host178.innovestsy*tems.com, neither of which are valid hosts on our network. If you search for the reverse DNS of those hostnames, rather than the root DNS of innovestsy*tems.com, you'll find they resolve to IP addresses outside of our network:
http://whois.xyz/65.200.157.177
http://whois.xyz/65.200.157.178
So the IPs you should be blocking are the above.
I will reiterate that we have no ownership of the 65.200.157.* network, and neither of the hosts mentioned above are valid hosts on our network. If I can be of any further assistance clarifying this matter, you may contact me.
Regards,
Raymond Umerley
Chief Security Officer
Innovest Systems, LLC
rumerley@<companyname>.com
p.s. I appreciate the administrators of this site obscuring the URL with the *s, unfortunately it was after Google had crawled the site and this thread now shows up on GoogleAlerts for innovestsy*tems.com (which brought it to my attention).
[edited by: jdMorgan at 5:32 pm (utc) on Dec. 25, 2009]
[edit reason] No domains or e-mails, please. [/edit]
According to your logs above the source of the scrapes is coming from host177.innovestsy*tems.com and host178.innovestsy*tems.com, neither of which are valid hosts on our network. If you search for the reverse DNS of those hostnames, rather than the root DNS of innovestsy*tems.com, you'll find they resolve to IP addresses outside of our network:
Except for the "*tems", the remainder resolves to Innovest Sy*tems on a reverse DNS.
It should be noted that checking DNS or IP ranges, or even reviewing logs does not require the expertise of "splitting the atom".
Nor does creating an htaccess which works effectively to prevent access to those a webmaster may not desire, rather people appear at Webmaster World because they are unaware of what terminology to use in their searches and how to locate relatively simple procedures (across the entire WWW) that have been used for a very long while.
htaccess [google.com]
[edited by: jdMorgan at 3:56 am (utc) on Dec. 27, 2009]
[edit reason] Obscured company name [/edit]
All evidence here points to some kind of DNS poisoning, as the rDNS lookups are apparently returning different results for different posters. If the rDNS is inconsistent, then rDNS lookups won't be useful for access control.
Thanks,
Jim
