Forum Moderators: phranque

Message Too Old, No Replies

.htaccess to ban spider

how to ban a whole range of IP's

         

jim_knopf

7:42 pm on Jun 19, 2008 (gmt 0)

10+ Year Member



Hi all,

to not repeat if you go to [webmasterworld.com...] to see my initial problem.
In general I want to ban with htaccess a whole IP range that people work with. I try robots.txt but that "Charlotte" spider not obey that.

Like others I observerd the following IP's

208.111.154.16
208.111.154.189
208.111.154.15
208.111.154.67
208.111.154.193
208.111.154.66
208.111.154.182
208.111.154.21
208.111.154.183
208.111.154.184
208.111.154.65
208.111.154.68
208.111.154.199
208.111.154.188
208.111.154.186
208.111.154.195
208.111.154.197
208.111.154.200
208.111.154.62
208.111.154.69
208.111.154.63
208.111.154.64
208.111.154.92

All related to limelight

According to wilderness who is the moderator over there I thingk I should "this line denied access foe limelight users in the following ranges:
208.111.128.0 - 208.111.191.255"

Now my question is: how would I add that to my htaccess? My htaccess files are created by Wordpress when doing the permalink update the first time. I never change anything to it. And since I use the same configuration for any of the domains on server they are for all similar. So if I get some help I can going with all of them.

Here a sample of one:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !http://www.#*$!-#*$!#*$!.com
RewriteRule (wp-content/uploads.*\.(gif¦jpg¦jpeg¦png)$)¦(wp-content/gallery/la-#*$!-mix.*\.(gif¦jpg¦jpeg¦png)$)¦(wp-content/gallery/historical.*\.(gif¦jpg¦jpeg¦png)$)¦(wp-content/gallery/gloria-trevi.*\.(gif¦jpg¦jpeg¦png)$)¦(wp-content/gallery/carnival_2008.*\.(gif¦jpg¦jpeg¦png)$)¦(wp-content/gallery/carnival-ii.*\.(gif¦jpg¦jpeg¦png)$) [#*$!-#*$!.com...] [L]
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

Where and how to add that range deny?

I would also try serverwide/global but was not comfortable with on suggestion I got on the other board. I'm ok with the apache httpconfig file to edit or similar but going any deeper without assistance not.

Thanks for any advise.

Greetings from a very hot La Paz down the Baja

wilderness

8:47 pm on Jun 19, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



According to wilderness who is the moderator over there

Hardly.
Neither of the following are listed below my pseudo:
Moderator This Forum
Moderator

----------------

In your Feb 2008 Inquiry
(http://www.webmasterworld.com/search_engine_spiders/3567029.htm)

I provided:
search forum for "Limelight"

[google.com...]

This message of that thread provides a BAD example of implemetation (perahps just a typo?)
[webmasterworld.com...]

In this line:
deny from 208.111.154
the REQUIRED trailing period has been omitted.
Should read:
deny from 208.111.154.

To deny the entire 208.111.128.0 - 208.111.191.255?
Simply replace the above line in the example at
[webmasterworld.com...]
with this line:
deny from 208.111.128.0/18
Using the remainder of the example in the process.

I've never had any WP implemented, however my gut feeling tells me that these additional htaccess lines
should follow your:
</IfModule>

jim_knopf

9:29 pm on Jun 19, 2008 (gmt 0)

10+ Year Member



Ok Wilderness - sorry must had you mixed up in regards "moderator"

I understand the example you pointed me.

<Limit GET HEAD POST>
order allow,deny
##--> Bye bye Limelight Networks. You are not welcome here.
deny from 208.111.128.0 - 208.111.191.255
allow from all
</LIMIT>

My concern is: WHERE to add that at my existing .htaccess file.

After the last line before </IfModule>?

After RewriteEngine On at the beginning?

Complete out of that container after </ifModule>?

Thanks for any advise

wilderness

9:51 pm on Jun 19, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I understand the example you pointed me.

<Limit GET HEAD POST>
order allow,deny
##--> Bye bye Limelight Networks. You are not welcome here.
deny from 208.111.128.0 - 208.111.191.255
allow from all
</LIMIT>

The above is incorrect.
Should read:

<Limit GET HEAD POST>
order allow,deny
##--> Bye bye Limelight Networks. You are not welcome here.
deny from 208.111.128.0/18
allow from all
</LIMIT>

My concern is: WHERE to add that at my existing .htaccess file.

After the last line before </IfModule>?

After RewriteEngine On at the beginning?

Complete out of that container after </ifModule>?

Thanks for any advise

1)This is NOT part of your WordPress module
2) This is not a rewrite and as a result does NOT follow or require "Rewrite on".

Complete out of that container after </ifModule>?

Believe what I previously provided!

should follow your:
</IfModule>

Takes the additional lines "out of" your WordPress module.

As an aside;
Apache support for mod_setenvif [httpd.apache.org]

jdMorgan

10:24 pm on Jun 19, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Do take advantage of the capabilities of regular-expressions patterns, too: that code can be shortened/corrected to:

RewriteEngine on
#
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^http://www\.example\.com
RewriteRule wp-content/(uploads¦gallery/la-xyz-mix¦gallery/historical¦gallery/gloria-trevi¦gallery/carnival_2008¦gallery/carnival-ii).*\.(gif¦jpe?g¦png)$ http://www.example.com/wp-content/plugins/hotlink-protection/nosteal.jpg [R=302,L]
#
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
#
Order allow,deny
Allow from all
Deny from 208.111.128.0/18

Neither <IfModule> nor RewriteBase are needed -- They do nothing for you as used here. Using <Limit> is also counter-productive in that there's no reason to *allow* PUT, DELETE, and the other methods.

It actually does not matter whether the mod_access code precedes or follows the mod_rewrite code. The lines of code are processed by both modules in turn; Each module runs in the order determined by the server configuration, and each module processes only the directives it understands. So the code order makes no difference, since the LoadModule order is what will control order of execution.

Jim

[edited by: jdMorgan at 10:24 pm (utc) on June 19, 2008]

jim_knopf

10:31 pm on Jun 19, 2008 (gmt 0)

10+ Year Member



Ok, I did understand the "these additional htaccess lines should follow" not correct.

Apologize, english also not first lingo here... I'll see what happend after placing that. It was quiet bad the last few days with that spider...

Thanks for the help

jim_knopf

10:32 pm on Jun 19, 2008 (gmt 0)

10+ Year Member



Thanks to you also JD

jim_knopf

5:19 am on Jun 23, 2008 (gmt 0)

10+ Year Member



Ok, I'm back, trying the code as JD wrote did give me a 500 error (yes - I made sure no broken pipes)

Doing as wilderness suggested did work in sense of no error.
I did it this way:

<IfModule mod_rewrite.c>
Options +FollowSymlinks
RewriteEngine On
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !http://www.#*$!x-#*$!xx.com
RewriteRule (wp-content/postpics.*\.(gif¦jpg¦jpeg¦png)$) [#*$!x-#*$!xx.com...] [L]
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

<Limit GET HEAD POST>
order allow,deny
##--> Bye bye Limelight Networks. You are not welcome here.
deny from 208.111.128.0/18
allow from all
</LIMIT>

Yet checking my stats today I see visitor v87.nat.svl.searchme.com (208.111.154.87)

Any suggestions please.....

Greetings

Key_Master

5:30 am on Jun 23, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



order allow,deny

The allow/deny directives aren't in the proper order. See:
[httpd.apache.org...]

Proper implementation:

<Limit GET HEAD POST>
order allow,deny
allow from all
##--> Bye bye Limelight Networks. You are not welcome here.
deny from 208.111.128.0/18
</Limit>

[edited by: Key_Master at 5:32 am (utc) on June 23, 2008]

wilderness

2:18 pm on Jun 23, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Yet checking my stats today I see visitor v87.nat.svl.searchme.com (208.111.154.87)

Have you confirmed that your seeing a "200" (actual visit), rather than a "403" (denied visit)?

Denial of access does not remove the request from you access logs.

The allow/deny directives aren't in the proper order. See:
[httpd.apache.org...]

Key_Master,
I've had those lines in place and functioning for nine years.
Although I'm aware that "Allow from all" is improperly used, it has not hindered functionality and thus I never felt any compelling need to make a change.

Don

jdMorgan

6:45 am on Jun 24, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The Order directive effectively declares a priority, and states whether Allows override Denys, or vice-versa.

If you use "Order allow,deny", then all Allows in the <container> (or the whole file if no container is used) are processed first, and then the Denys are processed, and can partially or fully override the Allows. So "Order" refers to the order in which the Allows and Denys are processed as groups, and not to any ordering requirement imposed upon the coder when writing the Allow and Deny lines.

Having declared an Order, you may write your Allows and Denys in any sequence.

Jim

jim_knopf

8:12 pm on Jul 9, 2008 (gmt 0)

10+ Year Member



Wilderness: in regards to your last reply - I just had no option to study logfiles. It shows "200" when I compare time/IP stats ( I use Statcounter) with the server log file.