Forum Moderators: phranque

Message Too Old, No Replies

htaccess question

         

rsmarsha

9:17 am on Mar 6, 2008 (gmt 0)

10+ Year Member



I a crawler visting with the following as it's user agent.

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; iOpus-Web-Automation; .NET CLR 2.0.50727

How do i block the anyone with iOpus-Web-Automation in their user agent string?

My current htaccess is below (with some url related rules removed:


<Files ~ "^.*$">
order allow,deny
allow from all
deny from env=ban
## Crawlers/IP bans
#
#Crawlers By Country
#china
deny from 202.115.128.0/20
#korea
deny from 211.232.0.0/15
#dom republic
#romania
deny from 89.120.0.0/14
#brazil
deny from 200.163.0.0/16
#nigeria
deny from 41.204.224.0/19

#Others
#twiceler, performance systems international
deny from 38.0.0.0/8
#BT crawler
deny from 86.138.53.15
#dynamic
deny from 81.6.218.37
#opal telecom
deny from 89.242.88.194
#bethere
deny from 87.194.67.30
#eclerx
deny from 62.197.4.0/24
deny from 41.211.0.0/19
deny from 63.238.0.0/16
#wolfenet
deny from 207.178.0.0/18
#UAE Crawler
deny from 195.229.242.154

#USA Crawler
deny from 72.232.81.78
#Swiss Crawler
deny from 80.254.74.3
#Some Argie crawler at 5am
deny from 200.51.41.0/24

#bt crawler
deny from 81.148.154.110

#Loads of single page viewing search
deny from 205.212.0.0/16
deny from 128.241.0.0/16
deny from 209.59.0.0/16
deny from 168.143.0.0/16
deny from 207.195.0.0/16
deny from 207.67.128.0/17
deny from 130.94.0.0/16
deny from 198.170.0.0/15
deny from 198.172.0.0/15
deny from 147.203.0.0/16

</Files>

<Files ~ "^robots\.txt$¦^favicon\.ico$">
allow from all
</Files>

RewriteEngine on
#current rules removed from post

<Files ~ "^\.ht">
Order allow,deny
Deny from all
</Files>

Thanks

wilderness

10:03 am on Mar 6, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Insert the line I've provided between the two lines from your example.

Files ~ "^.*$">
SetEnvIf User-Agent "Opus-Web-Automation" keep_out
order allow,deny
allow from all

IN addition, and in a line prior to youur closing </Files>

insert:

deny from env=keep_out

rsmarsha

11:26 am on Mar 6, 2008 (gmt 0)

10+ Year Member



Thanks. :)

I've added that as you suggested, hopefully that will stop them.

I had some user agent blocks before but I think i had the lines in the wrong order or something as it was causing problems with some IE users.