.htaccess ban list

Forum Moderators: phranque

Message Too Old, No Replies

.htaccess ban list

Line from 'Close to perfect 3'

Angonasec

3:11 am on Dec 3, 2004 (gmt 0)

Enjoyed the Close to Perfect ban List 3.

I've just added many lines in my root .htaccess blocking access based on referer, and they are working well, serving 403s.

Except this one based on REQUEST_URI :

RewriteCond %{REQUEST_URI} ^/(MSOffice¦_vti) [NC,OR]

But this got through to produce a 404 in my access log, not a 403, how come?

123.456.78.9 - - [01/Dec/2004:03:39:13 -0500] "GET /_vti_inf.html HTTP/1.0" 404 32744 "-" "Mozilla/4.0 (compatible; MS FrontPage 6.0)"

I really want to block this stuff, how do I code to give it a 403 properly?

jdMorgan

3:17 am on Dec 3, 2004 (gmt 0)

Your code looks correct, assuming that the broken pipe "¦" character is replaced by a solid one in your actual file.

Let me ask a possibly-relevant question: Is your custom 404 page really 32 kB in size?

Also, what does the RewriteRule following all of your RewriteConds look like, and are you sure you don't have an [OR] on the last RewriteCond?

<added> Another thing that could 'break' this code is if the line just before this one is missing its [OR] </added>

Jim

Angonasec

9:49 am on Dec 3, 2004 (gmt 0)

Ta Jim!

Here's the end of my root .htaccess file:

# Start Cyveillance blocks
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.2(2[4-9]¦[3-4][0-9]¦5[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.118\.41\.(19[2-9]¦2[0-1][0-9]¦22[0-3])$ [OR]
# End Cyveillance blocks
RewriteCond %{HTTP_REFERER} q=guestbook [NC,OR]
RewriteCond %{HTTP_REFERER} iaea\.org [NC]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
RewriteRule .* - [F]

No 'broken pipes', and I use the host default 404. It's American so it could well be err... bloated.

I'll check.

Here's the code around the line in question:

RewriteCond %{REQUEST_URI} (.?mail.?form¦form¦(GM)?form.?.?mail¦.?mail)(2¦to)?\.?(asp¦cgi¦exe¦php¦pl¦pm)?$ [NC,OR]
# MSOffice
RewriteCond %{REQUEST_URI} ^/(MSOffice¦_vti) [NC,OR]
# Nimda
RewriteCond %{REQUEST_URI} /(admin¦cmd¦httpodbc¦nsiislog¦root¦shell)\.(dll¦exe) [NC,OR]

Ta!

[edited by: Angonasec at 10:00 am (utc) on Dec. 3, 2004]

Angonasec

9:55 am on Dec 3, 2004 (gmt 0)

Apologies to bloated Americans.

I changed my host's default 404. I do in fact have a 32kb 404, I simply pasted the code from my index page into the 404.html page.

Is this a no-no?

I did it because I renamed a lot of files, and wanted to catch any I'd forgotten to 301.

Ta!

jdMorgan

2:30 pm on Dec 3, 2004 (gmt 0)

On the ban list problem, the second-to-last RewriteCond is missing a required [OR], and this will break the code, since anything below that will be ANDed. This section of code will only block requests from user-agents named EmailCollector that also meet any one of the preceding ORed conditions.

I changed my host's default 404. I do in fact have a 32kb 404, I simply pasted the code from my index page into the 404.html page.

On the 404 page:
> Is this a no-no?

Yes, it's potentially a massive duplicate-content problem, and could also cost you dearly in bandwidth if a badly-designed spider gets into your site and starts requesting non-existent URLs.

I'd suggest you create a nice, compact, concise, custom 404 error page that says,

We're sorry, but the page you requested cannot be found.
Please visit our site map [webmasterworld.com] for help finding the page you are looking for. Or visit our home page [webmasterworld.com].
We apologize for the inconvenience.
<signature line>

That will keep your visitors happy, and avoid confusing robots into messing up your search listings, flagging you for a duplicate-content penalty, or consuming massive bandwidth trying to fetch non-existent files.

Jim

Angonasec

2:56 am on Dec 4, 2004 (gmt 0)

Blimey, did I need your advice!

You're terrific...

Fixed.

Angonasec

5:04 pm on Dec 13, 2004 (gmt 0)

Thanks to WebmasterWorld the folowing bot blocking code has made a big improvement in sorting legit from illegit visitors.
However, I'm still confused:

I get too many hits like this: ie. Serving 'a good guy' a 403, I think based on the MSIECrawler part.

123.456.78.90 - - [12/Dec/2004:13:13:22 -0500] "GET /robots.txt HTTP/1.1" 403 306 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSIECrawler)"

So I've gone through my root htaccess bot blocking code, removing a line at a time, trying to *allow MSIECrawler*, but still the following is somehow blocking MSIECrawler, or summat else in the referrer string above.

Blowed if I know why, anyone help? I want to allow MSIECrawler, at least for a while.

(No 'broken pipes' etc, and the code is working as it s'posed to blocking nimda, Frontpage, etc)...

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^WebStripper/ [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTMLParser [OR]
RewriteCond %{HTTP_USER_AGENT} ^ZipppBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Deepnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GovindaBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSFrontPage/ [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper/ [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport*28 [OR]
RewriteCond %{HTTP_REFERER} jerseyfox [OR]
RewriteCond %{HTTP_REFERER} xopy [OR]
RewriteCond %{HTTP_REFERER} adminshop [OR]
RewriteCond %{HTTP_REFERER} linkerdome [OR]
RewriteCond %{REQUEST_URI} (.?mail.?form¦form¦(GM)?form.?.?mail¦.?mail)(2¦to)?\.?(asp¦cgi¦exe¦php¦pl¦pm)?$ [NC,OR]
# MSOffice
RewriteCond %{REQUEST_URI} ^/(MSOffice¦_vti) [NC,OR]
# Nimda
RewriteCond %{REQUEST_URI} /(admin¦cmd¦httpodbc¦nsiislog¦root¦shell)\.(dll¦exe) [NC,OR]
# CodeRed
RewriteCond %{REQUEST_URI} ^/default\.(ida¦idq) [NC,OR]
RewriteCond %{REQUEST_URI} ^/.*\.printer$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(BlackWidowICrescent¦Disco.?¦ExtractorPro¦HTML.?Works¦Franklin.?Locator¦HLoader¦http.?generic¦Industry.?Program¦IUPUI.?Research.?Bot¦Mac.?Finder¦NetZIP¦NICErsPRO¦NPBot¦Planty¦Production.?Bot¦Program.?Shareware¦Teleport.?Pro¦TE¦VoidEYE¦WebBandit¦WebCopier¦WebZIP¦WinMHT¦WEP.?Search¦Zeus) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cherry.?picker¦e?mail.?(collector¦extractor¦magnet¦reaper¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Educate.?Search¦Full.?Web.?Bot¦Indy.?Library¦IUFW.?Web [NC,OR]
RewriteCond %{HTTP_USER_AGENT} httrack¦larbin¦NaverRobot¦Siphon¦SURF [NC,OR]
RewriteCond %{HTTP_USER_AGENT} efp@gmx\.net [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.?URL.?Control [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Miss.*g.*.?Locat.* [NC,OR]
RewriteCond %{HTTP_REFERER} q=guestbook [NC,OR]
RewriteCond %{HTTP_REFERER} iaea\.org [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
RewriteRule .* - [F]

Angonasec

6:39 pm on Dec 18, 2004 (gmt 0)

Any ideas on why MSIECrawler is being blocked?

Furthermore...

This .htaccess code also seems to be blocking UAs containing the phrase 'BorderManager', witness this visit, which switched UAs after getting 403's for the BorderManager phrase.

This, presumably legitimate, visitor was served the html, but images were blocked when it switching to BorderManager UA.

What is happening here?

NB. I've changed the IP, search terms, and my domain to comply with WebmasterWorld protocol:

123.123.12.3 - - [17/Dec/2004:14:16:19 -0500] "GET /t/1.css HTTP/1.0" 200 3217 "http://www.my-domain.com/my.page.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
123.123.12.3 - - [17/Dec/2004:14:16:19 -0500] "GET /t/sta.gif HTTP/1.0" 403 293 "-" "Mozilla/4.0 (compatible; BorderManager 3.0)"
123.123.12.3 - - [17/Dec/2004:14:16:19 -0500] "GET /t/stp.gif HTTP/1.0" 403 293 "-" "Mozilla/4.0 (compatible; BorderManager 3.0)"
123.123.12.3 - - [17/Dec/2004:14:16:19 -0500] "GET /my.page.htm HTTP/1.0" 200 45309 "http://www.google.com/search?hl=en&lr=&q=%22fish+chips%22+%2B+vinegar" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
123.123.12.3 - - [17/Dec/2004:14:16:19 -0500] "GET /t/staon.gif HTTP/1.0" 200 514 "http://www.my-domain.com/my.page.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
123.123.12.3 - - [17/Dec/2004:14:16:19 -0500] "GET /t/stpon.gif HTTP/1.0" 200 512 "http://www.my-domain.com/my.page.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
123.123.12.3 - - [17/Dec/2004:14:16:20 -0500] "GET /t/leftco.gif HTTP/1.0" 403 296 "-" "Mozilla/4.0 (compatible; BorderManager 3.0)"
123.123.12.3 - - [17/Dec/2004:14:16:20 -0500] "GET /t/bar.gif HTTP/1.0" 403 293 "-" "Mozilla/4.0 (compatible; BorderManager 3.0)"
123.123.12.3 - - [17/Dec/2004:14:16:20 -0500] "GET /t/rightco.gif HTTP/1.0" 403 297 "-" "Mozilla/4.0 (compatible; BorderManager 3.0)"

The bar.gif and rightco.gif images are simple page formatting, no javascript used.

How is this .htaccess code causing this? Could it be due to something other than my root .htaccess file? It only seems to occur with the BorderManager UA. But as noted in my previous post, MSIECrawler UAs are also being blocked for some uknown reason too.

Can you help me tidy this up please?

Ta!

wilderness

7:56 pm on Dec 18, 2004 (gmt 0)

RewriteCond %{HTTP_USER_AGENT} ^(BlackWidowICrescent¦Disco.?¦ExtractorPro¦HTML.?Works¦Franklin.?Locator¦HLoader¦http.?generic¦Industry.?Program¦IUPUI.?Research.?Bot¦Mac.?Finder¦NetZIP¦NICErsPRO¦NPBot¦Planty¦Production.?Bot¦Program.?Shareware¦Teleport.?Pro¦TE¦VoidEYE¦WebBandit¦WebCopier¦WebZIP¦WinMHT¦WEP.?Search¦Zeus) [NC,OR]

You have an error on this rewrite:

Between the words BlackWidowand Crescent
you have the charcter "I" rather than a pipe.

wilderness

8:03 pm on Dec 18, 2004 (gmt 0)

RewriteCond %{HTTP_USER_AGENT} cherry.?picker¦e?mail.?(collector¦extractor¦magnet¦reaper¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]

This line is in error also:

This portion

cherry.?picker¦e?mail.?

fails to include any parenthenses, however you begin the second line addition with a parenthenses.
NOTE you'll also need to include a pipe between the current first and second lines correcting.

[edit]
possible the way your line current reads? That anything beginning the any of the first three words in the line minus parenthenses and ending with any word in the second line contained parenthenses would be denied.
Many of these associations are highly unlikely today. At least related to Cherry.

Angonasec

9:50 pm on Dec 18, 2004 (gmt 0)

Thank you Wilderness, I've fixed the vertical pipe, and decided to remove this line altogether because I didn't understand what it should look like:

Removed this line:

RewriteCond %{HTTP_USER_AGENT} cherry.?picker¦e?mail.?(collector¦extractor¦magnet¦reaper¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]

So now I'll wait and see what happens.

Thank you so much for taking the time to check the code for me, I really appreciate it.

Ta!

wilderness

7:29 pm on Dec 19, 2004 (gmt 0)

Removed this line:
RewriteCond %{HTTP_USER_AGENT} cherry.?picker¦e?mail.?(collector¦extractor¦magnet¦reaper¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]

Personally, I don't used Rewrite for these types of things.

I believe in your instance, you might be better off leaving the line, however modifying it to read:

RewriteCond %{HTTP_USER_AGENT} (collector¦extractor¦magnet¦reaper¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]

This line catches an abundance of harvesters.

kamikaze Optimizer

8:18 am on Dec 21, 2004 (gmt 0)

I think I like all of your code better then mine.

I am real new to this and found this thread because I am in some real trouble at the moment.

Mine is for a phpbb forum, hence all the rewrite stuff. Any takers on helping me clean it up?

I have deleted out about 100 "deny from" IP's that have been hitting me hard.

Also, does anyone know how I can prevent being hit on graphics via Google cache? Thats where I get hit the hardest. 43,000 hits in one hour today.

Also, I not sure, but I think I might have made msnbot mad with this code, he has not been around lately, which might not be a bad thing :)

Me .htaccess:

#*************************************************************************/
#* Googlifier 2.x */
#* for PHP-Nuke 6.5 and phpBB 2.1 */
#* Based on files and ideas listed in readme1st.file */
#* Maintained by Imago */
#* WARNING: this is a developing code. Please, regularly update from */
#* [orientalgate.org...] */
#* Support and discussion */
#* [orientalgate.org...] */
#*************************************************************************/

DirectoryIndex index.php index.cgi index.pl index.shtml index.html

<Limit GET PUT POST>
order allow,deny
deny from 123.456.789

allow from all
</Limit>

# deny most common except .php
<FilesMatch "\.(inc¦tpl¦h¦ihtml¦sql¦ini¦conf¦class¦bin¦spd¦theme¦module¦exe)$">
deny from all
</FilesMatch>

#Disable .htaccess viewing from browser
<Files ~ "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</Files>

# change to yoursite.com
ErrorDocument 404 [123-xyz.com...]
ErrorDocument 404 [123-xyz.com...]

RewriteEngine On

# Block site rippers and bots

RewriteCond %{HTTP_USER_AGENT} ^asterias [OR]
RewriteCond %{HTTP_USER_AGENT} ^BackDoorBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Black.Hole [OR]
RewriteCond %{HTTP_USER_AGENT} ^BlowFish [OR]
RewriteCond %{HTTP_USER_AGENT} ^BotALot [OR]
RewriteCond %{HTTP_USER_AGENT} ^BuiltBotTough [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bullseye [OR]
RewriteCond %{HTTP_USER_AGENT} ^BunnySlippers [OR]
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Cegbfeieh [OR]
RewriteCond %{HTTP_USER_AGENT} ^CheeseBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^CopyRightCheck [OR]
RewriteCond %{HTTP_USER_AGENT} ^cosmos [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Copier [OR]
RewriteCond %{HTTP_USER_AGENT} ^crawl [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DittoSpyder [OR]
RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Wonder [OR]
RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EroCrawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
RewriteCond %{HTTP_USER_AGENT} ^Foobot [OR]
RewriteCond %{HTTP_USER_AGENT} ^FrontPage [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetSmart [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^Harvest [OR]
RewriteCond %{HTTP_USER_AGENT} ^hloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^httplib [OR]
RewriteCond %{HTTP_USER_AGENT} ^humanlinks [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Indy\ Library [OR]
RewriteCond %{HTTP_USER_AGENT} ^InfoNaviRobot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetSeer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^Iria [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JennyBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
RewriteCond %{HTTP_USER_AGENT} ^Kenjin.Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Keyword.Density [OR]
RewriteCond %{HTTP_USER_AGENT} ^LexiBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkextractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkScan/8.1a.Unix [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^lwp-trivial [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
RewriteCond %{HTTP_USER_AGENT} ^likse [OR]
RewriteCond %{HTTP_USER_AGENT} ^Link [OR]
RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIIxpc [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister.PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^moget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/2 [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/3.Mozilla/2.01 [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*Indy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSFrontPage [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^ndy\ Library [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetMechanic [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^NPBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZip [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline.Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^Openfind [OR]
RewriteCond %{HTTP_USER_AGENT} ^ProPowerBot/2.14 [OR]
RewriteCond %{HTTP_USER_AGENT} ^ProWebWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ping [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^QueryN.Metasearch [OR]
RewriteCond %{HTTP_USER_AGENT} ^RepoMonkey [OR]
RewriteCond %{HTTP_USER_AGENT} ^RMA [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^robot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpankBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^spanner [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^suzuran [OR]
RewriteCond %{HTTP_USER_AGENT} ^Szukacz/1.4 [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Telesoft [OR]
RewriteCond %{HTTP_USER_AGENT} ^The.Intraformant [OR]
RewriteCond %{HTTP_USER_AGENT} ^TheNomad [OR]
RewriteCond %{HTTP_USER_AGENT} ^TightTwatBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Titan [OR]
RewriteCond %{HTTP_USER_AGENT} ^toCrawl/UrlDispatcher [OR]
RewriteCond %{HTTP_USER_AGENT} ^True_Robot [OR]
RewriteCond %{HTTP_USER_AGENT} ^turingos [OR]
RewriteCond %{HTTP_USER_AGENT} ^TurnitinBot/1.5 [OR]
RewriteCond %{HTTP_USER_AGENT} ^TurnitinBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^URLy.Warning [OR]
RewriteCond %{HTTP_USER_AGENT} ^VCI [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^WallPaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebBandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEnhancer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.Image.Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebmasterWorldForumBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website.Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^Webster.Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WISENutbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWW-Collector-E [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xenu's [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^ZyBorg
#RewriteRule ^.* - [F,L]
#RewriteRule ^.* /403.html [R,L]
RewriteRule ^.*$ [cannotlinktothisfile.com...] [L,R]

RewriteRule ^index.html index.php

RewriteRule ^gate.html modules.php

#Articles
RewriteRule ^article([1-9][0-9]*).* gate.html?name=News&file=article&sid=$1
RewriteRule ^article([1-9][0-9]*).html gate.html?name=News&file=article&sid=$1&mode=$2&order=$3&thold=$4
RewriteRule ^topic([1-9][0-9]*)-([0-9]*).* gate.html?name=News&new_topic=$1&pagenum=$2
RewriteRule ^topic([1-9][0-9]*).* gate.html?name=News&new_topic=$1
RewriteRule ^cat([1-9][0-9]*)-([0-9]*).* gate.html?name=News&file=categories&op=newindex&catid=$1&pagenum=$2
RewriteRule ^cat([1-9][0-9]*).* gate.html?name=News&file=categories&op=newindex&catid=$1
RewriteRule ^cat([1-9][0-9]*)-([0-9]*).* gate.html?name=News&file=article&op=newindex&catid=$1&pagenum=$2
RewriteRule ^cat([1-9][0-9]*).* gate.html?name=News&file=article&op=newindex&catid=$1
RewriteRule ^printout([1-9][0-9]*).* gate.html?name=News&file=print&sid=$1
RewriteRule ^newspage([1-9][0-9]*).* gate.html?name=News&pagenum=$1
RewriteRule ^comment([0-9]*)-([0-9]*)-([0-9]*)([a-zA-Z0-9]*)([0-9]*)([/:¦\s\-\'{}().&_0-9+=-]*).html gate.html?name=News&file=comments&op=showreply&tid=$1&sid=$2&pid=$3&mode=$4&order=$5&thold=$6
RewriteRule ^articles.* gate.html?name=News&file=index

#Content
RewriteRule ^contentid-([1-9][0-9]*).html gate.html?name=Content&pa=showpage&pid=$1
RewriteRule ^content-cat-([1-9][0-9]*).html gate.html?name=Content&pa=list_pages_categories&cid=$1

#Downloads
RewriteRule ^downloads.html gate.html?name=Downloads
RewriteRule ^ordercat([0-9]*)-([a-zA-Z_]*).html gate.html?name=Downloads&d_op=viewdownload&cid=$1&orderby=$2
RewriteRule ^topdowns.html gate.html?name=Downloads&d_op=TopRated
RewriteRule ^topdowns-(1¦5¦10¦25¦50)-(num¦percent).html topdowns.html&ratenum=$1&ratetype=$2
RewriteRule ^popdowns.html gate.html?name=Downloads&d_op=MostPopular
RewriteRule ^popdowns-(1¦5¦10¦25¦50)-(num¦percent).html popdowns.html&ratenum=$1&ratetype=$2
RewriteRule ^newdowns.html gate.html?name=Downloads&d_op=NewDownloads
RewriteRule ^newdownsw-([0-9]*).html gate.html?name=Downloads&d_op=NewDownloads&newdownloadshowdays=$1
RewriteRule ^newdownsd-([0-9]*).html gate.html?name=Downloads&d_op=NewDownloadsDate&selectdate=$1
RewriteRule ^downfile([0-9]*) gate.html?name=Downloads&d_op=getit&lid=$1
RewriteRule ^downcat([0-9]*)-([0-9]*)-([a-zA-Z0-9]*)-([0-9]*).html gate.html?name=Downloads&d_op=viewdownload&cid=$1&min=$2&orderby=$3&show=$4
RewriteRule ^downcat([0-9]*).html gate.html?name=Downloads&d_op=viewdownload&cid=$1

#Encyclopedia
RewriteRule ^dic([1-9][0-9]*)all([a-zA-Z0-9]*).* gate.html?name=Encyclopedia&op=terms&eid=$1&ltr=$2
RewriteRule ^term([1-9][0-9]*)-([1-9][0-9]*).* gate.html?name=Encyclopedia&op=content&tid=$1&page=$2
RewriteRule ^term([1-9][0-9]*).* gate.html?name=Encyclopedia&op=content&tid=$1
RewriteRule ^dic([1-9][0-9]*).* gate.html?name=Encyclopedia&op=list_content&eid=$1
RewriteRule ^encyclopedia.* gate.html?name=Encyclopedia&file=index

#FAQ
RewriteRule ^faq.html gate.html?name=FAQ&file=index
RewriteRule ^faq-([a-zA-Z0-9+]*)-([0-9]*)-([a-zA-Z0-9+]*) gate.html?name=FAQ&myfaq=$1&id_cat=$2&categories=$3

#Forums
RewriteRule ^forums.* gate.html?name=Forums&file=index
RewriteRule ^forums.* gate.html?name=Forums&file=index&mark=forums
RewriteRule ^fcat([0-9]*).* gate.html?name=Forums&file=index&c=$1
RewriteRule ^forum([0-9]*).* gate.html?name=Forums&file=viewforum&f=$1&mark=topic
RewriteRule ^viewforum([0-9]*)-([0-9]*)-([0-9]*).* gate.html?name=Forums&file=viewforum&f=$1&topicdays=$2&start=$3
RewriteRule ^forum([0-9]*).* gate.html?name=Forums&file=viewforum&f=$1
RewriteRule ^ptopic([0-9]*).* gate.html?name=Forums&file=viewtopic&t=$1&view=previous
RewriteRule ^ntopic([0-9]*).* gate.html?name=Forums&file=viewtopic&t=$1&view=next
RewriteRule ^watchtopic([0-9]*)-([0-9]*).* gate.html?name=Forums&file=viewtopic&t=$1&watch=topic&start=$2
RewriteRule ^unwatchtopic([0-9]*)-([0-9]*).* gate.html?name=Forums&file=viewtopic&t=$1&unwatch=topic&start=$2
RewriteRule ^ftopic([0-9]*)-([0-9]*)-([a-zA-Z]*)-([0-9]*).* gate.html?name=Forums&file=viewtopic&t=$1&postdays=$2&postorder=$3&start=$4
RewriteRule ^ftopic([0-9]*)-([0-9]*).* gate.html?name=Forums&file=viewtopic&t=$1&start=$2
RewriteRule ^ftopic([0-9]*).* gate.html?name=Forums&file=viewtopic&t=$1
RewriteRule ^ftopic([0-9]*).* gate.html?name=Forums&file=viewtopic&t=$1&highlight=$2
RewriteRule ^ftopic([0-9]*).html gate.html?name=Forums&file=viewtopic&t=$1&start=$2&postdays=$3&postorder=$4&highlight=$5
RewriteRule ^author-([a-zA-Z0-9+=]*).html gate.html?name=Forums&file=search&search_author=$1
RewriteRule ^member-([a-zA-Z0-9+=]*)-([0-9]*).html gate.html?name=Forums&file=profile&mode=$1&u=$2
RewriteRule ^sutra([0-9]*).* gate.html?name=Forums&file=viewtopic&p=$1&highlight=$2
RewriteRule ^sutra([0-9]*).* gate.html?name=Forums&file=viewtopic&p=$1

#Members
RewriteRule ^members.html gate.html?name=Members_List
RewriteRule ^members.html gate.html?name=Members_List&file=index
RewriteRule ^members-mode([a-zA-Z0-9]*)-([a-zA-Z0-9]*)-([0-9]*).html gate.html?name=Members_List&file=index&mode=$1&order=$2&start=$3

#NukeJokes
RewriteRule ^jokeid-([1-9][0-9]*).html gate.html?name=NukeJokes&func=JokeView&jokeid=$1
RewriteRule ^jokecat-([0-9]*).html gate.html?name=NukeJokes&func=CatView&cat=$1
RewriteRule ^jokes.html gate.html?name=NukeJokes&file=index

#Reviews
RewriteRule ^reviews.html gate.html?name=Reviews
RewriteRule ^review([0-9]*).html gate.html?name=Reviews&rop=showcontent&id=$1
RewriteRule ^reviews-new.html gate.html?name=Reviews&rop=write_review
RewriteRule ^reviews-sortby-([a-zA-Z0-9]*).html gate.html?name=Reviews&rop=$1
RewriteRule ^reviews-comment-([0-9]*)-([a-zA-Z0-9]*) gate.html?name=Reviews&rop=postcomment&id=$1&title=$2
RewriteRule ^review([0-9]*)-delete.html gate.html?name=Reviews&rop=del_review&id_del=$1
RewriteRule ^review([0-9]*)-del_comment([0-9]*).html gate.html?name=Reviews&rop=del_comment&cid=$1&id=$2
RewriteRule ^review([0-9]*)-edit.html gate.html?name=Reviews&rop=mod_review&id=$1

#Search
RewriteRule ^topic([0-9]*).html gate.html?name=Search&topic=$1

#Sections
RewriteRule ^secart([0-9]*)-page([0-9]*).html gate.html?name=Sections&op=viewarticle&artid=$1&page=$2
RewriteRule ^secart([0-9]*).html gate.html?name=Sections&op=viewarticle&artid=$1
RewriteRule ^secartprint([0-9]*).html gate.html?name=Sections&op=printpage&artid=$1
RewriteRule ^section([0-9]*).html gate.html?name=Sections&op=listarticles&secid=$1
RewriteRule ^sections.html gate.html?name=Sections

#Stories Archive
RewriteRule ^archive.html gate.html?name=Stories_Archive
RewriteRule ^allarticles.html gate.html?name=Stories_Archive&sa=show_all
RewriteRule ^allarticles-([0-9]*).html gate.html?name=Stories_Archive&sa=show_all&min=$1
RewriteRule ^archive([0-9]*)-([0-9]*).* gate.html?name=Stories_Archive&sa=show_month&year=$1&month=$2&month_l=$3

#Surveys
RewriteRule ^surveys.html gate.html?name=Surveys
RewriteRule ^survey([0-9]*)-results.html gate.html?name=Surveys&op=results&pollID=$1
RewriteRule ^survey([0-9]*).html gate.html?name=Surveys&pollID=$1
RewriteRule ^viewreply([0-9]*)-sur([0-9]*)-com([0-9]*)-mode([a-zA-Z0-9]*)-ord([0-9]*)-tre([0-9]*).html gate.html?name=Surveys&file=comments&op=showreply&tid=$1&pollID=$2&pid=$3&mode=$4&order=$5&thold=$6
RewriteRule ^recom([0-9]*)-sur([0-9]*)-mode([a-zA-Z0-9]*)-ord([0-9]*)-tre([0-9]*).html gate.html?name=Surveys&file=comments&op=Reply&pid=$1&pollID=$2&mode=$3&order=$4&thold=$5
RewriteRule ^commentsurvey([0-9]*)-post([0-9]*).html gate.html?name=Surveys&file=comments&pollID=$1&pid=$2
RewriteRule ^commentsurvey.html gate.html?name=Surveys&file=comments

#Top10
RewriteRule ^top.html gate.html?name=Top

#Topics
RewriteRule ^topics.html gate.html?name=Topics

#Web Links
RewriteRule ^addlink.html gate.html?name=Web_Links&l_op=AddLink
RewriteRule ^randomlink.html gate.html?name=Web_Links&l_op=RandomLink
RewriteRule ^toplinks.html gate.html?name=Web_Links&l_op=TopRated
RewriteRule ^poplinks.html gate.html?name=Web_Links&l_op=MostPopular
RewriteRule ^newlinks.html gate.html?name=Web_Links&l_op=NewLinks
RewriteRule ^newlinksw-([0-9]*).html gate.html?name=Web_Links&l_op=NewLinks&newlinkshowdays=$1
RewriteRule ^newlinksd-([0-9]*).html gate.html?name=Web_Links&l_op=NewLinksDate&selectdate=$1
RewriteRule ^vlink([0-9]*).html gate.html?name=Web_Links&l_op=visit&lid=$1
RewriteRule ^links([0-9]*).html gate.html?name=Web_Links&l_op=viewlink&cid=$1
RewriteRule ^linkorder([0-9]*)-([a-zA-Z0-9]*).html gate.html?name=Web_Links&l_op=viewlink&cid=$1&orderby=$2
RewriteRule ^links([0-9]*)-([0-9]*)-([a-zA-Z0-9]*)-([0-9]*).html gate.html?name=Web_Links&l_op=viewlink&cid=$1&min=$2&orderby=$3&show=$4

#Your Account
RewriteRule ^userinfo-([/:¦\s\-\'{}().&_a-zA-Z0-9+=-]*).html gate.html?name=Your_Account&op=userinfo&username=$1

wilderness

12:24 pm on Dec 21, 2004 (gmt 0)

Also, does anyone know how I can prevent being hit on graphics via Google cache? Thats where I get hit the hardest. 43,000 hits in one hour today.

It's not a simple task. Rather, a sound file and directory structure correction which involves chaning your file names and links.

For your images, rather than using names, use numbers and prefixes (letters for categories upon creation.)

Do not store your images in your local directories, rather in an folder specifically for images.

Disallow ALL bots from you newly created image folder in robots.txt

BTW in your Rewritwe lines there are numerous and unecesaary line duplications.

You may eliminate 25 lines from your htaccess with a single line:
RewriteCond %{HTTP_USER_AGENT} ^Web [OR]

This reads "everything that begins with Web" which is the 26 lines you will remove.
There are many other similar replications as well.
You do have to proceed with caution on this regrard. EX: "Go" would not be a sound elimination :)

kamikaze Optimizer

3:24 pm on Dec 21, 2004 (gmt 0)

I did that wilderness, thank you.

I notice, that no matter how I write the code for MS FrontPage, I can rip my site with Front page 2003. :(

Has anyone figured it out?

jdMorgan

5:34 pm on Dec 21, 2004 (gmt 0)

This code is wrong, and may cause you duplicate-content problems.


# change to yoursite.com
ErrorDocument 404 http://www.123-xyz.com
ErrorDocument 404 http://www.123-xyz.com/403.htm

If you read the Apache ErrorDocument description, you will find that this will return a 302 redirect and not a 404. The correct format is:


ErrorDocument 404 /403.htm

Do not use a full URL.

Correct code for MS Frontpage and MSFrontpage:


RewriteCond %{HTTP_USER_AGENT} MS\?FrontPage [OR]

Since your pattern was anchored, it didn't work.

Jim

kamikaze Optimizer

6:08 pm on Dec 21, 2004 (gmt 0)

Should'nt that look like this?, with the ^

RewriteCond %{HTTP_USER_AGENT} ^MS\?FrontPage [OR]

kamikaze Optimizer

6:22 pm on Dec 21, 2004 (gmt 0)

I tried that, and i can still rip the site. :(