CGI / VTI / 404 Error Issue

CGI Form Handler No Longer Working

[Jioffri]

After creating the .htaccess file below -- my cgi based form handler no longer works. What I am trying to accomplish is preventing bots from using my forms handler on my website similar to what this site currently does and to allow only those who manually type in the information for submission.

Furthermore, I am also trying to prevent hotlinking, indexing of certain file types, viewing of my .htaccess file via browsers, and a few other things as shown by the syntax below. The # characters below refer to IP addresses I am specifically blocking and were inserted here for example only. The? characters refer to my specific settings on my server and were inserted here for example only.

So, can anyone please tell me where and why the syntax below for the following rulesets may be incorrect? All feedback would be appreciated. I am still learning and would like to know where I have done something wrong and why. Thanks.

Jio

P.S. I know that the GET POST ruleset is not needed if I am using the POST ruleset at the bottom of the .htaccess file as shown below. I have only inserted it here for clarification along with everything else and to ask as to which would be best to use if I generally do not need such a restrictive parameter to prevent bot posting since it is really not an issue. However, it would be nice to learn how this bot post restriction works so that perhaps I can implement it in the future for some more advanced sites I have yet to develop. Furthermore, when I use either the GET POST or POST ruleset the 404 error shown below appears. It also appears even if I remove both. Perhaps the problem lies in the hotlinking ruleset since it also refers to the cgi I have in place?

------------------------------------------
# -FrontPage-

IndexIgnore *.htaccess *.gif *.jpg *.jpeg *.bmp *.wmv *.swf *.png *.pdf *.doc *.txt/.?* *~ *# */HEADER* */README* */_vti*

AuthName www.anywebsite.com
AuthUserFile /var/chroot/home/content/?/?/?/myuserid/html/_vti_pvt/service.pwd
AuthGroupFile /var/chroot/home/content/?/?/?/myuserid/html/_vti_pvt/service.grp

<Limit GET POST>
Order Allow,Deny
Allow from all
</Limit>

<Limit PUT DELETE>
Order Allow,Deny
Deny from all
</Limit>

<Files .htaccess>
Order Allow,Deny
Deny from all
</Files>

Order Allow,Deny
Deny from ###.##.###/18
Allow from all

Order Allow,Deny
Deny from ###.###.##/18
Allow from all

Order Allow,Deny
Deny from ###.###.##/18
Allow from all

Order Allow,Deny
Deny from .anywebsite.com
Allow from all

Order Allow,Deny
Deny from ###.###.##.##
Deny from ###.###.##.###
Deny from ###.###.##.###
Deny from ###.###.##.###
Deny from ###.###.##.#
Deny from ###.###.##.#
Allow from all

RewriteEngine On
RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://anywebsite.com [NC]
RewriteCond %{HTTP_REFERER}!^http://www.anywebsite.com [NC]
RewriteRule .*\.(gif¦jpg¦wmv¦swf¦png¦mp3¦pdf¦doc¦jpeg¦bmp¦zip¦css)$ - [NC,F]

RewriteEngine On
# redirect images
RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://(www\.)?anywebsite\.com/ [NC]
RewriteRule \.(gif¦jpe?g?)$ /no.$1 [NC,L]
# block scripts
RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://(www\.)?anywebsite\.com/ [NC]
RewriteRule \.(cgi¦pl)$ - [NC,F]

RewriteEngine On
RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://(www\.)?anywebsite.com/.*$ [NC]
RewriteRule \.(gif¦jpg¦wmv¦swf¦png¦mp3¦pdf¦doc¦jpeg¦bmp¦zip¦css)$ - [F]

RewriteEngine On
RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://anywebsite.com [NC]
RewriteCond %{REQUEST_METHOD} ^POST$
RewriteRule ^/POSTprocessing.cgi - [F]
------------------------------------------

As a result of the .htaccess file shown above -- whenever attempting to submit a form on my site the 404 server side page below is issued:

Forbidden
You don't have permission to access /_vti_bin/shtml.exe/feedback.htm on this server.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

Apache/1.3.31 Server at www.anywebsite.com Port 80

[jdMorgan]

Did you check the other thread you posted in?

You have a lot of duplicate rules in the post above, and also multiple RewriteEngine on directives, which are not needed - you only need one. You may use only one Order directive per .htaccess file, so that needs to be fixed, too.

The easiest way to identify which block of code is causing a problem is to comment them out one-by-one by putting a "#" at the beginning of the line. Then see if the problem goes away with a particular block of RewriteConds and RewriteRules commented-out.

Jim

[Jioffri]

Jim,

Thanks for the rapid response! So, as for the Order directives, I can just combine all of those under one? OK, cool. As for the Rewrite, as long as I have the Conditions followed by the corresponding Rule then it will be fine to have only one Rewrite? OK, cool too. Thanks for the tip on the # troubleshooting. I will try it and see if I can see what your experienced eyes see.

As for the other thread, you lost me on that one. Are you referring to the hotlinking issue which is different than this one entire issue overall? Thanks.

Jio

[Jioffri]

Jim,

Thanks for the info on the other issue. I will check it out.

Jio

[Jioffri]

Jim,

OK, after researching every resource you gave me I learned a lot and wanted to post with my results in case someone else needs help and to thank you for all of your feedback. Also, since the thread info you referred me to is no longer allowing posting I felt that maybe I could just resolve all of my nonsense here once and for all! ;)

So, after replacing Options None with “Options +FollowSymLinks” and adding

”<Files .htaccess>
order deny,allow
deny from all
</Files>”

at the end of the .htaccess files of the three primary vti folders under root and creating the following .htaccess file for root:

------------------------------------
# -FrontPage-

IndexIgnore *.htaccess *.gif *.jpg *.jpeg *.bmp *.wmv *.swf *.png *.pdf *.doc *.txt *.css */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
AuthName www.example.com
AuthUserFile /. . . . . . ./#/#/#/userid/html/_vti_pvt/service.pwd
AuthGroupFile /. . . . . . ./#/#/userid/html/_vti_pvt/service.grp
<Files .htaccess>
order deny,allow
deny from all
</Files>

order deny,allow
deny from .example.com
deny from xyz.yzx.xyz.xyz
deny from . . . . . . .
allow from all

RewriteEngine On
Options +FollowSymLinks
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_HOST} ^example\.com [NC]
RewriteRule (.*) http://www.example.com/$1 [R=301,L]
# block hotlinking
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^http://example.com [NC]
RewriteCond %{HTTP_REFERER} !^http://www.example.com [NC]
RewriteRule .*\.(gif¦jpg¦jpeg¦wmv¦¦pdf¦bmp¦swf¦png)$ - [NC,F]
# block redirect images
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^http://(www\.)?example\.com/ [NC]
RewriteRule \.(bmp¦wmv¦gif¦jpe?g?)$ /no.$1 [NC,L]
# block scripts
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^http://(www\.)?example\.com/ [NC]
RewriteRule \.(cgi¦pl)$ - [NC,F]
# block bot post
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} example.com [NC]
RewriteCond %{REQUEST_METHOD} ^POST$
RewriteRule ^/POSTprocessing.cgi - [F]
-----------------------------------

I was able to accomplish all of my goals as follows:

1.Prevent hotlinking of images and other media files based on some tests I did to analyze the syntax. I may be wrong with the tests and if there are resources you can provide to allow me to test the site better I would appreciate it.
2.Prevent bot posting.
3.Prevent 301/302 redirects from Alexa and skyscraper sites.
4.Prevent indexing of files I want to remain off limits.
5.Prevent viewing of my .htaccess files that have the info above.
6.Allow cgi based forms handling that worked correctly on the site using FrontPage extensions on an Apache server.
7.Allow all form info to be emailed correctly to the correct email addresses used by the forms.

I suppose it is not very important to prevent viewing of the .htaccess files (or is it?) but I did it anyway. So, my only questions for you now are:

A.Is the syntax I used above as correct as it should be in your opinion? If you see any errors can you please point out the resource(s) that will help me identify them?
B.Is the syntax I used streamlined enough to prevent unnecessary looping in your opinion? Considering of course how you advise that the syntax should be optimized for maximum efficiency?
C.When is it appropriate to group all of the Limit identifiers GET POST PUT DELETE into one directive and deny all or allow all?

Once again, Jim, thanks for all of your help and feedback.

Jio

[edited by: jdMorgan at 2:12 pm (utc) on Aug. 18, 2005]
[edit reason] Minor formatting cleanup. [/edit]

[jdMorgan]

Eliminate redundant RewriteConds, escape literal periods, remove unnecessary ".*" at beginning or end of unanchored patterns, reduce "jpg¦jpeg", remove double-¦ sequence in RewriteRule pattern:

# block hotlinking
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^http://[b](www\.)?[/b]example\.com [NC]
RewriteRule \.(gif¦[b]jpe?g[/b]¦wm[b]v¦p[/b]df¦bmp¦swf¦png)$ - [NC,F] 

The 'escape literal periods, remove unnecessary ".*" at beginning or end of unanchored patterns' changes should also be made to all your other rules.

The following constructs are a waste of CPU:

RewriteRule .*\.(gif¦jpg¦wmv¦swf¦png¦mp3¦pdf¦doc¦jpeg¦bmp¦zip¦css)$ - [NC,F]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?anywebsite.com/.*$ [NC]

and could be changed to

RewriteRule [b]\.(g[/b]if¦[b]jpe?g[/b]¦wmv¦swf¦png¦mp3¦pdf¦doc¦bmp¦zip¦css)$ - [NC,F]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?anywebsite.[b]com[/b] [NC]

Without access to your server (and a contract), I really can't tell if all this is completely "correct". The best way to tell is by testing it using your knowledge of the site and how it should work.

Jim