Please explain this htaccess file

htaccess, loops

[Ivanna]

I have issues with a site for which I am now admin. I run a site audit using a sitemap generator and it gives many hundreds of 400 errors and each has thousands of internal links. I think it is looping or something perhaps.

(I run this sitemap generator on other sites and they are ok, so the sitemap generator is not the problem)

All of the failed pages are error 400 Bad Request also they are all iso-8859-1. The pages that pass are utf-8. I have identified the places that make the iso-8859-1 pages.

When I analyse the first failed url

http://MyDomainName.com/../../

it then resolves to

http://MyDomainName.com/

and is ok but something makes all the errors before it resolves.

This is the content of public_html/.htaccess

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
AuthName MyDomainName.com
AuthUserFile /home/MyUserID/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/MyUserID/public_html/_vti_pvt/service.grp

I do not understand any of this but the vti bit stand up because one place that makes the iso charset is _vti_inf.html

Will somebody tell me if this htaccess is ok.

Thank you
Ivanna

[wilderness]

1) 400 errors are not loops (i. e., 500 errors) rather denials of access.

2) the 5th and 10th lines state "deny from all", you'll need to list exceptions or change the method.

3) MS-FrontPage has been obsolete for nearly a decade. Are you running FP servers?

[Ivanna]

Thank you for replying.

1. yes they are 400 errors. It says Bad Request. So why is the request bad or why is access denied. That is what I do not understand.Is it because something puts ../../ at the end of the failed URL?

2. I do not understand this either. order deny,allow and deny from all are written twice. Why? I do not know what the exceptions will be or to what method I must change.

3. I do not use FrontPage. I have just taken this site as admin. I do not think they are FP servers. I checked whois and the DNS server is in the name of the designer/host but I have suspicions it is GoDaddy.

thank you
Ivanna

[wilderness]

1. yes they are 400 errors. It says Bad Request. So why is the request bad or why is access denied. That is what I do not understand.Is it because something puts ../../ at the end of the failed URL?

2. I do not understand this either. order deny,allow and deny from all are written twice. Why? I do not know what the exceptions will be or to what method I must change.

The second statement is the answer to the 1st statement.

You have two options for the following, depending upon what you desire:

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>

1) list IP's that you wish to allow after the deny from all line
OR
2) simply remove the deny from all line and the next line (allow all) will take precedence (in its current state it doesn't function at all.

3) You have no way of knowing what old pages are on the site, unless you've redesigned them entirely, however you may be able to remove the FP lines and FP Servers completely. You may be able to use an html tool to search the files (offline) and determine if any use FP.

[Ivanna]

Thank you wilderness.

So, option 2. The rule should be:

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
</Limit>
AuthName MyDomainName.com
AuthUserFile /home/MyUserID/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/MyUserID/public_html/_vti_pvt/service.grp

Is that correct?

I asked questions on a thread last week about settings on htaccess and I want to add it on this site.

This is what I would add

# Redirect index.php and .ph or html and .htm to folder
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.(php|html?)\ HTTP/
RewriteRule ^(([^/]+/)*)index\.(php|html?)$ http://www.MyDomainName.com/$1 [R=301,L]

# Redirect non-canonical to www
RewriteCond %{HTTP_HOST} !^(www\.MyDomainName\.com)?$
RewriteRule (.*) http://www.MyDomainName.com/$1 [R=301,L] 

] 

Should this rule be first and then option 2. next?

Thank you for your help
Ivanna

[wilderness]

The following should remain the same.

<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>

Whatever mod_rewrite lines you add should come after the end of the others.

[Ivanna]

So, my htaccess should be like this, yes?

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
</Limit>
AuthName MyDomainName.com
AuthUserFile /home/MyUserID/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/MyUserID/public_html/_vti_pvt/service.grp 

# Redirect index.php and .ph or html and .htm to folder
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.(php|html?)\ HTTP/
RewriteRule ^(([^/]+/)*)index\.(php|html?)$ http://www.MyDomainName.com/$1 [R=301,L]

# Redirect non-canonical to www
RewriteCond %{HTTP_HOST} !^(www\.MyDomainName\.com)?$
RewriteRule (.*) http://www.MyDomainName.com/$1 [R=301,L]

] 

thank you
Ivanna

[wilderness]

Change this:

<Limit PUT DELETE>
order deny,allow
</Limit>

to this

<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>

Everything else appears O.K., with the exception of your followup to remove the FP lines.

[Ivanna]

Wilderness,
thank you for all your help. I am very grateful.
I have changed the code as directed and will add that to htaccess.

For the Front Page, it means knowing what to search for.

I made a search of the backups.
Front page tell me nothing but when I search for FP I found this - vti_webservertype:SR|apache-fp in file www\_vti_pvt\service.cnf

So it is perhaps a front page server?

thank you
Ivanna

[slipkid]

As Wilderness has explained, the .htaccess code you have was the default code placed on the website when the site was configured with Frontpage Extensions. Frontpage is a web authoring program that required a unique .htaccess file. In addition, Microsoft's Frontpage Extensions were required to be installed on the server for the website to be served to users.

As has been pointed out, Microsoft could not get the code for the .htaccess file correct. The suggestions offered rectify the errors.

Since you said you are not using Frontpage, I presume you are uploading changed and new pages via FTP. It is likely that much of the site has pages originally created with Frontpage, and these pages will likely not validate if run through the W3 Validator. One reason is that Microsoft used proprietary "webbots" to create special features.

FYI, one of the features of Frontpage was that it allowed you to upload pages directly from your desktop without the need of an FTP program.

Frontpage used a system of relative notation for links that allowed the webpage to be rendered on your desktop using your preferred browser. A normal link, e.g., www.example.com/directory/page.html might be coded as ../page.html depending on where you were on the directory tree. It appears to me that you may have a mix of Frontpage created pages and pages created by some other program or hand-coded. This may be why you are seeing 400 errors when you run the site through a sitemap generator.

When I dumped Frontpage as my site authoring tool, I notified my web hosting company to remove Frontpage Extensions. I had to go through each web page and remove any Microsoft webbots I was using and simultaneously repair all the links so that in the example given above the link reference now became /directory/page.html.

To create a sitemap using your sitemap generator may require you to go through the pages to remove any traces of Frontpage code and redo the links.

Make a backup of your site before making any changes. Any directory with a beginning underscore is a Frontpage directory which you will lose when your web hoster disables Frontpage Extensions (assuming that they are not already disabled).

[Ivanna]

Slipkid,
thank you for this explanation of FrontPage. I have never used this.

I presume you are uploading changed and new pages via FTP.

No, I have just taken this as new admin and have only made an audit of the site before I know what I must do.

My hope is to move this site to a new host and also a new cms/ecommerce format.(with the owner permission) That would be the easy thing to do but I need to show the problems and what causes the problem. There is also a problem working with the current designer/host for cooperation.

The owner may not move at this time so I must understand what else to do.

thank you
Ivanna

[lucy24]

<Limit GET POST>
order deny,allow
allow from all
</Limit>

Technically this envelope isn't necessary at all. "Order Deny,Allow" is the default [httpd.apache.org]. If you're not planning to block anyone using "Deny from..." directives-- which frankly seems insane to me, but it's your site-- you don't need to say anything.

Apache also says, quote,

In the general case, access control directives should not be placed within a <Limit> section.

and

A <LimitExcept> section should always be used in preference to a <Limit> section when restricting access, since a <LimitExcept> section provides protection against arbitrary methods.

[Ivanna]

Hello Lucy,
Please understand, I did not write this. It is already in the file. Also I did not understand it, therefore the title of this post.
Also, I do not understand who is to be allowed or whom to deny or how to do it correctly.

I think I do not ask questions correctly to receive the proper answer.

How do I write a correct allow/deny sentence please.

thank you
Ivanna

[slipkid]

Front page tell me nothing but when I search for FP I found this - vti_webservertype:SR|apache-fp in file www\_vti_pvt\service.cnf

Any filename with _vti in it is a Frontpage file.

I suspect that Frontpage Extensions are installed and working.

It would help your understanding of how to proceed if the prior webmaster would at least tell you if Frontpage Extensions are enabled, or if Frontpage was still being used to update the site.

Alternatively, you could email your web hoster and ask if Frontpage Extensions are enabled. It is possible that the prior webmaster had abandoned Frontpage in lieu of FTP for adding or making changes to HTML pages (not recommended practice). Perl scripts, on the other hand, in the cgi-bin would have to be uploaded using ASCII via FTP.

[Ivanna]

Hello Slipkid,
I did not learn Frontpage in school so I will believe what you said.
Unfortunately the prior webmaster is also the web hoster and does not answer my emails. I think the situation was that the site was set up and hosted by the same person without providing recurring support. The site is quite old but did not even have analytic code added. The only analysis is awstats.

I am trying to tell the owner she must move to have better support. I think it is a budget problem. Perhaps I will offer a site for free, or very low cost, and I will also gain experience. I want to offer a perfect solution and perfect support and get a good report from her and then more clients perhaps. I hope.

thank you all for helping me understand some more of htaccess. It is very perplex, but one day...

thank you
Ivanna