Major SEO problem with using directory ....

[editordude]

Hey everyone, I run a fansite and for ages now and wanted google to index the pages which I have used mod_rewrite to create and it looks like there is a major problem either concerning my .htaccess code or google itself accessing the code.

Here's the .htaccess file in the home root:

Options +FollowSymlinks
RewriteEngine on
RewriteCond %{HTTP_REFERER}!^http://domain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.domain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://otherdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.otherdomain.com/.*$ [NC]
RewriteRule .*\.(avi¦rar¦mp3¦wmv)$ http://www.example.com [R,NC]

And here is the .htaccess in the 'lyrics' directory:

Options +FollowSymLinks
RewriteEngine on
RewriteBase /lyrics
RewriteRule ^(.*)/song/$ view.php?song=$1 [L]
RewriteRule ^(.*)/album/$ index.php?album=$1

Now the lyrics page within the 'lyrics' folder ( index.php ) has the same links on page as the sitemap which is as so: http://www.example.com/lyrics/albumname/album/

All works well for the visitor but google seems to be stumbling as this morning I checked my error log to see many entries like this:

[Sun Jun 5 01:59:00 2005] [error] [client 65.19.169.234] File does not exist: /home/username/public_html/lyrics/albumnane/album/lyrics/songtitle/song/index.php
[Sun Jun 5 01:59:00 2005] [error] [client 65.19.169.234] File does not exist: /home/username/public_html/lyrics/albumname/album/lyrics/songtitle/song/sitemap.php

Is anyone able to please point out the problem here which I believe has to do with my.htaccess code, many thanks.

[edited by: jdMorgan at 2:02 am (utc) on June 7, 2005]
[edit reason] Example.com [/edit]

[jd01]

Hi editordude,

I do not use rewrite base, but my guess is there is some form of conflict between the base, relative URL's and/or the redirects...

The problem I see is the requests for certatian pages, are either being redirected, or through the use of relative URL's are being interpreted as continuing 'deeper' into the site from the current directory.

/lyrics/albumname/album/ lyrics/songtitle/song/sitemap.php

My guess is these are linked like this '../' or are being 'caught' by the rewrite rule and redirected to this location. Agian, I do not use a rewrite base, and not proficient with it...

Here's why...

I am sure rewritebase is helpful in some cases EG httpd.conf file, or cases where a large htaccess file is in the main directory and processed on every page request, but prefer when using htaccess to only process or include rules that are necessary in each directory to lower server load. IOW I use directory specific htaccess files, rather than a single file.

I believe the intended use of the rewrite base is to save characters and some processing while the file is active. I do not, personally, see the need for it if you are already in the directory, because a rule is only processed until it fails, therefore, as soon as a rule fails the ending (right side) of the rule is never processed, you may gain a slight advantage by not loading the (few) extra characters into memory, but I am not sure how much of that gain is realized when you have to store and retrieve another variable.

If you would like to know more about the function of the base, I suggest waiting for someone like jdMorgan, to respond, or possibly beginning a new thread.

Again, just my thoughts and guesses.

Hope this helps.

Justin

[editordude]

Ah I see, I had thought when using .htaccess in a directory it was essential, shall try without as I'm all for smaller .htaccess files.

So by default the RewriteRules will start from the directory you are in?

So for example I have this:

RewriteBase /lyrics
RewriteRule ^(.*)/song/?$ view.php?song=$1[L]

Would removing the rewritebase line mean I then need to change the rule to include the directory name at start of string?

Thank you very much for all the help given so far!

[jd01]

Would removing the rewritebase line mean I then need to change the rule to include the directory name at start of string?

Yes...

So the rule would look like this:

RewriteRule ^(.*)/song/?$ /the/path/to/view.php?song=$1[L]

OR

RewriteRule ^(.*)/song/?$ /lyrics/view.php?song=$1[L]

Justin

[editordude]

Have removed the base line from the .htaccess and strangely everything works well, but I shall add in the directory for good measure. Thanks for the response.

I final question in this thread on this issue:

Does the <base href=" .... html tag in the head of my document ever effect anything when it comes to mod_rewrite?

Cheers