robots.txt and htaccess

.htaccess circumvents to site root

[stefan009]

Hi,

I currently am using htaccess to cause a jump into a part of my site that is excluded from search robots etc.

This means the root: www.my_url.com/index.htm is redundant for the purposes of the search bot because the htaccess file grabs it first.

The trouble is I ONLY want the www.my_url.com/index.htm indexed.

Any help would be much appreciated.
I found this forum by accident last year, it is one of the better ones, thanks.

I have implemented the "perfect .htaccess banlist recently, still testing it.." [webmasterworld.com...]

Kind regards, Steve

[edited by: stefan009 at 5:36 pm (utc) on Jan. 24, 2004]

[tschild]

Wouldn't your purpose also served by not redirecting the home page (and the other pages) but just storing everything except the home page in a subdirectory (say /pages/) and disallowing this subdirectory to robots in the robots.txt?

[pendanticist]

First off, Welcome to WebmasterWorld [webmasterworld.com], stefan009 :)

Suggested Reading: TOS [webmasterworld.com], especially # 13.

You may still have time to edit your post. ;)

[stefan009]

Hello pendanticist,

sorry about that, I'd be happy if the post was deleted.

the re-direction... is dynamic using an error 404 trap, so it can go anywhere depending on the suffix at the end og the url so your_url/abc goes one place and your_url/xyz goes someplace else.

If the index.htm in the site root is bypassed because of what I have in my .htaccess there probably isn't much I can do without changing the set-up, unless the robot was to ask for your_url/index.htm instead of your_url period.

the htaccess will leave alone any request with a file specified.

regards, Steve.