homepage Welcome to WebmasterWorld Guest from 54.204.94.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt and htaccess
.htaccess circumvents to site root
stefan009




msg:1527521
 3:47 pm on Jan 24, 2004 (gmt 0)

Hi,

I currently am using htaccess to cause a jump into a part of my site that is excluded from search robots etc.

This means the root: www.my_url.com/index.htm is redundant for the purposes of the search bot because the htaccess file grabs it first.

The trouble is I ONLY want the www.my_url.com/index.htm indexed.

Any help would be much appreciated.
I found this forum by accident last year, it is one of the better ones, thanks.

I have implemented the "perfect .htaccess banlist recently, still testing it.." [webmasterworld.com...]

Kind regards, Steve

[edited by: stefan009 at 5:36 pm (utc) on Jan. 24, 2004]

 

tschild




msg:1527522
 4:02 pm on Jan 24, 2004 (gmt 0)

Wouldn't your purpose also served by not redirecting the home page (and the other pages) but just storing everything except the home page in a subdirectory (say /pages/) and disallowing this subdirectory to robots in the robots.txt?

pendanticist




msg:1527523
 4:06 pm on Jan 24, 2004 (gmt 0)

First off, Welcome to WebmasterWorld [webmasterworld.com], stefan009 :)

Suggested Reading: TOS [webmasterworld.com], especially # 13.

You may still have time to edit your post. ;)

stefan009




msg:1527524
 5:35 pm on Jan 24, 2004 (gmt 0)

Hello pendanticist,

sorry about that, I'd be happy if the post was deleted.

the re-direction... is dynamic using an error 404 trap, so it can go anywhere depending on the suffix at the end og the url so your_url/abc goes one place and your_url/xyz goes someplace else.

If the index.htm in the site root is bypassed because of what I have in my .htaccess there probably isn't much I can do without changing the set-up, unless the robot was to ask for your_url/index.htm instead of your_url period.

the htaccess will leave alone any request with a file specified.

regards, Steve.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved