homepage Welcome to WebmasterWorld Guest from 54.226.10.234
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Doorway / Hallway pages Vs. Robots.txt
Does one need to build "many" sites under one roof?
Garyh



 
Msg#: 73 posted 5:43 pm on Jan 13, 2001 (gmt 0)

A bit new to the game and have several thoughts (mostly confusion) I need comments on if anyone has the time. I would appreciate it.

If you optimize a page, say for AV, and want "other" spiders to not index it you would build a robots.txt file. I read, "This can work IF there are NO other links to that page from the rest of your site".

My literal interpretation is that one needs to build multiple web sites under one roof. The robots.txt file would keep AV from looking at optimized pages for Exite and so forth. It is my understanding you need hyperlinks in the "home" page for better ranking among other considerations.

If the optimized page has a link to the index.html and that home page has links or a site map to other pages set up for Google / Yahoo and AV (as an example), you are going to have quite a mess. Obviously, I don't fully understand the concept of doorway and hallway pages and how to implement them.

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 73 posted 12:54 pm on Feb 28, 2001 (gmt 0)

Hi Gary (apoligies for the overlooked post - it happens rarely)

Well, doorways and hallways work best with cloaking. You can control who gets what version of the page.

If you are not cloaking, then you can do the now you see me - now you don't tricks with robots.txt.

Build one set of directories for what ever engines you are targeting. Mask off the Alta directories in your robits for Excite, and mask off the google from everyone else. It can lead to a pretty complicated robots.txt, but it is worth it if you have enough content to warrent.

Garyh



 
Msg#: 73 posted 2:38 pm on Feb 28, 2001 (gmt 0)

Thanks for the reply. I need to reflect on your comments for a day or so. (Also busy as a one armed paper hanger) :)

g

Garyh



 
Msg#: 73 posted 3:35 am on Mar 20, 2001 (gmt 0)

Where can I find a decent document on cloaking.

Thanks.

2_much

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 73 posted 10:45 pm on Mar 20, 2001 (gmt 0)

Hi Gary, lots of GREAT info. in the cloaking forum, there you will find mentions of different cloaking softwares, styles of cloakings, comments on the ethics of cloaking, how SE's perceive cloaking, etc. If you are interested in this, I'd reccommend you start by checking out that forum to find comprehensive info.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved