Forum Moderators: goodroi
If you optimize a page, say for AV, and want "other" spiders to not index it you would build a robots.txt file. I read, "This can work IF there are NO other links to that page from the rest of your site".
My literal interpretation is that one needs to build multiple web sites under one roof. The robots.txt file would keep AV from looking at optimized pages for Exite and so forth. It is my understanding you need hyperlinks in the "home" page for better ranking among other considerations.
If the optimized page has a link to the index.html and that home page has links or a site map to other pages set up for Google / Yahoo and AV (as an example), you are going to have quite a mess. Obviously, I don't fully understand the concept of doorway and hallway pages and how to implement them.
Well, doorways and hallways work best with cloaking. You can control who gets what version of the page.
If you are not cloaking, then you can do the now you see me - now you don't tricks with robots.txt.
Build one set of directories for what ever engines you are targeting. Mask off the Alta directories in your robits for Excite, and mask off the google from everyone else. It can lead to a pretty complicated robots.txt, but it is worth it if you have enough content to warrent.
g
Thanks.
