A bit new to the game and have several thoughts (mostly confusion) I need comments on if anyone has the time. I would appreciate it.
If you optimize a page, say for AV, and want "other" spiders to not index it you would build a robots.txt file. I read, "This can work IF there are NO other links to that page from the rest of your site".
My literal interpretation is that one needs to build multiple web sites under one roof. The robots.txt file would keep AV from looking at optimized pages for Exite and so forth. It is my understanding you need hyperlinks in the "home" page for better ranking among other considerations.
If the optimized page has a link to the index.html and that home page has links or a site map to other pages set up for Google / Yahoo and AV (as an example), you are going to have quite a mess. Obviously, I don't fully understand the concept of doorway and hallway pages and how to implement them.
Hi Gary (apoligies for the overlooked post - it happens rarely)
Well, doorways and hallways work best with cloaking. You can control who gets what version of the page.
If you are not cloaking, then you can do the now you see me - now you don't tricks with robots.txt.
Build one set of directories for what ever engines you are targeting. Mask off the Alta directories in your robits for Excite, and mask off the google from everyone else. It can lead to a pretty complicated robots.txt, but it is worth it if you have enough content to warrent.
Hi Gary, lots of GREAT info. in the cloaking forum, there you will find mentions of different cloaking softwares, styles of cloakings, comments on the ethics of cloaking, how SE's perceive cloaking, etc. If you are interested in this, I'd reccommend you start by checking out that forum to find comprehensive info.