Forum Moderators: goodroi

Message Too Old, No Replies

Need help with a specific robots.txt question

         

Kurgano

9:16 pm on Jun 10, 2007 (gmt 0)

10+ Year Member



On my site I have a demo folder in which the index file allows people to see different versions of the same thing. It holds a few hundred "demos" of a certain product. The text/links on each demo page is virtualy the same but the graphics and layout are vastly different.

The index page of this demo folder is likely to be bookmarked and linked to quite a bit but I don't want or need google to index every single version of the same thing.

example.com/demo/index.html is one that I want showing up in results and I want PR to flow to it and out of it.

My problem is this, every demo page opens in a frame that covers the bottom 80% of the screen. The URI never changes, the controls are in the main frame along with some text etc. Will crawlers ignore the lower frame (hence each version of the demo) completely? or do I need to worry that the crawlers will find the frame and click on links in it?

If crawlers follow links from each demo they will visit redundant content to an extreme. Do i need to find a way to exclude the frame or am I good to go because the demos are in a frame?

I cant change the URI for each version right now, theyre all the same. Visitors only see one URI but can control the bottom demo portion.

trying to avoid future headaches.

goodroi

4:36 pm on Jun 12, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



My past experience has been that using frames which call on pages that are blocked from bots will avoid duplicate content issues.

I have not done this for a few months so it might be different now. If you want to be certain you could run a quick test on a throw away domain.

londrum

8:26 pm on Jun 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



not sure i've followed your question correctly... but if the lower frame just contains the controls, and you only really want the top frame indexed, then can't you just put a robots meta-tag in the header of the bottom page?, or maybe a robots nofollow meta-tag.

goodroi

9:18 pm on Jun 12, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



you can also place those pages that appear in the iframe in a subdirectory that is blocked by robots.txt