Forum Moderators: open
I just had a fast spider visit in the new section of my site. Apart from going through the whole new tree structure, he also spidered all the menu links on those pages in the following strange way:
www.starec.com/tree/branch1
(that's fine, he is welcome)
and then he goes for
www.starec.com/tree/branch1/menuitem1
and so on
then he goes deeper
www.starec.com/tree/branch1/branch11
(fine again)
and then he goes for the same menu links:
www.starec.com/tree/branch1/branch11/menuitem1
and it continues really deep down...
So, if I have a fairly structured tree structure and 10 menu links per page, about 5.000 duplicates got spidered.
It is related with the relative urls, in the rest of the web I use absolute urls and fast does not exhibit this behaviour.
What I would like to know: can I be punished because fast guys think that my site is a heap of duplicated pages, I am sure there must be some duplicate content checking mechanism implemented in the algo. If not, I will be the king in the new index:-).
Another question: how to benefit from this fast behaviour? Any ideas?