Forum Moderators: open

Message Too Old, No Replies

Fast and relative links spidering

strangely composed urls spidered

         

starec

11:52 am on May 3, 2001 (gmt 0)

10+ Year Member



It looks like fast does not understand the base and does not treat the relative links correctly. Or is me who does not understand it correctly?

I just had a fast spider visit in the new section of my site. Apart from going through the whole new tree structure, he also spidered all the menu links on those pages in the following strange way:
www.starec.com/tree/branch1
(that's fine, he is welcome)
and then he goes for
www.starec.com/tree/branch1/menuitem1
and so on

then he goes deeper
www.starec.com/tree/branch1/branch11
(fine again)
and then he goes for the same menu links:
www.starec.com/tree/branch1/branch11/menuitem1

and it continues really deep down...

So, if I have a fairly structured tree structure and 10 menu links per page, about 5.000 duplicates got spidered.

It is related with the relative urls, in the rest of the web I use absolute urls and fast does not exhibit this behaviour.

What I would like to know: can I be punished because fast guys think that my site is a heap of duplicated pages, I am sure there must be some duplicate content checking mechanism implemented in the algo. If not, I will be the king in the new index:-).

Another question: how to benefit from this fast behaviour? Any ideas?