Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

inktomi can spider links with GET information

/product.php?product=123 is OK for inktomi

10:16 am on Dec 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:June 21, 2003
votes: 0

Just in case anybody doesn't know, the inktomi robot (Slurp) can spider links with information after a question mark... this goes for both PFI and free inktomi.
4:28 pm on Dec 16, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 14, 2003
votes: 0

yes, but in many cases, INK is grabbing that info incorrectly. For example, it is treating dynamic links to outside websites as if they were duplicate copies of the sites they are linking to. This is a problem especially since INK just tightened their dupe content filter.