wasmith - 2:08 am on Jun 20, 2003 (gmt 0)

MSN had a spider that added SERP content several years ago, my guess is they are reworking that spider. If that were the case they could become live whenever they like what they see. If only to produce the first page, then hit inktomi for the fill in.

Maybe looksmart's days are numbered?

If i recall correctly (anybody who wrote down anything please come in and correct me) it liked keywords to be shown in the title / description (meta) and looked at on page link text (rating only, must have keywords in title / description). Based on the depth of the spidering i got i think they may be looking for off page content on the linked to page ... I tried a micro spider like that once, worked good, in tests of a few sites. Bandwidth becomes a problem if you want to index a billion pages and read all the pages of those billion pages link too! (honest for me it becomes a problem when indexing a single 20MB site); even if the pages only have 20 links.