Crawlers Out In Force.

[Brett_Tabke]

This week, we saw full crawls from:

Fast
Inktomi
Google
Wisenut
Altavista

That's all the major crawlers but Teoma. I think that is the first time in ages where 5 engines have all been actively crawling at the same time.

It's nice to see them all out fighting for it again!

[dantheman]

And inktomi just looped and looped while generating 20,000 404 hits. All this due to a handful of links that were out by one folder level. Just makes a huge mess of the logs!

But it sure was a busy week.

[mayor]

Brett, I haven't seen Inktomi in my logs, except PFI pages. I have some older sites that have some PFI pages, and get only PFI crawls, and new sites that, despite having good links from other established sites, haven't been getting crawled by Slurp.

Can you tell if the Ink crawling you are seeing includes new sites (three months old or newer), or free pages on PFI sites?

[Brett_Tabke]

Sorry, all older sites Mayor.

[john316]

Very nice activity from the major players, serps are improving as well.

I'm glad to see that there is still a race!

If AV would kill the poker/toner invitation links after each results page, they might get back faster, credibility is pretty well demolished with that nonsense.

[edited by: john316 at 3:36 pm (utc) on Nov. 28, 2002]

[creative craig]

I had Fast and Google taking near enough the same pages at the same time yesterday, and Wisenut is currently having a look around my site right now :)

[freddoo]

How can I know that a full craw was done on my site.
I have an old site but optimized it in mid octobre, no changes on the engines so far.

My cached page in google is still the old site.

When can I expect to see my modifications?

thank you.

[willtell]

We have Inktomi and LS on our site, every single day. Should I be concerned?

[martinibuster]

Even Robozilla stopped by to see if I was still around!

[Namaste]

All this is very nice, but the relevency of results from all except google is E grade.

Hands up all those that will use anything but google if looking for nightclubs and discount hotels for your next vacation to Spain.

Why don't the search engines understand that 1st it's relevancy . Just read all kinds of nonsense in Inktomi's press release about Geo relevance and blah blah. Typical kind of press release put out by an overpaid & disinterested PR department.

Then some user called "Inktomi" says "we're very excided about the future"...well my friend, we are not!