Search Engine Coordination

A question of timing

[MarkHutch]

The past couple of days we have seen Googlebot crawl some of our sites deeper than they have done the past three months. We are very thankful for that. Also, during this time I have noticed that Googlebot is the only major search engine that is deep crawling these sites.

This kind of made me wonder if the major search engines have a plan where they don't hit the same IP numbers at the same time when they are doing major crawling. About two weeks ago, Fast hit the same sites pretty hard and then last week AV hit them pretty hard. Surely this can't be a coincidence. It does make sense that major SE's wouldn't want to crash severs by hitting the same ones at the same time. Anyone have any thoughts about this?

[Filipe]

I doubt they would crash servers. I mean, you can have more than 3 users crawling your site right? Why couldn't you have 3 search engines crawling your site?

I would doubt that search engines would share this kind of information with each other, so the likeliness of coordination is slim.

[MarkHutch]

Maybe our situation is different, but when the major search engines hit our sites, they usually run very fast. Many times Googlebot, Fast and AltaVista will fetch 3-5 pages per second. I was just tossing this out for comment. The major search engines just don't seem to deep crawl at the same time on our sites.