Why do competing engines seem to crawl concurrently?

A.K.A. It never rains, it pours...

[uber_boy]

A phenomenon I've noticed over the past year that I've finally decided to raise here is that of being crawled simultaneously by the major search engines. Like most other people here, Google's bots are the only ones I really care about. But in my ongoing efforts to monitor their activities, I've repeatedly noticed that, when they visit, so do the bots of other search engines. To wit, I'm currently being crawled by Google's deep and fresh bots, Inktomi's bots, Fast's bots, LookSmart's bots, AOL's cache bots, and TurnItIn's bots. Yet, as soon as Google packs it in for the month, all except perhaps TurnItIn's will also pack it in. Thus, what I want to know is whether this is coincidence -- i.e., it's only happening to me -- or whether the controllers of the bots are intentionally unleashing their bots at the same time as Google. As our friends at National Geographic are fond of saying, inquiring minds want to know!

[Brett_Tabke]

Just be thankful that WiseNut and Teoma didn't join the crowd too ;-)

Ya, it is strange that they all gang up some times. I've often wondered if their access providers weren't offering some deal for that particular day or time. They all tend to come in through Exodus, so it very well could be something contractual.

[pendanticist]

As our friends at National Geographic are fond of saying, inquiring minds want to know!

Uhm, it's Enquirer....

Anyway, this situation you've described happens all the time at my domain. In fact, I've even had two of AV's visiting me simultainiously and they have three.

There is not a single day goes by without one of the 'majours' crawling through all me files.

(I should point out my domain is of the academic nature, currently contains some 5,000+/- links, is non-commercial (despite the .com at the end) and - while I do keep the links repaired - the crawls do not constitute 'fresh content' crawls.)

So...sit back, relax and enjoy the attention you've been afforded.....

Pendanticist.

[garylo]

Thus, what I want to know is whether this is coincidence

Yep, probably mere coincidence

[futureX]

Mine seem to come around at the same time too... Except for freshbot who visits daily :P

Although they are scattered around the month the most activity is around about the same time as google.