Flow of Robots and Spiders

[agerhart]

I was just checking our log files, looking for Googlebot and some other spiders and bots.

I noticed that some of the bots and spiders were deep crawling, but it was spaced out in some parts. Is this a bad thing?

Does this imply that the bots and spiders are having a tough time getting through the site?

[Will]

Shouldn't matter (it would largely depend on how the bot was programmed in any case).

Provided the bot visits pages that are two or three levels deep, it is safe to assume that they are reachable without problems. Some bots are known to grab only one or two pages on each visit then leave (e.g. Slurp).

[agerhart]

Thanks Will, I was looking for a steady flow list of pages that were grabbed....I guess I was looking for something that I shouldn't have been.