Slurp: Extremely fast and thorough

why is googlebot slower?

[walkman]

It's amazing how fast slurp works. I changed the path to my directories and within minutes slurp was following the new links. Over 3000 requests today alone.

I cannot say the same about GoogleBot; sometimes it takes weeks for Googlebot to pick up new pages. Any idea why the discrepancy? For the most part both engines have as full of a web catalog as possible so that cannot be explain Y's aggressiveness. I get about 4-5 times more slurp requests a day.

Am I the only one seeing this?

[BillyS]

I see this too. Yahoo is quicker to start showing a page in their index and they are much more aggressive than Google (it seems...).

For example, my stats package shows Yahoo spidering 14,000 pages so far this month (on a site with around 1,300 pages). To me this seems kind of overboard...

Googlebot, on the other hand shows only around 1,500 pages. But that's probably skewed because I run Adsense and as Matt Cutts has pointed out Google has made some changes that make spidering more efficient.