New googlebot crawlrate is unusually fast

[gn_wendy]

Looking at some of the sites i work on; google is crawling at unusually, and in some instances, crazy speeds.

One site is up by more than 15,000 crawled pages per hour. It is a huge site with lot's of traffic, so it's not affecting the servers.

All the sites can easily handle the extra traffic from GBot - so I am nothing but pleased that pages are getting refreshed and indexed faster.

My question: is anyone else seeing this (part of caffeine GFS-update) or is it only my sites?

[tedster]

Some other members have mentioned this new lightning fast crawl in the MAYDAY thread. And some clients of mine whose servers can handle it are also reporting very fast googlebot crawls. Since the new found speed seems to respect the servers involved, as you say, it can only be a good thing.

It looks like Google's crawl team has done some good work.

[cien]

They were all over my site last month. Sudden spike in G.bots crawling up to 60 at a time all day for a whole month and suddenly one day they disappeared. Didn't get absolutely any traffic while the crawling. Site is dead still since the beginning of March. Few visitors here and there from 1000s/day. Caffeine has been horrible to my site.

[Chrispcritters]

I've seen a huge increase in crawl as well. Interesting thing is that WMT showed a huge number of pages crawled, and then adjusted that number down about 50% with the next refresh of the data -- but still a big increase. Good or bad it just happens to correspond with a significant site overhaul.