Most aggressive crawl I have ever seen.

Gbot pulled 4700 pagesso far today; Request 3 pages a second

[vabtz]

I recently changed my site to reduce the page size drastically and clarified the links structure a bit.

Today gbot has pulled 4700 pages (so far) requesting 3 pages a second.

I verified its a real bot. Is this normal? My server can easily handle the load but just seems a little frightening and exciting at the same time.

[sandor]

yeah, i've noticed it crawling quite a lot lately. wonder what's up!

[itloc]

Same here ...

Never saw a crawl like that...

itloc

[walkman]

huge crawl on 3 sites. Barely any visits on the one that matters most though :). Of well.

[estaquieto]

Me six. About 1,500 pages crawled on mine so far. Unprecedented.

[victor]

Googlebot started rampaging across one of my sites at that sort of rate a few months ago.

I emailed Google and told them to behave or be banned.

They emailed back an apology, and all has been well since.

Their bots can be as buggy as their SERPs. If you don't tell them the problem, they may not notice it themslves.

[midlifecrisis]

On a brand new site (setup only last Friday) the bot pulled almost 3000 pages on Sunday. All of them already show up in the index.

[Marcia]

I also got the full treatment. Anyone notice any 304's around the same time? I posted here about it, it's got me wondering

[webmasterworld.com...]

[eyezshine]

I seen googlebot spidering my cgi pages deep today. Normaly they only spider one level deep into my cgi generated pages but today I seen it spidering up to 10 levels deep just like they were standard html pages?

Maybe they are trying to build an even bigger database? I never thought I would see them digging that deep on cgi generated pages?

[mcavill]

wonder if that's to correct the url only listings I'm seeing around the place.

[GoldFish]

I have the exact same thing. In this month so far, Googlebot has crawled about 20 times (!) the number of files than it did in january. IMHO, this probably has everything to do with their new algorythm, i.e. the Allegra update.

[Liane]

Its seems we are all experiencing the same thing. Its "that time of the month" for Googlebot to do its thing. Maybe we'll have another update around the end of the month.

It'll be really interesting to see if the 302 redirect problem has been solved. I have a gut feeling it just may be! :)

[GranPops]

Massive crawl everywhere, every last nook and cranny, looking good for another shuffle

[taps]

Seeing a very deep crawl too.

Lost a lot of positions after renaming all URLs and Allegra. My 301-redirects seem to work properly. Hope to be back soon. :-)

[BillyS]

The pattern for my website has always been a deep crawl a couple of days after an update and a couple of days before an update. My guess is that we will see more reshuffling this weekend.

[Marcia]

>>wonder if that's to correct the url only listings I'm seeing around the place

I've got a couple of pages that went URL only on a site. One was definitely something wierd on their end - it was a duplicate listing of the same URL/page on the SERP - repeated twice, so one was normal and the other was URL_only. I suspect that's a side effect of something they're doing "differently" in some way.

The other page is my fault for sure - accidentally put the same text on two pages, widgets-2.html and widgets-3.html instead of the final copy of the pages. All they did was have the widgets-2.html page go URL_only and the other is fine, as is the rest of the site. Rather, will be fine if it ever gets out of the "sandbox."

My gut feeling, which I can't shake and can't figure out, is that they're doing something different in the way they're crawling sites.

[xrtza]

Massive crawl here also. For about the past week 2-3 pages per second from 10 or more ip addresses. I want to believe all the attention is a good thing, but of the over 16,000 pages indexed by google none are rankng for anything including distinct company name. Very little traffic from G and intestingly more traffic from google.br than google.com (seems odd to be since it is .com and located in US). The not ranking may be in part due to a major rewrite of all URLs. Only a handful of pages have PR at this point. Hopefully a bigger better update is coming. As much as I have read about Allegra I don't quite understand exactly what it did other than make some happy and some sad. I can't help but to believe if Google feels my site is important enough to crawl and index that many pages it must be important enough to rank somewhere. Does anyone have any relative data to manage ranking expectations. What is reasonable? If on page one MSN page three Yahoo where would be realistic expectations for Google? page 10? 20? 1000? I would be curious to know if anyone has any data on this.

[Just Guessing]

Yes I saw huge crawls from the Mozilla agent before and after Allegra, and again now over the last couple of days.

Could also be a panic crawl to sort out a problem.

I have a site where a site:www.example.com search seems completely broken - loads of URL only listings, only returns 600 results even with &filter=0 and doesn't return the home page or any top level pages even though they are indexed and ranking well in other Google searches.

[midlifecrisis]

"If on page one MSN page three Yahoo where would be realistic expectations for Google? page 10? 20? 1000?"

I have one site that is #1 at Y and #2 at M for the same kw1 kw2 combo (500K SERPS). Also first page positions at Jeeves, AV etc. It's at a solid #75 at G. Sigh.

[trillianjedi]

they're doing something different in the way they're crawling sites.

I know exactly what you mean Marcia - I felt the same way about it. Couldn't spot any specific pattern though.

Actually, it looks like old-skool deepbot of about 2 years ago, but with some tweaks. I wonder if they dug some old code out?

Back to a monthly re-index maybe?

TJ

[midlifecrisis]

Another massive crawl yesterday: Googlebot more than 7000 pages, Teoma 10000 pages. Luckily the server didn't explode.

[taps]

Almost 40 percent of all my traffic this month results from bots. Lotta activity.

[Jakpot]

Now what are they going to do with this crawl results?
More surprises?

[midlifecrisis]

Given the fact that SE spamming, scraper sites etc. are such a widespread problem, could this be a sign of a concerted action?

Perhaps the beginning of something like the SEAA, Search Engine Association of America ;-)

[xrtza]

I just wish I could feel certain they will use all the pages the have crawled. Googlebot has settled down to a more "normal" rate for our main site now. Cached daily and deep crawled more than once this month. Last week was the big burst that the title of this thread describes. Still waiting for the update of all updates.

[MrSpeed]

Actually, it looks like old-skool deepbot of about 2 years ago, but with some tweaks. I wonder if they dug some old code out?

Back to a monthly re-index maybe?

I can't help but wonder if it's in response to Yahoo's crawling activity. It seems as though Yahoo is much more eager to crawl a new site or pages. Perhaps Yahoo saw an opportunity for sites that Google sandboxes?

[Seo1]

My two thoughts

With the index doubling in size recently, and as Mr Speed said with Yahoos crawls, and now from what I see MSN and AskJeeve page counts in the server stats Google needs to do this.

On subdomains the crawl is even crazier multiple visits per day

On a client site Googlebot ate up 13k pages.

Sweeeeeeeeeeettttt

[grail]

Yeah "me too". Googlebot had a real spurt, I like to think it was in direct response to my post on here just before it. 'Spider penalty' 'Googlebot on go slow'. :-)

Still has someway to beat Yahoo which is all day everyday. Infact I just started to put up some new stuff and yahoo was taking it before I had even finished tweaking the pages, i'm frankly amazed it found it it's almost like it spiders real time as soon as it finds a link off it goes. Rather than collecting a list of links which it intends to crawl later.

[trillianjedi]

Consider Block Rank:-

[webmasterworld.com...]

.... alongside furious deep-crawl behaviour.

TJ

[Dayo_UK]

Lets hope Google can start ranking pages like Yahoo - based on content again.

Googlebot is crawling like it did in November - just before the index size increased to 8 Billion.