No major Googlebot trawl

ages since the last big trawl from the big man

[TinkyWinky]

Anyone else experiencing an ever increasing lag between large crawls (we are talking about anything over a few hundred pages being spidered).

With 4 large sites, all have had little or no activity since turn of the year in terms of spidering - no wonder some of the SERPS are looking a little stale.

Anyone else?

[dickbaker]

Yes. Up until a few days ago, Google had only visited 150 times this month on my site, which has about 1500 pages.

Then, in one day, it visited 900 pages.

Since then, it's only done a page or two a day.

[inbound]

Very quiet

AND just noticed another bug in Google

When I was searching for a term that has over 500,000 pages from thousands of sites it returned a page with "Showing 1 to 29", #*$!.

29 results from 500,000?

I know there are issues but that is really poor.

[inbound]

can't replicate the problem on any other seach yet but does happen every time I search the broken phrase.

Anyone else seen this?/reported it?

[metrostang]

I've noticed the same thing for the past 6 month. Googlebot shows up every night, but only gets 100 to 200 pages. In the past, at least once a month Googlebot has picked up most of my 17,000 pages over a 2 to 3 day span. I don't remember the last time that happened.

The main pages still rank well, but only about half the site is indexed now.

[Imaster]

Yes, its been very slow this month. It seems to have followed the Googlers up on the hills.

Most of my sites see less activity as compared to other months, even though most of the sites have great PR and strong backlinks.

Could something major be in store? Only time will tell

[walkman]

been slow for me for 2-3 months. Today though it seems like one of those bi-weekly deep crawls

[suggy]

My pages also getting stale. Not a big site - 350 to 400 pages - but well placed in the SERPS and plenty of traffic.

Index page cache still showing 6 Jan!

Other parts of the site taken in chunks - say a section of the catalogue at a time. Very unusual. Google normally crawls the index every two or three days.

?

[PhraSEOlogy]

I thought gbot was sluggish too - lots of my sites getting a couple of pages taken per day - useless!

Then I checked one of my "out of a box" sites and gbot is swallowing the pages down at a rate of about 10,000 pages per day.

Go figure...

[Critter]

I've got pretty major bot traffic right now, and average 6K pages/day for the longest time (used to be 20K pages though so I can see a recent downturn).

[jo1ene]

Maybe this is my problem. I have a site that's been dropping for no apparent reason. G hasn't been getting very far with it, but I thought it was because I hadn't added much lately.

[Jakpot]

Over half my site shows url only. Why was cache and description dropped? When will it be corrected?

What's going on at the Plex?

[Imaster]

The following variation of Googlebot is hitting one of my site heavily and uses only one single IP address. This has been discussed here in the past as well, but I didn't exactly find whether its spoofed or genuine.

The log file entry:

66.249.65.236 - - [22/Jan/2005:13:07:37 -0500] "GET / HTTP/1.1" 200 3995 "-" "VM4050/132.037 UP.Browser/6.2.2.4.e.1.100 (GUI) MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The IP belong to Google.

Past thread on this topic:

Gbot Running Hard [webmasterworld.com]

Anyone noticing this?

[aeclark]

What's the bet Google will wait a couple of weeks for webmasters to implement their rel=nofollow attributes then start running hard?

[Localizer]

My big crawls also disappeared since the 1st this year.
Google does come to my site every day for some pages, but not the whole domain.

From de 1400 pages content online, I lost the half of it. Still indexed, but no cache, description, title etc.

I can't wait and sit down to wait till it will be fixed so I removed the "lost" links with the G's URL remover.
All went fine, but since my new removal request for another directory (requested at the 19th, this makes me think that G is really broken), there's still nothing to happen. Normally it will be fixed within 24 hours.

I made some new pages with quality content, with a link from the mainpage. Google did picked it up very quick including titles, descriptions etc. etc.

My personal current conclusion, was that I had a to high KW-density. I'll hang around at 10% density now.

FYI: My site is travel related.

Cheers

[suidas]

I connected this with the recent PR downgrade. If they double their index it makes sense that--all thing being equal--they need to downgrade everyone's PR, and crawl less. Right?

[larryhatch]

I check my access_log files daily. If anything, and today especially,
G is spidering as much as usual if not more so. -Larry