homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

No major Googlebot trawl
ages since the last big trawl from the big man

 6:20 pm on Jan 20, 2005 (gmt 0)

Anyone else experiencing an ever increasing lag between large crawls (we are talking about anything over a few hundred pages being spidered).

With 4 large sites, all have had little or no activity since turn of the year in terms of spidering - no wonder some of the SERPS are looking a little stale.

Anyone else?



 11:46 pm on Jan 20, 2005 (gmt 0)

Yes. Up until a few days ago, Google had only visited 150 times this month on my site, which has about 1500 pages.

Then, in one day, it visited 900 pages.

Since then, it's only done a page or two a day.


 11:53 pm on Jan 20, 2005 (gmt 0)

Very quiet

AND just noticed another bug in Google

When I was searching for a term that has over 500,000 pages from thousands of sites it returned a page with "Showing 1 to 29", #*$!.

29 results from 500,000?

I know there are issues but that is really poor.


 11:55 pm on Jan 20, 2005 (gmt 0)

can't replicate the problem on any other seach yet but does happen every time I search the broken phrase.

Anyone else seen this?/reported it?


 4:31 pm on Jan 21, 2005 (gmt 0)

I've noticed the same thing for the past 6 month. Googlebot shows up every night, but only gets 100 to 200 pages. In the past, at least once a month Googlebot has picked up most of my 17,000 pages over a 2 to 3 day span. I don't remember the last time that happened.

The main pages still rank well, but only about half the site is indexed now.


 8:21 pm on Jan 21, 2005 (gmt 0)

Yes, its been very slow this month. It seems to have followed the Googlers up on the hills.

Most of my sites see less activity as compared to other months, even though most of the sites have great PR and strong backlinks.

Could something major be in store? Only time will tell


 8:37 pm on Jan 21, 2005 (gmt 0)

been slow for me for 2-3 months. Today though it seems like one of those bi-weekly deep crawls


 9:03 pm on Jan 21, 2005 (gmt 0)

My pages also getting stale. Not a big site - 350 to 400 pages - but well placed in the SERPS and plenty of traffic.

Index page cache still showing 6 Jan!

Other parts of the site taken in chunks - say a section of the catalogue at a time. Very unusual. Google normally crawls the index every two or three days.



 9:18 pm on Jan 21, 2005 (gmt 0)

I thought gbot was sluggish too - lots of my sites getting a couple of pages taken per day - useless!

Then I checked one of my "out of a box" sites and gbot is swallowing the pages down at a rate of about 10,000 pages per day.

Go figure...


 12:18 am on Jan 22, 2005 (gmt 0)

I've got pretty major bot traffic right now, and average 6K pages/day for the longest time (used to be 20K pages though so I can see a recent downturn).


 12:43 am on Jan 22, 2005 (gmt 0)

Maybe this is my problem. I have a site that's been dropping for no apparent reason. G hasn't been getting very far with it, but I thought it was because I hadn't added much lately.


 11:03 am on Jan 23, 2005 (gmt 0)

Over half my site shows url only. Why was cache and description dropped? When will it be corrected?

What's going on at the Plex?


 11:46 am on Jan 23, 2005 (gmt 0)

The following variation of Googlebot is hitting one of my site heavily and uses only one single IP address. This has been discussed here in the past as well, but I didn't exactly find whether its spoofed or genuine.

The log file entry: - - [22/Jan/2005:13:07:37 -0500] "GET / HTTP/1.1" 200 3995 "-" "VM4050/132.037 UP.Browser/ (GUI) MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The IP belong to Google.

Past thread on this topic:

Gbot Running Hard [webmasterworld.com]

Anyone noticing this?


 12:29 pm on Jan 23, 2005 (gmt 0)

What's the bet Google will wait a couple of weeks for webmasters to implement their rel=nofollow attributes then start running hard?


 2:08 pm on Jan 23, 2005 (gmt 0)

My big crawls also disappeared since the 1st this year.
Google does come to my site every day for some pages, but not the whole domain.

From de 1400 pages content online, I lost the half of it. Still indexed, but no cache, description, title etc.

I can't wait and sit down to wait till it will be fixed so I removed the "lost" links with the G's URL remover.
All went fine, but since my new removal request for another directory (requested at the 19th, this makes me think that G is really broken), there's still nothing to happen. Normally it will be fixed within 24 hours.

I made some new pages with quality content, with a link from the mainpage. Google did picked it up very quick including titles, descriptions etc. etc.

My personal current conclusion, was that I had a to high KW-density. I'll hang around at 10% density now.

FYI: My site is travel related.



 9:21 pm on Jan 23, 2005 (gmt 0)

I connected this with the recent PR downgrade. If they double their index it makes sense that--all thing being equal--they need to downgrade everyone's PR, and crawl less. Right?


 3:14 am on Jan 24, 2005 (gmt 0)

I check my access_log files daily. If anything, and today especially,
G is spidering as much as usual if not more so. -Larry

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved