Decrease of Googlebot spidering

Googlebot cut down on visits in the last 3 days?

[mayday9]

Hello,

Googlebot has been spidering my pages average around 800 per day (2 sites) (all new ones). It has cut down in the last 3 days to around 60 per day. I'm still feeding it sites the same way as I did before but somehow it has calmed down.

Is anyone else experiencing cut down of visits by googlebot? I'm not refering to sites who get rare visits anyway, I'm talking about those who get regular visits.

Darko

[fabian]

Hello Mayday
Yes,I had 2 days without receiving googlebot visit, and the last ones the robot did not crawl deep on my pages, but asking only for robots.txt . (My robots.txt obviously is not filtering google...).

[mayday9]

Nicke:

I used to get a new fresh tag everyday, not the last two or three days, thought.

yup, so something is different again.. Maybe few days continuous update, then pause for data processing, then repeat update? who knows...

Darko

[alpine]

Same here. Average day is several hundred; last two/three days 75 each day. But the last few Sundays have been big, (around 1000) so we'll see.

[getvisibleuk]

is this the normal pre-update activity?

[mayday9]

We don't know yet, since google's roling update is new. We'll have an answer to that one in few weeks I guess

Darko

[getvisibleuk]

but do we know what we're seeing is the true rolling update or a transitions stage to it?

There are too many unanswered questions to be catagorical about anything google-related!

Lee

[mayday9]

that's why we open such threads to see what other people are experiencing and together we come to most probable conclusions which are as good as it will ever get and with any luck we benefit from them.

One thing at a time, at the moment it seems that Google bot has decreased visits to sites that it used to visit more. I'm seeing it on all my sites and so are others.

There could be a lot of reasons, but I suspect google pulled back to do something big and then hit again. That "big" could be anything like PR calculations, running everything trough their formulas, etc.. or god forbid they're approaching their capacity limit. We'll see soon

Darko

[Friday]

I submitted a new project to Google over 3 weeks ago and haven't been visited AT ALL by googlebot yet!
:(
Strange.
Maybe the bot is sick.

[amoore]

Here's how much googlebot has been by my sites in the last month:

+------+-------+ 
¦ day ¦ count ¦ 
+------+-------+ 
¦ 6/25 ¦ 23 ¦ 
¦ 6/26 ¦  5 ¦ 
¦ 6/27 ¦ 91 ¦ 
¦ 6/28 ¦ 519 ¦ 
¦ 6/29 ¦ 29 ¦ 
¦ 6/30 ¦  2 ¦ 
¦ 7/1 ¦  6 ¦ 
¦ 7/2 ¦ 79 ¦ 
¦ 7/3 ¦ 461 ¦ 
¦ 7/4 ¦ 17 ¦ 
¦ 7/5 ¦ 310 ¦ 
¦ 7/6 ¦ 107 ¦ 
¦ 7/7 ¦  9 ¦ 
¦ 7/8 ¦ 10 ¦ 
¦ 7/9 ¦ 32 ¦ 
¦ 7/10 ¦ 361 ¦ 
¦ 7/11 ¦ 137 ¦ 
¦ 7/12 ¦ 171 ¦ 
¦ 7/13 ¦ 248 ¦ 
¦ 7/14 ¦ 23 ¦ 
¦ 7/15 ¦ 69 ¦ 
¦ 7/16 ¦ 310 ¦ 
¦ 7/17 ¦ 183 ¦ 
¦ 7/18 ¦ 421 ¦ 
¦ 7/19 ¦ 13 ¦ 
¦ 7/20 ¦ 573 ¦ 
¦ 7/21 ¦ 160 ¦ 
¦ 7/22 ¦ 287 ¦ 
¦ 7/23 ¦ 219 ¦ 
¦ 7/24 ¦ 202 ¦ 
¦ 7/25 ¦ 413 ¦ 
¦ 7/26 ¦ 419 ¦ 
¦ 7/27 ¦ 24 ¦ 
¦ 7/28 ¦ 398 ¦ 
¦ 7/29 ¦ 307 ¦ 
¦ 7/30 ¦ 239 ¦ 
¦ 7/31 ¦ 513 ¦ 
¦ 8/1 ¦ 17 ¦ 
¦ 8/2 ¦ 372 ¦ 
+------+-------+ 

It doesn't really appear much slower to me, though yesterday (the first) was a slow day, like the 19th and the 27th.

[getvisibleuk]

Friday I agree - what theme was it as I think google alters its crawling patterns relating to what theme it views a site as.

[MonkeeSage]

Mabye this is a dumb question, but I only know how to find things using Google, and beyond that I'm pretty clueless as to what goes on behind the scenes. I am curious why in the world a person would would want / google would need to hit a site 800 times a day....or even 100 times a day...or even 10 times a day?!?

Why couldn't Google be adding 700-something previously unindexed pages instead? Does Google just site around indexing the same sites they think are going to make money, over and over, and ignore the others, or what?

If that is true (or mostly true), then it seems incorrect to call it a "internet" search service--it semms more accurate to call it an "intranet" search service in that case.

Have I misunderstood the hows, wheres and whys of Google's crawling or something? I'm pretty sure I'm misunderstanding something, somewhere, because I'm an SEO disaster (I have no keywords, I have obscure domains, I code with Mozilla in mind and tell IE users to go row a boat (unless of course I can find a solution that works in both), I use hidden content (menus and such), I have no ads at all, and the list goes on), but yet 4 or 5 of my pages are indexed on Google and get a fair amount of traffic.

So can someone please give me the cliffnotes version of how Google indexes and such? I'd be much obliged. Thanks.

Jordan

[mayday9]

Monkeesage:

You missunderstood. In this specific thread "googlebot hit" is either a new page spidering or rechecking old pages (to update the new content that might be there). In my case I was refering to newly spidered pages.

Your sites are probably user friendly and contain good amount of content, that's why you get good google hits. You're a good SEO by accident:)

Darko

[MonkeeSage]

Darko:

Ahh, well that makes more sense. I keep forgetting you guys have HUGE sites (I think I read that one of you had 1000 pages!). I, meanwhile, can barely count to 1000, hehe, let alone manage that much content. ;)

Ps. I think most of my traffic is click-trough from a couple of SourceForge forums (my pages relate to opensource software projects hosted on SF). One of them is a sort of unofficial builds page, and that one is the one that gets the most traffic. So I'm prolly just piggybacking on SF's ratings.

I have no idea how it all works out, really. I'm just glad that you guys do so's I can ask you! ;)

Jordan

[fabian]

Googlebot comes today and crawled 8 pages (the 8 previously appearing for my site) instead ask only for robots.txt.
It was done in two different visits crawling 4 pages on each. Despite no directory or still unlisted pages were crawled, it appears to be a different behavior from the last few days. Any new in your sites?

[mayday9]

9 of thousands of my pages got crawled today.. what a shame. It used to be between 1000 - 800 daily, now it's less than 10 since 5 days ago.

Darko

[Hardwood Guy]

They must be gearing up for the long awaited update? I'm crossing my fingers.

[Sharper]

Count of Googlebot page requests (includes Mediabot, although that's probably <10%) for one site in July. For reference, a relatively new site that has 110K+ pages, about 22K showing in Google results right now.

Log file end time, then count:
2003-07-01:00:18
2
2003-07-02:00:21
2
2003-07-03:00:19
5
2003-07-04:00:20
3
2003-07-05:00:18
0
2003-07-06:00:18
3
2003-07-07:00:19
30
2003-07-08:00:22
35
2003-07-09:00:20
92
2003-07-10:00:22
475
2003-07-11:07:22
64
2003-07-12:00:19
117
2003-07-13:00:18
244
2003-07-14:00:20
633
2003-07-15:00:20
3017
2003-07-16:00:22
3255
2003-07-17:00:20
1115
2003-07-18:00:20
1579
2003-07-19:00:20
1428
2003-07-20:00:18
10071
2003-07-21:00:19
9134
2003-07-22:00:21
11538
2003-07-23:00:24
6077
2003-07-24:00:23
4302
2003-07-25:00:22
1234
2003-07-26:00:18
8152
2003-07-27:00:21
2670
2003-07-28:00:20
6756
2003-07-29:00:23
2839
2003-07-30:00:23
1174
2003-07-31:00:23
1419
2003-08-01:00:26
212
2003-08-02:00:24
248
2003-08-09:42:15
0 (so far today)

It sure looks a lot more like an ongoing spider this month. Last month the same site had almost no hits from Googlebot, then 14K for a couple of days in the middle of June, then nothing again.

I noticed that refreshed pages for the site started showing up again in Google's results a few thousand at a time around July 28th.

Anyway, I'm picking sometime in the next couple of days to do server maintenance, since it appears that Google has slowed down a bit.

[mayday9]

Yup, your log shows the current situation well. I'm guessing that the rolling update which was put into action around 9th of July (according to your logs) has come to a serious slow-down because either they are now thoroughly processing what they spidered, or have runed into some trouble.. hope it's processing.

Darko

[IITian]

Last week it crawled most of my pages. On Friday it came mainly looking for 404s. Yesterday, it fetched a very few least popular pages and the index page. Today - nothing. None of the new content has showed up on serps. Some pages are stale by more than 2 weeks. Let me see what's next.

[mak445]

Googlebot is still comeing to my site new fresh tag today

[Net_Wizard]

Googlebot is in my site almost daily plus 'Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)' is also snooping around my site even when I'm not an adsense member.

And, for the first time for my site 'Mediapartners-Google/2.1' followed a jump.cgi URL never happened before with any googlebot.

So, what would happen to that jump URL? Would it now become part of the backlinks for the jump site? Hmmm, disallow jump.cgi then?

[mayday9]

Mediapartners-google bot as I understand it has a different agenda.. It checks to see if everything complies with adsense rules. So nothing happens with jump.cgi, the bot is probably making sure there are no rules broken on it

Darko

[mayday9]

some people are reporting new (fresh) backlinks showing up for their sites. Could that be the reason for googlebot slow down? Looks to me like PR calculations have started.

Still I hoped for more reports on "decreasing googlebot spiderings" and more reports of new backlinks showing, to make sure all this is not just a coincidence..

Darko

[Net_Wizard]

It checks to see if everything complies with adsense rules.

As I have posted above...I'm not an adsense affiliate. Just curious why Mediapartners-google is going around my site.

Cheers

[xlcus]

Googlebot is hitting my site harder today than it has in the last couple of weeks. It almost looks like an old style deep crawl.

[Friday]

Friday I agree - what theme was it as I think google alters its crawling patterns relating to what theme it views a site as.

Hi getvisibleuk,

To answer your question, my theme is "golf."
( a _TOUGH_ one, to be sure).