homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 68 message thread spans 3 pages: 68 ( [1] 2 3 > >     
Googlebot isn't crawling
sidney rascal

 9:26 pm on Apr 5, 2006 (gmt 0)

< note: related thread here: [webmasterworld.com...] >

The google bot is coming to the site looking at a couple of pages then dissapearing. I've tried site maps but they don't seam to be helping.

Any advice out there as the client is getting a bit pissed



[edited by: lawman at 9:38 pm (utc) on April 5, 2006]

[edited by: tedster at 12:36 am (utc) on April 8, 2006]
[edit reason] add link to related thread [/edit]



 6:22 am on Apr 7, 2006 (gmt 0)

Get more inbound links and read this forum regularly :)


 8:56 am on Apr 7, 2006 (gmt 0)

I don't suppose the bot in question is

Since Google switched to this bot for our site (a few days ago), we get crawled at a rate of about two pages an hour (way down from the norm).

Also, this bot seems to have a bug with 301 redirects. Every time it hits a 301 it follows it and then stalls (i.e. disapears for long periods).


 10:00 am on Apr 7, 2006 (gmt 0)

> Get more inbound links and read this forum regularly :)

this in not a very costructive comment....

The problem is a different problem.

For example I noted in few sites I monitor, that googlebot is visiting 1/4 than usual.
Those sites are good sites, no spamming sites having an excellent link popularity etc.
Those sites have Page Rank 6 and are not suffering in ranking...

But it's strange that googlebot is not crawling as usual....


 10:33 am on Apr 7, 2006 (gmt 0)

The problem is a different problem

How could you or I know? In initial post there was not mentioned that Google bot has changed its behavior. Just that it does not crawls deeply and site map does not help.

I read several times in this forum that the better page rank the more deeper Google bot crawls. PR depends directly on inbound links. So why my advice is irrelevant?

It may be not your case, but I answered not to you. I answered to the post that was unanswered two days, sorry.



 10:40 am on Apr 7, 2006 (gmt 0)

You are right about some arguments, but your post was not constructive...

There is another similar post explaining the problem:


I discovered thsi problem in another forum..


 11:00 am on Apr 7, 2006 (gmt 0)

Anyone want to clarify which bots are not crawling:

The old Googlebot/2.1 (+http://www.googlebot.com/bot.html)?
or the new "Googlebot/2.1"?


 11:07 am on Apr 7, 2006 (gmt 0)

your post was not constructive

I used to believe that constructive means suggest the solution and not to explain the problem. In the thread that you mentioned no solution was suggested.

Also, the problem is (or is not?) that I have no such a problem. My sitemap was fetched 16 hours ago, I had no supplemental and all seems OK.

Again, usually Google bot does not crawl the site when the site authority (used to be PR but now seems also other "signals of quality") is low.

Big Daddy seems considers all not original content as low quality (I don't mean the duplicate). Big Daddy basically gives good SERP, well, to Big Daddies (original authors).

So the constructive solution, again, are good inbound links and good original content. I understand that it is difficult but this seems the only way to become the Big Daddy.



 11:26 am on Apr 7, 2006 (gmt 0)

Dear Vadim I appreciate your last post.

I think that nobody here has an answer.
The question is: are the site decreasing a lot the Googlebot visit, going to be penalized?

I noted a site got supplemental few weeks ago ( and recovered ) that is receving now less googlebot visits.


 11:27 am on Apr 7, 2006 (gmt 0)

Anyone want to clarify which bots are not crawling

Out of curiosity I downloaded the most recent log that my host gave me (06/Apr) and saw that both Google bots crawled my site.

So I understand that it may be problem, but it seems not the general problem, i.e. not for all.



 11:32 am on Apr 7, 2006 (gmt 0)

I think that nobody here has an answer

I agree. Only Google knows. Still I beleave that it might be a sign that Google quality rank of the site became low. But I may be wrong, of course.



 1:46 pm on Apr 7, 2006 (gmt 0)

It may also be that the content on those pages rarely changes. If you change the content once a month, then google only needs to visit once a month.

Does not necessarily mean that its a bad page. Its just a way for googlebot to be free to visit pages where content is updated daily.


 4:48 pm on Apr 7, 2006 (gmt 0)

There is definitely something going on. I have something around 200 client sites hosted on our servers, some with thousands of pages that normally get crawled pretty fully every day. The past week or so, Googlebot visits have dropped to maybe 30 or 40 pages per site per day. That's WAY down. When I saw it happen on a couple of the larger site, I grepped the Googlebot entries out of all the logs and ran chart, and it was happening all over. 200 sites didn't just suddenly lose quality; Googlebot isn't even coming.


 5:40 pm on Apr 7, 2006 (gmt 0)

Yes, we have this crawl problem with 3 different large sites. Over the past couple of weeks the bots have indexed only a few pages. Saying that, Googlebot started indexing more pages on one of the sites yesterday so maybe things are starting to improve?

Who knows. Everything where Google is concerned is one long waiting game these days...


 8:10 pm on Apr 7, 2006 (gmt 0)

I'm seeing exactly the same thing, glad I'm not the only one! Only 70 pages fetched in the last 7 days compared with last month 5 or 6 times that amount.

It also seems to be taking along time to get a new domain indexed, Googlebot has found some links pointing to the new domain (cached) then heads off somewhere else, absolutely no sign in my logs. Maybe its needs a kick up the backside!


 8:31 pm on Apr 7, 2006 (gmt 0)

if someone kicked me in the backside id not be happy!


 8:37 pm on Apr 7, 2006 (gmt 0)

Googlebot has definitely slowed to a crawl for me. Also, cache dates go back to Mar 30th instead of say April 6. site: command's been returning almost identical set of pages across most DCs for the last week or two; nothing new seems to be getting indexed. All this stagnant inactivity actually made me file a reinclusion request :=)


 9:36 pm on Apr 7, 2006 (gmt 0)

Who would ever suspect that a cool nerve was a requirement in order to operate a fish-pond within web-waters?

The last time that that the old G-bot ("Googlebot/2.1 (+http://www.google.com/bot.html)") trawled my pond was 28 March. Previously it took about a thousand fish each month. It seems to have been retired, although time will show.

The 'new' M-bot ("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)") spent 40 days and 40 nights [webmasterworld.com] trying to get every fish from every pond. That also came to a full stop on 28 March, but only temporarily. It took it's full quota from my pond on 1 April, and is trawling full-blast as I write these words. It previously took ~60 fish each month, although it attempted >200,000 in March (restricted by fish-quotas [webmasterworld.com] to 25,000) and has attempted 35,000 in April (restricted to 4,000).

Google has just switched to a more modern fish-hook. Calm down, folks.


 6:37 am on Apr 8, 2006 (gmt 0)

Hmm, modern fish hook - but doesn't seem to notice the new frolicking fish in the pond since 29 March, even though the pond is PR5.

(the hook doesnt bother with PR 1-3 ponds as far as I can see..)

On the other hand, in another old PR5 pond, the hook catches fish every single day.

Weird hook.


 2:51 am on Apr 10, 2006 (gmt 0)

Could Google not be crawling sites that have a site map? No need?


 5:14 am on Apr 10, 2006 (gmt 0)

Could Google not be crawling sites that have a site map? No need?

I've the same problem. Ok I will delete my sitemap and monitor my logs closely and observe g bots patterns.


 9:00 am on Apr 10, 2006 (gmt 0)

I dont have a sitemap. No crawl.


 9:16 am on Apr 10, 2006 (gmt 0)

I'd say it's just another Google short term thing.

The bots will be back in force before long.


 9:26 am on Apr 10, 2006 (gmt 0)

From the level of complaint on the boards, it looks to me like this problem is only effecting a very small number of sites (mine included).

Begs the question, what do we all have in common?

In my case, the problem happened soon after switching hosts (from a US IP to a UK IP). After about a week, our regular Googlebot switched (presumably to one located in the UK), and ever since our crawl rate has plummeted: from an average of 500-1200 pages a day, down to around 40 a day.

So, in our case, the Googlebot is still coming regularly, it's just that it only takes one or two pages at a time instead of the usual 20-30. Every now and then it perks up for a couple of hours and takes 10 pages at a time (but only for short bursts).

Does this match anyone elses experience, or its it just a case of the Googlebot disapearing without trace?


 9:34 am on Apr 10, 2006 (gmt 0)

The only thing I did was take the Google analytics code out.


 11:40 am on Apr 10, 2006 (gmt 0)

Hello all webmasterworld users!

I have a question about googlebot, maybe someone could make this clear for me: GB is nowhere to find on my site since April 5th. I have a script that tracks robots, and it reports that April 5th was the last time GB visited my site. msnbot has no problem visiting, crawling and indexing my site. The site is not yet indexed in Google (it's normal, it's a very new site), but the thing that concerns me is that GB is not even trying. So, is it normal? Should I be worried? Is there something I can or should do? How long is the pause between two GB visits? I have two other sites that are crawled by GB many times a day, so I don't think the bot is on vacation.

Thank you.


 11:46 am on Apr 10, 2006 (gmt 0)

I think Googlebot may be on hiatus. Just kidding.

If you look at the PR update thread and a few others, I think you'll find that some of us believe there is a problem with some of the infrastructure associated with the last big crawl. My guess is that the techs are trying to make adjustments to fix whatever it is that is somehow "off".

The last big crawl on my site was on March 25th with 204 total hits from Googlebot.

195 of those were from
Mozilla/5.0 (compatible; Googlebot/2.1 and the rest were from a variety of other bots.

Since March 25th, hits drastically declined which is the norm, but the figures for my site are a bit lower than the norm ... particularly in the last week.

March 26 - 31
27 - 19
28 - 13
20 - 8
30 - 18
31 - 2

April 1 - 29
2 - 28
3 - 3
4 - 29
5 - 32
6 - 0
7 - 4
8 - 8
9 - 1

Give them time to fix whatever it is which is messed up.


 11:51 am on Apr 10, 2006 (gmt 0)

Yes, but two sites are getting a whole bunch of hits from Googlebot, and the third one is getting zero for 5 days. I guess I'll have to wait for a few more days before starting to worry. Thanks for the quick reply.

And thanks to everyone at WW for... well for being here and providing excellent content and tips


 11:52 am on Apr 10, 2006 (gmt 0)

I would imagine that resources are probably doing other things at the moment that is all.

Gbot has pretty much dried up on my site too, although 100 odd visits this morning.


 1:38 pm on Apr 10, 2006 (gmt 0)

I have been finding that Google is crawling my site more frequently and more pages than previously. I was questioning in another forum, if others are seeing an increase in crawling by Google - I did not get any responses.

April Hits
1 61
2 38
3 81
4 61
5 351
6 203
7 322
8 154
9 372

I do have a smaller site with about 500 pages.

This 68 message thread spans 3 pages: 68 ( [1] 2 3 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved