Forum Moderators: Robert Charlton & goodroi
The google bot is coming to the site looking at a couple of pages then dissapearing. I've tried site maps but they don't seam to be helping.
Any advice out there as the client is getting a bit pissed
Cheers
Sid
[edited by: lawman at 9:38 pm (utc) on April 5, 2006]
[edited by: tedster at 12:36 am (utc) on April 8, 2006]
[edit reason] add link to related thread [/edit]
Since Google switched to this bot for our site (a few days ago), we get crawled at a rate of about two pages an hour (way down from the norm).
Also, this bot seems to have a bug with 301 redirects. Every time it hits a 301 it follows it and then stalls (i.e. disapears for long periods).
this in not a very costructive comment....
The problem is a different problem.
For example I noted in few sites I monitor, that googlebot is visiting 1/4 than usual.
Those sites are good sites, no spamming sites having an excellent link popularity etc.
Those sites have Page Rank 6 and are not suffering in ranking...
But it's strange that googlebot is not crawling as usual....
The problem is a different problem
How could you or I know? In initial post there was not mentioned that Google bot has changed its behavior. Just that it does not crawls deeply and site map does not help.
I read several times in this forum that the better page rank the more deeper Google bot crawls. PR depends directly on inbound links. So why my advice is irrelevant?
It may be not your case, but I answered not to you. I answered to the post that was unanswered two days, sorry.
Vadim.
There is another similar post explaining the problem:
[webmasterworld.com...]
I discovered thsi problem in another forum..
Mozzybot?
The old Googlebot/2.1 (+http://www.googlebot.com/bot.html)?
or the new "Googlebot/2.1"?
your post was not constructive
Also, the problem is (or is not?) that I have no such a problem. My sitemap was fetched 16 hours ago, I had no supplemental and all seems OK.
Again, usually Google bot does not crawl the site when the site authority (used to be PR but now seems also other "signals of quality") is low.
Big Daddy seems considers all not original content as low quality (I don't mean the duplicate). Big Daddy basically gives good SERP, well, to Big Daddies (original authors).
So the constructive solution, again, are good inbound links and good original content. I understand that it is difficult but this seems the only way to become the Big Daddy.
Vadim.
I think that nobody here has an answer.
The question is: are the site decreasing a lot the Googlebot visit, going to be penalized?
I noted a site got supplemental few weeks ago ( and recovered ) that is receving now less googlebot visits.
Anyone want to clarify which bots are not crawling
So I understand that it may be problem, but it seems not the general problem, i.e. not for all.
Vadim.
I think that nobody here has an answer
I agree. Only Google knows. Still I beleave that it might be a sign that Google quality rank of the site became low. But I may be wrong, of course.
Vadim.
Does not necessarily mean that its a bad page. Its just a way for googlebot to be free to visit pages where content is updated daily.
Who knows. Everything where Google is concerned is one long waiting game these days...
It also seems to be taking along time to get a new domain indexed, Googlebot has found some links pointing to the new domain (cached) then heads off somewhere else, absolutely no sign in my logs. Maybe its needs a kick up the backside!
The last time that that the old G-bot ("Googlebot/2.1 (+http://www.google.com/bot.html)") trawled my pond was 28 March. Previously it took about a thousand fish each month. It seems to have been retired, although time will show.
The 'new' M-bot ("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)") spent 40 days and 40 nights [webmasterworld.com] trying to get every fish from every pond. That also came to a full stop on 28 March, but only temporarily. It took it's full quota from my pond on 1 April, and is trawling full-blast as I write these words. It previously took ~60 fish each month, although it attempted >200,000 in March (restricted by fish-quotas [webmasterworld.com] to 25,000) and has attempted 35,000 in April (restricted to 4,000).
Google has just switched to a more modern fish-hook. Calm down, folks.
(the hook doesnt bother with PR 1-3 ponds as far as I can see..)
On the other hand, in another old PR5 pond, the hook catches fish every single day.
Weird hook.
I've the same problem. Ok I will delete my sitemap and monitor my logs closely and observe g bots patterns.
The bots will be back in force before long.
Begs the question, what do we all have in common?
In my case, the problem happened soon after switching hosts (from a US IP to a UK IP). After about a week, our regular Googlebot switched (presumably to one located in the UK), and ever since our crawl rate has plummeted: from an average of 500-1200 pages a day, down to around 40 a day.
So, in our case, the Googlebot is still coming regularly, it's just that it only takes one or two pages at a time instead of the usual 20-30. Every now and then it perks up for a couple of hours and takes 10 pages at a time (but only for short bursts).
Does this match anyone elses experience, or its it just a case of the Googlebot disapearing without trace?
I have a question about googlebot, maybe someone could make this clear for me: GB is nowhere to find on my site since April 5th. I have a script that tracks robots, and it reports that April 5th was the last time GB visited my site. msnbot has no problem visiting, crawling and indexing my site. The site is not yet indexed in Google (it's normal, it's a very new site), but the thing that concerns me is that GB is not even trying. So, is it normal? Should I be worried? Is there something I can or should do? How long is the pause between two GB visits? I have two other sites that are crawled by GB many times a day, so I don't think the bot is on vacation.
Thank you.
If you look at the PR update thread and a few others, I think you'll find that some of us believe there is a problem with some of the infrastructure associated with the last big crawl. My guess is that the techs are trying to make adjustments to fix whatever it is that is somehow "off".
The last big crawl on my site was on March 25th with 204 total hits from Googlebot.
195 of those were from 66.249.65.104
Mozilla/5.0 (compatible; Googlebot/2.1 and the rest were from a variety of other bots.
Since March 25th, hits drastically declined which is the norm, but the figures for my site are a bit lower than the norm ... particularly in the last week.
March 26 - 31
27 - 19
28 - 13
20 - 8
30 - 18
31 - 2
April 1 - 29
2 - 28
3 - 3
4 - 29
5 - 32
6 - 0
7 - 4
8 - 8
9 - 1
Give them time to fix whatever it is which is messed up.
And thanks to everyone at WW for... well for being here and providing excellent content and tips
Gbot has pretty much dried up on my site too, although 100 odd visits this morning.
April Hits
1 61
2 38
3 81
4 61
5 351
6 203
7 322
8 154
9 372
I do have a smaller site with about 500 pages.
