Googlebot not crawling

Seeks index page, then leaves

[feeder]

Googlebot visits often. It requests the index page, but doesn't crawl any deeper. This happens two or three times a day.

The MediaBot crawls deeper into the site without issue. The site runs AdSense.

Could there be anything in the server config that is causing this? It isn't robots.txt. The index page is lo-fi and xenu crawls it fine, as does the searchengineworld sim spider.

Any ideas?

[jbgilbert]

Everybody appears to be seeing similar issues.

Googlebot is crawling, but not as aggressively or successfully as he has in the past.

Google's indexing seems to be working in line with the crawl activity.

BUT,

His update of backlinks and PR appear to be having an actual problem.

[mayor]

Maybe Googlebot is ill. Is there a Bot Doc in the house?

[internetheaven]

Google are putting the finishing touches to a new 'program'. I expect we will see it within the next week or so ... they're trying to get it out as quickly as possible to interupt the Yahoo media attention.

[internetheaven]

Wow, they actually let the post get listed this time ... fifth time lucky ....

[mayor]

Allright, Internetheaven, I'll bite ... what's the new program, and what's it got to do with a lethargic bot?

[internetheaven]

I tried to post that and it never got through, do you know who has the authority to remove messages and stop new threads before they start?

It seems as though every time I post anything it comes up 'On Hold for administrative review'.

[thumpcyc]

Pure Speculation:

Googlebot comes calling, but eats nothing, because it is till trying to digest that extra billion plus pages. What happens to you when you stuff more in your mouth than you can swallow.

Googlebot gobbled, until it had to take complete datacenters away from serving up SERPS, in an attempt to digest the data.

Florida, Brandy, The Non-Update, they were not updates, just indigestion, so those that think the directory ridden serps you are analyzing are cr@p, you are wrong, it came from the other end.
;-)

Just speculating like everyone else!
Thumpcyc

[jchance]

I tried to post that and it never got through, do you know who has the authority to remove messages and stop new threads before they start?

Ummm... the moderators.

This forum is pre-moderated meaning new posts wont show until they approve them.

[Stefan]

And Internetheaven, it's usually better to address that question to the forum mod via sticky... sort of an etiquette thing.

Forum3 is premoderated for a good reason; it used to be a total zoo during updates, with countless overlapping new threads being started. After the change happened, forum3 became much more informative.

ON TOPIC: Sorry to hear that many sites aren't getting crawled much, but ours is getting crawled and freshtagged better than ever. For example, I put two new pages up on Mar 8, and they were in the serps the next day with Mar 9 freshtags. In fact, all of the 30 or so pages that went online in the last few weeks were crawled within a couple of days and in the serps, to stay, soon after. Our index is a PR6... I don't know if that is a factor.

[steveb]

Anyone an exception to: if an index page is PR6 or higher, the first and second level pages on the domain that are PR5 or higher are crawled and freshed every day, while the PR4 and lower pages two clicks off the index page are freshed rarely.

[Stefan]

That is pretty much the case for us, Steveb, (meaning no exception). We have quite a few PR5 pages, all getting freshtagged regularly, and The PR4's only get looked at every few weeks or less. If I have a page that I want to be crawled often, I just link to it from one of the main pages rather than a couple of clicks down.

<edit>Clarity</edit>

[sandor]

same prob here with new sites

google's grabbing the index only .. everyday, sometimes several times per day

then suddenly it starts a 'mini-deep crawl' of a totally new site that had one link pointed at it .. meanwhile the others which have been around longer are still no being deepcrawled

[Snookered]

For me Googlebot only visited once last month and hasn't shown up this month yet. No page updates at all and I've lost much ground to rivals. I know not why. I'm thinking it will be more productive to focus on other SEs if this continues.

[Mohamed_E]

Today I noticed that a whole bunch of pages that I had added to two sites during the last month suddenly appeared in the index. All pages are a couple of clicks from the index page; one site is PR 5 at top, the other PR 4. In spite of the low PR new pages used to appear in the index much faster.

Is it just my sites, or did Google awaken from its deep sleep?

[IITian]

I have a small subdirectory of about 10 pages. Its index page is PR5 while all the rest (linked from the index page) are PR4. All the 10 pages are crawled daily and changes show up almost instantly.

BTW, my experience with Google spotting new pages to crawl and indexing pages of one of my sites - a few hundred-pages of content and mostly added within last 2 months - is very positive. All in done within a couple of days and they all show up in the serps almost immediately. (It's a PR5 site with most likely all other pages PR4 or lower.)

It is not refreshing my PR3 or below pages as much as I would like it to do but am not complaining expecially after realizing that Y! hasn't even crawled most of the pages even once. MSN was in the same situation as Y! but for some mysterious reasons or coincidence, just a couple of hours after I had submitted my site to Y! free addurl service announced a few days earlier, it showed up on my site grabbing all the pages.

All my sites are informational ones and that could be one of the reasons.

[mayor]

>> did Google awaken from its deep sleep

ZZZZZZZZZZZZZZZZZzzzzzzzzzz not here zzzzzzzzzzzzzzz

[MrSpeed]

I think the main issue is with new sites. My existing sites seem to be crawled as usual.

[minnapple]

Finally, a new site that was published Febuary 8th has been partially crawled by googlebot.

The site has one inbound link.

I have another site that was published in December that has not been crawled by googlebot. This has multiple inbound links.

Weird stuff.

M i n n a p p l e

[BallochBD]

Is everyone here getting some kind of visit from Googlebot? By this I mean the home page and/or robots.txt or is the bot not visiting some sites at all?

[BigJay]

Yes, Getting good GoogleBot hits, on all sites.

Added a new site last week, was partially spidered, and has shown up in the index already.

On all my sites I regularly ( weekly ) update content. This has helped "invite" the Googleot on a very regular basis.

[stcrim]

One of the most interesting things about this thread is, there has been no comment from Google on the matter. That, in itself is very telling...

Also, in the thread that only one "new site" has been spidered and indexed past the home page...

-s-

[stcrim]

How many of you have your sites all linked together in an effort to get google to find you?

-s-

[stcrim]

Another interesting point - this entire thread is already indexed in google - hmmmmmmmmmm

Search: Googlebot a morale boost

-s-

[barbeta]

Since last wednesday Googlebot is crawling and crawling my new sites... in fact... he(she)īs still crawling pages up to now :)

[cyberprofit]

I would like to add that Googlebot has been crawling my sites very slowly. I have a few new sites that seem to be taking forever to get fully crawled. It's apparant that Google has changed the deep crawl rate of new sites, but what is not so clear is why there is no representation by Google or GG? Where is Google Guy? What does he remain so silent even though he "IS" reading these threads? Is it because... even he himself don't know what the heck is going on anymore?
Your thought?

[MrSpeed]

How many of you have your sites all linked together in an effort to get google to find you

My sites are not all crazy linked together in any un-natural manner. I considered that perhaps I was getting penalized for having multiple sites on the same IP. The older sites are still getting spidered/indexed fine.

Does anyone think there is a penalty for having a site and links to it magically appear the same day?

[RussellC]

I have several 'old' sites that still get daily crawls. Just started a new site and it got a visit from googlebot 2 days after i launched it and got some links from other sites, but it has only picked up the robots.txt and the index page. It came by for 4 days and did the same thing, but i havent seen it for 3 days.

Also, the index page went into the index and had a cache etc 2 days ago, but now it's not in the index. Very strange. Seems like its gonna take some time to get into Google these days.

[jchance]

I too had been having the not being deep crawled issue, but that all inded last Friday.

Googlebot deepcrawled every page in my site (thousands of pages) and all of the new pages I had added now appear in the index.

[trimmer80]

For the people who are having this problem and the people who are not, can we post details in the hope a common link will be found.

My site that is affected.

PR - 4
Average before feb - 3000 googlebot hits/mth
Average Feb / Mar - 500 googlebot hits/mth
Size of Site - 1300 pages

[stevenb 1959]

To me Google must be crawling because I seen a new cache for my entry page, every day for the last week or so in the serps.
Anyone seen this or have any ideas and/or thoughts on why it is happening?
thanks
Steve