homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Don't crawl beyond index page syndrom

 2:55 pm on Mar 3, 2004 (gmt 0)

Anyone else seeing it?



 3:45 pm on Mar 3, 2004 (gmt 0)

yes, definitely. I thought it was because I had a couple of flaws in my linking logic - but maybe there's another reason.

Is your site new (or almost new)?


 3:53 pm on Mar 3, 2004 (gmt 0)


For about 3-4 weeks even, typical for a backdate/update though.


 4:16 pm on Mar 3, 2004 (gmt 0)

Put me down for two sites with this problem.

I know that they have some PR4 plus backlinks but non showing also no related:domain results.

I don't think I would crawl one of the sites if I was a self respecting robot but the other one has some merits and is very clean both codewise and link structurewise.


 5:23 pm on Mar 3, 2004 (gmt 0)

This is what I think is happening.

Google is in the process of testing several scoring system until they are sure of it then they will unleash googlebot to update the database.

Meanwhile, I think everybody is experiencing the same thing, just googlebot checking if your site was refreshed from your side and perhaps just checking if the domain is still online. But, not going deep beyond the index or the first level of the directory.

My thought of G testing on various scoring system is based on the flactuating traffic I get from Google. Like, weekends are historically low for me, but the past couple weekends I got weekends that are higher than my weekdays.

Or Mondays whichs are historically high for me, I've seen it goes down than any of the 6 days.

This bring to thought that it's kind of useless to do any tweaking right now.


 6:49 pm on Mar 3, 2004 (gmt 0)

We'd 6K pages crawled on the 29 Feb
New site went live 14 Feb. 1 incoming PR8 link.


 8:01 pm on Mar 3, 2004 (gmt 0)

More evidence for an impending update - hold onto your seats (but keep your brown trousers on :)


 8:18 pm on Mar 3, 2004 (gmt 0)

Why is a new update a reason to reduce spidering? Wouldn't I expect them to try and suck in everything that's out there BEFORE the update?


 9:36 pm on Mar 3, 2004 (gmt 0)

President day is near.


 10:41 pm on Mar 3, 2004 (gmt 0)

Yes, I am seeing this. 6 months old site. Recently aquired few good incoming links (in past 2 months), but no progress.
Total about 20 incoming links with 4 PR6, 1 PR5 and one PR4 link.

Only the Index page is being crawled once in two weeks.


 10:57 pm on Mar 3, 2004 (gmt 0)

Relax guys - GoogleBot has to take a breather too. :)

Google mentioned at PubCon that they had been focusing on depth for the past while, but are now focused on improving relevance scoring. This will explain the lack of deep spidering for the past few weeks.

Patience is a virtue. It will happen soon enough.


 1:44 am on Mar 4, 2004 (gmt 0)

Maybe Google is going PFI.

j/k :-)


 2:25 am on Mar 4, 2004 (gmt 0)

just a quick question.
does any of these site use the <noframes> or <noscript> tags to re-enforce nav structure. I have one the same and it uses frames.


 11:56 am on Mar 4, 2004 (gmt 0)

Yep i think that Net_Wizard is right on.

Exactly what i was thinking. They are experimenting with new algos now
and will unleash googleboot for a full update only once they feel the new algo is satisfactory.

Should happen soon! get ready for some major SERPs movement.

Current Googleboot visits are just to make sure your domain(site) is still online.

My 2 cents.


 12:22 pm on Mar 4, 2004 (gmt 0)

"Why is a new update a reason to reduce spidering"

It isn't. People are groping for easy answers that don't relate to the status quo.

Googlebot has been hitting all my sites like crazy for weeks now. Hundreds of pages, half or more of the domains.

This hyperactive googlebot activity is also obvious to observe in the Google Directory, dmoz and other higher PR sites that get fresh pages daily.

Better seek other explainations for your situations because Googlebot has been more active than it ever has been.


 12:31 pm on Mar 4, 2004 (gmt 0)

Mm - I agree I am not seeing to much Googlebot activity at the moment either (although I did have a little spike yesterday morning.)

But, it will come soon(ish) no doubt - in the meantime we should be all adding content, links etc to make sure we are ready.

A Webmasters job is never done - ever :).


 1:07 pm on Mar 4, 2004 (gmt 0)

I'm seeing this on a new site (launched middle of Feb) with several incoming links to it already. Googlebot is grabbing the index page every day, but going no further. robots.txt and linking structure are fine.

Must admit, it's a bit worrying.


 2:20 pm on Mar 4, 2004 (gmt 0)

Okay, I tried to start a new topic that Googlebot have started deep crawling but it seems that topic is stuck in limbo. Mods?

But, I say the deep crawling have started yesterday and I mean...DEEP including dynamic URLS.

For the full day yesterday, it's just 2.1 Googlebot all day long. Not sure this morning I haven't check my log yet.

I'll check again and check my other site as well.

We might be in for a huge update.


 6:30 pm on Mar 4, 2004 (gmt 0)

>>>We might be in for a huge update.

Great - thats all we need!

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved