homepage Welcome to WebmasterWorld Guest from 54.204.215.209
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Accredited PayPal World Seller

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Don't crawl beyond index page syndrom
SEOPTI




msg:87676
 2:55 pm on Mar 3, 2004 (gmt 0)

Anyone else seeing it?

 

the_nerd




msg:87677
 3:45 pm on Mar 3, 2004 (gmt 0)

yes, definitely. I thought it was because I had a couple of flaws in my linking logic - but maybe there's another reason.

Is your site new (or almost new)?

HayMeadows




msg:87678
 3:53 pm on Mar 3, 2004 (gmt 0)

YES!

For about 3-4 weeks even, typical for a backdate/update though.

Hissingsid




msg:87679
 4:16 pm on Mar 3, 2004 (gmt 0)

Put me down for two sites with this problem.

I know that they have some PR4 plus backlinks but non showing also no related:domain results.

I don't think I would crawl one of the sites if I was a self respecting robot but the other one has some merits and is very clean both codewise and link structurewise.
Sid

Net_Wizard




msg:87680
 5:23 pm on Mar 3, 2004 (gmt 0)

This is what I think is happening.

Google is in the process of testing several scoring system until they are sure of it then they will unleash googlebot to update the database.

Meanwhile, I think everybody is experiencing the same thing, just googlebot checking if your site was refreshed from your side and perhaps just checking if the domain is still online. But, not going deep beyond the index or the first level of the directory.

My thought of G testing on various scoring system is based on the flactuating traffic I get from Google. Like, weekends are historically low for me, but the past couple weekends I got weekends that are higher than my weekdays.

Or Mondays whichs are historically high for me, I've seen it goes down than any of the 6 days.

This bring to thought that it's kind of useless to do any tweaking right now.

johnser




msg:87681
 6:49 pm on Mar 3, 2004 (gmt 0)

We'd 6K pages crawled on the 29 Feb
New site went live 14 Feb. 1 incoming PR8 link.
J

SyntheticUpper




msg:87682
 8:01 pm on Mar 3, 2004 (gmt 0)

More evidence for an impending update - hold onto your seats (but keep your brown trousers on :)

the_nerd




msg:87683
 8:18 pm on Mar 3, 2004 (gmt 0)

Why is a new update a reason to reduce spidering? Wouldn't I expect them to try and suck in everything that's out there BEFORE the update?

otnot




msg:87684
 9:36 pm on Mar 3, 2004 (gmt 0)

President day is near.

sdani




msg:87685
 10:41 pm on Mar 3, 2004 (gmt 0)

Yes, I am seeing this. 6 months old site. Recently aquired few good incoming links (in past 2 months), but no progress.
Total about 20 incoming links with 4 PR6, 1 PR5 and one PR4 link.

Only the Index page is being crawled once in two weeks.

ryan26




msg:87686
 10:57 pm on Mar 3, 2004 (gmt 0)

Relax guys - GoogleBot has to take a breather too. :)

Google mentioned at PubCon that they had been focusing on depth for the past while, but are now focused on improving relevance scoring. This will explain the lack of deep spidering for the past few weeks.

Patience is a virtue. It will happen soon enough.

HayMeadows




msg:87687
 1:44 am on Mar 4, 2004 (gmt 0)

Maybe Google is going PFI.

j/k :-)

trimmer80




msg:87688
 2:25 am on Mar 4, 2004 (gmt 0)

just a quick question.
does any of these site use the <noframes> or <noscript> tags to re-enforce nav structure. I have one the same and it uses frames.

Max_mb




msg:87689
 11:56 am on Mar 4, 2004 (gmt 0)

Yep i think that Net_Wizard is right on.

Exactly what i was thinking. They are experimenting with new algos now
and will unleash googleboot for a full update only once they feel the new algo is satisfactory.

Should happen soon! get ready for some major SERPs movement.

P.S.
Current Googleboot visits are just to make sure your domain(site) is still online.

My 2 cents.

steveb




msg:87690
 12:22 pm on Mar 4, 2004 (gmt 0)

"Why is a new update a reason to reduce spidering"

It isn't. People are groping for easy answers that don't relate to the status quo.

Googlebot has been hitting all my sites like crazy for weeks now. Hundreds of pages, half or more of the domains.

This hyperactive googlebot activity is also obvious to observe in the Google Directory, dmoz and other higher PR sites that get fresh pages daily.

Better seek other explainations for your situations because Googlebot has been more active than it ever has been.

Dayo_UK




msg:87691
 12:31 pm on Mar 4, 2004 (gmt 0)

Mm - I agree I am not seeing to much Googlebot activity at the moment either (although I did have a little spike yesterday morning.)

But, it will come soon(ish) no doubt - in the meantime we should be all adding content, links etc to make sure we are ready.

A Webmasters job is never done - ever :).

mr_strong




msg:87692
 1:07 pm on Mar 4, 2004 (gmt 0)

I'm seeing this on a new site (launched middle of Feb) with several incoming links to it already. Googlebot is grabbing the index page every day, but going no further. robots.txt and linking structure are fine.

Must admit, it's a bit worrying.

Net_Wizard




msg:87693
 2:20 pm on Mar 4, 2004 (gmt 0)

Okay, I tried to start a new topic that Googlebot have started deep crawling but it seems that topic is stuck in limbo. Mods?

But, I say the deep crawling have started yesterday and I mean...DEEP including dynamic URLS.

For the full day yesterday, it's just 2.1 Googlebot all day long. Not sure this morning I haven't check my log yet.

I'll check again and check my other site as well.

We might be in for a huge update.

johnser




msg:87694
 6:30 pm on Mar 4, 2004 (gmt 0)

>>>We might be in for a huge update.

Great - thats all we need!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved