homepage Welcome to WebmasterWorld Guest from 54.237.99.131
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Don't crawl beyond index page syndrom
SEOPTI

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22417 posted 2:55 pm on Mar 3, 2004 (gmt 0)

Anyone else seeing it?

 

the_nerd

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22417 posted 3:45 pm on Mar 3, 2004 (gmt 0)

yes, definitely. I thought it was because I had a couple of flaws in my linking logic - but maybe there's another reason.

Is your site new (or almost new)?

HayMeadows

10+ Year Member



 
Msg#: 22417 posted 3:53 pm on Mar 3, 2004 (gmt 0)

YES!

For about 3-4 weeks even, typical for a backdate/update though.

Hissingsid

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22417 posted 4:16 pm on Mar 3, 2004 (gmt 0)

Put me down for two sites with this problem.

I know that they have some PR4 plus backlinks but non showing also no related:domain results.

I don't think I would crawl one of the sites if I was a self respecting robot but the other one has some merits and is very clean both codewise and link structurewise.
Sid

Net_Wizard



 
Msg#: 22417 posted 5:23 pm on Mar 3, 2004 (gmt 0)

This is what I think is happening.

Google is in the process of testing several scoring system until they are sure of it then they will unleash googlebot to update the database.

Meanwhile, I think everybody is experiencing the same thing, just googlebot checking if your site was refreshed from your side and perhaps just checking if the domain is still online. But, not going deep beyond the index or the first level of the directory.

My thought of G testing on various scoring system is based on the flactuating traffic I get from Google. Like, weekends are historically low for me, but the past couple weekends I got weekends that are higher than my weekdays.

Or Mondays whichs are historically high for me, I've seen it goes down than any of the 6 days.

This bring to thought that it's kind of useless to do any tweaking right now.

johnser

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22417 posted 6:49 pm on Mar 3, 2004 (gmt 0)

We'd 6K pages crawled on the 29 Feb
New site went live 14 Feb. 1 incoming PR8 link.
J

SyntheticUpper

10+ Year Member



 
Msg#: 22417 posted 8:01 pm on Mar 3, 2004 (gmt 0)

More evidence for an impending update - hold onto your seats (but keep your brown trousers on :)

the_nerd

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22417 posted 8:18 pm on Mar 3, 2004 (gmt 0)

Why is a new update a reason to reduce spidering? Wouldn't I expect them to try and suck in everything that's out there BEFORE the update?

otnot

10+ Year Member



 
Msg#: 22417 posted 9:36 pm on Mar 3, 2004 (gmt 0)

President day is near.

sdani

10+ Year Member



 
Msg#: 22417 posted 10:41 pm on Mar 3, 2004 (gmt 0)

Yes, I am seeing this. 6 months old site. Recently aquired few good incoming links (in past 2 months), but no progress.
Total about 20 incoming links with 4 PR6, 1 PR5 and one PR4 link.

Only the Index page is being crawled once in two weeks.

ryan26

10+ Year Member



 
Msg#: 22417 posted 10:57 pm on Mar 3, 2004 (gmt 0)

Relax guys - GoogleBot has to take a breather too. :)

Google mentioned at PubCon that they had been focusing on depth for the past while, but are now focused on improving relevance scoring. This will explain the lack of deep spidering for the past few weeks.

Patience is a virtue. It will happen soon enough.

HayMeadows

10+ Year Member



 
Msg#: 22417 posted 1:44 am on Mar 4, 2004 (gmt 0)

Maybe Google is going PFI.

j/k :-)

trimmer80

10+ Year Member



 
Msg#: 22417 posted 2:25 am on Mar 4, 2004 (gmt 0)

just a quick question.
does any of these site use the <noframes> or <noscript> tags to re-enforce nav structure. I have one the same and it uses frames.

Max_mb

10+ Year Member



 
Msg#: 22417 posted 11:56 am on Mar 4, 2004 (gmt 0)

Yep i think that Net_Wizard is right on.

Exactly what i was thinking. They are experimenting with new algos now
and will unleash googleboot for a full update only once they feel the new algo is satisfactory.

Should happen soon! get ready for some major SERPs movement.

P.S.
Current Googleboot visits are just to make sure your domain(site) is still online.

My 2 cents.

steveb

WebmasterWorld Senior Member steveb us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 22417 posted 12:22 pm on Mar 4, 2004 (gmt 0)

"Why is a new update a reason to reduce spidering"

It isn't. People are groping for easy answers that don't relate to the status quo.

Googlebot has been hitting all my sites like crazy for weeks now. Hundreds of pages, half or more of the domains.

This hyperactive googlebot activity is also obvious to observe in the Google Directory, dmoz and other higher PR sites that get fresh pages daily.

Better seek other explainations for your situations because Googlebot has been more active than it ever has been.

Dayo_UK

10+ Year Member



 
Msg#: 22417 posted 12:31 pm on Mar 4, 2004 (gmt 0)

Mm - I agree I am not seeing to much Googlebot activity at the moment either (although I did have a little spike yesterday morning.)

But, it will come soon(ish) no doubt - in the meantime we should be all adding content, links etc to make sure we are ready.

A Webmasters job is never done - ever :).

mr_strong

10+ Year Member



 
Msg#: 22417 posted 1:07 pm on Mar 4, 2004 (gmt 0)

I'm seeing this on a new site (launched middle of Feb) with several incoming links to it already. Googlebot is grabbing the index page every day, but going no further. robots.txt and linking structure are fine.

Must admit, it's a bit worrying.

Net_Wizard



 
Msg#: 22417 posted 2:20 pm on Mar 4, 2004 (gmt 0)

Okay, I tried to start a new topic that Googlebot have started deep crawling but it seems that topic is stuck in limbo. Mods?

But, I say the deep crawling have started yesterday and I mean...DEEP including dynamic URLS.

For the full day yesterday, it's just 2.1 Googlebot all day long. Not sure this morning I haven't check my log yet.

I'll check again and check my other site as well.

We might be in for a huge update.

johnser

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22417 posted 6:30 pm on Mar 4, 2004 (gmt 0)

>>>We might be in for a huge update.

Great - thats all we need!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved