Welcome to WebmasterWorld Guest from 54.145.53.251

Forum Moderators: open

Message Too Old, No Replies

Don't crawl beyond index page syndrom

     
2:55 pm on Mar 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:1825
votes: 21


Anyone else seeing it?
3:45 pm on Mar 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2003
posts:790
votes: 0


yes, definitely. I thought it was because I had a couple of flaws in my linking logic - but maybe there's another reason.

Is your site new (or almost new)?

3:53 pm on Mar 3, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Sept 7, 2002
posts:300
votes: 0


YES!

For about 3-4 weeks even, typical for a backdate/update though.

4:16 pm on Mar 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 31, 2001
posts:1357
votes: 0


Put me down for two sites with this problem.

I know that they have some PR4 plus backlinks but non showing also no related:domain results.

I don't think I would crawl one of the sites if I was a self respecting robot but the other one has some merits and is very clean both codewise and link structurewise.
Sid

5:23 pm on Mar 3, 2004 (gmt 0)

Preferred Member

joined:May 28, 2002
posts:377
votes: 0


This is what I think is happening.

Google is in the process of testing several scoring system until they are sure of it then they will unleash googlebot to update the database.

Meanwhile, I think everybody is experiencing the same thing, just googlebot checking if your site was refreshed from your side and perhaps just checking if the domain is still online. But, not going deep beyond the index or the first level of the directory.

My thought of G testing on various scoring system is based on the flactuating traffic I get from Google. Like, weekends are historically low for me, but the past couple weekends I got weekends that are higher than my weekdays.

Or Mondays whichs are historically high for me, I've seen it goes down than any of the 6 days.

This bring to thought that it's kind of useless to do any tweaking right now.

6:49 pm on Mar 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 2, 2002
posts:918
votes: 0


We'd 6K pages crawled on the 29 Feb
New site went live 14 Feb. 1 incoming PR8 link.
J
8:01 pm on Mar 3, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Feb 17, 2004
posts:261
votes: 0


More evidence for an impending update - hold onto your seats (but keep your brown trousers on :)
8:18 pm on Mar 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2003
posts:790
votes: 0


Why is a new update a reason to reduce spidering? Wouldn't I expect them to try and suck in everything that's out there BEFORE the update?
9:36 pm on Mar 3, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Jan 9, 2003
posts:235
votes: 0


President day is near.
10:41 pm on Mar 3, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 20, 2003
posts:450
votes: 0


Yes, I am seeing this. 6 months old site. Recently aquired few good incoming links (in past 2 months), but no progress.
Total about 20 incoming links with 4 PR6, 1 PR5 and one PR4 link.

Only the Index page is being crawled once in two weeks.

10:57 pm on Mar 3, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 12, 2003
posts:155
votes: 0


Relax guys - GoogleBot has to take a breather too. :)

Google mentioned at PubCon that they had been focusing on depth for the past while, but are now focused on improving relevance scoring. This will explain the lack of deep spidering for the past few weeks.

Patience is a virtue. It will happen soon enough.

1:44 am on Mar 4, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Sept 7, 2002
posts:300
votes: 0


Maybe Google is going PFI.

j/k :-)

2:25 am on Mar 4, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Oct 23, 2003
posts:314
votes: 0


just a quick question.
does any of these site use the <noframes> or <noscript> tags to re-enforce nav structure. I have one the same and it uses frames.
11:56 am on Mar 4, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 2, 2003
posts:39
votes: 0


Yep i think that Net_Wizard is right on.

Exactly what i was thinking. They are experimenting with new algos now
and will unleash googleboot for a full update only once they feel the new algo is satisfactory.

Should happen soon! get ready for some major SERPs movement.

P.S.
Current Googleboot visits are just to make sure your domain(site) is still online.

My 2 cents.

12:22 pm on Mar 4, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


"Why is a new update a reason to reduce spidering"

It isn't. People are groping for easy answers that don't relate to the status quo.

Googlebot has been hitting all my sites like crazy for weeks now. Hundreds of pages, half or more of the domains.

This hyperactive googlebot activity is also obvious to observe in the Google Directory, dmoz and other higher PR sites that get fresh pages daily.

Better seek other explainations for your situations because Googlebot has been more active than it ever has been.

Dayo_UK

12:31 pm on Mar 4, 2004 (gmt 0)

Inactive Member
Account Expired

 
 


Mm - I agree I am not seeing to much Googlebot activity at the moment either (although I did have a little spike yesterday morning.)

But, it will come soon(ish) no doubt - in the meantime we should be all adding content, links etc to make sure we are ready.

A Webmasters job is never done - ever :).

1:07 pm on Mar 4, 2004 (gmt 0)

New User

10+ Year Member

joined:Nov 11, 2003
posts:22
votes: 0


I'm seeing this on a new site (launched middle of Feb) with several incoming links to it already. Googlebot is grabbing the index page every day, but going no further. robots.txt and linking structure are fine.

Must admit, it's a bit worrying.

2:20 pm on Mar 4, 2004 (gmt 0)

Preferred Member

joined:May 28, 2002
posts:377
votes: 0


Okay, I tried to start a new topic that Googlebot have started deep crawling but it seems that topic is stuck in limbo. Mods?

But, I say the deep crawling have started yesterday and I mean...DEEP including dynamic URLS.

For the full day yesterday, it's just 2.1 Googlebot all day long. Not sure this morning I haven't check my log yet.

I'll check again and check my other site as well.

We might be in for a huge update.

6:30 pm on Mar 4, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 2, 2002
posts:918
votes: 0


>>>We might be in for a huge update.

Great - thats all we need!