Welcome to WebmasterWorld Guest from 54.226.194.180

Forum Moderators: open

Message Too Old, No Replies

Don't crawl beyond index page syndrom

     

SEOPTI

2:55 pm on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Anyone else seeing it?

the_nerd

3:45 pm on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



yes, definitely. I thought it was because I had a couple of flaws in my linking logic - but maybe there's another reason.

Is your site new (or almost new)?

HayMeadows

3:53 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



YES!

For about 3-4 weeks even, typical for a backdate/update though.

Hissingsid

4:16 pm on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Put me down for two sites with this problem.

I know that they have some PR4 plus backlinks but non showing also no related:domain results.

I don't think I would crawl one of the sites if I was a self respecting robot but the other one has some merits and is very clean both codewise and link structurewise.
Sid

Net_Wizard

5:23 pm on Mar 3, 2004 (gmt 0)



This is what I think is happening.

Google is in the process of testing several scoring system until they are sure of it then they will unleash googlebot to update the database.

Meanwhile, I think everybody is experiencing the same thing, just googlebot checking if your site was refreshed from your side and perhaps just checking if the domain is still online. But, not going deep beyond the index or the first level of the directory.

My thought of G testing on various scoring system is based on the flactuating traffic I get from Google. Like, weekends are historically low for me, but the past couple weekends I got weekends that are higher than my weekdays.

Or Mondays whichs are historically high for me, I've seen it goes down than any of the 6 days.

This bring to thought that it's kind of useless to do any tweaking right now.

johnser

6:49 pm on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We'd 6K pages crawled on the 29 Feb
New site went live 14 Feb. 1 incoming PR8 link.
J

SyntheticUpper

8:01 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



More evidence for an impending update - hold onto your seats (but keep your brown trousers on :)

the_nerd

8:18 pm on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Why is a new update a reason to reduce spidering? Wouldn't I expect them to try and suck in everything that's out there BEFORE the update?

otnot

9:36 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



President day is near.

sdani

10:41 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



Yes, I am seeing this. 6 months old site. Recently aquired few good incoming links (in past 2 months), but no progress.
Total about 20 incoming links with 4 PR6, 1 PR5 and one PR4 link.

Only the Index page is being crawled once in two weeks.

ryan26

10:57 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



Relax guys - GoogleBot has to take a breather too. :)

Google mentioned at PubCon that they had been focusing on depth for the past while, but are now focused on improving relevance scoring. This will explain the lack of deep spidering for the past few weeks.

Patience is a virtue. It will happen soon enough.

HayMeadows

1:44 am on Mar 4, 2004 (gmt 0)

10+ Year Member



Maybe Google is going PFI.

j/k :-)

trimmer80

2:25 am on Mar 4, 2004 (gmt 0)

10+ Year Member



just a quick question.
does any of these site use the <noframes> or <noscript> tags to re-enforce nav structure. I have one the same and it uses frames.

Max_mb

11:56 am on Mar 4, 2004 (gmt 0)

10+ Year Member



Yep i think that Net_Wizard is right on.

Exactly what i was thinking. They are experimenting with new algos now
and will unleash googleboot for a full update only once they feel the new algo is satisfactory.

Should happen soon! get ready for some major SERPs movement.

P.S.
Current Googleboot visits are just to make sure your domain(site) is still online.

My 2 cents.

steveb

12:22 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member



"Why is a new update a reason to reduce spidering"

It isn't. People are groping for easy answers that don't relate to the status quo.

Googlebot has been hitting all my sites like crazy for weeks now. Hundreds of pages, half or more of the domains.

This hyperactive googlebot activity is also obvious to observe in the Google Directory, dmoz and other higher PR sites that get fresh pages daily.

Better seek other explainations for your situations because Googlebot has been more active than it ever has been.

Dayo_UK

12:31 pm on Mar 4, 2004 (gmt 0)

10+ Year Member



Mm - I agree I am not seeing to much Googlebot activity at the moment either (although I did have a little spike yesterday morning.)

But, it will come soon(ish) no doubt - in the meantime we should be all adding content, links etc to make sure we are ready.

A Webmasters job is never done - ever :).

mr_strong

1:07 pm on Mar 4, 2004 (gmt 0)

10+ Year Member



I'm seeing this on a new site (launched middle of Feb) with several incoming links to it already. Googlebot is grabbing the index page every day, but going no further. robots.txt and linking structure are fine.

Must admit, it's a bit worrying.

Net_Wizard

2:20 pm on Mar 4, 2004 (gmt 0)



Okay, I tried to start a new topic that Googlebot have started deep crawling but it seems that topic is stuck in limbo. Mods?

But, I say the deep crawling have started yesterday and I mean...DEEP including dynamic URLS.

For the full day yesterday, it's just 2.1 Googlebot all day long. Not sure this morning I haven't check my log yet.

I'll check again and check my other site as well.

We might be in for a huge update.

johnser

6:30 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>>>We might be in for a huge update.

Great - thats all we need!

 

Featured Threads

Hot Threads This Week

Hot Threads This Month