Deep crawling behavior

Seems kind of strange

[Powdork]

I recently uploaded a new site and 216.239 spent quite a lovely morning with it today. Problem is, at this point she has not ventured to my nether regions. She has only stayed within one link of the homepage. Will she come back and follow links to the third level or as this site is new will she just index the url and not the content of the third level pages. This after all is where most of the meat of the content is.

[jdMorgan]

Add a site map!

Jim

[jatar_k]

Powdork,

a little more information would help. Are your links spiderable? Have you tested the sim spider [searchengineworld.com]? It can help give insight into what spiders see.

Is it the bot's first ever visit? Can you manage to wait and see? ;) What has been your experience with other sites that you have rolled out? do you have a site map linked from the main page?

just a few little questions that might help you find your answer.

[Powdork]

jatar k
The home page links to twenty or so pages. Many of these pages discuss a product (ceremony or reception location) and either link directly to the vendor's site or to a page with more information (these are what didn't get spidered). The links to these pages are all standard html. In each case there is the logo image which links to the page as well as a textual in content link.

Is it the bot's first ever visit?

64.68 strolled by once last month. Also, this is a domain I bought 4 months ago. The site has been down for about five months and was out of the index completely. It was indexed when under the previous owner. It is now indexed but only the url. For some reason as soon as I uploaded it it had a pr 4.

Can you manage to wait and see?

Thems fightin' words ;)

What has been your experience with other sites that you have rolled out?

First one, everything else has been picking up the pieces of someone elses improperly laid foundation.

do you have a site map linked from the main page?

No and i don't plan to. I will have a link to all the product info pages as part of the footer thingy to every page. Its still a bit unfinished but I had some good links to it and knew I gould get a bunch of content indexed for its December opening. Problem is that much of it is on the as yet unspidered pages. If it was just 5 pages that hadn't been spidered I wouldn't be worried. Its that there is a common thread between the pages that makes me think a new site may only be worth spidering to two levels at this point.

[Slade]

Googlebot probably considers this a new site. It'll pick up a few pages, then wander off to other parts of the net.

My guess is she doesn't like your site enough yet to do a full crawl. She should come back.

The reason a sitemap was suggested is to give googlebot the impression that every page is only 1 or 2 levels away from the index page. It's a method of getting deepcrawled without getting deepcrawled(does that make sense?).

[jatar_k]

I would say that you will have to wait and see. It sounds like everything is rolling along and I agree with Slade that googlebot will be back.

New sites are always a pain, waiting isn't really in seo nature. ;)

spend the waiting time link building so when it does get deep crawled the site pops and makes up for lost time.

[Powdork]

I didn't mean to imply that a site map was bad. It just won't solve my problem. The pages not being spidered are all in the third level(i.e. two clicks from the externally linked page). With a site map they would still be in the third level. I don't foresee going past the third level and I don't think the product pages will get so cumbersome as they can't be listed cleanly at the bottom of each page. I would love it if that should happen, though. $:)$:)$

[Powdork]

Ok, nevermind, she came by and picked up two of the five previously unspidered urls since our last chat. Now I'm curious as to how this happens. Does the spider leave pheromones behind at the uncrawled links or send back a message to the queen spider, or what.
Thanks everyone

[jatar_k]

I figure when it has 2,469,940,685 pages to spider it takes an interesting schedule and we must begrudge it a little time.

[coosblues]

well, looking at my site stats the bot has been one very busy lady. My site is very new with a low pr (my first month on google) so any green is a delight to me. Within the past day she's visited numerous times and it looks like a very deep crawl at that. Something a bit off topic but someone out there can explain this to me i'm sure. On my stats the server provides I get a list of bots and such under a section called BROWSERSUM.html - yesterday the googlebot count was 38 and today its 61. I have another section on my stats called Referrer.html. Again, good signs of the lady [216.239.35.120...] ETC. (this shows up as a link on my stats) Ok, my question is when i click to open that link my PR has gone up. Better than down I would assume or is it meaningless. Thanks to you all