How to Encourage a Deep Crawl?

Googlebot isn't indexing all of my site.

[WallaceCleaver]

My domain only has about two pages that googlebot keeps hitting. How do I get it to go for a deep crawl as the pages that it is crawling aren't exactly the best that my site has to offer. It's not even hitting the index page.

[jatar_k]

throw a link on those 2 pages that it is indexing to your sitemap.

[pageoneresults]

How do I get it to go for a deep crawl as the pages that it is crawling aren't exactly the best that my site has to offer.

Are they static (looking) URIs or dynamic? For example, here is a static (looking) URI...

www.example.com/product/widget/

Here is an example of dynamic...

www.example.com/product.asp&id=777&cat=widget&id=888...

[Reid]

use a bot simulator to check the site, especially the page it is asking for to see how bots view it.
Typiclly when googlebot is having problems with crawling the first page it finds it will keep asking for 2 pages, that page and robots.txt check the page it is asking for and as suggested above make sure that page has a link to a sitemap.

[BigDave]

How old is your site? It usually takes around a month for Google to decide to go much beyond the pages that it reaches from inbound links.

Other than waiting, about the only way to speed it up is to get inbound deep links to more of your pages.

[WallaceCleaver]

The links are not dynamic.

Not to sound like a newbie, but what is a deep link? How does it differ from a regular link to my home page?

Um, I don't have a sitemap. Should I get one? The pages that googlebot has been crawling have a link to my index and it has links to many of the other sections of my site.

[BigDave]

The way a deep link differs from a link to your home page is that it is a link to any page other than your home page.

If your site is a bunch of reviews of movies you have seen, you want to get a link from a page that is already in the index, to your review of Yellowbeard, and another one from a different page to your review of Traxx.

That will bring Googlebot directly to those pages.

You also want to have good internal navigation back up through your tree, not just back to the home page.

You still haven't told us how old your site is.

[WallaceCleaver]

Oh, sorry. I forgot to mention the age of the site. It has been online for about 6 months, but I haven't had much content on it until recently. I was working on other sites and only recently started really working on it. The problem is google won't crawl the new pages.

[BigDave]

How old are the new pages?

[WallaceCleaver]

About two weeks old.

[SEOtop10]

Creating a sitemap.xml as per Google instructions here <https://www.google.com/webmasters/sitemaps/login> should help getting your full site indexed.

Advanced warning - the instructions are slightly difficult for a novice webmaster. However you can search for "create google sitemap" and look at the results to find some more information.

I got very good results using Google sitemaps.

[pageoneresults]

It has been online for about 6 months, but I haven't had much content on it until recently.

That may be one of the main issues there. If you haven't had much content on it until recently, then you are pretty much starting from scratch. And, if the site doesn't have a decent level of PageRank, then the crawl pattern of Googlebot may be erratic and not as frequent as a site that has PR and content.

If you've got links from the pages that Googlebot has crawled and indexed that go to the other pages on your site, once you have content on them, they will get crawled eventually, it takes some time, especially with a new site that has little to no PR.

[BigDave]

Wait.

Googlebot is not the fastest at picking up new pages on low PR sites. And if your site has not changed at all in a long time, Google has probably noticed and does not have it on a frequent crawl schedule.