Major factor to deep crawl.

How to make google crawl all my website's pages?

[fischermx]

I recently started a new website, google soon showed the main page in their index. I did not have any backlinks at the begining, then I added some.
Then, google came and put like 50 more pages on their index, some started to show in SERPS. After a few days, google dropped all this extra pages and left the main page alone again.
I kept adding more backlinks, mainly directories, now Google came again and is showing back my 50 pages.
The 50 pages I mention are links from the main page. Each of this links has another 3-10 links to another level of pages.

I feel like if, each time I add some links, googlebot says, "well another vote for this idiot, let's see how does he have", then it comes and try, then it does not get convinced and drop my pages again.

What do you think would be the major factor in causing google to deep crawl my whole site? number of links? PR of those links? pages with more content? change the .aspx extension to .html (though, i don't use "?" parameters)?

[John Carpenter]

Have you tested their Sitemaps [google.com] program?

[fischermx]

Sorry, I didn't mean crawl, I meant indexing, I think.
My site is being crawled every day, I think Google has been already visited all my pages several times, the thing is they don't show up when looking for "site:example.com" and of course they don't show up in serps.

[soccrt]

How do you get Google to deep crawl your site often? Mine is updated a lot, yet Google doesnt seem to go past the sitemap very much.

[fischermx]

How do you get Google to deep crawl your site often?

Well, that's a good question, too.
I don't have too much backlinks, yet, say 50 at most, and mostly of them are from PR0 sites. I'm not listed in major directories either (no yahoo, no dmoz).
So, who knows.

Mine is updated a lot, yet Google doesnt seem to go past the sitemap very much.

Going past the sitemap? Isn't sitemaps to tell Google what to crawl? I would not expect that once having a site map, it would go and crawl additional pages ... but again, who knows.

[BigDave]

You don't give any sort of time frame for how long your site has been up, or when it has added the various pages. You also haven't mentioned if the pages that have added new links are already in the index or not.

Get backlinks. Get deep links. Wait. Repeat.

[soccrt]

The site has been up since 2003, has decent back links and added pages to the site every few days. The blog part of the site has no trouble being crawled and indexed, however, the other part takes a while longer.

Some of the pages have been indexed already, and I have changed some of them to better SEO, but they have not been re-indexed for a couple weeks.

[walkman]

Link to the main categories /sub cats from the front page, unless you have a lot of them. If you have to many, link to a dozen or so, and keep rotating.

[soccrt]

I believe I do have a very structurally sound site, with home page linking to sub categories then the subs linking to the mini pages. Yet it still does not crawl that much

[arran]

Link to the main categories /sub cats from the front page, unless you have a lot of them.

Definitely. From a crawling perspective, less levels is better. This structure coupled with one PR5+ link should get you deep crawled and fairly well indexed.

[bumpski]

After the Bourbon update it seemed like Google needed absolute links to keep pages fully indexed. I had a sitemap.html page with absolute links to all my site's pages. I was trying to boost my homepage PR by removing the links to my sitemap.html file from all my pages, etc. When I pulled these links all I had left was a set of hierarchical relative links, probably four layers deep. Over several weeks 80% of my pages became URL only, which is basically "crawled" but not "fully indexed". Google calls it "partially indexed".
I restored my links to my sitemap.html file which has absolute links to all my pages and was reindexed fully after 3 more crawls. I pulled my links to my sitemap.html page again, after having installed a sitemap.xml file, with prioritization, and still pages started to drop from the index. Needless to say my links on all pages to my sitemap.html page are back again, and so are most of my pages in the index.
I believe this type of "drop out" may have happened to many sites after the BourBAN update so I could be wrong and its all a coincidence.