Welcome to WebmasterWorld Guest from 3.227.208.153

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

How to Encourage a Deep Crawl?

Googlebot isn't indexing all of my site.

     
3:20 am on Jun 10, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:June 23, 2002
posts:98
votes: 0


My domain only has about two pages that googlebot keeps hitting. How do I get it to go for a deep crawl as the pages that it is crawling aren't exactly the best that my site has to offer. It's not even hitting the index page.
6:30 pm on June 10, 2005 (gmt 0)

Administrator

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 24, 2001
posts:15756
votes: 0


throw a link on those 2 pages that it is indexing to your sitemap.
6:38 pm on June 10, 2005 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts:12172
votes: 61


How do I get it to go for a deep crawl as the pages that it is crawling aren't exactly the best that my site has to offer.

Are they static (looking) URIs or dynamic? For example, here is a static (looking) URI...

www.example.com/product/widget/

Here is an example of dynamic...

www.example.com/product.asp&id=777&cat=widget&id=888...

7:01 pm on June 10, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2004
posts:693
votes: 0


use a bot simulator to check the site, especially the page it is asking for to see how bots view it.
Typiclly when googlebot is having problems with crawling the first page it finds it will keep asking for 2 pages, that page and robots.txt check the page it is asking for and as suggested above make sure that page has a link to a sitemap.
7:40 pm on June 10, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 19, 2002
posts:3454
votes: 0


How old is your site? It usually takes around a month for Google to decide to go much beyond the pages that it reaches from inbound links.

Other than waiting, about the only way to speed it up is to get inbound deep links to more of your pages.

12:31 am on June 11, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:June 23, 2002
posts:98
votes: 0


The links are not dynamic.

Not to sound like a newbie, but what is a deep link? How does it differ from a regular link to my home page?

Um, I don't have a sitemap. Should I get one? The pages that googlebot has been crawling have a link to my index and it has links to many of the other sections of my site.

12:52 am on June 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 19, 2002
posts:3454
votes: 0


The way a deep link differs from a link to your home page is that it is a link to any page other than your home page.

If your site is a bunch of reviews of movies you have seen, you want to get a link from a page that is already in the index, to your review of Yellowbeard, and another one from a different page to your review of Traxx.

That will bring Googlebot directly to those pages.

You also want to have good internal navigation back up through your tree, not just back to the home page.

You still haven't told us how old your site is.

4:22 am on June 11, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:June 23, 2002
posts:98
votes: 0


Oh, sorry. I forgot to mention the age of the site. It has been online for about 6 months, but I haven't had much content on it until recently. I was working on other sites and only recently started really working on it. The problem is google won't crawl the new pages.
4:37 am on June 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 19, 2002
posts:3454
votes: 0


How old are the new pages?
1:34 pm on June 11, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:June 23, 2002
posts:98
votes: 0


About two weeks old.
3:16 pm on June 11, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 20, 2003
posts:167
votes: 0


Creating a sitemap.xml as per Google instructions here <https://www.google.com/webmasters/sitemaps/login> should help getting your full site indexed.

Advanced warning - the instructions are slightly difficult for a novice webmaster. However you can search for "create google sitemap" and look at the results to find some more information.

I got very good results using Google sitemaps.

3:37 pm on June 11, 2005 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts:12172
votes: 61


It has been online for about 6 months, but I haven't had much content on it until recently.

That may be one of the main issues there. If you haven't had much content on it until recently, then you are pretty much starting from scratch. And, if the site doesn't have a decent level of PageRank, then the crawl pattern of Googlebot may be erratic and not as frequent as a site that has PR and content.

If you've got links from the pages that Googlebot has crawled and indexed that go to the other pages on your site, once you have content on them, they will get crawled eventually, it takes some time, especially with a new site that has little to no PR.

4:03 pm on June 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 19, 2002
posts:3454
votes: 0


Wait.

Googlebot is not the fastest at picking up new pages on low PR sites. And if your site has not changed at all in a long time, Google has probably noticed and does not have it on a frequent crawl schedule.