Welcome to WebmasterWorld Guest from 54.163.168.15

Google Indexing Large Sites Too Slowly

   
5:00 am on Jun 14, 2005 (gmt 0)

5+ Year Member


Hi all, I am new here.
I have site with more than 40.000 page, google index only 6980 pages only, every day I check google and found google add exactly 40 page each day, so it will takes more than three years to get all pages in the site indexed, how to make the google index my site faster?, I hear new google feature, google sitemap, is it help?

Regards,

3:28 pm on Jun 15, 2005 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Get more links, then get some more links and after that get even more links.

Sitemaps is still in the experimentation stage. The final results are unknown for the sitemap program.

3:44 pm on Jun 15, 2005 (gmt 0)

10+ Year Member



"I have site with more than 40.000 page"

Spam?

3:53 pm on Jun 15, 2005 (gmt 0)

5+ Year Member



not spam, product and its variation, but thanks for your comment and suggestion
3:56 pm on Jun 15, 2005 (gmt 0)

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Get deep links to some of the individual product pages, not just your home page.
4:39 pm on Jun 15, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Santip, this is one of the places where good page rank and fresh pages help a lot.

First -- do you pass parameters in the URL. If so, drop them and move to a mod_rewrite.

Second -- Do you have a high enough visible toolbar? 5-7?

Third -- Do you have any pages in the daily crawl? If yes, start rotating deep links to uncrawled pages from them. If you don't have pages in the daily crawl, try to keep your front page as fresh as possible and aim for a PR4-5.

Fourth -- Do you have a good on-site nav structure? Good breadcrumbs? A good site map?

Have you memorised the crawling pattern? There are a few things to be learnt from how the bot hits your site. The frequency of homepage visits and the pages collected later...

6:50 pm on Jun 15, 2005 (gmt 0)

5+ Year Member



BigDave,
do you mean cross linking on product page?
the current site structure was: index > category > product, no cross linking between product page.
Shri,
1. no pass parameter in url, I read in google guidelines, that's why the site has so many pages
2. pr was 4, expected to be 5 in next update, when?
3. yes, daily crawl from google added exactly 40 pages each day
4. maybe that is the answer, we had to work on site navigation, do you had any sample site with good site sturcture?
1:04 am on Jun 16, 2005 (gmt 0)



>> product, no cross linking between product page

you should do a "Random Page" thing and each page links to 2-5 of your other pages. Not spammy at all and everything is guaranteed to be eventually seen by Google.

1:26 am on Jun 16, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>> do you had any sample site with good site sturcture?

This site.

Also, take a look at theme pyramid documents (google them)

1:42 am on Jun 16, 2005 (gmt 0)

10+ Year Member



I agree with Shri.

Adding deep(er) links to daily crawl pages will definitely help. Rotating deep links in our sitemap/directory has been a surefire way of helping G find our deep pages. Naturally, having a site structure that is Googlebot-friendly is also important.

We recently streamlined the site structure of one of our portals and noticed significantly higher daily crawls. As a result, site indexing almost quadrupled from 120k -> 450k pages.

Also remember that being crawled doesn't necessitate inclusion in the index.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month