Page is a not externally linkable
-- Google SEO News and Discussion
---- How do huge sites get such complete index coverage?
deadsea - 11:01 am on Apr 5, 2012 (gmt 0) You don't need a special relationship with Google. I haven't worked on a big site yet that has one other than a large adwords budget (and that doesn't get you beans in terms of SEO).
I do SEO for such huge sites. Here is my take:
If you have millions of pages, Googlebot can do a lot of crawling. On a site with 10s of millions of page, at one point, 40% of the pages we served were going to Googlebot.
Google seems to be moving away from pagerank as a ranking signal, but googlebot still uses it to determine what to crawl. If you want to have a large site, you need high page rank to get it all crawled. If you have an important page, you need to have enough internal links to it that it gets recrawled frequently.
Googlebot has at least two crawl modes. "Freshbot" will greedily crawl all new pages. I experimented with creating chains of pages starting from a PR 5 page (each page links to the next and so on). In this mode, Googlebot may crawl a chain thousands of pages deep.
"Recrawlbot" will come back and recrawl pages with a frequency based on pagerank. A PR 7 page may be crawled hourly. A PR6 page may be crawled twice a day. A PR 5 page may be crawled every day. A PR 4 page every couple days. A PR 3 page every week. A PR 2 page every two weeks.
Sitemaps can help, but use internal links to highlight your best content.
If you are dealing with user generated content in large volumes you will have to separate your good content from your bad content. If you highlight all your content, even the poor quality stuff, you won't have a very compelling site. Wikipedia spends a lot of effort interlinking their articles such that every good article has many links from other articles. Stack overflow has a robust voting system such that important questions filter up to the top.
Thread source:: http://www.webmasterworld.com/google/4437301.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com