Forum Moderators: open
which sites manage to get most of their pages crawled? Is it PR related? (assuming the index PR gives a broad indication of the total site's PR)
The quick and dirty way I searched was:
Microsoft and then "more results from www.microsoft.com"
or site:www.microsoft.com microsoft
Some examples of big sites:
yahoo.com 3.8 mill (but contains clustered off-topic subs such as weather.yahoo)
web.mit.edu 852.200 PR10
nasa.gov 414.000 (including sub.nasa sites) PR10
dmoz.org 335.000 PR10
microsoft.com 176.000 PR10
bbc.co.uk 159.000 PR9
allafrica.com 105.000 PR8
nrc.nl 61.100 PR7
(the last two as examples of less high PR doing well)
Are there people here with PR5 or PR6 sites (not newspapers) still getting more than 10-20.000 pages indexed?
Or for people within that pagerank range (5-6) what percentage of your pages generally get taken into the index (is there a break-off value for which you find it stops? e.g. PR2)
I would assume Google groups would be the largest but i did not manage to find a way to search that. (searching for "RE" gives 198 mill.)
I personally had 700 of 750 pages indexed by Google (not after this months update though, but still...). If you just get a reasonable amount of PR (at least 5), and use good text links in your site, I'm sure that will help getting them indexed.
thats the type of info I was looking for.
Are those all the pages of your site (from the last monthly deep crawl)?
And are you getting complete deep crawls every month?
My guess is that Google attempts a rough estimate of a page's PR, based on the link that it's following, and if it looks like being reasonable then it will crawl the page. That will also help avoid getting trapped in infinite dynamic content.
Is page rank also dependant upon the category you are in? (i.e. kids space adventure doesn't have much competition so we got a high ranking).
I'm about to roll out 5,000 pages as my first big time content push. FYI these have been pulled from a unique web service in XML and transformed offline into static files. I'm looking forward to seeing how they go.
Are those all the pages of your site (from the last monthly deep crawl)?And are you getting complete deep crawls every month?
Yes thats the lot, the last deep crawl was 6th July
I've often wondered when this topic has come up, at the content found at these sites. Are there sites with more than 10,000 pages that are more than just catalogues or well constructed forums? Mom & Pop's, not .edu's, news sites, etc. Just curious...
Example would be a restaurant site, listing all restaurants in one country.. well those that pay you!
1 page per restaurant, hence 23,500 pages
* Minus four is my guess based on the number of links from a low PR1 page that Google will follow.
static 'looking' URL's
Yeah that has definitely been the key for me getting the sites with thousands of pages listed. I've not yet produced a site that Googlebot doesn't completely deep crawl.
1 page per restaurant, hence 23,500 pages
Nice example... I'm off to look for a list of restaurants ;)
It's clear to me that the home page PageRank not only limits the depth of the crawl, but on sites with many thousands of pages, these pages end up with a PR of zero or one. This is not a penalty, it's just that there's not enough PR to start with on the home page.
Yes, PR is per page, not per site. But consider a site with thousands of pages. What are the chances that deep pages are linked from outside? When they are, on rare occasions, it's true that they enjoy all the benefits of an increased PR due to one or more additional links.
Barring links from the outside, additional cross-linking within a site between its thousands of pages helps only a little. When you only have seven-tenths of a tub of margarine, it only spreads so far. I've tried it both ways. I think strong internal cross-linking on a site is, on balance, a good idea. But it doesn't help the internal PR much, if at all.
When internal pages aren't linked from outside, their PR derives from the home page. In that sense, it does become meaningful to speak about the importance of PR on the home page.
I speak for a site that gets over 90,000 Googlebot hits per month, with a home page PR of 7. I've been watching the crawl closely for nearly two years. My PR of low 7 or high 6 has been constant ever since the toolbar became available 18 months ago.
The depth of the crawl doubled once I made the transition from all-dynamic to all-static pages. I believe that this is due to the fact that Google can now crawl a lot faster and get deeper into the queue. My static pages are really static, not dynamic pages with static-looking URLs, and our bandwidth is adequate for Google -- which can crawl as fast as 10 hits per second.
On the other hand, a crawl that's twice as deep means my constant home page PR gets spread thinner. So the benefit, while real, is not terribly important.
It's also possible, I suspect, that Google does human eyeballing of sites with this many pages, and gives them a thumbs up or down for deep crawling, within the limits of their crawling schedule. This isn't a PageRank boost, but nevertheless, a tweak is a tweak by any other name. (While Google never messes with PageRank, it's ultimately meaningless to insist on the sanctity of PageRank when Google will, on occasion, impose penalties that deprive a site of PageRank consideration altogether.)
The next logical step is to tweak PageRank on a per-site basis, if Google isn't doing this already. Rather than a thumbs-up or thumbs-down for a site, you'd derive criteria for scaling the extent to which large sites are able to pass PageRank through to internal pages.
It's possible that this is happening already. Something strange is going on at a per-site level rather than a per-page level. For about three months now, a search for my name without quotes turns up my main site at number one in the SERPs out of 131,000. Now you may be suitably impressed and think that I'm "feeling lucky" over this, but consider further -- my name has never, since 1995, been on my home page in a form that Google can crawl. It's in a staff box inside of a GIF, which is also where I like to keep the email address so that it can't be harvested.
Google has made some sort of connection between internal pages and the home page. It would not surprise me if PageRank gets adjusted in the same manner. If not now, then soon.
looks like a reasonable PR can give tremendous milage if deep-crawling is concerned. Static links and a logical structure, possibly preventing Ciml's minus four suggestion.
Still makes me wonder why WebmasterWorld has relatively few (4890) of its threads indexed?
I'm interested in the case where, in a huge site, Google stops spidering where PageRank is higher than zero, the URLs look static and there are no frames.
But next crawl will be interesting - adding another 10k next weekend :)
please googlebot - stay away until they are loaded ;)
Thanks,
Linda
/cgi-bin/script.pl?category=people&page=linda is a dynamic looking url.
are your sites "regular sites" where you create the content, that is, not forums or sites where people can add their input themselves?
In this thread Grumpus has some findings for his large site.
[webmasterworld.com...]
His site seems to be a member fill in site as a forum would as well.
Could it just be that Google decides that forum type sites get less deep spidering/indexing (such as WebmasterWorld) unless there is some deeplinking or interlinking?
