Why hasn't Google indexed all of my pages?

post from a newer webmaster

[dellbert9111]

Hi all,

I am new to the search engine world. I created a site a little while ago and have some of the pages indexed but not all of them. I have about 35,000 pages on my site. It's a community site/blog site.

The URLs don't look dynamic. Google has spidered my site on six separate occassions and each time does about 1442 pages on day and 443 the next day. It seems to happen about every 10 days or so. The interesting part is it looks like Googlebot doesn't go any deeper into my site on each successive spider. When I search for my site on Google there about 3000 entries.

Any comments on how I can get Googlebot to spider my site more frequently and spider deeper into my site would be greatly appreciated!

Thanks and have a great weekend.

[Hanu]

Increase your PR. There are rumors that high PR sites are crawled deeper and more often.

[Ledfish]

dellbert

I'm wondering the same type of thing, actually I'm trying to understand what the max is per PR. I have a site with about 4000 pages, but it has been stuck at only having about 900 pages indexed so far.

[dellbert9111]

Thanks for the responses - I have heard the PR rumor and am working on increasing the rank. The best way to do this is by getting sites to link to my site correct? Higher ranked sites are better too.

I have another question, does it matter if my home page is "stale" would it help if I change the page by adding/changing links, content, etc.

[AthlonInside]

35,000 pages is a lot and it take time (IF only google wants them) to crawl all of them.

How to you link to these pages? Do a spider need to fetch 10 links down from your domain before it can reach that page? The more steps a spider need to take to reach a page, the more easy it will give up.

Do you use session IDs? Spiders don't like session IDs?

Are you pages in dynamic URLs? Spiders will crawl only a small number of dynamic URLs to protect them from crawling too much useless/similar/identical pages.

Again, 35,000 pages is a LARGE number. I thought you were building DMOZ or Yahoo! :)

[johnwards]

I've never look at how many pages google has had of my site in the db before...but if it helps.

My index has a PR of 5 and some of my internal pages have PRs of 3-5.

I think I have about 40,000 pages of info and about 500 message boards but these don't get indexed.

I have done a

allinurl: www.domain.com site:www.domain.com

and according to that I have 44,300 pages within google. Which is nice as that would nearly be everything!

My PR a few months ago was 6 or 7 so i dunno if thats helped.

John

[pcgamez]

you all are doing better than me, I have a ~45,000 page site that has ~50 hits by google in the last 2 weeks and only 300 links on site:www.domain.com (and all but a couple of those are broken).

probably due to the fact that the site is all php/mysql.

from my logs for the past 36 hours:

Links from an Internet Search Engine - Full list
- MSN 78 78
- Unknown search engines 6 6
- Looksmart 4 4
- search.bluewin.ch 2 2
- Web.de 2 2
- Google 1 1
93 2.9 % 93 0.9 %
Links from an external page (other

I have no idea why MSN loves me so much

[sidyadav]

I have no idea why MSN loves me so much

MSN doesn't - Inktomi does.

---
MSN uses Ink's search results
---

Sid

[pcgamez]

duh, but anyway..