homepage Welcome to WebmasterWorld Guest from 54.211.157.103
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
How Can I Get Google to Index all Pages of a Site?
Fortune Hunter




msg:3106783
 1:46 pm on Oct 3, 2006 (gmt 0)

I have been fighting with Google for almost 8 months now trying to get them to simply index all the pages of my site and I have to say I am losing this battle. I am not sure what else I need to do.

I have hand submitted every single page of the site to their site submission. I have created a Google approved XML site map with the software from Coffee cup software site and according to Google analytics it has accepted my site map. I have insured every page has a link to it that I want indexed from the home page. All pages are optimized with content, titles, etc. I have a regular site map (HTML) that has been created and hand submitted to them. I even re-submitted (by hand) all the pages multiple times over the course of months.

Despite all of this effort Google has only indexed about 3 pages out of 50, but has grabbed almost every PDF file on the site as a "supplemental" result. What am I missing here to get them to index every [HTML] page, I will worry where I am placing in the index once all the pages are actually indexed.

Any thoughts would be appreciated.

Fortune Hunter

 

Fortune Hunter




msg:3106791
 1:53 pm on Oct 3, 2006 (gmt 0)

I forgot to mention in my initial post that Yahoo has indexed almost every single page of the site. There are a few it isn't getting, but I would estimate it has over 80% indexed and most of the PDFs as well.

I don't think I have even worked at Yahoo nearly as hard as Google either. I think I simply went to their site and submitted my site map and home page links and forgot about it. Go figure.

Fortune Hunter

[edited by: jatar_k at 3:44 pm (utc) on Oct. 3, 2006]

Fortune Hunter




msg:3113063
 2:53 pm on Oct 8, 2006 (gmt 0)

Any thoughts on this from anyone? Also I was wondering if anyone can tell me what "supplemental results" mean in Google as opposed to normal results. Second why does Google appear to favor the PDF rather than the HTML pages

Fortune Hunter

Car_Guy




msg:3113083
 3:28 pm on Oct 8, 2006 (gmt 0)

Google explains all this if you go to google.com and click on "About Google", then on "Webmaster Central", then on "Webmaster tools (including Sitemaps)".

Here's one of the threads on here you can find by using the site search at the top of the page:

Supplemental Results: What exactly are they
[webmasterworld.com...]

tedster




msg:3113114
 4:18 pm on Oct 8, 2006 (gmt 0)

I have hand submitted every single page of the site to their site submission... I don't think I have even worked at Yahoo nearly as hard as Google either.

Hand submission is relatively futile. Google needs to see links from other domains. Your URLs only get into the index and stay there if your domain begins to show some signs that Google feels build "trust". Here's a related thread: Filters exist - the Sandbox doesn't. How to build Trust. [webmasterworld.com]

traffik daddy




msg:3113173
 5:06 pm on Oct 8, 2006 (gmt 0)

I rarely submit to Google, I get quality inbound links and let Googlebot find my site.

TD

BigDave




msg:3113190
 5:21 pm on Oct 8, 2006 (gmt 0)

I have been fighting with Google for almost 8 months now trying to get them to simply index all the pages of my site and I have to say I am losing this battle. I am not sure what else I need to do.

You can start by changing your attitude from "fighting" and "battle".

You can't force Google, or anyone else to index or rank your site. You can only encourage them to do it in their own good time.

Speaking of time, you don't mention how old your site is. That is very important with Google.

What I usually tell people is the first week or so, you should expect to get every page that has a reputable link from outside your site. If your only external links are to your home page, that is the only page to make it in quickly.

At about a month, you will get the pages that are linked from those pages with external links, unless your pages are loaded with too many links, which can slow things down a little.

Most sites with under 1000 pages will be indexed within 3 months.

The Google Sitemaps help speed things up a little, but they do not force Google to go any faster than Google wants. It does not force them to do anything, it only encourages them.

So, if your site is less than 3 months old, stop worrying about it. Work on your content and getting links, especially deep links.

Essex_boy




msg:3113271
 7:24 pm on Oct 8, 2006 (gmt 0)

Try, this is what works for me:

Internal sitemap, out bound links from from internal pages to other sites (widens the appeal to the spider and shows some depth of research), inbound links from DMOZ and other ontopic sites. This is really worth doing and pays off many times over. Do you have meta tags and descriptions?

Check your site has no duplicate content. Then wait

I find it takes 6 - 8 months to be spidered entirly.

trinorthlighting




msg:3113280
 7:38 pm on Oct 8, 2006 (gmt 0)

If you want your pages indexed, start using adsense and put it on every page. We did and every one of our pages started to index. It took a few months though.

lordvader




msg:3115367
 12:05 pm on Oct 10, 2006 (gmt 0)

Hello.

I have a couple of sites that have been running for about a year using a CMS I built myself in ASP. One of the sites does not seem to be indexed beyond the home page, the other seems to be reasonably well indexed.

Firstly the sites

Example1.com - this is the site that is not being indexed)
Example2.com - this is well indexed

If you look at example1.com, you will see that google sees the home page and 3 old links.

site:www.example2.com/ - Google has numerous links

Both sites (main page) have a PR of 2 according to googlebar, but nothing elsewhere.

Checking links via a search tool
<specific tool removed>

I used the above tool for both sites and the URLs appear to be well-formed

Sitemap
Both sites have a sitemap that is reasonably up to date and valid.

The logs
Having just looked at the web logs on october 2nd (when the cache for exmple1.com was updated) I have the following lines

2006-10-02 04:13:29 W3SVC1497435083 10.216.12.64 GET /robots.txt - 80 - 66.249.72.241 HTTP/1.1 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - - 404 0 2 1795 246 93
2006-10-02 04:13:30 W3SVC1497435083 10.216.12.64 GET /default.asp - 80 - 66.249.72.241 HTTP/1.1 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - - 200 0 0 10109 269 468

404 = Not found (correct I haven't got one)
200 = OK. So page returned OK.

Can anyone suggest to me why one site has been reasonably read by google and the other has been largely ignored.

Given that the software system is the same on both sites and they have existed similar length of time, have some similar incoming links and a similar page rank, what am I missing?

Yahoo and MSN have both indexed lots of pages on each site for quite a while now, this suggests a factor other than the software (ie. it confirms to me that the links my CMS generates are perfectly acceptable for a search engine).

Any suggestions greatfully recieved, I have tried to be thorough and scientific about this so that we can narrow down the issues.

[edited by: tedster at 6:17 pm (utc) on Oct. 10, 2006]
[edit reason] use example.com [/edit]

Adam_Lasnik




msg:3118684
 6:51 pm on Oct 12, 2006 (gmt 0)

If you want your pages indexed, start using adsense and put it on every page. We did and every one of our pages started to index. It took a few months though.

As much as we'd love to have all Webmasters with fine sites using AdSense, I've got to respectfully defuse this rumor. Putting AdSense on your pages is about as likely to cause your pages to get indexed or well-ranked as using the GooglePray metatag or wearing purple on full-moon days.

Fortune Hunter




msg:3120440
 9:36 pm on Oct 13, 2006 (gmt 0)

Thanks for the great responses. Some of this I discovered after I made my intial post and started combining different threads. I keep seeing a common theme about building incoming links.

I knew that was supposed to work to boost rankings, but I didn't realize it was also required just to get pages indexed in the first place.

Fortune Hunter

whitenight




msg:3120462
 10:02 pm on Oct 13, 2006 (gmt 0)

As much as we'd love to have all Webmasters with fine sites using AdSense, I've got to respectfully defuse this rumor. Putting AdSense on your pages is about as likely to cause your pages to get indexed or well-ranked as using the GooglePray metatag or wearing purple on full-moon days.

Wow, then what was all that gooblety-gook MC was talking about when mentioning BD new use of adsense bots to index..err cache pages?

I'm missing something here... (and we aren't talking about "ranking"...just indexing)

Whitey




msg:3120484
 10:20 pm on Oct 13, 2006 (gmt 0)

Are you free from supplementals and have you got all of your pages pointing to "/"? per [webmasterworld.com...]

This has been the biggest inhibiter to proper indexing that i know of in the recent months.

If you've done that, re indexing may be very slow. 8 months sounds like a bit of a worry, although several people appear to having probs, inclusing ourselves.

Whitey




msg:3120485
 10:21 pm on Oct 13, 2006 (gmt 0)

Are you free from supplementals and have you got all of your pages pointing to "/"? per [webmasterworld.com...]

This has been the biggest inhibiter to proper indexing that i know of in the recent months.

If you've done that, re indexing may be very slow. 8 months sounds like a bit of a worry, although several people appear to having probs, including ourselves.

goubarev




msg:3120615
 1:11 am on Oct 14, 2006 (gmt 0)

Hey Fortune Hunter, here is the simple answer - get links to your internal pages from other sites. Google will list them in no time...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved