homepage Welcome to WebmasterWorld Guest from 23.23.12.202
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 32 message thread spans 2 pages: 32 ( [1] 2 > >     
Most of my pages are in the Supplemental Index - why?
Mantees

5+ Year Member



 
Msg#: 34566 posted 1:08 pm on May 31, 2006 (gmt 0)

Hi all,
I seem to have a problem with my site being normally indexed by google but I can't understand the reasons.
if I search site:mysite.com I see that only my homepage and my RSS feed are indexed regularly, other 80-90 pages are Supplemental Results and the other 500+ of my site are just missing.

I understand that those informations are not enought to help me, but since I can't understand at all why this happened I really can't tell much else.

Mmmhhh..... help? Would be appreciated :)

 

Reid

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 34566 posted 1:42 am on Jun 1, 2006 (gmt 0)

sounds like you may have a crawling problem. If googlebot is unable to crawl the site then pages will start dropping into supplemental.
check your site with 'poodle predictor' and see if you can follow links to some of the troubled pages.
Also check your robots.txt file to see if there is some possible problem there.

Reid

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 34566 posted 1:44 am on Jun 1, 2006 (gmt 0)

oh yeah If you find the solution or not, please update this thread.

trinorthlighting

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 34566 posted 3:34 am on Jun 1, 2006 (gmt 0)

Or possibly a duplicate content issue.

Mantees

5+ Year Member



 
Msg#: 34566 posted 8:11 am on Jun 1, 2006 (gmt 0)

The website is quite treestyle, with no flash at all and almost no javascript. A text browser and poodle predictor can easly browse it, with the exception of the "email to a friend" links on each article.

In 500+ pages, I know I have 13 pages that are linked in two different ways inside the site. This is an issue of the CMS I am using (Joomla), but I know about other sites using the same CMS and no big Google index problems.
Since I noticed I had some probles with google, I tried to solve this possible duplicated content problem by banning duplicate URLs on my robots.txt
So I had the same page linked both as [---------.com...] and [---------.com...] and put on robots.txt a Disallow on the second URL.
After a week I removed the rules because I understood it would be better to put in act a 301 redirect, wich I still have to do.
Anyway I had the indexing problem before even touching my robots.txt

Another potential duplicate content problem could be that my homepage and my RSS feed, both indexed, looks very similar. I suppose anyway that google can recognize that they are different things.

Another note, maybe important, on my homepage I currently have 4 outbound external links and 57 internals. Sometimes I could even have less externals.
I have a bunch of inbound links pointing everywhere on my site and no link exchange.

Of course if someone wants to give a look at my website just send me an email.

ichthyous

10+ Year Member



 
Msg#: 34566 posted 9:29 pm on Jun 1, 2006 (gmt 0)

My pages all went supplemental too...I did notice something strange...when I do a site:www.mysite.com search in google my site comes up as having 209 pages, down from around 1500 before march. When I run a site:mysite.com search 1,200 pages show up. This is despite the fact that I have rewrite code in my htaccess file to force Google to index only "www". The worst thing is that most of the 1,200 pages indexed as non-"www" are now classified as supplemental. Has anyone else seen this happen?

[edited by: ichthyous at 9:34 pm (utc) on June 1, 2006]

ichthyous

10+ Year Member



 
Msg#: 34566 posted 9:32 pm on Jun 1, 2006 (gmt 0)

here is the actual code in my htaccess file:


RewriteEngine On
RewriteCond %{HTTP_HOST} "!^www\.mysite\.com"
RewriteRule (.*) http://www.mysite.com/$1 [R=301,L]

Does anyone know if this is causing problems these days?

anallawalla

WebmasterWorld Administrator anallawalla us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34566 posted 3:31 am on Jun 2, 2006 (gmt 0)

There are a few threads about supplemental pages in the Google Forum. Some were apparently a glitch and were meant to be fixed. The ones that still remain supplemental are most likely Thin Affiliates or other types of sites with duplicate content. If this applies to your site, don't expect to be unsupplementalled anytime soon, as it is a feature, not a bug.

Reid

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 34566 posted 5:10 am on Jun 2, 2006 (gmt 0)

don't expect to be unsupplementalled anytime soon

unsupplementalled I think we have a new e-word to add to the dictionary.

texasville

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 34566 posted 4:53 pm on Jun 3, 2006 (gmt 0)

>>>>The ones that still remain supplemental are most likely Thin Affiliates or other types of sites with duplicate content. <<<

absolutely not true. I have two clients sites that are in no way affiliates or duplication problems and are suffering from this.

outland88

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 34566 posted 5:56 pm on Jun 3, 2006 (gmt 0)

Gotta agree with Texasville. I've got totally unique pages full of content, that hasn't been scraped or duplicated yet, that went supplemental. This content isn't churn out a page a day type stuff.

The only thing I see is the cache dates go back to July of 2005 like some people have mentioned. Basically it seems like anything I turned out since August 2005 has gone supplemental.

ichthyous

10+ Year Member



 
Msg#: 34566 posted 7:26 pm on Jun 3, 2006 (gmt 0)

I also agree...none of my pages have anything to do with affiliates...duplicate content possibly as i have the same product entered several times in the store's database. A more likely scenario is that Google has just simply had to reallocate it's indexing resources to higher page rank pages...any page that isn't highly ranked is wiped. Why they even bother to replace them with old supplemental crap from 2005 is beyond me. Perhaps we'll get lucky and see these pages reindexed after a very long while, but I doubt it. I for one have taken to pulling apart my current site completely and reorganizing the content and getting away from dynamic URLS. Now that 85% of my traffic has vanished I have all the time in the world to rebuild!

sjwinslow

5+ Year Member



 
Msg#: 34566 posted 8:25 pm on Jun 4, 2006 (gmt 0)

To take a shot at answering the OP's original question (as no one but Google really knows), assuming the site is not being penalized for any black hat activity, here are the simplistic and general reasons your pages are supplemental.

1. At one time your pages were indexed and since Google has changed it aglo they no longer rate being in the active index. If you don't change anything these supplemental pages will eventually go away.

2. The reason the pages were changed in status from actively indexed to supplemental is because your site's real page rank is too low for the distance (the number of clicks) these pages are away from your home page for Google new algo.

3. If you want to change this, either increase your page rank or reduce the number of clicks away from the home page.

4. Or you can wait for Google to change their algo. They do it constantly and I'm sure they will do it again.

Reid

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 34566 posted 7:33 am on Jun 7, 2006 (gmt 0)

You could be onto something there sjwinslow, I have a PR4 site that was unaffected by the supplemental problem, maybe it's because it has a sitemap. any page is 2 clicks from home.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34566 posted 8:35 pm on Jun 10, 2006 (gmt 0)

You have done the 301 so you will soon see that the number of www pages indexed will increase.

Make sure that the response code for HTTP/1.1 accesses to non-www really is 301. Use WebBug, or similar, to check it.

Run Xenu LinkSleuth over the site and make sure that all internal links always point to www pages. Make sure that there are no links to any non-www URLs anywhere within the site.

The non-www pages may well take several years to drop out of the index, but don't worry about that. They are already marked as being Supplemental and are not causing any problems to you. If they ever appear in search results, then visitors will be redirected to the correct www page automatically by your redirect. That is what you want to happen.

ichthyous

10+ Year Member



 
Msg#: 34566 posted 5:06 pm on Jun 11, 2006 (gmt 0)

Well, I recently simplified my directory structure in my online store to remove subcategories, now all of the products are one click away from the home page in their respective categories. I am hoping this somehow kick starts a reindexing. The real problem from what I see is that Google's index seems to be frozen. I have implemented some major changes in some of my pages, including pulling whole categries by inseting noindex meta tags...not one has been dropped from Google's index yet. Only my home page and a few top level pages are crawled anymore, and that's why no changes are reflected. I hope that this is temporary bug on Google's side, but i somehow doubt it.

F_Rose

5+ Year Member



 
Msg#: 34566 posted 5:38 pm on Jun 11, 2006 (gmt 0)

"Run Xenu LinkSleuth over the site and make sure that all internal links always point to www pages. Make sure that there are no links to any non-www URLs anywhere within the site."

My internal pages do not link to wwww.mysite.com/internalpages.html.. It links to to /internalpages.html, (w/o the domain name) could that be a problem?

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34566 posted 6:00 pm on Jun 11, 2006 (gmt 0)

If you are on a www page than that URL links to a www page.

If you are on a non-www page then that page links to a non-www page.

Make sure that you have the 301 redirect from non-www to www in place so that you cannot access anything directly at the non-www locations.

nickied

10+ Year Member



 
Msg#: 34566 posted 7:18 pm on Jun 11, 2006 (gmt 0)

just thought i'd add my 2 cents on where my (small) domains are.

1st) 791 in index, down from +/- 1600 and still dropping. all but 1st 13 return supplemental on site:

2) 73 in index. all supplemental but main page.

the 1st, 4 years old, the 2nd 1 year+ old.

301s in place, don't have that problem, xenu run and ok, use that to assist generate site maps.

these are obviously very small sites vis. sites being run by the folks here. however, up until about last September, the 1,600 page site was getting about 6,000 googlebot hits a month. currently just 500 a month with more pages on site. all original content.

wmuser

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 34566 posted 8:33 pm on Jun 11, 2006 (gmt 0)

I guess its because of duplicate content

F_Rose

5+ Year Member



 
Msg#: 34566 posted 2:30 am on Jun 12, 2006 (gmt 0)

How can I test if Google bots are not having trouble indexing our site?

Our site has lots of text links on the home page. We have site map as well.

The site was designed with lots of text links for the crawlers to get through easily..

However, we currently have load of supplemental results and very few indexed pages..

So, again is thier a way I can check if the Google bots are having trouble crawling our site?

nickied

10+ Year Member



 
Msg#: 34566 posted 11:14 am on Jun 12, 2006 (gmt 0)

>>wmuser<<

I guess its because of duplicate content

well, i guess anything google these days is possible, but if it's duplicate content i cannot imagine where it's coming from. it's all original content. it's a small 1 man site that sells books. some terms naturally may occur in more than one listing on a page, such as "page" or "history" or whatever. the only duplication i can possibly see is the info on the "product detail" page for 1 product is also in the listing pages of 10 items per page.

Jakpot

10+ Year Member



 
Msg#: 34566 posted 12:01 pm on Jun 12, 2006 (gmt 0)

So nobody really knows and Google is not talking

F_Rose

5+ Year Member



 
Msg#: 34566 posted 1:03 pm on Jun 12, 2006 (gmt 0)

So, again is thier a way I can check if the Google bots are having trouble crawling our site?

Mantees

5+ Year Member



 
Msg#: 34566 posted 1:18 pm on Jun 12, 2006 (gmt 0)

F_Rose, if you check the first answer I got on my original post you will see that I have been suggested to check that using 'poodle predictor'

Eazygoin

5+ Year Member



 
Msg#: 34566 posted 1:55 pm on Jun 12, 2006 (gmt 0)

It seems that if you do a site:mydomain.com/ search the Supplementals dosappear [ don't forget the trailing slash ]

Apparently, Google are working on this. Someone posted about this before, but I don't have time to look it up.

F_Rose

5+ Year Member



 
Msg#: 34566 posted 2:43 pm on Jun 12, 2006 (gmt 0)

"It seems that if you do a site:mydomain.com/ search the Supplementals dosappear [ don't forget the trailing slash ]"

Not working for us-supplemental results-for / as well.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34566 posted 8:12 pm on Jun 12, 2006 (gmt 0)

Run your site through Xenu LinkSleuth to check that it can be spidered.

Look closely at the error messages, the reports, and the generated sitemap.

F_Rose

5+ Year Member



 
Msg#: 34566 posted 3:39 pm on Jun 13, 2006 (gmt 0)

g1smd,

I know that the browser automatically redirects to a / at the end of every domain name..

Do we have to have a 301 redirect from www.site.com to www.site.com/? Or does Google dedect that automatically?

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34566 posted 11:33 pm on Jun 13, 2006 (gmt 0)

Pfffft. I don't know the answer to that one. I see the / silently appear on just about every website that I try to access as either domain.com or as www.domain.com, and I always link to URLs that include the / at the end.

This 32 message thread spans 2 pages: 32 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved