Welcome to WebmasterWorld Guest from 54.224.158.39

Forum Moderators: Robert Charlton & andy langton & goodroi

Message Too Old, No Replies

Most of my pages are in the Supplemental Index - why?

     
1:08 pm on May 31, 2006 (gmt 0)

New User

10+ Year Member

joined:May 31, 2006
posts:7
votes: 0


Hi all,
I seem to have a problem with my site being normally indexed by google but I can't understand the reasons.
if I search site:mysite.com I see that only my homepage and my RSS feed are indexed regularly, other 80-90 pages are Supplemental Results and the other 500+ of my site are just missing.

I understand that those informations are not enought to help me, but since I can't understand at all why this happened I really can't tell much else.

Mmmhhh..... help? Would be appreciated :)

1:42 am on June 1, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2004
posts:693
votes: 0


sounds like you may have a crawling problem. If googlebot is unable to crawl the site then pages will start dropping into supplemental.
check your site with 'poodle predictor' and see if you can follow links to some of the troubled pages.
Also check your robots.txt file to see if there is some possible problem there.
1:44 am on June 1, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2004
posts:693
votes: 0


oh yeah If you find the solution or not, please update this thread.
3:34 am on June 1, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 5, 2006
posts:2095
votes: 2


Or possibly a duplicate content issue.
8:11 am on June 1, 2006 (gmt 0)

New User

10+ Year Member

joined:May 31, 2006
posts:7
votes: 0


The website is quite treestyle, with no flash at all and almost no javascript. A text browser and poodle predictor can easly browse it, with the exception of the "email to a friend" links on each article.

In 500+ pages, I know I have 13 pages that are linked in two different ways inside the site. This is an issue of the CMS I am using (Joomla), but I know about other sites using the same CMS and no big Google index problems.
Since I noticed I had some probles with google, I tried to solve this possible duplicated content problem by banning duplicate URLs on my robots.txt
So I had the same page linked both as [---------.com...] and [---------.com...] and put on robots.txt a Disallow on the second URL.
After a week I removed the rules because I understood it would be better to put in act a 301 redirect, wich I still have to do.
Anyway I had the indexing problem before even touching my robots.txt

Another potential duplicate content problem could be that my homepage and my RSS feed, both indexed, looks very similar. I suppose anyway that google can recognize that they are different things.

Another note, maybe important, on my homepage I currently have 4 outbound external links and 57 internals. Sometimes I could even have less externals.
I have a bunch of inbound links pointing everywhere on my site and no link exchange.

Of course if someone wants to give a look at my website just send me an email.

9:29 pm on June 1, 2006 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Oct 24, 2003
posts:590
votes: 4


My pages all went supplemental too...I did notice something strange...when I do a site:www.mysite.com search in google my site comes up as having 209 pages, down from around 1500 before march. When I run a site:mysite.com search 1,200 pages show up. This is despite the fact that I have rewrite code in my htaccess file to force Google to index only "www". The worst thing is that most of the 1,200 pages indexed as non-"www" are now classified as supplemental. Has anyone else seen this happen?

[edited by: ichthyous at 9:34 pm (utc) on June 1, 2006]

9:32 pm on June 1, 2006 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Oct 24, 2003
posts:590
votes: 4


here is the actual code in my htaccess file:


RewriteEngine On
RewriteCond %{HTTP_HOST} "!^www\.mysite\.com"
RewriteRule (.*) http://www.mysite.com/$1 [R=301,L]

Does anyone know if this is causing problems these days?

3:31 am on June 2, 2006 (gmt 0)

Moderator from AU 

WebmasterWorld Administrator anallawalla is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 3, 2003
posts:3744
votes: 13


There are a few threads about supplemental pages in the Google Forum. Some were apparently a glitch and were meant to be fixed. The ones that still remain supplemental are most likely Thin Affiliates or other types of sites with duplicate content. If this applies to your site, don't expect to be unsupplementalled anytime soon, as it is a feature, not a bug.
5:10 am on June 2, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2004
posts:693
votes: 0


don't expect to be unsupplementalled anytime soon

unsupplementalled I think we have a new e-word to add to the dictionary.
4:53 pm on June 3, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 29, 2005
posts:866
votes: 0


>>>>The ones that still remain supplemental are most likely Thin Affiliates or other types of sites with duplicate content. <<<

absolutely not true. I have two clients sites that are in no way affiliates or duplication problems and are suffering from this.

5:56 pm on June 3, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 22, 2003
posts:1230
votes: 0


Gotta agree with Texasville. I've got totally unique pages full of content, that hasn't been scraped or duplicated yet, that went supplemental. This content isn't churn out a page a day type stuff.

The only thing I see is the cache dates go back to July of 2005 like some people have mentioned. Basically it seems like anything I turned out since August 2005 has gone supplemental.

7:26 pm on June 3, 2006 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Oct 24, 2003
posts:590
votes: 4


I also agree...none of my pages have anything to do with affiliates...duplicate content possibly as i have the same product entered several times in the store's database. A more likely scenario is that Google has just simply had to reallocate it's indexing resources to higher page rank pages...any page that isn't highly ranked is wiped. Why they even bother to replace them with old supplemental crap from 2005 is beyond me. Perhaps we'll get lucky and see these pages reindexed after a very long while, but I doubt it. I for one have taken to pulling apart my current site completely and reorganizing the content and getting away from dynamic URLS. Now that 85% of my traffic has vanished I have all the time in the world to rebuild!
8:25 pm on June 4, 2006 (gmt 0)

New User

10+ Year Member

joined:May 27, 2006
posts:10
votes: 0


To take a shot at answering the OP's original question (as no one but Google really knows), assuming the site is not being penalized for any black hat activity, here are the simplistic and general reasons your pages are supplemental.

1. At one time your pages were indexed and since Google has changed it aglo they no longer rate being in the active index. If you don't change anything these supplemental pages will eventually go away.

2. The reason the pages were changed in status from actively indexed to supplemental is because your site's real page rank is too low for the distance (the number of clicks) these pages are away from your home page for Google new algo.

3. If you want to change this, either increase your page rank or reduce the number of clicks away from the home page.

4. Or you can wait for Google to change their algo. They do it constantly and I'm sure they will do it again.

7:33 am on June 7, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2004
posts:693
votes: 0


You could be onto something there sjwinslow, I have a PR4 site that was unaffected by the supplemental problem, maybe it's because it has a sitemap. any page is 2 clicks from home.
8:35 pm on June 10, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


You have done the 301 so you will soon see that the number of www pages indexed will increase.

Make sure that the response code for HTTP/1.1 accesses to non-www really is 301. Use WebBug, or similar, to check it.

Run Xenu LinkSleuth over the site and make sure that all internal links always point to www pages. Make sure that there are no links to any non-www URLs anywhere within the site.

The non-www pages may well take several years to drop out of the index, but don't worry about that. They are already marked as being Supplemental and are not causing any problems to you. If they ever appear in search results, then visitors will be redirected to the correct www page automatically by your redirect. That is what you want to happen.

5:06 pm on June 11, 2006 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Oct 24, 2003
posts:590
votes: 4


Well, I recently simplified my directory structure in my online store to remove subcategories, now all of the products are one click away from the home page in their respective categories. I am hoping this somehow kick starts a reindexing. The real problem from what I see is that Google's index seems to be frozen. I have implemented some major changes in some of my pages, including pulling whole categries by inseting noindex meta tags...not one has been dropped from Google's index yet. Only my home page and a few top level pages are crawled anymore, and that's why no changes are reflected. I hope that this is temporary bug on Google's side, but i somehow doubt it.
5:38 pm on June 11, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 9, 2006
posts:394
votes: 0


"Run Xenu LinkSleuth over the site and make sure that all internal links always point to www pages. Make sure that there are no links to any non-www URLs anywhere within the site."

My internal pages do not link to wwww.mysite.com/internalpages.html.. It links to to /internalpages.html, (w/o the domain name) could that be a problem?

6:00 pm on June 11, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


If you are on a www page than that URL links to a www page.

If you are on a non-www page then that page links to a non-www page.

Make sure that you have the 301 redirect from non-www to www in place so that you cannot access anything directly at the non-www locations.

7:18 pm on June 11, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 31, 2004
posts:43
votes: 0


just thought i'd add my 2 cents on where my (small) domains are.

1st) 791 in index, down from +/- 1600 and still dropping. all but 1st 13 return supplemental on site:

2) 73 in index. all supplemental but main page.

the 1st, 4 years old, the 2nd 1 year+ old.

301s in place, don't have that problem, xenu run and ok, use that to assist generate site maps.

these are obviously very small sites vis. sites being run by the folks here. however, up until about last September, the 1,600 page site was getting about 6,000 googlebot hits a month. currently just 500 a month with more pages on site. all original content.

8:33 pm on June 11, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 25, 2005
posts:677
votes: 0


I guess its because of duplicate content
2:30 am on June 12, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 9, 2006
posts:394
votes: 0


How can I test if Google bots are not having trouble indexing our site?

Our site has lots of text links on the home page. We have site map as well.

The site was designed with lots of text links for the crawlers to get through easily..

However, we currently have load of supplemental results and very few indexed pages..

So, again is thier a way I can check if the Google bots are having trouble crawling our site?

11:14 am on June 12, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 31, 2004
posts:43
votes: 0


>>wmuser<<

I guess its because of duplicate content

well, i guess anything google these days is possible, but if it's duplicate content i cannot imagine where it's coming from. it's all original content. it's a small 1 man site that sells books. some terms naturally may occur in more than one listing on a page, such as "page" or "history" or whatever. the only duplication i can possibly see is the info on the "product detail" page for 1 product is also in the listing pages of 10 items per page.

12:01 pm on June 12, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 31, 2002
posts:353
votes: 0


So nobody really knows and Google is not talking
1:03 pm on June 12, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 9, 2006
posts:394
votes: 0


So, again is thier a way I can check if the Google bots are having trouble crawling our site?
1:18 pm on June 12, 2006 (gmt 0)

New User

10+ Year Member

joined:May 31, 2006
posts:7
votes: 0


F_Rose, if you check the first answer I got on my original post you will see that I have been suggested to check that using 'poodle predictor'
1:55 pm on June 12, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Sept 20, 2005
posts:339
votes: 0


It seems that if you do a site:mydomain.com/ search the Supplementals dosappear [ don't forget the trailing slash ]

Apparently, Google are working on this. Someone posted about this before, but I don't have time to look it up.

2:43 pm on June 12, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 9, 2006
posts:394
votes: 0


"It seems that if you do a site:mydomain.com/ search the Supplementals dosappear [ don't forget the trailing slash ]"

Not working for us-supplemental results-for / as well.

8:12 pm on June 12, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Run your site through Xenu LinkSleuth to check that it can be spidered.

Look closely at the error messages, the reports, and the generated sitemap.

3:39 pm on June 13, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 9, 2006
posts:394
votes: 0


g1smd,

I know that the browser automatically redirects to a / at the end of every domain name..

Do we have to have a 301 redirect from www.site.com to www.site.com/? Or does Google dedect that automatically?

11:33 pm on June 13, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Pfffft. I don't know the answer to that one. I see the / silently appear on just about every website that I try to access as either domain.com or as www.domain.com, and I always link to URLs that include the / at the end.
This 32 message thread spans 2 pages: 32
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members