Not being fully indexed, Who is at fault?

Google or Webmaster

[F_Rose]

I need to understand this..

Google is only indexing a very small portion of our site...

Matt Cutts is claiming that BD is over..The reason websites are having trouble with supplemental results and not being indexed is due to spam..

Reading webmasterworld for the past few weeks, everyone claimed everything is Google's fault, just be patient with your site..Google needs to recover..

What is your opinion now..Not being fully indexed means that our site has duplicate or some other spam issues?

[jackson992]

I've got pages that still aren't indexed that I put up beginning of last month and the links are right on the home page. There is something seriously wrong at Google.

[SEOPimp]

Hey Jackson, do you have Goole Sitemaps?
It may solve your problem.

[asusplay]

I have a site that went 99% supplemental in mid April and it has not had the pages re-indexed or even crawled. It gets a visit from Googlebot, but that's about it. I've had a google site-map, placed 301's and haven't got anythign spammy. Since then traffic has nosedived...just when I thought i was getting out of the sandbox and i was beginning to rank for secondary keywords.

So much for keeping to the straight and narrow...Thanks Google.

[AustrianOak]

You're not alone.. it can be very frustrating once you'de done everything by the book and still get the shaft.

[Reid]

Cbartow wrote:
Googlebot hit up 251 of these pages in the new format again this morning, but it did this last weekend, and the weekend before that and they never made it into the index. It didn't hit any of the old pages.

You got me a little confused here.

old or new URLs listed in google?
googlebot hitting new urls but old ones are in the index?

If the new URL's are in the index and googlebot is hitting on the new ones and there are very few requests coming in for the old ones from low level SE's then I would just make them all 410 and get on with it. G Yahoo and MSN are importnt but the rest will just have to catch up (unless you are getting noticable traffic from some other.

[jackson992]

Hey:

I do have Google sitemaps but I haven't noticed that it's helped yet. What really bothers me is the links I have on the home page aren't even getting indexed and this is happening on all my sites.

[tigger]

it took G 3 weeks to follow a link from the index on one of my sites thats a PR4 and that page had loads of one way themed links pointing to it - in the good old days that would normally be crawled within a day or so.

(BTW MSN & Y took 2 days!)

regarding dropped pages well another 3 more got picked up today so thats 218 of a 600 page site hopefully by next year it should be fully back to where it was 4 weeks ago!

[Reid]

I've noticed that some of my pages that do not show up in site: do show up in link: as pages that link to me but they are actually internal pages.
maybe some of you have the same problem and google is filtering some of them out because they don't show all backlinks in Link:

[JuniorOptimizer]

Who is at fault?

Google. Their stated purpose is to organize the world's information. Not SOME of it. All of it. And they're not.

[jimpoo]

"Don't be evil"?

So what is evil? For me, Google is an evil to me.

My 3 years old site with 2000+ pages, many very competited keywords ranked #1 on Yahoo & MSN, but only 120 pages are indexed on Google, zero traffics from google.

[cbartow]

old or new URLs listed in google?
googlebot hitting new urls but old ones are in the index?

Google is crawling only the new URL's, but about 10% of the old ones are showing as supplemental, and the rest are just gone.

I'm now returning 410 on the old ones, but it doesn't really matter anymore cause Google isn't crawling them.

I think I have to just wait and hope. I sent email to bostonpubcon so hopefully I get some sort of response like others are getting.

[Reid]

but it doesn't really matter anymore cause Google isn't crawling them.

If they are in google then they will be crawled again. Just the supplemental index is probably crawled a lot less often.

[g1smd]

Pages that were previously Supplemental with either cache dates or snippets dated from 2004 January to 2005 June were all updated just a week or so ago. For pages that no longer exist, the pages were removed from Googles index. For pages that do still exist, the listing was updated. For URLs that have had a 301 redirect applied before 2005 July (maybe some later ones too), they also removed the redirect URLs from their index.

At the same time, Google created new Supplemental results for pages that have recently gone offline (last 9 months or so); the Supplemental Result having a cache of the final version that was ever online.

They also created Supplemental Results for pages that have been edited at any time after 2005 June. In this case the page shows as a normal result when you search for current content, and as a Supplemental Result when you search for anything that was on that previous version of the page.

In just the last few days, Matt Cutts has acknowledged that some Supplemental Results have recently been refreshed, but has also stated that this process will take a long time to complete for some sites, and that it will continue at least through the Summer. So, my advice is to check back some time near Christms to see where we are...

[g1smd]

>> >> It is a sad fact that systems like vBulletin, PHPbb, osCommerce, and a whole range of popular scripted sites, have a large number of SEO-related design errors built in to them. The designers are clever programmers, but have no clue about SEO or how their site will interact with search engines; and the situation isn't getting any better. << <<

>> I'd have to vote for "Google is at fault/broken" since we have to do all the things you list to "SEO" our sites even though Google says we should develop our sites for visitors, not search engines. <<

You wouldn't upload the same page of content to one site, twelve times with twelve different URLs and then work all of them into the site navigation, and expect it to work... so why do you expect Google to compensate when programs like vBulletin allow a single thread to have at least 12 different URLs that can access it, 24 if you haven't set up the non-www to www redirect, and thousands of URLs if you are showing session IDs to visitors, bots, and anyone not logged in?

Of course these things need fixing. They aren't even real SEO jobs, they are basic design principles, from the simplest of webmastering and hosting concepts.

[Reid]

Thanx for the info about supplemental results G1smd.

cbartow- you could always try the removal tool to remove them. (be very careful) Or you could just let them sit there till christmass. Just leave the 410 on them until the supplemental index gets updated again