Forum Moderators: Robert Charlton & goodroi
Seems to me that Matt's recent message confirms my theory. We're either all a bunch of moaning idiots with low quality sites with a few innapropriate, spammy links scattered here and there...or...
The more I think about it the more convinced I am that the missing pages problem is being caused by a Backlink/PR issue (see Msg #15).
Tying together all of the evidence from my own experience, and that of others gleaned from the forums, erroneous or out-of-date backlinks would explain all of the missing pages.The erroneous, or simply out-of-date, backlink information (which we cannot see) leads to insufficient PR (which we cannot see) and hence deep pages are not indexed.
We all know that a "link:www.mysite.com" does not show you the complete picture. But, since Big Daddy, it now shows just a tiny proportion of backlinks. Way less, than it used to show before Big Daddy. Why? Because either the backlink index hasn't been updated (and now dates back to mid 2005), or else because it has been updated, but the update process is buggy. Only a small handful of Google employees know which of these two possibilities is the case.
We know that the missing pages problem cannot be due to any kind of duplicate content filter, as some people are suggesting. If this were the case, then effected sites would see a proportion of their pages disapear. Some would lose 10%, some would lose 40%, and some would lose 95%. But that's not what we see. We see sites losing the vast majority of their pages or else losing no pages at all. The reason effected sites lose such high percentages of their pages is because of the hierarchical nature of a site. The number of pages increases with depth, and the artificially low PRs (based on innacurate and/or out-of-date backlink data) prevents the deeper content from being indexed.
The fact that Big Daddy was kick-started from an index dating back to the middle of last year, not only explains why the backlink data might be stale, but it also explains why ancient pages keep popping up on various data centres.
As further evidence: try a "link:www.mysite.com" and compare it to a search for "www.mysite.com". In my case, the "link:" search shows just 6 results, only one of which is external to my site. The one external backlink probably pre-dates when Big Daddy's index was seeded. The "www.mysite.com" search, on the other hand, finds hundreds of results representing hundreds of internal and external backlinks. Why aren't these showing up in the "link:" search? Is it because "link:" searches are well known for not showing you the complete picture? Or, has that well-known fact simply been obscuring the true cause of all of the problems? Namely, that the backlinks are simply missing from Google's backlink index.
[edited by: tedster at 8:25 pm (utc) on May 17, 2006]
about pages dropped. google, unless they are going private again, is gonna have to allow some percieved "junk" in, since it carries adsense. i'd say they will keep some good stuff with good click-through rates, and some junk stuff with fantastic click through rates. adsense ads in my opinion are the only motivator for any flux in rankings. e-commerce and other sites are just affected temporarily until google tries to make sure that adsense carrying sites with not completely junky content are presented first in search results, mix of authority sites with ads and without.
also, if they don't keep listing more pages with adsense, they will stop growing revenues, and shareholders will just appoint a new board of directors that will tell them to index all pages, period.
they will hopefully just present the most relevant results first, followed by crap. backlinks and all that crap are just tactics to get people to talk about things, while g is dancing.
so, sit tight.
One of my former competitors, who is no longer in business and no longer has a web site, ranks 5th on first page of search with key words. The site is Cached from last year.
Yep, and for us in UK and some other Europe countries (Germany I think is very much G orientated) Google market share is 80%+
For me:
German website: 96% G
English website: Yahoo MSN and G equal.
sample size around 12000 uniques a day.
Complained yesterday about my site going from 300+ to 50+ pages in the last 3 days.
I checked my weblogs and see that my overall hits are up from Google?
This made no sense, so I check individual search terms, and saw that I was coming up with terms I never used to rank for.
I then did a 'site:www.competitor.com' search on my closest competitors websites. They are gone! My closest competitor with a 5+ year old site has 40 pages left out of over 1000. ALL of the smaller mom and pop e-commercers in my field are de-indexed worse than I am!
Why am I not consoled?
I must have missed it and I am feeling rather dense here but where did Matt mention the problem with hyphenated domains and that they are aware?
My site is definitely suffering and we do have a hyphen in the domain name. We dropped overnight from 19k+ pages to just under 600 pages. Probably will lose the last 600 tonight but I am curious if there is any relation.
If you can direct me to where this topic was discussed, I would appreciate it.
Thanks
Berry
- The team refreshing our supplemental results checked out feedback, and on May 5th they discovered that a “site:” query didn’t return supplemental results. I think that they had a fix out for that the same day. Later, they noticed that a difference in the parser meant that site: queries didn’t work with hyphenated domains. I believe they got a quick fix out soon afterwards, with a full fix for site: queries on hyphenated domains in supplemental results expected this week.
mattcutts.com blog
---
Oops, thanks mattg3
Looks like we are just being de-indexed because we are small. *sigh* Once Google was small... They have already forgotten what it's like...
64.233.179.104
64.233.171.104
64.233.185.104
216.239.39.104
I could not find any more pages than the one I see on Google now
Our site is loosing more and more pages everyday
I removed site maps from Google Maps ... nothing seems to happen
Is there anyone at Google reading those threads?
sorry it was a false alert.
finally, my site is still deindexing on those datacenters as well as the others :'(
Lets start with dmoz.org homepage. I clicked "Video Games" link. That page is indexed. From there I went to "Action", indexed. Then moved onto "Browser Based", indexed as well. Then to "Asteroids". Aha (as MC says), that page isn't indexed, but has a PR3.
Now repeating the exercise -
Home>Arts>Animation>Anime>Fandom>Fan Works>Fan Art>Web Rings (Not Indexed, PR3)
Home>Science>Social Sciences>Geography>Cartography>Services (Not Indexed, PR5)
Home>Shopping>Vehicles>Autos>Custom and Collector Cars>Customized Cars>Rod and Custom (Not Indexed, PR3)
Do we have sufficient instances to convince ourselves? Pick any category that has many sub-levels, the deeper categories are not indexed.
Now, will Googl risk loosing pages from sites such as Dmoz? I wouldn't think so. If we take what Matt says as right (about sites not having enough link popularity of merit, link exchanges, low quality outbounds etc as reasons behind all these), it somehow doesn't corroborate with tests done on quality sites, and I would tend to think this is an issue that is fixed sooner than later. Relax guys :-)
Regional > Europe > United Kingdom > Recreation and Sports > Sports > Country Sports > Associations
Regional > Europe > United Kingdom > Recreation and Sports > Sports > Country Sports > Web Rings
Regional > Europe > Ukraine > Provinces > Khmelnytskyi Oblast > Letychiv
Hmmm...looks like Mcmohan is right try your selfs as well looks like Google deindexing Google :)
Last one out of the index turn off the lights please.
Repeated this one on Yahoo directory. But it is doing better than dmoz. However the paginated pages (after the 1st page) are not indexed in many instances I checked. Not enough trusted links perhaps to Yahoo :-)
I have never posted in this category before so I am a little nervous.
Some time ago (early May) GGuy gave a title code that we could use to contact the search team to report dropped pages (Cannot remember code word)
I wrote (expecting the usual cut-paste reply) However yesterday I had an email back saying that it is likely that the dropped pages may be back within the week. Something about datacenter fluctuations.
It is all double-dutch to me but it would be nice to see Google get back to a reliable index. I tend to use other engines if I am really looking for something.
try "crawlpages" within the title
Good for you. They must have spotted something there. At least you got a response.
may be back within the week
mmmm MAY come back and WHICH week, this ones done and dusted already, although most movements do happen over weekends I have noticed Mondays show changes, the first view we all got of the new SERP,s was 2 Mondays ago
"sorry Sense_able I miss understood your posting - need more coffee,..."
Coffee wouldn't do it. Try Cappuccino ;-)
I have submitted my sitemap to Google 2 days ago. Today I can able to see the report generated and it reads "No pages from your site are currently included in Google's index. Indexing can take time." and it also says that "Googlebot last successfully accessed your home page on May 16" i.e. the day which I have submitted my sitemap.
Can anyone here predict how long might take to index pages after submitting sitemap.
"Friends, don't worry over Google dropping pages. I was anxious too when I saw about 20% of my pages were lost, that was only until I stopped by DMoz and did a couple of tests. "
There are a vast amount of high level Dmoz pages not cached by Google when I last checked. You would assume therefore that dropped pages has something to do with a sites content rather than the inbound links. Dmoz must have some of the best incoming links on the web and still it has this problem. Obviously it is doing something Google doesn't like presently.
I am generally not a believer in conspiracy theories, but has anyone stopped to consider that this entire BigDaddy fiasco may actually have a motive? It's a very efficient way for Google to test how many webmasters and small businesses will convert into Adwords customers after their sites have been decimated.
Spot on! i just don't buy the crap and smoke screen various google reps are trying to build. They are openly saying that they are going to look closely into sites with affiliate links and incoming outgoing site links....opps there goes 95% of the web....how convenient.
Pathetic really!, i just hope those affected will not fall for it and (as a matter of principal) will use the Yahoo & MSN ad networks. Not Adwords.
Any webmaster that has some respect left for his/hers site(s) should immediately stop supporting this octopus. We do have the power to tip the scales if we realy want to. Do your part.
Meanwhile we are picking over the bones of hyphenation, 301 www/non-www, canonization.. phooey! Hate to put a spanner in the works but my $20k/mth adwords budget is moving steadily across to Yahoo and to MSN when AdCenter launches in the UK. Let's get moving!
