Welcome to WebmasterWorld Guest from 54.163.84.199

Pages Dropping Out of Big Daddy Index

Part 3

   
12:36 am on May 17, 2006 (gmt 0)

5+ Year Member



< continued from [webmasterworld.com...] >

Seems to me that Matt's recent message confirms my theory. We're either all a bunch of moaning idiots with low quality sites with a few innapropriate, spammy links scattered here and there...or...

The more I think about it the more convinced I am that the missing pages problem is being caused by a Backlink/PR issue (see Msg #15).
Tying together all of the evidence from my own experience, and that of others gleaned from the forums, erroneous or out-of-date backlinks would explain all of the missing pages.

The erroneous, or simply out-of-date, backlink information (which we cannot see) leads to insufficient PR (which we cannot see) and hence deep pages are not indexed.

We all know that a "link:www.mysite.com" does not show you the complete picture. But, since Big Daddy, it now shows just a tiny proportion of backlinks. Way less, than it used to show before Big Daddy. Why? Because either the backlink index hasn't been updated (and now dates back to mid 2005), or else because it has been updated, but the update process is buggy. Only a small handful of Google employees know which of these two possibilities is the case.

We know that the missing pages problem cannot be due to any kind of duplicate content filter, as some people are suggesting. If this were the case, then effected sites would see a proportion of their pages disapear. Some would lose 10%, some would lose 40%, and some would lose 95%. But that's not what we see. We see sites losing the vast majority of their pages or else losing no pages at all. The reason effected sites lose such high percentages of their pages is because of the hierarchical nature of a site. The number of pages increases with depth, and the artificially low PRs (based on innacurate and/or out-of-date backlink data) prevents the deeper content from being indexed.

The fact that Big Daddy was kick-started from an index dating back to the middle of last year, not only explains why the backlink data might be stale, but it also explains why ancient pages keep popping up on various data centres.

As further evidence: try a "link:www.mysite.com" and compare it to a search for "www.mysite.com". In my case, the "link:" search shows just 6 results, only one of which is external to my site. The one external backlink probably pre-dates when Big Daddy's index was seeded. The "www.mysite.com" search, on the other hand, finds hundreds of results representing hundreds of internal and external backlinks. Why aren't these showing up in the "link:" search? Is it because "link:" searches are well known for not showing you the complete picture? Or, has that well-known fact simply been obscuring the true cause of all of the problems? Namely, that the backlinks are simply missing from Google's backlink index.

[edited by: tedster at 8:25 pm (utc) on May 17, 2006]

7:30 pm on May 18, 2006 (gmt 0)

5+ Year Member



just a thought about all of us that will be financially ruined. nope. it takes a few days to rebuild a site on a new domain. if you have decent ideas you can survive no matter what.

about pages dropped. google, unless they are going private again, is gonna have to allow some percieved "junk" in, since it carries adsense. i'd say they will keep some good stuff with good click-through rates, and some junk stuff with fantastic click through rates. adsense ads in my opinion are the only motivator for any flux in rankings. e-commerce and other sites are just affected temporarily until google tries to make sure that adsense carrying sites with not completely junky content are presented first in search results, mix of authority sites with ads and without.

also, if they don't keep listing more pages with adsense, they will stop growing revenues, and shareholders will just appoint a new board of directors that will tell them to index all pages, period.

they will hopefully just present the most relevant results first, followed by crap. backlinks and all that crap are just tactics to get people to talk about things, while g is dancing.

so, sit tight.

8:32 pm on May 18, 2006 (gmt 0)

5+ Year Member



Seems like Matt Cutts team is repairing Google. Some of my pages are reappearing on the following datacenters :
64.233.179.104
64.233.171.104
64.233.185.104
216.239.39.104
8:38 pm on May 18, 2006 (gmt 0)

10+ Year Member



Is there any difference with singular or plural search terms. I have noticed that the words are highlighted as plural yet our site has now dropped for the plural search term but is still listed for the singular although it has a low search term
10:23 pm on May 18, 2006 (gmt 0)

5+ Year Member



Why don't most of you believe that Matt Cutts is just giving us spin? One of my sites is down to the home page only. One that is mostly scholarly content lost 50% of the pages. One that is a links based web site with minimal content remains fully indexed.

One of my former competitors, who is no longer in business and no longer has a web site, ranks 5th on first page of search with key words. The site is Cached from last year.

11:16 pm on May 18, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Yep, and for us in UK and some other Europe countries (Germany I think is very much G orientated) Google market share is 80%+

For me:
German website: 96% G
English website: Yahoo MSN and G equal.

sample size around 12000 uniques a day.

11:19 pm on May 18, 2006 (gmt 0)

10+ Year Member



Wow folks!

Complained yesterday about my site going from 300+ to 50+ pages in the last 3 days.

I checked my weblogs and see that my overall hits are up from Google?
This made no sense, so I check individual search terms, and saw that I was coming up with terms I never used to rank for.

I then did a 'site:www.competitor.com' search on my closest competitors websites. They are gone! My closest competitor with a 5+ year old site has 40 pages left out of over 1000. ALL of the smaller mom and pop e-commercers in my field are de-indexed worse than I am!

Why am I not consoled?

11:32 pm on May 18, 2006 (gmt 0)

5+ Year Member



Hi,

I must have missed it and I am feeling rather dense here but where did Matt mention the problem with hyphenated domains and that they are aware?

My site is definitely suffering and we do have a hyphen in the domain name. We dropped overnight from 19k+ pages to just under 600 pages. Probably will lose the last 600 tonight but I am curious if there is any relation.

If you can direct me to where this topic was discussed, I would appreciate it.

Thanks

Berry

12:21 am on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



- The team refreshing our supplemental results checked out feedback, and on May 5th they discovered that a “site:” query didn’t return supplemental results. I think that they had a fix out for that the same day. Later, they noticed that a difference in the parser meant that site: queries didn’t work with hyphenated domains. I believe they got a quick fix out soon afterwards, with a full fix for site: queries on hyphenated domains in supplemental results expected this week.

mattcutts.com blog

12:21 am on May 19, 2006 (gmt 0)

5+ Year Member



never mind.. found it.

---

Oops, thanks mattg3

Looks like we are just being de-indexed because we are small. *sigh* Once Google was small... They have already forgotten what it's like...

3:25 am on May 19, 2006 (gmt 0)

5+ Year Member



>> Frederic 1

64.233.179.104
64.233.171.104
64.233.185.104
216.239.39.104

I could not find any more pages than the one I see on Google now

Our site is loosing more and more pages everyday

I removed site maps from Google Maps ... nothing seems to happen

Is there anyone at Google reading those threads?

4:43 am on May 19, 2006 (gmt 0)

5+ Year Member



my site is entirely back in the index now. All I had to do was duplicate all my content into a static format instead of dynamic. Now I have duplicates of every page..one static and one dynamic..but all are listed now.
5:23 am on May 19, 2006 (gmt 0)

5+ Year Member



> supermanu

sorry it was a false alert.
finally, my site is still deindexing on those datacenters as well as the others :'(

6:46 am on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Friends, don't worry over Google dropping pages. I was anxious too when I saw about 20% of my pages were lost, that was only until I stopped by DMoz and did a couple of tests.

Lets start with dmoz.org homepage. I clicked "Video Games" link. That page is indexed. From there I went to "Action", indexed. Then moved onto "Browser Based", indexed as well. Then to "Asteroids". Aha (as MC says), that page isn't indexed, but has a PR3.

Now repeating the exercise -

Home>Arts>Animation>Anime>Fandom>Fan Works>Fan Art>Web Rings (Not Indexed, PR3)

Home>Science>Social Sciences>Geography>Cartography>Services (Not Indexed, PR5)

Home>Shopping>Vehicles>Autos>Custom and Collector Cars>Customized Cars>Rod and Custom (Not Indexed, PR3)

Do we have sufficient instances to convince ourselves? Pick any category that has many sub-levels, the deeper categories are not indexed.

Now, will Googl risk loosing pages from sites such as Dmoz? I wouldn't think so. If we take what Matt says as right (about sites not having enough link popularity of merit, link exchanges, low quality outbounds etc as reasons behind all these), it somehow doesn't corroborate with tests done on quality sites, and I would tend to think this is an issue that is fixed sooner than later. Relax guys :-)

7:28 am on May 19, 2006 (gmt 0)

5+ Year Member



Some bigger number of indexed pages are back :).
7:40 am on May 19, 2006 (gmt 0)

5+ Year Member



McMohan
"Friends, don't worry over Google dropping pages. I was anxious too when I saw about 20% of my pages were lost, that was only until I stopped by DMoz and did a couple of tests. "
Did you try those tests at Google Directory :)
just some random I did and of cource no cache
Regional/Europe/United_Kingdom/Recreation_and_Sports/Pets/Reptiles_and_Amphibians/

Regional > Europe > United Kingdom > Recreation and Sports > Sports > Country Sports > Associations
Regional > Europe > United Kingdom > Recreation and Sports > Sports > Country Sports > Web Rings
Regional > Europe > Ukraine > Provinces > Khmelnytskyi Oblast > Letychiv

Hmmm...looks like Mcmohan is right try your selfs as well looks like Google deindexing Google :)

7:58 am on May 19, 2006 (gmt 0)

5+ Year Member



The whole thing looks more like "Google Self Destruct" to me.

Last one out of the index turn off the lights please.

8:03 am on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



1984bb, Google deindexing G directory pages maybe some how correlated to dmoz getting deindexed, am not sure though.

Repeated this one on Yahoo directory. But it is doing better than dmoz. However the paginated pages (after the 1st page) are not indexed in many instances I checked. Not enough trusted links perhaps to Yahoo :-)

8:29 am on May 19, 2006 (gmt 0)

10+ Year Member



Hello all

I have never posted in this category before so I am a little nervous.

Some time ago (early May) GGuy gave a title code that we could use to contact the search team to report dropped pages (Cannot remember code word)

I wrote (expecting the usual cut-paste reply) However yesterday I had an email back saying that it is likely that the dropped pages may be back within the week. Something about datacenter fluctuations.

It is all double-dutch to me but it would be nice to see Google get back to a reliable index. I tend to use other engines if I am really looking for something.

9:58 am on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>Some time ago (early May) GGuy gave a title code that we could use to contact the search team to report dropped pages (Cannot remember code word)

try "crawlpages" within the title

10:11 am on May 19, 2006 (gmt 0)

5+ Year Member



Sense_able,

Good for you. They must have spotted something there. At least you got a response.

10:15 am on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member



sorry Sense_able I miss understood your posting - need more coffee, well done on getting a reply
10:32 am on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



may be back within the week

mmmm MAY come back and WHICH week, this ones done and dusted already, although most movements do happen over weekends I have noticed Mondays show changes, the first view we all got of the new SERP,s was 2 Mondays ago

10:37 am on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



tigger

"sorry Sense_able I miss understood your posting - need more coffee,..."

Coffee wouldn't do it. Try Cappuccino ;-)

10:44 am on May 19, 2006 (gmt 0)

5+ Year Member



Since two weeks my site pages to dropped out of index and showing only the home page in Index.

I have submitted my sitemap to Google 2 days ago. Today I can able to see the report generated and it reads "No pages from your site are currently included in Google's index. Indexing can take time." and it also says that "Googlebot last successfully accessed your home page on May 16" i.e. the day which I have submitted my sitemap.

Can anyone here predict how long might take to index pages after submitting sitemap.

10:49 am on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I found the sitemap offered no benefits at all, so long as you have your own thats good enough IMHO
12:16 pm on May 19, 2006 (gmt 0)

5+ Year Member



reseler"Coffee wouldn't do it. Try Cappuccino ;-) "
I unfortunatelly can't drink anymore coffeine coffee because some time ago Google send me to the Hospital with Heart palpitations , since then the doctors advice me not to Smoke ,Drink ,no coffeine ,no stress with Google ,I rememper the date it was just before xmas 2005 when one night I sudenly show my site's top possitions disappearing and go at page 80.....
Since then my life has change a lot ,after a period of deep depression and nearly on edge of divorce because I had become an old grumpy old man and all just because of Google ,i have invested all my last 3 years internet revenues on building an existense on something that is controled only by me (thanks to CJ I made a small hotel in a small Greek island)
12:31 pm on May 19, 2006 (gmt 0)

10+ Year Member



"Friends, don't worry over Google dropping pages. I was anxious too when I saw about 20% of my pages were lost, that was only until I stopped by DMoz and did a couple of tests. "

There are a vast amount of high level Dmoz pages not cached by Google when I last checked. You would assume therefore that dropped pages has something to do with a sites content rather than the inbound links. Dmoz must have some of the best incoming links on the web and still it has this problem. Obviously it is doing something Google doesn't like presently.

1:29 pm on May 19, 2006 (gmt 0)

10+ Year Member



I am generally not a believer in conspiracy theories, but has anyone stopped to consider that this entire BigDaddy fiasco may actually have a motive? It's a very efficient way for Google to test how many webmasters and small businesses will convert into Adwords customers after their sites have been decimated. Google is giving us all a good hard PUSH in that direction by "suddenly" screwing up the index. "Ooops! Sorry your traffic has vanished inexplicably...NOW PAY US." I for one plan to rebuild my site to try an rank better with Yahoo and MSN rather than pay Google to advertise. The most sensible thing we can all to is try to become less dependent on any one SE for our business.
2:09 pm on May 19, 2006 (gmt 0)



I am generally not a believer in conspiracy theories, but has anyone stopped to consider that this entire BigDaddy fiasco may actually have a motive? It's a very efficient way for Google to test how many webmasters and small businesses will convert into Adwords customers after their sites have been decimated.

Spot on! i just don't buy the crap and smoke screen various google reps are trying to build. They are openly saying that they are going to look closely into sites with affiliate links and incoming outgoing site links....opps there goes 95% of the web....how convenient.

Pathetic really!, i just hope those affected will not fall for it and (as a matter of principal) will use the Yahoo & MSN ad networks. Not Adwords.

Any webmaster that has some respect left for his/hers site(s) should immediately stop supporting this octopus. We do have the power to tip the scales if we realy want to. Do your part.

2:41 pm on May 19, 2006 (gmt 0)

5+ Year Member



Agree entirely with previous few posts. Is their prime objective to rid the web of affiliate sites? Their money comes from adwords not adsense. Judging by the flotsam and jetsam in search results over the last two months, what a lot of collateral damage. And the worst sites are still up there!

Meanwhile we are picking over the bones of hyphenation, 301 www/non-www, canonization.. phooey! Hate to put a spanner in the works but my $20k/mth adwords budget is moving steadily across to Yahoo and to MSN when AdCenter launches in the UK. Let's get moving!

This 213 message thread spans 8 pages: 213
 

Featured Threads

Hot Threads This Week

Hot Threads This Month