| This 249 message thread spans 9 pages: < < 249 ( 1 2 3 4 5  7 8 9 ) > > || |
|Pages Dropping Out of Big Daddy Index|
Continued from: [webmasterworld.com...]
internetheaven, you said:
|I had 20,300 pages showing for a site:www.example.com search yesterday and for the past month. Today it dropped to 509 but my traffic is still pretty constant. I normally get around 4,500 - 5,000 to that site per day and today I've already got 4,000. |
So, either Google doesn't account for even a small percentage of my traffic (which I doubt) or the way Google stores information about my site has changed. i.e. the 20,300 pages are still there, Google will only tell me about 509 of them. As far as I can tell, I think the other pages have been supplemented.
That resonated with something that I was talking about with the crawl/index team. internetheaven, was that post about the site in your profile, or a different site? Your post aligns exactly with one thing I've seen in a couple ways. It would align even more if you were talking about a different site than the one in your profile. :) If you were talking about a different site, would mind sending the site name to bostonpubcon2006 [at] gmail.com with the subject line of "crawlpages" and the name of your site, plus the handle "internetheaven"? I'd like to check the theory.
Just to give folks an update, we've been going through the feedback and noticed one thing. We've been refreshing some (but not all) of the supplemental results. One part of the supplemental indexing system didn't return any results for [site:domain.com] (that is, a site: search with no additional terms). So that would match with fewer results being reported for site: queries but traffic not changing much. The pages are available for queries matching the supplemental results, but just adding a term or stopword to site: wouldn't automatically access those supplemental results.
I'm checking with the crawl/index folks if this might factor into what people are seeing, and I should hear back later today or tomorrow. In the mean time, interested folks might want to check if their search traffic has gone up/down by a major amount, and see if there are fewer/more supplemental results for a site: search for their domain. Since folks outside Google couldn't force the supplemental results to return site: results, it needed a crawl/index person to notice that fact based on the feedback that we've gotten.
Anyone that wants to send more info along those lines to bostonpubcon2006 [at] gmail.com with the subject line "crawlpages" is welcome to. So you might send something like "I originally wrote about domain.com. I looked at my logs and haven't seen a major decrease in traffic; my traffic is about the same. I used to have about X% supplemental results, and now I hardly see any supplemental results with a site:domain.com query."
I've still got someone reading the bostonpubcon email alias, and I've worked with the Sitemaps team to exclude that as a factor. The crawl/index folks are reading portions of the feedback too; if there's more that I notice, I'll stop by to let you know.
[edited by: Brett_Tabke at 8:07 pm (utc) on May 8, 2006]
That affiliate statement is very broad. There is no way Google is going to kick out all affiliates as that would be financial suicide for them. Plus the user experience would become horrendous.
On a side note, my site: command shows no change in the # of pages and I have checked to make sure some of my newer pages wereen't indexed. Note that they are not in supplemental so there is no way they were just penalized either.
I agree it is very broad and like everything else it leaves us webmasters having to try and second guess what G/MC is going on about
Please treat this post with caution - I've subsequently got my doubts as to whether it's genuine - maybe the moderators can take it out if necc.
I thought this forum was a hoax as i saw no response on any of the 100 or so messages.
Hopefully, this communication isn't a hoax either - it doesn't look like it, but who knows, there's a lot of uncertainty.
Can i just take the opportunity to thank all of the team at Google [ Vanessa Fox , Matt Cutt's - before you head off to Utah for hols and the other team members if indeed the content on this is true ] - I'm sure the rest of WebmasterWorld members would agree.
Not sure that loosing our top 10 positions and indexed pages is making us jump for joy at the moment - but we're hoping the prob will be fixed.
Some specific mention of the processes that you're going through and how it is effecting us would likely take a bit of the sting away as well.
Google Employee just posted this :
Hey guys, it's not a hoax and Ms "Google Employee" IS listening to
these posts and sending them off to the appropriate places [groups.google.com]. They just
aren't going to answer any single reports here.
"We do look into every report, though we may not have time to respond
to them all individually. "
The sitemaps team has done some things that were never done before, and
has done many things - handled many exceptions - faster than I would
ever have imagined possible within a large Corp. like Google. Google
used to be a super-secret "we ain't saying nuthin'" system, now we are
getting feedback like never before. Let's give the Sitemaps team a
little respect - they can't handle this issue by themselves alone, it
is something that is probably buried elsewhere. Kudos to Vanessa and
the team for putting up a thread about it here and for taking our
I know you all are hurting and it's been quite some time, but these
things sometimes take their time and IMHO it's worth working out the
details if the index is better afterwards: I would rather be listed
under the 10 results for my niche instead of being listed in a pile of
100 spam results.
Let's continue to keep this thread clean, keep listing the sites that
got hit and the information that is required. [feel free to flame me in
a new thread :-)]
If this isn't a hoax, then there's hope.
[edited by: Whitey at 8:31 am (utc) on May 15, 2006]
To me what Cutts is saying is Google is on another witch-hunt with its new infrastructure. Look for even more quality web sites to bite the bullet and make less money.
Iím just now fully encountering the problem. Itís baffling me how some of my one of a kind articles are going supplemental. These articles often take a month or more to create and donít have Adsense. Again other search engines are handling it flawlessly and ranking the pages well. Why must Google have sitemaps and nobody else? Google just creates an nver ending flow of work for webmasters so they can line their own pockets. Has anybody told them itís all right for others to make money, not just them and their buddies.
This situation is becoming a REAL NIGHTMARE:
Two of of my main sites have lost between 47% and 70% of their pages in Google. Find below the number of occurences I find in Google now...
I have Site Map, Big content, no duplicates, no tricks, real clean
Website 1 (5 years): 300,000 => 250,000 => 161,000 (-47%)
Website 2 (1 year): 99,000 => 40,000 => 29,900 (-70%)
Those numbers did increase a bit last week but now it is a total disaster.
Any solution? What do you think?
Please help me!
welcome to the club, as for a fix or a solution right now I don't there is one, all I'm doing is still adding content and hoping that G sorts itself out
Additionally, apart from pages dropping, backlinks therefore not working and all the rest of the problems we're experiencing just look at Google's flip side.
-Google's Adsense revenues will be hit without content pages.
-Webmasters will surely start to switch to other ad networks. Why use Adsense if the pages aren't showing on Google - may as well use Yahoo.
-Why not diversify advertising away from Google if there is a chance the pages are declining.
When leading SEO's in my region, who manage PPC for Blue Chip Co's start talking to me like this on their initiation, i know the thinking is starting to occur elsewhere. And Yahoo is not so bad that some of those folks will come back when it's all fixed [ when?!? ]
I think Google have a problem on their side.
|Google just creates an never ending flow of work for webmasters |
Idle hands are the devils work I believe the saying goes.
That wasn't a post by "Google Employee". It was a post by someone with the nickname "softplus".
Google haven't responded to anything in over a week.
|Basically, Google doesn't want 1000 sites in the index selling the same Pocahantas shirt with the same Disney description. |
This kind of statement is made so often now that it's almost as if people are starting to believe that it's true.
They may not want 1000 sites in their index selling the same shirt with the same description, but unfortunately in order to qualify as a search engine, they need to.
How annoyed would you be with Google if you were looking to buy this particular shirt, you search on Google, and they present you with one cherry-picked, "original" result? You click it and discover that they are asking $900 for it, they don't ship to your area, and they are currently out of stock. I guess you are just out of luck. Or else you need to find yourself a real search engine that presents you with all of the options, ranked by percieved relevancy. Even Google aren't this dumb.
|How annoyed would you be with Google if you were looking to buy this particular shirt, you search on Google, and they present you with one cherry-picked, "original" result? You click it and discover that they are asking $900 for it, they don't ship to your area, and they are currently out of stock. I guess you are just out of luck. Or else you need to find yourself a real search engine that presents you with all of the options, ranked by percieved relevancy. Even Google aren't this dumb. |
I second every word.
thanx, great post, all of truth.
May a add that different poeple like different layouts? Some prefer it with navigation on top some with navigation
on left and so on. The users wonīt have a chance to chose a page of their prefered layout/navigation. That canīt be in the interest of google.
It seems to me that pages are not dropping from the index for affiliate content, duplicate content or anything particularly "wrong". As I've posted before, all my dropped pages were written in about the last five months and none were linked to my home page. Last Thursday, I put up a list of "New pages on mysite.com", linked to the dropped pages. Today they are all in the index, although not ranking very well. At least they're there! Which is a step. Curiously, in combing through my new pages last week, I discovered that one was scraped. That page was indexed both on my site and the scraper site! Go figure.
"Page removed from index" is the most serious punishment from google. So if affiliate page is the problem, they can only set a lower ranking property for that site. The current situation is some affiliate or non-affiliate pages (for example, my forum pages) were removed from google index completely and the remainning affiliate pages of the same site rank very well.
I am surprised to see this thread, I thought all sites that went supplemental all came back a few months ago. I know my sites did after a 10 day stay in 'supplemental hell'.
Or is this a different issue? I notice I don't have 60K+ pages indexed anymore, now I am closer to 25K but I believe that is a result of not having www, non-www, http, and https versions of the same pages indexed. Now I only have the proper www results indexed. That is a good thing.
Well, I guess it is time to go get a full time job. Google has completely erased one site I built three months ago. For a client. At first, it was almost fully indexed and then went down to two pages. Now it is gone.
Another site I put up for another client is down to one page with only a few others showing and all of those are supplemental.
A long time site has gone down the tubes for another client. All but two pages supplemental.
These are all sites for small businesses. No black hat tactics. No real optimization. Right by the book.
My heart sunk when I saw that the one had been deindexed. No reason for it.
Not going to be able to get any new clients with results like that. "See what I did for them?". lmao.
Maybe I will try something new.
Thanks for letting me rant.
|Google has completely erased one site I built three months ago. |
Wouldn't this be the standard Google sandbox? Sometimes a new site will show up for a 2-4 weeks then go to the sandbox for 18+ months. Three month old sites pretty much don't show up in Google anymore unless you know the secret to avoiding the sandbox.
Thought I would just add in my experience.
Either late last week or at the start of the weekend Google reduced our site from several hundred pages indexed to just two, the domain name and index.php.
Let me just say that our site has never done well on Google, despite ranking near top on most searches in MSN and Yahoo. Every 3-6 months I do some tweaks to try and make the site more Google-friendly - I should point out that these tweaks are all content and presentation based and not to do with fake links or cloaking.
Here is a rough chronology of what we have done to the site in the last two weeks.
1) Moved the osCommerce store to the root directory, placed an .htaccess 301 redirect to make requests for the old urls go to the right place.
2) Installed a contribution to generate static .html product urls rather than the dynamic php ones. This generates 301's at all the old php pages.
3) Installed another contribution to generate a Google sitemap.
4) Uploaded the Google sitemap.
And then the camel's back broke and we went down to two pages indexed.
My gut feeling is that the sitemap submission has caused all the indexed pages that did not have a PR to fall out of the index. Also possible is that Google feels the number of 301's is spammy.
So I'm pretty frustrated with Google at the moment. Even more galling is that they do a decent trade from our Adwords account each month.
My plan now is to make our case on the Google Groups and by email and see if we can get crawled properly again. I don't care if our pages are last on the index, just as long as the content is available!
><>>>Three month old sites pretty much don't show up in Google anymore unless you know the secret to avoiding the sandbox. <<<<<
Nope. Not sandboxed. Gone. As in "no information for this url...".
If I was doing something no-no I would expect this but there is nothing to be penalized for.
This isn't even a competitive niche.
texasville, i have the same problem. A new site of mine that took forever to get indexed finally did, 750 pages. After 3 or 4 weeks of being fully indexed I'm now down to 100 pages and dropping.
I have no idea why.
My site pages r drops today?
Convenient "nofollow" post on MC blog [mattcutts.com...]
jd & texas
Your saying that its very recent you lost the pages?
I and others on this board have been in this state for well over a month now, it will be interesting to see if when we recover you recover, or, you take as long as we did, you should see fluctuations on page count every week or so but as yet (after a month) still not showing all pages that were previousley in the index dropped.
is google reset his index?
I'm seeing some dramatic drops over the last 2-3 days also. 2 weeks ago I updated page titles, keywords and some copy, then uploaded a sitemap. The pages I changed jumped up to the 1st page within a few days.
Last week, I updated another set of pages and resubmitted the sitemap.
Last night, the main page I was concerned with (which was on the 1st page) is now gone.
The latest crawl found some problems with URL case-sensitivity in the sitemap. I fixed those today, but could that have caused a ton of properly listed pages to drop?
I may need to go back to my standard line of "I'll submit and do my best to optimize, but no promises, guarantees or refunds on SEO".
My site pages r drops today? Why?
Everybody asks this question!
wheelie34, I noticed a 50% reduction in indexed pages on the 9th May. site:domain.com search is not something i do every day but as soon as my traffic plummeted it was the first thing i did. As of today the 350 pages that were left over on the 9th are now down to 100 or so.
Ironically I built this site with hundreds of keywords that would not be sandboxed (very uncompetitive) I figured a few hundred visitors a week would be better then sitting in the sandbox for a year waiting/hoping for the big money terms. So to have almost all the pages taken out of the index so soon is a real choker.
I guess trying to be smart didn't help after all!
I have lost 60% of my indexed pages in last couple days on all domains, but strange thing i can't see drop in traffic.
A LOT of osCommerce sites (and some other scripted sites) have been badly whacked in the last few months. Their multiple-URLs-for-the-same-content problem, is a factor, as is the not having a unique title and meta description per page, that some sites have.
If you are showing session IDs as well, then it is "Game Over".
The contributions you have added will work in the long term, but in the short term you are going to need to allow several months for Google to sort out where your site has gone (changing the URL for every page of the site means that it looks like it has "moved"; and it might even be flagged as a "new site" now).
"I have lost 60% of my indexed pages in last couple days on all domains, but strange thing i can't see drop in traffic. "
Maybe the site: command is broke, not unlike the link: command. Perhaps the pages are actually there.
|Maybe the site: command is broke, not unlike the link: command. Perhaps the pages are actually there. |
Here we go again...
| This 249 message thread spans 9 pages: < < 249 ( 1 2 3 4 5  7 8 9 ) > > |