Forum Moderators: Robert Charlton & goodroi
I had 20,300 pages showing for a site:www.example.com search yesterday and for the past month. Today it dropped to 509 but my traffic is still pretty constant. I normally get around 4,500 - 5,000 to that site per day and today I've already got 4,000.So, either Google doesn't account for even a small percentage of my traffic (which I doubt) or the way Google stores information about my site has changed. i.e. the 20,300 pages are still there, Google will only tell me about 509 of them. As far as I can tell, I think the other pages have been supplemented.
That resonated with something that I was talking about with the crawl/index team. internetheaven, was that post about the site in your profile, or a different site? Your post aligns exactly with one thing I've seen in a couple ways. It would align even more if you were talking about a different site than the one in your profile. :) If you were talking about a different site, would mind sending the site name to bostonpubcon2006 [at] gmail.com with the subject line of "crawlpages" and the name of your site, plus the handle "internetheaven"? I'd like to check the theory.
Just to give folks an update, we've been going through the feedback and noticed one thing. We've been refreshing some (but not all) of the supplemental results. One part of the supplemental indexing system didn't return any results for [site:domain.com] (that is, a site: search with no additional terms). So that would match with fewer results being reported for site: queries but traffic not changing much. The pages are available for queries matching the supplemental results, but just adding a term or stopword to site: wouldn't automatically access those supplemental results.
I'm checking with the crawl/index folks if this might factor into what people are seeing, and I should hear back later today or tomorrow. In the mean time, interested folks might want to check if their search traffic has gone up/down by a major amount, and see if there are fewer/more supplemental results for a site: search for their domain. Since folks outside Google couldn't force the supplemental results to return site: results, it needed a crawl/index person to notice that fact based on the feedback that we've gotten.
Anyone that wants to send more info along those lines to bostonpubcon2006 [at] gmail.com with the subject line "crawlpages" is welcome to. So you might send something like "I originally wrote about domain.com. I looked at my logs and haven't seen a major decrease in traffic; my traffic is about the same. I used to have about X% supplemental results, and now I hardly see any supplemental results with a site:domain.com query."
I've still got someone reading the bostonpubcon email alias, and I've worked with the Sitemaps team to exclude that as a factor. The crawl/index folks are reading portions of the feedback too; if there's more that I notice, I'll stop by to let you know.
[edited by: Brett_Tabke at 8:07 pm (utc) on May 8, 2006]
Google has completely erased one site I built three months ago.
Wouldn't this be the standard Google sandbox? Sometimes a new site will show up for a 2-4 weeks then go to the sandbox for 18+ months. Three month old sites pretty much don't show up in Google anymore unless you know the secret to avoiding the sandbox.
Either late last week or at the start of the weekend Google reduced our site from several hundred pages indexed to just two, the domain name and index.php.
Let me just say that our site has never done well on Google, despite ranking near top on most searches in MSN and Yahoo. Every 3-6 months I do some tweaks to try and make the site more Google-friendly - I should point out that these tweaks are all content and presentation based and not to do with fake links or cloaking.
Here is a rough chronology of what we have done to the site in the last two weeks.
1) Moved the osCommerce store to the root directory, placed an .htaccess 301 redirect to make requests for the old urls go to the right place.
2) Installed a contribution to generate static .html product urls rather than the dynamic php ones. This generates 301's at all the old php pages.
3) Installed another contribution to generate a Google sitemap.
4) Uploaded the Google sitemap.
And then the camel's back broke and we went down to two pages indexed.
My gut feeling is that the sitemap submission has caused all the indexed pages that did not have a PR to fall out of the index. Also possible is that Google feels the number of 301's is spammy.
So I'm pretty frustrated with Google at the moment. Even more galling is that they do a decent trade from our Adwords account each month.
My plan now is to make our case on the Google Groups and by email and see if we can get crawled properly again. I don't care if our pages are last on the index, just as long as the content is available!
Nope. Not sandboxed. Gone. As in "no information for this url...".
If I was doing something no-no I would expect this but there is nothing to be penalized for.
This isn't even a competitive niche.
I have no idea why.
Your saying that its very recent you lost the pages?
I and others on this board have been in this state for well over a month now, it will be interesting to see if when we recover you recover, or, you take as long as we did, you should see fluctuations on page count every week or so but as yet (after a month) still not showing all pages that were previousley in the index dropped.
Last week, I updated another set of pages and resubmitted the sitemap.
Last night, the main page I was concerned with (which was on the 1st page) is now gone.
The latest crawl found some problems with URL case-sensitivity in the sitemap. I fixed those today, but could that have caused a ton of properly listed pages to drop?
I may need to go back to my standard line of "I'll submit and do my best to optimize, but no promises, guarantees or refunds on SEO".
Puzzled...
Chris.
Why?
Everybody asks this question!
Ironically I built this site with hundreds of keywords that would not be sandboxed (very uncompetitive) I figured a few hundred visitors a week would be better then sitting in the sandbox for a year waiting/hoping for the big money terms. So to have almost all the pages taken out of the index so soon is a real choker.
I guess trying to be smart didn't help after all!
If you are showing session IDs as well, then it is "Game Over".
The contributions you have added will work in the long term, but in the short term you are going to need to allow several months for Google to sort out where your site has gone (changing the URL for every page of the site means that it looks like it has "moved"; and it might even be flagged as a "new site" now).
Maybe the site: command is broke, not unlike the link: command. Perhaps the pages are actually there.
