Forum Moderators: Robert Charlton & goodroi
I tried searching for snippets of my content in quotes to see if there are any scrapers outranking me. None.
Any ideas why this would happen.
You've said the pages are different ... but do they overlap in content?
For example, are the pages machine generated or human-made?
Is the content per page fairly small?
Does each page carry a heavy (and identical) load of site navigation, headers, footers, promos etc?
Code bloat - if that's the issue - is a major cause of sick site syndrome.
It sounds like you've triggered a filter which has caused Google to believe your site/page is of low quality. Not always an easy problem to troubleshoot. A meta description with the 'wrong' words in it is enough to get a page dropped in this way, for instance.
Added: it sounds like you mean the entire site from the above, which would certainly be a severe reaction to your changes!
[edited by: Receptional_Andy at 7:59 pm (utc) on June 18, 2008]
Is the content per page fairly small?
If the text content is small per page, that would be enough to banish the pages from many search results.
textex said:yes...unique titles and descriptions...
I changed the entire structure of the site creating folders and moving pages there using redirect 301s.
I also changed the footer on my site...removing a plug "top supplier of widgets since 1998!" to thewebsite.com copyright 2008.
Seems like we have been sandboxed?
Anyone?
[edited by: textex at 3:13 pm (utc) on June 19, 2008]
Another place that's worth a look is messages within Google webmaster tools.
It certainly sounds a little more serious.
I'd do the standard Google problem check - every outgoing link, starting with reciprocals. Make sure that all are going where they should, and all are sites that you'd be happy with if you were Google.
Then check them all again. Have no mercy. "Links" has to be the first place to look with a serious, unexplained Google problem. And the sooner the better.
I changed the entire structure of the site creating folders and moving pages there using redirect 301s.I also changed the footer on my site...removing a plug "top supplier of widgets since 1998!" to thewebsite.com copyright 2008.
Seems like we have been sandboxed?
your site is either spidered *right now* or put up for manual review because of the irregularity. if by sandboxing you meant 'put on hold until further notice'... could be. Chances are you reset most of your deep links' age with this move ( if you had any, that would be a trust issue. if not... see below )
Let me clarify, the site is indexed
I'd say it *was* indexed.
Basically you removed most of your content and left notices where Google can grab the pages from. While they may do it eventually, it's not like it's instant...
also this 'change' fits the 'domain changed hands is now MFA' profile all too well. Right? All pages gone, completely new structure, different site-wide footer copyright notice...
You could be up for a manual review that'll get you out.
How long has it been like that?
I'd wait and see how actively Googlebot is fetching the new URLs and if after it re-read your entire new set 3 times for 2 weeks and still no listings, I'd file a reinclusion-consideration request to ASK for a manual review. Would do the same if it doesn't request the new URLs at all.
...
I'd file a reinclusion-consideration request to ASK for a manual review.
Do that ONLY if the problem is as described by Miamacs (which seems quite possible!).
If there's ANY chance that the site has incurred a penalty, deal with the CAUSE of the that penalty FIRST.
Then you are experiencing normal behavior. Things should return to normal once all old pages get purged and new locations found. Helping speed that along is what you need to focus on now.
Get a google site map.
Check your site navigation and clean up widows and orphans (xenu is your friend).
Where you have used 301 permanent redirects, check that they actually work!
Be sure that domain.com forwards to www.domain.com (or vice versa).
Continue safe and quality link building; see if you can get quality +expired links renewed with the new URLs
Read this before you next do any serious reorganizing: Cool URIs don't change [w3.org]
Then you are experiencing normal behavior
Even with changing every URL on a site, I've never come across an effect like this. For me 'normal behaviour' after URL migration is a little bit of instability: you may see both old and new URLs in the SERPs for a while as Google finds new locations via direct links, before finding the redirect at the old location.
My opinion is that unless you have a real need for one, avoid XML sitemaps, since they add an extra complication in troubleshooting spidering issues.
So, unless there were problems in how the migration was implemented, I think there is something else at play here.
I would second the advice to get any 'controllable' external links pointing at the new locations, and also the suggestion to avoid further significant change (other than technical fixes) until you're confident you have found the source of the current problem.
Even that is no guarantee, but it can be a valuable and effective way to get new URLs spidered, provided that (as I suggested), your site navigation has been previously checked.
If you use webmaster tools (and if you don't, now is the time to start!), then using a Google sitemap can aid diagnostics too.
Basically, when you have a catastrophic problem like this, you have two choices; treat Google and all its works as your enemy - or use Google as an ally to both diagnose and manage the problem. Personally, I always advise the latter, and have never had cause to regret it. Your mileage may vary ;)
If the site has largely disappeared from Google, then you have a real and urgent need for an XML sitemap.
I'm afraid I disagree ;) My take is, if your site has disappeared from Google, you need to work out why. A sitemap might give more comforting results in site:searches, but it risks covering up the genuine problem, and making it much harder to troubleshoot. It isn't going to fix ranking problems.
This isn't to do with working with or against Google - the issue is ensuring the domain is correctly spidered (by links) and is of sufficient 'quality' to be included in results. I personally don't think a sitemap helps with either. But I agree that YMMV ;)
I changed the entire structure of the site creating folders and moving pages there using redirect 301s.
Conceivably, there's something you've done in the redirect process that's preventing old link reputation from being transferred... and also causing Google to regard the new pages as dupes and the old pages as original sources.
I'm assuming you've checked server header responses on your new urls and have gotten a 200.
I myself don't redirect all old pages when I make a structural change... only the important ones... those which had inbound links and/or good rankings. Whichever way you handled it, take a look at those "important" old urls and make sure that they have been properly redirected... and that a header check on those shows a 301.
If you'd done something like redirect all of your old pages to the home page in your new site structure, that might perhaps create problems.
Conceivably, a sequence of redirects in your .htaccess could also be creating problems.
[edited by: Robert_Charlton at 10:59 pm (utc) on June 22, 2008]
Link to the old URLs until the old URLs are removed from the index. Never link to the new pages directly until you change all the URLs to the new ones (after the old pages are out of the index).
Again, this is totally normal for Google now. If they find a new duplicate of content on an existing URL, they will cache the new page but not have it appear in the results. It should be nothing to worry about if you do as above and only link to the old URLs... even then google will sometimes get confused for a brief while, but they are picking up 301s pretty quick now so there should be only a short time (say under a couple months) where you might have content go into lost purgatory.
Ancient history now. Google is far more likely now to cache the duplicate but not display it in the results --- essentially the new page is like a supplemental, it can't rank for anything.
"I think there is something else at play here."
Yes, a more ruthless approach to duplicates, which is overall good, but means 301 changes must be done much more carefully.
"I would second the advice to get any 'controllable' external links pointing at the new locations"
Definitely backwards. Remove any links to any new URLs. Force Google to discover the new URLs via following the 301s. (This also means avoid using an XML sitemap. This will cause total grief. Wait to do an XML sitemap for the new URLS until the old ones are gone.)
