Redesign with 301 redirects - rankings are still vanished

[sett]

Hi All,

I have redesigned my website few months back and placed permanent 301 redirect on old pages to corresponding new pages. Before redesigning, my websites was appearing in top search results of Google.com for the most popular search terms of my business industry.

But now my rankings have been vanished from Google SERPs since my website is live with new look & feel.

Can anyone please let me know why it happened and what should I do from here on?

[tedster]

This sounds suspiciously like a technical problem. First I'd check these factors:

1. robots.txt to make sure you are allowing indexing of the new pages
2. meta robots tags to make sure there's not accidental noindex or nofollow
3. http headers to be sure you are returning a 301 status on the redirects
4. 404 handling to be sure the server is returning a 404 status
5. Google Webmaster Tools for their crawling error reports
6. server logs to see what requests googlebot is making and what responses the server is giving

[jdMorgan]

1a. robots.txt to make sure you are allowing indexing of the new URLs
1b. robots.txt to make sure you are allowing indexing of the old URLs, so that 'bots will receive the redirect responses.

Jim

[g1smd]

Run Xenu LinkSleuth over the entire site, starting at example.com/ and then again starting at www.example.com/ and look very carefully at all of the reports.

[sett]

Thanks for your replies.

My robots.txt file contains:

User-agent: *
Allow: /.
Allow: /

My all pages have:

<meta name="robots" content="index,follow">

I have checked server header of my all old URLs they are showing proper 301 redirect towards new corresponding pages.

There is no crawler is appearing in webmaster tools.

My site is designed in volusion and hosted on Microsoft IIS server, Can you please tell me how to check my server logs?

[tedster]

We have a dedicated IIS forum [webmasterworld.com] where you might get some in-depth technical help - it;s not really a Google topic. As a top view, the server must first be configured to create the log files. And then there are a few options that can be important - such as recording the referer string and not just the request, user agent and IP address.

However, googlebot won't send a referer, so it wouldn't matter if this is the only reason you want to use it. However, you certainly need to look at the IP address - there are lots of rogue bots that spoof the googlebot user agent string [webmasterworld.com].

[edited by: tedster at 5:26 am (utc) on May 5, 2010]

[TheMadScientist]

Robots.txt might be part of the problem...

Emphasis Mine:

This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory:

robots.txt [robotstxt.org]

This is correct to allow complete access:
User-agent: *
Disallow:

[tedster]

However, googlebot does accept the Allow directive as an extension to standard robots.txt syntax. In fact, they even suggest this syntax in their Webmaster Help and they use it in their own robots.txt file.

I don't understand the first directive however - the one with the period. As an Allow directive, it adds nothing and it "might" be a problem. Do you have URLs that look like example.com/.filename ?

Webmaster Tools will tell you if they wanted to crawl some file that was restricted by robots.txt

[sett]

I have changed my robots.txt file content to

User-agent: *
Disallow:

[sett]

I would also like to add here is that some of my old URLs redirecting permanently towards [mysite.com...] (i.e. my home page).

Should it be only towards [mysite.com...] or [mysite.com...] is OK?

[milosevic]

This could be other things than just spidering errors.

I'd say issues with content would be the next most likely reason. Does your new site "look and feel" involve loading content with JavaScript? Does your new site have the same text content as the old site?

I'd say contact a professional SEO with something this serious, getting to the bottom of problems like this is the bread and butter of a good SEO but there are too many potential red herrings for someone who just dabbles.

[sett]

No javascript involved in content loading.

I have posted new content on my website couple of months ago.

[g1smd]

The canonical URL for the root of your site should be www.example.com/ with trailing slash and without name of default or index file.

[milosevic]

g1smd, I feel that would be unlikely to be a serious issue here unless for example the site had changed from www to non-www with the relaunch and the www now produces an error.

Backlink validation would be the next place to go presuming spidering and content are OK, any problems like that mentioned above would immediately become apparent.

[TheMadScientist]

I feel that would be unlikely to be a serious issue here unless for example the site had changed from www to non-www with the relaunch and the www now produces an error.

Except that using no / invokes a double redirect, which is advised against and, I think, those have been known to cause issues with link weight passing to the final landing page. It should not cause issues with all pages though, AFAIK.

Intra-domain it should be fine to even redirect http://example.com/ to http://example.com/page-name.html, even though IMO it's not usually the best course of action, and as long as there is only one redirect, it should transfer all (most of the) weight just fine.

The best reason I can think of outside of a Google issue is a spidering issue. I'd start viewing source code and double checking those robots meta tags, even though it looks like you already have.

Out of sheer curiosity, do you run AdSense and if so, how many blocks of ads do you have on each page?

If the issue is not technical, could you describe the scope of the redesign... Did page titles and content change too? Were site-wide backlinks added anywhere in the footer or anything? How much and what changed?

[sett]

I don't run any Adsense program on my website.

I have implemented new meta tags and page tittles on my pages few months back.

When I was analysing the source code of my internal pages and found that <title> and <meta description> tags are appearing on 659 and 660 line respectively and there is huge amount of CMS code that is appearing from 1st line to 658th line in page source.

Is deep appearance of title, meta tags and content in source code could be a reason for no rankings in Google?

one more thing, below tags are appearing twice in source code:

<meta name="robots" content="index,follow">
<meta name="GOOGLEBOT" content="INDEX, FOLLOW">
<meta name="robots" content="index,follow" />
<meta name="GOOGLEBOT" content="INDEX, FOLLOW" />

[milosevic]

I'd say remove those tags altogether, all they do is tell search engines to carry out what is their default behaviour anyway.

What language is the code in the <head> of your page in?

[sett]

My <head> section have <script> tag, <link> tag, <style> tag, <title>, meta descriptiin, keywords, robots and googlebot tags.

[tedster]

It is best to have <title> element and meta description near the top. Sounds like you have your JavaScript and CSS all in the <head> section of the page. Move them to external .js and .css files and you've resolved at least this one issue.