301 Redirects, SSL and New sitemap pending over 3 weeks, lost over 50%

301 Redirect, Sitemap, HTTPS, Fetch

[WebCrawlerUK]

My URL is: https://www.example.co.uk

Greetings fellow WM's esp Phranque!

My first post so please be kind :)

Hoping someone may be able to shed some light on this this one...

Had HTML site running for over 4 years, (http non secure) cirxa 100 backlinks including DMOZ. 22-23 pages indexed. Sitemap updated every 6 months.

With the looming "mobilegeddon" decided to revamp website and switch over to WordPress.

In addition, although not necessary as we don't collect any sensitive data, I wanted to experiment with SSL as per Google/Matt Cutts indication that the future ranking consideration will be given more to SSL enables sites etc...

New site went live on 22nd April, created a 301 redirect in .htaccess for http to https. In addition, as URLs have been updated to comply with best practice (underscores replaces with hyphens) had to include redirect for 22 of the indexed pages to new url example as below

Redirect /wooden_flushcasement_windows.html http://www.example.co.uk/wooden-flush-casement-window/

Note: I wasn't sure if I should redirect to http or https hence just left it as http.

The WordPress has Yoast plug -in with auto sitemap enabled, this was new style of sitemap hence I disabled and simply created sitemap using xml-sitemaps.com. Tested & Uploaded via webmaster tools. 48hrs later still pending...?

Thought it might be site url, note apparently https is not currently supported in webmaster tools to change URL so no joy there.

Over the last 3 weeks I have noticed index has fallen fro 22 to 8 (screen shot attached [snip])

I have re-uploaded sitemap 3 times to no avail, each time it is in pending state?

Desperately used Google Fetch to try and encourage bot to index new structure. (screenshot: [snip])

Noticed in WMT yesterday that all back links have disappeared? (screenshot: [snip]) No search info so the downward spiral continues...

No crawl errors (screenshot: [snip])

Would like to consider myself as having a pretty good understanding of webmaster tools, Google, crawlers, bots, blah blah blah however on this occasion I am just unable to identify what the issue is?

I would be MOST grateful to anyone that might have some idea of what is going on...?

Many thanks

Irfan

[edited by: phranque at 9:58 am (utc) on May 13, 2015]
[edit reason] exemplified domain [/edit]

[phranque]

welcome to WebmasterWorld, Irfan!

...created a 301 redirect in .htaccess for http to https.

you should add another site in GWT for the https: version and start tracking that as well.

In addition, as URLs have been updated to comply with best practice (underscores replaces with hyphens) had to include redirect for 22 of the indexed pages to new url example as below

Redirect...

did you use mod_rewrite for the http: to https: redirect?
your "Redirect" directives are mod_alias directives and shouldn't be mixed with mod_rewrite.


if you start separately watching 4 sites in GWT you should learn few things about what is happening:
https://www.example.co.uk
https://example.co.uk
http://www.example.co.uk
http://example.co.uk

they should all show different and expected results.

[WebCrawlerUK]

Hi Phranque,

Thanks for the reply. I have added HTTPS version to WMT and submitted sitemap so hope it will get picked up soon.

Regarding the redirect current entry is

RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteRule ^(.*)$ https://example.com/$1 [R=301,L]

Should I leave or change it to https://www.example.com ?

Thanks!

Irfan

[WebCrawlerUK]

Sorry one more thing to add, the 301 URL redirects are to http

e.g. Redirect /wooden_flushcasement_windows.html http://www.example.co.uk/wooden-flush-casement-window/

Should it be to https?

[not2easy]

Hi Irfan and welcome to the forums. First thing I suggest is that if you are using Yoast and this is old static html pages moved to WP URLs that you might want to use the features in Yoast's software to generate your sitemaps because it lets you control which URLs are submitted and which are canonical. WP will generate several different URLs for the same content and you probably don't want to submit all the possible URLs to be indexed. If you create a page called http://www.example.co.uk/blue-widgets/ you don't want to also have http://www.example.co.uk/widgets/blue-widgets/ and http://www.example.co.uk/blue/blue-widgets/ and http://www.example.co.uk/2015/blue-widgets/ and http://www.example.co.uk/products/blue-widgets/ listed. Limit the sitemaps to one permalink structure (one type of URL) and add the canonical meta tag to the rest of the URLs for the same content.

As phranque mentioned, you will need to add the https version of your site to GWT as a different site and the issues with your sitemaps may be caused because GWT has not yet verified the new https: site while the URLs in your sitemaps would be at the "new" domain. There are some steps to take for sitewide URL changes: [support.google.com...]

It is only natural that the number of indexed pages listed in the old domain GWT account are fewer, but by adding a "new" domain with the https: you can see the number increasing.

How and where are these Redirects applied? Have you tested to ensure that they are all working as expected - I mean tested by using a tool to view headers? As phranque says, you should not mix Rewrite and Redirect in your htaccess files and it is important to have Rewrites in the proper order to avoid rewriting the same request under multiple rules. It may take you where intended, but with less efficiency.

[WebCrawlerUK]

Hi not2easy,

Thanks for the welcome!

For now I actually disabled the sitemap feature in Yoast and instead generated sitemap manually and submitted on the new HTTPs site version.

Still waiting for the new HTTPS pages to register and show up in GWT.

Regarding the .htaccess, I do have both Rewrite and Redirect in the same file.

First ReWrite is the HTTPS at start of the file

RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteRule ^(.*)$ https://example.com/$1 [R=301,L]

followed by # BEGIN W3TC Browser Cache
<IfModule mod_mime.c>

with list of file types

then in order of: (have left out the conditions just the headings)

<IfModule mod_mime.c>
<IfModule mod_expires.c>
<IfModule mod_deflate.c>
<IfModule mod_headers.c>

2nd mod_rewrite

# BEGIN W3TC Page Cache core
<IfModule mod_rewrite.c>

3rd mod_rewrite with example line of redirect

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
Redirect /Hammersmith_W14_SashWindows01.html http://www.example.com/gallery/hammersmith-w14-double-glazed-sash-windows/

From what you have mentioned, I suspect that I shouldn't be doing it this way?

Thanks

[edited by: Ocean10000 at 7:59 pm (utc) on May 13, 2015]

[edited by: phranque at 7:35 am (utc) on May 18, 2015]
[edit reason] examplfied. [/edit]

[lucy24]

Should I leave or change it to https:/ /www.example.com ?

There is no evidence whatsoever that with-or-without www makes any difference of any kind anywhere. Pick the version you like best and stick with it.

Redirect /

Do not repeat DO NOT do this. If you have any RewriteRules at all, you must use mod_rewrite for all redirects. Mixing mod_alias (Redirect or RedirectMatch by that name) with mod_rewrite will not break your server. But the rules will not execute in the order you want, so things will get mixed up.

If you have a great many existing redirects using mod_alias, you can convert them globally in any text editor that uses Regular Expressions.
:: shuffling papers ::
My text editor uses the \1 locution for captures; replace with $1 if needed.

# change . to \. in pattern
^(Redirect \d\d\d \S+?[^\\])\.
TO
\1\\.

# now change Redirect to Rewrite
^Redirect(?:Match)? 301 /(.+)
TO
RewriteRule \1 [R=301,L]

# and if needed
^Redirect(?:Match)? 410 /(.+)
TO
RewriteRule \1 - [G]

^Redirect(?:Match)? 403 /(.+)
TO
RewriteRule \1 - [F]

[not2easy]

You can create and check your sitemaps manually if you prefer, the reason I suggested using Yoast for that task is that it lets you select the URL format you prefer as the primary URL for your content and it can automatically add the canonical meta tags to all other URLs for the same content.

IF your old site at that domain was previously redirected to www it might make a difference, otherwise, no difference at all. Google views the URL changes as a "New" domain.

When editing your htaccess file with WP installed, you will see a section of WP generated code in the htaccess file, usually after all your rewrite rules. It is best to leave that section to whatever is generated because it can and will be overwritten during core WP updates.

[guggi2000]

Sorry one more thing to add, the 301 URL redirects are to http

Why? Weren't you moving everything to HTTPS?