MSN spider and https and 302s

Spider not moving from home page

[bmsd33]

My client's site is entirely secure, i.e., "https" from home page on (don't ask me why, no one knows and he isn't going to pay to change it now). MSN indexes the home page URL only (no page title or description) and it starts with "http." Log records indicate the spider is now only visiting once every couple of weeks, going to the robots text, then home page and that's it.

Looking through an http header response viewer, I saw that the home page that starts with a "http" employs a 302 redirect to the same home page under "https."

Does anyone know if either the 302 redirect or https pages is causing a problem for the MSN spider?

Thanks

[Span]

Anything in robots.txt that is stopping the bot? And I think the 302 should be a 301 Permanent.

[bmsd33]

There's nothing in the robots.txt to cause it.

I would love to change the 302 to a 301, but I cannot figure out where I go to do that. Any Idea?

[Receptional]

A major search engine spider is unlikely to spider an https site I would have thought. It would not be a good "image" for secure pages to start appearing in the serps, so if you change the 302 to a 301, my guess is that you will lose the only page you have got in the index! What you need to do instead is to bring the pages that you want indexed OUT of the secure url.

My guess is that you could choose and https:// page, take the S out of the URL and it will still parse? If so, then you problem seems easy enough to fix - just stop using https until you really need it.

[jd01]

1. Not specific to MSN... try 302 redirect and search engines in your favorite SE, it should give you something to go on.

A 302 says temporary move to a search engine, so whatever shows on the page will not be parsed or included in the index because that page has been temporarily moved to an https location. SEE 2 below.

To change this depends on where the redirect is set. If it is in the root - httpd.conf file - you will need to ask the host to change it or remove it and create your own. If it is in the .htaccess file, you will need to add R=301 to the RewriteRule or 301 to the Redirect that does the redirecting.

2. https is not crawled, indexed or parsed for content. If your client wants pages in the search engines he will pay to have them moved to http pages. If not - there is not much you can do, unless you can convince G, M and Y to change their policy.

Justin

[bmsd33]

Thanks for the feedback. It looks like a follow up post I sent indicating the site was built on ASP.Net was not posted.

I think I found the source of the redirects: it is from the default.aspx.cs file. Now I need to figure out how to code it, but I think I know how.

I thought search engines had a problem with https pages, too, but Yahoo and Google are indexing it pretty good with no problems. MSN and Ask seem to be the ones with an issue.

Unfortunately, the site was built before I came onboard, and, without going into detail, there are certain reasons we have to keep it all secure at this point.

I believe we can fix the possible MSN issue by 301 redirecting the http home page they have indexed to https and then, hopefully, it will crawl more.

Thanks again.

[msndude]

Hi bmsd33,
Thanks for taking the time to write about this issue. At this point we do not support https. I cannot say for certainty what is causing the issue you are seeing, however, that is a definite possibility.

We do plan on adding support for https in the near future.

- msndude (msd)

[bmsd33]

Thanks for your feedback.