Missing homepages: MSN bug with index.html?

[Marcia]

I've had a missing homepage on a site for months (and another that slipped out 100%), and the homepage on another couple of sites just went URL-only, meaning they are also about to disappear. Checking into it, this is what I've discovered:

http://www.example.com = no record at all of that URL in the index
http://www.example.com/ = no record at all of that URL in the index

http://www.example.com/index.html - shows cache, title and description snippet

And when doing a site: search - other pages turn up but NOT the homepage.

There is not one single, solitary link to the index.html on any of them, they are all to the actual root. Then either the site slips out altogether or the rankings tank. Obviously, there are no links to index.html - so consequently, no link juice to pass on to interior pages. Besides, the main index.html doesn't come up at all with the site: search either.

Not every webmaster can re-configure Apache at root level to force a redirect, nor should people have to. I know .htaccess can *sometimes* solve it on SOME servers, but not all - and webmasters should not have to call in and pay tech people to adapt to a problem like this.

So - if you have a missing homepage, check for the root URL with and without index.html and see if that's what made it disappear. That's what I've found on mine that disappeared - and the .htaccess 301 doesn't work on all servers, I've tried.

MSN Live Search is getting it wrong with figuring out what the actual root of sites is - which I'm sure is fixable.

[edited by: Marcia at 9:18 pm (utc) on Feb. 5, 2007]

[Marcia]

Oh yeah - also, I'd like to personally request that some courtesy be shown, and that this not be turned into yet another useless MSN bashing session. They are NOT the only search engine to have an occasional bug - not the first and not the last, and we know some others by name. ;)

This thread was started to address a specific issue that will hopefully lead to solving some people's problem (including my own), once we can get some input.

[gehrlekrona]

Yeah, I agree (not bashing MSN) even if I do it a lot but I can't help getting annoyed when other search engines can get it right and they can't.
I do not mind them removing sites that they think is spammy, even my own, but when they remove sites that are totally clean whatsoever then *I* get annoyed with them.
If you have a couple of sites that are listed among the top ten sites in a theme, and MSN bans them then there's got to be something wrong with how MSN indexes sites. It can be 301's that they can't handle, it can be links that are not counted for anymore. It can be almost anything and it sure would be nice to know so we could do something to fix our sites if there is the problem on our end, or if MSNDude or someone else would look into it and see if it is on their end.
The results I see now doesn't make any sense.

I tried your suggestion but didn't work for my site(s). You might have a little piece to the puzzle but I am guessing there are more pieces missing to get the whole picture of what went wrong with MSN in December.

[Marcia]

OK... let's stay on this one specific issue because if it is definitely affecting me, then it's affecting other people also.

I tried your suggestion but didn't work for my site(s).

There can be differences in how the servers are configured. The redirect fix is working with one host, but not another.

This is a specific issue, and I'll be letting the manager of the hosting company know so they can check it out on their end. I'll show him the specifics on sites, but if someone can come up with some more info on causes, then there will be more to tell him and it may also solve the problem for other people too.

[CainIV]

The redirect fix is working with one host, but not another.

I suspect this could be part of the issue. Sites where I use a 301 in htaccess list two copies of the root - one as only the url and cached, which shows a document moved page.

As well, the second copy is the real root url, with current cache date.

Marcia have you checked the redirect with a head checker, such as the live heard checker in firefox?

[Marcia]

I've checked headers and they were normal - thanks for the reminder. But what I found is that redirecting from .html works on some hosts - but on another one it doesn't, I had to change the root index page to index.htm and that worked when redirected. I tested using both and only one worked, the other didn't. I'm use this right after the non-www to www. in htaccess (either htm or html or both)

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index\.html\ HTTP/
RewriteRule ^index\.html$ http://www.example.com/ [R=301,L]

So let's see if this helps with a few sites, though I really don't think we should have to do this. BUT one homepage came back correctly after implementing, so I'm assuming that was the problem and it'll reflect now that I've done it on the others.

[Marcia]

Both homepages are back, one has today's cache date and the other is yesterday's date. One is with redirecting index.htm and the other index.html - since I moved hosting on one of them it's working differently.