Marcia - 1:04 pm on May 2, 2007 (gmt 0)

>>Can I see one such example, may you can pm me. I have never seen such a problem.

No, because after it caused a disaster at MSN I've 301'd index.html or index.htm to the site root/ so no one will be able to see it, it'll redirect. You'll just have to take my word for it that exactly that very issue caused a couple of sites to totally bomb because the engine got it wrong (not Google, but why play with fire - MSN Search sends BUYERS). Besides which, I don't lift my skirt in front of strangers, and I don't know you. ;)

>>I have always Google taking care of it.

No, it isn't always taken care of, and for very good reasons. index.html may or may not be the root of a site, they are not necessarily the same page and often aren't. The root could be index.php or default.htm or whatever - and don't think for one minute that there haven't been people who spammed the h*ll out of the engines with multiples, using massive numbers of IBLs with different anchor text.

Let's not expect the crawlers to guess what we mean, let's be specific on what the root of a site is so it's made very clear. That's up to us, on our end.

http://www.example.com/ may or may not be the same as http://www.example.com/index.html or
http://www.example.com/index.htm - or any number of variations, which are all different pages.

I've had an instance where MSN Search *lost" the root of a site of mine and instead indexed http://www.example.com/index.html which totally messed things up - there was not one link in the world to index.html and what I had to do was 301 from that to the actual root. It killed the site, because the internal navigation structure and IBLs were totally skewed on their end because of the mistake. The site has never fully recovered, and it's a huge Q4 traffic loss.

Rule #1: Never confuse the bot or make her wonder, think or second-guess. She is not a mind reader, able to second-guess what the webmaster's intentions are.

[edited by: Marcia at 1:33 pm (utc) on May 2, 2007]