Reno - 9:54 pm on Oct 29, 2007 (gmt 0)

did MSN ever stated that they support wildcards and/or regular expression syntax in robots.txt?

That's a good question and I admit I don't know -- to be honest I thought it was a standard format for keeping a bot out of a specific folder.

And I do want to emphasize, until this latest weirdness they did more or less stay out. What is really strange is how almost all of the straight-forward html dropped out (which WAS well indexed), and the cgi pages came in. It was like a complete reversal! Very strange.

.................