But it can be included in the programming, can't it? "IF you're redirected from your target URL to the identical URL plus or minus www THEN follow the redirect, ELSE report back to me."
Search-engine robots definitely distinguish between www redirects and "real" redirects. You can see them hippity-hopping in the logs, making two consecutive requests for the same page: a 301 followed by a 200.
what's that Jack Benny line?
"Your money or your life!" "... I'm thinking! I'm thinking!"
I went off to investigate a few random logs and instead found another goofy UA from a blocked IP (China).
Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 4.0) Opera 7.0 [en]
Uh... Make up your mind, willya? Come to think of it, didn't someone post about this variant recently?
Anyway, quick eyeballing suggests that what I get the most of is not 301-plus-200-- except Yandex, which can't get it into its head that it's with www-- but 301-plus-403.
The ones that intrigue me are the ones that ask for robots.txt at the wrong address, get redirected, and never come back. Makes it seem as if they never really wanted it in the first place, doesn't it?