Maurice - 11:51 am on Oct 31, 2012 (gmt 0)

@lucy24 the problem is a lot of sites are using poor quality CMS (some one mentioned joomla) which default to EN or they are devlopers who have ZERO knowledge of the varuious language encodings (and this is a very complex subject even for UTF8 let alone UTF16 and Non latin languages)

If part of the html spec is so debased by using it incorectly then just ignoring it makes sense - less chance of a false posative.