Forum Moderators: open
The basics of character encoding - US-ASCII
In the beginning there was binary - all information is stored as a series of ones and zeros, or "on" and "off" - the heart of computing and electronics. In order to display alphanumeric characters, a standard was created which defined which binary sequence represented which character. This was the American Standard Code for Information Interchange, or ASCII. There were a few variants, the most well-known by far being US-ASCII, still in widespread use today.
With ASCII, each character is represented by a single-octet sequence. One byte, one letter. The biggest weakness with US-ASCII is that it only includes characters used in English, excluding any accented letters or regional variations such as the German double S.
Stage two - the ISO standards
To fulfil the demands on users which required more than the basic a-z / A-Z sequence, extensions to ASCII were developed and approved by the ISO. The best known are the ISO-8859 series, which used the same sequences as ASCII but added extra characters for accented letters and regional variations. ISO-8859-1 is for most western European languages such as English, French, Italian...
ISO-8859-1 versus windows-1252
ISO-8859-1 became the standard encoding for most Unix and Unix-like systems. However when Microsoft developed Windows, it used a slight variation on ISO-8859-1 commonly known as windows -1252. the differences between the two boil down to 27 characters (including the Euro symbol, certain angled quote marks, the ellipsys and the conjoined oe or "oe ligature") which windows-1252 uses in the place of 27 control characters in ISO-8859-1. Within Windows, ISO-8859-1 is silently replaced by windows-1252, which often means that copy/pasting content from, say, a Word document left the web page with validation errors. Many web authors incorrectly assume that the fault is with the characters themselves and that using entity references is the only way for accented characters. In fact, if you are using a western European language and ISO-8859-1 most accented characters suh as é è û î etc. can be used without resorting to entities such as
éœCharacter encoding on the web - HTML entity references
In order to get around character encoding problems on the web, a method was introduced to "encode" non-ASCII characters in HTML without having to change charsets away from the widely-supported US-ASCII. Accented characters such as é (e acute) can be encoded as
éThere are several weaknesses with the entity references approach. Firstly, they are excessively verbose - in ISO-8859-1 an e acute takes up one byte of space, whereas the entity reference takes up 8 bytes. The second problem is that the are only useful in the context of a parsed HTML document - read the source code as plain text and the result can end up verging on gibberish, especially of you are using a language which relies heavily on accents, such as Polish. Even in French, if you want to write the phrase à côté it ends up as à côté.
HTML entities are a tag soup solution to a tag soup problem, and this is seen clearest with the third problem with entity references - XML.
HTML entity references, RSS and XML
The entity references "solution" falls down once you start working with XML. Unlike HTML 4.0 or XHTML 1.0, which have DTDs which define the entity references, most XML does not have a doctype declaration, so none of those entity references are valid. What's worse, as XML doesn't share HTML's liberal error-handling, using undefined entities will break the document.
XML actually has ony five defined entity references, the bare minimum required for functionality. They are:
& ' " <>The most popular use of XML on the web at the moment is RSS and syndication. RSS is an XML format, so if you are using entity references, for example held in a database, then you will have difficulties producing a valid RSS feed. What's more, encoding directly in, say, ISO-8859-1 doesn't completely solve your problem as you are limited in the character you can use. Want to add a copyright notice in you feed? In HTML you can use
©One encoding for every language - Unicode and UTF-8
In order to overcome the hodge-podge of incomplete, conflicting and aging standards (the ISO-8859 series date from the early 1980s), the notion of Unicode was developed. The differing versions of the ISO-10646 standard (Unicode has been approved by the ISO) are beyond the scope of this very brief introduction, but the important thing to note that is different with Unicode is that it offers one single character encoding for all of the world's languages. The second difference is that it is a multi-byte implementation rather than a simple one-byte per character representation.
By far the most important Unicode version on the web is UTF-8. This standard have numerous advantages, the most important of which is that it remains compatible with the much earlier US-ASCII standard. In fact, all of the single-byte ASCII characters are represented in exactly the same way in UTF-8. Only extended characters are different, made from multi-byte strings defined for each character, whether an e acute, an oe ligature, or characters from Arabic, Russian, Urdu or Japanese.
UTF-8 is especialy important for XML as it is the default encoding for all XML documents. And as you can't use HTML entity references and earlier ISO-8859 standards are incomplete, UTF-8 is the only logical choice when dealing with XML formats such as RSS or Atom which, even if you are only using English, are more than likely to eventually need more than the basic ASCII charset can offer.
UTF-8 is incredibly useful in HTML/XHTML too - no more entity references, the possibility to use extended characters such as curly quotes or long dashes, the possibility of using one charset across a multi-lingual site.
The downsides to UTF-8
There remain a few hurdles to UTF-8 acceptance, most of which can be minimsed or overcome.
- Browser support is excellent, with IE5.x up supporting UTF-8 fully, as do Mozilla/Firefox, Opera, Safari, Konqueror, etc. However earlier browsers such as IE4 and NN4 have problems, and IE3/NN3 and earlier lack support. Bear in mind that documents using markup older than HTML 4.0 cannot use UTF-8.
- The scripting language PHP (and some others) can have problems with multi-byte strings. See an excellent earlier WebmasterWorld thread by ergophobe: UTF-8, ISO-8859-1, PHP and XHTML [webmasterworld.com]. However if you check out how beautifully the PHP-driven WordPress handles UTF-8 content, it is clear that UTF-8 and PHP can successfully mix.
- Just because you can add content in, say, traditional Chinese to your site doesn't mean that the end-user has an appropriate font to display it - you still need to test and ensure compatibility when it comes to defining font families and such for your target audience.
How to implement UTF-8 on your site
If your site's language is English, simply swapping your ISO-8859-1 meta tags to UTF-8 goves the impression that you have succeeded. However, there is a little more to it than that. You still need to ensure that any non-ASCII content is correctly encoded. Users of other languages will almost certainly need to convert their files to UTF-8.
Most modern text and wysiwyg editors handle UTF-8 perfectly - in most cases, it is simply a case of going to "Save As" and choosing "UTF-8" or "Unicode" from the options. From then on, you can tidy up any entity references and start using the true characters. One useful tip is to copy/paste from a word-processing program such as Word which automagically replaces, for example, straight quotes with the appropriate "curly" opening and closing quotes.
If you are using a Linux or similar Unix-like server or desktop, you can use
iconvConclusion
If you are serious about standards, character encoding matters - even if you are just producing content in English. UTF-8 offers huge advantages, and you have everything to gain by moving to UTF-8 for new content.
Further reading
If you want a better or more detailed introduction to Unicode and character encoding in general, try some of these links:
if I do it in an .php file, I get a box with 4 letters in it.
Do not use UTF-8 encoding in the PHP-script file or, more accurately, do not use hi-bit chars within the script.
PHP is in the same situation as Visual Basic 3 - the file must be 8-bit, and the chars used must lie within the 0-126 range (accurately, 20-126 + CR, LF, Tab etc., etc.). Within those limits it may be UTF-8, Windows-1252, ISO-8859-x, whatever. Although, including the UTF-8 BOM may well cause problems (I have no personal experience of that). I produce php-scripts under UK-localised Windows with no problems. They are saved with Unix line-endings, but otherwise are bog-standard Windows files.
A php-script such as the previous paragraph can handle *any* encoding without error, as long as the script file itself is 8-bit, etc. etc.. There are 2 considerations:
Clear as mud?
Added: "mbstring" == "multi-byte string" and is principally designed for 16-bit encodings, which use 2-bytes for each char, thus allowing ~65,000 different chars.
I make most of pages like regular html files, but I use php includes for navigation and stuff, so the files have an extension of .php. Are you saying that alone is enough to make it impossible to use UTF-8 on those pages without changing the file extension? If so, then I guess there is no way I can change to unicode then - most of my sites are that way - if I would have to change the extension, that changes the url, and would cause all sorts of search engine problems.
Are you saying that alone is enough to make it impossible to use UTF-8 on those pages without changing the file extension?
I did not say that PHP-script files cannot be UTF-8. What I said was: "do not use hi-bit chars within the script." You can use \"][()$!;'. _A-Z0-9a-z- etc but not any characters above decimal: 126 (hex: 7E) (oct: 176) (binary: 01111110) (look at an ASCII chart [lookuptables.com])! As soon as you do the PHP-parser will choke on it.
What it means is, that you can declare $char and $number but not $çhar or $ñumber.
Still clear as mud?
Still clear as mud?
yes!
I guess I don't understand what you mean by '"do not use hi-bit chars within the script."'.
If the file has the extension .php, does that not make it 'within the script'?
And the hi-bit characters part: using 'ctrl/shift:f then 1' (=n with tilde) works when the file has the extension .html, but not .php.
works when the file has the extension .html, but not .php
When the file has a php extension it gets sent first to the php-program (which parses the file for PHP-script elements) and then, eventually, the output gets sent to the client-browser. The problem is the stage in the middle with the PHP-program parsing the file. That program will only accept lo-bit bytes, which is to say bytes in the range (decimal) 0-126.
Getting clearer?
<?php ... ?>
you can't have the n tilde between <?php ...?> tags
everything outside of a pair of opening and closing tags is ignored by the PHP parser...so that answers that.
So far so good. Now for a whoopsie [php.net]...
A valid variable name starts with a letter or underscore, followed by any number of letters, numbers, or underscores. As a regular expression, it would be expressed thus: '[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*'...which contradicts everything I've so far said!
.
Note: For our purposes here, a letter is a-z, A-Z, and the ASCII characters from 127 through 255 (0x7f-0xff).
...
OK, I am discovering the scope of my ignorance of UTF-8 [uk2.php.net]:
PHP encodes UTF-8 characters in up to four bytes
However, it turns out that what has been stated so far is accurate:
bytes bits representation(Each b represents a bit that can be used to store character data.)
1 7 0bbbbbbb
So, a 7-bit char (decimal 0-126) is stored in one-byte with the hi-bit set to zero. This is identical to ALL 8-bit charset-encodings. However, any 8-bit char (referring to the so-called extended ASCII range) (such as ç or ñ, in the 128-256 range) is stored in 2 bytes, or 11 bits ("110bbbbb 10bbbbbb"), and so mind-bogglingly on.
So, what has been stated so far is accurate, it is simply that there is more to UTF-8 than I realised. If you want to use UTF-8 with PHP, do not use the hi-bit chars between PHP-tags.
<?php header('Content-Type: text/html; charset=UTF-8');?>Also, are you specifying a different font in your PHP file?
Like I said in the original post I use the blog CMS WordPress: it handles UTF-8 flawlessly and is written in PHP, so cohabitation between the two must be possible. I'm not a PHP expert, but it would be an interesting exercise to see how WordPress manages UTF-8-encoded strings within the program scripts.
In bluefish (gedit too) when I hold down the ctrl and shift at the same time, then type 'f' and '1' - then let the keys up - I see an n with a tilde above it.
.
If I do that in an .html file and view it in firefox - I see the n with the tilde just fine - but if I do it in an .php file, I get a box with 4 letters in it.
If the former (seen at the time of editing) then the problem is coming from a mis-configured or buggy editor. I think that you should now understand why there are "4 letters in it" (UTF-8 uses up to four bytes for each character) although--without checking--I doubt that UTF-8 actually uses 4-bytes for an n-tilde; 2-bytes seems far more likely to me.
If the latter (as output from the server) then there must be a mis-match between the server-declared charset-encoding (Response header) and the document-declared charset-encoding (<meta> statement) (Fischerlaender spoke of this in msg#6). The livehttpheaders [livehttpheaders.mozdev.org] extension for Mozilla and Firefox is brilliant for checking Request- and Response-Headers.
The final item to check is whether your editor is adding the UTF-8 BOM ("Byte-Order Mark") (which I believe is to declare Big-Endian or Little-Endian byte-order) (don't ask!). The BOM is not required and, generally, you do not want it since it will also screw the PHP-parser.
Finally, as a comment, there are so many references to UTF-8 within PHP bug-reports that it is quite out of order. I do think, however, that this is because of bugs within the Linux utilities re: uft-8 as much as php itself.
