Character encoding problem when taking input from emails

[Fyse]

I'm using PHP to run my weblog (in the form of Wordpress) and am trying to set up posting via email, so I can can post from my mobile phone. I am using the wp-gmail script so that I can use a Gmail account for posting.

[snip]

When emailing posts from Gmail, or my email client, I have no problems and the post comes out fine. When sending from my phone, however, the post comes out garbled.

[snip]

The process of saving a copy of the site seems to have mucked up the special characters in the rest of the site, but it's just the first post that is actually the problem.

I checked, and it appears that my phone sends emails encoded in UTF-8, so I think this is the problem. I have tried using utf8_decode in the script, but have had no success. I'm a novice with PHP, and would appreciate any advice on a possible solution to the problem.

Thanks!

PS I tried the Wordpress support forum first, but without luck.

PPS If a copy of an email from my phone would be helpful, contact me at fyse [at] floatingface [dot] com

[edited by: ergophobe at 7:05 pm (utc) on July 9, 2005]
[edit reason] no personal URLs please - see site Usage Agreement [/edit]

[Fyse]

Apologies. I'm a newbie round here, and obviously didn't read the agreement closely enough. I vaguely remember reading about a site allowing you to upload code which can then be linked to, but can't seem to find it now. The script I'm using can easily be found through Google, however (it's the one produced by StealthAngel).

[ergophobe]

No problem and welcome.

I was thinking about the problem, but I'm having trouble figuring out where the conversion is happening. In other words, something is clearly happening when you send from your phone. Can you figure out at which point along the trail it is getting garbled? If you send it to a regular email address, what happens? What if you then take something sent from your phone to your regular email and forward it to your script?

It seems like maybe at some point it's a multi-byte encoding getting interpreted as single-byte or vice versa, or perhaps it's a different Unicode encoding. If you get some funny characters right at the beginning, like a left angle double quote (<< character) and a question mark, that's usually an indication that a byte-order mark is getting sent (which wouldn't be the case with UTF-8, but woudl with other Unicode encodings).

[Fyse]

I'll try to give as much detail as possible, but apologies if I don't accurately discern what is relevant.

I changed the settings in my email client (Thunderbird) to send emails encoded in UTF-8 (default was ISO-8859-1), and these were still posted fine onto the blog. I also copied it back to myself, and the last few lines of the source are...

Subject: Just a test...
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Yet another test post. I'll get this bloody posting via email working 
properly in the end!
Yet another test post. I'll get this bloody posting via email working 
properly in the end!

I sent an identically worded email from my phone, and viewed the source in my email client. The last few lines were...

Subject: =?utf-8?B?SnVzdCBhIHRlc3QuLi4=?=
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64
WWV0IGFub3RoZXIgdGVzdCBwb3N0LiBJJ2xsIGdldCB0aGlzIGJsb29keSBwb3N0aW5nIHZpYSBl
bWFpbCB3b3JraW5nIHByb3Blcmx5IGluIHRoZSBlbmQhCgpZZXQgYW5vdGhlciB0ZXN0IHBvc3Qu
IEknbGwgZ2V0IHRoaXMgYmxvb2R5IHBvc3RpbmcgdmlhIGVtYWlsIHdvcmtpbmcgcHJvcGVybHkg
aW4gdGhlIGVuZCE=

Thunderbird is smart enough to display the email ungarbled when I view it, however, and reads exactly as it should. The garbled version I observe in the source code of the email (through Thunderbird) is exactly what appears on the blog when I email directly from my phone. I assume this indicates they are both mis-interpreting them in the same way?

You mentioned a possible problem with multi-byte encoding, so would it be right in guessing that the base64 encoding in the email from my phone is the problem? Thunderbird sophisticated enough to deal with the difference, where Wordpress is not?

(I wasn't sure of the conventions when posting script/source to the forum. I put it between code tags, but it seem to have come out strangely. Any pointers?)

[ergophobe]

Ahh! Now I see. The output you're getting from your phone emails is base64 encoded. This is commonly used to send emails, especially with extended character sets beyond plain ASCII (aka 7bit), but also binary data of various sorts that then gets sorted out with mime type headers and whatnot. Your email client knows how to decode it, but your blog thingie does not.

You probably need to intercept the messages, check for the base64 header, and if present decode them before sending them on to your blog.

Sure enough, if you run the sample text you gave through a base64 decoder [makcoder.sourceforge.net], you get your text message exactly as it looks when sent from an email client.

See
[us3.php.net...]

Some other reading that may help

[en.wikipedia.org...]
[fourmilab.ch...]
[ietf.org...]

[Fyse]

Ok, so I'll rewrite the script so it checks for base64 encoding, and converts the 'subject' and 'body' strings if necessary, using the function base64_decode(). Right?

Great stuff. I'll let you know if it works... :)

[ergophobe]

Since I'm just theorizing, definitely let us know how it works out. Seems like it should work though.

[Fyse]

Yep, that wasn't too tricky in the end. Decoding the body of the email was a simple as...

$email['body'] = base64_decode(imap_fetchbody($serverHandler,$msgNum,'1'));

For the subject I had to take a substring...

$email['subject'] = base64_decode(substr($email['subject'], 10, strlen($email['subject'])-3));

I haven't yet written it to automatically discern between emails that need base64 decoding and those that don't, but at least I have a script that'll work from my phone now.

Thanks very much for your help!