Author Topic: strange Base64/UTF-8 behaviour (Umlaute)  (Read 10380 times)

Offline Thomas.Schroeder

  • Newbie
  • *
  • Posts: 4
strange Base64/UTF-8 behaviour (Umlaute)
« on: April 04, 2011, 09:27:45 AM »
Hi,

using Roundcube 0.5.1 I experience a strange behaviour:

I start a new conversation using a simple text containing "Umlaute", lets say:

Code: [Select]
äöüÄÖÜß

My partner answers me using Outlook 2007 (English version). The message I received is messed up when using Umlaut or other special characters. The message reads like this (my partner answered with 'äöüÄÖÜß' too, for demonstration):

Code: [Select]

äöüÄÖÜß


If I answer this message again and get another reply, everything is back to normal. So, msg #1, #2 are corrupt, msg #3, #4, ... are okay.

If I check the corrupt text using Thunderbird, all is okay and readable. When I check the source of the mail, I see that the answer of my partner (who's using Outlook) is base64 encoded. So I tried die decode using different web-tools. The "äöüÄÖÜß" I should receive is encoded to
Code: [Select]
77u/DQrDpMO2w7zDhMOWw5zDnw0K

Decoding "77u/DQrDpMO2w7zDhMOWw5zDnw0K" using PHP-Einfach.de - Base64 kodieren/dekodieren (php based) comes out with crap. Using the javascript-based decoder at Kodierung und Dekodierung mit base64 gives the same messed up result.

But if I use the asp-decoder at Safe Base 64 Decoder, it gives the correct result. The same if I use a windows executable program for decoding.

Countercheck: if i use a web-tool to encode "äöüÄÖÜß" I get
Code: [Select]
5Pb8xNbc3w== which should be the right base64 encoded result.

What I can say from comparing some mail sources is that the starting "77u/" seems to cause the trouble. It doesnt matter what text is encoded, if it starts with "77u/" it will decoded incorrect.

So what is the solution? And remember, the 3rd, 4th... reply of the message-thread is correctly showing umlaut and special chars.

Cheers & thnx in advance

Thomas Schröder

Offline alec

  • Hero Member
  • *****
  • Posts: 1,363
strange Base64/UTF-8 behaviour (Umlaute)
« Reply #1 on: April 04, 2011, 01:12:07 PM »
It could be a unicode BOM (Byte order mark). Provide a sample message.

Offline Thomas.Schroeder

  • Newbie
  • *
  • Posts: 4
strange Base64/UTF-8 behaviour (Umlaute)
« Reply #2 on: April 05, 2011, 03:22:04 AM »
Quote from: alec;34087
It could be a unicode BOM (Byte order mark). Provide a sample message.


Here's a sample mail. I sent the starting mail with RC 0.5.1 to my girlfriend, and she answered using Outlook 2007, I guess (Englisch version). If I reply to that message and get an answer, the answer I can read normally, but msg #1 and #2 (see attachment) are still messed up.

I used Roundcube to save the .eml and the FAR built-in editor to edit & save the message.
« Last Edit: April 05, 2011, 05:12:09 AM by Thomas.Schroeder »

Offline Thomas.Schroeder

  • Newbie
  • *
  • Posts: 4
strange Base64/UTF-8 behaviour (Umlaute)
« Reply #3 on: April 05, 2011, 03:29:40 AM »
Quote from: alec;34087
It could be a unicode BOM (Byte order mark). Provide a sample message.


Hm, you might be right with the BOM. I've just checked Byte order mark - Wikipedia, the free encyclopedia, and in the provide table there are the chars I always see at the beginning of a corrupt msg: "" (the representation of UTF-8 BOM in ISO-8859-1).

Offline alec

  • Hero Member
  • *****
  • Posts: 1,363
strange Base64/UTF-8 behaviour (Umlaute)
« Reply #4 on: April 05, 2011, 04:57:42 AM »
It works for me. BOM is removed by Roundcube properly. Maybe this is environment-dependent or config-dependent issue.

Offline Thomas.Schroeder

  • Newbie
  • *
  • Posts: 4
strange Base64/UTF-8 behaviour (Umlaute)
« Reply #5 on: April 05, 2011, 05:11:09 AM »
Quote from: alec;34100
It works for me. BOM is removed by Roundcube properly. Maybe this is environment-dependent or config-dependent issue.


Ok, thnx. So I'll get in contact with my server admin. I found no solution/workaround for that issue on the net.