The final word on unicode

[Francis]

Hi guys

I noticed that the topic of unicode support in Winamp 5 has been mentionned.

Just so that the matter is settled (and to be clear, I'm not writing this to debate *about* unicode), there is no support of unicode coming up in Winamp 5. That's it. End of story.

So some of you may bring this up :



or this :



and say, "to do this, it must be supporting unicode!".

Well no, what you see here are captures I have made of the freeform skinning rendering by itself (as in: not using Win32's DrawText functions) the default charset for an operating system set to japanese or hebrew. Support for showing the correct international strings in the default language selected in the "regional settings" box of Windows is *not* unicode support. What unicode support allows you to do is to display these strings even if your Windows default os language is set to english. Mixing up japanese AND hebrew in the same program is a feature of unicode. Windows explorer, for instance, can do it, so does Internet Explorer, as long as you have the language installed (but not necessarilly set as default).

This means that, as it has been since a long time in Winamp 2, Winamp 5 can display strings in japanese or in hebrew (freeform skins couldn't previously do it, hence my captures, which I have slightly regreted making since), so long as your Windows is set to japanese or hebrew as your default language.

The reason for this has to do with the size of character strings: if you use 8bit character strings to encode text (what Winamp 2 has been doing since the beginning), you must observe an encoding scheme (even if you do not know about it and just take all chars to simply be 0...255), and there is no way to detect an encoding scheme from a given encoded string (it's just random bytes, it could just be ascii after all, ie: no encoding at all), the operating system needs to know *somehow* in what encoding you are when it wants to print the string, so that it knows wether this code is rather an accented e in french, or a 'ia' in russian (i'm making this correspondance up, btw) and so that it can print the correct characters, not just merely ascii. It does this by reading the default language setting in Windows. In 8bits-strings programs, that is being done by the classical Win32 DrawText functions, they do the character decoding on their own, and then print the characters with the requested font (usually unicode charset, but not always, but that's another story). In contrast, when you support unicode internally, there are four bytes to represent each character and these are enough to store all the unicode characters, and there is therefore no need for a notion of character encoding, everything can be in just one native encoding (unicode, in which case the encoding and the charmap 'fuse together' in a sense, or something still 8bits, like UTF-8), and you can stop thinking about it altogether, because all characters in all languages have a unique number. That's what is sometimes called "proper" unicode support.

Now, to be completely accurate, I must say that the freeform engine has some sort of unicode support, but *in the font rendering engine*, in the sense that, since we cannot rely on DrawText to make the character decoding (decoding to unicode according to the default language set in the system) because we do not use DrawText to draw our fonts (we use Freetype which does not provide the conversion), we have to provide our own conversion of encoded strings to unicode, in order to print the japanese or hebrew characters (we also for instance had to implement our own right-to-left routines). This conversion we provide uses the functions that DrawText uses, and these uses the default language setting to do the decoding, we therefore support japanese and hebrew chatacters only if you have the right language set, we cannot mix the two. True unicode support could.

Summed up, the reason true unicode support cannot be added simply (as in, just recompile, silly!) in the Winamp 2 system is that it would render our plugins incompatible, since they would have to all be recompiled for unicode support as well.

There are thoughts being given about what should be done about it, I certainly would be willing, once Winamp 5 is out, to try and make it truly support unicode without breaking everything (ie: supporting legacy encoding, UTF-8 seems to be the key here), but it would certainly be a lengthy task. Perhaps it is worth it.

Oh and let me be clear again, that's not to start a debate about unicode, if you want to have one amongst yourselves, that's fine, but I won't be participating yes, I have my own view about wether or not it should be undertaken, but we're several in the team, and this is a work that would impact everybody, so unless we all agree that this can be done safely (ie: without breaking winamp2) we probably won't be doing it.

Francis.

[SailorH]

Thanks for the information.
So, you are saying that Winamp 5 Beta 1 supports Japanese if my default language is set to Japanese in Regional Options? Can I have my region "Your Locale" set to English still?

[iron2000]

If what Sailor H says is true, I think mine is already set that way.

Most program display Jap, normal desktop icons, WinMX, IE but Winamp still needs a font change to do that.

But I can type Jap into the ID tag window and see Jap characters.

[gugu]

I have some russian musics with some non-standard characters. When I open them in winamp5, both in the playlist and the main window in the modern skin or the classic one, I get wrong characters.
For example, instead of "Dobranotch - ?? ????", I get "Dobranotch - 0 ;CF5". That's very anoying

Will this ever be supported in winamp5?

[DrPizza]

Summed up, the reason true unicode support cannot be added simply (as in, just recompile, silly!) in the Winamp 2 system is that it would render our plugins incompatible, since they would have to all be recompiled for unicode support as well.

What drivel. I mean, it's not like looking for a different entrypoint would be impossible.

[xdcdx]

If what Sailor H says is true, I think mine is already set that way.

Most program display Jap, normal desktop icons, WinMX, IE but Winamp still needs a font change to do that.

But I can type Jap into the ID tag window and see Jap characters.

I have exactly the same problem, I have set 'use japanese as the default language for non-unicode programs', and it works fine for the file system, for soulseek and mirc.
However I can't get Winamp 2 nor Winamp 5 to show japanese characters correctly, any help would be appreciated.

Best regards.

[SailorH]

Originally posted by xdcdx
I have exactly the same problem, I have set 'use japanese as the default language for non-unicode programs', and it works fine for the file system, for soulseek and mirc.
However I can't get Winamp 2 nor Winamp 5 to show japanese characters correctly, any help would be appreciated.

Best regards.

There is an option in Beta 2 to overide (some of the) fonts to replace with international supported fonts. This fixes the problem in some places, but the Playlist still doesn't seem to work. Give it a try, good luck!

[xdcdx]

Hello,

Thanks for the help! But it isn't working yet...
I have downloaded beta 2, and I'm playing with these settings but I can't get it to work properly.
I select the MS Gothic font, and then the main windows shows some japanese characters, but shows them screwed. On the other hand, for example, I have selected the same MS Gothic font on Total Commander (a windows file manager) and all the names are fine there.
I think I have tried every combination of the modern skin internationalization options with no results.

And there's another weird thing, when I select the MS Gothic font, exit from the option screen and enter again, it says I have selected the Wingdings font, but it isn't true, because there aren't wingdings characters in the song name.

What font are you selecting for winamp ? Does it show the names correctly on the main window ?

Best regards.

[SailorH]

I have been using MS-PRGothic as my Japanese font. When I open up the options though it always resets to "Larabie Font" even though it is still using MS-PRGothic. I think that is just a bug in the beta.

As for why you are getting incorrect Japanese, I've found that some of my files also do not show proper Japanese. There are several ways to encode Japanese text, in EUC, ISO and SJIS modes. This bug might be related to Winamp not recognizing which mode the text is in... I'm not really sure what the deal is though. Hopefully it will be fixed for the next release.

[xdcdx]

Originally posted by SailorH
I have been using MS-PRGothic as my Japanese font. When I open up the options though it always resets to "Larabie Font" even though it is still using MS-PRGothic. I think that is just a bug in the beta.

As for why you are getting incorrect Japanese, I've found that some of my files also do not show proper Japanese. There are several ways to encode Japanese text, in EUC, ISO and SJIS modes. This bug might be related to Winamp not recognizing which mode the text is in... I'm not really sure what the deal is though. Hopefully it will be fixed for the next release.

Could you send me your MS-PRGothic font ?
If you can upload it to some webspace, that would be great, if not I'll tell you my e-mail in private message.

Thanks in advance.

[EDIT]
Oh, you are right, some files (very few ones actually) show its names correctly!
I hope this bug gets fixed soon Also, support for this in the playlist would be great.
The font isn't necessary now.
Thanks again and best regards.

[Canar]

Furthermore, it's very difficult to reconcile both non-Unicode and Unicode builds of the same program, especially when they allow for text manipulation through plugins.

To incorporate Unicode, Winamp would likely need to do the following:[list=1][*]Rewrite (or at best heavily rework) all code having to do with string handling (except possibly if they were using UTF-8; there may be less work involved there)[*]Break backwards compatibility to allow newer plugins to work[*]Add bloat knee-deep to allow different plugins to co-exist well[*]Or some combination of the above.[/list=1]

[peter]

Originally posted by Francis
Hi guys

I noticed that the topic of unicode support in Winamp 5 has been mentionned.

Just so that the matter is settled (and to be clear, I'm not writing this to debate *about* unicode), there is no support of unicode coming up in Winamp 5. That's it. End of story.

So some of you may bring this up :



or this :



and say, "to do this, it must be supporting unicode!".

Well no, what you see here are captures I have made of the freeform skinning rendering by itself (as in: not using Win32's DrawText functions) the default charset for an operating system set to japanese or hebrew. Support for showing the correct international strings in the default language selected in the "regional settings" box of Windows is *not* unicode support. What unicode support allows you to do is to display these strings even if your Windows default os language is set to english. Mixing up japanese AND hebrew in the same program is a feature of unicode. Windows explorer, for instance, can do it, so does Internet Explorer, as long as you have the language installed (but not necessarilly set as default).

This means that, as it has been since a long time in Winamp 2, Winamp 5 can display strings in japanese or in hebrew (freeform skins couldn't previously do it, hence my captures, which I have slightly regreted making since), so long as your Windows is set to japanese or hebrew as your default language.

The reason for this has to do with the size of character strings: if you use 8bit character strings to encode text (what Winamp 2 has been doing since the beginning), you must observe an encoding scheme (even if you do not know about it and just take all chars to simply be 0...255), and there is no way to detect an encoding scheme from a given encoded string (it's just random bytes, it could just be ascii after all, ie: no encoding at all), the operating system needs to know *somehow* in what encoding you are when it wants to print the string, so that it knows wether this code is rather an accented e in french, or a 'ia' in russian (i'm making this correspondance up, btw) and so that it can print the correct characters, not just merely ascii. It does this by reading the default language setting in Windows. In 8bits-strings programs, that is being done by the classical Win32 DrawText functions, they do the character decoding on their own, and then print the characters with the requested font (usually unicode charset, but not always, but that's another story). In contrast, when you support unicode internally, there are four bytes to represent each character and these are enough to store all the unicode characters, and there is therefore no need for a notion of character encoding, everything can be in just one native encoding (unicode, in which case the encoding and the charmap 'fuse together' in a sense, or something still 8bits, like UTF-8), and you can stop thinking about it altogether, because all characters in all languages have a unique number. That's what is sometimes called "proper" unicode support.

Now, to be completely accurate, I must say that the freeform engine has some sort of unicode support, but *in the font rendering engine*, in the sense that, since we cannot rely on DrawText to make the character decoding (decoding to unicode according to the default language set in the system) because we do not use DrawText to draw our fonts (we use Freetype which does not provide the conversion), we have to provide our own conversion of encoded strings to unicode, in order to print the japanese or hebrew characters (we also for instance had to implement our own right-to-left routines). This conversion we provide uses the functions that DrawText uses, and these uses the default language setting to do the decoding, we therefore support japanese and hebrew chatacters only if you have the right language set, we cannot mix the two. True unicode support could.

Summed up, the reason true unicode support cannot be added simply (as in, just recompile, silly!) in the Winamp 2 system is that it would render our plugins incompatible, since they would have to all be recompiled for unicode support as well.

There are thoughts being given about what should be done about it, I certainly would be willing, once Winamp 5 is out, to try and make it truly support unicode without breaking everything (ie: supporting legacy encoding, UTF-8 seems to be the key here), but it would certainly be a lengthy task. Perhaps it is worth it.

Oh and let me be clear again, that's not to start a debate about unicode, if you want to have one amongst yourselves, that's fine, but I won't be participating yes, I have my own view about wether or not it should be undertaken, but we're several in the team, and this is a work that would impact everybody, so unless we all agree that this can be done safely (ie: without breaking winamp2) we probably won't be doing it.

Francis.

*standing ovation*

[amano]

*lol*

peter is back. as we like him!

[DJ Egg]

Originally posted by amano
peter is back

I wish

[amano]

at least back in these forums

[OCedHrt]

Just out of curiousity, what exactly does the Microsoft Unicode Layer do then? Or what purpose does it serve? Is it just to enable unicode on 9x versions of Windows? Oh and one other thing, Winamp5 still doesn't play the right file sometimes =P It plays the first file under the unicode names portions of the filesystem that has the same # of unicode characters at the same places. Ex. Playing F:\My Music\Audio\Chinese\±i´f©f\¯u¹ê\¯u¹ê - 04 - ±i´f©f - ¯u¹ê.mp3 actually plays instead F:\My Music\Audio\Chinese\¤B¤åµX\ÄF¤l\ÄF¤l - 04 - ¤B¤åµX - ¬K_Ë.mp3 since they all look the same to the mp3 decoder =P (Yes, I know changing the default language fixes that, but my Japanese files still won't work right =P) Big thanks for all the effort though! (They are in Big5 because UTF-8 doesn't seem to display certain characters right in this forum) (Ok, Big5 doesn't really work right either..last Chinese character can't be displayed ><)

[lemiel]

There is problem with OGG Vorbis files with UTF in tags in Winamp5 (rc8 rc666). There is no this problem in mp3 files.
In tag editor for ogg files is allright, but in Winamp main window or in playlist chars are not displayed correctly.

This picture is a part of playlist window, the same names in tags but different displayed. First line - ogg vorbis file, second line mp3 file.

ADD after edit:
In Winamp 2.92 in both file types is OK.
In Winamp 3 depends on internalization options, but mostly not correct.

[will]

Originally posted by DJ Egg
I wish

I don't.

[DJ Egg]

umm, and why not exactly?


irrespective of any other issues....

in_vorbis, in_midi, in_wave, in_mod, out_wave, out_ds
The coder? The (instant) fixer?

[will]

Neither.
The person who tried to crash my mIRC a few weeks back.

[DJ Egg]

Aah... I see, lol
So it's a personal grudge kinda thing?

[will]

Originally posted by DJ Egg
Aah... I see, lol
So it's a personal grudge kinda thing?

personal?
The whole of #nullsoft.

[DJ Egg]

ick

bad pp

[james]

Originally posted by will
personal?
The whole of #nullsoft.

hey come on, I did that too.

*runs away*

[CraigF]

yes, and i believe i probably klined or killed you for it. have fun.

[jcsston]

Originally posted by OCedHrt
Just out of curiousity, what exactly does the Microsoft Unicode Layer do then? Or what purpose does it serve? Is it just to enable unicode on 9x versions of Windows?

It allows the same Unicode supporting .exe to work under Win9x (with the Unicows.dll) and WinNT+ (without the Unicows.dll). It mostly saves developers time in supporting non-Unicode OS's.

Just so that the matter is settled (and to be clear, I'm not writing this to debate *about* unicode), there is no support of unicode coming up in Winamp 5. That's it. End of story.

And so was WA5 developed in 1999 and just now released? IMHO Today any 'new' app needs support Unicode or UTF-8 at the least.

Furthermore, it's very difficult to reconcile both non-Unicode and Unicode builds of the same program, especially when they allow for text manipulation through plugins.

To incorporate Unicode, Winamp would likely need to do the following:


1. Rewrite (or at best heavily rework) all code having to do with string handling (except possibly if they were using UTF-8; there may be less work involved there)
2. Break backwards compatibility to allow newer plugins to work
3. Add bloat knee-deep to allow different plugins to co-exist well
4. Or some combination of the above.

Using UTF-8 would solve the issue of backwards compatiblility with the API remaining the same and ASCII text wouldn't be treated any differently. Any new plugin supporting UTF-8 could expand the char * UTF-8 strings to the Unicode strings used in the Win32 API.

[vlease]

Lack of Unicode support is sort of sad...

It's certainly nice to know that one can make any single language display properly by adjusting font and locale settings. That *is* some level of consolation. I expect, though, that I'm not the only person in the world who knows more than one language. Having to adjust my OS locale anytime I'd like to find a song in a language other than the one that's currently playing is a little on the unwieldy side.

Actually, the ogg tag bug in wa5 makes it so that wa5 does not even display the OS' natural language properly for ogg files! This does not even qualify as a multilingual issue, as no non-English data is properly displayed for oggs.

I was so excited about WA5, too! It looks *so* nice otherwise! A very nice product, assuming you have no multilingual needs. I would *so* pay money for WA5 if it could properly display all of my songs' titles. I want to like it...

[vptam]

I posted this up in the Bugs section... but maybe I should have put it here first... but just to clarify now:

Is it because of the lack of Unicode support, files w/ unicode filenames will not load up correctly, i.e. double clicking two different files plays the same song?

[Knocks]

Just tried out 5.01, and Cyrillic support is really weak so far. ID3 tags created in Cyrillic cannot be edited in Winamp ("file is read-only" error). Burning songs with Cyrillic filenames or ID3 tags is also extremely buggy. Most of the time the Sonic engine just refuses to start when you click "Burn." No error messages, it just won't start burning the CD.

[luke977]

Hallo everybody,
here a picture from my Winamp 5.01.
Note the different title between main and playlist windows.
Above in 'xdcdx' picture it seems to me he has same issue.
cya

luke

[luke977]

sorry it is my second post i forgot to attach screenshot.

cya
Luke

[_Reloaded_]

the operating system needs to know *somehow* in what encoding you are when it wants to print the string, so that it knows wether this code is rather an accented e in french, or a 'ia' in russian (i'm making this correspondance up, btw)

and will be any patches for shoutcast to fix 'ia' russian symbols?

[cybermarz]

I'm not sure if my contribution will help this discussion.

Anyway on a Win2K SP2 machine (English, Western Europe and United States Language setting). I can execute "unicode-m3u.bat" which will create a playlist in any MP3 directory.

Unfortunatly Winamp can not handle these Unicode generated playlists, no matter western or non-western characters.

Let see what the future will bring.

code:
@ECHO OFF

ECHO  unicode-m3u.bat



:startUnicode

if "%1"=="yes" goto next

cmd /u /c %0 yes

goto end



:next

for /R %%I IN (*.mp3) do (

        if not exist "%%~dpI/album.m3u" (

                dir /B /S "%%~dpI/*.mp3" > "%%~dpI/album.m3u"

        )

)

:end

Result for a greek album:

H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 01 - Trip-Side.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 02 - Ο πρίγκιπας των fast-food.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 03 - Αρένα.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 04 - Γλυφιτζουράκι.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 05 - Σοκ.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 06 - Λύκος.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 07 - Έμμονες ιδέες.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 08 - Το ψέμα.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 09 - Αποστολή.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 10 - Η θλίψη του κλόουν.mp3
H:\Anima\ΠΑΡΑΞΕΝΕΣ ΜΕΡΕΣ\Anima - 11 - Σύρμα.mp3

[choden_1]

The internet is a WORLD WIDE web.

Although the Americanisation of the known universe is progressing fast, there are still more languages than just English, some of which do not use a standard latin set of characters.

Accept it or not, there are people who speak/listen to Russian/Greek/etc. and these people both can be both internet users and listen to mp3's.

Heck, I'm a fan of Russian pop music, and it took me about 2 hours work to transliterate the songs into a semi-usable state... editing ID3 tags alone took most of my time, and effort just to make Winamp write to it.

To be a true "universal" media player, support for unicode is a requirement.

/end of my 2 cents

[jettoblack]

Here's MY final word. I'll put it simply: if Winamp adds Unicode support, I will be willing to pay US$30-50 for it.

If not, I'll look into an alternative.

I suspect that ignoring the needs of 90% of the world's population isn't exactly a great business plan.

Oh, and I don't care if it breaks every old plugin (though, it would be great if it didn't). How about a checkbox option? "Enable Unicode Support? WARNING!!! WILL BREAK OLD PLUGINS!!!! OK/Cancel"

[pstub]

I have the same problem, and seriously, all this posts didn't really help me all that much. I still don't understand much!
So far, WinAmp5 works great for me (started using it 3 days ago), but I still have the same problem that I always have had: In the full play-list the Norwegian characters (ÆæØøÅå) display just fine, and also in the ID3 tags. The problem arises when the playlist is minimized and only the playing song is showing: The Norwegian letters turn blank!

I hate this, and really don't understand why it is like this! Call me stupid, but so it is...

Preben

[lunarboy1]

because when the normal playlist is drawn, an actual font is used, which supports the characters, however, when the PL is minimized it uses a Bitmap based font, which doesn't conatin the letters.

[Russ]

Originally posted by jettoblack
I suspect that ignoring the needs of 90% of the world's population isn't exactly a great business plan.

They are not ignoring 90% of the world's population - if your computer has the correct charset installed and set as default, then it will display the characters properly.

Who they are ignoring are the 0.5% or so of people who are using one charset and playing files with tags in a different charset - such as someone with an English locale playing songs in Russian or Hebrew.

If you do want full Unicode now, then I would suggest foobar2000 for all your Unicode needs.

The final word is this:

Originally posted by FrancisThere are thoughts being given about what should be done about it, I certainly would be willing, once Winamp 5 is out, to try and make it truly support unicode without breaking everything (ie: supporting legacy encoding, UTF-8 seems to be the key here), but it would certainly be a lengthy task. Perhaps it is worth it.

If you want to press the issue further, please feel free to PM me. Until then,

*whacked*