HydrogenAudio

Lossless Audio Compression => FLAC => Topic started by: ktf on 2022-01-09 20:52:50

Title: FLAC unicode patch: some help wanted
Post by: ktf on 2022-01-09 20:52:50
Hi all,

As some of you might know, FLAC and Unicode support on Windows have a bit of a past. Through the last few years, various windows-specific bits were added and changed. Some of these changes were visible on the outside (i.e. the functions exposed by the DLL, the interface), most of them were not.

Anyway, I've proposed yet another change (https://github.com/xiph/flac/blob/master/src/libFLAC/windows_unicode_filenames.c) that I hope will be the last change that is visible on the DLL interface. With this change, the libFLAC interface should now be the same cross-platform. However, that involves a change that might not be fully backward compatible: filenames passed to libFLAC have always been UTF-8 on all platforms except Windows. On Windows, it was dependent on the active codepage. With this change, filenames passed to libFLAC are always UTF-8, also on Windows.

I don't think this change will affect a lot of programs, because unicode-aware programs probably weren't using these functions at all (because they weren't usable with unicode on Windows), and I expect non-unicode-aware programs using libFLAC are extinct.

That's why I'd like to ask for your help: I've prepared a libFLAC.dll with quite a few patches that are not yet applied to git. Patches included are:
- UTF-8 on Windows change (https://github.com/xiph/flac/pull/272) (which is what this topic is about)
- Compression improvement patch (https://github.com/xiph/flac/pull/245), discussed on HA here (https://hydrogenaud.io/index.php?topic=120158.msg999746#msg999746)
- Bug fix (https://github.com/xiph/flac/pull/252), discussed on HA here (https://hydrogenaud.io/index.php?topic=121349.0)
- Fixed subframe speed improvement (https://github.com/xiph/flac/pull/255)
- some build system improvement (https://github.com/xiph/flac/pull/267) (which made it possible for me to build this DLL with MinGW)

If you know a program that uses libFLAC as a DLL, please test whether it still works if you replace that DLL and encode and decode files with non-ASCII characters (Cyrillic, Greek, Hanzi, emoji, etc.) in its filename, and let me know.
Title: Re: FLAC unicode patch: some help wanted
Post by: lvqcl on 2022-01-12 17:34:48
Is it 64-bit DLL? Is it possible to make and share 32-bit one?
Title: Re: FLAC unicode patch: some help wanted
Post by: Porcus on 2022-01-12 22:13:26
And the issues @Rollin reported in that thread discussing the above mentioned "Bug fix" (https://hydrogenaud.io/index.php?topic=121349.msg1001274#msg1001274), were with a 32-bit build. No idea whether that was just a coincidence, but ...
Title: Re: FLAC unicode patch: some help wanted
Post by: ktf on 2022-01-13 10:10:05
Yes, the one in the first post was 64-bit. Didn't occur to me that a 32-bit version is probably very handy to test with programs that are a little less recent. Here is the 32-bit version.
Title: Re: FLAC unicode patch: some help wanted
Post by: doccolinni on 2022-01-24 12:06:46
- UTF-8 on Windows change (https://github.com/xiph/flac/pull/272) (which is what this topic is about)
- Compression improvement patch (https://github.com/xiph/flac/pull/245), discussed on HA here (https://hydrogenaud.io/index.php?topic=120158.msg999746#msg999746)
- Bug fix (https://github.com/xiph/flac/pull/252), discussed on HA here (https://hydrogenaud.io/index.php?topic=121349.0)
- Fixed subframe speed improvement (https://github.com/xiph/flac/pull/255)
- some build system improvement (https://github.com/xiph/flac/pull/267) (which made it possible for me to build this DLL with MinGW)

What's taking so long for these to be merged, anyway?

Also, could you provide flac.exe as well instead of just a .dll?
Title: Re: FLAC unicode patch: some help wanted
Post by: ktf on 2022-01-24 12:26:27
The point of this discussion is to test the DLL interface for compatibility, so providing an exe would be off-topic and probably derail the discussion.

Seems the FLAC maintainer hasn't got time/motivation to work on FLAC. He merged a bunch of PRs on March 15 2021, and his last commits before that are from May 14 2020. Someone else involved with Xiph/Mozilla has merged a few PRs, but the last activity (merging, commenting etc.) by anyone with write access to the FLAC repository has been well over 6 months ago.
Title: Re: FLAC unicode patch: some help wanted
Post by: doccolinni on 2022-01-24 12:51:46
Seems the FLAC maintainer hasn't got time/motivation to work on FLAC. He merged a bunch of PRs on March 15 2021, and his last commits before that are from May 14 2020. Someone else involved with Xiph/Mozilla has merged a few PRs, but the last activity (merging, commenting etc.) by anyone with write access to the FLAC repository has been well over 6 months ago.

Well that's a shame. Might be time for a fork.
Title: Re: FLAC unicode patch: some help wanted
Post by: Rollin on 2022-01-24 14:38:17
I tested 32-bit dll with qaac, refalac, freac and xrecode 3 on 32-bit windows 7 with file named "青木ヶ原樹海-Hliðskjálf-Δ δ-λάμβδα" and it works.
dll also works with SoX (for non-unicode filenames, because SoX itself doesn't support unicode)
Title: Re: FLAC unicode patch: some help wanted
Post by: ktf on 2022-01-24 18:36:48
Thanks for posting the results. I myself tested CDex and CUETools, which all worked fine. It seems various developers have taken the warning in the api documentation (https://xiph.org/flac/api/group__flac__stream__decoder.html#ga4021ead5cff29fd589c915756f902f1a) seriously and not used FLAC__stream_decoder_init_file and FLAC__stream_encoder_init_file

Quote
If POSIX fopen() semantics are not sufficient, (for example, with Unicode filenames on Windows), you must use FLAC__stream_decoder_init_FILE(), or FLAC__stream_decoder_init_stream() and provide callbacks for the I/O.

With this patch it becomes

Quote
On Windows, filename must be a UTF-8 encoded filename, which libFLAC  internally translates to a appropriate  presentation to use with _wfopen
Title: Re: FLAC unicode patch: some help wanted
Post by: XPEHOPE3 on 2022-02-08 01:56:54
Seems the FLAC maintainer hasn't got time/motivation to work on FLAC. He merged a bunch of PRs on March 15 2021, and his last commits before that are from May 14 2020. Someone else involved with Xiph/Mozilla has merged a few PRs, but the last activity (merging, commenting etc.) by anyone with write access to the FLAC repository has been well over 6 months ago.
Did you try to just ask those guys (rillian and erikd) to simply commit whatever you want?) You seem to be the most involved in FLAC now, and competent enough to prepare next release, and they probably know this.
Title: Re: FLAC unicode patch: some help wanted
Post by: doccolinni on 2022-02-08 10:55:45
Did you try to just ask those guys (rillian and erikd) to simply commit whatever you want?) You seem to be the most involved in FLAC now, and competent enough to prepare next release, and they probably know this.

Agreed. @ktf if the previous/current maintainers don't have the time or motivation to maintain the repository but you do, they should just make you the new maintainer. I have no idea how one goes about requesting to become the FLAC repository maintainer however, but the amount of work and dedication you've put into developing FLAC the past several months is exceptional and should serve to validate the qualification.
Title: Re: FLAC unicode patch: some help wanted
Post by: ktf on 2022-02-08 11:05:45
I've been busy reaching out and have high hopes a new release will be out soon.