HydrogenAudio

Lossless Audio Compression => FLAC => Topic started by: ktf on 2021-08-28 19:08:00

Title: FLAC decoder testbench
Post by: ktf on 2021-08-28 19:08:00
In a recent post on this board, I mentioned that there might be quite a few hardware decoders out there that do not support certain features of the FLAC format, probably because the reference FLAC encoder never created files using those features. For example, the FLAC format allows a variable blocksize and certain encoders do create such files, but the reference encoder never did. I cannot blame decoder developers for not properly testing or even implementing decoding of these features, if there is nothing to test it with.

Moreover, I am also interested in knowing how good or bad the average FLAC hardware player is. If it is known that a certain feature with potential is poorly supported by hardware decoders, a developer of a FLAC encoder might choose not to pursue implementing it, as it would produce incompatible files.

That's why I assembled a FLAC decoder testbench: a set of 46 short FLAC files containing Creative Commons licensed music of which every file tests a certain decoder feature. I've ran the testbench on devices I had access to and added the information to a wiki page: https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench

TL;DR
Please take a close look at the files in the testbench and provide feedback about whether I missed any features that need testing. Also, try these files on your hardware devices and add the results to the wikitable. The testbench is linked on the wikipage. Thanks a lot!
Title: Re: FLAC decoder testbench
Post by: Porcus on 2021-08-28 20:58:09
I don't know if any of the following are interesting to consider, but:

* Block size. It is possible to get a common block size (like 512) from end of header. At worst, a decoder that understands 1001 might not find the same from end of header?
* Similar for sample rate. There is a possible difference between accepting 1010 and accepting that one has to find 48 kHz from the end of header?
* Unknown total samples in stream info?

And:
* Testing how the players support or react to pictures (strange mime types?), cuesheets, gigantic padding, repeating metadata blocks, blah blah blah?
Even, it is known that some players want ID3 (https://hydrogenaud.io/index.php?topic=82716.0)...
Title: Re: FLAC decoder testbench
Post by: Nick.C on 2021-08-28 22:11:00
There was a case here (https://hydrogenaud.io/index.php?topic=120420.0) where an Android decoder didn't seem to cope very well with different wasted bits values for different channels.
Title: Re: FLAC decoder testbench
Post by: Rollin on 2021-08-28 23:49:39
About variable blocksize. Is it stated clearly somewhere in documentation that variable blocksize is "subset feature"? If this is not stated, how it was detected that it is "subset feature" actually?
And stupid question: what is difference in meaning between NOK and dash in wiki table?
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-08-29 09:24:37
* Block size. It is possible to get a common block size (like 512) from end of header. At worst, a decoder that understands 1001 might not find the same from end of header? [
* Similar for sample rate. There is a possible difference between accepting 1010 and accepting that one has to find 48 kHz from the end of header?
I don't think adding those would change much. Any reasonable encoder would not create such files (needless filesize increase) and any player that is able to read non-common blocksizes should work with this as well. Same for sample rate.

* Unknown total samples in stream info?

And:
* Testing how the players support or react to pictures (strange mime types?), cuesheets, gigantic padding, repeating metadata blocks, blah blah blah?
Even, it is known that some players want ID3 (https://hydrogenaud.io/index.php?topic=82716.0)...
I wouldn't include anything with ID3. The FLAC documentation strongly discourages using it, most FLAC tools won't work with them and including a file with ID3 might give people the idea that it is required/endorsed/supported. Testing an incomplete STREAMINFO block is a good idea. I'm not yet sure what tests to add for PICTURE blocks, the specification is not as clear as for the stream format, but I'll figure something out.

There was a case here (https://hydrogenaud.io/index.php?topic=120420.0) where an Android decoder didn't seem to cope very well with different wasted bits values for different channels.
That should be tested with file 14. I just checked, about 1/3th of the frames has different wasted bit values for both channels.

About variable blocksize. Is it stated clearly somewhere in documentation that variable blocksize is "subset feature"? If this is not stated, how it was detected that it is "subset feature" actually?
No, it is not clearly stated, it is simply not listed as a limitation of the subset, much like a feature like wasted bits isn't excluded. There is, however, a diff from git (https://github.com/xiph/flac/commit/09229aa967251ce840e43d300d27a915495e75db#diff-ebae485d99a90977f92d1b7a943437b1d9207df1500e9c12bdecd3ce79390d0b) that gave me the idea variable blocksize streams are meant to be subset.

Before this diff, format.html stated about the subset:
Code: [Select]
The blocksize bits in the frame header must be 0001-0101 or 1000-1110, specifying a fixed-blocksize stream (the exception being the last block as described in the table) and a few allowable blocksizes.  This also means that the STREAMINFO metadata block must specify equal mininum and maximum blocksizes.  If the sample rate is <= 48000Hz, the blocksize must be <=4608, i.e. blocksize bits 0001-0101 or 1000-1100.
After it, format.html stated
Code: [Select]
The blocksize bits in the frame header must be 0001-1110.  The blocksize must be <=16384; if the sample rate is <= 48000Hz, the blocksize must be <=4608.
This was changed with the introduction of a 'blocking strategy bit'. Furthermore, looking at commit messages from the Flake encoder, it seems Justin Ruggles and Josh Coalson did some discussing. After this change, Justin removed the non-subset warning from Flake on encoding variable-blocksize streams.

And stupid question: what is difference in meaning between NOK and dash in wiki table?
The items with a dash weren't tested. The JVC KD-R871BT was in the car of a friend of mine, and I had little time to test. When the device froze on file with a non-standard blocksize, simply restarting didn't unfreeze the unit, I needed to wait for a minute. So, I hurried the rest of the testing a little. I will do more thorough testing later.
Title: Re: FLAC decoder testbench
Post by: Rollin on 2021-08-29 12:57:32
If payer only plays 16 bit files , what should i write to Bitdepths column: NOK or Limited?
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-08-29 13:32:06
I did the following:
- If it plays everything as intended: OK (green)
- If it plays a single (minor) thing not as intended: Limited (yellow)
- If it plays something major or multiple things not as intended: NOK (orange)
- If it freezes on anything: Freezes (red)

What is considered major or minor is up for debate though. In any case, place anything that does not play as intended under remarks. I'd say, not playing 12-bit, blocksize 16, non-standard samplerate, old-format variable blocksize or escape codes are things I would consider minor.

Perhaps I should change all Limited to NOK though, or add some comment to the wikipage to make it consistent for everyone.
Title: Re: FLAC decoder testbench
Post by: Rollin on 2021-08-29 14:07:21
By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-08-29 19:14:57
Perhaps I should change all Limited to NOK though, or add some comment to the wikipage to make it consistent for everyone.
I just thought of the following: perhaps it is better to change the gradations to
So, instead of making a distinction based on whether the tester thinks the problem is minor, annotate whether the player recognizes the problem itself. The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table should read 'Freezes' for that category. What do you think? Right now it seems worse that the Cowon iAudio7 doesn't play high-res files than that it freezes on non-standard blocksizes, while the first is a design choice that is well defendable, while the latter is a implementation oversight.

By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.
Even with parts of the same file played back-to-back? There is a key change right after the last non-44.1kHz file indeed, maybe that wasn't the best possible place to cut, or maybe I should make the difference even more extreme.
Title: Re: FLAC decoder testbench
Post by: Rollin on 2021-08-30 18:10:44
I just thought of the following: perhaps it is better to change the gradations to
    Plays OK (green): all files in this category are played as intended, without glitches, changes etc.
    Rejects (yellow): One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported"
    Fails (orange): One or more files are not played back as intended, for example by stuttering, garbling or playing at incorrect speed
    Freezes (red): On one or more files, the player crashes on decoding, leaving the unit inoperable until reboot

So, instead of making a distinction based on whether the tester thinks the problem is minor, annotate whether the player recognizes the problem itself. The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table should read 'Freezes' for that category. What do you think?
Yes, maybe such gradation is better or more honest.

But, about blocksize...
Definitions of blocksize for subset stream in specification and in help from flac.exe are conflicting.
Specification says
Quote
The blocksize bits in the frame header must be 0001-1110. The blocksize must be <=16384; if the sample rate is <= 48000Hz, the blocksize must be <=4608.
But help from flac.exe says:
Quote
must be one of 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096 (and 8192 or 16384 if the sample rate is >48kHz) for Subset streams.
If info in help is indeed incorrect, it must be corrected.
Title: Re: FLAC decoder testbench
Post by: Octocontrabass on 2021-08-31 19:30:27
It looks like the specification is correct: the encoder was changed in 2008 to match the specification (https://github.com/xiph/flac/commit/d8a6f4aaf70ee550b308963dc80ab97c2fbf59f8), but the help text was never updated to reflect the new requirements (https://github.com/xiph/flac/blob/b358381a102a2c1c153ee4cf95dfc04af62faa1a/src/libFLAC/format.c#L209).
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-08-31 20:49:29
I just sent a pull request to fix this documentation issue: https://github.com/xiph/flac/pull/253
Title: Re: FLAC decoder testbench
Post by: Rollin on 2021-09-04 18:03:51
Interesting that while iAudio 7 refuses to play files with variable blocksize, it has no problems with files created with --lax -l 32 -r 0,15.
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-09-05 21:06:06
To me, that doesn't really come as a surprise. --lax -l 32 -r 0,15 is simply a higher LPC order and a rice partition with more switches. Besides that, it could very well be the encoder hasn't used these wider limits: have you tried creating an analysis file with flac -a and checking that file for orders above 12 and partition orders above 8?

Moreover, it is simply more of the same thing. A higher LPC order certainly requires more processing power, but FLAC decoding isn't processor intensive by any means on current devices, even embedded ones. There isn't anything special to program: simply run the loop that is already in place for order 1 to 12 a few more times. Simply put: (almost) no extra bugs possible.

The variable blocksize however, needs specific programming to handle it: reading the blocksize bit, decoding a part of the header differently (sample number instead of frame number), changing buffer lengths maybe. That needs debugging, and for debugging one needs input material to test it. That's one of the reasons this testbench might come in handy for people programming a FLAC decoder.
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-09-06 19:29:31
I just updated the testbench. Changes are:
- Multichannel files now contain spoken names of channels, so correct channel lay-out can be verified
- 15 files were added to check various metadata extremes, as @Porcus suggested

Any suggestions are still very welcome. The link is once again on the wikipage (https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench)
Title: Re: FLAC decoder testbench
Post by: Porcus on 2021-09-13 20:17:45
But, about blocksize...
Definitions of blocksize for subset stream in specification and in help from flac.exe are conflicting.
Specification says
Quote
The blocksize bits in the frame header must be 0001-1110. The blocksize must be <=16384; if the sample rate is <= 48000Hz, the blocksize must be <=4608.
But help from flac.exe says:
Quote
must be one of 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096 (and 8192 or 16384 if the sample rate is >48kHz) for Subset streams.
If info in help is indeed incorrect, it must be corrected.

Also, https://xiph.org/flac/documentation_tools_flac.html , in the table for -b #, --blocksize=#   says (and have said for years):
Quote
Specify the block size in samples. Subset streams must use one of 192/576/1152/2304/4608/256/512/1024/2048/4096 (and 8192/16384 if the sample rate is >48kHz). The reference encoder uses the same block size for the entire stream.

That agrees with the help file and disagrees with the format specification document.
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-09-13 20:34:02
I would argue that as Josh has specifically stated that these were added to the subset later, this is probably simply something he forgot to update. See the three commits linked here: https://github.com/xiph/flac/pull/253

The title of one of the commits is also very clear
Quote
fix problem with encoder being too strict about subset blocksizes
starting with that commit, flac accepted 'odd' blocksizes without requiring --lax. His intention seems quite clear to me from this.
Title: Re: FLAC decoder testbench
Post by: Porcus on 2021-09-13 23:02:31
Not my decision, but I would suggest that the devs sit on their hands until the results from your test bench are in and evaluated.

If it so turns out that the odd block sizes cause too many failures in decoders(/devices) that are already out in the wild, one might want to consider further restrictions; be it by (I) redefining the subset more restrictively, or (II) keeping the subset but recommending even stricter choices in encoders' standard presets with override switches like --semilax, or (III) by encouraging --safeguards N switches to be employed when problems arise.
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-09-14 08:33:16
The only change that is being proposed is updating the docs. The flac encoder has been doing this since 1.2.1, so for 14 years now. As most people are sticking with the default presets anyway, and there is no good reason to deviate from the 'standard' blocksizes in the presets (because the frame header is larger with the 'non-standard' blocksizes) I don't see much of a problem.
Title: Re: FLAC decoder testbench
Post by: Brand on 2021-09-18 13:11:10
Just a comment on the foobar2000 entry in the table: for me it doesn't actually play file 55 properly. I get "Unsupported format or corrupted file" errors for the indexes, after the first 10 seconds. Version 1.6.7 (the latest stable).
I wonder if it's something on my end or is that a mistake.
EDIT: tried with a clean foobar install and it still doesn't play it properly, with big glitches/pauses between tracks/indexes (sometimes it shows errors, sometimes not).
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-09-18 13:51:00
I did the testing with foobar2000 also with 1.6.7, on a not too recent Windows 10 laptop. While it didn't play track 55 gapless, I didn't count that as an error, as quite a lot of players don't play tracks gapless. My thinking was, just because foorbar2000 honours the cuesheet by splitting the file into separate tracks, doesn't mean I should mark this track as fail because it can't play this track gapless.

However, I just tested again: when simply selecting a track in the playlist and letting it play, I don't get any errors. I do get an error when I try to start a 'cuesheet-track' of track 55 while it is still trying to open a different 'cuesheet-track'. Do you see the same behaviour?
Title: Re: FLAC decoder testbench
Post by: Brand on 2021-09-18 16:50:58
However, I just tested again: when simply selecting a track in the playlist and letting it play, I don't get any errors. I do get an error when I try to start a 'cuesheet-track' of track 55 while it is still trying to open a different 'cuesheet-track'. Do you see the same behaviour?
That's one possible way to trigger it too, but it's inconsistent, not always reproducible for me. Sometimes the errors just don't happen and sometimes they happen even during simple playback from the start, without touching anything. There's also high CPU and RAM usage... so the issues could be somewhat system-related.
But the gaps between chapters are already enough for a Fail, IMO. Compare it with mpv, which plays the entire file smoothly with no interruptions and with low CPU usage. I can't make it pause even if I try.
Title: Re: FLAC decoder testbench
Post by: Porcus on 2021-09-18 16:54:09
My fb2k had issues with track 55, in particular on verifying integrity - and the problems are not even consistent on my computer. Also a minor on 54, it shows track number "0".

Anyway, @Peter is aware of it, so I have just waited for updates rather than testing what behaviour is fb2k-outofthebox vs what might possibly depend on (even his own) components.


Title: Re: FLAC decoder testbench
Post by: Rollin on 2021-10-12 18:08:36
As for malformed/corrupted FLAC files...

Reference decoder doesn't report about truncation in some files if they don't have md5. But foobar2000 with foo_verifier and AudioTester can detect that such file is truncated. Sample - https://www.dropbox.com/s/n9awti95o8kt88v/TRUNCATED%20%40%203m%2031s.flac?dl=1

There is (was?) some software (i don't know exactly what it is/was) that ignores restriction for maximal embedded picture size 16 MB. As a result, metadata in FLAC file becomes corrupted. In some very pathological cases it almost impossible to decode such files. See next sample. Reference decoder, foobar2000, ffmpeg are unable to decode this file. VLC can decode it, but resulting file is truncated. The only decoder that i could find that can successfully and properly decode audio from this file is one from XMplay. If XMplay can do this, why reference decoder should be unable to do this? Sample - https://www.dropbox.com/s/1qllhlcxphkam6h/Metadata%20corrupted%20by%20gigantic%20picture.flac?dl=1
Title: Re: FLAC decoder testbench
Post by: Porcus on 2021-10-12 18:26:30
Also, it is damn hard to get anything to remove that embedded picture ...
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-10-12 20:37:58
Sample - https://www.dropbox.com/s/1qllhlcxphkam6h/Metadata%20corrupted%20by%20gigantic%20picture.flac?dl=1
There are also 2 STREAMINFO blocks in that file. I just did some hacking, if libFLAC is persuaded to ignore the second STREAMINFO block and flac -F is used, it decodes fine.
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-10-13 09:55:48
I just sent in a patch to fix both problems in flac: https://github.com/xiph/flac/pull/257 If anyone wants a binary let me know.
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-11-02 12:30:42
I've just uploaded a new testbench, revision 5. Link can be found on the wikipage (https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench).

This revision has all testfiles replaced, the testbench is now licensed CC0 1.0 (https://creativecommons.org/publicdomain/zero/1.0/). This revision should also fix the following two issues:

By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.
Just a comment on the foobar2000 entry in the table: for me it doesn't actually play file 55 properly
The first because all files now include a spoken description and there is no chiptune used, the second because current stable Foobar2000 always crashes on file 55 on my end. It doesn't even import properly now.

I'll redo the hardware player checks I've done previously. If you have any hardware capable of playing FLAC, testing it with this testbench and sharing the results would be most welcome.
Title: Re: FLAC decoder testbench
Post by: Porcus on 2021-11-02 13:40:07
Opened it and saw something I didn't even think of: there is no mono file. Maybe an insult to the decoders ... but then I also thought of: left/side, right/side, mid/side channel assigments 1000/1001/1010?

As for foobar2000, 1.6.8 beta 5 handles it; foo_verifier gives a warning over file 45 (no total number of samples set), but I just let it ReplayGain scan and tag the files, and rewrite tags; then started afresh and rewrote tags and then ReplayGain scanned and tagged. Works fine, but once it gets to writing 48 to, it spends time. It doesn't support AVIF, but says it is present.
Pretty much the same with Mp3tag (3.08) too.

Oh, and at least when the test bench goes final, it might be an idea to have the readme available separately, so one doesn't have to download everything to get it. I took the liberty to attach it here in case it is useful to anyone.
Title: Re: FLAC decoder testbench
Post by: ktf on 2021-11-03 12:22:41
Opened it and saw something I didn't even think of: there is no mono file. Maybe an insult to the decoders ... but then I also thought of: left/side, right/side, mid/side channel assigments 1000/1001/1010?
Yes, I was thinking of adding a mono file. The stereo decorrelation things are simply 'in the mix': they are so common I don't expect any developer needing a specific test file for it. This is also the case with things like LPC orders: there is no file that specifically test all orders, but they're all in there, somewhere.

I was just thinking of something else: the FLAC specification isn't quite clear on whether or not certain parameters are allowed to change mid-file, like bitdepth, number of channels and samplerate. There are certain codecs that do things like that, for example AC3 in TV streams does it all the time, switching from stereo to multichannel.

Perhaps not something to add to this testbench, but to a different one more focused on potential security issues with crafted files.
Title: Re: FLAC decoder testbench
Post by: Porcus on 2021-11-22 18:15:48
The following tests done on a Dell laptop (running Windows 10) through its stereo speakers, no fancier audio devices. I cannot tell what is supposed to be "expected downmixing to stereo", and I cannot know for sure whether Spotify generally refuses to play multichannel, or because it can figure out it is talking to a stereo device.

Spotify for Windows version 1.1.72.439.gc253025e: Here I had to let the library point at the folder for local files.
Tracks 48 through 52 are rejected and so is 55; they are not even indexed by the player. 
Tracks 22 and 36 through 44 skip.  44 after seemingly playing silence for four or five seconds before skipping the last ten-ish - but I cannot rule out that the progress bar just runs while it in reality Spotify spends this time to figure out that it is something it wants to skip.

fb2k 1.6.8: like the beta but unlike previous versions, it now handles everything.

Can also confirm the wiki entries for: VLC and SMPlayer 21.8.0 using mpv 0.32.0.

Title: Re: FLAC decoder testbench
Post by: Porcus on 2021-11-22 20:31:05
Oh, wait. While VLC for all "end-user-practical" purposes hangs at the infamous 55, it is actually "just" spending minutes processing the file before playing it.

Reason I found out: I tried ffplay, which does the same thing but starts playback much faster.  Went back to VLC then. Also tried MPlayer.exe (in the SMPlayer bundle). And mpv.exe, which was so stuck forever that I'm killing it rather than waiting overnight.

I'd say nobody wants to wait neither one nor four minutes though.


VLC on files 27 and 55, opening the messages and media information windows:
27 - old format variable blocksize file created with Flake 0.11.flac Gnaws at the file but one can skip back and forth in the playlist. After some three minutes it makes a short noise, and then apparently it consider itself done with the file.
55 - file 48-53 combined.flac : Starts playing after three to four minutes. If I have another window on top, a popup with metadata infomation shows up in around 100 seconds, that is way earlier than metadata shows up in the Ctrl-i information window.
51 - Extremely large VORBISCOMMENT.flac : Traces of "as 55" - but hangs only for three to four seconds, that is tolerable. (Also 54 hangs for a second.)


ffplay version 4.4.1-essentials_build-www.gyan.dev . Invoked through a FOR loop with -autoexit
03 - blocksize 16.flac plays at like 1/6th of intended speed. Totally unlistenable, of course.
55: Because I see metadata output to terminal window, I can tell it is technically not hanging. Without it, I wouldn't have had the patience. But after 70 seconds, it plays.


MPlayer.exe reported to be the following:
MPlayer Redxii-SVN-r37955-6.2.0 (x86_64) (C) 2000-2017 MPlayer Team
Using FFmpeg N-87137-g6ccd32c367 (2017-08-30 20:24:28 +0200)
Compiled on 2017-09-09 13:22:37 EDT (rev. 1)

36: Stuck at outpt like A:  -1.0 (unknown) of 8.0 (08.0) ??,?%, but I can use right arrow (which is supposed to seek 10 seconds!) to get to 6 seconds-ish. It then plays the last couple of seconds.
I think that is a "Fails" when user can skip and get to next without program restart?
55: Plays - but spends around 80 seconds before playing audio.

(MPlayer makes some other quirks that do not affect playback, like showing the GIF as a green rectangle. And I've seen strange times on the 27 on other players too.)