Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: FLAC decoder testbench (Read 2885 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

FLAC decoder testbench

In a recent post on this board, I mentioned that there might be quite a few hardware decoders out there that do not support certain features of the FLAC format, probably because the reference FLAC encoder never created files using those features. For example, the FLAC format allows a variable blocksize and certain encoders do create such files, but the reference encoder never did. I cannot blame decoder developers for not properly testing or even implementing decoding of these features, if there is nothing to test it with.

Moreover, I am also interested in knowing how good or bad the average FLAC hardware player is. If it is known that a certain feature with potential is poorly supported by hardware decoders, a developer of a FLAC encoder might choose not to pursue implementing it, as it would produce incompatible files.

That's why I assembled a FLAC decoder testbench: a set of 46 short FLAC files containing Creative Commons licensed music of which every file tests a certain decoder feature. I've ran the testbench on devices I had access to and added the information to a wiki page: https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench

TL;DR
Please take a close look at the files in the testbench and provide feedback about whether I missed any features that need testing. Also, try these files on your hardware devices and add the results to the wikitable. The testbench is linked on the wikipage. Thanks a lot!
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #1
I don't know if any of the following are interesting to consider, but:

* Block size. It is possible to get a common block size (like 512) from end of header. At worst, a decoder that understands 1001 might not find the same from end of header?
* Similar for sample rate. There is a possible difference between accepting 1010 and accepting that one has to find 48 kHz from the end of header?
* Unknown total samples in stream info?

And:
* Testing how the players support or react to pictures (strange mime types?), cuesheets, gigantic padding, repeating metadata blocks, blah blah blah?
Even, it is known that some players want ID3 ...
High Voltage socket-nose-avatar

Re: FLAC decoder testbench

Reply #2
There was a case here where an Android decoder didn't seem to cope very well with different wasted bits values for different channels.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

Re: FLAC decoder testbench

Reply #3
About variable blocksize. Is it stated clearly somewhere in documentation that variable blocksize is "subset feature"? If this is not stated, how it was detected that it is "subset feature" actually?
And stupid question: what is difference in meaning between NOK and dash in wiki table?

Re: FLAC decoder testbench

Reply #4
* Block size. It is possible to get a common block size (like 512) from end of header. At worst, a decoder that understands 1001 might not find the same from end of header? [
* Similar for sample rate. There is a possible difference between accepting 1010 and accepting that one has to find 48 kHz from the end of header?
I don't think adding those would change much. Any reasonable encoder would not create such files (needless filesize increase) and any player that is able to read non-common blocksizes should work with this as well. Same for sample rate.

* Unknown total samples in stream info?

And:
* Testing how the players support or react to pictures (strange mime types?), cuesheets, gigantic padding, repeating metadata blocks, blah blah blah?
Even, it is known that some players want ID3 ...
I wouldn't include anything with ID3. The FLAC documentation strongly discourages using it, most FLAC tools won't work with them and including a file with ID3 might give people the idea that it is required/endorsed/supported. Testing an incomplete STREAMINFO block is a good idea. I'm not yet sure what tests to add for PICTURE blocks, the specification is not as clear as for the stream format, but I'll figure something out.

There was a case here where an Android decoder didn't seem to cope very well with different wasted bits values for different channels.
That should be tested with file 14. I just checked, about 1/3th of the frames has different wasted bit values for both channels.

About variable blocksize. Is it stated clearly somewhere in documentation that variable blocksize is "subset feature"? If this is not stated, how it was detected that it is "subset feature" actually?
No, it is not clearly stated, it is simply not listed as a limitation of the subset, much like a feature like wasted bits isn't excluded. There is, however, a diff from git that gave me the idea variable blocksize streams are meant to be subset.

Before this diff, format.html stated about the subset:
Code: [Select]
The blocksize bits in the frame header must be 0001-0101 or 1000-1110, specifying a fixed-blocksize stream (the exception being the last block as described in the table) and a few allowable blocksizes.  This also means that the STREAMINFO metadata block must specify equal mininum and maximum blocksizes.  If the sample rate is <= 48000Hz, the blocksize must be <=4608, i.e. blocksize bits 0001-0101 or 1000-1100.
After it, format.html stated
Code: [Select]
The blocksize bits in the frame header must be 0001-1110.  The blocksize must be <=16384; if the sample rate is <= 48000Hz, the blocksize must be <=4608.
This was changed with the introduction of a 'blocking strategy bit'. Furthermore, looking at commit messages from the Flake encoder, it seems Justin Ruggles and Josh Coalson did some discussing. After this change, Justin removed the non-subset warning from Flake on encoding variable-blocksize streams.

And stupid question: what is difference in meaning between NOK and dash in wiki table?
The items with a dash weren't tested. The JVC KD-R871BT was in the car of a friend of mine, and I had little time to test. When the device froze on file with a non-standard blocksize, simply restarting didn't unfreeze the unit, I needed to wait for a minute. So, I hurried the rest of the testing a little. I will do more thorough testing later.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #5
If payer only plays 16 bit files , what should i write to Bitdepths column: NOK or Limited?

Re: FLAC decoder testbench

Reply #6
I did the following:
- If it plays everything as intended: OK (green)
- If it plays a single (minor) thing not as intended: Limited (yellow)
- If it plays something major or multiple things not as intended: NOK (orange)
- If it freezes on anything: Freezes (red)

What is considered major or minor is up for debate though. In any case, place anything that does not play as intended under remarks. I'd say, not playing 12-bit, blocksize 16, non-standard samplerate, old-format variable blocksize or escape codes are things I would consider minor.

Perhaps I should change all Limited to NOK though, or add some comment to the wikipage to make it consistent for everyone.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #7
By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.

Re: FLAC decoder testbench

Reply #8
Perhaps I should change all Limited to NOK though, or add some comment to the wikipage to make it consistent for everyone.
I just thought of the following: perhaps it is better to change the gradations to
  • Plays OK (green): all files in this category are played as intended, without glitches, changes etc.
  • Rejects (yellow): One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported"
  • Fails (orange): One or more files are not played back as intended, for example by stuttering, garbling or playing at incorrect speed
  • Freezes (red): On one or more files, the player crashes on decoding, leaving the unit inoperable until reboot
So, instead of making a distinction based on whether the tester thinks the problem is minor, annotate whether the player recognizes the problem itself. The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table should read 'Freezes' for that category. What do you think? Right now it seems worse that the Cowon iAudio7 doesn't play high-res files than that it freezes on non-standard blocksizes, while the first is a design choice that is well defendable, while the latter is a implementation oversight.

By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.
Even with parts of the same file played back-to-back? There is a key change right after the last non-44.1kHz file indeed, maybe that wasn't the best possible place to cut, or maybe I should make the difference even more extreme.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #9
I just thought of the following: perhaps it is better to change the gradations to
    Plays OK (green): all files in this category are played as intended, without glitches, changes etc.
    Rejects (yellow): One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported"
    Fails (orange): One or more files are not played back as intended, for example by stuttering, garbling or playing at incorrect speed
    Freezes (red): On one or more files, the player crashes on decoding, leaving the unit inoperable until reboot

So, instead of making a distinction based on whether the tester thinks the problem is minor, annotate whether the player recognizes the problem itself. The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table should read 'Freezes' for that category. What do you think?
Yes, maybe such gradation is better or more honest.

But, about blocksize...
Definitions of blocksize for subset stream in specification and in help from flac.exe are conflicting.
Specification says
Quote
The blocksize bits in the frame header must be 0001-1110. The blocksize must be <=16384; if the sample rate is <= 48000Hz, the blocksize must be <=4608.
But help from flac.exe says:
Quote
must be one of 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096 (and 8192 or 16384 if the sample rate is >48kHz) for Subset streams.
If info in help is indeed incorrect, it must be corrected.


Re: FLAC decoder testbench

Reply #11
I just sent a pull request to fix this documentation issue: https://github.com/xiph/flac/pull/253
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #12
Interesting that while iAudio 7 refuses to play files with variable blocksize, it has no problems with files created with --lax -l 32 -r 0,15.

Re: FLAC decoder testbench

Reply #13
To me, that doesn't really come as a surprise. --lax -l 32 -r 0,15 is simply a higher LPC order and a rice partition with more switches. Besides that, it could very well be the encoder hasn't used these wider limits: have you tried creating an analysis file with flac -a and checking that file for orders above 12 and partition orders above 8?

Moreover, it is simply more of the same thing. A higher LPC order certainly requires more processing power, but FLAC decoding isn't processor intensive by any means on current devices, even embedded ones. There isn't anything special to program: simply run the loop that is already in place for order 1 to 12 a few more times. Simply put: (almost) no extra bugs possible.

The variable blocksize however, needs specific programming to handle it: reading the blocksize bit, decoding a part of the header differently (sample number instead of frame number), changing buffer lengths maybe. That needs debugging, and for debugging one needs input material to test it. That's one of the reasons this testbench might come in handy for people programming a FLAC decoder.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #14
I just updated the testbench. Changes are:
- Multichannel files now contain spoken names of channels, so correct channel lay-out can be verified
- 15 files were added to check various metadata extremes, as @Porcus suggested

Any suggestions are still very welcome. The link is once again on the wikipage
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #15
But, about blocksize...
Definitions of blocksize for subset stream in specification and in help from flac.exe are conflicting.
Specification says
Quote
The blocksize bits in the frame header must be 0001-1110. The blocksize must be <=16384; if the sample rate is <= 48000Hz, the blocksize must be <=4608.
But help from flac.exe says:
Quote
must be one of 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096 (and 8192 or 16384 if the sample rate is >48kHz) for Subset streams.
If info in help is indeed incorrect, it must be corrected.

Also, https://xiph.org/flac/documentation_tools_flac.html , in the table for -b #, --blocksize=#   says (and have said for years):
Quote
Specify the block size in samples. Subset streams must use one of 192/576/1152/2304/4608/256/512/1024/2048/4096 (and 8192/16384 if the sample rate is >48kHz). The reference encoder uses the same block size for the entire stream.

That agrees with the help file and disagrees with the format specification document.
High Voltage socket-nose-avatar

Re: FLAC decoder testbench

Reply #16
I would argue that as Josh has specifically stated that these were added to the subset later, this is probably simply something he forgot to update. See the three commits linked here: https://github.com/xiph/flac/pull/253

The title of one of the commits is also very clear
Quote
fix problem with encoder being too strict about subset blocksizes
starting with that commit, flac accepted 'odd' blocksizes without requiring --lax. His intention seems quite clear to me from this.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #17
Not my decision, but I would suggest that the devs sit on their hands until the results from your test bench are in and evaluated.

If it so turns out that the odd block sizes cause too many failures in decoders(/devices) that are already out in the wild, one might want to consider further restrictions; be it by (I) redefining the subset more restrictively, or (II) keeping the subset but recommending even stricter choices in encoders' standard presets with override switches like --semilax, or (III) by encouraging --safeguards N switches to be employed when problems arise.
High Voltage socket-nose-avatar

Re: FLAC decoder testbench

Reply #18
The only change that is being proposed is updating the docs. The flac encoder has been doing this since 1.2.1, so for 14 years now. As most people are sticking with the default presets anyway, and there is no good reason to deviate from the 'standard' blocksizes in the presets (because the frame header is larger with the 'non-standard' blocksizes) I don't see much of a problem.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #19
Just a comment on the foobar2000 entry in the table: for me it doesn't actually play file 55 properly. I get "Unsupported format or corrupted file" errors for the indexes, after the first 10 seconds. Version 1.6.7 (the latest stable).
I wonder if it's something on my end or is that a mistake.
EDIT: tried with a clean foobar install and it still doesn't play it properly, with big glitches/pauses between tracks/indexes (sometimes it shows errors, sometimes not).

Re: FLAC decoder testbench

Reply #20
I did the testing with foobar2000 also with 1.6.7, on a not too recent Windows 10 laptop. While it didn't play track 55 gapless, I didn't count that as an error, as quite a lot of players don't play tracks gapless. My thinking was, just because foorbar2000 honours the cuesheet by splitting the file into separate tracks, doesn't mean I should mark this track as fail because it can't play this track gapless.

However, I just tested again: when simply selecting a track in the playlist and letting it play, I don't get any errors. I do get an error when I try to start a 'cuesheet-track' of track 55 while it is still trying to open a different 'cuesheet-track'. Do you see the same behaviour?
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #21
However, I just tested again: when simply selecting a track in the playlist and letting it play, I don't get any errors. I do get an error when I try to start a 'cuesheet-track' of track 55 while it is still trying to open a different 'cuesheet-track'. Do you see the same behaviour?
That's one possible way to trigger it too, but it's inconsistent, not always reproducible for me. Sometimes the errors just don't happen and sometimes they happen even during simple playback from the start, without touching anything. There's also high CPU and RAM usage... so the issues could be somewhat system-related.
But the gaps between chapters are already enough for a Fail, IMO. Compare it with mpv, which plays the entire file smoothly with no interruptions and with low CPU usage. I can't make it pause even if I try.

Re: FLAC decoder testbench

Reply #22
My fb2k had issues with track 55, in particular on verifying integrity - and the problems are not even consistent on my computer. Also a minor on 54, it shows track number "0".

Anyway, @Peter is aware of it, so I have just waited for updates rather than testing what behaviour is fb2k-outofthebox vs what might possibly depend on (even his own) components.


High Voltage socket-nose-avatar

 

Re: FLAC decoder testbench

Reply #23
As for malformed/corrupted FLAC files...

Reference decoder doesn't report about truncation in some files if they don't have md5. But foobar2000 with foo_verifier and AudioTester can detect that such file is truncated. Sample - https://www.dropbox.com/s/n9awti95o8kt88v/TRUNCATED%20%40%203m%2031s.flac?dl=1

There is (was?) some software (i don't know exactly what it is/was) that ignores restriction for maximal embedded picture size 16 MB. As a result, metadata in FLAC file becomes corrupted. In some very pathological cases it almost impossible to decode such files. See next sample. Reference decoder, foobar2000, ffmpeg are unable to decode this file. VLC can decode it, but resulting file is truncated. The only decoder that i could find that can successfully and properly decode audio from this file is one from XMplay. If XMplay can do this, why reference decoder should be unable to do this? Sample - https://www.dropbox.com/s/1qllhlcxphkam6h/Metadata%20corrupted%20by%20gigantic%20picture.flac?dl=1

Re: FLAC decoder testbench

Reply #24
Also, it is damn hard to get anything to remove that embedded picture ...
High Voltage socket-nose-avatar