In a recent post on this board, I mentioned that there might be quite a few hardware decoders out there that do not support certain features of the FLAC format, probably because the reference FLAC encoder never created files using those features. For example, the FLAC format allows a variable blocksize and certain encoders do create such files, but the reference encoder never did. I cannot blame decoder developers for not properly testing or even implementing decoding of these features, if there is nothing to test it with.
Moreover, I am also interested in knowing how good or bad the average FLAC hardware player is. If it is known that a certain feature with potential is poorly supported by hardware decoders, a developer of a FLAC encoder might choose not to pursue implementing it, as it would produce incompatible files.
That's why I assembled a FLAC decoder testbench: a set of 46 short FLAC files containing Creative Commons licensed music of which every file tests a certain decoder feature. I've ran the testbench on devices I had access to and added the information to a wiki page: https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench
TL;DR
Please take a close look at the files in the testbench and provide feedback about whether I missed any features that need testing. Also, try these files on your hardware devices and add the results to the wikitable. The testbench is linked on the wikipage. Thanks a lot!
I don't know if any of the following are interesting to consider, but:
* Block size. It is possible to get a common block size (like 512) from end of header. At worst, a decoder that understands 1001 might not find the same from end of header?
* Similar for sample rate. There is a possible difference between accepting 1010 and accepting that one has to find 48 kHz from the end of header?
* Unknown total samples in stream info?
And:
* Testing how the players support or react to pictures (strange mime types?), cuesheets, gigantic padding, repeating metadata blocks, blah blah blah?
Even, it is known that some players want ID3 (https://hydrogenaud.io/index.php?topic=82716.0)...
There was a case here (https://hydrogenaud.io/index.php?topic=120420.0) where an Android decoder didn't seem to cope very well with different wasted bits values for different channels.
About variable blocksize. Is it stated clearly somewhere in documentation that variable blocksize is "subset feature"? If this is not stated, how it was detected that it is "subset feature" actually?
And stupid question: what is difference in meaning between NOK and dash in wiki table?
* Block size. It is possible to get a common block size (like 512) from end of header. At worst, a decoder that understands 1001 might not find the same from end of header? [
* Similar for sample rate. There is a possible difference between accepting 1010 and accepting that one has to find 48 kHz from the end of header?
I don't think adding those would change much. Any reasonable encoder would not create such files (needless filesize increase) and any player that is able to read non-common blocksizes should work with this as well. Same for sample rate.
* Unknown total samples in stream info?
And:
* Testing how the players support or react to pictures (strange mime types?), cuesheets, gigantic padding, repeating metadata blocks, blah blah blah?
Even, it is known that some players want ID3 (https://hydrogenaud.io/index.php?topic=82716.0)...
I wouldn't include anything with ID3. The FLAC documentation strongly discourages using it, most FLAC tools won't work with them and including a file with ID3 might give people the idea that it is required/endorsed/supported. Testing an incomplete STREAMINFO block is a good idea. I'm not yet sure what tests to add for PICTURE blocks, the specification is not as clear as for the stream format, but I'll figure something out.
There was a case here (https://hydrogenaud.io/index.php?topic=120420.0) where an Android decoder didn't seem to cope very well with different wasted bits values for different channels.
That should be tested with file 14. I just checked, about 1/3th of the frames has different wasted bit values for both channels.
About variable blocksize. Is it stated clearly somewhere in documentation that variable blocksize is "subset feature"? If this is not stated, how it was detected that it is "subset feature" actually?
No, it is not clearly stated, it is simply not listed as a limitation of the subset, much like a feature like wasted bits isn't excluded. There is, however, a diff from git (https://github.com/xiph/flac/commit/09229aa967251ce840e43d300d27a915495e75db#diff-ebae485d99a90977f92d1b7a943437b1d9207df1500e9c12bdecd3ce79390d0b) that gave me the idea variable blocksize streams are meant to be subset.
Before this diff, format.html stated about the subset:
The blocksize bits in the frame header must be 0001-0101 or 1000-1110, specifying a fixed-blocksize stream (the exception being the last block as described in the table) and a few allowable blocksizes. This also means that the STREAMINFO metadata block must specify equal mininum and maximum blocksizes. If the sample rate is <= 48000Hz, the blocksize must be <=4608, i.e. blocksize bits 0001-0101 or 1000-1100.
After it, format.html stated
The blocksize bits in the frame header must be 0001-1110. The blocksize must be <=16384; if the sample rate is <= 48000Hz, the blocksize must be <=4608.
This was changed with the introduction of a 'blocking strategy bit'. Furthermore, looking at commit messages from the Flake encoder, it seems Justin Ruggles and Josh Coalson did some discussing. After this change, Justin removed the non-subset warning from Flake on encoding variable-blocksize streams.
And stupid question: what is difference in meaning between NOK and dash in wiki table?
The items with a dash weren't tested. The JVC KD-R871BT was in the car of a friend of mine, and I had little time to test. When the device froze on file with a non-standard blocksize, simply restarting didn't unfreeze the unit, I needed to wait for a minute. So, I hurried the rest of the testing a little. I will do more thorough testing later.
If payer only plays 16 bit files , what should i write to Bitdepths column: NOK or Limited?
I did the following:
- If it plays everything as intended: OK (green)
- If it plays a single (minor) thing not as intended: Limited (yellow)
- If it plays something major or multiple things not as intended: NOK (orange)
- If it freezes on anything: Freezes (red)
What is considered major or minor is up for debate though. In any case, place anything that does not play as intended under remarks. I'd say, not playing 12-bit, blocksize 16, non-standard samplerate, old-format variable blocksize or escape codes are things I would consider minor.
Perhaps I should change all Limited to NOK though, or add some comment to the wikipage to make it consistent for everyone.
By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.
Perhaps I should change all Limited to NOK though, or add some comment to the wikipage to make it consistent for everyone.
I just thought of the following: perhaps it is better to change the gradations to
- Plays OK (green): all files in this category are played as intended, without glitches, changes etc.
- Rejects (yellow): One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported"
- Fails (orange): One or more files are not played back as intended, for example by stuttering, garbling or playing at incorrect speed
- Freezes (red): On one or more files, the player crashes on decoding, leaving the unit inoperable until reboot
So, instead of making a distinction based on whether the tester thinks the problem is minor, annotate whether the player recognizes the problem itself. The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table should read 'Freezes' for that category. What do you think? Right now it seems worse that the Cowon iAudio7 doesn't play high-res files than that it freezes on non-standard blocksizes, while the first is a design choice that is well defendable, while the latter is a implementation oversight.
By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.
Even with parts of the same file played back-to-back? There is a key change right after the last non-44.1kHz file indeed, maybe that wasn't the best possible place to cut, or maybe I should make the difference even more extreme.
I just thought of the following: perhaps it is better to change the gradations to
Plays OK (green): all files in this category are played as intended, without glitches, changes etc.
Rejects (yellow): One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported"
Fails (orange): One or more files are not played back as intended, for example by stuttering, garbling or playing at incorrect speed
Freezes (red): On one or more files, the player crashes on decoding, leaving the unit inoperable until reboot
So, instead of making a distinction based on whether the tester thinks the problem is minor, annotate whether the player recognizes the problem itself. The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table should read 'Freezes' for that category. What do you think?
Yes, maybe such gradation is better or more honest.
But, about blocksize...
Definitions of blocksize for subset stream in specification and in help from flac.exe are conflicting.
Specification says
The blocksize bits in the frame header must be 0001-1110. The blocksize must be <=16384; if the sample rate is <= 48000Hz, the blocksize must be <=4608.
But help from flac.exe says:
must be one of 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096 (and 8192 or 16384 if the sample rate is >48kHz) for Subset streams.
If info in help is indeed incorrect, it must be corrected.
It looks like the specification is correct: the encoder was changed in 2008 to match the specification (https://github.com/xiph/flac/commit/d8a6f4aaf70ee550b308963dc80ab97c2fbf59f8), but the help text was never updated to reflect the new requirements (https://github.com/xiph/flac/blob/b358381a102a2c1c153ee4cf95dfc04af62faa1a/src/libFLAC/format.c#L209).
I just sent a pull request to fix this documentation issue: https://github.com/xiph/flac/pull/253
Interesting that while iAudio 7 refuses to play files with variable blocksize, it has no problems with files created with --lax -l 32 -r 0,15.
To me, that doesn't really come as a surprise. --lax -l 32 -r 0,15 is simply a higher LPC order and a rice partition with more switches. Besides that, it could very well be the encoder hasn't used these wider limits: have you tried creating an analysis file with flac -a and checking that file for orders above 12 and partition orders above 8?
Moreover, it is simply more of the same thing. A higher LPC order certainly requires more processing power, but FLAC decoding isn't processor intensive by any means on current devices, even embedded ones. There isn't anything special to program: simply run the loop that is already in place for order 1 to 12 a few more times. Simply put: (almost) no extra bugs possible.
The variable blocksize however, needs specific programming to handle it: reading the blocksize bit, decoding a part of the header differently (sample number instead of frame number), changing buffer lengths maybe. That needs debugging, and for debugging one needs input material to test it. That's one of the reasons this testbench might come in handy for people programming a FLAC decoder.
I just updated the testbench. Changes are:
- Multichannel files now contain spoken names of channels, so correct channel lay-out can be verified
- 15 files were added to check various metadata extremes, as
@Porcus suggested
Any suggestions are still very welcome. The link is once again on the wikipage (https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench)
But, about blocksize...
Definitions of blocksize for subset stream in specification and in help from flac.exe are conflicting.
Specification says
The blocksize bits in the frame header must be 0001-1110. The blocksize must be <=16384; if the sample rate is <= 48000Hz, the blocksize must be <=4608.
But help from flac.exe says: must be one of 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096 (and 8192 or 16384 if the sample rate is >48kHz) for Subset streams.
If info in help is indeed incorrect, it must be corrected.
Also, https://xiph.org/flac/documentation_tools_flac.html , in the table for -b #, --blocksize=# says (and have said for years):
Specify the block size in samples. Subset streams must use one of 192/576/1152/2304/4608/256/512/1024/2048/4096 (and 8192/16384 if the sample rate is >48kHz). The reference encoder uses the same block size for the entire stream.
That agrees with the help file and disagrees with the format specification document.
I would argue that as Josh has specifically stated that these were added to the subset later, this is probably simply something he forgot to update. See the three commits linked here: https://github.com/xiph/flac/pull/253
The title of one of the commits is also very clear
fix problem with encoder being too strict about subset blocksizes
starting with that commit, flac accepted 'odd' blocksizes without requiring --lax. His intention seems quite clear to me from this.
Not my decision, but I would suggest that the devs sit on their hands until the results from your test bench are in and evaluated.
If it so turns out that the odd block sizes cause too many failures in decoders(/devices) that are already out in the wild, one might want to consider further restrictions; be it by (I) redefining the subset more restrictively, or (II) keeping the subset but recommending even stricter choices in encoders' standard presets with override switches like --semilax, or (III) by encouraging --safeguards N switches to be employed when problems arise.
The only change that is being proposed is updating the docs. The flac encoder has been doing this since 1.2.1, so for 14 years now. As most people are sticking with the default presets anyway, and there is no good reason to deviate from the 'standard' blocksizes in the presets (because the frame header is larger with the 'non-standard' blocksizes) I don't see much of a problem.
Just a comment on the foobar2000 entry in the table: for me it doesn't actually play file 55 properly. I get "Unsupported format or corrupted file" errors for the indexes, after the first 10 seconds. Version 1.6.7 (the latest stable).
I wonder if it's something on my end or is that a mistake.
EDIT: tried with a clean foobar install and it still doesn't play it properly, with big glitches/pauses between tracks/indexes (sometimes it shows errors, sometimes not).
I did the testing with foobar2000 also with 1.6.7, on a not too recent Windows 10 laptop. While it didn't play track 55 gapless, I didn't count that as an error, as quite a lot of players don't play tracks gapless. My thinking was, just because foorbar2000 honours the cuesheet by splitting the file into separate tracks, doesn't mean I should mark this track as fail because it can't play this track gapless.
However, I just tested again: when simply selecting a track in the playlist and letting it play, I don't get any errors. I do get an error when I try to start a 'cuesheet-track' of track 55 while it is still trying to open a different 'cuesheet-track'. Do you see the same behaviour?
However, I just tested again: when simply selecting a track in the playlist and letting it play, I don't get any errors. I do get an error when I try to start a 'cuesheet-track' of track 55 while it is still trying to open a different 'cuesheet-track'. Do you see the same behaviour?
That's one possible way to trigger it too, but it's inconsistent, not always reproducible for me. Sometimes the errors just don't happen and sometimes they happen even during simple playback from the start, without touching anything. There's also high CPU and RAM usage... so the issues could be somewhat system-related.
But the gaps between chapters are already enough for a Fail, IMO. Compare it with mpv, which plays the entire file smoothly with no interruptions and with low CPU usage. I can't make it pause even if I try.
My fb2k had issues with track 55, in particular on verifying integrity - and the problems are not even consistent on my computer. Also a minor on 54, it shows track number "0".
Anyway,
@Peter is aware of it, so I have just waited for updates rather than testing what behaviour is fb2k-outofthebox vs what might possibly depend on (even his own) components.
As for malformed/corrupted FLAC files...
Reference decoder doesn't report about truncation in some files if they don't have md5. But foobar2000 with foo_verifier and AudioTester can detect that such file is truncated. Sample - https://www.dropbox.com/s/n9awti95o8kt88v/TRUNCATED%20%40%203m%2031s.flac?dl=1
There is (was?) some software (i don't know exactly what it is/was) that ignores restriction for maximal embedded picture size 16 MB. As a result, metadata in FLAC file becomes corrupted. In some very pathological cases it almost impossible to decode such files. See next sample. Reference decoder, foobar2000, ffmpeg are unable to decode this file. VLC can decode it, but resulting file is truncated. The only decoder that i could find that can successfully and properly decode audio from this file is one from XMplay. If XMplay can do this, why reference decoder should be unable to do this? Sample - https://www.dropbox.com/s/1qllhlcxphkam6h/Metadata%20corrupted%20by%20gigantic%20picture.flac?dl=1
Also, it is damn hard to get anything to remove that embedded picture ...
Sample - https://www.dropbox.com/s/1qllhlcxphkam6h/Metadata%20corrupted%20by%20gigantic%20picture.flac?dl=1
There are also 2 STREAMINFO blocks in that file. I just did some hacking, if libFLAC is persuaded to ignore the second STREAMINFO block and flac -F is used, it decodes fine.
I just sent in a patch to fix both problems in flac: https://github.com/xiph/flac/pull/257 If anyone wants a binary let me know.
I've just uploaded a new testbench, revision 5. Link can be found on the wikipage (https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench).
This revision has all testfiles replaced, the testbench is now licensed CC0 1.0 (https://creativecommons.org/publicdomain/zero/1.0/). This revision should also fix the following two issues:
By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.
Just a comment on the foobar2000 entry in the table: for me it doesn't actually play file 55 properly
The first because all files now include a spoken description and there is no chiptune used, the second because current stable Foobar2000 always crashes on file 55 on my end. It doesn't even import properly now.
I'll redo the hardware player checks I've done previously. If you have any hardware capable of playing FLAC, testing it with this testbench and sharing the results would be most welcome.
Opened it and saw something I didn't even think of: there is no mono file. Maybe an insult to the decoders ... but then I also thought of: left/side, right/side, mid/side channel assigments 1000/1001/1010?
As for foobar2000, 1.6.8 beta 5 handles it; foo_verifier gives a warning over file 45 (no total number of samples set), but I just let it ReplayGain scan and tag the files, and rewrite tags; then started afresh and rewrote tags and then ReplayGain scanned and tagged. Works fine, but once it gets to writing 48 to, it spends time. It doesn't support AVIF, but says it is present.
Pretty much the same with Mp3tag (3.08) too.
Oh, and at least when the test bench goes final, it might be an idea to have the readme available separately, so one doesn't have to download everything to get it. I took the liberty to attach it here in case it is useful to anyone.
Opened it and saw something I didn't even think of: there is no mono file. Maybe an insult to the decoders ... but then I also thought of: left/side, right/side, mid/side channel assigments 1000/1001/1010?
Yes, I was thinking of adding a mono file. The stereo decorrelation things are simply 'in the mix': they are so common I don't expect any developer needing a specific test file for it. This is also the case with things like LPC orders: there is no file that specifically test all orders, but they're all in there, somewhere.
I was just thinking of something else: the FLAC specification isn't quite clear on whether or not certain parameters are allowed to change mid-file, like bitdepth, number of channels and samplerate. There are certain codecs that do things like that, for example AC3 in TV streams does it all the time, switching from stereo to multichannel.
Perhaps not something to add to this testbench, but to a different one more focused on potential security issues with crafted files.
The following tests done on a Dell laptop (running Windows 10) through its stereo speakers, no fancier audio devices. I cannot tell what is supposed to be "expected downmixing to stereo", and I cannot know for sure whether Spotify generally refuses to play multichannel, or because it can figure out it is talking to a stereo device.
Spotify for Windows version 1.1.72.439.gc253025e: Here I had to let the library point at the folder for local files.
Tracks 48 through 52 are rejected and so is 55; they are not even indexed by the player.
Tracks 22 and 36 through 44 skip. 44 after seemingly playing silence for four or five seconds before skipping the last ten-ish - but I cannot rule out that the progress bar just runs while it in reality Spotify spends this time to figure out that it is something it wants to skip.
fb2k 1.6.8: like the beta but unlike previous versions, it now handles everything.
Can also confirm the wiki entries for: VLC and SMPlayer 21.8.0 using mpv 0.32.0.
Oh, wait. While VLC for all "end-user-practical" purposes hangs at the infamous 55, it is actually "just" spending minutes processing the file before playing it.
Reason I found out: I tried ffplay, which does the same thing but starts playback much faster. Went back to VLC then. Also tried MPlayer.exe (in the SMPlayer bundle). And mpv.exe, which was so stuck forever that I'm killing it rather than waiting overnight.
I'd say nobody wants to wait neither one nor four minutes though.
VLC on files 27 and 55, opening the messages and media information windows:
27 - old format variable blocksize file created with Flake 0.11.flac Gnaws at the file but one can skip back and forth in the playlist. After some three minutes it makes a short noise, and then apparently it consider itself done with the file.
55 - file 48-53 combined.flac : Starts playing after three to four minutes. If I have another window on top, a popup with metadata infomation shows up in around 100 seconds, that is way earlier than metadata shows up in the Ctrl-i information window.
51 - Extremely large VORBISCOMMENT.flac : Traces of "as 55" - but hangs only for three to four seconds, that is tolerable. (Also 54 hangs for a second.)
ffplay version 4.4.1-essentials_build-www.gyan.dev . Invoked through a FOR loop with -autoexit
03 - blocksize 16.flac plays at like 1/6th of intended speed. Totally unlistenable, of course.
55: Because I see metadata output to terminal window, I can tell it is technically not hanging. Without it, I wouldn't have had the patience. But after 70 seconds, it plays.
MPlayer.exe reported to be the following:
MPlayer Redxii-SVN-r37955-6.2.0 (x86_64) (C) 2000-2017 MPlayer Team
Using FFmpeg N-87137-g6ccd32c367 (2017-08-30 20:24:28 +0200)
Compiled on 2017-09-09 13:22:37 EDT (rev. 1)
36: Stuck at outpt like A: -1.0 (unknown) of 8.0 (08.0) ??,?%, but I can use right arrow (which is supposed to seek 10 seconds!) to get to 6 seconds-ish. It then plays the last couple of seconds.
I think that is a "Fails" when user can skip and get to next without program restart?
55: Plays - but spends around 80 seconds before playing audio.
(MPlayer makes some other quirks that do not affect playback, like showing the GIF as a green rectangle. And I've seen strange times on the 27 on other players too.)
Even if the digital silence thread (https://hydrogenaud.io/index.php?topic=121807.0) may lead to further revision of the testbench, I updated the table with further results.
A couple of questions on distinguishing between Rejects/Skips on one hand, and Fails.
* I have taken that wrong playback means Fails. And outputting silence is wrong playback.
* Edit: JRiver lowers the volume on 8 and 12 bits. I tagged that "Plays", but left a tooltip. Objectionable, but nothing like garbling or wrong speed.
* Some "Rejects" do however tell require you to acknowledge by hitting an OK. Whether that is a "Fails" as in never started playing ... I would disagree. I am a bit more in doubt for the variable block size cell for Exaile should be Rejects or Fails though; it stops, and you have to manually skip to next. I gave it a yellow, thinking that a "Fails" by never starting to play should mean "pretends to be about to start but never does", not an "OK, I give up" (in both cases without hanging), but that is kinda an "err on the side of caution" judgement - where "caution" means "caution against crying wolf and badmouthing for acknowledging not to be able to play".
* Quod Libet had a creative way of "failing" on variable blocksize: it would evidently think it was way further down in its playlist, and so skip past next files which would normally play fine. Anyway, it doesn't matter whether that is "Fails": it makes a chirp sound it on the flake 0.11 and then stops.
Updates committed, in brief:
* Foobar2000: updated with 1.6.8, left a tooltip that 1.6.7 objects.
* Filled in the "new" players mentioned in previous couple of posts. Thinking a bit back and forth, I went with specifying OS as e.g. "Spotify (Windows)", they might differ across platforms - especially the next ones:
* Filled in for JRiver (in the trial period, taking for granted that this does not differ) and Windows versions of several multiplatform players.
Even if the digital silence thread (https://hydrogenaud.io/index.php?topic=121807.0) may lead to further revision of the testbench, I updated the table with further results.
Thanks!
A couple of questions on distinguishing between Rejects/Skips on one hand, and Fails.
* I have taken that wrong playback means Fails. And outputting silence is wrong playback.
I agree
* Edit: JRiver lowers the volume on 8 and 12 bits. I tagged that "Plays", but left a tooltip. Objectionable, but nothing like garbling or wrong speed.
By how much is the volume reduced? One conceivable failure mode is when 8 bits is played back as the lower 8 bits of 16 bits, then volume is reduced by 48dB, which I would find a serious problem. I assume the volume reduction isn't that extreme?
* Some "Rejects" do however tell require you to acknowledge by hitting an OK. Whether that is a "Fails" as in never started playing ... I would disagree.
The four categories do have some grey areas. Not really a problem when the description is in the tooltip I'd say. The categories and colors are mainly there to easily spot differences and see the big picture as the table grows larger.
By how much is the volume reduced? One conceivable failure mode is when 8 bits is played back as the lower 8 bits of 16 bits, then volume is reduced by 48dB, which I would find a serious problem. I assume the volume reduction isn't that extreme?
No, but it could have been 4 bits on the 12 bit file.
I don't remember exactly; I tried to reinstall it to test again, and now it consistently crashes on me.
Here's the thing about those reduced bit rate files, the decoded 32 bit integers always include the samples in the lower n bits, as I found. macOS, on the other hand, expects even n bit to float conversions for the samples to be included in the upper n bits.
I've managed to play the entire test set with Cog. The only current problem is that the 7.1 channel file doesn't output anything but the front left and right outputs, which means that it's doing something wrong with multi-channel output mapping. Cog doesn't currently support specifying speaker mappings, and nor does it support asking the output device for any specific mappings. It only deals in channel counts.
Cog is currently the only other player that plays revision 5 of the testbench perfectly. And it helped me greatly in improving my player's playback queue handling, as it tended to choke badly on playlists of lots of small files.
Here's the thing about those reduced bit rate files, the decoded 32 bit integers always include the samples in the lower n bits, as I found. macOS, on the other hand, expects even n bit to float conversions for the samples to be included in the upper n bits.
If I understand correctly, AudioStreamBasicDescription of Mac can describe bits alignment in PCM samples (either high or low) by kLinearPCMFormatFlagIsAlignedHigh flag.
Uninformed question on WAVEFORMATEXTENSIBLE, that has been a discussion.
If I understand from changelogs, the remark at https://wiki.hydrogenaud.io/index.php?title=Lossless_comparison#Free_Lossless_Audio_Codec_.28FLAC.29 is up-to-date: reference FLAC does the right thing - but there is no expectation on a formats-compliant FLAC decoder that it shall honour a WAVEFORMATEXTENSIBLE channel mask, and assign channels accordingly?
Are there any ... known and relevant consequences here?
I should admit that my entries to the table are based on "do I hear it?". I might have given false green light if some player messed up channel orderings, but I am kinda semi-convinced that I would have picked up a "left" to the right.
(Oh, and ... to #12 and #13, I wonder if there was any opportunity missed here: Once making a test bench, why not include non-subset samples too? That would give an indication on whether/what subset restrictions are significant to playback. I agree with ktf in #13 that it is not surprising that some are not. But then part of the subset's purpose is to ensure streamability, and a bunch of files played offline don't test that.)
By the way, fb2k decodes 3-channels file from testbench incorrectly. According to specification (https://xiph.org/flac/format.html#frame_header) 3 channels file
without channelmask in waveformatextensible must be decoded as left, right, center, but fb2k decodes it as left, right, LFE. 5-channels file
without channelmask in waveformatextensible is decoded by fb2k incorrectly too: FL, FR,
LFE, BL, BR instead of front left, front right,
front center, back/surround left, back/surround right.
@Peter , can you look into this?
3 channels: left, right, center
5 channels: front left, front right, front center, back/surround left, back/surround right
Didn't test any other player for this
Cog doesn't use channel masks yet, as I haven't implemented support for that yet, but will eventually. Currently, it treats:
1 channel: Front center
2 channel: Front left/right
3 channel: Front left, right, center
4 channel: Front left, right, Back left, right
5 channel: Front left, right, center, Back left, right
6 channel: Front left, right, center, LFE, Back left, right
7 channel: Front left, right, center, LFE, Back center, Side left, right
8 channel: Front left, right, center, LFE, Back left, right, Side left, right
I will consider adding support for more channel layouts as they become necessary, but I don't currently have a way to downmix or upmix them dynamically. I currently only have a downmix/upmix processor that supports:
HRIR enabled: those 1-8 channel configs, outputting to a 2 channel device
Disabled, or not matching those rules: those 1-8 channel configs, outputting to an arbitrary 1-8 channel output, with several manual mappings
Contributions of dynamic mappings welcome. Source lives here: https://github.com/losnoco/Cog/blob/main/Audio/Chain/Downmix.m
The Ogg Vorbis and Opus inputs are the only ones that don't map files as-is, and use a Ogg to WAVE channel remapping for up to 8 channel configurations. This briefly had an error mapping 8 channel files, as I checked the channel count incorrectly with a < MAX_CHANNELS instead of <= MAX_CHANNELS.
reference FLAC does the right thing - but there is no expectation on a formats-compliant FLAC decoder that it shall honour a WAVEFORMATEXTENSIBLE channel mask, and assign channels accordingly?
It isn't even included in the FLAC specification, so you can't expect players other than libFLAC to make work of this.
I should admit that my entries to the table are based on "do I hear it?". I might have given false green light if some player messed up channel orderings, but I am kinda semi-convinced that I would have picked up a "left" to the right.
I didn't really check channel orderings either. The only hardware player I had that played these files at all dowmixed them to stereo, and all software players I checked were tested on computers with only stereo output. So, I focussed on "Does it play at all" mostly.
(Oh, and ... to #12 and #13, I wonder if there was any opportunity missed here: Once making a test bench, why not include non-subset samples too? That would give an indication on whether/what subset restrictions are significant to playback. I agree with ktf in #13 that it is not surprising that some are not. But then part of the subset's purpose is to ensure streamability, and a bunch of files played offline don't test that.)
I could include a bunch of headerless files cut somewhere halfway a block to emulate streaming, but this isn't really a good way to test indeed. One can only test streaming behaviour with an actual stream, not a file, as a decoder might incorporate knowledge about whether something is a file or a stream into its decoding logic.
2008 thread, gstreamer and mplayer allegedly affected then: https://hydrogenaud.io/index.php?topic=63269.0
Cog now supports arbitrary channel mappings, and uses WAV channel mapping when FLAC doesn't specify WFX speaker usage bits.
Is there any reason why files 45-54 are 48 kHz and not 44.1 kHz in testbench r5?
No, there is no particular reason. It wasn't intentional, but it doesn't really seem out of place either.
Cog renders the whole set just fine, including the massive cuesheet mess, even with resampling each individual range of audio to the output rate.
Okay, I need to amend Cog, it reads the cuesheets to memory for every file, and every span of every track. This leads to massive memory consumption. I'll deal with this soonish.
Meanwhile, I now do AVIF image decoding, so the AVIF artwork works. Consider adding WebP as well, since Big Sur supports that, and therefore so does Cog on Big Sur.
I've recently uploaded a new version of the testbench, revision 6. It can be found on FLAC decoder testbench page on the wiki (https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench).
4 files have been added, all mono. The first is simply mono (that wasn't already in the testbench), the three other test predictor overflow handling. The last file could not be decoded by ffmpeg until recently, so it is likely ffmpeg-based players not using the very latest git will fail this test until updated.
Updated the Cog entry. All files play correctly in the latest version, though one of the new test files would throw old versions for a loop because it won't support FLAC files whose bit depth isn't a multiple of 8. And that's just a shortcoming of the frontend code of the FLAC input plugin, not because it used an incompatible version of libFLAC.
Edited the introductory text. I think these edits are in line with the facts?
https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench&diff=35608&oldid=35607
I think these edits are in line with the facts?
Yes, that is OK.
Not completely true though, seems that files 26, 27 and 55 differ. (26 and 27 are not bit-identical files, but same size. 55 is different size.)
I should have put forth the point: could one merely test 60 to 63?
Filled in some fields:
* Amarok. I see you have edited that at some point. Anyway, on this computer, it returns silence (not skipping) on the 20-bit file number 37, and then it certainly also does the same on a 20-bit predictor overflow test too.
* mpv and Mplayer based on old ffmpeg fail the last file. Not sure if it was worth mentioning since it is kinda known ... but so I did.
The silence on 20 bit files would be the same reason that Cog failed those files. Arbitrarily only supporting multiples of 8 bits in the code interfacing libFLAC to the player.
It must be really fun where mishandling libFLAC can result in outright freezes or crashes of the player, or worse, crashing the entire embedded OS.
Not completely true though, seems that files 26, 27 and 55 differ. (26 and 27 are not bit-identical files, but same size. 55 is different size.)
I already forgot about those. Yes, there were some minor corrections. See https://github.com/ietf-wg-cellar/flac-test-files/pull/3 for details.
I should have put forth the point: could one merely test 60 to 63?
Yes, I don't think the changes to files 26, 27 and 55 should cause any decoder to fail them where they didn't fail before. The reverse is the case though, as explained in the PR linked above, one player complained about this that weren't strictly according to spec.
Haha, so someone reads specs with a harsher eye than the reference implementation does?
I did some updates, but in case there is a revision 7 ... volunteers, don't bother to do JRiver yet unless you are a paying user. It is only trialware and I couldn't kick it from my computer altogether, so each user who wants to test it, only has 30 days. Also this computer isn't friends with MediaMonkey anymore it seems.
Added VUPlayer, and also: VUPlayer's audiotester.exe verifies them all except complains "EXTRA SAMPLES" on number 45 (not to whine over people who do a job - but if every track had started with speaking out its number, I would have detected skipping without staring at it. Oh, and I have compared the "gritty" the last four seconds of 44 across players.
I just uploaded revision 7 of the testbench, see the wiki page for the link: https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench
Following a bug report (https://github.com/xiph/flac/issues/393) by
@kode54 it turned out file 63 of r6 was faulty, so this is replaced. Also, file 64 is added, testing rice partition with escape code 0. This works in most decoders, but is not decodeable by libFLAC 1.3.3 (that specific version only) because of a bug.
Files 54 and 55 make foobar2000 v2 beta 6 on 32bit windows 7 force runtime error. They are OK in 1.6.12.
And file 64 is detected as "Unsupported format or corrupted file" and unplayable in both fb2k 1.6.12 and v2 beta 6.
Winamp 5.9 Final released.
http://forums.winamp.com/showthread.php?p=3223376#post3223376
Time to run the testbench there.
Please include Mpxplay.
http://mpxplay.sourceforge.net/
Care to do the test then?
Looking back at it, I think the testbench could have included non-subset files: if player P cannot handle file X, is that due to this or that? If it can handle file Y that uses all but one of the features of X and also goes to -l 16 and -r13, that would inform about whether the failures occur due to
* particular features considered unnecessary, like multi-channel playback
* ... or high resolution
* "arbitrary bad design" with no relation to subset bounds other than in coincidences with the previous two items
Files 54 and 55 make foobar2000 v2 beta 6 on 32bit windows 7 force runtime error. They are OK in 1.6.12.
Is that with some custom components?
And file 64 is detected as "Unsupported format or corrupted file" and unplayable in both fb2k 1.6.12 and v2 beta 6.
There will be update for 1.6 series later with new libflac. foobar2000 v2.0 beta 7 was just released that now decodes all these test tracks and 32-bit FLACs.
I think the testbench could have included non-subset files
That's on the TODO list (notice that the files in https://github.com/ietf-wg-cellar/flac-test-files are in a directory called subset), but I don't think will happen soon.
Is that with some custom components?
Yes, indeed. Clean installation has no such problem.
I found problematic component. It was foo_multisource. Notice, that problem only occurs with fb2k v2, not 1.6.12.
File 64 passed in fb2k v2 beta 7!
File 64 fails with XMPlay FLAC decoder revision 13.
All flac files plays just fine with mpv and ffplay.
All flac files plays just fine with mpv and ffplay.
Versions? Do you happen to know what is the earliest ffmpeg version whose ffplay handles everything - since ffmpeg is truly a Big Thing?
Their failures in the current table are with (copy/paste from the margin box note):
* mpv-x86_64-20211031-git-4a80de9 on Windows, testbench revision 5 & files 60-63
* ffmpeg version 4.4.1-essentials_build-www.gyan.dev with testbench revision 5
Also I see that the foobar2000 line is now v2 beta. Given the number of alien cat owners here, I think there at least should be a margin note on how fb2k-stable behaves.
Updates:
* MusicBee - shining colours!
* MediaMonkey has improved between 5.01 and 5.04 ... or maybe it is just how it interacts with other hardware
"Non-updates", just verified for other versions:
* foobar2000: files 60 and up verified with stable version 1.6.14
* ffplay: blocksize 16 still is broken as of today's build N-109368-gcc11afe502-20221206.
ffplay requires block size at least 128, at least on CDDA.
And its errors are apparently in the
player only - not in ffmpeg's "decoder". Nor the encoder.
(1) Initial test: blocksizes 16 and up in steps of 4.
Result: gradually less annoying up to 128, which sounds fine.
(2) Same results when trying all blocksizes 120 to 130 in steps of 1, encoded as follows:
* Using reference flac.exe with both fixed predictors (-2) and various attempted harassments up to -q 15 -l 32 (and some -r that would affect 128 only)
* Using ffmpeg's FLAC encoder, no other options set than -frame_size
(3) Verified the ffmpeg-encodes with flac -t. All fine!
(3) Then used ffmpeg to
decode convert files ffplay couldn't play. (That is, convert to ffmpeg's default blocksize.) All fine!
This indicates that ffmpeg encodes and decodes correctly, it is just ffplay that chokes on too low blocksize.
(
@mycroft , is ffplay your department?)
ffplay is toy and demo app, use mpv always.
I wonder where you can decode FLAC encoded on a Ogg container else than Foobar (which decodes FLAC too - which are smaller in size too)
I tried to have these files opened by some old wave editor that decoded .ogg files to see if it improved the usability of the FLAC format backward but couldn't succeed in any.
Hope to have added something for your testing...
In my opinion them work well (haven't gone into the compression algorythm though, but just run some rip/encode/decode/test routines at level 7 of compression and haven't found anything wrong - from library version 2007 (maybe 1.0.4) to 1.4.1)
good vibes
Today I tested the FLAC testbench v6 on a Toyota Entune 3.0 unit. The older 2.0 units only supported MP3, AAC(LC), and WMV. The new 3.0 units support that plus OGG-Vorbis, hi--res WAV (LPCM), FLAC and ALAC, up to 192/24.
*Anything above 48KHz will be down-sampled to 48KHz/24-bit.
I'll only list what files has issues, anything not listed played fine.
Track 19 - Skip
Track 20 - Skip
Track 35 - Skip
Track 36 - Skip
Track 42 - Skip
Track 43 - Skip
Track 44 - Skip
Track 45 - Plays, but no progress indication.
Track 46 - Skip
Track 51 - Skip
Track 54 - Skip
Are you sure you tested revision 6? r7 has been out for a while.
Assuming you did, this is only the second hardware unit tested that has no failures or freezes in the list. So, apparently most hardware implementations have problems with one or more files that are not gracefully handled.
The file I downloaded only has 63 FLAC files in it. Seems I used the link in post #50, which is a revision of the page that links to v6. I can test again later with v7.
Only file 63 differs, and file 64 has been added between revision 6 and revision 7. Anyway, thanks for testing!
Tested files 63 and 64 from v7. They play fine.
Added to the table. Interpreting "will be down-sampled" as not being part of the decoder (rather subsequent processing) I have not remarked those. Could be filled in "Remarks" if one wants to.
Added to the table. Interpreting "will be down-sampled" as not being part of the decoder (rather subsequent processing) I have not remarked those. Could be filled in "Remarks" if one wants to.
BTW, ability to correctly play files 19-21 depends of device/program to do resampling rather than ability to correctly decode FLAC
Yes, but it would be nice to recognize during parsing of the FLAC file already that the device is unable to render a file properly. In other words, a file with a samplerate that a device cannot play should be gracefully skipped, informing the user the file is unplayable, instead of mangling or even crashing.
I am not sure if resampling is "mangling" when it comes to a hardware player. To pinpoint it: what if the device doesn't even have any digital output? If it decodes the FLAC file then it decodes the FLAC file, even if what the device outputs is analog-only.
Even if an in-car player has a digital connection, how many in-car DACs support sampling rates of 54321 kHz? I'd say it is fair game for the purpose to resample to 48000 or 44100.
With mangling I actually meant playback at wrong speed, not resampling.
54321 is still not as malicious as 76543 (prime). SSRC in foobar supports 54321 but not 76543, setting the output to 76543 the result will be an untouched file with no warning or error. RetroArch and foo_dsp_resampler for example have no issue with 76543.
In the case of hardware resampler chips there are also limitations on maximum resampling ratios. For example Cirrus Logic CS8421, CS8422 and TI SRC4192, SRC4193. I have a very old Roland synth/interface with a very old CS8420.
I was able to test a Mazda Connect unit today in a 2024 CX-90. I'm not sure if Mazda Connect has generations like the Toyota Entune I tested. (Entune 2.0, 3.0 etc)
This unit is a Panasonic unit, running Mazda Connect software version 10010 (https://car.panasonic.jp/oss/h04lmm2/)
Tested with revision 7 of the Flac testbench. (64 tracks)
All files played!
The only minor issue I experienced is when track 27 finishes, it doesn't automatically move onto the next track. The unit doesn't appear to hang, but does require me to manually advance to the next track.
Edit: I'll see if I can find anything in the manual for any limitations, like if any tracks get down sampled...
I've added your results to the wikipage (https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench). Thanks! Nice to see a hardware device has been found that really does playback all files properly.
I couldn't find anything in the manual, or online, about what codecs are supported or any limitations of those codecs. Looking through the licenses, the unit looks to be ARM based, likely a Linux based OS. Seems to be using gstreamer for decoding audio files. I thinks there's support for ALAC as well.