Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: FLAC decoder testbench (Read 33771 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: FLAC decoder testbench

Reply #26 –
I just sent in a patch to fix both problems in flac: https://github.com/xiph/flac/pull/257 If anyone wants a binary let me know.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #27 –
I've just uploaded a new testbench, revision 5. Link can be found on the wikipage.

This revision has all testfiles replaced, the testbench is now licensed CC0 1.0. This revision should also fix the following two issues:

Quote from: Rollin on
By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.
Quote from: Brand on
Just a comment on the foobar2000 entry in the table: for me it doesn't actually play file 55 properly
The first because all files now include a spoken description and there is no chiptune used, the second because current stable Foobar2000 always crashes on file 55 on my end. It doesn't even import properly now.

I'll redo the hardware player checks I've done previously. If you have any hardware capable of playing FLAC, testing it with this testbench and sharing the results would be most welcome.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #28 –
Opened it and saw something I didn't even think of: there is no mono file. Maybe an insult to the decoders ... but then I also thought of: left/side, right/side, mid/side channel assigments 1000/1001/1010?

As for foobar2000, 1.6.8 beta 5 handles it; foo_verifier gives a warning over file 45 (no total number of samples set), but I just let it ReplayGain scan and tag the files, and rewrite tags; then started afresh and rewrote tags and then ReplayGain scanned and tagged. Works fine, but once it gets to writing 48 to, it spends time. It doesn't support AVIF, but says it is present.
Pretty much the same with Mp3tag (3.08) too.

Oh, and at least when the test bench goes final, it might be an idea to have the readme available separately, so one doesn't have to download everything to get it. I took the liberty to attach it here in case it is useful to anyone.

Re: FLAC decoder testbench

Reply #29 –
Quote from: Porcus on
Opened it and saw something I didn't even think of: there is no mono file. Maybe an insult to the decoders ... but then I also thought of: left/side, right/side, mid/side channel assigments 1000/1001/1010?
Yes, I was thinking of adding a mono file. The stereo decorrelation things are simply 'in the mix': they are so common I don't expect any developer needing a specific test file for it. This is also the case with things like LPC orders: there is no file that specifically test all orders, but they're all in there, somewhere.

I was just thinking of something else: the FLAC specification isn't quite clear on whether or not certain parameters are allowed to change mid-file, like bitdepth, number of channels and samplerate. There are certain codecs that do things like that, for example AC3 in TV streams does it all the time, switching from stereo to multichannel.

Perhaps not something to add to this testbench, but to a different one more focused on potential security issues with crafted files.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #30 –
The following tests done on a Dell laptop (running Windows 10) through its stereo speakers, no fancier audio devices. I cannot tell what is supposed to be "expected downmixing to stereo", and I cannot know for sure whether Spotify generally refuses to play multichannel, or because it can figure out it is talking to a stereo device.

Spotify for Windows version 1.1.72.439.gc253025e: Here I had to let the library point at the folder for local files.
Tracks 48 through 52 are rejected and so is 55; they are not even indexed by the player. 
Tracks 22 and 36 through 44 skip.  44 after seemingly playing silence for four or five seconds before skipping the last ten-ish - but I cannot rule out that the progress bar just runs while it in reality Spotify spends this time to figure out that it is something it wants to skip.

fb2k 1.6.8: like the beta but unlike previous versions, it now handles everything.

Can also confirm the wiki entries for: VLC and SMPlayer 21.8.0 using mpv 0.32.0.

Re: FLAC decoder testbench

Reply #31 –
Oh, wait. While VLC for all "end-user-practical" purposes hangs at the infamous 55, it is actually "just" spending minutes processing the file before playing it.

Reason I found out: I tried ffplay, which does the same thing but starts playback much faster.  Went back to VLC then. Also tried MPlayer.exe (in the SMPlayer bundle). And mpv.exe, which was so stuck forever that I'm killing it rather than waiting overnight.

I'd say nobody wants to wait neither one nor four minutes though.


VLC on files 27 and 55, opening the messages and media information windows:
27 - old format variable blocksize file created with Flake 0.11.flac Gnaws at the file but one can skip back and forth in the playlist. After some three minutes it makes a short noise, and then apparently it consider itself done with the file.
55 - file 48-53 combined.flac : Starts playing after three to four minutes. If I have another window on top, a popup with metadata infomation shows up in around 100 seconds, that is way earlier than metadata shows up in the Ctrl-i information window.
51 - Extremely large VORBISCOMMENT.flac : Traces of "as 55" - but hangs only for three to four seconds, that is tolerable. (Also 54 hangs for a second.)


ffplay version 4.4.1-essentials_build-www.gyan.dev . Invoked through a FOR loop with -autoexit
03 - blocksize 16.flac plays at like 1/6th of intended speed. Totally unlistenable, of course.
55: Because I see metadata output to terminal window, I can tell it is technically not hanging. Without it, I wouldn't have had the patience. But after 70 seconds, it plays.


MPlayer.exe reported to be the following:
MPlayer Redxii-SVN-r37955-6.2.0 (x86_64) (C) 2000-2017 MPlayer Team
Using FFmpeg N-87137-g6ccd32c367 (2017-08-30 20:24:28 +0200)
Compiled on 2017-09-09 13:22:37 EDT (rev. 1)

36: Stuck at outpt like A:  -1.0 (unknown) of 8.0 (08.0) ??,?%, but I can use right arrow (which is supposed to seek 10 seconds!) to get to 6 seconds-ish. It then plays the last couple of seconds.
I think that is a "Fails" when user can skip and get to next without program restart?
55: Plays - but spends around 80 seconds before playing audio.

(MPlayer makes some other quirks that do not affect playback, like showing the GIF as a green rectangle. And I've seen strange times on the 27 on other players too.)

Re: FLAC decoder testbench

Reply #32 –
Even if the digital silence thread may lead to further revision of the testbench, I updated the table with further results.

A couple of questions on distinguishing between Rejects/Skips on one hand, and Fails.
* I have taken that wrong playback means Fails. And outputting silence is wrong playback.
* Edit: JRiver lowers the volume on 8 and 12 bits. I tagged that "Plays", but left a tooltip. Objectionable, but nothing like garbling or wrong speed.
* Some "Rejects" do however tell require you to acknowledge by hitting an OK. Whether that is a "Fails" as in never started playing ... I would disagree. I am a bit more in doubt for the variable block size cell for Exaile should be Rejects or Fails though; it stops, and you have to manually skip to next. I gave it a yellow, thinking that a "Fails" by never starting to play should mean "pretends to be about to start but never does", not an "OK, I give up" (in both cases without hanging), but that is kinda an "err on the side of caution" judgement - where "caution" means "caution against crying wolf and badmouthing for acknowledging not to be able to play".
* Quod Libet had a creative way of "failing" on variable blocksize: it would evidently think it was way further down in its playlist, and so skip past next files which would normally play fine. Anyway, it doesn't matter whether that is "Fails": it makes a chirp sound it on the flake 0.11 and then stops.



Updates committed, in brief:

* Foobar2000: updated with 1.6.8, left a tooltip that 1.6.7 objects.
* Filled in the "new" players mentioned in previous couple of posts. Thinking a bit back and forth, I went with specifying OS as e.g. "Spotify (Windows)", they might differ across platforms - especially the next ones:
* Filled in for JRiver (in the trial period, taking for granted that this does not differ) and Windows versions of several multiplatform players.

Re: FLAC decoder testbench

Reply #33 –
Quote from: Porcus on
Even if the digital silence thread may lead to further revision of the testbench, I updated the table with further results.
Thanks!

Quote
A couple of questions on distinguishing between Rejects/Skips on one hand, and Fails.
* I have taken that wrong playback means Fails. And outputting silence is wrong playback.
I agree

Quote
* Edit: JRiver lowers the volume on 8 and 12 bits. I tagged that "Plays", but left a tooltip. Objectionable, but nothing like garbling or wrong speed.
By how much is the volume reduced? One conceivable failure mode is when 8 bits is played back as the lower 8 bits of 16 bits, then volume is reduced by 48dB, which I would find a serious problem. I assume the volume reduction isn't that extreme?

Quote
* Some "Rejects" do however tell require you to acknowledge by hitting an OK. Whether that is a "Fails" as in never started playing ... I would disagree.
The four categories do have some grey areas. Not really a problem when the description is in the tooltip I'd say. The categories and colors are mainly there to easily spot differences and see the big picture as the table grows larger.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #34 –
Quote from: ktf on
By how much is the volume reduced? One conceivable failure mode is when 8 bits is played back as the lower 8 bits of 16 bits, then volume is reduced by 48dB, which I would find a serious problem. I assume the volume reduction isn't that extreme?
No, but it could have been 4 bits on the 12 bit file.

I don't remember exactly; I tried to reinstall it to test again, and now it consistently crashes on me.

Re: FLAC decoder testbench

Reply #35 –
Here's the thing about those reduced bit rate files, the decoded 32 bit integers always include the samples in the lower n bits, as I found. macOS, on the other hand, expects even n bit to float conversions for the samples to be included in the upper n bits.

I've managed to play the entire test set with Cog. The only current problem is that the 7.1 channel file doesn't output anything but the front left and right outputs, which means that it's doing something wrong with multi-channel output mapping. Cog doesn't currently support specifying speaker mappings, and nor does it support asking the output device for any specific mappings. It only deals in channel counts.

Re: FLAC decoder testbench

Reply #36 –
Cog is currently the only other player that plays revision 5 of the testbench perfectly. And it helped me greatly in improving my player's playback queue handling, as it tended to choke badly on playlists of lots of small files.

Re: FLAC decoder testbench

Reply #37 –
Quote from: kode54 on
Here's the thing about those reduced bit rate files, the decoded 32 bit integers always include the samples in the lower n bits, as I found. macOS, on the other hand, expects even n bit to float conversions for the samples to be included in the upper n bits.
If I understand correctly, AudioStreamBasicDescription of Mac can describe bits alignment in PCM samples (either high or low) by kLinearPCMFormatFlagIsAlignedHigh flag.

Re: FLAC decoder testbench

Reply #38 –
Uninformed question on WAVEFORMATEXTENSIBLE, that has been a discussion.
If I understand from changelogs, the remark at https://wiki.hydrogenaud.io/index.php?title=Lossless_comparison#Free_Lossless_Audio_Codec_.28FLAC.29 is up-to-date: reference FLAC does the right thing - but there is no expectation on a formats-compliant FLAC decoder that it shall honour a WAVEFORMATEXTENSIBLE channel mask, and assign channels accordingly?
Are there any ... known and relevant consequences here?

I should admit that my entries to the table are based on "do I hear it?". I might have given false green light if some player messed up channel orderings, but I am kinda semi-convinced that I would have picked up a "left" to the right.


(Oh, and ... to #12 and #13, I wonder if there was any opportunity missed here: Once making a test bench, why not include non-subset samples too? That would give an indication on whether/what subset restrictions are significant to playback.  I agree with ktf in #13 that it is not surprising that some are not. But then part of the subset's purpose is to ensure streamability, and a bunch of files played offline don't test that.)

Re: FLAC decoder testbench

Reply #39 –
By the way, fb2k decodes 3-channels file from testbench incorrectly.  According to specification 3 channels file without channelmask in waveformatextensible must be decoded as left, right, center, but fb2k decodes it as left, right, LFE. 5-channels file without channelmask in waveformatextensible is decoded by fb2k incorrectly too: FL, FR, LFE, BL, BR instead of front left, front right, front center, back/surround left, back/surround right. @Peter , can you look into this?
Code: [Select]
    3 channels: left, right, center
    5 channels: front left, front right, front center, back/surround left, back/surround right
Didn't  test any other player for this

Re: FLAC decoder testbench

Reply #40 –
Cog doesn't use channel masks yet, as I haven't implemented support for that yet, but will eventually. Currently, it treats:

1 channel: Front center
2 channel: Front left/right
3 channel: Front left, right, center
4 channel: Front left, right, Back left, right
5 channel: Front left, right, center, Back left, right
6 channel: Front left, right, center, LFE, Back left, right
7 channel: Front left, right, center, LFE, Back center, Side left, right
8 channel: Front left, right, center, LFE, Back left, right, Side left, right

I will consider adding support for more channel layouts as they become necessary, but I don't currently have a way to downmix or upmix them dynamically. I currently only have a downmix/upmix processor that supports:

HRIR enabled: those 1-8 channel configs, outputting to a 2 channel device
Disabled, or not matching those rules: those 1-8 channel configs, outputting to an arbitrary 1-8 channel output, with several manual mappings

Contributions of dynamic mappings welcome. Source lives here: https://github.com/losnoco/Cog/blob/main/Audio/Chain/Downmix.m

The Ogg Vorbis and Opus inputs are the only ones that don't map files as-is, and use a Ogg to WAVE channel remapping for up to 8 channel configurations. This briefly had an error mapping 8 channel files, as I checked the channel count incorrectly with a < MAX_CHANNELS instead of <= MAX_CHANNELS.

Re: FLAC decoder testbench

Reply #41 –
Quote from: Porcus on
reference FLAC does the right thing - but there is no expectation on a formats-compliant FLAC decoder that it shall honour a WAVEFORMATEXTENSIBLE channel mask, and assign channels accordingly?
It isn't even included in the FLAC specification, so you can't expect players other than libFLAC to make work of this.

Quote
I should admit that my entries to the table are based on "do I hear it?". I might have given false green light if some player messed up channel orderings, but I am kinda semi-convinced that I would have picked up a "left" to the right.
I didn't really check channel orderings either. The only hardware player I had that played these files at all dowmixed them to stereo, and all software players I checked were tested on computers with only stereo output. So, I focussed on "Does it play at all" mostly.

Quote
(Oh, and ... to #12 and #13, I wonder if there was any opportunity missed here: Once making a test bench, why not include non-subset samples too? That would give an indication on whether/what subset restrictions are significant to playback.  I agree with ktf in #13 that it is not surprising that some are not. But then part of the subset's purpose is to ensure streamability, and a bunch of files played offline don't test that.)
I could include a bunch of headerless files cut somewhere halfway a block to emulate streaming, but this isn't really a good way to test indeed. One can only test streaming behaviour with an actual stream, not a file, as a decoder might incorporate knowledge about whether something is a file or a stream into its decoding logic.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #43 –
Cog now supports arbitrary channel mappings, and uses WAV channel mapping when FLAC doesn't specify WFX speaker usage bits.

Re: FLAC decoder testbench

Reply #44 –
Is there any reason why files 45-54 are 48 kHz and not 44.1 kHz in testbench r5?

Re: FLAC decoder testbench

Reply #45 –
No, there is no particular reason. It wasn't intentional, but it doesn't really seem out of place either.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #46 –
Cog renders the whole set just fine, including the massive cuesheet mess, even with resampling each individual range of audio to the output rate.

Re: FLAC decoder testbench

Reply #47 –
Okay, I need to amend Cog, it reads the cuesheets to memory for every file, and every span of every track. This leads to massive memory consumption. I'll deal with this soonish.

Meanwhile, I now do AVIF image decoding, so the AVIF artwork works. Consider adding WebP as well, since Big Sur supports that, and therefore so does Cog on Big Sur.

Re: FLAC decoder testbench

Reply #48 –
I've recently uploaded a new version of the testbench, revision 6. It can be found on FLAC decoder testbench page on the wiki.

4 files have been added, all mono. The first is simply mono (that wasn't already in the testbench), the three other test predictor overflow handling. The last file could not be decoded by ffmpeg until recently, so it is likely ffmpeg-based players not using the very latest git will fail this test until updated.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #49 –
Updated the Cog entry. All files play correctly in the latest version, though one of the new test files would throw old versions for a loop because it won't support FLAC files whose bit depth isn't a multiple of 8. And that's just a shortcoming of the frontend code of the FLAC input plugin, not because it used an incompatible version of libFLAC.