Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: FLAC decoder testbench (Read 4100 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: FLAC decoder testbench

Reply #26
I just sent in a patch to fix both problems in flac: https://github.com/xiph/flac/pull/257 If anyone wants a binary let me know.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #27
I've just uploaded a new testbench, revision 5. Link can be found on the wikipage.

This revision has all testfiles replaced, the testbench is now licensed CC0 1.0. This revision should also fix the following two issues:

By the way, evaluation of correct/incorrect playback speed for non-44.1 files (19-21) on unfamiliar material, especially on chiptune-like music, is not very straightforward.
Just a comment on the foobar2000 entry in the table: for me it doesn't actually play file 55 properly
The first because all files now include a spoken description and there is no chiptune used, the second because current stable Foobar2000 always crashes on file 55 on my end. It doesn't even import properly now.

I'll redo the hardware player checks I've done previously. If you have any hardware capable of playing FLAC, testing it with this testbench and sharing the results would be most welcome.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #28
Opened it and saw something I didn't even think of: there is no mono file. Maybe an insult to the decoders ... but then I also thought of: left/side, right/side, mid/side channel assigments 1000/1001/1010?

As for foobar2000, 1.6.8 beta 5 handles it; foo_verifier gives a warning over file 45 (no total number of samples set), but I just let it ReplayGain scan and tag the files, and rewrite tags; then started afresh and rewrote tags and then ReplayGain scanned and tagged. Works fine, but once it gets to writing 48 to, it spends time. It doesn't support AVIF, but says it is present.
Pretty much the same with Mp3tag (3.08) too.

Oh, and at least when the test bench goes final, it might be an idea to have the readme available separately, so one doesn't have to download everything to get it. I took the liberty to attach it here in case it is useful to anyone.
High Voltage socket-nose-avatar

Re: FLAC decoder testbench

Reply #29
Opened it and saw something I didn't even think of: there is no mono file. Maybe an insult to the decoders ... but then I also thought of: left/side, right/side, mid/side channel assigments 1000/1001/1010?
Yes, I was thinking of adding a mono file. The stereo decorrelation things are simply 'in the mix': they are so common I don't expect any developer needing a specific test file for it. This is also the case with things like LPC orders: there is no file that specifically test all orders, but they're all in there, somewhere.

I was just thinking of something else: the FLAC specification isn't quite clear on whether or not certain parameters are allowed to change mid-file, like bitdepth, number of channels and samplerate. There are certain codecs that do things like that, for example AC3 in TV streams does it all the time, switching from stereo to multichannel.

Perhaps not something to add to this testbench, but to a different one more focused on potential security issues with crafted files.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #30
The following tests done on a Dell laptop (running Windows 10) through its stereo speakers, no fancier audio devices. I cannot tell what is supposed to be "expected downmixing to stereo", and I cannot know for sure whether Spotify generally refuses to play multichannel, or because it can figure out it is talking to a stereo device.

Spotify for Windows version 1.1.72.439.gc253025e: Here I had to let the library point at the folder for local files.
Tracks 48 through 52 are rejected and so is 55; they are not even indexed by the player. 
Tracks 22 and 36 through 44 skip.  44 after seemingly playing silence for four or five seconds before skipping the last ten-ish - but I cannot rule out that the progress bar just runs while it in reality Spotify spends this time to figure out that it is something it wants to skip.

fb2k 1.6.8: like the beta but unlike previous versions, it now handles everything.

Can also confirm the wiki entries for: VLC and SMPlayer 21.8.0 using mpv 0.32.0.

High Voltage socket-nose-avatar

Re: FLAC decoder testbench

Reply #31
Oh, wait. While VLC for all "end-user-practical" purposes hangs at the infamous 55, it is actually "just" spending minutes processing the file before playing it.

Reason I found out: I tried ffplay, which does the same thing but starts playback much faster.  Went back to VLC then. Also tried MPlayer.exe (in the SMPlayer bundle). And mpv.exe, which was so stuck forever that I'm killing it rather than waiting overnight.

I'd say nobody wants to wait neither one nor four minutes though.


VLC on files 27 and 55, opening the messages and media information windows:
27 - old format variable blocksize file created with Flake 0.11.flac Gnaws at the file but one can skip back and forth in the playlist. After some three minutes it makes a short noise, and then apparently it consider itself done with the file.
55 - file 48-53 combined.flac : Starts playing after three to four minutes. If I have another window on top, a popup with metadata infomation shows up in around 100 seconds, that is way earlier than metadata shows up in the Ctrl-i information window.
51 - Extremely large VORBISCOMMENT.flac : Traces of "as 55" - but hangs only for three to four seconds, that is tolerable. (Also 54 hangs for a second.)


ffplay version 4.4.1-essentials_build-www.gyan.dev . Invoked through a FOR loop with -autoexit
03 - blocksize 16.flac plays at like 1/6th of intended speed. Totally unlistenable, of course.
55: Because I see metadata output to terminal window, I can tell it is technically not hanging. Without it, I wouldn't have had the patience. But after 70 seconds, it plays.


MPlayer.exe reported to be the following:
MPlayer Redxii-SVN-r37955-6.2.0 (x86_64) (C) 2000-2017 MPlayer Team
Using FFmpeg N-87137-g6ccd32c367 (2017-08-30 20:24:28 +0200)
Compiled on 2017-09-09 13:22:37 EDT (rev. 1)

36: Stuck at outpt like A:  -1.0 (unknown) of 8.0 (08.0) ??,?%, but I can use right arrow (which is supposed to seek 10 seconds!) to get to 6 seconds-ish. It then plays the last couple of seconds.
I think that is a "Fails" when user can skip and get to next without program restart?
55: Plays - but spends around 80 seconds before playing audio.

(MPlayer makes some other quirks that do not affect playback, like showing the GIF as a green rectangle. And I've seen strange times on the 27 on other players too.)

High Voltage socket-nose-avatar

Re: FLAC decoder testbench

Reply #32
Even if the digital silence thread may lead to further revision of the testbench, I updated the table with further results.

A couple of questions on distinguishing between Rejects/Skips on one hand, and Fails.
* I have taken that wrong playback means Fails. And outputting silence is wrong playback.
* Edit: JRiver lowers the volume on 8 and 12 bits. I tagged that "Plays", but left a tooltip. Objectionable, but nothing like garbling or wrong speed.
* Some "Rejects" do however tell require you to acknowledge by hitting an OK. Whether that is a "Fails" as in never started playing ... I would disagree. I am a bit more in doubt for the variable block size cell for Exaile should be Rejects or Fails though; it stops, and you have to manually skip to next. I gave it a yellow, thinking that a "Fails" by never starting to play should mean "pretends to be about to start but never does", not an "OK, I give up" (in both cases without hanging), but that is kinda an "err on the side of caution" judgement - where "caution" means "caution against crying wolf and badmouthing for acknowledging not to be able to play".
* Quod Libet had a creative way of "failing" on variable blocksize: it would evidently think it was way further down in its playlist, and so skip past next files which would normally play fine. Anyway, it doesn't matter whether that is "Fails": it makes a chirp sound it on the flake 0.11 and then stops.



Updates committed, in brief:

* Foobar2000: updated with 1.6.8, left a tooltip that 1.6.7 objects.
* Filled in the "new" players mentioned in previous couple of posts. Thinking a bit back and forth, I went with specifying OS as e.g. "Spotify (Windows)", they might differ across platforms - especially the next ones:
* Filled in for JRiver (in the trial period, taking for granted that this does not differ) and Windows versions of several multiplatform players.
High Voltage socket-nose-avatar

Re: FLAC decoder testbench

Reply #33
Even if the digital silence thread may lead to further revision of the testbench, I updated the table with further results.
Thanks!

Quote
A couple of questions on distinguishing between Rejects/Skips on one hand, and Fails.
* I have taken that wrong playback means Fails. And outputting silence is wrong playback.
I agree

Quote
* Edit: JRiver lowers the volume on 8 and 12 bits. I tagged that "Plays", but left a tooltip. Objectionable, but nothing like garbling or wrong speed.
By how much is the volume reduced? One conceivable failure mode is when 8 bits is played back as the lower 8 bits of 16 bits, then volume is reduced by 48dB, which I would find a serious problem. I assume the volume reduction isn't that extreme?

Quote
* Some "Rejects" do however tell require you to acknowledge by hitting an OK. Whether that is a "Fails" as in never started playing ... I would disagree.
The four categories do have some grey areas. Not really a problem when the description is in the tooltip I'd say. The categories and colors are mainly there to easily spot differences and see the big picture as the table grows larger.
Music: sounds arranged such that they construct feelings.

Re: FLAC decoder testbench

Reply #34
By how much is the volume reduced? One conceivable failure mode is when 8 bits is played back as the lower 8 bits of 16 bits, then volume is reduced by 48dB, which I would find a serious problem. I assume the volume reduction isn't that extreme?
No, but it could have been 4 bits on the 12 bit file.

I don't remember exactly; I tried to reinstall it to test again, and now it consistently crashes on me.
High Voltage socket-nose-avatar

Re: FLAC decoder testbench

Reply #35
Here's the thing about those reduced bit rate files, the decoded 32 bit integers always include the samples in the lower n bits, as I found. macOS, on the other hand, expects even n bit to float conversions for the samples to be included in the upper n bits.

I've managed to play the entire test set with Cog. The only current problem is that the 7.1 channel file doesn't output anything but the front left and right outputs, which means that it's doing something wrong with multi-channel output mapping. Cog doesn't currently support specifying speaker mappings, and nor does it support asking the output device for any specific mappings. It only deals in channel counts.

Re: FLAC decoder testbench

Reply #36
Cog is currently the only other player that plays revision 5 of the testbench perfectly. And it helped me greatly in improving my player's playback queue handling, as it tended to choke badly on playlists of lots of small files.

 

Re: FLAC decoder testbench

Reply #37
Here's the thing about those reduced bit rate files, the decoded 32 bit integers always include the samples in the lower n bits, as I found. macOS, on the other hand, expects even n bit to float conversions for the samples to be included in the upper n bits.
If I understand correctly, AudioStreamBasicDescription of Mac can describe bits alignment in PCM samples (either high or low) by kLinearPCMFormatFlagIsAlignedHigh flag.