Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Vorbis, Opus and Musepack transparency test (Read 17626 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Vorbis, Opus and Musepack transparency test

I've been doing a personal listening test to pick a format to encode my library into for mobile listening, so I figured I might as well share it with people 8) This test focuses on the high end of the audible range.

Software: Windows 10, foobar2000 1.3.10 with ABX Comparator 2.0.2, pass is 13 out of 16 trials
Hardware: Laptop's onboard sound chip, Superlux 668B
Song: Noshi - Realizing The Life (uplifting trance), flac rip
Encoders: Vorbis aoTuV beta5.7, Opus 1.1.3, Musepack 1.30.0


Vorbis -q 5 (175 kbps)
Difficulty: easy
Spoiler (click to show/hide)

Vorbis -q 6 (206 kbps)
Difficulty: medium
Spoiler (click to show/hide)

Vorbis -q 7 (249 kbps)
Difficulty: hard
Spoiler (click to show/hide)

Vorbis -q 8 (279 kbps)
Difficulty: hard
Spoiler (click to show/hide)

Vorbis -q 9 (351 kbps)
Difficulty: failed
Spoiler (click to show/hide)


Opus --bitrate 176 (179 kbps)
Difficulty: medium
Spoiler (click to show/hide)

Opus --bitrate 192 (196 kbps)
Difficulty: hard
Spoiler (click to show/hide)

Opus --bitrate 224 (229 kbps)
Difficulty: failed
Spoiler (click to show/hide)


Musepack --standard (180 kbps)
Difficulty: medium
Spoiler (click to show/hide)

Musepack --extreme (210 kbps)
Difficulty: medium
Spoiler (click to show/hide)

Musepack --insane (236 kbps)
Difficulty: failed
Spoiler (click to show/hide)


Conclusion
There's no denying that Opus is the state of the art right now, it sounds amazing for the bitrate and even when it does alter the sound the cut is clean and undistracting. Vorbis, on the other hand, turns the high end into a mush of white noise until it has enough bits to encode it properly. Musepack sits nicely between the two, altering the sound but in a way that doesn't stand out. Opus really impressed me in this test and I'll be using it from now on at 256kbps.

30s sample file is attached. The full song was used in the test, so bitrates might be slightly different. The (in my opinion) killer sound is the offbeat buzz that kicks in 10s into this preview.
 

Re: Vorbis, Opus and Musepack transparency test

Reply #1 –
Thanks for sharing those results. I'm looking forward to do a listening test myself, especially to try Vorbis and Opus. Actually never tested those formats before, always used LAME MP3 until now.

Did you consider also comparing LAME against the other ones? Or are you one of those people who can still hear a difference between LAME and FLAC at -V0?

Re: Vorbis, Opus and Musepack transparency test

Reply #2 –
Quote from: Tear♪ on
[...] Opus [...] sounds amazing for the bitrate and even when it does alter the sound the cut is clean and undistracting.

In what way does it alter the sound to your ears?

Re: Vorbis, Opus and Musepack transparency test

Reply #3 –
The problem is again 256k ~ 320k is much larger than 176k.  Some codecs performed well @176 like opus and mpc have potential for efficiency /quality tradeoff. Second issue is that @ 256k~320 its possible that even mp3 may be competitive. 

Re: Vorbis, Opus and Musepack transparency test

Reply #4 –
I would love to see where qaac stands among those with this sample.

Re: Vorbis, Opus and Musepack transparency test

Reply #5 –
Quote from: pmattke93 on
Thanks for sharing those results. I'm looking forward to do a listening test myself, especially to try Vorbis and Opus. Actually never tested those formats before, always used LAME MP3 until now.

Did you consider also comparing LAME against the other ones? Or are you one of those people who can still hear a difference between LAME and FLAC at -V0?

I'm not interested in patented codecs, I've heard stories of patent holders demanding a cut of mp3 sales from artists, it's like a paper producer asking for a cut of book sales. And free ones work well enough.

Honestly, the question of "can you hear a difference between A and B" by itself isn't any more meaningful than "what is the fps of human eye", it all depends on the content. I'm sure there's some killer samples that trip up mp3 320kbps vbr but for most songs 128kbps opus or even 96 is transparent to me. I picked one that's very difficult for codecs to handle on purpose, so that if this is transparent then everything else in my library should be. Ideally one would ABX every song to find the lowest setting that works great, or have an encoder with a reliable "quality" setting that maintains constant fidelity with no regard for bitrate (which to my knowledge doesn't exist yet).

Re: Vorbis, Opus and Musepack transparency test

Reply #6 –
Quote from: Tear♪ on
I'm not interested in patented codecs, I've heard stories of patent holders demanding a cut of mp3 sales from artists
https://en.wikipedia.org/wiki/MP3#Licensing.2C_ownership_and_legislation

Quote
it's like a paper producer asking for a cut of book sales.
No, it's not like that at all.

Quote from: Tear♪ on
And free ones work well enough.
Quote
Today, LAME is considered the best MP3 encoder at mid-high bitrates and at VBR, mostly thanks to the dedicated work of its developers and the open source licensing model that allowed the project to tap into engineering resources from all around the world. Both quality and speed improvements are still happening, probably making LAME the only MP3 encoder still being actively developed.

If you mean free, as in beer, then just about everything can be done without the end user having to pay a cent.  While in some places there may be some gray spots to the legality, the same goes for getting legally obtained content on to your hard drive in order to encode it.

Re: Vorbis, Opus and Musepack transparency test

Reply #7 –
Hi Tear♪, can you tell us how old are you? I am interested to know about the relationship between hearing ability and age. You seem to be quite sensitive about the differences of different codecs.

Re: Vorbis, Opus and Musepack transparency test

Reply #8 –
Isn't it aoTuVb6.03 the latest build? I tried for myself the test sample with aoTuVb6.03 (libVorbis 1.3.5) at Q7 and Q5, and I couldn't find any difference.

Re: Vorbis, Opus and Musepack transparency test

Reply #9 –
Quote from: Tear♪ on
Quote from: pmattke93 on
Thanks for sharing those results. I'm looking forward to do a listening test myself, especially to try Vorbis and Opus. Actually never tested those formats before, always used LAME MP3 until now.

Did you consider also comparing LAME against the other ones? Or are you one of those people who can still hear a difference between LAME and FLAC at -V0?

I'm not interested in patented codecs, I've heard stories of patent holders demanding a cut of mp3 sales from artists, it's like a paper producer asking for a cut of book sales. And free ones work well enough.

Honestly, the question of "can you hear a difference between A and B" by itself isn't any more meaningful than "what is the fps of human eye", it all depends on the content. I'm sure there's some killer samples that trip up mp3 320kbps vbr but for most songs 128kbps opus or even 96 is transparent to me. I picked one that's very difficult for codecs to handle on purpose, so that if this is transparent then everything else in my library should be. Ideally one would ABX every song to find the lowest setting that works great, or have an encoder with a reliable "quality" setting that maintains constant fidelity with no regard for bitrate (which to my knowledge doesn't exist yet).

Outside of the (dead) 5.1 multichannel mp3 extension, I think all mp3 essential patents have expired, at least in the US and EU. 

Re: Vorbis, Opus and Musepack transparency test

Reply #10 –
Looks like patents of MP3 decoding have already expired. 

As for encoder MP3 patents they expire in April 2017.
https://hydrogenaud.io/index.php/topic,106938.msg895903.html#msg895903

Also some interesting formats will be patent free soon (if are not already). G.729 patents expire in 2016 and AC-3 in 2017.

Re: Vorbis, Opus and Musepack transparency test

Reply #11 –
Also using only 1 sample maybe deceptive - you should not draw a conclusion based solely on that.

If 128k is fine for most material then going up twice to 256k is very large step. It could also be that it isn't enough for undiscovered killer samples. One should think 150 .. 190k range would be the headroom setting. Opus is much less compatible than mp3 and other some lossy. IMO this is not an win situation. It may even be for many listeners mp3 176k - V3 or even 150k V4 are fine for vast majority of material. The other alternative is going lossless for limited collections . flac is a more supported open format.

Another option if you have the CD's or lossless versions at hand is to encode everything @ opus 160k .  If you find a rare problem you can always re-encode that track at different setting. If the difference is slight and not annoying [as it should be] you can just ignore it. Many times for older mobile devices I used mp3 160k CBR [GOGO] with very reasonable results.

Re: Vorbis, Opus and Musepack transparency test

Reply #12 –
I tested this sample using mp3  128 and 160 abr gogo. Also lame 3.99 V5, V4, V3, V4 -f, V3 -f

@ 128 there are obvious issues and some can be ugly - pre echo and ringing. Gogo 160 still help up respectably. Lame vbr had issues in certain sports. V5 [160k] was not transparent obviously but I thought not bad until the 2.2 sec mark. Its is kind of ugly even with V4 / V4 -f [180k].  With V3 / V3 -f [194k], I thought the results are quite good and not annoying - the best of the lot.

I can see that with this type of electronic music there can be problems and its hard to be 100% pleased depending on the listener and the situation. Despite this I remain convinced that a good encode of 160..190k is the best tradeoff (IMO) .

Re: Vorbis, Opus and Musepack transparency test

Reply #13 –
Opus is not for music, it use resampling (48 Khz), right ?
so i guess Ogg is better....
but maybe i'm wrong
we have to test at 224 kps with the latest build of ogg aoTuVb6.03...

sorry for my ugly english ;)

Re: Vorbis, Opus and Musepack transparency test

Reply #14 –
Quote from: david-lisb on
Opus is not for music, it use resampling (48 Khz), right ?
so i guess Ogg is better....
but maybe i'm wrong
we have to test at 224 kps with the latest build of ogg aoTuVb6.03...

sorry for my ugly english ;)

48kHz is fine for music.  Ogg is not a codec, it is a container.  I'll assume that you meant Vorbis.

Opus has demonstrably better audio quality at bitrates up to perhaps 160 kbps.  Above that, it becomes difficult to reliably distinguish them from eachother or from a lossless source.  If you read this thread (go on, that's the whole point!) then you'll see that an experienced listener with great care (and a suitable sample that is difficult to encode) can distinguish Vorbis from a lossless sample even above 200 kbps, while Opus is indistinguishable at 196 kbps.  If those small fractions are important to you then you should probably choose Opus, although your experience may differ and the encoded quality definitely varies with the type of music.  Otherwise, check carefully how well Opus is supported on your hardware or software.  Features like replaygain are still very patchy in Opus and even on relatively new hardware you will often find that Opus isn't supported.

Lastly, aoTuV beta 6.03 does not offer any quality improvements over 5.7, so no real problem there.

Re: Vorbis, Opus and Musepack transparency test

Reply #15 –
Actually the latest aoTuV b6.03 has more consistent quality than b5.7.  It boosts bitrate more on critic parts to keep the quality.
http://d.hatena.ne.jp/kamedo2/20110409/1302373616 

While the average scores are practically the same a quality is more constant amongst different samples.

P.S. It's prefferable to have a codec which produces the same quality for two samples (4.0 and 4.0) instead of (3.0 and 5.0, or 3.5 and 4.5 etc...).
More constant quality means better quality.

Re: Vorbis, Opus and Musepack transparency test

Reply #17 –
I wonder how results would differ if the sample would be normalized to -1 dB peak or lower, it makes a huge difference for encoding efficiency in my experience, at least at lower qualities.