I've been doing a personal listening test to pick a format to encode my library into for mobile listening, so I figured I might as well share it with people 8) This test focuses on the high end of the audible range.
Software: Windows 10, foobar2000 1.3.10 with ABX Comparator 2.0.2, pass is 13 out of 16 trials
Hardware: Laptop's onboard sound chip, Superlux 668B
Song: Noshi - Realizing The Life (uplifting trance), flac rip
Encoders: Vorbis aoTuV beta5.7, Opus 1.1.3, Musepack 1.30.0
Vorbis -q 5 (175 kbps)
Difficulty:
easyfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 10:59:26
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: q5.ogg
SHA1: c15863217f856cd1c86c632c1a1ee43f7478bde5
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
10:59:26 : Test started.
11:00:29 : 01/01
11:01:04 : 02/02
11:01:13 : 03/03
11:01:20 : 04/04
11:01:29 : 05/05
11:01:35 : 06/06
11:01:43 : 07/07
11:01:49 : 08/08
11:01:55 : 09/09
11:02:01 : 10/10
11:02:05 : 11/11
11:02:11 : 12/12
11:02:21 : 13/13
11:02:35 : 14/14
11:02:41 : 15/15
11:02:49 : 16/16
11:02:49 : Test finished.
----------
Total: 16/16
Probability that you were guessing: 0.0%
-- signature --
9034b2bd3d9f7c9bb00872769b0ab067d51865f7
Vorbis -q 6 (206 kbps)
Difficulty:
mediumfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 11:04:21
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: q6.ogg
SHA1: 8673bdf9f4a944048cd6c4e8f5d74edd02b187cb
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
11:04:21 : Test started.
11:05:20 : 01/01
11:05:30 : 02/02
11:05:43 : 03/03
11:05:51 : 04/04
11:06:02 : 05/05
11:06:06 : 06/06
11:06:11 : 07/07
11:06:19 : 08/08
11:07:38 : 09/09
11:07:43 : 10/10
11:07:49 : 11/11
11:07:54 : 12/12
11:07:59 : 13/13
11:08:05 : 14/14
11:08:14 : 15/15
11:08:19 : 16/16
11:08:19 : Test finished.
----------
Total: 16/16
Probability that you were guessing: 0.0%
-- signature --
5078b8e1d476316f8f02e6df16246a8c80504dc4
Vorbis -q 7 (249 kbps)
Difficulty:
hardfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 11:17:23
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: q7.ogg
SHA1: 39ee020c20ff8a38bf6fd7001d6ac6d02cb8b6e8
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
11:17:23 : Test started.
11:20:10 : 01/01
11:20:20 : 02/02
11:23:11 : 03/03
11:23:20 : 04/04
11:23:31 : 05/05
11:23:44 : 06/06
11:27:14 : 07/07
11:27:26 : 08/08
11:27:39 : 09/09
11:27:44 : 10/10
11:27:52 : 11/11
11:28:12 : 11/12
11:28:22 : 12/13
11:28:30 : 13/14
11:31:32 : 14/15
11:31:36 : 15/16
11:31:36 : Test finished.
----------
Total: 15/16
Probability that you were guessing: 0.0%
-- signature --
aff6dcef53d361ea43df301ab73887fae66f35bc
Vorbis -q 8 (279 kbps)
Difficulty:
hardfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 12:08:11
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: q8.ogg
SHA1: 0863d4be7b459076913da56c44c430dbdfa46ada
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
12:08:11 : Test started.
12:12:31 : 01/01
12:13:02 : 02/02
12:13:21 : 03/03
12:13:57 : 03/04
12:14:21 : 04/05
12:14:44 : 05/06
12:14:56 : 06/07
12:15:21 : 07/08
12:16:35 : 08/09
12:16:49 : 09/10
12:17:02 : 10/11
12:17:52 : 10/12
12:21:49 : 11/13
12:21:56 : 12/14
12:22:05 : 13/15
12:23:52 : 14/16
12:23:52 : Test finished.
----------
Total: 14/16
Probability that you were guessing: 0.2%
-- signature --
cebc6e14d326dd9fa85a3f9508dbe1d4a7451c6b
Vorbis -q 9 (351 kbps)
Difficulty:
failedfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 12:32:47
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: q9.ogg
SHA1: 772d02625f2d1ce174eb32de0de9a67778f13238
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
12:32:47 : Test started.
12:33:57 : 01/01
12:34:04 : 02/02
12:34:20 : 03/03
12:34:30 : 04/04
12:36:12 : 04/05
12:36:20 : 04/06
12:36:49 : 05/07
12:37:30 : 06/08
12:40:17 : 07/09
12:41:03 : 07/10
12:41:29 : 08/11
12:42:11 : 09/12
12:43:06 : 10/13
12:43:16 : 11/14
12:43:34 : 11/15
12:43:58 : 12/16
12:43:58 : Test finished.
----------
Total: 12/16
Probability that you were guessing: 3.8%
-- signature --
63e54a157d10dd09b2fc7b138227f339665eed13
Opus --bitrate 176 (179 kbps)
Difficulty:
mediumfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 12:56:17
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: 176.opus
SHA1: 9f281654a248b522db7b9b4e86bd00979feeac12
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
12:56:17 : Test started.
12:56:49 : 01/01
12:56:57 : 02/02
12:57:03 : 03/03
12:57:08 : 04/04
12:57:11 : 05/05
12:57:18 : 06/06
12:57:24 : 07/07
12:57:44 : 08/08
12:57:50 : 09/09
12:57:55 : 10/10
12:58:00 : 11/11
12:58:05 : 12/12
12:58:08 : 13/13
12:58:12 : 14/14
12:58:15 : 15/15
12:58:19 : 16/16
12:58:19 : Test finished.
----------
Total: 16/16
Probability that you were guessing: 0.0%
-- signature --
3ed943029184ed7ea5143218aeea0877f76e956b
Opus --bitrate 192 (196 kbps)
Difficulty:
hardfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 13:04:28
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: 192.opus
SHA1: 9575757b6987bc2f9276284b9ce7694be580cbc9
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
13:04:28 : Test started.
13:04:48 : 01/01
13:05:05 : 02/02
13:05:10 : 03/03
13:05:17 : 04/04
13:05:29 : 05/05
13:05:34 : 06/06
13:05:39 : 07/07
13:05:50 : 08/08
13:08:29 : 09/09
13:08:36 : 10/10
13:08:47 : 11/11
13:09:06 : 12/12
13:11:00 : 13/13
13:11:18 : 14/14
13:11:28 : 14/15
13:11:35 : 15/16
13:11:35 : Test finished.
----------
Total: 15/16
Probability that you were guessing: 0.0%
-- signature --
2bd1e359556f97dd9374d493396a00697f5eab7d
Opus --bitrate 224 (229 kbps)
Difficulty:
failedfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 13:16:41
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: 224.opus
SHA1: 7e3ef06c89764e433336cc87b56b37ece29b4d3f
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
13:16:41 : Test started.
13:17:49 : 00/01
13:18:16 : 00/02
13:18:35 : 01/03
13:18:43 : 02/04
13:19:41 : 02/05
13:19:48 : 03/06
13:19:56 : 04/07
13:20:07 : 05/08
13:22:55 : 06/09
13:23:03 : 07/10
13:23:17 : 08/11
13:23:26 : 09/12
13:23:43 : 09/13
13:23:57 : 10/14
13:24:11 : 11/15
13:24:19 : 12/16
13:24:19 : Test finished.
----------
Total: 12/16
Probability that you were guessing: 3.8%
-- signature --
a46da06b997542c4a28b4de3fbfd3f66f404f37d
Musepack --standard (180 kbps)
Difficulty:
mediumfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 13:31:39
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: standard.mpc
SHA1: c822d9874d4dd7569bb3dc0748a936f8cc5ab3d9
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
13:31:39 : Test started.
13:32:56 : 01/01
13:33:03 : 02/02
13:33:08 : 03/03
13:33:12 : 04/04
13:33:19 : 05/05
13:33:26 : 06/06
13:33:41 : 07/07
13:33:48 : 08/08
13:33:53 : 09/09
13:35:58 : 10/10
13:36:02 : 11/11
13:36:08 : 12/12
13:36:13 : 13/13
13:36:18 : 14/14
13:36:27 : 15/15
13:36:34 : 16/16
13:36:34 : Test finished.
----------
Total: 16/16
Probability that you were guessing: 0.0%
-- signature --
438d6f82033872169a5b5557c0c2cdf0ba389aa6
Musepack --extreme (210 kbps)
Difficulty:
mediumfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 13:40:30
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: extreme.mpc
SHA1: 2db71a0522b4fde5f9b5ee423c108871a5e936ad
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
13:40:30 : Test started.
13:40:52 : 01/01
13:41:06 : 02/02
13:41:16 : 03/03
13:41:24 : 04/04
13:43:01 : 05/05
13:43:09 : 06/06
13:43:15 : 07/07
13:43:22 : 08/08
13:43:37 : 09/09
13:44:11 : 10/10
13:44:17 : 11/11
13:44:26 : 12/12
13:44:36 : 13/13
13:44:47 : 14/14
13:44:55 : 15/15
13:45:05 : 16/16
13:45:05 : Test finished.
----------
Total: 16/16
Probability that you were guessing: 0.0%
-- signature --
dc2e19cc507408d081f549e6ab8254e0414759b4
Musepack --insane (236 kbps)
Difficulty:
failedfoo_abx 2.0.2 report
foobar2000 v1.3.10
2016-09-02 13:49:32
File A: orig.flac
SHA1: 9aa7852a5a35d6c8416f430dd02891c73acddfcb
File B: insane.mpc
SHA1: 94b1a5fa426fe829defc29c8fa19a1e33c698aad
Output:
DS : Podstawowy sterownik dźwięku
Crossfading: NO
13:49:32 : Test started.
13:49:55 : 01/01
13:50:06 : 02/02
13:50:17 : 02/03
13:50:29 : 03/04
13:50:46 : 03/05
13:50:56 : 04/06
13:51:13 : 05/07
13:51:23 : 06/08
13:52:39 : 07/09
13:52:51 : 07/10
13:53:15 : 07/11
13:53:23 : 08/12
13:53:50 : 09/13
13:54:03 : 10/14
13:54:53 : 11/15
13:55:14 : 12/16
13:55:14 : Test finished.
----------
Total: 12/16
Probability that you were guessing: 3.8%
-- signature --
3e6b9b86b3f02a5b47122279de1bb81b8140f483
ConclusionThere's no denying that Opus is the state of the art right now, it sounds amazing for the bitrate and even when it does alter the sound the cut is clean and undistracting. Vorbis, on the other hand, turns the high end into a mush of white noise until it has enough bits to encode it properly. Musepack sits nicely between the two, altering the sound but in a way that doesn't stand out. Opus really impressed me in this test and I'll be using it from now on at 256kbps.
30s sample file is attached. The full song was used in the test, so bitrates might be slightly different. The (in my opinion) killer sound is the offbeat buzz that kicks in 10s into this preview.
Thanks for sharing those results. I'm looking forward to do a listening test myself, especially to try Vorbis and Opus. Actually never tested those formats before, always used LAME MP3 until now.
Did you consider also comparing LAME against the other ones? Or are you one of those people who can still hear a difference between LAME and FLAC at -V0?
[...] Opus [...] sounds amazing for the bitrate and even when it does alter the sound the cut is clean and undistracting.
In what way does it alter the sound to your ears?
The problem is again 256k ~ 320k is much larger than 176k. Some codecs performed well @176 like opus and mpc have potential for efficiency /quality tradeoff. Second issue is that @ 256k~320 its possible that even mp3 may be competitive.
I would love to see where qaac stands among those with this sample.
Thanks for sharing those results. I'm looking forward to do a listening test myself, especially to try Vorbis and Opus. Actually never tested those formats before, always used LAME MP3 until now.
Did you consider also comparing LAME against the other ones? Or are you one of those people who can still hear a difference between LAME and FLAC at -V0?
I'm not interested in patented codecs, I've heard stories of patent holders demanding a cut of mp3 sales from artists, it's like a paper producer asking for a cut of book sales. And free ones work well enough.
Honestly, the question of "can you hear a difference between A and B" by itself isn't any more meaningful than "what is the fps of human eye", it all depends on the content. I'm sure there's some killer samples that trip up mp3 320kbps vbr but for most songs 128kbps opus or even 96 is transparent to me. I picked one that's very difficult for codecs to handle on purpose, so that if this is transparent then everything else in my library should be. Ideally one would ABX every song to find the lowest setting that works great, or have an encoder with a reliable "quality" setting that maintains constant fidelity with no regard for bitrate (which to my knowledge doesn't exist yet).
I'm not interested in patented codecs, I've heard stories of patent holders demanding a cut of mp3 sales from artists
https://en.wikipedia.org/wiki/MP3#Licensing.2C_ownership_and_legislation
it's like a paper producer asking for a cut of book sales.
No, it's not like that at all.
And free ones work well enough.
Today, LAME is considered the best MP3 encoder at mid-high bitrates and at VBR, mostly thanks to the dedicated work of its developers and the open source licensing model that allowed the project to tap into engineering resources from all around the world. Both quality and speed improvements are still happening, probably making LAME the only MP3 encoder still being actively developed.
If you mean free, as in beer, then just about everything can be done without the end user having to pay a cent. While in some places there may be some gray spots to the legality, the same goes for getting legally obtained content on to your hard drive in order to encode it.
Hi Tear♪, can you tell us how old are you? I am interested to know about the relationship between hearing ability and age. You seem to be quite sensitive about the differences of different codecs.
Isn't it aoTuVb6.03 the latest build? I tried for myself the test sample with aoTuVb6.03 (libVorbis 1.3.5) at Q7 and Q5, and I couldn't find any difference.
Thanks for sharing those results. I'm looking forward to do a listening test myself, especially to try Vorbis and Opus. Actually never tested those formats before, always used LAME MP3 until now.
Did you consider also comparing LAME against the other ones? Or are you one of those people who can still hear a difference between LAME and FLAC at -V0?
I'm not interested in patented codecs, I've heard stories of patent holders demanding a cut of mp3 sales from artists, it's like a paper producer asking for a cut of book sales. And free ones work well enough.
Honestly, the question of "can you hear a difference between A and B" by itself isn't any more meaningful than "what is the fps of human eye", it all depends on the content. I'm sure there's some killer samples that trip up mp3 320kbps vbr but for most songs 128kbps opus or even 96 is transparent to me. I picked one that's very difficult for codecs to handle on purpose, so that if this is transparent then everything else in my library should be. Ideally one would ABX every song to find the lowest setting that works great, or have an encoder with a reliable "quality" setting that maintains constant fidelity with no regard for bitrate (which to my knowledge doesn't exist yet).
Outside of the (dead) 5.1 multichannel mp3 extension, I think all mp3 essential patents have expired, at least in the US and EU.
Looks like patents of MP3 decoding have already expired.
As for encoder MP3 patents they expire in April 2017.
https://hydrogenaud.io/index.php/topic,106938.msg895903.html#msg895903
Also some interesting formats will be patent free soon (if are not already). G.729 patents expire in 2016 and AC-3 in 2017.
Also using only 1 sample maybe deceptive - you should not draw a conclusion based solely on that.
If 128k is fine for most material then going up twice to 256k is very large step. It could also be that it isn't enough for undiscovered killer samples. One should think 150 .. 190k range would be the headroom setting. Opus is much less compatible than mp3 and other some lossy. IMO this is not an win situation. It may even be for many listeners mp3 176k - V3 or even 150k V4 are fine for vast majority of material. The other alternative is going lossless for limited collections . flac is a more supported open format.
Another option if you have the CD's or lossless versions at hand is to encode everything @ opus 160k . If you find a rare problem you can always re-encode that track at different setting. If the difference is slight and not annoying [as it should be] you can just ignore it. Many times for older mobile devices I used mp3 160k CBR [GOGO] with very reasonable results.
I tested this sample using mp3 128 and 160 abr gogo. Also lame 3.99 V5, V4, V3, V4 -f, V3 -f
@ 128 there are obvious issues and some can be ugly - pre echo and ringing. Gogo 160 still help up respectably. Lame vbr had issues in certain sports. V5 [160k] was not transparent obviously but I thought not bad until the 2.2 sec mark. Its is kind of ugly even with V4 / V4 -f [180k]. With V3 / V3 -f [194k], I thought the results are quite good and not annoying - the best of the lot.
I can see that with this type of electronic music there can be problems and its hard to be 100% pleased depending on the listener and the situation. Despite this I remain convinced that a good encode of 160..190k is the best tradeoff (IMO) .
Opus is not for music, it use resampling (48 Khz), right ?
so i guess Ogg is better....
but maybe i'm wrong
we have to test at 224 kps with the latest build of ogg aoTuVb6.03...
sorry for my ugly english ;)
Opus is not for music, it use resampling (48 Khz), right ?
so i guess Ogg is better....
but maybe i'm wrong
we have to test at 224 kps with the latest build of ogg aoTuVb6.03...
sorry for my ugly english ;)
48kHz is fine for music. Ogg is not a codec, it is a container. I'll assume that you meant Vorbis.
Opus has demonstrably better audio quality at bitrates up to perhaps 160 kbps. Above that, it becomes difficult to reliably distinguish them from eachother or from a lossless source. If you read this thread (go on, that's the whole point!) then you'll see that an experienced listener with great care (and a suitable sample that is difficult to encode) can distinguish Vorbis from a lossless sample even above 200 kbps, while Opus is indistinguishable at 196 kbps. If those small fractions are important to you then you should probably choose Opus, although your experience may differ and the encoded quality definitely varies with the type of music. Otherwise, check carefully how well Opus is supported on your hardware or software. Features like replaygain are still very patchy in Opus and even on relatively new hardware you will often find that Opus isn't supported.
Lastly, aoTuV beta 6.03 does not offer any quality improvements over 5.7, so no real problem there.
Actually the latest aoTuV b6.03 has more consistent quality than b5.7. It boosts bitrate more on critic parts to keep the quality.
http://d.hatena.ne.jp/kamedo2/20110409/1302373616
While the average scores are practically the same a quality is more constant amongst different samples.
P.S. It's prefferable to have a codec which produces the same quality for two samples (4.0 and 4.0) instead of (3.0 and 5.0, or 3.5 and 4.5 etc...).
More constant quality means better quality.
Tear, could you also do the test with FDK?
http://wiki.hydrogenaud.io/index.php?title=Fraunhofer_FDK_AAC
I wonder how results would differ if the sample would be normalized to -1 dB peak or lower, it makes a huge difference for encoding efficiency in my experience, at least at lower qualities.