MP3/AAC/Vorbis/Opus vs lossless ABX test (one song)

2016-07-10 02:14:50

Encoders used: LAME 3.99.5, QAAC 2.59, oggenc v2.87, opusenc 2016-05-22
Headphones: One EHP-303E in-ear headphones (costed 1500BsF/$1.5)
Equalizer: Equalizer APO (software), screenshot (imgur)
Song: USAO - Dynamite

MP3 V2 (~192kbps) vs FLAC
Spoiler (click to show/hide)

Code: [Select]

foo_abx 2.0.2 report
foobar2000 v1.3.10
2016-07-09 16:12:37

File A: 29.Dynamite.flac
SHA1: 28121ff891814d3eac827bce2cd1aedb42324d96
File B: 29.Dynamitev2.mp3
SHA1: b47d1aec6144b41b9f5dde5d1ac57fe2aa69b811

Output:
DS : Altavoces (Realtek High Definition Audio)
Crossfading: NO

16:12:37 : Test started.
16:12:47 : 01/01
16:12:50 : 01/02
16:12:53 : 02/03
16:12:56 : 03/04
16:12:59 : 04/05
16:13:01 : 05/06
16:13:04 : 06/07
16:13:07 : 07/08
16:13:10 : 08/09
16:13:13 : 09/10
16:13:16 : 09/11
16:13:19 : 10/12
16:13:22 : 11/13
16:13:25 : 12/14
16:13:28 : 13/15
16:13:31 : 14/16
16:13:31 : Test finished.

 ---------- 
Total: 14/16
Probability that you were guessing: 0.2%

 -- signature -- 
502cc0d0e77f398c4835a3626e8500b2cc7cb324

Note: Start time is 1:07.5, the sound effect on the right channel around 1:08.5 is a bit muffled on the MP3. It sounds a bit more "centered" on the MP3. I only listened to the "X" sample.

AAC VBR Q82 (~160kbps) vs FLAC
Spoiler (click to show/hide)

Code: [Select]

foo_abx 2.0.2 report
foobar2000 v1.3.10
2016-07-09 19:01:27

File A: 29.Dynamite.flac
SHA1: 28121ff891814d3eac827bce2cd1aedb42324d96
File B: 29.DynamiteAACQ82.m4a
SHA1: 0d0fe0b0e3ec190d00af66563a19603b0a0ee3cb

Output:
DS : Altavoces (Realtek High Definition Audio)
Crossfading: NO

19:01:27 : Test started.
19:01:46 : 00/01
19:01:49 : 01/02
19:01:54 : 02/03
19:01:57 : 03/04
19:02:00 : 04/05
19:02:04 : 05/06
19:02:07 : 06/07
19:02:11 : 07/08
19:02:14 : 08/09
19:02:17 : 09/10
19:02:21 : 10/11
19:02:24 : 11/12
19:02:32 : 12/13
19:02:35 : 13/14
19:02:39 : 14/15
19:02:43 : 15/16
19:02:43 : Test finished.

 ---------- 
Total: 15/16
Probability that you were guessing: 0.0%

 -- signature -- 
ed26f778b42d58bb07f55394350a61b234dbb64c

Note: Start time is 1:05.0, there's a sound effect around 1:06.2 that has a slightly lower volume in the AAC file.

Vorbis Q5 (~160kbps) vs FLAC
Spoiler (click to show/hide)

Code: [Select]

foo_abx 2.0.2 report
foobar2000 v1.3.10
2016-07-09 20:23:22

File A: 29.Dynamite.flac
SHA1: 28121ff891814d3eac827bce2cd1aedb42324d96
File B: 29.DynamiteQ5-160.ogg
SHA1: 70d6b35d929d7e90764ab33096e25a16de0b6be8

Output:
DS : Altavoces (Realtek High Definition Audio)
Crossfading: NO

20:23:22 : Test started.
20:23:41 : 01/01
20:23:44 : 02/02
20:23:50 : 02/03
20:23:54 : 02/04
20:23:57 : 02/05
20:24:00 : 03/06
20:24:03 : 04/07
20:24:07 : 05/08
20:24:10 : 06/09
20:24:15 : 07/10
20:24:18 : 08/11
20:24:21 : 09/12
20:24:24 : 10/13
20:24:43 : 11/14
20:24:46 : 12/15
20:24:49 : 13/16
20:24:49 : Test finished.

 ---------- 
Total: 13/16
Probability that you were guessing: 1.1%

 -- signature -- 
77bdfaede0b3028c2eaabf9d1d13c018edd75e76

Note: Start time is 1:05.0, sounds a bit different.

Opus 144kbps VBR vs FLAC
Spoiler (click to show/hide)

Code: [Select]

foo_abx 2.0.2 report
foobar2000 v1.3.10
2016-07-09 19:46:33

File A: 29.Dynamite.flac
SHA1: 28121ff891814d3eac827bce2cd1aedb42324d96
File B: 29.Dynamite144.opus
SHA1: c874f4ef75791714fae3c5ed3a3c13628997de22

Output:
DS : Altavoces (Realtek High Definition Audio)
Crossfading: NO

19:46:33 : Test started.
19:46:42 : 01/01
19:46:47 : 01/02
19:46:51 : 02/03
19:46:54 : 03/04
19:46:57 : 04/05
19:47:00 : 05/06
19:47:04 : 05/07
19:47:08 : 06/08
19:47:11 : 06/09
19:47:15 : 07/10
19:47:18 : 08/11
19:47:21 : 09/12
19:47:24 : 10/13
19:47:29 : 11/14
19:47:33 : 12/15
19:47:36 : 13/16
19:47:36 : Test finished.

 ---------- 
Total: 13/16
Probability that you were guessing: 1.1%

 -- signature -- 
a2ba542fb6d71c834789b2048b465edaa39ad063

Note: Start time is 1:05.0, sounds a bit different. The difference is more noticeable if I start on X then click on A. I can't pass an ABX test between Opus 144kbps and Vorbis Q5 for some reason.

MP3 V1 (~224kbps) vs FLAC: failed
AAC Q91 (~192kbps) vs FLAC: failed
Vorbis Q6 (~192kbps) vs FLAC: failed
Opus 160kbps VBR vs FLAC: failed

1:00 in the full song is 0:00 in the attached file.

Re: MP3/AAC/Vorbis/Opus vs lossless ABX test (one song)

Reply #1 – 2016-07-10 10:12:00

That's some very significant EQ, was it done to specifically target problem frequencies, or are those the general settings you use for listening?

Re: MP3/AAC/Vorbis/Opus vs lossless ABX test (one song)

Reply #2 – 2016-07-10 17:07:08

Quote from: KozmoNaut on 2016-07-10 10:12:00

That's some very significant EQ, was it done to specifically target problem frequencies, or are those the general settings you use for listening?

Those are the settings that I use for listening. Without the EQ, I can notice the difference between AAC Q64 (~128kbps) and FLAC but not Opus 96kbps and FLAC.

Re: MP3/AAC/Vorbis/Opus vs lossless ABX test (one song)

Reply #3 – 2016-07-10 17:18:15

Very interesting :-)

Did you try LAME -V0 as well, just for curiosity?

Re: MP3/AAC/Vorbis/Opus vs lossless ABX test (one song)

Reply #4 – 2016-07-10 17:36:51

Quote from: KozmoNaut on 2016-07-10 17:18:15

Very interesting :-)

Did you try LAME -V0 as well, just for curiosity?

Yes. My final pval value was over 30%, same with the other formats that I marked as "failed"

Re: MP3/AAC/Vorbis/Opus vs lossless ABX test (one song)

Reply #5 – 2016-07-10 17:51:41

Ah OK, I didn't get that initially :-)

Re: MP3/AAC/Vorbis/Opus vs lossless ABX test (one song)

Reply #6 – 2016-08-16 15:36:27

Today I managed to ABX a 128 kbps vrb qaac, and I was going to post my results, but another venezuelan have already done it. jajaja ¿De qué ciudad eres? outstanding results, I actually tried to replicate your results twice, both ended up being 7/10.

The track you provided is a lot harder for me.

If you're interested, I can link you the track which I'm able to ABX, to see if you can ABX it at even higher bitrates since I can't replicate your results.

Notice