New Public Multiformat Listening Test (Jan 2014) 2013-12-08 16:57:00 Hi, Guys It's time to discuss an upcoming listening test. It will be a multiformat test at 96 kbps as a logical continuation to the last public AAC listening test from 2011 year.There are few things we need to talk about:1.Amount of codecs. I think the possible amount of them is around 3-5. 2. Selection of codecs. Please, propose a codecs You want to test. AAC, MP3, Opus, Vorbis, ...We already had some discussion here http://www.hydrogenaudio.org/forums/index....c=92490&hl=But since it was 2 years ago it will be good to start from scratch. I will be updating this list "choice of codecs"Also I think it will be more interesting to compare MP3 128-135 kbps and AAC/AoTuV 96-100 kbps. Probably a lot of people are interesting to trade off between compability/compression efficiency. But it's my point of view. Let's discuss.Also this time Steve Forte Rio will be conducting the test. He has helped a lot to organize and conduct the last AAC public test.I just organize the discussion and help him. He will receive a results from a participants and keep a dialogue with them when the test will be open. Last Edit: 2013-12-08 17:09:17 by IgorC
New Public Multiformat Listening Test (Jan 2014) Reply #1 – 2013-12-08 17:31:52 I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.Also, AAC is a mature, strong codec, have good compatibility, and especially Apple AAC encoder is known for its very high quality. Will Opus beat Apple?And MP3 128kbps is also interesting. It has the best compatibility, and many people know what to expect from MP3 128kbps, so it's a good "anchor".And we need a low anchor as well. FAAC 96kbps cbr has a bad quality, good for a low anchor.
New Public Multiformat Listening Test (Jan 2014) Reply #2 – 2013-12-08 17:35:41 My choices would be:MP3 (Helix or LAME or Fraunhofer) - Which one is better at 96Kbps?AoTuV VorbisApple AACOpusAll of them running at 96KbpsEDIT: And FAAC 96Kbps low anchor Last Edit: 2013-12-08 17:37:06 by LithosZA
New Public Multiformat Listening Test (Jan 2014) Reply #3 – 2013-12-08 18:23:19 My choice:1. MP3@128 kbps (96 kbps will surely lead to complete defeat). We can use Fraunhofer IIS MP3 Surround encoder, which is sometimes better than LAME at low bitrates beginning from 128 kbps, but I consider LAME much more popular, so maybe we should use LAME 3.99.5 -b 128 -q0.2. OGG Vorbis aoTuV b6.03 3. QuickTime AAC TVBR.4. WMA Pro5. Opus 1.1, of course. Last Edit: 2013-12-08 18:26:24 by Steve Forte Rio
New Public Multiformat Listening Test (Jan 2014) Reply #4 – 2013-12-08 18:30:01 I think we should increase the number of samples. More samples leads to more statistically valid results.And I think we should choose the samples so that the average bitrate of the samples tested, and average bitrate of albums, is roughly equal, like I did;http://www.hydrogenaudio.org/forums/index....howtopic=100896If the average bitrate of albums is 96k and the average bitrate of tested samples is 144k, the corpus is overrepresented by critical samples.
New Public Multiformat Listening Test (Jan 2014) Reply #5 – 2013-12-08 18:33:54 Thank you for organizing this, my choices:AAC (Apple/qaac)AAC (Fraunhofer/fhgaacenc)AAC (Fraunhofer/fdkaac)Opus (1.1)Vorbis (libvorbis 1.3.3)Vorbis (aoTuV b6.03)WMA StandardWMA ProDon't care about MP3, don't care about MPC.edit: good observation from IgorC, let me the "optional" ones. I don't know much about WMA, just want to see how it performs even if nothing changed in the last few years (I'm not even really sure about this, Microsoft is a mess)? Is Stardard or Pro more compatible? Last Edit: 2013-12-08 18:46:17 by eahm
New Public Multiformat Listening Test (Jan 2014) Reply #6 – 2013-12-08 18:40:09 eahm,OK. But some obseravtions.We already know that Apple was a best AAC encoder. It represents all AAC format very well. No need to test again FhG encoder. The same for Vorbis. Only aoTuv. We can't test 10 codecs. . The optimal number is 3-5.Guys, correct me if I'm wrong. Last Edit: 2013-12-08 18:41:59 by IgorC
New Public Multiformat Listening Test (Jan 2014) Reply #7 – 2013-12-08 18:48:58 Quote from: Kamedo2 on 2013-12-08 17:31:52I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.Do You mean include Opus 1.0 and 1.1? Quote from: Kamedo2 on 2013-12-08 17:31:52Also, AAC is a mature, strong codec, have good compatibility, and especially Apple AAC encoder is known for its very high quality. Will Opus beat Apple?Good question. Let's see.Quote from: Kamedo2 on 2013-12-08 17:31:52And MP3 128kbps is also interesting. It has the best compatibility, and many people know what to expect from MP3 128kbps, so it's a good "anchor".And we need a low anchor as well. FAAC 96kbps cbr has a bad quality, good for a low anchor.Agree.
New Public Multiformat Listening Test (Jan 2014) Reply #8 – 2013-12-08 18:49:45 Opus, QT-AAC, Musepack and Vorbis (AoTuV). Musepack really deserves to be tested again vs. other modern codecs. Last Edit: 2013-12-08 18:54:33 by Kohlrabi
New Public Multiformat Listening Test (Jan 2014) Reply #9 – 2013-12-08 18:56:00 I would like to point to one comment of member Gecko. He has participated in previous tests:http://www.hydrogenaudio.org/forums/index....st&p=780195Quote+1 for keeping the number of codecs small. Consider only three perhaps? Four is stretching it. I really struggled with the five in the last test (AAC @ ~96 kbps [July 2011]).
New Public Multiformat Listening Test (Jan 2014) Reply #10 – 2013-12-08 18:57:35 Just to be difficult, I propose 80kbps instead of 96kbps (easier for testers), and Apple LC-AAC, Apple HE-AAC, FhG (libfdk) LC-AAC, FhG (libfdk) HE-AAC, Opus 1.1, Vorbis aoTuV.The AAC encoders could/should be a seperate pre-test, especially if FhG wants to send in a newer encoder than what is in libfdk. I'd favor libfdk over anything else AAC as it's used a lot together with ffmpeg now.Edit: This isn't 100% a serious suggestion, but I want people to think about some things. Last Edit: 2013-12-08 19:04:05 by Garf
New Public Multiformat Listening Test (Jan 2014) Reply #11 – 2013-12-08 18:59:12 Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).MPC and WMA is not interesting imo
New Public Multiformat Listening Test (Jan 2014) Reply #12 – 2013-12-08 19:00:47 Quote from: IgorC on 2013-12-08 18:48:58Quote from: Kamedo2 on 2013-12-08 17:31:52I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.Do You mean include Opus 1.0 and 1.1? The latest one, Opus 1.1. Testing Opus 1.0 is likely to lead to the redundant duplicate of http://listening-tests.hydrogenaudio.org/igorc/results.html and http://www.hydrogenaudio.org/forums/index....showtopic=97913
New Public Multiformat Listening Test (Jan 2014) Reply #13 – 2013-12-08 19:03:55 Quote from: Kohlrabi on 2013-12-08 18:49:45Musepack really deserves to be tested again vs. other modern codecs.Doesn't MPC really ony shine at settings that are intended to deliver transparent results which are like 3x what is being proposed for this test?
New Public Multiformat Listening Test (Jan 2014) Reply #14 – 2013-12-08 19:08:15 Quote from: Kamedo2 on 2013-12-08 19:00:47The latest one, Opus 1.1. Testing Opus 1.0 is likely to lead to the redundant duplicate of http://listening-tests.hydrogenaudio.org/igorc/results.html and http://www.hydrogenaudio.org/forums/index....showtopic=97913Agree, only Opus 1.1
New Public Multiformat Listening Test (Jan 2014) Reply #15 – 2013-12-08 19:13:11 Quote from: Steve Forte Rio on 2013-12-08 18:23:19My choice:1. MP3@128 kbps (96 kbps will surely lead to complete defeat). We can use Fraunhofer IIS MP3 Surround encoder, which is sometimes better than LAME at low bitrates beginning from 128 kbps, but I consider LAME much more popular, so maybe we should use LAME 3.99.5 -b 128 -q0.2. OGG Vorbis aoTuV b6.03 3. QuickTime AAC TVBR.4. WMA Pro5. Opus 1.1, of course.I think we should probably keep in mind both TVBR and CVBR. Because if TVBR will end up with ~94 kbps and other codecs at ~96-100 kbps then we probably should go to CVBR ~100 kbps. Anyway both Apple TVBR and CVBR are great.
New Public Multiformat Listening Test (Jan 2014) Reply #16 – 2013-12-08 19:18:09 Quote from: Garf on 2013-12-08 18:57:35Just to be difficult, I propose 80kbps instead of 96kbps (easier for testers), and Apple LC-AAC, Apple HE-AAC, FhG (libfdk) LC-AAC, FhG (libfdk) HE-AAC, Opus 1.1, Vorbis aoTuV.The AAC encoders could/should be a seperate pre-test, especially if FhG wants to send in a newer encoder than what is in libfdk. I'd favor libfdk over anything else AAC as it's used a lot together with ffmpeg now.Edit: This isn't 100% a serious suggestion, but I want people to think about some things.Why not? I agree that it will be interesting to see it. So let's see what people propose.However also let's see where we're coming from. We've tested AAC encoders at 96 kbps and it's logical to test the best AAC encooder, Apple, against the rest of codecs. Personally it's hard for me to do pre-test and then test, but, yeah, let's test in future or even now . It's all up to people decision. Last Edit: 2013-12-08 19:18:36 by IgorC
New Public Multiformat Listening Test (Jan 2014) Reply #17 – 2013-12-08 19:20:30 [quote author=o-l-a-v link=msg=852293 date=1386529152]Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).MPC and WMA is not interesting imo[/quote]MP3 at 96 or 128 kbps?
New Public Multiformat Listening Test (Jan 2014) Reply #18 – 2013-12-08 19:27:51 Quote from: IgorC on 2013-12-08 19:20:30[quote author=o-l-a-v link=msg=852293 date=1386529152]Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).MPC and WMA is not interesting imoMP3 at 96 or 128 kbps?[/quote]Let's say 128. If I were to convert music in low bitrate mp3, I would never even think about going lower than that.
New Public Multiformat Listening Test (Jan 2014) Reply #19 – 2013-12-08 19:36:57 I will participate in this test too so here is my wishlist.1. MP3 128 kbps. LAME 3.99.5 -V5 (high anchor)2. MP3 96 kbps . LAME ABR is better than VBR (?)3. Apple AAC 96 kbps (QAAC highest quality TVBR or CVBR.)4. Opus 1.1 vbr 96 kbps.5. Vorbis AoTuv 6.0.3 vbr 96 kbps.low anchor - FAAC CBR 96 kbps, as Kamedo2 said. It has a reasonably low quality.We had also discussion to have 2 low anchors. Actually low anchor and low-middle anchor. It's good to have two acnhors to validate results. Low-middle anchor should be better than low anchor. It can be: FAAC 64 (low anchor) and FAAC 96 (low-middle anchor). Last Edit: 2013-12-08 19:40:47 by IgorC
New Public Multiformat Listening Test (Jan 2014) Reply #20 – 2013-12-08 20:02:36 Quote from: Kamedo2 on 2013-12-08 18:30:01I think we should increase the number of samples. More samples leads to more statistically valid results.And I think we should choose the samples so that the average bitrate of the samples tested, and average bitrate of albums, is roughly equal, like I did;http://www.hydrogenaudio.org/forums/index....howtopic=100896If the average bitrate of albums is 96k and the average bitrate of tested samples is 144k, the corpus is overrepresented by critical samples.More than 20 samples? hm, maybe, I don't know.20 is already enough high number. During the last we've waited a little bit more than month to get enough results. What do other think about it?
New Public Multiformat Listening Test (Jan 2014) Reply #21 – 2013-12-08 21:10:29 Quote from: Steve Forte Rio on 2013-12-08 18:23:19... so maybe we should use LAME 3.99.5 -b 128 -q0.What about vbr. V5 ~128 kbps?
New Public Multiformat Listening Test (Jan 2014) Reply #22 – 2013-12-08 21:46:43 What I'd like to see in the test:LAME MP3 -V5Apple AAC @96Opus 1.1 @96Muespack @96
New Public Multiformat Listening Test (Jan 2014) Reply #23 – 2013-12-08 21:58:54 1) Musepack at 96kbps will have lowpass ~14kHz. That's too low IMHO.2) Bitrates of WMA VBR:WMA Std Q50: 74 kbps;WMA Std Q75: 115 kbps;WMA Pro Q25: 83 kbps;WMA Pro Q50: 113 kbps.None of them is close to the target bitrate.3) IMHO for LAME 3.99.x low-bitrate VBR is better than low-bitrate ABR.
New Public Multiformat Listening Test (Jan 2014) Reply #24 – 2013-12-08 22:41:06 For WMA you can also set the bit rate at 96.OTlvqcl, where did you get the bit rates for the WMA quality settings? Do you have all of them (Std and Pro)? Last Edit: 2013-12-08 22:42:25 by eahm