Hi, Guys 

It's time to discuss an upcoming listening test. It will be a multiformat test at 96 kbps as a logical continuation to the last public AAC listening test from 2011 year.


There are few things we need to talk about:

1.Amount of codecs.
I think the possible amount of them  is around 3-5. 

2. Selection of codecs.
Please, propose  a codecs You want to test.  AAC, MP3, Opus, Vorbis, ...
We already had some discussion here http://www.hydrogenaudio.org/forums/index....c=92490&hl=
But since it was 2 years ago it will be good to start from scratch. 
I will be updating this list "choice of codecs"


Also I think it will be more interesting to compare MP3 128-135 kbps and AAC/AoTuV 96-100 kbps. Probably a lot of people are interesting to trade off between compability/compression efficiency. But it's my point of view.

Let's discuss.

Also this time Steve Forte Rio will be conducting the test. He has helped a lot to organize and conduct the last  AAC public test.
I just organize the discussion and help him. He will receive a results from a participants and keep a dialogue with them when the test will be open.

I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.
Also, AAC is a mature, strong codec, have good compatibility, and especially Apple AAC encoder is known for its very high quality. Will Opus beat Apple?
And MP3 128kbps is also interesting. It has the best compatibility, and many people know what to expect from MP3 128kbps, so it's a good "anchor".

And we need a low anchor as well. FAAC 96kbps cbr has a bad quality, good for a low anchor.

My choices would be:
MP3 (Helix or LAME or Fraunhofer) - Which one is better at 96Kbps?
AoTuV Vorbis
Apple AAC
Opus

All of them running at 96Kbps
EDIT: And FAAC 96Kbps low anchor

My choice:

1. MP3@128 kbps (96 kbps will surely lead to complete defeat). We can use  Fraunhofer IIS MP3 Surround encoder, which is sometimes better than LAME at low bitrates beginning from 128 kbps, but I consider LAME much more popular, so maybe we should use LAME 3.99.5 -b 128 -q0.

2. OGG Vorbis aoTuV b6.03

3. QuickTime AAC TVBR.

4. WMA Pro

5. Opus 1.1, of course.

I think we should increase the number of samples. More samples leads to more statistically valid results.
And I think we should choose the samples so that the average bitrate of the samples tested, and average bitrate of albums, is roughly equal, like I did;
http://www.hydrogenaudio.org/forums/index....howtopic=100896
If the average bitrate of albums is 96k and the average bitrate of tested samples is 144k, the corpus is overrepresented by critical samples.

Thank you for organizing this, my choices:

AAC (Apple/qaac)

AAC (Fraunhofer/fhgaacenc)

AAC (Fraunhofer/fdkaac)

Opus (1.1)

Vorbis (libvorbis 1.3.3)

Vorbis (aoTuV b6.03)

WMA Standard

WMA Pro


Don't care about MP3, don't care about MPC.

edit: good observation from IgorC, let me the "optional" ones. I don't know much about WMA, just want to see how it performs even if nothing changed in the last few years (I'm not even really sure about this, Microsoft is a mess)? Is Stardard or Pro more compatible?

eahm,

OK. But some obseravtions.

We already know that Apple was a best AAC encoder. It represents all AAC format very well. No need to test again FhG encoder.
The same for Vorbis. Only aoTuv. 

We can't test 10 codecs.  . The optimal number is 3-5.

Guys, correct me if I'm wrong.

I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.

Do You mean include  Opus 1.0 and 1.1?

Also, AAC is a mature, strong codec, have good compatibility, and especially Apple AAC encoder is known for its very high quality. Will Opus beat Apple?

Good question. Let's see.

And MP3 128kbps is also interesting. It has the best compatibility, and many people know what to expect from MP3 128kbps, so it's a good "anchor".

And we need a low anchor as well. FAAC 96kbps cbr has a bad quality, good for a low anchor.

Agree.

Opus, QT-AAC, Musepack and Vorbis (AoTuV). Musepack really deserves to be tested again vs. other modern codecs.

I would like to point to one comment of member Gecko. He has participated in previous tests:

http://www.hydrogenaudio.org/forums/index....st&p=780195

+1 for keeping the number of codecs small. Consider only three perhaps? Four is stretching it. I really struggled with the five in the last test (AAC @ ~96 kbps [July 2011]).

Just to be difficult, I propose 80kbps instead of 96kbps (easier for testers), and Apple LC-AAC, Apple HE-AAC, FhG (libfdk) LC-AAC, FhG (libfdk) HE-AAC, Opus 1.1, Vorbis aoTuV.

The AAC encoders could/should be a seperate pre-test, especially if FhG wants to send in a newer encoder than what is in libfdk. I'd favor libfdk over anything else AAC as it's used a lot together with ffmpeg now.

Edit: This isn't 100% a serious suggestion, but I want people to think about some things.

Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).
MPC and WMA is not interesting imo

I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.

Do You mean include  Opus 1.0 and 1.1?

The latest one, Opus 1.1. Testing Opus 1.0 is likely to lead to the redundant duplicate of http://listening-tests.hydrogenaudio.org/igorc/results.html and http://www.hydrogenaudio.org/forums/index....showtopic=97913

Musepack really deserves to be tested again vs. other modern codecs.

Doesn't MPC really ony shine at settings that are intended to deliver transparent results which are like 3x what is being proposed for this test?

The latest one, Opus 1.1. Testing Opus 1.0 is likely to lead to the redundant duplicate of http://listening-tests.hydrogenaudio.org/igorc/results.html and http://www.hydrogenaudio.org/forums/index....showtopic=97913

Agree, only Opus 1.1

My choice:

1. MP3@128 kbps (96 kbps will surely lead to complete defeat). We can use  Fraunhofer IIS MP3 Surround encoder, which is sometimes better than LAME at low bitrates beginning from 128 kbps, but I consider LAME much more popular, so maybe we should use LAME 3.99.5 -b 128 -q0.

2. OGG Vorbis aoTuV b6.03

3. QuickTime AAC TVBR.

4. WMA Pro

5. Opus 1.1, of course.

I think we should probably keep in mind both TVBR and CVBR. Because if TVBR will end up with ~94 kbps and other codecs at ~96-100 kbps then we  probably should go to CVBR ~100 kbps. Anyway both Apple  TVBR and CVBR are great.

Just to be difficult, I propose 80kbps instead of 96kbps (easier for testers), and Apple LC-AAC, Apple HE-AAC, FhG (libfdk) LC-AAC, FhG (libfdk) HE-AAC, Opus 1.1, Vorbis aoTuV.

The AAC encoders could/should be a seperate pre-test, especially if FhG wants to send in a newer encoder than what is in libfdk. I'd favor libfdk over anything else AAC as it's used a lot together with ffmpeg now.

Edit: This isn't 100% a serious suggestion, but I want people to think about some things.

Why not?

I agree that it will be interesting to see it. So let's see what people propose.

However also let's see where we're coming from. We've tested AAC encoders at 96 kbps and it's logical to test the best AAC encooder, Apple, against the rest of codecs.

Personally it's hard for me to do pre-test and then test, but, yeah, let's test in future or even now . It's all up to people decision.

Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).
MPC and WMA is not interesting imo

MP3 at 96 or 128 kbps?

Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).
MPC and WMA is not interesting imo

MP3 at 96 or 128 kbps?

Let's say 128. If I were to convert music in low bitrate mp3, I would never even think about going lower than that.

I will participate in this test too so here is my wishlist.

1. MP3 128 kbps. LAME 3.99.5 -V5 (high anchor)
2. MP3 96 kbps . LAME ABR is better than VBR (?)
3. Apple AAC 96 kbps (QAAC highest quality TVBR or CVBR.)
4. Opus 1.1 vbr 96 kbps.
5. Vorbis AoTuv 6.0.3 vbr 96 kbps.

low anchor - FAAC CBR 96 kbps, as Kamedo2 said. It has a reasonably low quality.
We had also discussion to have 2 low anchors. Actually low anchor and low-middle anchor.  It's good to have two acnhors to validate results. Low-middle anchor should be better than low anchor.
It can be: FAAC 64 (low anchor) and FAAC 96 (low-middle anchor).

I think we should increase the number of samples. More samples leads to more statistically valid results.
And I think we should choose the samples so that the average bitrate of the samples tested, and average bitrate of albums, is roughly equal, like I did;
http://www.hydrogenaudio.org/forums/index....howtopic=100896
If the average bitrate of albums is 96k and the average bitrate of tested samples is 144k, the corpus is overrepresented by critical samples.

More than 20 samples? hm, maybe, I don't know.
20 is already enough high number. During the last we've waited a little bit more than month to get enough results. 


What do other think about it?

... so maybe we should use LAME 3.99.5 -b 128 -q0.

What about vbr. V5 ~128  kbps?

What I'd like to see in the test:

• LAME MP3 -V5
• Apple AAC @96
• Opus 1.1 @96
• Muespack @96

1) Musepack at 96kbps will have lowpass ~14kHz. That's too low IMHO.

2) Bitrates of WMA VBR:
WMA Std Q50: 74 kbps;
WMA Std Q75: 115 kbps;
WMA Pro Q25: 83 kbps;
WMA Pro Q50: 113 kbps.

None of them is close to the target bitrate.

3) IMHO for LAME 3.99.x low-bitrate VBR is better than low-bitrate ABR.

For WMA you can also set the bit rate at 96.

OT
lvqcl, where did you get the bit rates for the WMA quality settings? Do you have all of them (Std and Pro)?