Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: New Public Multiformat Listening Test (Jan 2014) (Read 163452 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

New Public Multiformat Listening Test (Jan 2014)

Hi, Guys 

It's time to discuss an upcoming listening test. It will be a multiformat test at 96 kbps as a logical continuation to the last public AAC listening test from 2011 year.


There are few things we need to talk about:

1.Amount of codecs.
I think the possible amount of them  is around 3-5. 

2. Selection of codecs.
Please, propose  a codecs You want to test.  AAC, MP3, Opus, Vorbis, ...
We already had some discussion here http://www.hydrogenaudio.org/forums/index....c=92490&hl=
But since it was 2 years ago it will be good to start from scratch. 
I will be updating this list "choice of codecs"


Also I think it will be more interesting to compare MP3 128-135 kbps and AAC/AoTuV 96-100 kbps. Probably a lot of people are interesting to trade off between compability/compression efficiency. But it's my point of view.

Let's discuss.

Also this time Steve Forte Rio will be conducting the test. He has helped a lot to organize and conduct the last  AAC public test.
I just organize the discussion and help him. He will receive a results from a participants and keep a dialogue with them when the test will be open.

New Public Multiformat Listening Test (Jan 2014)

Reply #1 –
I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.
Also, AAC is a mature, strong codec, have good compatibility, and especially Apple AAC encoder is known for its very high quality. Will Opus beat Apple?
And MP3 128kbps is also interesting. It has the best compatibility, and many people know what to expect from MP3 128kbps, so it's a good "anchor".

And we need a low anchor as well. FAAC 96kbps cbr has a bad quality, good for a low anchor.

New Public Multiformat Listening Test (Jan 2014)

Reply #2 –
My choices would be:
MP3 (Helix or LAME or Fraunhofer) - Which one is better at 96Kbps?
AoTuV Vorbis
Apple AAC
Opus

All of them running at 96Kbps
EDIT: And FAAC 96Kbps low anchor

New Public Multiformat Listening Test (Jan 2014)

Reply #3 –
My choice:

1. MP3@128 kbps (96 kbps will surely lead to complete defeat). We can use  Fraunhofer IIS MP3 Surround encoder, which is sometimes better than LAME at low bitrates beginning from 128 kbps, but I consider LAME much more popular, so maybe we should use LAME 3.99.5 -b 128 -q0.

2. OGG Vorbis aoTuV b6.03

3. QuickTime AAC TVBR.

4. WMA Pro

5. Opus 1.1, of course.
🇺🇦 Glory to Ukraine!

New Public Multiformat Listening Test (Jan 2014)

Reply #4 –
I think we should increase the number of samples. More samples leads to more statistically valid results.
And I think we should choose the samples so that the average bitrate of the samples tested, and average bitrate of albums, is roughly equal, like I did;
http://www.hydrogenaudio.org/forums/index....howtopic=100896
If the average bitrate of albums is 96k and the average bitrate of tested samples is 144k, the corpus is overrepresented by critical samples.

New Public Multiformat Listening Test (Jan 2014)

Reply #5 –
Thank you for organizing this, my choices:

AAC (Apple/qaac)

AAC (Fraunhofer/fhgaacenc)

AAC (Fraunhofer/fdkaac)

Opus (1.1)

Vorbis (libvorbis 1.3.3)

Vorbis (aoTuV b6.03)

WMA Standard

WMA Pro


Don't care about MP3, don't care about MPC.

edit: good observation from IgorC, let me the "optional" ones. I don't know much about WMA, just want to see how it performs even if nothing changed in the last few years (I'm not even really sure about this, Microsoft is a mess)? Is Stardard or Pro more compatible?

New Public Multiformat Listening Test (Jan 2014)

Reply #6 –
eahm,

OK. But some obseravtions.

We already know that Apple was a best AAC encoder. It represents all AAC format very well. No need to test again FhG encoder.
The same for Vorbis. Only aoTuv. 

We can't test 10 codecs.  . The optimal number is 3-5.

Guys, correct me if I'm wrong.

New Public Multiformat Listening Test (Jan 2014)

Reply #7 –
Quote from: Kamedo2 on
I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.

Do You mean include  Opus 1.0 and 1.1?

Quote from: Kamedo2 on
Also, AAC is a mature, strong codec, have good compatibility, and especially Apple AAC encoder is known for its very high quality. Will Opus beat Apple?

Good question. Let's see.

Quote from: Kamedo2 on
And MP3 128kbps is also interesting. It has the best compatibility, and many people know what to expect from MP3 128kbps, so it's a good "anchor".

And we need a low anchor as well. FAAC 96kbps cbr has a bad quality, good for a low anchor.

Agree.



New Public Multiformat Listening Test (Jan 2014)

Reply #8 –
Opus, QT-AAC, Musepack and Vorbis (AoTuV). Musepack really deserves to be tested again vs. other modern codecs.
It's only audiophile if it's inconvenient.

New Public Multiformat Listening Test (Jan 2014)

Reply #9 –
I would like to point to one comment of member Gecko. He has participated in previous tests:

http://www.hydrogenaudio.org/forums/index....st&p=780195
Quote
+1 for keeping the number of codecs small. Consider only three perhaps? Four is stretching it. I really struggled with the five in the last test (AAC @ ~96 kbps [July 2011]).

New Public Multiformat Listening Test (Jan 2014)

Reply #10 –
Just to be difficult, I propose 80kbps instead of 96kbps (easier for testers), and Apple LC-AAC, Apple HE-AAC, FhG (libfdk) LC-AAC, FhG (libfdk) HE-AAC, Opus 1.1, Vorbis aoTuV.

The AAC encoders could/should be a seperate pre-test, especially if FhG wants to send in a newer encoder than what is in libfdk. I'd favor libfdk over anything else AAC as it's used a lot together with ffmpeg now.

Edit: This isn't 100% a serious suggestion, but I want people to think about some things.

New Public Multiformat Listening Test (Jan 2014)

Reply #11 –
Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).
MPC and WMA is not interesting imo

New Public Multiformat Listening Test (Jan 2014)

Reply #12 –
Quote from: IgorC on
Quote from: Kamedo2 on
I'd like to see the results of Opus. The development is very active, and I'm very interested to see the progress since two years ago, especially at high bitrates.

Do You mean include  Opus 1.0 and 1.1?

The latest one, Opus 1.1. Testing Opus 1.0 is likely to lead to the redundant duplicate of http://listening-tests.hydrogenaudio.org/igorc/results.html and http://www.hydrogenaudio.org/forums/index....showtopic=97913

New Public Multiformat Listening Test (Jan 2014)

Reply #13 –
Quote from: Kohlrabi on
Musepack really deserves to be tested again vs. other modern codecs.

Doesn't MPC really ony shine at settings that are intended to deliver transparent results which are like 3x what is being proposed for this test?

New Public Multiformat Listening Test (Jan 2014)

Reply #15 –
Quote from: Steve Forte Rio on
My choice:

1. MP3@128 kbps (96 kbps will surely lead to complete defeat). We can use  Fraunhofer IIS MP3 Surround encoder, which is sometimes better than LAME at low bitrates beginning from 128 kbps, but I consider LAME much more popular, so maybe we should use LAME 3.99.5 -b 128 -q0.

2. OGG Vorbis aoTuV b6.03

3. QuickTime AAC TVBR.

4. WMA Pro

5. Opus 1.1, of course.

I think we should probably keep in mind both TVBR and CVBR. Because if TVBR will end up with ~94 kbps and other codecs at ~96-100 kbps then we  probably should go to CVBR ~100 kbps. Anyway both Apple  TVBR and CVBR are great.

New Public Multiformat Listening Test (Jan 2014)

Reply #16 –
Quote from: Garf on
Just to be difficult, I propose 80kbps instead of 96kbps (easier for testers), and Apple LC-AAC, Apple HE-AAC, FhG (libfdk) LC-AAC, FhG (libfdk) HE-AAC, Opus 1.1, Vorbis aoTuV.

The AAC encoders could/should be a seperate pre-test, especially if FhG wants to send in a newer encoder than what is in libfdk. I'd favor libfdk over anything else AAC as it's used a lot together with ffmpeg now.

Edit: This isn't 100% a serious suggestion, but I want people to think about some things.

Why not?

I agree that it will be interesting to see it. So let's see what people propose.

However also let's see where we're coming from. We've tested AAC encoders at 96 kbps and it's logical to test the best AAC encooder, Apple, against the rest of codecs.

Personally it's hard for me to do pre-test and then test, but, yeah, let's test in future or even now . It's all up to people decision.

New Public Multiformat Listening Test (Jan 2014)

Reply #17 –
Quote from: o-l-a-v on
Opus, MP3(LAME), AAC(QAAC) and Vorbis(aoTuv).
MPC and WMA is not interesting imo


MP3 at 96 or 128 kbps?

New Public Multiformat Listening Test (Jan 2014)

Reply #19 –
I will participate in this test too so here is my wishlist.

1. MP3 128 kbps. LAME 3.99.5 -V5 (high anchor)
2. MP3 96 kbps . LAME ABR is better than VBR (?)
3. Apple AAC 96 kbps (QAAC highest quality TVBR or CVBR.)
4. Opus 1.1 vbr 96 kbps.
5. Vorbis AoTuv 6.0.3 vbr 96 kbps.

low anchor - FAAC CBR 96 kbps, as Kamedo2 said. It has a reasonably low quality.
We had also discussion to have 2 low anchors. Actually low anchor and low-middle anchor.  It's good to have two acnhors to validate results. Low-middle anchor should be better than low anchor.
It can be: FAAC 64 (low anchor) and FAAC 96 (low-middle anchor).

New Public Multiformat Listening Test (Jan 2014)

Reply #20 –
Quote from: Kamedo2 on
I think we should increase the number of samples. More samples leads to more statistically valid results.
And I think we should choose the samples so that the average bitrate of the samples tested, and average bitrate of albums, is roughly equal, like I did;
http://www.hydrogenaudio.org/forums/index....howtopic=100896
If the average bitrate of albums is 96k and the average bitrate of tested samples is 144k, the corpus is overrepresented by critical samples.

More than 20 samples? hm, maybe, I don't know.
20 is already enough high number. During the last we've waited a little bit more than month to get enough results. 


What do other think about it?

New Public Multiformat Listening Test (Jan 2014)

Reply #22 –
What I'd like to see in the test:
  • LAME MP3 -V5
  • Apple AAC @96
  • Opus 1.1 @96
  • Muespack @96

New Public Multiformat Listening Test (Jan 2014)

Reply #23 –
1) Musepack at 96kbps will have lowpass ~14kHz. That's too low IMHO.

2) Bitrates of WMA VBR:
WMA Std Q50: 74 kbps;
WMA Std Q75: 115 kbps;
WMA Pro Q25: 83 kbps;
WMA Pro Q50: 113 kbps.

None of them is close to the target bitrate.

3) IMHO for LAME 3.99.x low-bitrate VBR is better than low-bitrate ABR.

New Public Multiformat Listening Test (Jan 2014)

Reply #24 –
For WMA you can also set the bit rate at 96.

OT
lvqcl, where did you get the bit rates for the WMA quality settings? Do you have all of them (Std and Pro)?