Skip to main content

Topic: Nero 48 kbit/s HE-AAC (+v2) Test - STARTED! (Read 16870 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Dear All,

I would like to announce that the 48 kbps test of upcoming Nero AAC encoder has just started.  The purpose of this test is to determine which bit allocation mode gives highest quality, and how big the quality difference is between ABR, CBR and VBR.

With this test, I wish to include true audiophile and scientific community into making the HE-AAC /v2 solutions even better in the future.  Many thanks in advance for your help!

Test runs for 10 days, Until february 3rd 2006.  It should be much easier and more interesting to the wide audience than the last 128 kbps test, since I hope many more people would be able to actually judge the quality

So, What's in?

There are two tests actually - Test 1 and Test 2.

- Test 1 contains HE-AAC v1 tracks (AAC+SBR)
- Test 2 contains HE-AAC v2 tracks (AAC+SBR+Parametric Stereo)

The reason why they are split is that rating HE-AAC vs HE-AAC v2 is not the point of this test (it is the point of the next public 48 kbps multivendor AAC test coming in February)

Which codecs are tested?

For both tests, following codecs are being tested:

- HE-AAC (or v2) CBR - Constant Bit Rate, optimized for online streaming
- HE-AAC (or v2) ABR - Average Bit Rate, little bit more pre-buffer delay, more room for "hard to encode" frames
- HE-AAC (or v2) VBR v1 - Quality iterated VBR 1
- HE-AAC (or v2) VBR v2 - Quality iterated VBR 2
- MP3 at 128 kbps (LAME -b128)  *** High Anchor ***
- MP3 at 48 kbps (LAME -b48)  *** Low Anchor ***

Difference between quality iterated VBR modes is in the bit allocation strategy. All encoding modes (except high anchor) should produce clips of average bit rate close to the 48 kbps.

How To Test

1. Download test1.rar and test2.rar  (links in the next message)

2. Extract these two RAR files in separate directories (for example, C:\TEST1 and C:\TEST2)

3. Download samples for the test, and extract them in the directory of the test (so, test1_atrain.rar should go to C:\TEST1 and test2_atrain.rar should go to C:\TEST2)

4. When you download all the samples, in each test directory (C:\TEST1 or C:\TEST2 in example) execute the decode_all.bat files.  This will decode all the files and prepare the test.  Note that you have to do it for both tests separately.

5. Alternatively, you can download and decode each sample, and execute individual bat files to decode just that sample (e.g. for test1_atrain.rar, unpacked in C:\TEST1, you have to execute C:\TEST1\atrain.bat)

6. Now everything is ready for the test, and the file structure should look like:

Code: [Select]
C:\TEST1\
C:\TEST1\ATRAIN\
                           atrain.wav
                           atrain_codeca.wav
                           atrain_codecb.wav
                           atrain_codecc.wav
                           atrain_codecd.wav
                           atrain_codece.wav
                           atrain_codecf.wav



7. Open ABC/HR by executing abchr.jar - and, to test each sample, open up the individual .ecf files.  Each test set (test1 or test2) has an .ecf file for each sample. 

8. From now on, typical ABC/HR methodology applies, you need to judge the quality of individual files - save the results after you're done, and send the files to me, either by PP or directly to:  idimkovic (at) nero.com (replace at with @ to get an e-mail address)

Samples coming in the next post...

Happy Testing
  • Last Edit: 24 January, 2006, 06:47:51 AM by Ivan Dimkovic

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #1
Sample selection

Total 14 samples are tested, exposing all relevant issues in audio coding domain (pre-echoes, high tonal content, human speech, strong stereo separation, complex harmonic structure, etc...)  - Two of these samples were transcoded from the lossy source (Yello and Sympathy) in order to reflect typical 48 kbps codec usage even more.

So, here are the links:

ftp://ftp.morbo.org/pub/Nero_48_Test/

TEST 1

ftp://ftp.morbo.org/pub/Nero_48_Test/test1.rar  -- Test 1 SETUP
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_atrain.rar -- Test 1, Sample 1
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_biniou.rar -- Test 1, Sample 2
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_clapton_44k.rar -- Test 1, Sample 3
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_es02.rar -- Test 1, Sample 4
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_french_ad.rar -- Test 1, Sample 5
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_harp40_1.rar -- Test 1, Sample 6
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_mahler.rar -- Test 1, Sample 7
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_sandman.rar -- Test 1, Sample 8
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_si02.rar -- Test 1, Sample 9
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_sm01.rar -- Test 1, Sample 10
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_sympathy.rar -- Test 1, Sample 11
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_te09_44k.rar -- Test 1, Sample 12
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_yello.rar -- Test 1, Sample 13
ftp://ftp.morbo.org/pub/Nero_48_Test/test1_youcantdothat.rar -- Test 1, Sample 14

TEST 2

ftp://ftp.morbo.org/pub/Nero_48_Test/test2.rar  -- Test 2 SETUP
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_atrain.rar -- Test 2, Sample 1
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_biniou.rar -- Test 2, Sample 2
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_clapton_44k.rar -- Test 2, Sample 3
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_es02.rar -- Test 2, Sample 4
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_french_ad.rar -- Test 2, Sample 5
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_harp40_1.rar -- Test 2, Sample 6
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_mahler.rar -- Test 2, Sample 7
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_sandman.rar -- Test 2, Sample 8
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_si02.rar -- Test 2, Sample 9
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_sm01.rar -- Test 2, Sample 10
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_sympathy.rar -- Test 2, Sample 11
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_te09_44k.rar -- Test 2, Sample 12
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_yello.rar -- Test 2, Sample 13
ftp://ftp.morbo.org/pub/Nero_48_Test/test2_youcantdothat.rar -- Test 2, Sample 14

Edit - typo.
  • Last Edit: 24 January, 2006, 09:20:44 AM by Ivan Dimkovic

  • Shade[ST]
  • [*][*][*][*][*]
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #2
Finally it started! I'm going to take part in this ASAP!

  • IgorC
  • [*][*][*][*][*]
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #3
  • Last Edit: 24 January, 2006, 09:19:59 AM by IgorC

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #4
Thanks IgorC for noticing it - It is corrected now.

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #5
Quote
8. From now on, typical ABC/HR methodology applies, you need to judge the quality of individual files - save the results after you're done, and send the files to me, either by PP or directly to:  idimkovic (at) nero.com (replace at with @ to get an e-mail address)[a href="index.php?act=findpost&pid=359374"][{POST_SNAPBACK}][/a]

What are the requirements for producing valid results?

For example:
- Will ranking any reference make the result invalid?
- Will it be acceptable to mistake between the high anchor and reference?
- Will results with 5.0 ratings be invalid?

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #6
Quote
- Will ranking any reference make the result invalid?


I would say - yes.  Ranking reference in such a low bit rate test is clearly a sign of either a mistake or lack of listening training.

Quote
- Will it be acceptable to mistake between the high anchor and reference?


I will have to think about that - and discuss with some other people, too.

Quote
- Will results with 5.0 ratings be invalid?


5.0 results will be valid.

  • Firon
  • [*][*][*][*][*]
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #7
Do we have to perform all the tests for the results to count or can we do say, only test2? (I'm of course planning to do both)

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #8
You can do both the tests, or a single test, or only few samples of any set (of course, I'd prefer full results, but we also accept single sample results, too

  • vinouz
  • [*]
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #9
suggestion : a little  shell script instead of the batch file, for linux users...

  • IgorC
  • [*][*][*][*][*]
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #10
Maybe it is my ignorance but it seems to me there is no big difference between Nero VBR1/2 , ABR, CBR  on some (not on all?) samples like Sadman and Beatles (You can do it)
I also tried to ABX between Nero's VBR1&2 /CBR/ABRsamples . It does not help.

Code: [Select]
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname: Beatles

1L = C:\48\test2_youcantdothat\005 youcantdothat_codecD.wav
2L = C:\48\test2_youcantdothat\001 youcantdothat_codecE.wav
3L = C:\48\test2_youcantdothat\006 youcantdothat_codecF.wav
4R = C:\48\test2_youcantdothat\002 youcantdothat_codecA.wav
5R = C:\48\test2_youcantdothat\004 youcantdothat_codecC.wav
6R = C:\48\test2_youcantdothat\003 youcantdothat_codecB.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: C:\48\test2_youcantdothat\005 youcantdothat_codecD.wav
1L Rating: 2.3
1L Comment:
---------------------------------------
3L File: C:\48\test2_youcantdothat\006 youcantdothat_codecF.wav
3L Rating: 1.0
3L Comment:
---------------------------------------
4R File: C:\48\test2_youcantdothat\002 youcantdothat_codecA.wav
4R Rating: 2.3
4R Comment:
---------------------------------------
5R File: C:\48\test2_youcantdothat\004 youcantdothat_codecC.wav
5R Rating: 2.3
5R Comment:
---------------------------------------
6R File: C:\48\test2_youcantdothat\003 youcantdothat_codecB.wav
6R Rating: 2.3
6R Comment:
---------------------------------------
ABX Results:
Original vs C:\48\test2_youcantdothat\005 youcantdothat_codecD.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_youcantdothat\001 youcantdothat_codecE.wav
   5 out of 8, pval = 0.363
Original vs C:\48\test2_youcantdothat\006 youcantdothat_codecF.wav
   8 out of 8, pval = 0.004
Original vs C:\48\test2_youcantdothat\002 youcantdothat_codecA.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_youcantdothat\004 youcantdothat_codecC.wav
   8 out of 8, pval = 0.004
Original vs C:\48\test2_youcantdothat\003 youcantdothat_codecB.wav
   5 out of 5, pval = 0.031


Code: [Select]
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname: Sandman

1R = C:\48\test2_sandman\sandman\004 sandman_codecD.wav
2L = C:\48\test2_sandman\sandman\002 sandman_codecB.wav
3R = C:\48\test2_sandman\sandman\006 sandman_codecF.wav
4R = C:\48\test2_sandman\sandman\003 sandman_codecC.wav
5L = C:\48\test2_sandman\sandman\001 sandman_codecA.wav
6L = C:\48\test2_sandman\sandman\005 sandman_codecE.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: C:\48\test2_sandman\sandman\004 sandman_codecD.wav
1R Rating: 2.0
1R Comment:
---------------------------------------
2L File: C:\48\test2_sandman\sandman\002 sandman_codecB.wav
2L Rating: 2.0
2L Comment:
---------------------------------------
3R File: C:\48\test2_sandman\sandman\006 sandman_codecF.wav
3R Rating: 1.0
3R Comment:
---------------------------------------
4R File: C:\48\test2_sandman\sandman\003 sandman_codecC.wav
4R Rating: 2.0
4R Comment:
---------------------------------------
5L File: C:\48\test2_sandman\sandman\001 sandman_codecA.wav
5L Rating: 2.0
5L Comment:
---------------------------------------
6L File: C:\48\test2_sandman\sandman\005 sandman_codecE.wav
6L Rating: 4.7
6L Comment:
---------------------------------------
ABX Results:
Original vs C:\48\test2_sandman\sandman\004 sandman_codecD.wav
   8 out of 8, pval = 0.004
Original vs C:\48\test2_sandman\sandman\002 sandman_codecB.wav
   8 out of 8, pval = 0.004
Original vs C:\48\test2_sandman\sandman\006 sandman_codecF.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_sandman\sandman\003 sandman_codecC.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_sandman\sandman\001 sandman_codecA.wav
   5 out of 5, pval = 0.031
Original vs C:\48\test2_sandman\sandman\005 sandman_codecE.wav
   7 out of 8, pval = 0.035
  • Last Edit: 28 January, 2006, 09:49:19 PM by IgorC

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #11
Quote
suggestion : a little shell script instead of the batch file, for linux users...


Huh... must be because I am dumb windows user  If someone is willing to share some shell scripts doing this job with me - I could update the archives of course

Quote
IgorC
Maybe it is my ignorance but it seems to me there is no big difference between Nero VBR1/2 , ABR, CBR on some (not on all?) samples like Sadman and Beatles (You can do it)
I also tried to ABX between Nero's VBR1&2 /CBR/ABRsamples . It does not help.


Huh... "discussing of the results ist verboten" 

Anyway - this is the sole purpose of the test - to be honest, I am not at all amused in how would VBR perform on 48 kbps compared to ABR or even CBR - since there is not so much space to actually "adapt' the bit rate, and SBR anyway takes half of the spectrum (and SBR is not scalable) - but let's see after the test ends.

Also, too bad there is no time to test 2-pass CBR, too - but maybe in one of the future tests later this year

I would like to ask for more people to do the test, please  I only got 2 incomplete results so far - and if this trend continues, there won't be too much to discuss and analyse... thanks in advance!

  • archish
  • [*]
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #12
which which version of nero encoder is HE-AAC v2 supplied?
thanks

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #13
Basically all recent Nero7 AAC encoders have HE-AAC v2.

What we are testing here is completely new AAC encoder - which will be releaesed in short time - so its performance is quite different (I believe) than the current public N7 AAC encoder.

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #14
2 more days left for the test.

Due to lack of results (3 for test 1, 1 for test 2,  many incomplete) I might consider postponing finishing of this test until Sunday evening (5th of February), hopefully more people will have time to do it during weekend.

Please if you have time, do the test - I would expect 2 or 3 certain results for test 1 and test 2, but that is still not enough for reliable analysis.

Thanks a lot in advance!

  • Alex B
  • [*][*][*][*][*]
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #15
I am not promising anything, but I could give it a try next weekend. I have been very busy during the last few weeks and had no time for hobbies like this.

  • Gabriel
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #16
Quote
Due to lack of results (3 for test 1, 1 for test 2, many incomplete)

Does it means that right now you do not even have full results from you, Menno, Garf and Juha? 

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #17
 

Actually the results I mentioned are excludig @nero.com results.

I will refrain from testing the clips I authored because I can tell which codec is which and I feel I would be biased by that fact.

  • Firon
  • [*][*][*][*][*]
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #18
I'm still working on it, I've been busy unfortunately... I've finished test1, done 2 samples in test 2, so I've got a ways to go. I'm hoping to finish tomorrow.

I have a question though, what's the point of the low anchor in listening tests? So you can establish a scale or...? Pardon the n00b question.
And when the test is finished, do you get to see individual results? (so you can see if you screwed up and so on)
  • Last Edit: 02 February, 2006, 09:32:05 PM by Firon

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Nero 48 kbit/s HE-AAC (+v2) Test - STARTED!
Reply #19
Just a quick note - test is definitely extended until Sunday, 5th of February!

(I got more results, but still I need more for reliable analysis) - your feedback will be greatly appreciated!

Quote
I have a question though, what's the point of the low anchor in listening tests? So you can establish a scale or...? Pardon the n00b question. tongue.gif


Yep - it is to establish a reliable scale, since IMHO most of the codecs could be very tied.

Quote
And when the test is finished, do you get to see individual results? (so you can see if you screwed up and so on)


Yes, I will upload all individual results - and post the test keys, so everyone could check their own test locally, too.