Hydrogenaudio Forums

Hydrogenaudio Forum => Listening Tests => Topic started by: menno on 2006-12-21 10:20:40

Title: 80 kbps LC AAC listening test
Post by: menno on 2006-12-21 10:20:40
Dear All,

I would like to announce that the 80 kbps test of upcoming Nero AAC encoder has just started. The purpose of this test is to determine if PNS (Perceptual Noise Substitution) or IS (Intensity Stereo) gives any advantage when used at 80 kbps.
With this test, I wish to include true audiophile and scientific community into making the LC-AAC solutions even better in the future. Many thanks in advance for your help!
Test runs for 18 days, until January 8th 2007.

Which codecs are tested?

Following codecs are being tested:

- LC-AAC 80 kbps CBR – No PNS no IS
- LC-AAC 80 kbps CBR – No PNS, using  IS
- LC-AAC 80 kbps CBR – No IS, using PNS
- LC-AAC 80 kbps CBR – Using both PNS and IS
- Hidden codec 1 80 kbps
- Hidden codec 2 (high anchor)
- Hidden codec 3 (low anchor)


How to Test

1. Download http://s4.quicksharing.com/v/5647513/ABC_HR.ZIP.html (http://s4.quicksharing.com/v/5647513/ABC_HR.ZIP.html)
2. Extract ABC_HR.ZIP in folder of your choice
3. Read Readme.txt from ABC_HR.ZIP
4. Download samples for the test (links are in ABC_HR.ZIP and in next post), and extract them in the directory where ABC_HR.zip was extracted
5. When you download and extract all the samples, execute the decode_all.bat files from bin folder. This will decode all the files and prepare the test
6. Now everything is ready for the test
7. Open ABC/HR by executing abchr.jar - and, to test each sample, open up the individual .ecf files. Each test set (Sample1, Sample2,…) has an .ecf file for each sample
8. From now on, typical ABC/HR methodology applies, you need to judge the quality of individual files - save the results after you're done, and send the files to me, either by PP or directly to: mbakker (at) nero.com (replace at with @ to get an e-mail address)

Links to samples coming in the next post...

Happy Testing 


Here are the samples for the test. Samples from previous Mares and Robertos test at 48 kbps and 128 kbps were chosen. Test with lowest scores on those tests were chosen. This is because even at 48 kbps for many people it was hard to distinguish encoded samples from the original. Also most genres of music are present.

http://s19.quicksharing.com/v/9204458/Sample1.zip.html (http://s19.quicksharing.com/v/9204458/Sample1.zip.html)
http://s4.quicksharing.com/v/801115/Sample2.zip.html (http://s4.quicksharing.com/v/801115/Sample2.zip.html)
http://s4.quicksharing.com/v/7815217/Sample3.zip.html (http://s4.quicksharing.com/v/7815217/Sample3.zip.html)
http://s4.quicksharing.com/v/3713672/Sample4.zip.html (http://s4.quicksharing.com/v/3713672/Sample4.zip.html)
http://s4.quicksharing.com/v/5514459/Sample5.zip.html (http://s4.quicksharing.com/v/5514459/Sample5.zip.html)
http://s4.quicksharing.com/v/5814487/Sample6.zip.html (http://s4.quicksharing.com/v/5814487/Sample6.zip.html)
http://s4.quicksharing.com/v/7761119/Sample7.zip.html (http://s4.quicksharing.com/v/7761119/Sample7.zip.html)
http://s4.quicksharing.com/v/3128033/Sample8.zip.html (http://s4.quicksharing.com/v/3128033/Sample8.zip.html)
http://s4.quicksharing.com/v/5707785/Sample9.zip.html (http://s4.quicksharing.com/v/5707785/Sample9.zip.html)
http://s4.quicksharing.com/v/8088598/Sample10.zip.html (http://s4.quicksharing.com/v/8088598/Sample10.zip.html)
http://s4.quicksharing.com/v/6741335/Sample11.zip.html (http://s4.quicksharing.com/v/6741335/Sample11.zip.html)
http://s4.quicksharing.com/v/1364761/Sample12.zip.html (http://s4.quicksharing.com/v/1364761/Sample12.zip.html)
http://s4.quicksharing.com/v/5209480/1Sample13.zip.html (http://s4.quicksharing.com/v/5209480/1Sample13.zip.html)
http://s4.quicksharing.com/v/1426353/Sample14.zip.html (http://s4.quicksharing.com/v/1426353/Sample14.zip.html)
Title: 80 kbps LC AAC listening test
Post by: dand on 2006-12-21 11:20:35
Good idea to clarify the is&pns issues.

Myself: somehow sceptical on IS, some more expectation from PNS (certain type of samples only), even, as everything else, it all depends on implementation.

But lets wait fot the test results!
Title: 80 kbps LC AAC listening test
Post by: Sebastian Mares on 2006-12-21 11:42:21
Can anyone confirm that downloads work only with IE?
Title: 80 kbps LC AAC listening test
Post by: menno on 2006-12-21 11:45:41
Can anyone confirm that downloads work only with IE?


If you block the google ads the download link doesn't show up.
Title: 80 kbps LC AAC listening test
Post by: Maurits on 2006-12-21 11:50:21
Can anyone confirm that downloads work only with IE?

Works with Firefox 2.0.0.1 on OS X 10.4.8...
Title: 80 kbps LC AAC listening test
Post by: guruboolez on 2006-12-21 11:59:42
Opera 9.1 is OK too.
Title: 80 kbps LC AAC listening test
Post by: Sebastian Mares on 2006-12-21 12:14:26

Can anyone confirm that downloads work only with IE?


If you block the google ads the download link doesn't show up.


OK, Adblock's fault then.
Title: 80 kbps LC AAC listening test
Post by: IgorC on 2006-12-21 14:57:50
menno
Only LC is included to test?
Is the reason that you want to see how PNS and IS perform on LC part only and not on SBR or  new encoder will bring better quality with LC profile?
Title: 80 kbps LC AAC listening test
Post by: spockep on 2006-12-21 18:37:26
So in short you just need the .erf files zipped then?
Title: 80 kbps LC AAC listening test
Post by: Firon on 2006-12-22 04:47:25
Quicksharing isn't loading for me at all...
Can someone upload these to another site? Like zshare.
Title: 80 kbps LC AAC listening test
Post by: menno on 2006-12-22 07:45:06
menno
Only LC is included to test?
Is the reason that you want to see how PNS and IS perform on LC part only and not on SBR or  new encoder will bring better quality with LC profile?


The use of PNS and IS with SBR is questionable, because it will usually be used in the same range. And anyway, if PNS or IS doesn't help for LC only it will also not help when using SBR and vice versa.
Title: 80 kbps LC AAC listening test
Post by: halb27 on 2006-12-22 09:14:33
Can a user expect that his LC decoder (for instance the one in my Nokia 6230) makes use of PNS and/or IS?
Title: 80 kbps LC AAC listening test
Post by: menno on 2006-12-22 09:25:23
Can a user expect that his LC decoder (for instance the one in my Nokia 6230) makes use of PNS and/or IS?


All AAC decoders must support IS, all MPEG-4 AAC decoders must support PNS.
To answer your question, try these samples on iPod and weep  (it's sad that Apple did not fix this yet), for other decoders the only way to find out is to test it, but besides iPod I am not aware of any uncompliant decoders.
Title: 80 kbps LC AAC listening test
Post by: muaddib on 2006-12-22 09:45:17
Here, I have uploaded all the files to zshare:

abc-hr.zip - 1.07MB (http://www.zshare.net/download/abc-hr-zip.html)
sample1.zip - 4.36MB (http://www.zshare.net/download/sample1-zip.html)
sample2.zip - 2.38MB (http://www.zshare.net/download/sample2-zip.html)
sample3.zip - 4.06MB (http://www.zshare.net/download/sample3-zip.html)
sample4.zip - 4.12MB (http://www.zshare.net/download/sample4-zip.html)
sample5.zip - 2.11MB (http://www.zshare.net/download/sample5-zip.html)
sample6.zip - 4.56MB (http://www.zshare.net/download/sample6-zip.html)
sample7.zip - 2.81MB (http://www.zshare.net/download/sample7-zip.html)
sample8.zip - 5.10MB (http://www.zshare.net/download/sample8-zip.html)
sample9.zip - 4.49MB (http://www.zshare.net/download/sample9-zip.html)
sample10.zip - 5.06MB (http://www.zshare.net/download/sample10-zip.html)
sample11.zip - 5.59MB (http://www.zshare.net/download/sample11-zip.html)
sample12.zip - 1.66MB (http://www.zshare.net/download/sample12-zip.html)
sample13.zip - 3.07MB (http://www.zshare.net/download/sample13-zip.html)
sample14.zip - 1.90MB (http://www.zshare.net/download/sample14-zip.html)
Title: 80 kbps LC AAC listening test
Post by: muaddib on 2006-12-22 10:02:55
For most of the tests it is hard for me to determine what is an encoded sample and what is the original.
What do you people think: should results where only low anchor is determined also be sent and counted as relevant data?
Title: 80 kbps LC AAC listening test
Post by: menno on 2006-12-22 12:03:00
So in short you just need the .erf files zipped then?

Yeah just zip them and send them to me.
Title: 80 kbps LC AAC listening test
Post by: IgorC on 2006-12-23 02:39:29
For most of the tests it is hard for me to determine what is an encoded sample and what is the original.
What do you people think: should results where only low anchor is determined also be sent and counted as relevant data?

That would a half of problem.  On some samples after to be enable to abxed them (even high anchor) it's still hard to judge wich one was slightly better than another one.
Title: 80 kbps LC AAC listening test
Post by: muaddib on 2006-12-23 19:47:20

For most of the tests it is hard for me to determine what is an encoded sample and what is the original.
What do you people think: should results where only low anchor is determined also be sent and counted as relevant data?

That would a half of problem.  On some samples after to be enable to abxed them (even high anchor) it's still hard to judge wich one was slightly better than another one.


Yes, that is a big problem. That is especially evident when average rating for a sample is between 2 and 3. But that problem IMO does not have a solution.

p.s. All your bases are belong to us!
Title: 80 kbps LC AAC listening test
Post by: SirChristof on 2007-01-05 08:07:49
My main problem is, in order to properly prepare myself for the "Official" ABC/HR test, I always do very very many ABX tests---and since there are so many samples being tested (7 in this case), by the time I have finished training myself on them via ABX, my ears are too fatigued to take the "official" test in an ideal fashon.

Part of the issue (at least for me) is that some artifacts in some of the samples, are nearly inaudible unless listened to a bit on the loud side.  So I get fatigued rather quickly.

I do have solid gear and a very quiet listening environment, so I will make my best attempt to contribute regardless. 
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-01-05 10:43:53
First of all, thanks to everyone who has already contributed!

For the people who still want to participate, please remember that the test runs until the 8th, which is next monday!
Title: 80 kbps LC AAC listening test
Post by: IgorC on 2007-01-06 19:51:17
menno
Can you confirm if you received my results? Often the messages end in spam folder. Thank you.
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-01-07 10:16:10
menno
Can you confirm if you received my results? Often the messages end in spam folder. Thank you.

Yes got them  thanks.
Title: 80 kbps LC AAC listening test
Post by: muaddib on 2007-01-08 09:49:04
My main problem is, in order to properly prepare myself for the "Official" ABC/HR test, I always do very very many ABX tests---and since there are so many samples being tested (7 in this case), by the time I have finished training myself on them via ABX, my ears are too fatigued to take the "official" test in an ideal fashon.

Part of the issue (at least for me) is that some artifacts in some of the samples, are nearly inaudible unless listened to a bit on the loud side.  So I get fatigued rather quickly.


IMO it is not important to do so much detailed ABX test.
In tests conducted by ITU (for development of PEAQ) testers were asked to give scores even if they are not certain what is the original and what is a test sample. It was even desired that they don't give 5 but rather something between 4 and 5. So people sometimes judged incorrectly what is the original and what is a test. There are results in ITU tests with positive SDG (meaning that a test sample sounds better than the original).
IMO it is not good to have this kind of results, but it is also better to have results with 5 than not to have any results at all. 5 is also good because it means that you can not hear the difference. So if it is hard for you to ABX simple give up and give 5
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-01-08 12:35:03
Because we didn't get so many results yet, we decided to let the test run for one week longer. So now you can send in your results until the 15th of January.
Thanks for your help.
Title: 80 kbps LC AAC listening test
Post by: SirChristof on 2007-01-08 13:07:14
IMO it is not important to do so much detailed ABX test.
In tests conducted by ITU (for development of PEAQ) testers were asked to give scores even if they are not certain what is the original and what is a test sample. It was even desired that they don't give 5 but rather something between 4 and 5. So people sometimes judged incorrectly what is the original and what is a test. There are results in ITU tests with positive SDG (meaning that a test sample sounds better than the original).
IMO it is not good to have this kind of results, but it is also better to have results with 5 than not to have any results at all. 5 is also good because it means that you can not hear the difference. So if it is hard for you to ABX simple give up and give 5


Determining that I hear a difference is easy.  Say we have original, and 2 encoded "test files", O, A and B respectively.  Even in such a case where I can hear a difference between O & A, O & B, and A & B, it can be difficult to truly give a "better" answer.  Doing very many ABX tests between them (assuming you hear a difference), better allows you to "rate" them.  Without such attention to detail, the test becomes less meaningfull, at least in my opinion.

I am still in the process of doing some of these test samples for this listening test.

A small handful of the given test samples, were transparent to me upon my initial listen.

If I did not do many ABX tests to first prepare myself to get accustomed to what I should be hearing, I could simply say 5/5 and move on.  When people don't give a full effort, and just listen a couple times then determine "It is transparent", that severely waters down the results from people like me who actually did everything in their power to not only hear the difference, but rate it as honestly as they can.

On a side-note, I did not hear about this test until the 5th, so the extension from the 8th to 15th is welcomed.  Thanks menno.
Title: 80 kbps LC AAC listening test
Post by: muaddib on 2007-01-08 13:27:49
Doing very many ABX tests between them (assuming you hear a difference), better allows you to "rate" them.  Without such attention to detail, the test becomes less meaningfull, at least in my opinion.
...
If I did not do many ABX tests to first prepare myself to get accustomed to what I should be hearing, I could simply say 5/5 and move on.  When people don't give a full effort, and just listen a couple times then determine "It is transparent", that severely waters down the results from people like me who actually did everything in their power to not only hear the difference, but rate it as honestly as they can.

Your metodology for participating in listening test is meaningful and it would be really nice to have some test with people that are doing just like you described. But how many people would do that? IMO not many even if you pay them.
In public listening test many people don't give as much effort as you. But this does not "water down" your results. Public listening test produce average scores (that is average scores among people interested in doing test which should extrapolate to whole population that listens music). Most people don't hear differences without full effort, but their grades also count.
Also have in mind that in another time in another place in different circuimstances, you wouldn't give exactly the same grades  People are just not accurate enough.
In short: if you are deciding between 4.6 or 4.7, just give any of those.
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-01-15 12:32:35
Again thanks to all who have participated. Unfortunately we still don't have enough results at the moment. We will let the test run a little longer, until 29th of january for now.
Without your help it is not possible to determine if we should enable PNS and/or IS or not.
Title: 80 kbps LC AAC listening test
Post by: starcy on 2007-01-16 10:46:09
Why not just user can decide this?
Title: 80 kbps LC AAC listening test
Post by: Ivan Dimkovic on 2007-01-16 11:21:04
Because we want to avoid flood of suboptimal encodings (e.g. 192 kbps, PNS and IS ON) 

If we put PNS and IS in the encoder, they will either be off by default, or there will be a switch to easily disable them because one very widespread portable player (Apple iPod) appears not to support them well.
Title: 80 kbps LC AAC listening test
Post by: dand on 2007-01-16 16:13:17
...because one very widespread portable player (Apple iPod) appears not to support them well.

This is so hard to believe... Can't they (Apple) just fix this and issue a firmware update...
Title: 80 kbps LC AAC listening test
Post by: IgorC on 2007-01-16 21:50:17

...because one very widespread portable player (Apple iPod) appears not to support them well.

This is so hard to believe... Can't they (Apple) just fix this and issue a firmware update...

Considering that Apple always goes to very easy and low complexity codecs like ASP without  Qpel, GMC, trellis and H.264 only up to not full Main Profile it's not surprise if they doesn't support IS and PNS.  Don't talk about SBR.
Title: 80 kbps LC AAC listening test
Post by: Garf on 2007-01-16 22:34:26


...because one very widespread portable player (Apple iPod) appears not to support them well.

This is so hard to believe... Can't they (Apple) just fix this and issue a firmware update...

Considering that Apple always goes to very easy and low complexity codecs like ASP without  Qpel, GMC, trellis and H.264 only up to not full Main Profile it's not surprise if they doesn't support IS and PNS.  Don't talk about SBR.


Well, this is different. An AAC codec without IS is just broken, period.

And you can't call it "MP4" without PNS, either.

This isn't about some optional extensions, those tools are REQUIRED even in the low-complexity profile.
Title: 80 kbps LC AAC listening test
Post by: dand on 2007-01-17 12:54:44


...because one very widespread portable player (Apple iPod) appears not to support them well.

This is so hard to believe... Can't they (Apple) just fix this and issue a firmware update...

Considering that Apple always goes to very easy and low complexity codecs like ASP without  Qpel, GMC, trellis and H.264 only up to not full Main Profile it's not surprise if they doesn't support IS and PNS.  Don't talk about SBR.


Well, this is different. An AAC codec without IS is just broken, period.

And you can't call it "MP4" without PNS, either.

This isn't about some optional extensions, those tools are REQUIRED even in the low-complexity profile.


We should be looking at encoder and decoder separately here. Not supporting a feature in ENcoder is completely different than doing the same in DEcoder (i.e. failing to build a standard-conformant DEcoder).
Title: 80 kbps LC AAC listening test
Post by: Garf on 2007-01-17 17:08:53

Well, this is different. An AAC codec without IS is just broken, period.

And you can't call it "MP4" without PNS, either.

This isn't about some optional extensions, those tools are REQUIRED even in the low-complexity profile.


We should be looking at encoder and decoder separately here. Not supporting a feature in ENcoder is completely different than doing the same in DEcoder (i.e. failing to build a standard-conformant DEcoder).


IS and PNS (in MPEG-4) are REQUIRED in the DEcoder.

The iPod is not a standard-conformant decoder.
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-01-17 17:45:39
I think IS does work, just the PNS gave troubles
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-01-25 15:04:51
It would be great if we could get some more results for samples 4,5,7,9,10,11 and 13
Title: 80 kbps LC AAC listening test
Post by: sketchy_c on 2007-01-28 22:20:07
I should have read that last post before testing tonight. 
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-01-29 18:38:13
Since we still don't have enough results, we will simply let the test run until we do have enough (we do need an answer). I will let you know when we have enough results and from that moment people who are still doing the test will get a week to finish it.
Thanks again to everyone who already participated.
Title: 80 kbps LC AAC listening test
Post by: gameplaya15143 on 2007-01-31 00:25:16
Cool.. now I just have to stop forgetting this test is going on
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-02-01 07:42:49
Cool.. now I just have to stop forgetting this test is going on


bump
Title: 80 kbps LC AAC listening test
Post by: IgorC on 2007-02-06 18:41:27
German speech sample N5  was very low scored in 48 kbps test. While in this test it's transparent for me almost for all encode modes. Of course, 80 is a higer bitrate but, for example, LAME -V5 and Vorbis at 80 also have some problems with this sample.
Good work.
Title: 80 kbps LC AAC listening test
Post by: gameplaya15143 on 2007-02-09 01:38:30
I got through samples 14-8 before I started to lose my mind  (yes I'm going backwards)
I think I started hearing things that weren't even there  time for a little break.
Title: 80 kbps LC AAC listening test
Post by: Diow on 2007-02-09 02:06:01
I'm save my results in .erf but don't save them in .abc now I can't open them [the .erf] how can I open these .erf?
Title: 80 kbps LC AAC listening test
Post by: Firon on 2007-02-09 02:12:26
You can't open them. THey're encrypted result files that you need to send to menno.
Title: 80 kbps LC AAC listening test
Post by: Diow on 2007-02-09 11:53:29
You can't open them. THey're encrypted result files that you need to send to menno.

Thanks, indeed I Know about this the problem that's I don't save the sessions (I save but delete without want but made this)...
Title: 80 kbps LC AAC listening test
Post by: sketchy_c on 2007-02-13 12:17:35
Menno, is this still ongoing?  Not sure if the 1.0.7.0 release was dependent on these results.
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-02-13 14:50:30
Yes, test is still going on. We are very thankful for everyone submitting their test results. The results of this test will be used for our next release.
Title: 80 kbps LC AAC listening test
Post by: Jerethi on 2007-03-04 00:54:30
Currently working my way through the samples.  Completed 4 samples so far.
Title: 80 kbps LC AAC listening test
Post by: Diow on 2007-03-04 14:31:40
Is the test now ended or still running?
Title: 80 kbps LC AAC listening test
Post by: Jerethi on 2007-03-04 19:01:48
Is the test now ended or still running?


My understanding is that it is open until enough people submit their results.
Title: 80 kbps LC AAC listening test
Post by: menno on 2007-03-05 09:43:38
We now have about enough results, test will end on 12th of march. So people still working on the test have until that day to send in their results. Thanks everyone for the help!
Title: 80 kbps LC AAC listening test
Post by: The Sheep of DEATH on 2007-03-15 15:55:02
So how did it go?  (I mean, were you able to answer the initial question "Are PNS and IS useful at 80kbps?")
And will we avid NeroDigital fans soon be treated to a smooth LC encoder update?

edit: never mind, I saw the results thread here: http://www.hydrogenaudio.org/forums/index....showtopic=53340 (http://www.hydrogenaudio.org/forums/index.php?showtopic=53340)
SimplePortal 1.0.0 RC1 © 2008-2020