HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: IgorC on 2011-07-17 22:55:37

Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-07-17 22:55:37
Today we have a chance to see the performance of well known AAC encoders like Nero, Apple and Coding Technologies (previous versions of Winamp). 
Also Fraunhofer have released their totally new AAC encoder.


The following codecs are presented in this test:
Nero 1.5.4
Apple QuickTime 7.6.9  true VBR
Apple QuickTime 7.6.9  constrained VBR
Fraunhofer (Winamp 5.62)
Coding Technologies (Winamp 5.61)
ffmpeg's AAC (low anchor)

The homepage of the test:
Public AAC Listening Test @ ~96 kbps [July 2011] (http://listening-tests.hydrogenaudio.org/igorc/aac-96-a/)


There are 20 samples.  Please, don't hurry. Take your time as there will be enough of it. It's better to do fewer samples per day and avoid a fatigue.
The test will be open during 10 days (until July 27)

As always every single result is helpful.
Have a good time testing it!
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: benski on 2011-07-18 17:53:41
Thanks so much, Igor.

Hope you don't mind, but I mirrored the large ZIP file with all the samples here:
http://download.nullsoft.com/listening-tes...all_samples.zip (http://download.nullsoft.com/listening-tests/aac-96-a/all_samples.zip)
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: Garf on 2011-07-18 19:54:37
I added the code to randomize the sample list, hope that's OK with you.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-07-18 20:01:53
Hope you don't mind, but I mirrored the large ZIP file with all the samples here:
http://download.nullsoft.com/listening-tes...all_samples.zip (http://download.nullsoft.com/listening-tests/aac-96-a/all_samples.zip)

Thank you.

I added the code to randomize the sample list, hope that's OK with you.

Yes, it's better this way
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-07-19 01:13:33
Ok, some listeners have started to send their results.
It will be better if somebody who has already any results send them. It helps to prevent some simple errors related to ABC-HR application or any other at early stage. 
The listeners usually will get answer quickly if everything is fine.

Also it is recommended to use headphones during the test.


Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: kennedyb4 on 2011-07-20 12:11:05
Hello. I am trying to participate. I downloaded the sample pack above and used the decode all bat file. I can't get abc/hr to play anything yet.

Can someone point me to a tutorial? I see a training mode but I don't really know what I'm doing.

Edit  Ok I checked out ff123 page. I have opened the config file for sample 1 after running decode all as well as decode sample 1 bat files. there are 6 wav files in the directory and they all play in winamp OK.

The main abc/hr page is blank with no samples selected or able to be played. I have tried to load the config file over and again with no result.  Still can't play anything.

Edit 2  ABC/HR auto selected spdif out on my m-audiophile 24/96 rather than line. I can play and hear now. Still not sure what I am doing though
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: xnor on 2011-07-20 15:53:29
I've taken the liberty to mention this test over at head-fi (link (http://www.head-fi.org/forum/thread/563566/public-aac-listening-test-96-kbps-july-2011)), hope that's ok IgorC.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-07-20 22:34:30
Edit 2  ABC/HR auto selected spdif out on my m-audiophile 24/96 rather than line. I can play and hear now.

So does it mean you have managed to fix the problems?
ABC-HR application has presented the problem in past.  See the link below to fix some problems.
http://www.hydrogenaudio.org/forums/index....st&p=749194 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=87785&view=findpost&p=749194)


I've taken the liberty to mention this test over at head-fi (link (http://www.head-fi.org/forum/thread/563566/public-aac-listening-test-96-kbps-july-2011)), hope that's ok IgorC.

Thank You, I haven't time to do it yet.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: naylor83 on 2011-07-21 16:48:21
Was a long time since I was here ... Just found out about the listening test. Will definitely join in. I'm very curious to see how the best AAC codec fares against AoTuV b6, in a future test.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: kennedyb4 on 2011-07-21 17:30:55
Edit 2  ABC/HR auto selected spdif out on my m-audiophile 24/96 rather than line. I can play and hear now.

So does it mean you have managed to fix the problems?
ABC-HR application has presented the problem in past.  See the link below to fix some problems.
http://www.hydrogenaudio.org/forums/index....st&p=749194 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=87785&view=findpost&p=749194)


I can hear the samples now,yes. Finding fault with them is proving much harder in some cases. I can't even hear the lowpass which is really suprising me. Much more difficult than I anticipated. Not like listening to Blade at 96kbps
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: naylor83 on 2011-07-21 18:06:59
Finding fault with them is proving much harder in some cases. I can't even hear the lowpass which is really suprising me. Much more difficult than I anticipated. Not like listening to Blade at 96kbps


Don't like the sound of that 
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: guruboolez on 2011-07-21 21:57:46
My first listening tests since many years
I probably lost most of my skill, but it's very interesting for me to see if I'm still able to make a distinction between ~100 kbps encoding and an uncompressed PCM sound. I guess encoders have progressed during the last years. I remember that last time I tested LC-AAC at this bitrate, I was rather disappointed. But is was four or five years ago and I was at the apogee of my skill.

For now, I completed the five first samples. Well, it's interesting! It's nice to see that so many implementations in one test (but it makes the evaluation longer and harder).
Thank you IgorC and other people involved for organizing it!
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: xnor on 2011-07-21 23:30:20
It's definitely not an easy test.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: kennedyb4 on 2011-07-22 00:28:02
Question. I have used a short selection in the abx mode for a given sample. Once I use test mode,not training mode, I cant erase my results and try again in test mode. I can only use training mode even though I am checking a different time frame of the sample.

Is it supposed to be this way or can I reset back to test mode?
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: muaddib on 2011-07-22 09:19:22
Question. I have used a short selection in the abx mode for a given sample. Once I use test mode,not training mode, I cant erase my results and try again in test mode. I can only use training mode even though I am checking a different time frame of the sample.

Is it supposed to be this way or can I reset back to test mode?

It is supposed to be that way and does exactly what it should. Don't start test mode until you have heard a complete sample and choose your "worst" section before starting the test mode.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: kennedyb4 on 2011-07-22 14:02:46
Question. I have used a short selection in the abx mode for a given sample. Once I use test mode,not training mode, I cant erase my results and try again in test mode. I can only use training mode even though I am checking a different time frame of the sample.

Is it supposed to be this way or can I reset back to test mode?

It is supposed to be that way and does exactly what it should. Don't start test mode until you have heard a complete sample and choose your "worst" section before starting the test mode.


Thanks. I tested a few sections that I had been able to abx in practice mode and failed during the actual test. I was then able to pick another section that I was able to detect in practice mode but could not repeat the testing.

This is the first time I have used abx/hr. I will run more practice tests in the future before making a definitive test.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: kennedyb4 on 2011-07-22 15:37:38
OK. Mission accomplished and results sent in. Did 20 samples over 3 days. I'm on vacation so what the heck. Very interesting process but a bit grueling too.

I had  lots of trouble. Figured I would at least be able to detect a 16K lowpass regularly but I can't anymore. I was at a Rockfest for 4 days last weekend which I hope is part of the problem.

I hope my results are not rubbish because I put in alot of time.

Can't wait to see how it all turns out. Thanks to all concerned for an interesting experience.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: Zarggg on 2011-07-22 16:35:58
I'm going to try finishing this one; I never finished the first test I participated in here, and I didn't participate in the last one at all. (48kbps, which I might have actually had fun with).

One slight issue I've noticed so far: I was not able to identify the low anchor at all on Sample 3. Does this predict a problem for my results? The quality this test focuses is right on the edge of my transparent zone, so I don't want to risk tainting the results.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: kennedyb4 on 2011-07-22 17:32:26
I'm going to try finishing this one; I never finished the first test I participated in here, and I didn't participate in the last one at all. (48kbps, which I might have actually had fun with).

One slight issue I've noticed so far: I was not able to identify the low anchor at all on Sample 3. Does this predict a problem for my results? The quality this test focuses is right on the edge of my transparent zone, so I don't want to risk tainting the results.


You'll get it. Its not mangled like the other samples but its there. Its definitely harder than the others.

If I remember the other listening test discussions, they reject results on certain criteria, like consistently incorrectly identifying the reference or giving it scores too low to be credible. Honest results should not taint but contribute to the experiment.

PS I think I'll get my son to join HA and give this a try. He is a music major and interested in music and music compression. Maybe he could get his friends interested....
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-07-22 19:24:52
My first listening tests since many years

I'm glad to see You back. 

As first observation,  some listeners are expecting to hear the same artifacts as in past. And of course there are new versions of codecs and artifacts are different.
The more listener tries to listen some particular artifacts the less chance to spot any of them. But if the listener just tries to enjoy the music itself then the artifacts appear themselves.

Also it's very important to mention about some practice with headphones. It's known that even good headphones can sound bad because of not optimal position on the head. It's easy to check. While the music is playing the listener should try different positions and distances (different adjustment of the rim) without set the headphones completely on the head.
It might sound crazy but it works.
More here:

http://www.meier-audio.homepage.t-online.de/tipstricks.htm (http://www.meier-audio.homepage.t-online.de/tipstricks.htm)
Quote
headphone position   

   It's so easy, but did you ever test for the optimal position of your headphone? With many headphones the soundstage and the balance of sound are strongly dependent on the position of the driver to our ears.

Scientific research has shown that our perception of depth is increased when the driver is placed more forward and lower with respect to the entrance of the ear channel.

Experimenting doesn't cost you anything and may improve sound considerably.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: muaddib on 2011-07-23 08:43:15
The more listener tries to listen some particular artifacts the less chance to spot any of them. But if the listener just tries to enjoy the music itself then the artifacts appear themselves.

Depends on a listener: how well he can hear, how much concentration he has, his experience, familiarity with the sample,...
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: naylor83 on 2011-07-23 23:29:44
Have only had time to do two samples so far. (16 & 20) (I found this listening test just a few days ago.)

Igor, which samples should I focus on if I have to chose? Any with fewer results at the moment?
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-07-24 10:54:01
Have only had time to do two samples so far. (16 & 20) (I found this listening test just a few days ago.)

You can send a few results already now to be sure that everything is ok.

Quote
Igor, which samples should I focus on if I have to chose?

All of them.
It's very unlikely to have enough results during first 10 days so it's more likely (almost sure) that test will be extended for another 10 days.


The rules are the same as from previous test with the small change of possibilities for ranking the references (3 instead of 4) because now there are less samples .

Quote
If the listener ranked the reference worse than 4.5 on a sample, the listener's results for that sample were discarded.
If the listener ranked the low anchor at 5.0 on a sample, the listener's results for that sample were discarded.
If the listener ranked the reference below 5.0 on more than 3 samples, all of that listener's results were discarded.

These rules aren't extremely strict in order to allow for simple human
error while still excluding careless participants (*).


A stricter procedure to exclude all ranked references risks a systemic
bias against any codec which are very good on a few samples and thus
subject to more reference confusion by causing those samples to be excluded
and weighing the test towards other samples.


(*)During the previous test some listeners have made the reasonable errors by accident (for example, have forgotten to rank the samples accidentally). But next time the listener wouldn't do such errors again.


Some detailed description of the rules from previous test. http://www.hydrogenaudio.org/forums/index....st&p=749501 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=87785&view=findpost&p=749501)
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: naylor83 on 2011-07-24 11:35:35
Ok, thanks.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: jukkap on 2011-07-25 18:50:01
This is the 1st listening test that I am trying to do and I found it hard !! I am impressed !! It is extremely difficult to hear differences.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: kennedyb4 on 2011-07-25 22:09:29
This is the 1st listening test that I am trying to do and I found it hard !! I am impressed !! It is extremely difficult to hear differences.


Go slow and take lots of breaks. Whatever you do, don't guess because the rules are strict about ranking the reference in error. The encoders are really great at this bitrate aren't they?
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-07-27 02:15:45
The test is extended until August 5.

We need more participants.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: Zarggg on 2011-07-27 03:27:39
Just sent off my results.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: mjb2006 on 2011-07-27 06:18:38
If the deadline is approaching and we can't get through all 20 samples, is it OK to send results for just the tests we've completed?

I had already saved my results files before realizing I could've put my nickname in them. Is it possible to add the nickname to the files after-the-fact?
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: Alexxander on 2011-07-27 11:09:14
Yes, every single result is very helpful.

Haven't you saved the individual Sessions? If you did, you can reopen it and add your nick, but I don't know how much work it would save for IgorC.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: mjb2006 on 2011-07-28 00:31:45
Haven't you saved the individual Sessions?

No, once I was done setting all the sliders for a sample, I didn't think there'd be any reason to save the session. There was no mention of it in the readme or the practice page. Oh well. I am just kind of curious to know e.g. how much better/worse I am than others at noticing certain differences.

Is it necessary or meaningful to use the ABX features?
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-07-28 03:52:56
Is it possible to add the nickname to the files after-the-fact?

The results of anonymous listener will be named as 01_anon.txt

If the listener put the name during the session or says that he wants to put the name then the results will be like 01_John.txt

Is it necessary or meaningful to use the ABX features?

If You aren't sure 100% that You actually hear the difference then You should use ABX button.
After successful ABX session in Testing Mode (5 valid tries is minimum - 5/5)  ABC-HR application will show you which is reference and which is compressed. It will lock the the slider of the reference. This way it's impossible to make a mistake.

Some useful guides
http://www.ecodis.de/audio/guideline_high.html (http://www.ecodis.de/audio/guideline_high.html)
and
http://ff123.net/64test/practice.html (http://ff123.net/64test/practice.html)
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: lvqcl on 2011-07-28 14:38:21
I noticed that faac doesn't support gapless decoding and adds up to 1600 samples to the beginning of WAV files. IMHO this should be avoided in future listening tests.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: Alex B on 2011-07-28 19:19:40
ABC/HR for JAVA has the Calculate Offsets feature. It should be able to handle the encoder and decoder delays (assuming the feature was used when the encrypted test configuration files were created).


IgorC,

Thanks for the extension. I have been on vacation, but I can do at least some testing during the next weekend.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: lvqcl on 2011-07-28 19:27:04
Thanks for clarification.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-07-29 01:06:10
I noticed that faac doesn't support gapless decoding and adds up to 1600 samples to the beginning of WAV files. IMHO this should be avoided in future listening tests.

The low anchor is ffmpeg's AAC encoder which is not FAAC.

ABC/HR for JAVA has the Calculate Offsets feature. It should be able to handle the encoder and decoder delays (assuming the feature was used when the encrypted test configuration files were created).

Yes, the offsets were calculated + additional offset (ABC/HR application) that cuts first 1 second of the sample.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: lvqcl on 2011-07-29 14:06:02
The low anchor is ffmpeg's AAC encoder which is not FAAC.

I meant decoder faad...
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-08-02 17:34:38
Many listeners ask for some extra time. I guess it's because of the holidays. 

Probably it won't be bad to extend the test during the weekend. The new listeners are still required.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: Larson on 2011-08-04 13:04:31
Just a note guys, Quicktime has been updated on Windows as well, 7.7 like on Lion; I don't know if it contains changes to AAC encoding, but for those who are testing for the listening test it may be interesting.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: Gecko on 2011-08-04 19:55:10
I'm through most samples but prematurely aborted some due to fatigue. I would like to return to them but can not do so before the weekend.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: no404error on 2011-08-04 23:22:31
Just a note guys, Quicktime has been updated on Windows as well, 7.7 like on Lion; I don't know if it contains changes to AAC encoding, but for those who are testing for the listening test it may be interesting.


QuickTime 7.7 improves security. That's all.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-08-05 00:27:39
The test is extended at least until  August 8, Monday (9:00 of GMT -3:00)
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: Alexxander on 2011-08-08 09:21:08
I have done 7 so far, but this week I have days off and would be able to do possibly all samples befor next weekend. So if it's possible to extend a bit more...
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-08-08 12:54:46
The test is extended until August 20 as some listeners are sending their partial results in these days. New participants are required.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: albertoh on 2011-08-08 13:30:51
Hi. I am new in this forum.

I'd like to know where I would send the results of the listening test.

Thank you in advance.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: lvqcl on 2011-08-08 15:02:28
I'd like to know where I would send the results of the listening test.

http://listening-tests.hydrogenaudio.org/i...96-a/readme.txt (http://listening-tests.hydrogenaudio.org/igorc/aac-96-a/readme.txt)

Quote
7. After you finish the test, save the results, (7-)ZIP, RAR or ACE them together and mail the file to listening.tests.ha@gmail.com
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: albertoh on 2011-08-08 16:44:46
Thank you
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: dispatcher7007 on 2011-08-13 11:57:30
Hi, this is the first listening test i participate in, and I must admit, I find it very hard. Most of the times I cant even tell the difference for sure, in which times i prefer to do nothing. Except for this, one example is obviously very bad.

And the more natural a source sounds, the easier is it to tell the difference. Half of the Samples are sythesizer/MIDI-generated or othewise highly distorted music, where i almost never can tell a difference between reference and coded.

Is it useful to submit my results, if they are most of the times 5,0 to 5,0? Isn't there a piece of organ-music? I bet I would find more significant results in that.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: Gecko on 2011-08-13 13:35:21
See post #32 in this thread for some useful links and information:
http://www.hydrogenaudio.org/forums/index....st&p=764406 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=89765&view=findpost&p=764406)

All properly attained results are meaningful, including 5/5 rankings.

One of the samples -- the low anchor -- is intentionally bad. It helps to put things into perspective.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-08-13 18:38:33
Half of the Samples are sythesizer/MIDI-generated or othewise highly distorted music, where i almost never can tell a difference between reference and coded.

The samples contain the different music genres + female and male speech. http://www.hydrogenaudio.org/forums/index....st&p=763168 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=89208&view=findpost&p=763168)
Synthesizer's  samples are reasonably presented in the test. Totally 5 samples (03, 04, 08, 10, 13)
Most of them are actually hard samples.

Is it useful to submit my results, if they are most of the times 5,0 to 5,0?

Some codecs can be transparent for some samples. So it's ok if You can't tell the difference for few samples.
But You should be able to spot artifacts for most of the other samples. You can leave the highest score (5.0) if You can't ABX some of the competitors.
The low anchor should be always ranked.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-08-15 20:41:37
Update:

We need more results for samples 12,15,16,17,18.  The results for other samples are welcome as well.
New listeners are required.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-08-20 23:04:47
The test is closed.

Thank You to ALL who have participated or supported the test. 

The test was open during 34-35 days. So the listeners have plenty time.
Now it's me who have some limited time but I will do my best to publish results as soon as possible. If it won't be this weekend then it will be during the week.
Title: Public AAC Listening Test @ ~96 kbps [July 2011]
Post by: IgorC on 2011-08-23 20:01:07
I will ask the administrators to close this topic.

The topic with results is open here http://www.hydrogenaudio.org/forums/index....showtopic=90403 (http://www.hydrogenaudio.org/forums/index.php?showtopic=90403)