Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: AAC at 128kbps v2 listening test - FINISHED (Read 66088 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

AAC at 128kbps v2 listening test - FINISHED

The AAC at 128kbps listening test is finished at last.

The summary is: iTunes and Nero are tied at first place, although iTunes is quite above Nero in the final ranking. And Compaact!, Faac and Real are tied at second place.

The results:
http://www.rjamorim.com/test/aac128v2/results.html

For those in a hurry:


Thank-you very much for everyone that participated.

AAC at 128kbps v2 listening test - FINISHED

Reply #1 –
BTW, the obvious outcome of this is: iTunes will be featured at the 128kbps multiformat test, that shall start next month.

AAC at 128kbps v2 listening test - FINISHED

Reply #2 –
And just as I expected FAAC did a lot better job than most people thought it would. Props to Knik for his awesome work.
And, of course, respect and thanks to rjamorim for his amazing efforts.
"To understand me, you'll have to swallow a world." Or maybe your words.

AAC at 128kbps v2 listening test - FINISHED

Reply #3 –
thanks to everyone who worked on this.  from what i've read a lot of hard work and listening was done.

AAC at 128kbps v2 listening test - FINISHED

Reply #4 –
What did the sample numbers equate to? (e.g. sample_3 = Real, sample_2 = iTunes... etc.)

AAC at 128kbps v2 listening test - FINISHED

Reply #5 –
1 = Nero
2 = Real
3 = Faac
4 = Compaact!
5 = iTunes

Tomorrow I'll upload the decryption key to the site.

AAC at 128kbps v2 listening test - FINISHED

Reply #6 –
First of all, thanks for all your hard work on conducting this test, Roberto.    And thanks to everyone who participated.

I tried 4 times to do the test, but the medication I've been taking has hopelessly affected my hearing sensitivity, and I wasn't able to differentiate anything. 

About the results, I'm no AAC expert, and I can only go on what I've heard here over the past several months.  Based on that, I'm not surprised at the outcome of QT or Nero, but FAAC surprised me a little, and Real surprised me quite a bit to be ranked as highly as it was.  I guess I was only able to remember the reputation RA had over the past couple of years, so I wasn't expecting Real's AAC encoder to do so well.  (Although, finding out since that it's Coding Technologies who created it, it's not so surprising after all.)


Edit:

@Roberto:  Can you post the actual average bitrates for each codec, for reference?

AAC at 128kbps v2 listening test - FINISHED

Reply #7 –
Quote
Edit:

@Roberto:  Can you post the actual average bitrates for each codec, for reference?

Yes. Tomorrow.

I'll get some sleep now

AAC at 128kbps v2 listening test - FINISHED

Reply #8 –
IMHO, iTunes won. Though not rated higher than Nero with 95% confidence, the difference does seem significant, with iTunes clearly getting first place on 4 of the samples, slightly edging out Nero on 6 others, and being slightly beaten byNero on only 2 samples. These results also imply that iTunes is more efficient, seeing as it stayed at a constant 128 kbps, while Nero went above 140 kbps on some occasions. All in all, I think iTunes should be the recommended AAC encoder if one hopes to obtain the highest quality on average .
Happiness - The agreeable sensation of contemplating the misery of others.

AAC at 128kbps v2 listening test - FINISHED

Reply #9 –
That's true. I tried to clarify the issue in the results page, in this part:

Quote
iTunes is more or less tied to Nero. The security margin (LSD/2 overlapping with ranking) is very small, which would likely indicate iTunes is the winner.


Thank-you (and thanks to JohnV) for your comments on this subject.

AAC at 128kbps v2 listening test - FINISHED

Reply #10 –
Cool. iTunes is first placed and its free too.  Great combination
 

AAC at 128kbps v2 listening test - FINISHED

Reply #11 –
Quote
IMHO, iTunes won. Though not rated higher than Nero with 95% confidence, the difference does seem significant, with iTunes clearly getting first place on 4 of the samples, slightly edging out Nero on 6 others, and being slightly beaten byNero on only 2 samples. These results also imply that iTunes is more efficient, seeing as it stayed at a constant 128 kbps, while Nero went above 140 kbps on some occasions. All in all, I think iTunes should be the recommended AAC encoder if one hopes to obtain the highest quality on average .

I agree with this.

AAC at 128kbps v2 listening test - FINISHED

Reply #12 –
Roberto, didn't you get my results?

AAC at 128kbps v2 listening test - FINISHED

Reply #14 –
Quote
Roberto, didn't you get my results?

I guess I did. Why?

Edit: Oh, harashin found them

Now that participant name can be entered in the ABC/HR results, I guess I will change the file naming method. Instead of anon, I'll just add "participantXX" before each result file. (Except the ones that send the results files with their login in the filename)

AAC at 128kbps v2 listening test - FINISHED

Reply #15 –
Quote
IMHO, iTunes won. Though not rated higher than Nero with 95% confidence, the difference does seem significant

It's not statistically significant still. Stricly by the books there's no winner. Roberto was unsure about this because there were not so many people taking the test, LSD/2 overlap being 0.03 points according to him.
But I said that it still can't be said that it is statistically significant (something either is or isn't significant), only that iTunes is a likely winner because the LSD/2 overlap is pretty small.
Still, stricly speaking statistically there is no winner.

One has to remember that with a different sample set the results could have been different. 12 samples is pretty close to the practical limit this kind of group test can have, but 12 samples is in no way "enough" considering that many deficiencies of all codecs are not revealed here. This is somekind of average indication, but I think with different set of samples there could have caused more or less difference between the contenders.

Another thing which will likely draw attention is that Nero had with these samples higher bitrates than most other codecs, although its average with lots of samples according to spoon's test was about 131kbps. One has to remember that VBR is quality based encoding, so what you see according to the VBR principle is the average quality for a VBR codec for its overall average bitrate, but with only 12 samples (the average quality is based on very small amount of samples but what it shows is the average quality of the overall average bitrate=131kbps in Nero's case).  Compaact is an exception in certain cases: it scales the bitrate up very heavily with short blocks, which means that it gets good ranking with velvet. But this behaviour can't be mirrored directly to the general VBR behaviour and to the principle of VBR coding (constant quality principle).
Of course the constant quality doesn't in practise happen always (even if you exclude compaact's short block scaling), because VBR is hard to control, so things are quite complicated..  It's by no mean clear that if QT/iTunes gets VBR mode that it will be better than its CBR. Good VBR quality in all circumstances is hard to achieve and control at mid-low bitrates, and that's why my opinion is that ABR would be probably the best coding method for mid-low bitrate like 128kbps.
Juha Laaksonheimo

AAC at 128kbps v2 listening test - FINISHED

Reply #16 –
Anyone want to post the comments archive zipped?

AAC at 128kbps v2 listening test - FINISHED

Reply #17 –
Quote
Quote
Roberto, didn't you get my results?

I found your name in a result.
http://www.rjamorim.com/test/aac128v2/comm...2_decrypted.txt

Thank you. I looked for my result for sample 5 (Hongroise) and couldn't find it. All other results are there (with anon03) but this one is missing! 

(Maybe I ranked the reference 5 times... )

AAC at 128kbps v2 listening test - FINISHED

Reply #18 –
Quote
Anyone want to post the comments archive zipped?

Well, you can get UnRAR and RAR for MacOS X, and I think StuffIt supports it too.

I use Rar because, in solid mode, it gets up to 3 times smaller than zip.

@Continuum: I'm afraid I messed the anon ordering a little. Sorry :/

AAC at 128kbps v2 listening test - FINISHED

Reply #19 –
Quote
Quote
Anyone want to post the comments archive zipped?

Well, you can get UnRAR and RAR for MacOS X, and I think StuffIt supports it too.

Okay, I tried stuffit and gumby, they both report errors with the format

:-\

UnRAR cli app works

AAC at 128kbps v2 listening test - FINISHED

Reply #20 –
Quote
Quote
Quote
Anyone want to post the comments archive zipped?

Well, you can get UnRAR and RAR for MacOS X, and I think StuffIt supports it too.

Okay, I tried stuffit and gumby, they both report errors with the format

:-\

UnRAR cli app works

UnRarX should probably work too, and is a little nicer than using the commandline tool.

AAC at 128kbps v2 listening test - FINISHED

Reply #21 –
Some comments:

1) iTunes result is nice, but I think our progress is even nicer! In the previous test iTunes clearly beat Nero. Now it's getting quite arguable whether it's really better  Next major Nero release will be very intresting.

2) Somebody said that iTunes is more efficient because of bitrate - that's simply not true. The average bitrates of both codecs with the used settings is within +-3 kbps.

3) About interpreting the results. Is the graph correct? It is stated that the overlap is really small. But on the graph it is very large.

AAC at 128kbps v2 listening test - FINISHED

Reply #22 –
Well, thankyou to rjamorim for running the test

Was very interesting, I'm looking forward to the multi-format test now

AAC at 128kbps v2 listening test - FINISHED

Reply #23 –
Quote
2) Somebody said that iTunes is more efficient because of bitrate - that's simply not true. The average bitrates of both codecs with the used settings is within +-3 kbps.

Depends of the genre.
I've encoded ~1500 files with latest Nero build (and a VBR profile especially optimized for this test), only classical, and bitrate is close to 140 kbps. And quality have serious issues. I really hope that problems will quickly be solved. But in my opinon, current encoder have still problems with efficiency AND with quality.


Anyway, faac surprises me. I really wonder if -q115 is the best preset for this encoder. I remember that in my last test, I've used ABR, safer. Is the bitrate table ready? It wouldn't surprises me if faac obtained good results on sample with bitrate > 140, and bad one with bitrate < 120.

Good job for all AAC developers. Many people discovered that 128 kbps are close to be transparent, if not totally transparent. And thanks to Roberto, for this nice test.
Wavpack Hybrid: one encoder for all scenarios
WavPack -c4.5hx6 (44100Hz & 48000Hz) ≈ 390 kbps + correction file
WavPack -c4hx6 (96000Hz) ≈ 768 kbps + correction file
WavPack -h (SACD & DSD) ≈ 2400 kbps at 2.8224 MHz

AAC at 128kbps v2 listening test - FINISHED

Reply #24 –
Yes, it is true that some codecs are more fit to a genre than others. This does not change my point at all - on a sample of various genres, the bitrate was not abnormally high, and apparently the quality is very good, too, so the efficiency argument is false.