Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: AAC at 128kbps test - FINISHED (Read 63828 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

AAC at 128kbps test - FINISHED

Reply #50
Quote
Thales uses AAC encoder from FhG and SBR encoder from CT.


That's pretty cool stuff.  One probably costs more then what I'll make in an entire year, but its damn cool stuff.

Too bad we can't encode with those codecs.  It would be interesting to see where "satellite radio" falls in the group.

- Agent 86

AAC at 128kbps test - FINISHED

Reply #51
Quote
Does ANOVA take into consideration the varying scales that people use when they grade the test samples? For example someone who thinks that the goal of the codec is to achive total transparancy may grade everything between 1 and 3 when someone else who doesn't care that much would give grades between 4 and 5 to the same set.

So could someone very briefly explain how ANOVA works (or post some good links)? Which types of data sets is it possible to use ANOVA on?

This particular type of ANOVA does take into account the varying scales.  That's because I treat each listener as a "block."  Here is a general description of blocked ANOVA:

http://www.basic.nwu.edu/statguidefiles/on...ay_b_anova.html

ff123

AAC at 128kbps test - FINISHED

Reply #52
I updated the overall graph, following a suggestion by ff123 and Garf. Now, there's no more results being "eyeballed"

As you will notice, Faac got a little worse in this updated overall, and Nero is now almost perfectly tied to Psytel AACenc. Sorenson also got closer to QuickTime (or maybe QuickTime got ranked lower).

For those feeling nostalgic of the old plot:
http://rarewares.hydrogenaudio.org/test/aa...t/plot10old.png

Regards;

Roberto.

AAC at 128kbps test - FINISHED

Reply #53
Thanks again for organising this most useful test Roberto, and thanks to the others who helped with the setup, and post analysis.

Based on these results, which AAC encoder will be the "champion" to go up against the next round of contenders, namely vorbis, mpc and mp3?

It concerns me a little, that a particular AAC encoder will be chosen as the representative, but in reality, it may not be the most logical choice for the average user. For example, I don't think QT will be used as often as some of the others by Mr/Ms Joe Windows-User in its current state, due to the crappy "one file at a time" user interface. Even Sorenson may be a great encoder, but its cost may keep people away? 

I for one will always favour an encoder that allows me to access it from the command line, and feed it from a range of front ends of my choosing depending on what I need at the time, ie foobar, EAC, CDex, Frontah, home made batch files, "insert you favourite media player/front end/ripping/compression package here"... 

If all the command line optioned encoders sound like complete crap, I would be forced to use one of the superior encoders as the difference will be worth it, but I suspect from these results, Sorenson, Quicktime, Nero and Psytel all have some potential to compete against the others at ~ 128 kbit, especially if they are allowed to go VBR. If you choose QT to be the AAC representative, I will still be keen to be involved, and see the results, but if it is close to the other non-AAC formats in quality, I'll have this nagging question in the back of my head. "I wonder how Psytel would have gone..."

I appreciate that these listening tests can not accomodate everyone's wishes, and I suppose ultimately you need to pick the best performing encoder, regardless of its expense or convenience...

Just my 2 Australian cents.

Den.

AAC at 128kbps test - FINISHED

Reply #54
Quote
Based on these results, which AAC encoder will be the "champion" to go up against the next round of contenders, namely vorbis, mpc and mp3?

Add WMA to that list.

And atrac3 is another possibility, but that's only a speculation right now. I don't even plan to take this decision now, I'd rather go take a looong nap after I finish dealing with the test results. Preferably at the beach.

After I'm sufficiently rested, I'll start thinking about codecs, bitrates, settings, etc.

Anyway, ATM there are two options for the AAC: Nero and QuickTime

QuickTime because it kicks ass. Nero because Ivan reportedly has been improving his codec, spurred on by this test results. If he can deliver an improved codec on time, it might be considered for the extension test.

Quote
It concerns me a little, that a particular AAC encoder will be chosen as the representative, but in reality, it may not be the most logical choice for the average user. For example, I don't think QT will be used as often as some of the others by Mr/Ms Joe Windows-User in its current state, due to the crappy "one file at a time" user interface. Even Sorenson may be a great encoder, but its cost may keep people away? 


Right, I don't think people will be letting go 400 hard earned smackers just so that they can encode their audio tunes to AAC.

Besides, I think Sorenson suffers from the same problem as QuickTime, of "one file at a time". Gotta check that later.

Quote
I for one will always favour an encoder that allows me to access it from the command line, and feed it from a range of front ends of my choosing depending on what I need at the time, ie foobar, EAC, CDex, Frontah, home made batch files, "insert you favourite media player/front end/ripping/compression package here"...


I understand you are talking about AACenc, but including AACenc wouldn't be a good idea. From the development point of view, AACenc is now dead, since it can't be developed furthermore. Nero has the downside of being GUI-only, but at least it's being actively developed, more so these days.

Quote
I appreciate that these listening tests can not accomodate everyone's wishes, and I suppose ultimately you need to pick the best performing encoder, regardless of its expense or convenience...


Well, by far the most important aspect is the quality, but price and convenience also have their importance. For instance, I would hardly pick Sorenson for the extension test even if it had won, because it's an encoder few people have access to, it's damn expensive and you can only encode one file at a time.

QuickTime, on the other hand, is cheaper, widely available and reportedly you don't even need to register it if you access the encoding component directly through another program. (maybe a CLI program  )

Thanks a lot for your remarks.

Regards;

Roberto.

AAC at 128kbps test - FINISHED

Reply #55
Quote
For instance, I would hardly pick Sorenson for the extension test even if it had won, because it's an encoder few people have access to, it's damn expensive and you can only encode one file at a time.

If you use Squeeze 3.5, you can "Open Watch Folder" to do a batch encode. It's very odd name for batch encoding until you realize that Squeeze will remain open and "watch" for any new WAV files you drop into the Watch Folder and immediately start encoding them according to your previous settings. It's weird, but kinda cool in the right situation. Squeeze will also separate the WAV files and MP4 files into separate sub-folders called "Work" and "Done", which may or may not drive you insane.

AAC at 128kbps test - FINISHED

Reply #56
We can consider the possibility, for the next test, to include two AAC codecs :

- VBR one, in fast development (don't forget HE-AAC), easy to use, pretty GUI (I can transcode my MPC to MP4 with Nero)
- Winner of this test : CBR QT 128, for people that want a very good quality, without consideration of time encoding, GUI annoyance, etc...


Other codecs may be mpc, vorbis (official), wma (standard, pro ?), mp3 (lame). Maybe ATRAC3...
But that's maybe too much challengers (12 samples x 6-7 codecs = 72...84 files).
(don't forget that you need a CLI decoder, if you want to reduce the package size with a .bat file : is there any WMA9 or ATRAC3 separate decoder available ?)


Problem is : what's the purpose of this test ?
Is it a practical one : what is the best audio format for portable use ? MPC is actually useless here...
Is it a theorical one : what is the best audio format at 128 kbps, regardless of any portable available, or GUI, or encoding speed ?
Is it a mix of the two ?

We need to clearly answer to these questions in order to make the codec/setting choices.

AAC at 128kbps test - FINISHED

Reply #57
Quote
We can consider the possibility, for the next test, to include two AAC codecs :

- VBR one, in fast development (don't forget HE-AAC), easy to use, pretty GUI (I can transcode my MPC to MP4 with Nero)
- Winner of this test : CBR QT 128, for people that want a very good quality, without consideration of time encoding, GUI annoyance, etc...


Other codecs may be mpc, vorbis (official), wma (standard, pro ?), mp3 (lame). Maybe ATRAC3...
But that's maybe too much challengers (12 samples x 6-7 codecs = 72...84 files).
(don't forget that you need a CLI decoder, if you want to reduce the package size with a .bat file : is there any WMA9 or ATRAC3 separate decoder available ?)


Problem is : what's the purpose of this test ?
Is it a practical one : what is the best audio format for portable use ? MPC is actually useless here...
Is it a theorical one : what is the best audio format at 128 kbps, regardless of any portable available, or GUI, or encoding speed ?
Is it a mix of the two ?

We need to clearly answer to these questions in order to make the codec/setting choices.

I was talking with Roberto earlier today exactly about this. We have a plan how to proceed, but I don't know when Roberto wants to make it public.

I'm also wondering if MPC is necessary for low-medium (around 128kbps) vbr test. But Roberto claims that Musepack users will be very upset if it's not included. I'm not so sure about this, because people already know that Musepack is very good at higher bitrates, and I'm not sure if MPC users are even interested in the bitrates this low.
Juha Laaksonheimo

AAC at 128kbps test - FINISHED

Reply #58
Quote
I'm also wondering if MPC is necessary for low-medium (around 128kbps) vbr test. But Roberto claims that Musepack users will be very upset if it's not included. I'm not so sure about this, because people already know that Musepack is very good at higher bitrates, and I'm not sure if MPC users are even interested in the bitrates this low.


Most current Musepack users (audiophiles/HA-whores ) are definetly not interested in anything below --standard, but it has been mentioned before that it will probably do a lot better than people would expect it to at around 128kbps, so this might bring some clarity to the situation and possibly make Musepack an option for a new group of users.

dev0
"To understand me, you'll have to swallow a world." Or maybe your words.

AAC at 128kbps test - FINISHED

Reply #59
Excluding MPC makes no sense - it's definetely a contender at 128kbps.

AAC at 128kbps test - FINISHED

Reply #60
Quote
Excluding MPC makes no sense - it's definetely a contender at 128kbps.

I'm partially agree with you. I did some tests long time ago, and I found mpc --radio to be the most pleasant format to my ears : PsyTEL -streaming was second, followed by LAME ABR. I didn't like Vorbis (rank #5 if I remember correctly).
Nevertheless, if the new test has for purpose PORTABLE abilities, musepack is a contestable choice... It may be a better choice to include ATRAC3 (MiniDisc portable) or WMA9, even if theses competitor are worse than mpc.

There are some problems with mpc too. Accoding to my experience, the amplitude of musepack is the higher I saw on 'common' music. A piano Goldberg Variations at --radio is ~100 kbps (Gould 81 for exemple) ; on harpsichord, it's ~170-180 kbps. Some isolated tracks are ~90 kbps with piano, and ~200 with harpsichord. Difference isn't so big with other codecs : it's easier to reach an approx. 130 kbps, or fit them to an exact 128 kbps without losing too much in quality.
Here are the bitrate for each samples of the previous test :
http://membres.lycos.fr/guruboolez/AUDIO/a...pack_report.htm
(--radio, but don't remember if it's 1.14 or 1.15r)

AAC at 128kbps test - FINISHED

Reply #61
Quote
Excluding MPC makes no sense - it's definetely a contender at 128kbps.

Do you know anybody who uses MPC at these bitrates? Do you think thousands of new people will start using MPC even if MPC wins that test? Imo MPC is clearly a niche format, not even targeted at masses. The (sad) fact is MPC does not have mass appeal, starting from its website...

I think many people here already know that MPC is surprisingly good at average 128kbps. Not sure what would it serve to include it, simply because even though MPC is very good, it is generally unknown format used by small niche group and lacks the appeal ouside hifi quality high bitrate use.

Sure, maybe MPC gets few dozen of new users who will start using it at higher bitrates, but I'm not sure if it makes any difference regarding MPC's position as a format.

I don't have anything against the inclusion of MPC, on the contrary, but if we have to chop some codecs, imo we should start from those codecs which have overall the least interestingness globally.
Juha Laaksonheimo

AAC at 128kbps test - FINISHED

Reply #62
I don't get it. Why should we include Vorbis then? What's the point of the test?

AAC at 128kbps test - FINISHED

Reply #63
Quote
I don't get it. Why should we include Vorbis then? What's the point of the test?

Xiph tries to make Vorbis appeal to masses. It's relatively good at all bitrates, it has multichannel encoding, it has directshow filters, it is used in movie encoding, it has an order of magnitude bigger userbase than MPC, Vorbis has had quite a lot of media coverage (MPC once introduced in c't), there are several broadcasters streaming vorbis etc. etc.
In short, Vorbis is trying to be in the same league with the big formats, when MPC is not even trying and it isn't in the same league, except quality wise. I'm not sure if the knowledge that MPC is good at 128kbps is very relevant for the masses outside HA, and most HA people interested in MPC know that MPC is good at near 128kbps already, so why include MPC?
Juha Laaksonheimo

AAC at 128kbps test - FINISHED

Reply #64
Quote
and most HA people interested in MPC know that MPC is good at near 128kbps already, so why include MPC?

Well, I for one would like to know how good or bad it is.

But you did not really answer my question, what is the use of the second test? Is it to compare AAC to it's direct competitors, and see who has the best quality? In that case, MPC would not be a competitor indeed.

AAC at 128kbps test - FINISHED

Reply #65
Quote
But you did not really answer my question, what is the use of the second test? Is it to compare AAC to it's direct competitors, and see who has the best quality? In that case, MPC would not be a competitor indeed.

Umm, I don't think it's the idea that the cross-format test is AAC vs others. Rather I'd see it as comparison of the big players against each others - the formats which are the most widely used in PC, so that the test has as much significance as possible for as many people as possible.

I'm not deciding anything here. Besides, I sure will be talking a lot about this with Roberto. I just think we should consider carefully which formats will be included.
Juha Laaksonheimo

AAC at 128kbps test - FINISHED

Reply #66
Just to chime in with Garf... MPC is used by exactly as many portable devices as Vorbis. There is only those who say they are working on it and that it will be released later - that goes for Vorbis as well as MPC. So from that is definately not an argument for excluding one from the test and keeping the other.

If the test is setup to answer which is the best choice for portables both WMA8 and ATRAC3 have to be included. Neither Vorbis nor MPC has anything to do in such test.

AAC at 128kbps test - FINISHED

Reply #67
Quote
Just to chime in with Garf... MPC is used by exactly as many portable devices as Vorbis. There is only those who say they are working on it and that it will be released later - that goes for Vorbis as well as MPC. So from that is definately not an argument for excluding one from the test and keeping the other.

If the test is setup to answer which is the best choice for portables both WMA8 and ATRAC3 have to be included. Neither Vorbis nor MPC has anything to do in such test.

As far as I can see I didn't say anything about portables at all.
Portable usage is one aspect which should be considered also though.
Juha Laaksonheimo

AAC at 128kbps test - FINISHED

Reply #68
Quote
As far as I can see I didn't say anything about portables at all.
Portable usage is one point of view also.

guruboolez did.

In case it will be narrowed down to portable use you could afford to have two AAC codecs in the test as well.
- QT
- Nero VBR
- LAME ABR
- WMA
- ATRAC3

More than five codecs is too much IMO.

AAC at 128kbps test - FINISHED

Reply #69
The objectives of this test are not that clear, at least to me anyway. 

Despite being an ATRAC3 user (dodges rotten tomatoes)    , I'm not sure if it belongs here. I hate to generalise, but most if not all ATRAC3 users do so because they have no other choice, ie they own an ATRAC3 only Sony/Minidisc product. OK, so Sony now have these Discmans with ATRAC and MP3, but I'd bet my left one    that savvy owners would use MP3/LAME alt presets. You simply have more control over your encode...

I would expect in the longer term that ATRAC will be left to die when Minidisc eventually goes away...

I suggest we drop ATRAC3 from the list, unless you really need another codec as a control, or comparison for the test.

I think WMA, Vorbis, Musepack and the omnipresent MP3 should stay in. In fact LAME and FHG should probably be tested as some here swear (substantiated or otherwise) about FHG being better at 128 kbits than LAME. They all offer creditable, albeit sometimes unused performance at this bitrate, even if they lack of portability. While Vorbis and MPC have similar, virtually non-existant portable support, it will probably come in the future, even if only on PocketPCs, etc (Vorbis is there now.)

With Musepack in particular, I came away pleasantly surprised with mpc -radio during recent tests. It isn't regularly used or recommended by HA, but that could be from habit rather than the facts. (In my tests, I actually preferred it over Vorbis...)

Just another 2 Australian cents wirth (which currently buys a lot more than it did 12 months ago...)  B)


Den.

AAC at 128kbps test - FINISHED

Reply #70
Wow. I wake up, and there's almost a format war going on.

Well, people, first, I'd like to repeat that I'm not willing to make decisions ATM. We have a full month ahead until the next test starts, it's now time to chill out.  B)

Some misconceptions that happened here:

1) Where the heck did you guys came from that this test is intened to address formats with portable support? I never wrote it, either at posts or at the test pages. The main purpose of these tests is verify what format performs best at quality. And only that. Other issues like usability, popularity and availability might be taken in consideration, but quality weights in about 90% in my decisions. Only when two formats/codecs are tied in quality that I will consider other features.

2) I definitely don't plan to allow a format to be represented twice in the extension test (I.E, QT and Nero). I remember how ff123 suffered from his decision of representing vorbis twice on his 64 test, and I don't plan to go the same way.

3) atrac3 is only speculation ATM. There'll be lots of things to consider.

4) It's likelier that MPC will be at the test instead of atrac3

5) No Lame vs. FhG MP3 at this time. I decided that this can be handled in a separate test.

Now, the idea JohnV and I had was to conduce a fast listening test (5 days, 8 samples, 2/3 codecs) at the beginning of July, QuickTime 128 vs. Nero -streaming (and maybe a 3rd codec kept in secrecy ATM). If Nero wins, it'll be used on the extension test. If QT wins, it'll be used and God help us all.

Again, it's only an idea. It's not definitive

And this is my planned schedule:
June 28th to July 3rd: The quick AAC test
July 16th to July 27th: The extension test

And it's only a plan. Don't go planning your schedule based on that, and don't start complaining about the dates now. You'll have time to do that later.

As a bottom line, the formats that are already guaranteed at this test are Vorbis (official CVS version), AAC (QT or Nero), MP3 and VQF. I mean, WMA v9.

Regards;

Roberto.

AAC at 128kbps test - FINISHED

Reply #71
What about MP3Pro ??

AAC at 128kbps test - FINISHED

Reply #72
There isn't much point testing mp3pro at 128kpbs...

AAC at 128kbps test - FINISHED

Reply #73
I'd like to better understand the purpose of this test, too. I know Roberto said "the purpose is to find what encoder performs best encoding AAC at 128kbps", but that still leaves me with some questions.

CBR was chosen because some codecs are incapable of making VBR audio, and a level playing field was desired, correct? Well, this isn't a friendly, sporting match between AAC implementations, this is a codec war! Let every codec employ its best tools for the job, I say, and that would mean using VBR wherever possible. I know there was a discussion back with the 64kbps test about setting the Vorbis q-value so that the samples' bitrates averaged to 64kbps, and I think ff123's conclusion was that q0 should be used simply because it's the most likely setting for a typical user to choose. But that decision was not a quality-based one, and since this test is all about quality, the question should be re-visited. Or maybe you've already dealt with the issue by planning a quick AAC test at the end of the month. In that case... never mind.

Another question: I don't recall reading why 128kbps was chosen. Is it just a convenient number chosen for historical reasons? Are the codecs optimized for this specific bitrate? Different bitrates can be used for streaming, file distribution, digital radio, portable playback, and archival. Is 128kbps considered to be the best compromise for all these applications? Or are people just more likely to pay attention to the results of a 128kbps test?

Roberto, I'm not trying to discount your listening test. I still think it was (and will be) very well-designed. It has already given us valuable information, and I know it will continue to do so.

AAC at 128kbps test - FINISHED

Reply #74
MP3pro isn't considered good above 96kbps. The SBR codec starts degrading the audio quality more than mp3 alone would do.