Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps) (Read 18915 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Hello guys,

I did a 'pseudo-scientific' test to check the bitrate vs size vs quality performance of AAC. AAC section doesn't get much love so I'll post it here.

I'm very tired right now so I'll keep it short, I'll add more information as needed.

Methodology:

I ripped one (1) second of audio from a CD (random song, The Offspring - The Future Is Now) to uncompressed WAV (16-bit, 44100hz, stereo) using GoldWave CD-Reader tool.

Then I encoded the resulting WAV file to...

Winamp FhG via Foobar2000
AAC CBR@512kbps
AAC CBR@384kbps
AAC CBR@320kbps
AAC VBR@6
AAC CBR@160kbps

and saved those as WAV.

Using GoldWave, I mixed (inverted mix) the files with the original uncompressed WAV. The resulting output would be the difference between the uncompressed file and the encoded file (whatever info is lost or changed during encoding; the bigger the difference, the bigger the degradation/deviation). These files were saved as .txt files, they only include this "difference".

(Encoded files were slightly longer, I fixed it before mixing).

Using Excel, I added this "difference" information to columns, each value (row) represents a sample of the "difference" audio file (88,200 samples in one (1) second of audio).

These values were then added (absolute values so they don't cancel each other out). Basically, the resulting value (sum of all absolute values) is a number that represents how different the encoded file is compared to the original uncompressed WAV. The larger the number, the larger the deviation from the WAV file, and thus, worse quality.

Please note that a larger "Difference" value doesn't *necessarily* mean worse quality, technically yes, but... let's try this analogy:
When someone changes the settings in an equalizer, the output is very different from the original audio (more bass or more trebble), but for some people it might sound 'better', even though the output is very, very different from the original audio (with equalizer off). So, this test considers the deviation as a degradation in sound quality.

Ok now the good stuff...





Basically, the lower the bitrate the lower the performance of the codec. From the numbers, it looks like that below VBR6, the codec performance decreases dramatically.

You can see how the mp3 does compared to AAC.

From 160kbps to VBR6, performance gain is about 145%, from there to 320kbps is 14.7%, and 4.23% to 320kbps.



I believe CBR@256 or VBR6 (even better) is the sweet spot for me: transparence to my ears, the best performance gain at a given bitrate.

I hope you find this test helpful. I will be adding the values corresponding to lower bitrates in the next days.
GusNV

Re: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Reply #1 –
Quote
Using GoldWave, I mixed (inverted mix) the files with the original uncompressed WAV. The resulting output would be the difference between the uncompressed file and the encoded file (whatever info is lost or changed during encoding; the bigger the difference, the bigger the degradation/deviation).
The sound of the difference isn't the same as the difference in the sound.

Yes, if the files are identical there will be silence when you subtract so if you get silence the files are identical.   But the reverse is not true...   You can get a big (subtraction) difference when there is no difference in sound.

MP3 introduces a time shift and phase shifts.   (I'm not sure about AAC.)

For example If you add a few milliseconds to the beginning of the file, there is no difference in sound but subtraction will create a "loud" comb-filtered difference file.

Or, think about what happens if invert the file before subtracting.   Again, no difference in sound but a difference-file that's acrually a sum-file with twice the amplitude (and probably clipped).

Re: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Reply #2 –
Unfortunately, these criteria are mostly meaningless.
 

Re: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Reply #3 –
Quote from: DVDdoug on
Quote
Using GoldWave, I mixed (inverted mix) the files with the original uncompressed WAV. The resulting output would be the difference between the uncompressed file and the encoded file (whatever info is lost or changed during encoding; the bigger the difference, the bigger the degradation/deviation).
The sound of the difference isn't the same as the difference in the sound.

Yes, if the files are identical there will be silence when you subtract so if you get silence the files are identical.   But the reverse is not true...   You can get a big (subtraction) difference when there is no difference in sound.

MP3 introduces a time shift and phase shifts.   (I'm not sure about AAC.)

For example If you add a few milliseconds to the beginning of the file, there is no difference in sound but subtraction will create a "loud" comb-filtered difference file.

Or, think about what happens if invert the file before subtracting.   Again, no difference in sound but a difference-file that's acrually a sum-file with twice the amplitude (and probably clipped).

Yes, I agree that the sound of the difference isn't the same as the difference in the sound, and that this difference doesn't contain the exact info lost during encoding (some things are even added -aac adds a few ms of noise(?) at the beginning and the end of the file).

Added *noise* (removed before mixing, resulting time identical to WAV):


When I mixed the files I considered all these variables, there was no phase shift (otherwise the files wouldn't cancel each other out -I did even try to mix inverted files with phase shifts, they didn't cancel).

Beginning and end of files (WAV vs CBR@160):





This test helped me compare the performance of a codec, probably not the best way to do it, but I now feel that I'm using the right bitrate and codec based on my results and on my ears.

I can safely say that the resulting Lame MP3@320kbps file deviates more from the original than the (FhG) AAC at the same bitrate. At that bitrate even with a nice set of headphones I can't tell the difference, but I know why I should still go with AAC over MP3.

This is the updated chart:



Includes excel file.
GusNV

Re: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Reply #4 –
Quote
This test helped me compare the performance of a codec, probably not the best way to do it, but I now feel that I'm using the right bitrate and codec based on my results and on my ears.

This test only tells you that higher bitrate files have less compression than lower bitrate files. This is true, but did you really need to run all those pointless calculations to find out that higher bitrate have less quantization noise?

The question most people are interested in, which actually sounds better, cannot be answered using this method.

Re: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Reply #5 –
Quote
The question most people are interested in, which actually sounds better, cannot be answered using this method.

I do not rely on this test alone, I trust my ears. But as an exercise I learned new things, and I believe the results are consistent with my definition of "quality". The higher the "Difference" value, the worse the quality, that's a fact, because the difference between the original and the encoded files is greater (assuming same volume, etc.).

I can't use this test alone to determine which codec/bitrate sounds better, but it gives me a rough idea of how a given rate compares to another, or how two different codecs at the same bitrate compare to each other.
GusNV

Re: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Reply #6 –
Quote from: gusnv on
how two different codecs at the same bitrate compare to each other.
That's what really wrong.

Re: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Reply #7 –
Quote from: gusnv on
The higher the "Difference" value, the worse the quality, that's a fact, because the difference between the original and the encoded files is greater (assuming same volume, etc.).

This is wrong too. 

Re: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Reply #8 –
Quote from: gusnv on
[...] as an exercise I learned new things, and I believe the results are consistent with my definition of "quality". The higher the "Difference" value, the worse the quality, that's a fact, because the difference between the original and the encoded files is greater (assuming same volume, etc.).

I can't use this test alone to determine which codec/bitrate sounds better, but it gives me a rough idea of how a given rate compares to another, or how two different codecs at the same bitrate compare to each other.

A simple shift by a couple hundred samples would yield a vast "difference" between the two input signals - even though they are identical. You could use an auto correlation analysis on it, to remove the time-domain shift, but comparing the "difference function" between the two would give you nothing. At the same time: both inputs are identical, and would sound exactly the same, one would only have a set of zero-value samples to either side of the samples which would be of question.

You could for instance FFT both signals and look at the result of that, to have a rough estimate if they'd sound roughly similar. However that all is also kindof a fool's errand: at the end of the day, you'd be estimating how either signal would sound to human ears.
In essence what you want is an estimation function that models human hearing and is able to create a result set for both input signals. So the actual effort (which would have some merit) would be to model the perception of sound mathematically (as is done in psychoacoustic modelling). The only efforts in that area that I'm currently aware of, is to use machine learning algorithms to create a modeled average perceived audio quality.

Using a mere difference function is kindof starting on the wrong foot here.

What you're actually comparing, is how encoders re-arrange noise mixed into the signal. It'd be possible to create a codec which is quite horrible in perceived audio quality, with a relatively low arithmetic difference per sample in the time domain.

Re: AAC bitrate quality comparison (512/384/320/VBR6/160) (bonus MP3@320kbps)

Reply #9 –
Thank you everyone for your input! I guess I'll go to the more advanced sections of this forum so I can get a more clear understanding of how encoders work.
GusNV