Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Another Comparison test (Read 5709 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Another Comparison test

Someone posted a link to ff123's 64Kbps test results over at Arstechnica, and a few people were duking it out over there about numerous things dealing with Audio. (link here: http://arstechnica.infopop.net/OpenTopic/p...31&m=3710918725)

Anyways, during the course of the flaming/discussion, someone asked for a test similiar to r3mix.net's Analysis because they thought ff123's wasn't very scientific. So I looked into r3mix.net's test, and I did the same tests. Only this time I used Ogg Vorbis at three different qualities, MPEG4 AAC at 64Kbps, Mp3Pro at 64Kbps, and Monkey's Audio. I really wanted to use the same exact programs as ff123 did, but I couldn't get a hold of MMJB or Quicktime 6, so I did my best using PsyTEl for AAC and a standalone Thomson encoder/decoder for MP3Pro. Anyways, here are my results and test:

http://home.insightbb.com/~marshallp/vorbistest/

Oddly enough, the results are basically the same at 64Kbps as ff123's test. MP3Pro and Ogg Vorbis are a close match and AAC is far behind.

Any comments, suggestions, rants about this method and how I did it?

Another Comparison test

Reply #1 –
Quote
Originally posted by greenirft

Anyways, during the course of the flaming/discussion, someone asked for a test similiar to r3mix.net's Analysis because they thought ff123's wasn't very scientific.


In reality, it's the other way around. ff123's test was well conducted and yours is fatally flawed.

(and so are the r3mix.net ones)

Take a look at http://sjeng.sourceforge.net/audio/codecs.html to get some clues why.

--
GCP

Another Comparison test

Reply #2 –
He, he, he! There are some funny posts over at ars technica. Though, I can't quite disagree with certain person's comment about subjectivity in ff123's method of testing. I mean, it's easy enough when you can say that one encoded sample sounds transparent (indistinguishable from the original sample), whereas one sounds different/worse (from the original). Clearly you can then say that encoder A performed better on sample X than encoder B did. But if both encoded samples sounds different/worse than the original sample -- that's where the subjectivity kicks in. But given the fact that the tests are double-blind (which also makes the comment about being biased flawed), and that several people's opinions on which sounds better than what, what are the chances of several people rating one encoded sample (encoder A) as sounding better than the same sample encoded with another encoder (encoder B)? Also, when testing on several types of samples, and gathering results from several "test persons", doesn't that make it fair to reach a conclusion that one encoder might be better than the other (at a certain setting/bitrate)?

Another Comparison test

Reply #3 –
*sigh*
it was a nice thought, but in the future I suggest you leave this job to the author of the project.
You didn't mention any background or provide enough information.
let ff do this

Another Comparison test

Reply #4 –
People there didn't know at all what the test was about.

A) They keep saying that there was no reference sample to compare with. Huh? have they missed the point at all?

B) They keep saying that a studio equipment is needed for such a tests. If an OGG file sounds better than AAC/WMA/whatever else on a crappy 9€ soundcard, it will NOT sound worse in a 500€ soundcard+2000€ Speakers!!! Somebody whould say, common sense.

C) Some even say that he is trying to say that AAC is worse than OGG. This is not the case at all. We know that the test was done because OGG 1.0 was much better at -q0 than before, and we wanted to do a side by side test with other codecs worth at this bitrate.

D) You might remember that test that with WMA, they substracted the encoded signal from the original, and showed that WMA was "closer" to the original". That would be an objective test with no sense at all, as we know. Closer bit by bit does not mean closer in the frequency range, which is what matters.

Can someone post something like this in there? I don't want to register there just for this.

Another Comparison test

Reply #5 –
Quote
Originally posted by MTRH
*sigh*
it was a nice thought, but in the future I suggest you leave this job to the author of the project.
You didn't mention any background or provide enough information.
let ff do this


if you notice the r3mix.net "analysis" article, I tried to provide more than r3mix did. It wasn't a test I devised or thought of, simply copied the steps for the people at arstechnica.

I wasn't questioning ff123's methods of testing, I was simply supplying what someone asked for

Quote
These surveys have absolutely zero scientific worth and are non-reproducable. If you want to read an article that uses good methodology and testing prodecures, go to http://www.r3mix.net and click on the analysis button.

Another Comparison test

Reply #6 –
Quote:
Quote
These surveys have absolutely zero scientific worth and are non-reproducable. If you want to read an article that uses good methodology and testing prodecures, go to http://www.r3mix.net and click on the analysis button.


Ahahaha.....

Oh man...

Whoever posted this is in serious need of a reality check.

Another Comparison test

Reply #7 –
sigh

going in...

Edit:  check that:  got to JohnV's post.  Thanks.

ff123

Another Comparison test

Reply #8 –
Quote
Originally posted by greenirft
if you notice the r3mix.net "analysis" article, I tried to provide more than r3mix did. It wasn't a test I devised or thought of, simply copied the steps for the people at arstechnica. 
R3mix.net's analysis method.. .
HAHA, I guess this is what r3mix (Roel) asked for... 

Obviously the "General Note" in his analysis section is not enough, just like everyone has said for a long time. Now r3mix is paying the price...
Juha Laaksonheimo

Another Comparison test

Reply #9 –
Quote
Originally posted by greenirft
Oddly enough, the results are basically the same at 64Kbps as ff123's test. MP3Pro and Ogg Vorbis are a close match and AAC is far behind. 

Any comments, suggestions, rants about this method and how I did it?
Your method is totally flawed. The reason why it gives similar results to the listening test is that the bitrate is very low and because of that the distortions reasonably high.
AAC is so bad, that it has to show also in the FFT graphs. When analysing moderate to high bitrate, graph method doesn't show much, and in any case, is not to be trusted.
Juha Laaksonheimo

Another Comparison test

Reply #10 –
I decided to make a criticism and response page.  I had not anticipated so many negative reactions to this test.

http://ff123.net/64test/criticism.html

ff123

Another Comparison test

Reply #11 –
You know, too many clueless people out there...

Another Comparison test

Reply #12 –
Quote
Originally posted by KikeG
You know, too many clueless people out there...


Objective comparison is very seductive, because it doesn't leave room for human subjectivity, whereas subjective comparison must not be scientific because it actually relies on subjective responses, which can vary wildly.

"Any dolt with any knowledge of objectivity and the scientific method can see that it's [the 64 kbit/s listening test] completely invalid."  -- a physics major

It doesn't help that a certain popular website out there (r3mix.net) promotes the graphic analysis method even though Roel knows better.  He has all but admitted in the past (and in fine print on this site) that listening tests are the final arbiter, but deliberately chooses to push the graphics because it's easier to reach newbies that way and "show" them the horrors of Xing.

ff123

Another Comparison test

Reply #13 –
Reading the follow ups, to JohnVs and ff123s replies, I would assume they're suffering under FAS.

I don't know what's so hard understand about the idea that people try to find the best sounding codec, and not the best looking on a spectrum analyzer?

Another Comparison test

Reply #14 –
Found another one for Garf..

This one is again a true classic. Quality evaluated using CoolEdit spectral view and rated based on high freq response..

http://www.serverkrash.nu/audioformat/
Juha Laaksonheimo

Another Comparison test

Reply #15 –
from http://www.serverkrash.nu/audioformat/
Quote
Personally I will be doing my archiving with the fifth (-q 5) quality setting, as I personally think this gave the best quality / size ratio for my need and purpose.

Same here
Chaintech AV-710

Another Comparison test

Reply #16 –
Quote
Originally posted by bubka
Same here


I really hope you didn't take this decision only by reading that page.

Another Comparison test

Reply #17 –
what is your opinion on what i should use? or with ogg, what setting?  And if its ACC or MPC i dont care.  thanks
Chaintech AV-710

Another Comparison test

Reply #18 –
Quote
Originally posted by bubka
what is your opinion on what i should use? or with ogg, what setting?  And if its ACC or MPC i dont care.  thanks


My opinion is always the same: Test the formats by yourself and decide which one pleases you the most.

My favorite is AAC, but that's just a personal opinion.

After you choose a format, test quality settings that generate files around 170 - 200kbps. Taking in consideration these 3 formats, that should be enough for almost all music out there.

Hope that helped.

Regards;

Roberto.
 

Another Comparison test

Reply #19 –
[Sigh], this frequently seems to be the case when discussing audio compression on tech-related sites.  These people don't understand the first thing about audio compression, but since they're computer-people they pretend they do.  So they use methods that a tech-person would use (like comparing frequency histograms) that are completely unsuited to lossy audio compression.

The discussions on slashdot on audio compression are generally pretty disturbingly inaccurate as well.