Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multiformat@128kbps listening test - FINISHED (Read 181337 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

Multiformat@128kbps listening test - FINISHED

Reply #250 –
no audiophile will encode in 128kbps anyway...so that was a real world test with real people that use that bitrate...nothing wrong with it and well done
--alt-presets are there for a reason! These other switches DO NOT work better than it, trust me on this.
LAME + Joint Stereo doesn't destroy 'Stereo'

Multiformat@128kbps listening test - FINISHED

Reply #251 –
Quote
Yes, there are many factors and variables that ruin the validity of the test. One being, which you named, the audio equipment being used to do the testing. Most users have shit audio equipment, therefore their results are pretty poor and innacurate. Secondly, many people, like you said, don't even know what the hell ABXing is, so you can tell by that that they don't know much about audio. Their ears, and/or listening skills probably suck. This would dramatically alter the results of the test.

Anyways, the test is better than no test. It gives us a reasonable idea, but not accurate enough, in my opinion, to really make any conclusive judgements.

I would be interested in gathering a group of good listeners that have quality equipment. I think we should have enough here. I myself have Etymotic Research ER-4s, which are basically the best you can get as far as equipment goes.
[a href="index.php?act=findpost&pid=274122"][{POST_SNAPBACK}][/a]


You keep saying "invalid" and "inaccurate."  But in what way?  As in the rankings would have produced a different order of rankings, or another winner or loser?  I don't think so.  The effect of having different setups, in my opinion, is to add random variability to the results, so that the uncertainty is greater.  But I don't think it would add a bias, i.e., change the order of the rankings by much, if any.

ff123

Multiformat@128kbps listening test - FINISHED

Reply #252 –
Quote
Secondly, many people, like you said, don't even know what the hell ABXing is, so you can tell by that that they don't know much about audio. Their ears, and/or listening skills probably suck. This would dramatically alter the results of the test.

I just meant that most of these people will not bother testing, not that they will do the test WITHOUT knowing what an ABX test is.

Multiformat@128kbps listening test - FINISHED

Reply #253 –
Quote
This test is very good and I commend rjamorim for taking his time to conduct it, however I don't believe there was nearly enough testers to validate any accurate data, and therefore come to any valid conclusions.


Quote
I would be interested in gathering a group of good listeners that have quality equipment. I think we should have enough here. I myself have Etymotic Research ER-4s, which are basically the best you can get as far as equipment goes.


First you say there are not enough participants, then you want to conduct a test with a very selective group... 

The test does not claim to be more than it is, it is not the definite anwer to which codec is "best". However, it does give an good indiciation what is good and bad (or perhaps i should say "not so good"?) for the average user. There is no more inaccuracy in the test than the error bars in the graphs suggest.

A tests which proves codec A does better than codec B on some 10.000$ piece of equiptment is completely useless for most people.
"We cannot win against obsession. They care, we don't. They win."

Multiformat@128kbps listening test - FINISHED

Reply #254 –
Quote
Quote
I would be interested in gathering a group of good listeners that have quality equipment.[a href="index.php?act=findpost&pid=274122"][{POST_SNAPBACK}][/a]


And, by doing that, you would conduce a test that would only have meaning to people with good listening and quality equipment


It would have meaning for people with poor equipment as well; the quality headroom will be bigger with the winning codec. Even if the current equipment isn't good enough to reveal flaws at 128kbps, I bet most people would always want to encode with the best format at that bit-rate, rated by people where the equipment/hearing isn't the bottleneck.

When hifi mags test speakers, they tend to use the best possible cables, amplifiers and most trained ears. That doesn't make the test useless to people with average hearing/equipment

Multiformat@128kbps listening test - FINISHED

Reply #255 –
Quote
A tests which proves codec A does better than codec B on some 10.000$ piece of equiptment is completely useless for most people.


This is exactly my point. Crappy equipment, and poor ears don't provide and accurate data. Yes, it may be real world to the majority of listeners. But, nevertheless, it does not prove anything substantial. For all we know these users were guessing. I would trust good ears and good equipment with a small majority of users, over poor ears and crappy equipment with a large majority of listeners.

Multiformat@128kbps listening test - FINISHED

Reply #256 –
Quote
For all we know these users were guessing. [a href="index.php?act=findpost&pid=274486"][{POST_SNAPBACK}][/a]


Do you even know how ABC/HR testing works?

Multiformat@128kbps listening test - FINISHED

Reply #257 –
Quote
For all we know these users were guessing.[a href="index.php?act=findpost&pid=274486"][{POST_SNAPBACK}][/a]

Nice one...

Multiformat@128kbps listening test - FINISHED

Reply #258 –
Quote
Quote
A tests which proves codec A does better than codec B on some 10.000$ piece of equiptment is completely useless for most people.


This is exactly my point. Crappy equipment, and poor ears don't provide and accurate data. Yes, it may be real world to the majority of listeners. But, nevertheless, it does not prove anything substantial. For all we know these users were guessing.


Please read up on how the test was performed. You cannot make any valid conclusion from it if you do not understand how to interpret the data.
"We cannot win against obsession. They care, we don't. They win."

Multiformat@128kbps listening test - FINISHED

Reply #259 –
Quote
Quote
For all we know these users were guessing. [a href="index.php?act=findpost&pid=274486"][{POST_SNAPBACK}][/a]


Do you even know how ABC/HR testing works?
[a href="index.php?act=findpost&pid=274491"][{POST_SNAPBACK}][/a]



@rjamorim: As I see, you couldn´t resist the oportunity. 

@jmitch: Well, that´s what a listening test´s supposed to be, isn't it? A "so good" audio equipment is not that relevant.


SoNiX 
SACD/DVD-A are useless, because...

"I am not a bat to hear the 100Khz frequency response of SACD technology :D;" - Gray Wolf

Multiformat@128kbps listening test - FINISHED

Reply #260 –
Quote
Crappy equipment, and poor ears don't provide and accurate data. [{POST_SNAPBACK}][/a]


The accuracy of the answers is given in the test results. It is 95 %.

Explanations : [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=16295&]http://www.hydrogenaudio.org/forums/index....howtopic=16295&[/url]
The ABC/HR method : http://ff123.net/abchr/abchr.html
The Anova analysis (which gives the 95 % above) : http://www.psychstat.smsu.edu/introbook/sbk27.htm

Multiformat@128kbps listening test - FINISHED

Reply #261 –
Quote
For all we know these users were guessing. I would trust good ears and good equipment with a small majority of users, over poor ears and crappy equipment with a large majority of listeners.
[a href="index.php?act=findpost&pid=274486"][{POST_SNAPBACK}][/a]


Just to satisfy yourself, why don't you take the test with your gear?  All the samples and ABX software are still there.  when you are done, just for yuks, go slumming and borrow someones "crappy" sub $100 phones and see if that makes a difference (not in how good things sound, just in how they affect your ability to abx a codec)

Let us know how it went.

edit: fixed quote markers

Multiformat@128kbps listening test - FINISHED

Reply #262 –
Quote
Crappy equipment, and poor ears don't provide and accurate data.


I participated in this test myself (when it was run). I used:

- RME Digi 96/8 PAD (professional level 96kh/24bit sound card with 1:1 bit accuracy and very nice analog measurements)
- High quality minimum capacitance shielded interconnects
- Meier Audio Pre head (very high quality solid state headphone amp)
- Sennheiser HD600, AKG K271s, Ultrasone HFI-650 and Etymotics ER4p/s headphones (Ultrasones and etymotics getting most of the listening time)
- A quiet room with a silenced computer

My ears have been tested to be flat to 11 kHz (with less than average attenuation after that up to 14kHz, the maximum that the test equipment at national hearing clinic was able to test).

I have c. 4 years in trying to get into lossy audio, perhaps 3 of that with slowly increasing listening acuity. I've gone through several training sessions with the AES "Perceptual Audio encoders - what to listen for" CD, as well as many example samples from here, mpeg development archives and previous listening tests. I've also purchased and gone through the Moulton labs "Golden Ears" hearing training cd set. In addition, I regularly audition new hifi and high end gear (and also write about hifi to a national publication). I think my hearing (both as an instrument and as a skill) is better than average.

While I'm far from being a "golden ear" I can invalidate the above argument by saying that neither my equipment or hearing is crap.

My results didn't significantly differ from that of the statistical averages in this test.

While neither my hearing or equipment are not "best in class", I think they are clearly better than average population in both cases. It is debatable how good they are, but surely not crap.

As such, I don't think the test can be in invalidated by only referring to "crappy equipment and poor listeners".

Had I magnificiently surpassed every other listener in this test by picking out artifacts other couldn't hear, I could _perhaps_ be willing to entertain the possibility of the argument being right.

But alas, I wasn't even among the best listeners in the test. Surely equipment at least wasn't a limiting factor in my case.

I must say that I was also a wee surprised that spotting artifacts in a 128kbps ABR test was so difficult. I knew it was going to be difficult, but it was even more so than I initially had imagined.

friendly regards,
halcyon

PS I really should not even have needed to reply with this defense, as ad hominem type attacks don't really need refutation. I think arguments should be evaluted based on the evidence available (and the logic of reasoning). Not on the basis who makes the argument, UNLESS there is strong proof to show that the author is not to be trusted (which in this case is non-existent). Conjecture is not enough. Arguments need evidence, not prejudice as their support.
 

Multiformat@128kbps listening test - FINISHED

Reply #263 –
Quote
When hifi mags test speakers, they tend to use the best possible cables, amplifiers and most trained ears. That doesn't make the test useless to people with average hearing/equipment
[a href="index.php?act=findpost&pid=274192"][{POST_SNAPBACK}][/a]


Tell me you're kidding.  Hifi mags may *claim* that the ancillary equipment they use is the 'best possible ' (though it's almost never been subjected to a controlled listening test) and that their listeners are 'trained'  (no proof of that either...but  I guess it's how guys like Robert Harley can hear the directionality of the crystalline structure of cables)... but that's no reason to believe what they claim. 

And it's always funny when the caution that the listener who isn;'t using a $10,000 amp and $100/ft cabling might not hear the amazing microdynamics they hear...thereby covering their asses.

FWIW, Floyd Toole, Sean Olive, et al, who are doing controlled comparisons of speakers  using trained listeners at their facility at Harman/JBL, seem rather more credible to me than any 'hi fi' mag in this area