results out today
http://www.reddit.com/r/audiophile/comment...ssless_results/ (http://www.reddit.com/r/audiophile/comments/1903bi/lossy_vs_lossless_results/)
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I don’t like this.
I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that there’s a good amount of scientific discussion and scepticism there.
I’m sure there are lots of
delightful comments there if I were to read far enough, though. I don’t really do Reddit. And I try to avoid comment sections in general.
Anyway, I’ll wait with interest to see what other users here think about this.
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I don’t like this.
3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I don’t like this.
I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that there’s a good amount of scientific discussion and scepticism there.
I’m sure there are lots of delightful comments there if I were to read far enough, though. I don’t really do Reddit. And I try to avoid comment sections in general.
Anyway, I’ll wait with interest to see what other users here think about this.
There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.
edit: btw, it occurs to me that maybe this should have bene in the Listening Tests subforum...sorry, mods, please move as needed.
There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.
Sure, and that’s what I meant:
I had a sinking feeling […] but I was pleasantly surprised […] I’m sure there are lots of delightful comments there if I were to read far enough, though.
I was referring to the same uppermost comments as you and my reluctance to read much further down.
results out today
http://www.reddit.com/r/audiophile/comment...ssless_results/ (http://www.reddit.com/r/audiophile/comments/1903bi/lossy_vs_lossless_results/)
In as few words as possible - how were the evaluations done? I looked at the site and nothing about that popped out at me.
In as few words as possible - how were the evaluations done? I looked at the site and nothing about that popped out at me.
See http://www.reddit.com/r/audiophile/comment...st_details_and/ (http://www.reddit.com/r/audiophile/comments/17zm5a/raudiophile_lossless_vs_lossy_test_details_and/)
3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?
No, the test didn't show that. If you squash all listeners together, and hence pretend only a single person took the test, then yes. You can also squash the samples together, and then say FLAC won only 1 test That's not the right way to analyze this.
If we instead consider there are 62 listeners, times 5 samples, each of which did a block of 4 codecs, then we get this:
./bootstrap.py --compare-all --blocked reddit2.csv
bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto <gcp@sjeng.org>
License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html>
Reading from: reddit2.csv
Read 4 treatments, 310 samples => 6 comparisons
Means:
MP3_320 FLAC AAC_192 MP3_128
2.406 2.165 2.694 2.761
Unadjusted p-values:
FLAC AAC_192 MP3_128
MP3_320 0.013* 0.005* 0.001*
FLAC - 0.000* 0.000*
AAC_192 - - 0.520
FLAC is worse than MP3_320 (p=0.013)
AAC_192 is better than MP3_320 (p=0.005)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.001)
MP3_128 is better than FLAC (p=0.000)
p-values adjusted for multiple comparison:
FLAC AAC_192 MP3_128
MP3_320 0.028* 0.014* 0.004*
FLAC - 0.000* 0.000*
AAC_192 - - 0.521
FLAC is worse than MP3_320 (p=0.028)
AAC_192 is better than MP3_320 (p=0.014)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.004)
MP3_128 is better than FLAC (p=0.000)
Note that better really means worse here - Reddit had good=1 and bad=4. The test result is basically:
FLAC > MP3_320 > AAC_192 = MP3_128
I'm surprised that they're apparently able to tell 320kbps MP3 from a FLAC, but fail to tell an 192kbps AAC from an 128kbps MP3. (Unless 192kbps iTunes has the same lowpass as 128kbps LAME, of course )
Three of these sections were then re-encoded from that lossless WAV to MP3-320 Joint Stereo, AAC-192 and MP3-128 Non-joint Stereo (via Goldwave) while the last section remains lossless.
What encoders goes Goldwave use? Also why force full stereo on an 128kbps MP3 encode, that does not make sense.
I hope they didn't use something like FAAC for AAC encoding...
Also why force full stereo on an 128kbps MP3 encode, that does not make sense.
I agree, that doesn't make sense at all.
I hope they didn't use something like FAAC for AAC encoding...
Seems to be QuickTime (which should be fine?) and some version of LAME (potentially an alpha).