results of lossless vs lossy listening test on reddit
Reply #7 – 2013-02-26 10:19:17
3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf? No, the test didn't show that. If you squash all listeners together, and hence pretend only a single person took the test, then yes. You can also squash the samples together, and then say FLAC won only 1 test That's not the right way to analyze this. If we instead consider there are 62 listeners, times 5 samples, each of which did a block of 4 codecs, then we get this:./bootstrap.py --compare-all --blocked reddit2.csv bootstrap.py v1.0 2011-02-03 Copyright (C) 2011 Gian-Carlo Pascutto <gcp@sjeng.org> License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html> Reading from: reddit2.csv Read 4 treatments, 310 samples => 6 comparisons Means: MP3_320 FLAC AAC_192 MP3_128 2.406 2.165 2.694 2.761 Unadjusted p-values: FLAC AAC_192 MP3_128 MP3_320 0.013* 0.005* 0.001* FLAC - 0.000* 0.000* AAC_192 - - 0.520 FLAC is worse than MP3_320 (p=0.013) AAC_192 is better than MP3_320 (p=0.005) AAC_192 is better than FLAC (p=0.000) MP3_128 is better than MP3_320 (p=0.001) MP3_128 is better than FLAC (p=0.000) p-values adjusted for multiple comparison: FLAC AAC_192 MP3_128 MP3_320 0.028* 0.014* 0.004* FLAC - 0.000* 0.000* AAC_192 - - 0.521 FLAC is worse than MP3_320 (p=0.028) AAC_192 is better than MP3_320 (p=0.014) AAC_192 is better than FLAC (p=0.000) MP3_128 is better than MP3_320 (p=0.004) MP3_128 is better than FLAC (p=0.000) Note that better really means worse here - Reddit had good=1 and bad=4. The test result is basically: FLAC > MP3_320 > AAC_192 = MP3_128 I'm surprised that they're apparently able to tell 320kbps MP3 from a FLAC, but fail to tell an 192kbps AAC from an 128kbps MP3. (Unless 192kbps iTunes has the same lowpass as 128kbps LAME, of course )