Skip to main content
Topic: results of lossless vs lossy listening test on reddit (Read 6620 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

results of lossless vs lossy listening test on reddit

Reply #1
Quote
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I don’t like this.

I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that there’s a good amount of scientific discussion and scepticism there.

I’m sure there are lots of delightful comments there if I were to read far enough, though. I don’t really do Reddit. And I try to avoid comment sections in general.

Anyway, I’ll wait with interest to see what other users here think about this.

results of lossless vs lossy listening test on reddit

Reply #2
Quote
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I don’t like this.
3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?
It's only audiophile if it's inconvenient.

results of lossless vs lossy listening test on reddit

Reply #3
Quote
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I don’t like this.

I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that there’s a good amount of scientific discussion and scepticism there.

I’m sure there are lots of delightful comments there if I were to read far enough, though. I don’t really do Reddit. And I try to avoid comment sections in general.

Anyway, I’ll wait with interest to see what other users here think about this.



There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.

edit: btw, it occurs to me that maybe this should have bene in the Listening Tests subforum...sorry, mods, please move as needed.

results of lossless vs lossy listening test on reddit

Reply #4
There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.
Sure, and that’s what I meant:
I had a sinking feeling […] but I was pleasantly surprised […] I’m sure there are lots of delightful comments there if I were to read far enough, though.
I was referring to the same uppermost comments as you and my reluctance to read much further down.



 

results of lossless vs lossy listening test on reddit

Reply #7
3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?


No, the test didn't show that. If you squash all listeners together, and hence pretend only a single person took the test, then yes. You can also squash the samples together, and then say FLAC won only 1 test  That's not the right way to analyze this.

If we instead consider there are 62 listeners, times 5 samples, each of which did a block of 4 codecs, then we get this:

Code: [Select]
./bootstrap.py --compare-all --blocked reddit2.csv 
bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto <gcp@sjeng.org>
License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html>

Reading from: reddit2.csv
Read 4 treatments, 310 samples => 6 comparisons
Means:
MP3_320      FLAC   AAC_192   MP3_128
   2.406     2.165     2.694     2.761

Unadjusted p-values:
          FLAC      AAC_192   MP3_128  
MP3_320   0.013*    0.005*    0.001*  
FLAC      -         0.000*    0.000*  
AAC_192   -         -         0.520    

FLAC is worse than MP3_320 (p=0.013)
AAC_192 is better than MP3_320 (p=0.005)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.001)
MP3_128 is better than FLAC (p=0.000)

p-values adjusted for multiple comparison:
          FLAC      AAC_192   MP3_128  
MP3_320   0.028*    0.014*    0.004*  
FLAC      -         0.000*    0.000*  
AAC_192   -         -         0.521    

FLAC is worse than MP3_320 (p=0.028)
AAC_192 is better than MP3_320 (p=0.014)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.004)
MP3_128 is better than FLAC (p=0.000)


Note that better really means worse here  - Reddit had good=1 and bad=4. The test result is basically:

FLAC > MP3_320 > AAC_192 = MP3_128

I'm surprised that they're apparently able to tell 320kbps MP3 from a FLAC, but fail to tell an 192kbps AAC from an 128kbps MP3. (Unless 192kbps iTunes has the same lowpass as 128kbps LAME, of course  )

results of lossless vs lossy listening test on reddit

Reply #8
Quote
Three of these sections were then re-encoded from that lossless WAV to MP3-320 Joint Stereo, AAC-192 and MP3-128 Non-joint Stereo (via Goldwave) while the last section remains lossless.


What encoders goes Goldwave use? Also why force full stereo on an 128kbps MP3 encode, that does not make sense.

results of lossless vs lossy listening test on reddit

Reply #9
I hope they didn't use something like FAAC for AAC encoding...

Quote
Also why force full stereo on an 128kbps MP3 encode, that does not make sense.

I agree, that doesn't make sense at all.

results of lossless vs lossy listening test on reddit

Reply #10
I hope they didn't use something like FAAC for AAC encoding...


Seems to be QuickTime (which should be fine?) and some version of LAME (potentially an alpha).

 
SimplePortal 1.0.0 RC1 © 2008-2020