Skip to main content

Topic: results of lossless vs lossy listening test on reddit (Read 5669 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • krabapple
  • [*][*][*][*][*]
results of lossless vs lossy listening test on reddit

  • db1989
  • [*][*][*][*][*]
  • Global Moderator
results of lossless vs lossy listening test on reddit
Reply #1
Quote
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I don’t like this.

I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that there’s a good amount of scientific discussion and scepticism there.

I’m sure there are lots of delightful comments there if I were to read far enough, though. I don’t really do Reddit. And I try to avoid comment sections in general.

Anyway, I’ll wait with interest to see what other users here think about this.

  • Kohlrabi
  • [*][*][*][*][*]
  • Global Moderator
results of lossless vs lossy listening test on reddit
Reply #2
Quote
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I don’t like this.
3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?
  • Last Edit: 22 February, 2013, 02:15:35 PM by Kohlrabi
It's only audiophile if it's inconvenient.

  • krabapple
  • [*][*][*][*][*]
results of lossless vs lossy listening test on reddit
Reply #3
Quote
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I don’t like this.

I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that there’s a good amount of scientific discussion and scepticism there.

I’m sure there are lots of delightful comments there if I were to read far enough, though. I don’t really do Reddit. And I try to avoid comment sections in general.

Anyway, I’ll wait with interest to see what other users here think about this.



There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.

edit: btw, it occurs to me that maybe this should have bene in the Listening Tests subforum...sorry, mods, please move as needed.
  • Last Edit: 22 February, 2013, 02:19:06 PM by krabapple

  • db1989
  • [*][*][*][*][*]
  • Global Moderator
results of lossless vs lossy listening test on reddit
Reply #4
There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.
Sure, and that’s what I meant:
I had a sinking feeling […] but I was pleasantly surprised […] I’m sure there are lots of delightful comments there if I were to read far enough, though.
I was referring to the same uppermost comments as you and my reluctance to read much further down.

results of lossless vs lossy listening test on reddit
Reply #5
results out today

http://www.reddit.com/r/audiophile/comment...ssless_results/


In as few words as possible - how were the evaluations done?  I looked at the site and nothing about that popped out at me.

  • mjb2006
  • [*][*][*][*][*]
results of lossless vs lossy listening test on reddit
Reply #6
In as few words as possible - how were the evaluations done?  I looked at the site and nothing about that popped out at me.

See http://www.reddit.com/r/audiophile/comment...st_details_and/

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
results of lossless vs lossy listening test on reddit
Reply #7
3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?


No, the test didn't show that. If you squash all listeners together, and hence pretend only a single person took the test, then yes. You can also squash the samples together, and then say FLAC won only 1 test  That's not the right way to analyze this.

If we instead consider there are 62 listeners, times 5 samples, each of which did a block of 4 codecs, then we get this:

Code: [Select]
./bootstrap.py --compare-all --blocked reddit2.csv 
bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto <gcp@sjeng.org>
License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html>

Reading from: reddit2.csv
Read 4 treatments, 310 samples => 6 comparisons
Means:
MP3_320      FLAC   AAC_192   MP3_128
   2.406     2.165     2.694     2.761

Unadjusted p-values:
          FLAC      AAC_192   MP3_128  
MP3_320   0.013*    0.005*    0.001*  
FLAC      -         0.000*    0.000*  
AAC_192   -         -         0.520    

FLAC is worse than MP3_320 (p=0.013)
AAC_192 is better than MP3_320 (p=0.005)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.001)
MP3_128 is better than FLAC (p=0.000)

p-values adjusted for multiple comparison:
          FLAC      AAC_192   MP3_128  
MP3_320   0.028*    0.014*    0.004*  
FLAC      -         0.000*    0.000*  
AAC_192   -         -         0.521    

FLAC is worse than MP3_320 (p=0.028)
AAC_192 is better than MP3_320 (p=0.014)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.004)
MP3_128 is better than FLAC (p=0.000)


Note that better really means worse here  - Reddit had good=1 and bad=4. The test result is basically:

FLAC > MP3_320 > AAC_192 = MP3_128

I'm surprised that they're apparently able to tell 320kbps MP3 from a FLAC, but fail to tell an 192kbps AAC from an 128kbps MP3. (Unless 192kbps iTunes has the same lowpass as 128kbps LAME, of course  )
  • Last Edit: 26 February, 2013, 06:16:09 AM by Garf

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
results of lossless vs lossy listening test on reddit
Reply #8
Quote
Three of these sections were then re-encoded from that lossless WAV to MP3-320 Joint Stereo, AAC-192 and MP3-128 Non-joint Stereo (via Goldwave) while the last section remains lossless.


What encoders goes Goldwave use? Also why force full stereo on an 128kbps MP3 encode, that does not make sense.

  • LithosZA
  • [*][*][*]
results of lossless vs lossy listening test on reddit
Reply #9
I hope they didn't use something like FAAC for AAC encoding...

Quote
Also why force full stereo on an 128kbps MP3 encode, that does not make sense.

I agree, that doesn't make sense at all.

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
results of lossless vs lossy listening test on reddit
Reply #10
I hope they didn't use something like FAAC for AAC encoding...


Seems to be QuickTime (which should be fine?) and some version of LAME (potentially an alpha).