HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: Archimago on 2013-01-15 20:11:43

Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-01-15 20:11:43
Happy New Year and greetings everyone!  I know the folks here at Hydrogen audio have been around and back with regards to these tests... Nonetheless, I've been collecting data from around the audio forums since early Dec 2012.

Since early December 2012, I've opened up a survey to see if music lovers & "audiophiles" around the world can tell the difference between high bitrate ~320kbps MP3 against the original un-lossy-compressed CD audio. I'm only planning to collect data until the end of January 2013.

So far, I have >70 responses from around the world with 5 continents represented. Folks have been using everything from inexpensive (but good) headphones to megabuck separates >$50K for this. There's 2 weeks left so if you've ever been curious about participating in a blind test or just want to add to the dataset, here's your chance :-).

The "test" is relatively simple and consists of 3 musical passages encoded as "Set A" vs. the same songs "Set B"; one of which was MP3 encoded.

Come to my blog to download the music and fill out the survey:
Archimago's Musings (http://archimago.blogspot.com/)

Thanks and have a wonderful 2013!

OOps - the title should have read 'ends Jan 31'... [fixed]
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: pdq on 2013-01-15 20:31:23
Just out of curiosity, which forums have you been posting to, and what kind of responses are you getting?
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: krabapple on 2013-01-16 08:02:13
Just out of curiosity, which forums have you been posting to, and what kind of responses are you getting?


From his blog one can see that he's posted it to Audio Asylum too.  I can well imagine at least one species of response there:  "bah, CD audio is lossy too!"
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Destroid on 2013-01-16 11:02:09
Quote
I've opened up a survey to see if music lovers & "audiophiles" around the world can tell the difference between high bitrate ~320kbps MP3 against the original un-lossy-compressed CD audio.

Meaning no disrespect, but comparing those options are not indicative of a solution. I wonder what is the objective. To clarify:

1. To use such high bit-rates is to achieve transparency/archival.

2. Any user using either of those options is likely dealing with higher space confines.

Putting all cards on the table, I would not entrust to archive with 320kpbs MP3, nor I would put those massive MP3's on my player (i.e. LAME -V3 thru -V5 is "good enough" on my portable). Currently, I am content with AAC ~125kbps without any qualms.

If I misunderstood that this was a listening challenge, then I apologize for my interruption. I probably could not ABX MP3 320kpbs and lossless.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: db1989 on 2013-01-16 12:00:45
1. To use such high bit-rates is to achieve transparency/archival a warm and fuzzy feeling.
FTFY

I hope Moore’s Law has taken us past the point where people feel a need to connect any lossy configuration to archival. IMO, the middle ground makes no sense: either choose a bitrate that (aside from in possible highly rare cases) is transparent, or use lossless. Space confines, obviously possible, but then don’t waste space on 320 kbps for listening! If someone lacks the space to archive, then that’s a bit of a shame, but there’s no point in settling for anything other than lossless if one wants music in a reliable and future-proof form.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Destroid on 2013-01-16 12:34:07
Ok, thanks for catching my rash assumption.

So, I guess it will be interesting in reading about why a music lover would go after 320kbps, and, whether that audio aficionado could detect a difference to the lossless source.

My best guess why this is issue is of concern: buying music online. Except, I doubt the major vendors would have knowledge of LAME/halb27's cutting-edge contributions.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Kohlrabi on 2013-01-16 12:57:19
Reading the test description one flaw seems to be that you encoded one set with a lossless encoder (let's call it "A"), and the other with a lossy MP3 encoder ("B"). Since you don't enforce the use of a software which ensures the test is being conducted in a double-blind fashion, so that the experimenter himself doesn't know which file is which until the test is finished, the following chain of events might occur:

Someone might analyse the spectra of the files and look for MP3 signatures beforehand, or conduct a successful test, and thus be able to determine correctly that file "B" is the lossy one. He then might publish it, which will affect any listeners who don't use a DBT software.

To me, the only way to ensure a proper test is to ensure that no cheating is possible in the first place, i.e. the test is done by enforcing the use of a DBT software, and ideally the result cannot be tampered with. Similar to how the ABC/HR tests have been done here in the past.

The questionnaire reduces the test to a single question about whether "A" or "B" is the inferior source. If the listener did not perform a full ABX test with a high enough number of trials, but rather a single listening or guess, this is the same as conducting only a single ABX test, where a person who guessed has a 50% chance to guess correctly. How will you be able to exclude guesses from your results, if you only have this single question at the beginning, and not a full test result with a number of trials that reduce your likelihood of the result occurring by chance to below 0.05?
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: pdq on 2013-01-16 13:40:48
There is a further problem. Suppose that a listener is able to reliably differentiate between the two versions, but he prefers the sound of the lossy encoded file. This will then skew the results toward listeners NOT being able to hear a difference.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: DonP on 2013-01-16 13:50:13
My best guess why this is issue is of concern: buying music online. Except, I doubt the major vendors would have knowledge of LAME/halb27's cutting-edge contributions.


Properties in foobar shows all my emusic files as encoded by Lame.  Older ones (3.96) just show an average bit rate ~170-190kb.  Newer ones (3.98r) are listed as -V0 or V2. I don't know if Lame started explicitly tagging the encode setting?

Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: 2Bdecided on 2013-01-16 15:00:46
I'm sure there are examples of 320kbps mp3s being ABXed right here on HA.

I don't think anyone here would claim a common/universal night-and-day difference with all tracks though.

Cheers,
David.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: DonP on 2013-01-16 15:39:39
Someone might analyse the spectra of the files and look for MP3 signatures beforehand, or conduct a successful test, and thus be able to determine correctly that file "B" is the lossy one. He then might publish it, which will affect any listeners who don't use a DBT software.


Even if someone guesses wrongly, publishing that guess would introduce bias.

ANother interesting quirk, compared to ABX, is that there's no reference.  Even if you can detect a difference (likely to be pretty subtle), you are also tasked with figuring out which one is "better." 
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: testyou on 2013-01-17 03:28:12
I'm going to decline your invitation Archimago.
You don't want to actually post at a reasonable place like HA, you're just trying to get hits on your blog.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-01-19 23:57:55
I'm going to decline your invitation Archimago.
You don't want to actually post at a reasonable place like HA, you're just trying to get hits on your blog.


Ain't that a bit cynical?

I know this place is "reasonable" but just the same, I am interested in an honest evaluation for anyone who hasn't already tried a blind test or just wants to be charitable in devoting some time and let me borrow the use of their ears/brain and equipment. Although I love audio gear, I know most of the world doesn't care for the bizarre/cultish/tweaky nature of much of it (frankly neither do I!).

You'll note that my blog has nothing on it other than this.  Also, I have no ads and I care not about the # of hits, haven't even signed up for Google Analytics or anything like that...

As of today, there are just >100 responses from around the world!  There's about 1.5 weeks left so I would love to get more responses to beef up the stats analysis.

Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-01-20 00:08:17
Someone might analyse the spectra of the files and look for MP3 signatures beforehand, or conduct a successful test, and thus be able to determine correctly that file "B" is the lossy one. He then might publish it, which will affect any listeners who don't use a DBT software.


Even if someone guesses wrongly, publishing that guess would introduce bias.

ANother interesting quirk, compared to ABX, is that there's no reference.  Even if you can detect a difference (likely to be pretty subtle), you are also tasked with figuring out which one is "better."


Yup, no doubt there will be criticisms of the methodology. My hope is that there will be enough responses to statistically differentiate any bias towards A or B. Also, I would love to see if there's a significant bias for those who thought it was easy to tell the difference (ie. can the confident audiophile pick out the MP3?).  Also, I can look at results vs. cost and presumably quality of the equipment used.

Ideally, an ABX methodology would be great.  However practically, this would mean a 30% increase in the download size. Furthermore, given the nature of the test; it would be WAY TOO EASY for anyone to compare X with A / B without listening :-).  Impossible that I would trust that people would not "cheat" when it's so easy. 
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-01-20 00:11:46
Reading the test description one flaw seems to be that you encoded one set with a lossless encoder (let's call it "A"), and the other with a lossy MP3 encoder ("B"). Since you don't enforce the use of a software which ensures the test is being conducted in a double-blind fashion, so that the experimenter himself doesn't know which file is which until the test is finished, the following chain of events might occur:

Someone might analyse the spectra of the files and look for MP3 signatures beforehand, or conduct a successful test, and thus be able to determine correctly that file "B" is the lossy one. He then might publish it, which will affect any listeners who don't use a DBT software.

To me, the only way to ensure a proper test is to ensure that no cheating is possible in the first place, i.e. the test is done by enforcing the use of a DBT software, and ideally the result cannot be tampered with. Similar to how the ABC/HR tests have been done here in the past.

The questionnaire reduces the test to a single question about whether "A" or "B" is the inferior source. If the listener did not perform a full ABX test with a high enough number of trials, but rather a single listening or guess, this is the same as conducting only a single ABX test, where a person who guessed has a 50% chance to guess correctly. How will you be able to exclude guesses from your results, if you only have this single question at the beginning, and not a full test result with a number of trials that reduce your likelihood of the result occurring by chance to below 0.05?


Good points.  Again, I think such is the limitation of open tests like this over the 'Net.

Why don't you analyze the files and tell me which is the MP3 by PM? I've tried my best to make it not easy to "cheat" the usual way :-)
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-01-25 20:10:59
FINAL CALL!

Closes next week. So far 124 respondents to the test / survey.

Thanks to all participants.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-02-01 21:39:33
Study complete!  Total respondents - 151.

Follow on my blog as I post the procedure and analysis in the days ahead.  Thanks for all the input.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: C.R.Helmrich on 2013-02-02 17:07:17
From your blog (http://archimago.blogspot.de/):

Quote
This utilizes LAME's "free format" to create initially a 400kbps MP3 without the usual lowpass filter in place, then runs the resulting file through the MP3 encoder again but at a lower 350kbps bitrate (again with low-pass turned off) which closer approximates the 320kbps target bitrate for the test. By doing this, even though the resulting MP3 size is slightly larger by 30kbps, the degradation in sound quality by objective measures is in fact approximately the same or slightly worse than if the audio were processed directly through 320kbps but without the tell-tale sign of the strong low-pass filter.

  Wait... so:

Sorry, you idea was great, but I can't see how you'll be able to draw any conclusion from your test which cannot be torn into pieces in a subsequent discussion.

Chris
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: greynol on 2013-02-02 17:14:07
Wow, I can't believe I once had this topic in validated news.

Two words for you, Archimago: temporal smearing

You should have studied up on the block-size limitations of mp3 and how that makes it particularly vulnerable to the pitfalls of lossy-lossy transcoding before making such a bone-headed mistake.

Any sort of statistical results are going to determine the audibility of transcoded mp3 against lossless long before they can even begin to apply to high-bitrate mp3 against lossless.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Kohlrabi on 2013-02-02 19:07:07
I (and others) did question the approach before the test was conducted, too bad not more regulars decide to chime in. Now the damage is done. It's a pity with all the effort that went into organizing this. I mean, this after all shows that high-priced gear "audiophiles" might have worse hearing than the general populace, so this would have been valuable.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: greynol on 2013-02-02 19:16:22
It's pretty obvious the OP registered here only to solicit participants.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-02-03 06:57:18
Hang on guys, you're perhaps misinterpreting the test and jumping to conclusions...

This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

Of course people could look at the spectrum while testing, there's no way I can control that EXCEPT by masking it with turning off the lowpass filter, but I could not do that and have the spectrum "look right" unless I went slightly above 320kbps - hence the double pass so I could get the "error" within the 320kbps level.

Even with the double LAME compression passes, people still thought the MP3 sounded better than lossless! As the report shows, those using >$6000 gear got it even more wrong than those using <$500.

Is it because those owning >$6000 gear were older and had bad hearing?  Who knows, but the point was that I wondered if MP3 was really that atrocious compared to lossless as many claim (I know you guys here generally do not believe this). Clearly even running the music through 2 psychoacoustic passes with all the horrors of smearing, and hindering the algorithm further with turning off the lowpass filter, it was undetectable and in fact more respondents PREFERRED it.

An ABX methodology with a test like this would not be practical on a large scale IMO and doomed for multiple reasons.

No, I did not register just to solicit subjects...  Been reading here for years in fact, so wanted to have some input from some of the folks here whom I honestly highly respect. If you haven't seen the results I suggest having a gander.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: ExUser on 2013-02-03 07:40:30
An ABX methodology with a test like this would not be practical on a large scale IMO and doomed for multiple reasons.

Been reading here for years in fact, so wanted to have some input from some of the folks here whom I honestly highly respect.
I cannot rationalize the existence of these two claims with respect to each other. "An ABX methodology" is the foundational concept of this forum, for heaven's sake. This is a flagrant breach of TOS8. You're showing one of the most massive disrespects possible.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-02-03 07:54:01
An ABX methodology with a test like this would not be practical on a large scale IMO and doomed for multiple reasons.

Been reading here for years in fact, so wanted to have some input from some of the folks here whom I honestly highly respect.
I cannot rationalize the existence of these two claims with respect to each other. "An ABX methodology" is the foundational concept of this forum, for heaven's sake. This is a flagrant breach of TOS8. You're showing one of the most massive disrespects possible.


Really? I'm not criticizing or questioning the ABX methodology, am I? I think the ABC/HR idea earlier looks very interesting and maybe if I had been aware of some software that could have approached it from that angle, it would have been the way to go.

My goal was to have an easily accessible test to reach the "audiophile" crowd (as can be seen from most of the forums solicited and the test introduction) since the idea and disagreement which started it all began there. In order to get input from the cohort, the test needed to be flexible enough so some could burn to CD, others could upload to music servers, etc...  Not sit around a computer to flip between A & B. Sure, a proper ABX would be significantly more reliable but looking at the equipment people used to test (remember, many of these guys use stripped down computers, insist on Amarra/JPlay/Decibel, etc.), a large proportion unfortunately would not have accepted this type of methodology.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: C.R.Helmrich on 2013-02-03 15:01:26
Fair enough, given that you apparently didn't know about ABC-HR and given that the "result says no difference" issue pdq first mentioned (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=98890&view=findpost&p=820873) did not occur. But the bottom line is: your listeners heard a difference between A and B. Proponents of lossy (but transparent) coding can now argue, "of course they heard a difference, the MP3 encoding was badly done", while opponents might argue, "this test used some enhanced (free-format, not low-passed) 350-kbps MP3 and doesn't tell you anything about whether listeners would have preferred real-life non-tandem-coded MP3". See what I mean?

Quote
This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

I think we all understood this perfectly. The problem is that when you have to modify a lossy encoder to hide what it's doing from your test participants, your test is flawed.

Chris
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Kohlrabi on 2013-02-03 15:56:27
This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

Of course people could look at the spectrum while testing, there's no way I can control that EXCEPT by masking it with turning off the lowpass filter, but I could not do that and have the spectrum "look right" unless I went slightly above 320kbps - hence the double pass so I could get the "error" within the 320kbps level.
I will be glad the day when everybody accepts that MP3 encoding is more than just a lowpass. By obsessing over the lowpass of LAME you destroyed everything else that makes LAME a good lossy encoder, like the psychoacoustic model.

Even with the double LAME compression passes, people still thought the MP3 sounded better than lossless! As the report shows, those using >$6000 gear got it even more wrong than those using <$500.
I always assumed those overpaying audiofools had bad hearing, thanks for showing at least that.

Is it because those owning >$6000 gear [...] had bad hearing?
Well, this might be the only real outcome of this test, if any.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-02-03 16:32:22
Fair enough, given that you apparently didn't know about ABC-HR and given that the "result says no difference" issue pdq first mentioned (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=98890&view=findpost&p=820873) did not occur. But the bottom line is: your listeners heard a difference between A and B. Proponents of lossy (but transparent) coding can now argue, "of course they heard a difference, the MP3 encoding was badly done", while opponents might argue, "this test used some enhanced (free-format, not low-passed) 350-kbps MP3 and doesn't tell you anything about whether listeners would have preferred real-life non-tandem-coded MP3". See what I mean?

Quote
This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

I think we all understood this perfectly. The problem is that when you have to modify a lossy encoder to hide what it's doing from your test participants, your test is flawed.

Chris


Yes, Helmrich, I do fully understand this and talked about the "unorthodox" encoding and how it could affect the audio quality in a negative way. The hope in the test was to demonstrate that lossy encoding does not deserve a "bad rap" as has been generally portrayed and I think this has been achieved when I look at the response from the audiophile crowd.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-02-03 16:37:09
This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

Of course people could look at the spectrum while testing, there's no way I can control that EXCEPT by masking it with turning off the lowpass filter, but I could not do that and have the spectrum "look right" unless I went slightly above 320kbps - hence the double pass so I could get the "error" within the 320kbps level.
I will be glad the day when everybody accepts that MP3 encoding is more than just a lowpass. By obsessing over the lowpass of LAME you destroyed everything else that makes LAME a good lossy encoder, like the psychoacoustic model.

Even with the double LAME compression passes, people still thought the MP3 sounded better than lossless! As the report shows, those using >$6000 gear got it even more wrong than those using <$500.
I always assumed those overpaying audiofools had bad hearing, thanks for showing at least that.

Is it because those owning >$6000 gear [...] had bad hearing?
Well, this might be the only real outcome of this test, if any.


I did not say that MP3 was somehow just lowpass. Lowpass is just the "pathognomonic" sign of MP3 encoding; the nasty fingerprint by which anyone who has an interest in looking at the spectrum would detect. Knowing that in an open survey like this where people are free to look, this somehow must be hidden using the MP3 encoder itself.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: db1989 on 2013-02-03 18:05:08
I question whether some factor related to the specific processing methodology might be responsible for the skew of listeners in favour of the encoded files. One would more logically expect an approximately 50/50 split between lossless and lossy when the latter is at a generally transparent bitrate such as 320 kbps. The fact that 52% thought MP3 sounded better, whereas only 18% perceived no difference, suggests either some effect of your double conversion that is making the MP3s sound more appealing in some way or simply really bad and somewhat arbitrary judgements by the relevant respondents.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: C.R.Helmrich on 2013-02-03 19:48:50
The hope in the test was to demonstrate that lossy encoding does not deserve a "bad rap" as has been generally portrayed and I think this has been achieved when I look at the response from the audiophile crowd.

Alright. Once some (positive or negative) criticism from that audiophile crowd appeared, it would be nice if you could summarize it or link to it in this thread. I'd be curious to know whether you convinced some long-term sceptics. And please, call me Chris

Chris
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-02-03 22:56:59
The hope in the test was to demonstrate that lossy encoding does not deserve a "bad rap" as has been generally portrayed and I think this has been achieved when I look at the response from the audiophile crowd.

Alright. Once some (positive or negative) criticism from that audiophile crowd appeared, it would be nice if you could summarize it or link to it in this thread. I'd be curious to know whether you convinced some long-term sceptics. And please, call me Chris

Chris


Hi Chris,
I don't know if I have time ahead to to summarize in great detail the response from audiophiles...  Since this test started on AudioAsylum in the "PC Audio" subforum, it might be worthwhile following the thread there.

Nonetheless I have already received E-mail such as the little snip below...

"This test was VERY VERY educating. I write this not because I am in the
wrong-guessers camp, but because it demonstrated to me the following:
1. My very minor exposure to mp3 material had led me to assume that mp3
sounds BAD without actual listening to it in my life - less than several
dozen times and through smallish desktop speakers.
Truth is mp3 can sound very good."

He goes on to describe tests he did himself with the LAME procedure I used and his further thoughts... Great to see that this test has catalyzed some consideration on the matter.

This test was never aimed at scientific rigor (although I have tried to document and describe the procedure as best I can for full disclosure and critique); but rather allow questioning of a certain commonly held sentiment.  Like the response above shows, it has at least affected some people who I suspect would never in their life sit down to try a properly controlled ABX.
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Dynamic on 2013-02-04 10:48:52
It's always going to be tough to let people look at the spectrum and disguise the MP3. The only way to do so without transcoding the MP3 would probably be to high-pass filter the original (or a freeformat version of it) with a very close-matching opposite filter with zero delay and arithmetically add the waveform containing just the ultrasonics back onto the 320kbps LAME decode or to restrict the test to 320kbps LAME handicapped by having the low-pass disabled. The other approach is to apply the same low-pass filter to the lossless original, which is likely to draw criticisms of its own.

Despite the methodological difficulties, which make the test less definitive and scientifically valid than typical single-encode ABX comparisons, I think the effort may have opened some eyes among audiophiles who simply assume that MP3 is different or worse, so this 'outreach' may have been useful.

I think a 4th option saying "Different, but no idea which is better" would have been problematic in providing a 'get out' for people who instinctively assume MP3 is inferior, so making them pick the better sound unless they said they were identical was necessary to counter that intellectual subconscious bias.

If we consider the Null Hypothesis that "nobody could reliably distinguish A from B" and assume for a moment that it might be true, a statistically significant preference for A over B or B over A might still arise from any number of reasons why people might prefer to choose A over B or B over A (in the same way that people in general asked to choose a random two-digit number that's not easy for the experimenter to guess frequently choose thirty-seven in psychology experiments because it 'feels' more random than most others with repeated, even or consecutive digits or 'round numbers' or those near the middle of the distribution, people might choose A or choose B more frequently)

It might be that a number of replications of your test with the same methodology but a random assignment of A or B to lossless and encoded music might show a similar preference for A over B or B over A even when the audio is swapped, for example.

So the result is not necessarily inconsistent with the Null Hypothesis and due to lack of randomization of A and B between different people's trials, it's not possible to rule out the Null Hypothesis that "no difference could be discerned", nor is it possible to rule out an Alternative Hypothesis that "the encoded file sounds better than the lossless file" or a Second Alternative Hypothesis that "the files sounded different but one was not clearly better than the other - just different"

I certainly think the test has some value and short of programming a server to randomly assign audio to A & B and somehow track that to the participants, it's about the best you can fairly do without ABX or ABC/HR type methodology.

It also has some educational value in experimental design and compromises that may be necessary, so thanks for that.

I think however that the conclusion drawn should be more nuanced than simply stating that the participants preferred MP3 to lossless by an apparently significant margin.

It's an interesting experiment but the correct conclusion to draw isn't obvious in the light of this one test.

Different kinds of tests might have had more power to rule out some of the possible conclusions as false conclusions from the same number of participants, but might not have garnered so many participants.

We're fortunate within HydrogenAudio to have a decent number of participants who are willing to use double-blinded ABC/HR or ABX, which help to eliminate more potential biases or confounding factors and help us to gain reliable conclusions frequently (albeit that it's rare to get a statistically significant result with normal music at medium to high bitrates)
Title: High Bitrate ~320kbps MP3 vs. Lossless audio
Post by: Archimago on 2013-02-05 00:38:08
Thanks for the eloquent response Dynamic.

Would you be OK if I took this message and added it to the "comments" on the blog page? As you alluded to, the experimental design is absolutely important and I certainly want to make sure readers get a chance to review what you just said.