Skip to main content ↑ ↓
Topic: High bitrate MP3 vs. Lossless ABX Tests? (Read 30048 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

High bitrate MP3 vs. Lossless ABX Tests?

Hi All,

I've been corrected before on this site for making the statement: "I can tell the difference between an MP3 and lossless". I've never ABX'ed myself but I was wondering if there has been a statistically significant (with regard to population) ABX study done with (say) Lame insane vs. Lossless?

Clearly for myself it makes more sense to do a personal ABX but I was wondering in general about the population.

I searched manually through the posts in this forum (since search seems useless for the term 'mp3') and didn't see anything.

Thanks for help,

ss.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #1 –
You couldn't do an accurate statistically significant test with average users because they'd all be using different (and much of the time lousy) equipment.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #2 –
Quote
Hi All,

I've been corrected before on this site for making the statement: "I can tell the difference between an MP3 and lossless". I've never ABX'ed myself but I was wondering if there has been a statistically significant (with regard to population) ABX study done with (say) Lame insane vs. Lossless?

Clearly for myself it makes more sense to do a personal ABX but I was wondering in general about the population.

I searched manually through the posts in this forum (since search seems useless for the term 'mp3') and didn't see anything.

Thanks for help,

ss.
[a href="index.php?act=findpost&pid=302604"][{POST_SNAPBACK}][/a]


try searching for lossless lossy abx

High bitrate MP3 vs. Lossless ABX Tests?

Reply #3 –
Quote
You couldn't do an accurate statistically significant test with average users because they'd all be using different (and much of the time lousy) equipment.
[a href="index.php?act=findpost&pid=302669"][{POST_SNAPBACK}][/a]


And that is a perfectly accurate reflection of the real world.  If your statement was true, any of the group blind listening tests conducted here would be invalid.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #4 –
Quote
And that is a perfectly accurate reflection of the real world.  If your statement was true, any of the group blind listening tests conducted here would be invalid.
[a href="index.php?act=findpost&pid=302676"][{POST_SNAPBACK}][/a]
No, you're not listening to what he is saying. He wants to test the limits of people's hearing, not their equipment. If you did a blind test with $50 headphones and onboard sound, and the people taking the test were unable to determine a difference, does that mean they cannot detect the difference between mp3 and lossless? Not necessarily, it means they cannot on that equipment.

Group tests conducted here are different, they are set up to determine whether people can tell the difference on their equipment because that's what they will be listening on. They do not determine whether the person is capable of percieving a difference.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #5 –
Quote
I've been corrected before on this site for making the statement: "I can tell the difference between an MP3 and lossless". I've never ABX'ed myself but I was wondering if there has been a statistically significant (with regard to population) ABX study done with (say) Lame insane vs. Lossless?[a href="index.php?act=findpost&pid=302604"][{POST_SNAPBACK}][/a]

"lossy vs. lossless ABX" doesn't make any sense. You simply ABX lossy codecs against the original, be it a wav file or a losslessly encoded file. The term "lossless" here is irrelevant. ABX tests of various lossy codecs at various settings have been made several times, search the forum (just don't search "lossy vs. lossless" specifically, because of the reason I just gave).

High bitrate MP3 vs. Lossless ABX Tests?

Reply #6 –
Quote
"lossy vs. lossless ABX" doesn't make any sense.


Come on. I think every single person who read my post understood what I intended here. You are arguing semantics instead of giving information.

Quote
try searching for lossless lossy abx


As noted in my original post, this doesn't work because the (in my opinion insanely dumb) search restrictions on the board prevent searching for any term shorter than 4 characters.

Quote
You couldn't do an accurate statistically significant test with average users because they'd all be using different (and much of the time lousy) equipment.


I guess the study that I was looking for would use standardized equipment for all participants thus eliminating this factor.

Thanks for all the information... but I still haven't seen a link to an actual study.

If indeed there is no study like this, I'm a little shocked. With all the ABX tests that have been done before (on this board and other sites) why has no one used a lossless (ok ok "orginal") sample as reference? Maybe I'm missing something here... I'm by no means a statistical or testing expert.

ss.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #7 –
Why make it complicated:

Comparing lossy VS lossless is common practive when ABXing. This is because a lossy sample is usually compared against the original(which is lossless). So, what you are asking about is not an exception but just normal standard practice in almost every ABX-test.

Concerning numbers: taking high-bitrate ABXes on only this forum into account, they would total in hundreds if not come near thousand.

So yes, statistical evidence(for non-transparency in everyday-listening) after dozens of tests is quite low.

Of course, it is always possible to find single fragments of music, where a difference can be heard. However, most people dont care if of 10 full CDs there may be 10seconds where a difference can be noticed when listening to the 10sec sample in a loop a dozen times! The background noisefloor in your room in that case is a much bigger hazard.

So, the evidence currently is low enough that one could ask "Is the background-noise in your room(a room is almost never absolutely silent) a problem to you? If no, then properly encoded high-bitrate lossy audio should be transparent to you during everyday-listening."

- Lyx

Edit: notice that you are asking the question the wrong way around: It is not possible to prove that something doesn't exist with absolute certainity. It is only possible to prove that something DOES exist. So, the question should not be if there are enough negative ABXes, but instead if there are enough positive ABXes under conditions which can be applied to normal listening. Thus, you cannot ask others to prove non-existence - the burden is on the one who claims that there IS a difference.
I am arrogant and I can afford it because I deliver.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #8 –
Quote
If indeed there is no study like this, I'm a little shocked. With all the ABX tests that have been done before (on this board and other sites) why has no one used a lossless (ok ok "orginal") sample as reference? Maybe I'm missing something here... I'm by no means a statistical or testing expert.

ss.
[a href="index.php?act=findpost&pid=302874"][{POST_SNAPBACK}][/a]

I've done the search I referred to you, and found several topics compating lame preset insane to wav.  I suggest you do the same.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #9 –
Quote
I've done the search I referred to you, and found several topics compating lame preset insane to wav. I suggest you do the same.

Why do you not post links to the studies you found? Saying that you found them does not benifit anyone. I have done similar searches are found individual results but no -studies- as I asked for at the beginning of this thread.

Quote
Edit: notice that you are asking the question the wrong way around: It is not possible to prove that something doesn't exist with absolute certainity. It is only possible to prove that something DOES exist. So, the question should not be if there are enough negative ABXes, but instead if there are enough positive ABXes under conditions which can be applied to normal listening. Thus, you cannot ask others to prove non-existence - the burden is on the one who claims that there IS a difference.
[a href="index.php?act=findpost&pid=302884"][{POST_SNAPBACK}][/a]

Thank you for the explanation. I agree with you completely. My quote from the original post was merely a summary of a viewpoint I expressed in another post not really the question I was trying to answer. I agree that you can not prove non-existence.

I guess I was hoping that there was some summarized data along these lines instead of having to go through a bunch of individual ABXs in order to determine how significant the differences were between the lossy and the lossless/original sample(s).

Quote
Comparing lossy VS lossless is common practive when ABXing.

This seems to be the case and indeed it is noted as a key part of the ABX by pab in his introduction to ABXing. However, all the studies I've found (including the extremely well constructed studies by rjamorim), these never show the results for the original sample in their results. The results always compare different formats but never give us a control to compare against... or perhaps I am reading the results incorrectly...?

ss.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #10 –
I believe that all of the listening tests compare the lossy file to the lossless original.  It is then given a rating on how well it did.

Then, the ratings of each lossy codec are compared to each other so as to see which one most accurately reproduced the lossless file.

Just to be sure that everyone understands...

Comparing the lossy file to a lossless file is no different than comparing the lossy file to the actual CD.
flac>fb2k>kernel streaming>audiophile 2496>magni>dt990 pro

High bitrate MP3 vs. Lossless ABX Tests?

Reply #11 –
Basically, any codec that is transparent versus the original sample, will be un-abx-able.  The results will be regarded as unsignificant and discarded, because most of the time, abx results are to compare version evolutions, or different encoders.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #12 –
To summarize:

a) A "lossless to high-bitrate MP3 ABX" test = a "high-bitrate MP3 ABX" test. No need to specify lossless, all ABX tests are done against a lossless source by definition. A test comparing two codecs (again, against a lossless reference) would be an ABC test.

b) Even the 128kbps tests usually come up with statistically insignificant results because the public just isn't able to distinguish. So the only way you'll ever get interesting results at high-bitrates is to either do it yourself, or read the results of someone who did. Guruboolez is about the best person here for this sort of thing... he can tell the difference on some problem samples, but not must samples.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #13 –
Quote
To summarize:

a) A "lossless to high-bitrate MP3 ABX" test = a "high-bitrate MP3 ABX" test. No need to specify lossless, all ABX tests are done against a lossless source by definition. A test comparing two codecs (again, against a lossless reference) would be an ABC test.
No 
mp3 ABX test does mean ABX vs original source (or lossless, it's the same), but nothing prevents you from ABXing mp3 vs mpc, mp3@128 vs mp3@256 and so on...

Quote
b) Even the 128kbps tests usually come up with statistically insignificant results because the public just isn't able to distinguish. So the only way you'll ever get interesting results at high-bitrates is to either do it yourself, or read the results of someone who did. Guruboolez is about the best person here for this sort of thing... he can tell the difference on some problem samples, but not must samples.
[a href="index.php?act=findpost&pid=302925"][{POST_SNAPBACK}][/a]
True.... Even in this forum, which probably is the place where most motivated and skilled people can be found, 128> listening tests don't involve many people, because all codecs are nowadays excellent (I didn't say perfect, or CD quality...) at any bitrate >= 128...

High bitrate MP3 vs. Lossless ABX Tests?

Reply #14 –
Quote
True.... Even in this forum, which probably is the place where most motivated and skilled people can be found, 128> listening tests don't involve many people, because all codecs are nowadays excellent (I didn't say perfect, or CD quality...) at any bitrate >= 128...

Thats certainly true for recent times(i think the fact that most lossy codecs are "too good" is also a factor which seems demotivating to even try ABXes for many people. Multiple people mentioned that they got the feeling that "there isn't much going on in the lossy department anymore, because it is already so efficient"). However, i've heard so often that the tests on lame 3.90.3 were massive in numbers - did those also include many high-bitrate tests? I dont know, because i wasn't there, but i would guess yes, considering the reputation of preset standard? What about the pre 1.0 tests on vorbis? Were there many high-bitrate tests? And MPC? I think it would be interesting to get some info from long-time board-members, because i have the feeling the time where most of the majority of tests were done is a few years back in time. And during this "peak-time" codecs just became so good that ABXing them at high bitrates became very difficult? Insight from someone more knowledgeable would be greatly appreciated.

- Lyx
I am arrogant and I can afford it because I deliver.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #15 –
Quote
Quote
To summarize:

a) A "lossless to high-bitrate MP3 ABX" test = a "high-bitrate MP3 ABX" test. No need to specify lossless, all ABX tests are done against a lossless source by definition. A test comparing two codecs (again, against a lossless reference) would be an ABC test.
No 
mp3 ABX test does mean ABX vs original source (or lossless, it's the same), but nothing prevents you from ABXing mp3 vs mpc, mp3@128 vs mp3@256 and so on...


Okay, fair enough - around here, the ACCEPTIBLE process is to ABX vs the original, since ABXing two different lossy codecs doesn't tell you much of anything, just which one sounds better to you (which could be the OPPOSITE of which one is closer to the source. Some people like the sound of certain codecs).

High bitrate MP3 vs. Lossless ABX Tests?

Reply #16 –
Related to the first posts...
Without the intention to violate the AUP 8 with this assesment, I've heard artifacts using the onboard audio and onboard speakers of my laptop that were not audible using better equipment.

The same applies for the car, sometimes on the puter I can't hear some artifacts but on the car it is a different story, and some artifacts I hear on the car I can not perceive on the puter (I'm talking about the same file, but just decoded to CD-audio for play in the car).

I believe that the explanation behind this situation is that different equipment will create different masks, poor quality equipment might not have the fidelity needed at certain frequencies making a worse mask than that obtained with hi-fi equipment. Likewise, poor quality equipment might produce a higher mask within certain frequencies making some artifacts inaudible.

Just a thought thou.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #17 –
Quote
Okay, fair enough - around here, the ACCEPTIBLE process is to ABX vs the original, since ABXing two different lossy codecs doesn't tell you much of anything, just which one sounds better to you (which could be the OPPOSITE of which one is closer to the source. Some people like the sound of certain codecs).
[a href="index.php?act=findpost&pid=302949"][{POST_SNAPBACK}][/a]


No this is not correct... ABX tells you *nothing* about your personal preference. It only tells you can hear a difference between the two files you are ABXing. While you can ABX two lossy files against one another, after doing this you can only show people that you can hear a difference between the two files, not which one you prefer.

[edit] removed some unclarities
"We cannot win against obsession. They care, we don't. They win."

High bitrate MP3 vs. Lossless ABX Tests?

Reply #18 –
I'm new, hi everyone, i want to make a suggestion on this subject.

This is my slant:

In a few years ill be buying either an sacd or dvd-a player depending on the climate, which of course will be a wireless device and send the digital data straight to the amplifier, which will also be wifi connected to my media storage device, containing my music collection, very likely with storage in the realm of terabytes.

When i play my SACDs, and then flip to my mp3 encoded tracks, they are going to sound terrible. Even cds will begin to sound a little lifeless. I'm putting all my CDs on my HD with Flac, and thats that - it makes sense for the future, sooner or later we'll all have media devices instead of dvd players.

Don't spoil your expensive systems you'll buy in the future with a poor substitute for lossless when hard drives are getting so cheap and so big.

Soonish, I'm going to mirror raid myself up a terabyte archive beast (current costs = 8*250 gig HDs - Im waiting till i can run 4*500 gig hds without breaking the bank) and once ive got that there is no excuse for worrying about the space my music collection takes up. Some of you may have enormous collections, which causes a problem, but i still like the idea of wifi music throughout my house at CDDA quality with no cds in sight!!

This applies in this thread because I think these days we should be trying move away from lossy over processed codecs, to take advantage of this cheap storage with the highest quality media. What you pump out to portable devices could very easily be done in realtime, and set to encode different bitrates dependant on the device. So your ipod, your car and your house can sync up the audio library, but in whatever bitrates is best for that device.  Maybe, 320kbit in the car, 192 on the ipod, reflecting available storage on media devices.

When this sort of system becomes available, you'll want a good quality source.

Just my perspective, in respect of yours

High bitrate MP3 vs. Lossless ABX Tests?

Reply #19 –
yes yes, but what is the point of that? Okay, MP3 "a" sounds different than MPC "b". This means.... nothing.

So why would such a test be performed in the first place? Because the tester wants to prove that one of the two tracks sounds better. But of course, as you said, this just proves that they are DIFFERENT, not which is closer to the original. Tester however concludes that this difference supports their preference. A preference they are entitled, to, I might add, but it doesn't prove much else.

Give me another example of why you'd ABX two lossy copies of a file. I can't think of one.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #20 –
Quote
I'm new, hi everyone, i want to make a suggestion on this subject.

This is my slant:

In a few years ill be buying either an sacd or dvd-a player depending on the climate, which of course will be a wireless device and send the digital data straight to the amplifier, which will also be wifi connected to my media storage device, containing my music collection, very likely with storage in the realm of terabytes.

When i play my SACDs, and then flip to my mp3 encoded tracks, they are going to sound terrible. Even cds will begin to sound a little lifeless. I'm putting all my CDs on my HD with Flac, and thats that - it makes sense for the future, sooner or later we'll all have media devices instead of dvd players.

Don't spoil your expensive systems you'll buy in the future with a poor substitute for lossless when hard drives are getting so cheap and so big.

Soonish, I'm going to mirror raid myself up a terabyte archive beast (current costs = 8*250 gig HDs - Im waiting till i can run 4*500 gig hds without breaking the bank) and once ive got that there is no excuse for worrying about the space my music collection takes up. Some of you may have enormous collections, which causes a problem, but i still like the idea of wifi music throughout my house at CDDA quality with no cds in sight!!

Just my perspective, in respect of yours
[a href="index.php?act=findpost&pid=302961"][{POST_SNAPBACK}][/a]


*sigh*. No one has yet been able to prove they can hear a difference between a PROPERLY MASTERED CD (that's a hard thing to come by, unfortunately) and a DVD-A or SACD disc. If you can, please do so. If not, don't spout such misinformation. TOS #8 violation and whatnot.

I'm really getting sick of saying this. I think this is like the 5th time in 2 days.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #21 –
Quote
When i play my SACDs, and then flip to my mp3 encoded tracks, they are going to sound terrible.
[a href="index.php?act=findpost&pid=302961"][{POST_SNAPBACK}][/a]

God bless placebo - because otherwise, imagine the pain of paying for all that expensive equipment, and then noticing that it doesn't sound any better than normal audio-cds :-) (except of maybe multichannel tracks).

Did this post mean, that i fed a troll? :)
- Lyx
I am arrogant and I can afford it because I deliver.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #22 –
Hmm, there's a bit of confusion in this thread.

All of the MPEG verification tests were made against the original, which is the same thing to compare to as a bitwise-exact coding method.

All of the MPEG verification tests I'm aware of, which would include MPEG-1, MPEG-2 BC, MPEG-2 NBC, various and sundry MPEG-4 verification tests, have shown that at least some listeners appear to distinguish between the original and the coded at the test rate, even for rates like 256 and 192 for layers 1 and 2, and for 192 as Layer 3, if I recall correctly.

The CRC test for the american DAR project, the one done a long time ago when the audio quality was of the 160kb/s variety, also showed that 160kb/s PAC and some higher rate of (I think it was AC3, but I frankly don't remember) could be distinguished from the original.

So, I think it's confirmed via some very sophisticated tests, done by world experts, that the usual rate of most codecs is not "transparent" in the ABX sense, although MPEG has used a standard wherein anything above the 4.0 CCIR scale mark is considered 'acceptable'.
-----
J. D. (jj) Johnston

High bitrate MP3 vs. Lossless ABX Tests?

Reply #23 –
Quote
So, I think it's confirmed via some very sophisticated tests, done by world experts, that the usual rate of most codecs is not "transparent" in the ABX sense, although MPEG has used a standard wherein anything above the 4.0 CCIR scale mark is considered 'acceptable'.
[a href="index.php?act=findpost&pid=303569"][{POST_SNAPBACK}][/a]

Well, not only the bitrate average matters, but also the encoder and method. Is Fhg CBR competitive against LAME VBR at high-bitrates? How high was the amount of positive ABX-tests compared to total trials? What kind of samples were used?

The most sophisticated tests done by world-experts can only tell something about "what" they tested. If the tested encoder/settings were inferior compared to high-bitrate LAME VBR, then the tests only concluded just that: That with an inferior encoder some people were able to tell the difference.

- Lyx
I am arrogant and I can afford it because I deliver.

High bitrate MP3 vs. Lossless ABX Tests?

Reply #24 –
Quote
The most sophisticated tests done by world-experts can only tell something about "what" they tested. If the tested encoder/settings were inferior compared to high-bitrate LAME VBR, then the tests only concluded just that: That with an inferior encoder some people were able to tell the difference.

- Lyx

Well, in those tests, the encoders were built by the algorithm proponents, with everything on the line, based primarily on encoded quality. 

I don't think it's possible to get evidence at this point to show exactly what was tested, but the people doing the encoding were the inventors of the coding methods, and the ones with the most to lose if the test results weren't good. The encoders were generally not restricted to real time, not restricted to simple, and were supposed to show the "ultimate" quality.

That's all I can offer you.

None the less, I don't think there is a 128kb/s MP3 encoder that meets the "ABX transparent" standard, or at least I haven't meant one, even in tests where all of the degradations were reported as being over 4.0 on the scale, the ID for the 'reference' signal in an ABC/hr test wasn't even close to any reasonable random hypothesis.  The MPEG test results, at least, as well as the CRC results that are on much, much newer encoders, have been published. I don't know the citations offhand, but the results were not ambiguous in any real fashion.
-----
J. D. (jj) Johnston
 
SimplePortal 1.0.0 RC1 © 2008-2019