Does subtracting MP3 from WAV reveal artifacts?

greynol, have you even listened to the square wave example? If you try his method with the V5 square wave + V5 noise you obtain the original PCM square wave, and by muting the noise you will instantaneously notice the artifacts. This is not about ABX.

Sure and I've created quite a few difference files of real music as well. I would never be so daft as to suggest that this is how one should judge the quality of a perceptual encoding, however.

At the music conservatory, students don't take ear tests out of the blue, just like that, without any training and practice. In fact they train for years and their abilities grow enormously. Musical dictation, chord identification, interval comparison are very difficult without training.

People don't really know it but I am a musician and come from a family of musicians. I understand what you're talking about but as far as this discussion is concerned it's a non-sequitur.

So don't expect people to be proficient at ABX tests without proper training, listening time, controlled conditions etc.

I don't.

I believe listening to sound differences are a proper tool, one of many, that can teach or least indicate what kind of noises and artifacts to look for in an ABX test. I'm not trying to replace your ABX testing method that you seem to praise so wildly, i'm giving you something to complement it.

Do you have any proof that your method works? Even something anecdotal would be fine. Right now we have HibyPrime looking quite foolish covering his ass by asking how to perform tests that he says are helpful which he's never actually conducted. Perhaps you'll be able to tell him that he isn't wasting his time, though the same can't be said for those who have had to read what has transpired since you decided to resurrect this discussion.

>"praise so wildly"
Really?!? Do you know some alternative method of performing an objective evaluation of the sound quality of perceptual encoding that isn't double-blind which is deserving of any praise at all?

Does subtracting MP3 from WAV reveal artifacts?

Reply #54 – 2010-11-08 04:52:22

I believe listening to sound differences is a proper tool, one of many, that can teach or at least indicate what kind of noises and artifacts to look for in an ABX test. I'm not trying to replace your ABX testing method that you seem to praise so wildly, i'm giving you something to complement it.

His point is that you are basing this whole "tool" on one absurdly artificial example (square waves aren't music!) and trying to generalize it to all music based on sighted tests and a suspect understanding of what an encoder actually does. Thats not a good start.

I really don't know what happens when you use LAME to encode a square wave, and I don't really care because its a silly example of something the encoder was never meant to do and should not be expected to work correctly. I suspect that if you took greynol's advice and tried this with a sample thats actually very hard to ABX you would find it doesn't work nearly so well in a proper test. Of course if it does then you're welcome to "give" it all to us, but one thing at a time

Does subtracting MP3 from WAV reveal artifacts?

Reply #55 – 2010-11-08 05:07:21

Heaven forbid we try this with something encoded at -V2.

What happens with a difference sample where the RMS levels can be as high as -25dB, if not higher?

Such an encoding can't possibly be transparent, now can it? Spoiler (click to show/hide)

Does subtracting MP3 from WAV reveal artifacts?

Reply #56 – 2010-11-08 05:37:21

I've taken too much of my time trying to explain this very simple concept, by providing countless examples founded on mathematical theorems that have been known for centuries, and yet you seem not to have the slightest grasp of what so many people understand instinctively. What do i get? A reference to a Wikipedia article (A+), and the same bottomless counterarguments over and over.

Well if someone else wants to take up the torch, be my very welcome guest. Now, I'll leave you two to your metaphysical world in which nothing can be falsified and where everything has a shade of greynol.

Does subtracting MP3 from WAV reveal artifacts?

Reply #57 – 2010-11-08 05:45:21

I've taken too much of my time trying to explain this very simple concept, by providing countless examples founded on mathematical theorems that have been known for centuries,

Since I think you're misunderstanding everyone else here, we are not saying superposition is wrong, just saying its not a universal property of systems. This is one of those things you learn in an introductory signals course. You can apply superposition only to linear systems. Try applying to a system in which linearity is violated (such as one in which masking effects can occur) and you'll see that you get the wrong answer!

This should be obvious. If masking occurs, then superposition cannot hold. Think about it.

What do i get? A reference to a Wikipedia article (A+), and the same bottomless counterarguments over and over.

Did you take a look at the wikipedia article? I really do think it would help you. I'm not interested in these arguments, but if you have questions I'm willing to take the time to explain these things to you. They're quite easy once you get your head around them.

Quote from: JapanAudio on 2010-11-08 04:02:01

Well if someone else wants to take up the torch, be my very welcome guest. Now, I'll leave you two to your metaphysical world in which nothing can be falsified and where everything has a shade of greynol.

I'm afraid I don't follow you here. I'm just suggesting that you need to apply these " mathematical theorems that have been known for centuries" correctly, not that they don't exist or something like that.

Does subtracting MP3 from WAV reveal artifacts?

Reply #58 – 2010-11-08 05:57:51

This is a great idea. But make sure you convert all files to WAV before importing them to Audacity; this way they will line up perfectly by themselves. So do WAV<->MP3 conversions with foobar. Play noise + MP3, mute noise as you wish, notice the difference.

Converting the mp3 back to wav doesn't seem to remove the line-up problem. I must be doing it wrong - is there any way to convert to MP3 and back without exporting?

Quote from: greynol on 2010-11-08 04:07:10

Quote from: HibyPrime on 2010-11-08 03:30:04
Ok, so here is what I did, hopefully someone more knowledgeable than I will decide if this methodology is sound.

Sighted tests are absolutely not sound.

Foobar2000 + ABX plugin or something that performs the same function is what you need.

Hmm, I prefer using both methods personally. I wouldn't want to limit myself to only one source of information, that would be just bad practice

Quote

The two of you seem so unwilling to discuss, let alone make use of double-blind testing. Why is this?

Thats a two way street, saying that we're unwilling to discuss.

I'm new here, I'm not used to everyone expecting double blind tests to be done before you are taken seriously - I'm not saying thats a bad thing, just that it's a bit foreign to me. I do use double-blind tests, I just don't consider them the end-all be-all of what is true. Also blind testing myself solves one problem and introduces another; many people feel 'pressure' to make sure they get the right answers in a blind situation, and since I have anxiety problems to begin with, ABX testing only makes that worse.

Anyway, BACK ON TOPIC: Audacity is giving me some weird issues, I'll save a file then open it a minute later and the track is silent when it shouldn't be, so it's making it difficult to try out multiple songs... Aside from that, I still managed to get Joe Satriani's Always with me, Always with you into the method I described earlier at 256Kbps. The difference is small, but still clear enough for me to hear 'something' different, not enough to say it's any better or worse - but enough to say it is there. The difference is most clear in the maracas throughout the song. After noticing this, I fired up foobar and ABX'ed away. Due to reasons described above, I failed miserably.

I also added the original flac track into the same audacity project as the difference and MP3 tracks. I accidentally discovered that if you solo and mute a track, it still plays that track as if it was solo'd, not sure why this is. So I solo + muted the FLAC track, and checked/unchecked the solo button, and could not hear any difference between the FLAC track and the difference+MP3 tracks combined.

Does subtracting MP3 from WAV reveal artifacts?

Reply #59 – 2010-11-08 06:04:09

Also blind testing myself solves one problem and introduces another; many people feel 'pressure' to make sure they get the right answers in a blind situation, and since I have anxiety problems to begin with, ABX testing only makes that worse.

Just repeat the test until its routine enough that you don't feel anxious.

Aside from that, I still managed to get Joe Satriani's Always with me, Always with you into the method I described earlier at 256Kbps. The difference is small, but still clear enough for me to hear 'something' different, not enough to say it's any better or worse - but enough to say it is there. The difference is most clear in the maracas throughout the song. After noticing this, I fired up foobar and ABX'ed away. Due to reasons described above, I failed miserably.

If you can't ABX it I would not assume you can hear a difference. It sounds like its in your head.

I also added the original flac track into the same audacity project as the difference and MP3 tracks. I accidentally discovered that if you solo and mute a track, it still plays that track as if it was solo'd, not sure why this is. So I solo + muted the FLAC track, and checked/unchecked the solo button, and could not hear any difference between the FLAC track and the difference+MP3 tracks combined.

Not sure what you're asking here? Do you mean to ask if adding the difference to the MP3 gives you the lossless copy back? It does, at least to within rounding error. FWIW this is actually a good part of how quite a few lossless codecs work. FWIW I'd probably use something a little easier to compare samples in though. IMO matlab works great for this if you have access to it and are familiar with its syntax.

Does subtracting MP3 from WAV reveal artifacts?

Reply #60 – 2010-11-08 06:11:38

Thats a two way street, saying that we're unwilling to discuss.

Perhaps if you brought a new argument to the table which is actually valid, then I'm sure it would be discussed. So far I haven't seen a new and valid argument, at least not to my satisfaction. I recommend using the forum's search function. The fact of the matter is that many of us aren't interested in hashing over the same tired subject.

I'm new here, I'm not used to everyone expecting double blind tests to be done before you are taken seriously

That's the rule to which you agreed in order to participate on this forum.

Also blind testing myself solves one problem and introduces another; many people feel 'pressure' to make sure they get the right answers in a blind situation, and since I have anxiety problems to begin with, ABX testing only makes that worse.

It would be much easier if you simply accept that the differences you think you hear during a sighted test are imagined rather than conjure excuses. For those who insist that the differences are night and day, I believe these excuses should be summarily dismissed without a second-thought.

Anyway, BACK ON TOPIC: Audacity is giving me some weird issues

Actually this is off-topic.

Quote from: greynol on 2010-11-08 04:47:53

Due to reasons described above, I failed miserably.

I think that placebo effect isn't allowed to work in an ABX test is the most likely reason here.

Does subtracting MP3 from WAV reveal artifacts?

Reply #61 – 2010-11-08 06:45:47

oy.. Started with logical fallacies, now we're at name calling?

I'm stepping out here, it was fun while it lasted (not really).

Does subtracting MP3 from WAV reveal artifacts?

Reply #62 – 2010-11-08 07:00:41

Name calling???

FWIW, I did edit the first paragraph in my previous post but I can assure you that I didn't call you any names. If anything the edit resulted in a tougher stance than what was originally there.

Does subtracting MP3 from WAV reveal artifacts?

Reply #63 – 2010-11-08 07:26:41

Right now we have HibyPrime looking quite foolish covering his ass by asking how to perform tests that he says are helpful which he's never actually conducted. Perhaps you'll be able to tell him that he isn't wasting his time

Quote from: greynol on 2010-11-08 07:00:41

Name calling???

FWIW, I did edit the first paragraph in my previous post but I can assure you that I didn't call you any names. If anything the edit resulted in a tougher stance than what was originally there.

Does subtracting MP3 from WAV reveal artifacts?

Reply #64 – 2010-11-08 07:35:21

Telling you that you're looking foolish and calling you a fool are two different things. Perhaps you don't like my language, but I can't think of a more fitting description of your behavior, especially considering your evasion of the questions I asked you earlier:
http://www.hydrogenaudio.org/forums/index....st&p=730222

FWIW, I look foolish more often than I'd like. The trick is to learn from your mistakes.

Does subtracting MP3 from WAV reveal artifacts?

Reply #65 – 2010-11-08 08:52:56

Greynol, please never give up your tireless dedication to logic and reason on this board. And I'm not talking about sequencers. The whole battle is reminiscent of humanities struggle between superstition and science except for a change, here, science is winning. Bravo.

Does subtracting MP3 from WAV reveal artifacts?

Reply #66 – 2010-11-08 10:42:39

Quote from: Alexey Lukin on 2010-11-07 21:21:44

I've taken too much of my time trying to explain this very simple concept, by providing countless examples founded on mathematical theorems that have been known for centuries, and yet you seem not to have the slightest grasp of what so many people understand instinctively. What do i get? A reference to a Wikipedia article (A+), and the same bottomless counterarguments over and over.

You should have spent this time to learn. Basic DSP theory (you are so proud of) has little in common with actual problems present in audio quality assessment. Once you don't know what the problem is and don't respect people trying to explain you the situation the conversation has no sense.

Does subtracting MP3 from WAV reveal artifacts?

Reply #67 – 2010-11-08 12:39:31

1. Check if the signals are aligned in time.

LAME takes care of this.

Quote

2. Phase spectrum of FFT is not the same as "phase information".

Well then, I'd like to know what exactly do you mean by "phase-preserving quantization", and how it can be proven mathematically.

Quote

You have to understand what you are looking for and take care of several things: weighting windows; analysis of spectrum sparsity; only paying attention to phase of significant spectrum components; etc.

I'm looking for a difference, and I see difference. You are right that I probably had to take amplitude spectrum into consideration as well (if amplitude component is close to zero, phase component is practically meaningless).

At least I believe it is obvious that mp3 does not preserve the phase spectrum in exact sense. The question is how significant the introduced difference is, in the context of this thread.

For what I know, phase spectrum distortion is almost universally inaudible, yet it can result in significant change of the waveform shape. So if mp3 encoding can, at least potentially, indroduce phase distortion resulting in such changes, then it would invalidate the "test" being discussed even further. That's why I'd like to clarify this issue.

Does subtracting MP3 from WAV reveal artifacts?

Reply #68 – 2010-11-08 12:51:21

Quote from: JapanAudio on 2010-11-08 00:52:24

OK. I did the experiment suggested by Arnold B. Krueger, with the square wave, and i think it has been most revealing in terms of favorably validating the statement "Subtracting MP3 from WAV reveals artifacts". Judge for yourself:

http://www.multiupload.com/YDPTV9XPH8

Can't download it. I keep getting offered a download of "whitesmokewirter.exe".

Quote

Method: An "ideal" PCM square wave is generated with MATLAB @ 440Hz (although a mathematical continuous reconstruction of the signal bears a pseudo-square wave suffering from aliasing.) The wave is then converted to LAME CBR 320kbps (best quality), and V5 VBR 130kbps ('threshold' quality). Finally, both signals are subtracted from the original PCM source to (hypothetically) reveal their artifacts.

I don't know if the Lame parameters are optimal, but at least this is some kind of practical example.

Quote

Observations:
1. MATLAB PCM square wave rings @ A440, sounding very digital.

Likely a false claim. If you do this experiment properly there will be ringing on the 440 Hz square wave, but the ringing will be at the Nyquist frequency which in this case is 22050 Hz. There is a considerable history of people comparing square waves coded 44/16 and comparing them to square waves generated in the analog domain with accurate sources. Results of any reasonable subjective comparison has always been random guessing.

Quote

2. LAME 320kbps MP3 sounds indiscernibly like the original PCM signal.

OK.

Quote

3. LAME V5 130kbps sound very similar to the original PCM signal, with a slight added 'hush'.

If memory serves, 128 kbps is not generally considered to be a sufficient bitrate for reliable transparent coding in general.

Quote

4. Error signal PCM-320kbps is inaudible, explaining the absence of audible discrepancies described in (2), thus supporting transparency in this instance.

My perspective on this is that we know that if *all* kinds of error are small enough, then audible differences for sure won't be heard.

However, some errors are far more audible than others, so this kind of a criteria will turn out to be overkill in many cases.

For example, putting *all* errors 100 dB down is always sufficient to *guarantee* audible transparency, but putting them *only* 80 dB down generally gets the job done. OTOH, errors that are 60 dB down will often be audible. But, if I get to pick the kind of error, I can put it only 60 dB down and everybody will be fooled all of the time. An example of an error that is 60 dB down is a broadband 0.1 dB level shift. Nobody ever hears it. Period.

Quote

5. Error signal PCM-V5 sounds exactly like the noisy 'hush' described in (5), supporting the fact that artifacts are indeed exposed by this method, in this instance.

This statement may be true, but it is a very narrow statement.

Because of the extreme variation in the audibility of various common kinds of errors, simple signal subtraction is not a very efficient way to rate audio equipment and software. I think that it is even a far worse criteria than THD, certainly no better.

Does subtracting MP3 from WAV reveal artifacts?

Reply #69 – 2010-11-08 12:59:44

Well then, I'd like to know what exactly do you mean by "phase-preserving quantization", and how it can be proven mathematically.

Consider quantization of waveform samples of a signal. This operation is considered phase-preserving because there's nothing that can flip or rotate phase in it. This quantization can be considered as "result = original + quantization noise", where quantization noise is typically small.
Mp3 is similar: it splits the signal into a few hundred subband signals and quantizes them. There's nothing that can flip or rotate phases in this operation.

I'm looking for a difference, and I see difference. You are right that I probably had to take amplitude spectrum into consideration as well (if amplitude component is close to zero, phase component is practically meaningless).

Yes, that's what I meant.

At least I believe it is obvious that mp3 does not preserve the phase spectrum in exact sense.

Of course it doesn't. But even adding a tiny amount of noise alters the phase spectrum of the signal. By "preserving phase information" I mean "preserving phase information of significant spectral components, down to a quantization step". There's no strict definition of this concept.