HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: Jax184 on 2011-03-03 22:32:29

Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-03 22:32:29
This is my first post, and I have no professional background in audio, but I love music and try hard to treat it with respect. So I offer you something I've been working on.

I've always had a problem with traditional A/B or ABX listening tests. They're great when the differences are pretty big, but as they get smaller it becomes harder and harder to remember precisely what one track sounded like when you're listening to another. I for one just can't think about audio with the kind of detail that I can with visual information. Now I will confess that by the time this becomes a problem, the quality difference is probably not very significant. But perhaps I'll get better equipment later and suddenly the difference will be more obvious. Or perhaps I'll play the file to someone with better ears and they'll find it unpleasant. Whatever the case, I wanted a way to detect even small differences in audio quality. My concern was not so much with being blind or even with determining which audio file sounded better, just to determine if there was a discernible difference in the first place. In my mind there's no point trying to say which file sounds better if they've just been proven to sound exactly the same. But this method could easily be performed under double blind conditions.

Anyway, I think I've come up with a solution. So far it's worked nicely in my tests, but it seems far too easy. I can't help but suspect I'm overlooking something. So I'm asking for your help to validate the technique. Tell me if you can see a flaw in the process.

My inspiration came from the classic method of inverting one stereo channel and combining it with the other to eliminate center channel vocals and allow analysis of the subtler instrumental stuff. I've heard it called OOPS or Out Of Phase Stereo. I always liked the elegance of it. But then I realized I could do something similar to analyze the impact of various levels of audio compression.

To start with I ripped a single track from a CD as a wav file, then created several versions of it compressed with the latest LAME library at different bitrates. VBR 0, CBR 320, CBR 128 and finally CBR 56 to provide a really glaring example if need be.
Then I loaded the wav into Audacity along with one of the compressed versions, carefully measured and cut off the padding created by the Mp3 compression so the two waveforms were aligned not simply to the second but to the very sample, inverted the compressed version and mixed it with the wav. This produced a file which contained only the portions of the audio which had been removed or altered by LAME during compression. It was really cool to listen to, a hissy scratchy sort of racket. But it's not the difference file itself I was interested in.

I then loaded the corresponding Mp3 back into audacity below this "difference" file. With the two in perfect waveform alignment, playing them together produced something which was mathematically identical to the source CD. But at any time during the playback I could mute the "difference" file and switch instantly to hearing just the Mp3. If there was a discernible quality difference between the Mp3 and the source CD, it would show up at the point of muting as a change in volume, background hiss, clarity, etc. But if the aspects of the file which had been removed or altered were imperceptible to my ears (and thus having no impact on audio quality,) there would be no change when I muted/unmuted the "difference" track.

So far my testing has revealed absolutely no detectable change on the half dozen songs I've tried at CBR 320 or VBR -0. To prove that the method's working though I also made 128 bitrate and 56 bitrate Mp3s. Those produce a dramatic drop in quality every time the "difference" track is muted, but sound just like the CD when it's unmuted. So it seems the method works correctly, and LAME works really damn good these days.

I've also found that it's much easier to detect an increase in quality from unmuting the "difference" track than a decrease in quality from muting it.

So what do you guys think? Has anyone else tried this before? Am I on to something here, or have I misunderstood something? I do hope I'm correct in this because the method is so simple and elegant. It only takes a minute to set up.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-03 23:02:26
It's a sighted test, so you cannot guarantee that the results will be free from expectation bias.

Using foobar2000's ABX utility provides exactly the same functionality but eliminates the possibility for expectation bias.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-03 23:07:19
Is it actually able to switch between a compressed and non-compressed version of a song in mid playback? I've never seen something which could do that without introducing a delay or other change of its own.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-03 23:09:55
Try it for yourself.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-03 23:19:12
For a moment I was quite excited, but every time I switch tracks in Foobar's ABX during playback there's a very distinct popping sound. 1 in 3 times there's even a jitter. I have a hunch that Audacity is waiting for the waveform to cross the center before it applies the mute, since it always pulls it off without a pop.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 00:34:52
Perhaps I can clarify my intent with this a little. I know things get rocky when you start to work outside of blind tests.

My major complaint with traditional testing is not the blindness but rather the reliance on memory. I know I can't trust my memory to accurately remember precisely how clear a given horn blast in a song was 3 and a half minutes previous when I fire up a track to compare with it. So what I'm trying to do is change the test from one relying on memory and a vague feeling of "which sounds better" to a test of the listener's ability to notice a change in the audio track as it plays. A change to pure perception without requiring recollection.

I further wanted to do it in a way that would be very simple and transparent, relying on basic tools that the person orchestrating the test would understand, instead of a magic box process that did god only knows what to the audio in the process.

I'm sure there are other methods to switch seamlessly between audio sources, but I'm not aware of any which work this well.

The part I find particularly interesting is that even with my sightedness, even with my unfair advantage, my knowledge of when the switches occurred and in which direction, even when fully expecting to hear a change, I found I was unable to. CBR 320 and VBR -0 produced no change when compared to the source CD that I could detect. If I had detected a difference I would have followed it up with a traditional ABX test. But since I didn't, I suspect that it means there isn't one. And that's something I wouldn't have felt comfortable saying if I had gone straight to a traditional ABX test, due to its reliance on memory.
Title: I think I've discovered a new listening test method
Post by: DVDdoug on 2011-03-04 00:42:55
I'll try to take a few minutes later to convert your "word problem" into regular math problem (or expression/equation).   

But if I'm following what you are doing, I think you are simply doing a normal A/B test  (sighted), comparing the original file to the MP3.  Except you've done some round-about operations to re-create the original, rather than just using the "original-original"...  Is that what you're doing?

BTW (this doesn't directly relate to what you are doing) - The sound of the difference file does not represent the difference in the sound!  The fact that you had to time-align the MP3 is a big clue as to why this doesn't work.    Try recording yourself and someone else reading the same sentence and subtract...  See if what you hear sounds like the difference in the two files.  Or record yourself twice and subtract.  Or, try subtracting two completely different songs (or different versons of the same song) and see if that sounds like the difference between the two files...    It turns-out that the difference sounds exactly like the sum (with two uncorrelated files). 
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 00:52:22
Yes, it is a roundabout way to do an a/b, but the most important detail is that it allows an instant and undistorted ability to change between the two sources. So far I've never encountered that anywhere else.

I'm afraid I'm a little unclear about your second part, but it's late for me and I'm on the verge of going to bed. I had to time correct them because the Mp3 compression pads the beginning and end of the file. The whole gapless playback problem. Once the beginning of the files were aligned, they remained aligned for the duration of the files. It's not like I had to alter the speed of one to keep it lined up with the other.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-04 00:56:05
If you cannot remember what something sounds like, how on earth are you going to know if you're listening to a altered version of it?

This is one of the classic excuses given by people who are afraid of ABX tests.  It simply does not hold water.
Title: I think I've discovered a new listening test method
Post by: FumbleFinger on 2011-03-04 03:12:02
Interesting. I've no professional background in audio, and never used anything like Audacity either.

But I can understand well enough what you've done there. I must admit I'm surprised to learn there's a problem finding hardware & software to switch instantly & silently between audio streams. Maybe the pros have stuff they don't noise about.

DVDdoug seems preoccupied with the fact that what you've come up with is still fundamentally an A/B comparison. Er... what should we be looking for instead? The only thing we're interested in here is any audible difference between A and B, to which end anything that simplifies comparison must surely be a useful addition to the toolkit.

I disagree with greynol's idea that your perceptions are somehow 'invalidated' simply because you have concious control of the source switch. Sure, they wouldn't constitute a strict scientific proof unless someone else toggled the input unseen by you. Even then you might be lying, and claim not to be aware of any switching at all. But obviously you yourself would know what if any differences you heard, and any sane person should believe what their own senses tell them, not someone else's convoluted & agenda-ridden arguments to the contrary.

As to the notion that you should be able to remember the audio quality of a sample for perhaps several minutes before you finally get to hear the alternative you want to compare it with...  Words (almost) fail me.

Human brains are very good at noticing subtle differences when there are real-time transitions - we've evolved that way because it's seriously pro-survival to know about changes going on around us. But when it comes to comparing the current environment with the memory of a similar previous one, our brains are heavily biased to find similarities, not differences. Evolution again, because matching & categorising the present against stored historical knowledge makes us smarter, more adaptive, and more likely to survive until parenthood.

In short, Jax184, I think you raise a very interesting issue. I'll be fascinated to know what the real professionals have to say.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-04 03:26:56
I disagree with greynol's idea that your perceptions are somehow 'invalidated' simply because you have concious control of the source switch.

I said nothing of the sort.

any sane person should believe what their own senses tell them

Unfortunately for your argument there are plenty of experiments that demonstrate just the opposite when it comes to audio.  You can begin by googling the McGurk effect.

As to the notion that you should be able to remember the audio quality of a sample for perhaps several minutes before you finally get to hear the alternative you want to compare it with...

Can you say straw man?
Title: I think I've discovered a new listening test method
Post by: saratoga on 2011-03-04 03:30:34
I disagree with greynol's idea that your perceptions are somehow 'invalidated' simply because you have concious control of the source switch.


I suspect you have never tried to compare audio sources in a blind test.  One's (non-audio) perception has an extraordinarily powerful effect on ones hearing.
Title: I think I've discovered a new listening test method
Post by: Polouess on 2011-03-04 09:00:12
If the glitches and jitter are the OPs main concern about foobars ABX tool, I suggest to try "Advanced ABX".
http://www.hydrogenaudio.org/forums/index....&pid=703733 (http://www.hydrogenaudio.org/forums/index.php?showtopic=80673&mode=threaded&pid=703733)
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 12:04:49
I am painfully aware of how fallible our senses are. I actually prefaced the page I wrote about this technique on my website with a discussion of optical illusions and such. But our memories are even more fallible. Ask any police officer, they'll tell you all about the 5 different unique stories that 5 different people who witnessed the same car crash will tell. So perception+memory has the potential to introduce far more mistakes than pure perception. I'm just trying to get it down to its simplest form. The sightedness of the method I outlined is entirely optional. You can blind it just as well. But I find it very interesting that even with my sight I still wasn't able to detect a difference in high quality Mp3s. I think there's something to be said for that.


Quote
Can you say straw man?


I'm sorry, but it sounded to me like that's what you were saying as well. Care to clarify?


That Advanced ABX looks interesting, but so far it's been a pain. It requires the two steams to be in the same format and have the same length. In other words, I need to do most of the work of the technique I outlined above, then export the files, download a seperate program and use it. Why not just keep going with Audacity at that point? To be a full replacement it would need to align the waveforms internally and only switch between them during a center cross.

BTW, I tried that McGurk effect video and found that it didn't work on me. Perhaps it was because of the subtle timing errors in that particular example, but I heard it as ba no matter what he was mouthing.


Title: I think I've discovered a new listening test method
Post by: FumbleFinger on 2011-03-04 14:06:11
I disagree with greynol's idea that your perceptions are somehow 'invalidated' simply because you have concious control of the source switch.

I said nothing of the sort.

Well, I don't want to get in a big argument about this - my main concern was to endorse the OP approach IF it's really true there's no existing easily-implemented way of seamlessly switching between audio sources for subjective comparison.

But you DID say any differences Jax184 perceived would be subject to expectation bias, and I assumed the implication was this would make his conclusions less meaningful. Which I don't agreee with. Apart from anything else, he could easily get a friend to do 'blind' source switching once he's established the basic principles.

As to the epistemological issues raised by perception of differences between current input and memory, I stand by my contention that memory is far less reliable, and should be factored out of any comparison process so far as possible. Obviously in some (often contrived or pathological) cases people tend to misinterpret what their own senses are telling them. But that's no reason to big up the role of memory, which I firmly believe is even less reliable.


I'm interested in this thread because I'd like to know if OP's method is technically sound (no pun intended!), and because it just seems odd to me if there isn't already a standard way of doing the seamless source switching. With disc space so cheap today, I don't really care how big my audio files are any more. But if I could prove to myself that I consistently hear the difference between 128kbps mp3 and lossless formats, I might want to identify my personal "equivalence threshhold" bitrate with a view to replacing inferior quality items already in my music library.
Title: I think I've discovered a new listening test method
Post by: 2Bdecided on 2011-03-04 14:19:50
It doesn't really matter whether you're switching between original and encoded using buttons marked A and B, or a button marked mute.

What matters is that you demonstrate you can tell a difference when you don't know what you're listening to and that includes there being no way you could be unconsciously led to know which one you're listening to. A double-blind test.

This is entirely absent from your sighted A/B test.

It's not that easy to prove you hear a difference in a simple A/B test even if you do get a friend to help. If you think it through, it's far more work (and far less reliable) than a straightforward ABX test.

That said, a simple sighted A/B, sometimes with no switch glitch, is one of the first things I usually try. Or sometimes I loop the two different versions, switch the monitor off, turn the volume down (so I lose track of which is which), then turn it back up again later. If I think I can tell which one is which, and/or hear a difference, I'll do an ABX.

Cheers,
David.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 14:28:35
Ahh, but that's what I'm trying to build! As I've said a few times now, this is the only method I am aware of which does not cause some form of distortion at the moment of transition. Nothing to inform the listener that the file being played has been changed, and nothing to mask the change in quality that might come with it. If you can point me in the direction of another piece of software which will take two versions of a track, put them into perfect alignment and switch between them with no pauses, changes in volume, pops, crackles, jitters or other artifacts introduced by the method, then I would welcome it as a far more straightforward method. But until then, this is still the only method I know of which can switch between two tracks in real time in such a clean manner that it can be used for blind testing.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 15:00:15
Here, how about I include a practical example to make things a bit clearer. Here's 15 seconds of a Sarah McLachlan song. In the first example it switches twice between a VBR -0 stream and the CD audio. See if you can spot it.
http://www.jax184.com/projects/Mp3s/Buildi...R%20Toggle).wav (http://www.jax184.com/projects/Mp3s/Building%20a%20Mystery/Building%20a%20Mystery%20Clip%20(VBR%20Toggle).wav)

In this second example it switches between a CBR 56k stream and the CD audio.
http://www.jax184.com/projects/Mp3s/Buildi...k%20Toggle).wav (http://www.jax184.com/projects/Mp3s/Building%20a%20Mystery/Building%20a%20Mystery%20Clip%20(56k%20Toggle).wav)

Spoiler (click to show/hide)


Notice how smooth the transition is? To my ears with a pair of Sennheiser HD-280s, there's no change at all in the first one, despite switching twice between different quality streams.

(PS, I hope I'm not violating any forum rules by linking to these)
Title: I think I've discovered a new listening test method
Post by: 2Bdecided on 2011-03-04 15:17:15
I agree it would be nice to have an ABX tool with inaudible switching like this.

However, I don't use ABX like that (I always play from start for each click), so haven't tried advanced ABX to see if it works.

Cheers,
David.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-04 17:10:41
I'm sorry, but it sounded to me like that's what you were saying as well. Care to clarify?

Allow me to add emphasis to the quote to which I responded:
As to the notion that you should be able to remember the audio quality of a sample for perhaps several minutes before you finally get to hear the alternative you want to compare it with...

With foobar2000's ABX, which is the most suitable test for checking encoders (as opposed to hardware-based ABX), the user can choose any clip of any duration and switch between the two versions of it.  Let's compare this with the method you currently have where the tester must be a different person from the testee and the tester chooses what and where the change is made without the testee being given the opportunity to hear any specific region in either version.  Until your method is made to be double-blind, perhaps exactly as what has been done with foobar2000's ABX comparator, it will not be as effective.

Ask any police officer, they'll tell you all about the 5 different unique stories that 5 different people who witnessed the same car crash will tell.

Tell them ahead of time that they will be witnessing an accident and will be given a video that perfectly captures it, any part of which they can replay any number of times they like and you'll see that your analogy quickly falls apart.

So far your only complaint with foobar2000 is a split-second glitch when transitioning.  Considering how many people report reliable results with this tool, many (most?) of whom probably never made use of the transitioning functionality, I think your complaint is quite minor.  Let's take a step back and actually consider the fact that any audible artifact can be pinpointed to an exact piece of audio.  It should then make sense to select just that point for testing by using the comparator's start and stop functionality.  If the audible artifact is something more general like the presence of a low pass filter, the same method can still be applied.  The piece of the sample selected for playback need not be longer than just a few seconds.

FWIW, I have dabbled with your method of subtracting an error signal in the past.
Title: I think I've discovered a new listening test method
Post by: Garf on 2011-03-04 17:54:41
For a moment I was quite excited, but every time I switch tracks in Foobar's ABX during playback there's a very distinct popping sound. 1 in 3 times there's even a jitter. I have a hunch that Audacity is waiting for the waveform to cross the center before it applies the mute, since it always pulls it off without a pop.


I would be good to check this, as simply unmuting the difference track should be identical to a seamless switch in foobar2000's ABX component. So your method must be doing something different if it never produces an audible distortion, which by the way is perfectly expected, because the instant switch can introduce frequencies not present in either original track.

Quote
I've also found that it's much easier to detect an increase in quality from unmuting the "difference" track than a decrease in quality from muting it.


You're doing this sighted, so your observation is completely and utterly unreliable. Please see TOS 8. Your brain is much more powerful than you think it is.
Title: I think I've discovered a new listening test method
Post by: SamDeRe81 on 2011-03-04 18:36:14
Alright listen, this guy http://www.youtube.com/watch?v=WmIAJeaKQys (http://www.youtube.com/watch?v=WmIAJeaKQys) already did what you're suggesting. I also saw someone else that inverted the lossless signal to compare to a lossy one using Audacity, so umm nothing new man
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-04 18:57:46
I don't see an error signal being mixed in and out to switch back and forth in that video, so no, that is not what Jax184 is doing.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 19:15:58
For a moment I was quite excited, but every time I switch tracks in Foobar's ABX during playback there's a very distinct popping sound. 1 in 3 times there's even a jitter. I have a hunch that Audacity is waiting for the waveform to cross the center before it applies the mute, since it always pulls it off without a pop.


I would be good to check this, as simply unmuting the difference track should be identical to a seamless switch in foobar2000's ABX component. So your method must be doing something different if it never produces an audible distortion, which by the way is perfectly expected, because the instant switch can introduce frequencies not present in either original track.

I don't know why, but in the dozen or so songs I've tested so far I've never heard an artificial pop or a click or anything of the sort when unmuting the difference track, but I do clearly hear a change in quality if switching between two files with sufficient differences. As I said, I'm assuming Audacity is waiting for a center cross before applying/removing the mute to avoid false frequencies. I'll see if I can dig up any information on it.


I've also found that it's much easier to detect an increase in quality from unmuting the "difference" track than a decrease in quality from muting it.


You're doing this sighted, so your observation is completely and utterly unreliable. Please see TOS 8.


So I'm not permitted to say I can't hear something?

Your brain is much more powerful than you think it is.


I am autistic, I was abused as a child, I went through a severe depression in my teens, and had to reassemble myself into a person after it was all said and done. That's none of your business and not relevant to this topic, but I want you to know that I'm very very much aware of how fallible our minds are. I Know our perception is highly distorted. I know our thought processes are strange things. You don't need to tell me. I live with the proof every day. Please don't tell me what I think.

Back on topic. The fact that a test is sighted does not automatically discredit its results. Especially if the results are negative. I am not saying I have used a sighted test to find a difference which accepted wisdom says shouldn't exist. Quite the opposite! I thought there would be a small difference if a really detailed test could be set up, but I didn't find one. To borrow an audio term, I will fully agree that the signal to noise ratio of sighted audio tests is very low. But in this case the outcome was no signal at all. Even when I expected to find something I still found I could not. I Think that's a legitimate result.

But all this ignores something of an important point. As I've said something like 3 times now, this test does not need to be sighted. Obviously I knew the answers to the first few runs as I was assembling the process. But after that there's no need for sightedness. I have already asked people to listen to files like the ones I linked to above, which switch sources at multiple points, and to report if they could detect a change in audio quality at any point. So far no one has been able to spot the transitions in the high quality samples, while the low quality ones have produced correct answers. To me, that suggests I'm on to something. But since I know how hard it is to be sure of these things, and because I know I'm not a professional audio engineer, I came here to ask for help. So can we please stop arguing over what I heard during the process of putting the method together and move on to what others hear with it under more controlled conditions?
Title: I think I've discovered a new listening test method
Post by: DonP on 2011-03-04 19:25:34
I wonder if the smoothness of Audacity is really that it does a soft transition (ramp up/down) when switching the difference track in and out.  If it turns out the mute transition takes a significant fraction of a second, that's messing with your memory more than a slight click.

Can you make audacity just switch between A and B tracks instead of muting one of them?  It seems like while adding the difference file back in works, it is prone to error in execution which could make some encoders/settings appear less transparent.



Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 19:39:25
I wonder if the smoothness of Audacity is really that it does a soft transition (ramp up/down) when switching the difference track in and out.  If it turns out the mute transition takes a significant fraction of a second, that's messing with your memory more than a slight click.

Can you make audacity just switch between A and B tracks instead of muting one of them?  It seems like while adding the difference file back in works, it is prone to error in execution which could make some encoders/settings appear less transparent.


Just ran a few tests and confirmed that the mute does not wait for a zero crossing. So then I suspect the reason I haven't heard any odd pops or such yet is because, so far, I've only been working with very similar files. That's interesting. I wonder why the other programs I used were having such a hard time with it?

If it's possible to just switch between two tracks I haven't found it yet. Would certainly simplify things.
Title: I think I've discovered a new listening test method
Post by: AndyH-ha on 2011-03-04 19:47:31
It isn’t so much the senses per se but expectation and belief that can make perception unreliable. This is by no means limited to hearing, it effects most, perhaps all the senses.

Wine “experts,” as well as most other people, tend to judge a wine superior when they’ve been told beforehand that it is: this glass contains rare and expensive wine (see, read  the bottle label), this one vin ordinaire. Sure enough they tastes that way to everyone -- behind the curtain, both glasses were actually poured from the same bottle.

The same principal applies to most things people purchase for different prices. People also often see (or do not see) what they have some reason to expect to see, even when it isn’t there (or actually is there) -- ask any stage magician if you haven’t been paying enough attention to your own life to know it is also true out on the street.

In the controlled laboratory situation, people are trained with different sounds and then test, blind, as being consistently able to identify them correctly. Then the subjects are given expectations about what they are about to hear and usually do believe they hear what they expect, even when a different sound is actually being played.

I much doubt this would apply to gross differences; church bells vs fire engines at normal volumes are unlikely to be so easily manipulated. These tests are with slight differences, small but real (as is the subject matter presently under consideration) that the subjects can correctly differentiate between when expectation and belief don’t foul up the process.

I suggest, if jax184 really wants to verify his results, he carry the tests a bit further, to an easy to manage ABX: A small audible difference is identified in Audacity (or wherever) with the sighted comparison. Around that difference, select the same short sample of both original wav and decoded-back-to-wav versions (generally just a few seconds), and do a real ABX test on those. This eliminates memory difficulties and removes sighted cues.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 19:53:01
I actually used the wine example in the article I wrote about this method on my website before I posted about it on this forum.

The point which so far exactly zero people reading this thread have noticed though is that I've said I did NOT find a difference where I expected to find one. I came here to announce my pleasant surprise at discovering Mp3s are really damn good, and the method by which I discovered this. The only differences I can find are with really crap low bitrate Mp3s which should also show up in traditional ABX testing.
Title: I think I've discovered a new listening test method
Post by: FumbleFinger on 2011-03-04 20:01:14
...so far exactly zero people ... noticed ... I did NOT find a difference where I expected to find one

I noticed. That's why I'm interested. You make some very cogent points, btw.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-04 20:41:26
If you chose a part of your adulterated sample to be played back as lossless where lossy would have exposed an artifact and all the parts chosen as lossy were otherwise transparent then clearly your test will have given the wrong conclusion.  By allowing the testees the ability to choose their own sample to be played back both ways for comparison this problem is eliminated.  Until this is incorporated into your method it is patently inferior to the testing methodology currently in place.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 20:43:32
This method would be pointless for comparing the same section under differing compression levels. We already have standard ABX testing for that. It's only suitable for searching for a change in quality at the moment of transition.
I seek to augment traditional ABX testing in times of very small differences, not replace it in general use.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-04 20:47:46
It's only suitable for searching for a change in quality at the moment of transition.

Which is arbitrary, not to mention that it is obvious that you really don't know for sure what's going on at the point that you think the transition is being made.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-04 21:11:07
It's only suitable for searching for a change in quality at the moment of transition.

Which is arbitrary, not to mention that it is obvious that you really don't know for sure what's going on at the point that you think the transition is being made.


I'm working on that. The whole reason I'm here is to find and correct any faults in the method, not to get browbeaten. I am only doing this because I enjoy music and want to make it easier to have high quality music to enjoy.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-04 21:28:06
I think you need to demonstrate objectively that the current system causes an under-reporting of positive results due to a less than perfect transition first if you have any hope that your method would get adopted.  Even before that, I think you need to demonstrate that there are artifacts that can only be distinguished resulting from a transition and that the artifacts aren't being caused by the transition itself; never mind the fact that people do not listen to music that has been encoded so that there are transitions between lossless and lossy, except for possibly LossyWAV.
Title: I think I've discovered a new listening test method
Post by: dhromed on 2011-03-04 21:38:38
Just ran a few tests and confirmed that the mute does not wait for a zero crossing. So then I suspect the reason I haven't heard any odd pops or such yet is because, so far, I've only been working with very similar files. That's interesting. I wonder why the other programs I used were having such a hard time with it?


I took a short clip and added a track with a pure sine wave, so that can hardly be called similar. I played it, switching the sine track on and off just as you have been doing.

I then exported both versions to WAV and ABXed then in foobar.

Foobar's ABXer switched much faster but produced a clear pop or tiny break on every single switch, while Audacity switced relatively slowly and only produced a barely perceptible pop occasionally. I currently assume it's performing a very short fade. This is beta 1.3.12, by the way.

I was unable to find a setting that related to buffering or fading. A quick google turns up nothing.

Edit:
This is trivial to create for anyyone here, but here's my samples anyway:
http://www.hydrogenaudio.org/forums/index....showtopic=87205 (http://www.hydrogenaudio.org/forums/index.php?showtopic=87205)
Title: I think I've discovered a new listening test method
Post by: FumbleFinger on 2011-03-05 02:49:33
people do not listen to music that has been encoded so that there are transitions between lossless and lossy

Surely vbr encoding inherently switches between less / more lossy within certain material, subjectively no different to lossless / lossy somewhere within the quality range we're mainly interested in here.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-05 03:16:30
subjectively no different to lossless / lossy somewhere within the quality range we're mainly interested in here.

On what basis?

Perhaps my point has flown over your head.  Please tell me how Jax184's sample preparation mimics real-world listening.  We either listen to files that are lossily encoded or losslessly encoded.  Objectively testing for audible problems between adjacent frames in a lossy file are only complicated by Jax184's method.  Anyway this is only one of many glaring flaws with this type of test; one to which I explicitly suggested people pay no mind.

Let me lay my opinion out plainly.  I don't care what people do in an effort to train themselves to hear or attempt to spot audible differences with lossy encoding.  At the end of the day the only acceptable means to communicate about audible differences on this forum is through the results of a properly controlled double-blind test.  If you guys want to do things like listen to difference signals, artifact amplification* or perform seamless switching back and forth between lossless and lossy in the hopes to magically find some artifact, that's great, but I think you're wasting your time.

As I stated earlier, audible artifacts can be precisely pinpointed to a particular event in the audio stream.  General artifacts caused by such things as overly aggressive low-pass filtering, high frequency synthesis or collapsing of the sound field can also be tied to any specific event in the audio stream.  If your memory is too poor to account for a couple of seconds of audio in order to detect a change, I don't how you can complain about degraded audio quality.  Rather, to me it sounds like a pathetic justification for a belief in placebo.  This rant isn't necessarily intended for anyone specific; those who take offense suffer from a guilty conscience.  I appreciate Jax184's desire to be free from expectation bias, though it would appear by his TOS8 violation as quoted by Garf that he is not free from expectation bias.  He is more than welcome to conduct an ABX test to confirm the differences he thinks he can hear when performing a sighted test, of course.  Regarding expectation bias, Woodinville nailed it when he said (I'm going to paraphrase) that not only can people be steered into hearing differences that don't exist, they can also be steered into thinking things sound exactly the same when audible differences do exist.

(*) while this hasn't been suggested here, training by listening to difference signals and artifact amplification has been raised in other discussions.
Title: I think I've discovered a new listening test method
Post by: Jax184 on 2011-03-05 03:58:35
This is absurd.

1. The test can be performed double blind! I've already performed it single blind. I've never set up a double blind test before, so I had hoped the people here would help.

2. I DID NOT HEAR A DIFFERENCE. I AM NOT CLAIMING TO HEAR A DIFFERENCE. CAN YOU HEAR THAT?? I heard NO DIFFERENCE. NONE. That was the unexpected result! I had come into the test expecting to hear a difference, but found I could not. I AM NOT CLAIMING TO HEAR A DIFFERENCE WHERE OTHERS CANNOT.

You know what, you guys are on your own. I came here because I wanted to share something cool and get help from people with more experience than me. Instead this is feeling like the "who's on first" routine with added attitude.
Title: I think I've discovered a new listening test method
Post by: Soap on 2011-03-05 04:08:28
You know what, you guys are on your own. I came here because I wanted to share something cool and get help from people with more experience than me. Instead this is feeling like the "who's on first" routine with added attitude.


Running off in a huff serves no one.
Dismissing the criticisms raised serves no one.
Instead of taking offense where none was intended how about defending "a new listening test method" against the (valid) critiques raised so far?
Would you rather praise from the blind or critical dialogue from the sighted?*


*bad pun. 
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-05 04:08:30
You're trying to reinvent the wheel when there is no credible evidence demonstrating that the current wheel doesn't work.

I've also found that it's much easier to detect an increase in quality from unmuting the "difference" track than a decrease in quality from muting it.
This needs to be ABXed.  Doing so is quite simple:

1) Create a ten second adulterated file where you have five seconds of lossy transitioning to five seconds of lossless.

2) Create a ten second file over the exact same section of audio that is pure lossless.

3) ABX the two files. 

To make sure you aren't simply hearing the transition:

4) Create a ten second file over the exact same section of audio that is pure lossy to be ABXed with the file that is pure lossless.

5) Submit your results in addition to the two files for peer-review.  The files themselves should all be in a lossless format that are time aligned (already covered) and have not had their levels adulterated beyond what was done by the lossy encoder.  I suppose the lossless to lossy transition can be cross-faded which is probably preferable to one that is abrupt.

I wanted to [...] get help from people with more experience than me.

You're getting help; just that you need to know how a plane works before you try to fly one.

Instead this is feeling like the "who's on first" routine with added attitude.

I wouldn't feel the need to repeat myself if you would simply acknowledge what are legitimate criticisms.
Title: I think I've discovered a new listening test method
Post by: sthayashi on 2011-03-05 06:51:04
This is absurd.

1. The test can be performed double blind! I've already performed it single blind. I've never set up a double blind test before, so I had hoped the people here would help.

I believe that the theory can be done simply.  The actual implementation may not be as simple, but I'm not a software guy.

Diff+subject is known to be A. Subject is known to be B.  The computer randomly selects either Diff+subject or subject alone to be X.  You have 3 transition buttons A<->X, B<->X, A<->B.  A<->B mutes or unmutes the diff file.  A<->X and B<->X mutes/unmutes if appropriate or does nothing if not.  You don't know which.  The task: reliably identify whether X is A or B.

The implementation details?  Well, you're responsible for creating the diff and subject files and ensuring that playback can happen exactly anticipated.

Functionally, it is identical to a traditional ABX test.  In practice, it's a different way of getting the job done.  Rather than switching audio streams, you're simply modifying a single audio stream.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-05 07:54:30
You also need buttons to make sure playback is begun as either A, B or X, correct?

The more I consider my previous reply the more I am absolutely sure you need to cross-fade the transition!  If the transition button creates an audible click then you know the results straight away.  This is a very real problem since lossy encoders don't care about preserving sample amplitudes and is often demonstrated in the "If Lame is supposed to be gapless, how come I hear clicks when the track changes?" threads.  There are also the DBT skeptics who will dismiss null results on the grounds that the samples had to be mixed and would therefore not be "bit-perfect", though I could care less about that.  Add the ability to select a range for playback and except to also say that the controls may be confusing for some, I can't think of any other objections to this method of making such a test double-blind.

Will it result in more people being able to hear the difference than traditional ABX?  The jury is out, but I doubt it.  If data comes back indicating that I'm wrong I will gladly apologize for being a skeptic.
Title: I think I've discovered a new listening test method
Post by: Garf on 2011-03-05 09:05:34
That's interesting. I wonder why the other programs I used were having such a hard time with it?


As already stated, there is no guarantee that such a switch could be seamless. I would be much more included to believe Audacity is messing with the signal (fadein/fadout) rather than that the foobar2000 component for example has obvious bugs.

Edit: Ah, I see that someone already confirmed that what I said is exactly right...So not only is your method flawed, it produces a result opposite of what your goal was!

I've also found that it's much easier to detect an increase in quality from unmuting the "difference" track than a decrease in quality from muting it.


You're doing this sighted, so your observation is completely and utterly unreliable. Please see TOS 8.


So I'm not permitted to say I can't hear something?


You already forgot what you said yourself? You did in fact claim to be hearing a difference but didn't substantiate it through a reliable method.

There's a reason TOS 8 has been there since the beginning of this site: it's because the founders knew and had experienced beforehand that discussions based on flawed, nonscientific methods like yours could never possibly go anywhere.

Quote
You know what, you guys are on your own. I came here because I wanted to share something cool and get help from people with more experience than me.


Quote
So what do you guys think? Has anyone else tried this before? Am I on to something here, or have I misunderstood something? I do hope I'm correct in this because the method is so simple and elegant. It only takes a minute to set up.


You asked for feedback if you were on to something, and that's what you got.

Title: I think I've discovered a new listening test method
Post by: Garf on 2011-03-05 09:13:57
By the way, the ABX program included with LAME (I'm not sure if it still is), but which only worked under Linux, supported many additional modes like comparing files with the differences added and amplified, or only listening for the differences, so it would most probably already handle what you want.

It only supported that to help one to nail down the differences before doing a proper test, of course.
Title: I think I've discovered a new listening test method
Post by: FumbleFinger on 2011-03-05 15:12:55
This is absurd.

Couldn't agree more.

I started following this thread because I have a peripheral interest in finding a simple tool to help me identify my personal threshold for gaining any benefit from mp3 encoding with higher bitrates. Jax184's method of toggling between two samples with bitrates close to the threshhold seems like a potentially user-friendly way to approach this problem.

Having looked into ABX comparisons, I'm left with the impression they're targetted at making statistically valid pronouncements about whether most people can detect differences between two samples. This may well be scientifically valid, but speaking as a man with only one working ear (which is getting on a bit, I'm afraid), I don't care much what other people can or can't hear.

Presumably this thread will either wither away or get deleted / locked soon, flirting as it does with TOS #8. Which saddens me, because I feel Jax184's input would have been an asset to the forum.

I appreciate that many of the people who do have specialist skills and/or 'inside information' here at HA are busy people, and don't always have the time or inclination to bring others up to speed. But I really wish the forum could be more welcoming and informative.
Title: I think I've discovered a new listening test method
Post by: dhromed on 2011-03-05 15:47:56
Having looked into ABX comparisons, I'm left with the impression they're targetted at making statistically valid pronouncements about whether most people can detect differences between two samples. This may well be scientifically valid, but speaking as a man with only one working ear (which is getting on a bit, I'm afraid), I don't care much what other people can or can't hear.


If you perform a complete ABX session, you can make a valid pronouncement about whether you can detect differences between two samples. It's a completely isolated experiment. Have you actually tried it?

This has nothing to do with "most" people. "Most" people is simply an aggregate of of all the ABX tests done by a lot of people.
Title: I think I've discovered a new listening test method
Post by: gaekwad2 on 2011-03-05 16:14:25
Having looked into ABX comparisons, I'm left with the impression they're targetted at making statistically valid pronouncements about whether most people can detect differences between two samples. This may well be scientifically valid, but speaking as a man with only one working ear (which is getting on a bit, I'm afraid), I don't care much what other people can or can't hear.


If you perform a complete ABX session, you can make a valid pronouncement about whether you can detect differences between two samples. It's a completely isolated experiment. Have you actually tried it?

This has nothing to do with "most" people. "Most" people is simply an aggregate of of all the ABX tests done by a lot of people.

I think a lot of people are getting confused by the whole subjective vs. objective argument. ABX is a subjective test. It's testing whether a person's subjective perception is able to reliably detect a difference.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-05 18:23:33
I think a lot of people are getting confused by the whole subjective vs. objective argument. ABX is a subjective test. It's testing whether a person's subjective perception is able to reliably detect a difference.

Subjective in that results are subject to an individual's specific abilities, sure, but that is not the way the word is typically used with respect to ABX's ability to eliminate expectation bias.  Unlike ABC(/HR), ABX does not provide for the ability to rank test samples and as such is merely an objective measure of one's ability to spot differences.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-05 18:32:48
This may well be scientifically valid, but speaking as a man with only one working ear (which is getting on a bit, I'm afraid), I don't care much what other people can or can't hear.

Having the ability to hear with just one ear which may or may not be limited has no bearing in this discussion.

Having looked into ABX comparisons, I'm left with the impression they're targetted at making statistically valid pronouncements about whether most people can detect differences between two samples.

It appears that you really don't understand that ABX is a personal test and that no individual test can negate the possibility that an audible difference might actually exist, whether it be for the same person under a different set of circumstances or for someone else entirely.  Have you ever used foobar2000's ABX plugin?  If not then I respectfully suggest that you refrain from contributing to this topic until you do.
Title: I think I've discovered a new listening test method
Post by: FumbleFinger on 2011-03-07 00:54:35
...If not then I respectfully suggest that you refrain from contributing to this topic until you do.

Well I suppose I will, but I must respectfully point out that I didn't get involved in this topic in order to dispense my pearls of wisdom. I was interested in what Jax184 had to say, but I ended up feeling that he was effectively shouted down.

I'd still rather see this forum adopt a more welcoming and inclusive attitude.
Title: I think I've discovered a new listening test method
Post by: 2Bdecided on 2011-03-07 09:28:23
The more I consider my previous reply the more I am absolutely sure you need to cross-fade the transition!
If you were testing the audibility of inverting the signal, that would pretty much destroy the test. You'd get brief silence cross fading original>inverted, but not original>original, or inverted>inverted.


I think ABX is fine - with the choice of doing a cross fade, an instantaneous-but-glitch-free switch, or a glitchy/stopped transition. The tester should know the pros and cons of each, and the log should report which was used.

Cheers,
David.
Title: I think I've discovered a new listening test method
Post by: NullC on 2011-03-07 15:53:05
I agree it would be nice to have an ABX tool with inaudible switching like this.
However, I don't use ABX like that (I always play from start for each click), so haven't tried advanced ABX to see if it works.


Squishyball does glitchless switching:  http://svn.xiph.org/trunk/squishyball/ (http://svn.xiph.org/trunk/squishyball/)

It's not hidden from the user— you tell it which file to play, but you can flip back and forth without any discontinuity (or, if you like, it can insert silence or a beep in the switch).

I use a patched up version for codec development which can play the difference signal. (Handy when the files are almost identical except for a few points— I'll use difference mode to set a loop around a difference then blind A/B as normal)
Title: I think I've discovered a new listening test method
Post by: googlebot on 2011-03-07 16:15:41
Squishyball implements "glitchlessness" by just inserting silence into the transition. This can still add correlated pops. A short cross-fade can deliver much better results. I have implemented a tool for all kind of advanced transitions once. But partly due to the choice of Java it wasn't easily integrable into the existing playback ecosystems. It never found that much interest, so I moved on.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-07 17:27:22
If you were testing the audibility of inverting the signal, that would pretty much destroy the test. You'd get brief silence cross fading original>inverted, but not original>original, or inverted>inverted.

If one is trying to invert the signal by mixing an error signal it better be done in 32-bit float.
Title: I think I've discovered a new listening test method
Post by: 2Bdecided on 2011-03-07 19:31:55
If you were testing the audibility of inverting the signal, that would pretty much destroy the test. You'd get brief silence cross fading original>inverted, but not original>original, or inverted>inverted.

If one is trying to invert the signal by mixing an error signal it better be done in 32-bit float.
Really? Apart from the fact that digital full scale +ve is one less than digital full scale -ve, inversion is lossless with 16-bit integers.

As for the cross fade - pass!

Cheers,
David.
Title: I think I've discovered a new listening test method
Post by: greynol on 2011-03-07 19:35:44
Simple example:
-0.6 requires summing 1.2 in order to get 0.6.

Is is not clear to everyone that the OP is interested in summing an error signal in order to swap versions?

But you're right though, David, a single method of seamless switching will not always be possible depending on the samples being tested.  As I said earlier, if the transition itself poses an audible difference and can be used to determine which is which, clearly we no longer have a blind comparison.
Title: I think I've discovered a new listening test method
Post by: NullC on 2011-03-07 20:38:23
Squishyball implements "glitchlessness" by just inserting silence into the transition. This can still add correlated pops. A short cross-fade can deliver much better results. I have implemented a tool for all kind of advanced transitions once. But partly due to the choice of Java it wasn't easily integrable into the existing playback ecosystems. It never found that much interest, so I moved on.


Er.  It has three modes:

  -B --beep-flip          : Mark transitions between samples with a short beep
  -M --mark-flip        : Mark transitions between samples with a short period of silence
  -S --seamless-flip  : Do not mark transitions between samples; flip with a seamless crossfade (default)

(of course, these can be switched at runtime too)

The seamless is as it says, its a crossfade using a power complementary window.

I toyed around with nudging the transition point to the position with the least difference but couldn't find any realistic samples where it made a difference.

For ABX  the seamless option has the risk of making it possible to distinguish X from A/B based on transition artifacts, so the beep/mark options are probably better there.


Title: I think I've discovered a new listening test method
Post by: googlebot on 2011-03-07 20:53:25
Oh, ok. I only knew -B and -M.