Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: New to empicism (Read 15933 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

New to empicism

Hi all,

I'm conducting a test as part of my degree course to research any perceived differences in analog and digital audio. I'm proposing a double-blind within subjects ABX test with both vinyl and CD stimuli over a number of genres, recorded to 24-bit/96kHz (audible clicks/pops removed) and administered in a controlled listening environment (studio containing Blue Sky 5.1 monitor system and Pro Tools HD with D-Command console).

Like the subject header says, I'm new to all this, so was just wondering if anyone had any advice/constructive criticism for me please?

New to empicism

Reply #1 –
My advice is don't bother. There will be no problem successfully ABXing vinyl vs, CD, and all you will be left with is asking people's preference for one over the other.

New to empicism

Reply #2 –
I'm sure this must have been already done, but wouldn't ABXing a digital recording of a vinyl recording against the vinyl recording itself be worthwhile?

New to empicism

Reply #3 –
Quote from: knucklehead on
I'm sure this must have been already done, but wouldn't ABXing a digital recording of a vinyl recording against the vinyl recording itself be worthwhile?

That's been tried, with an analog signal either going through ADC -> DAC or directly, and they were unable to ABX it.

New to empicism

Reply #4 –
Quote from: pdq on
Quote from: knucklehead on
I'm sure this must have been already done, but wouldn't ABXing a digital recording of a vinyl recording against the vinyl recording itself be worthwhile?

That's been tried, with an analog signal either going through ADC -> DAC or directly, and they were unable to ABX it.


I'm not at all surprised it's been done, and that those were the (worthwhile) results. 

New to empicism

Reply #5 –
I think an interesting research question would be to try to figure out what is behind people's preferences.

Analyze what kinds of distortion are present in playback from Vinyl and simulate each of those with your digital system and see which enhance preference and which detract from it.

It would also be fun to do an experiments where you play back a digitized version of the Vinyl with and without a turntable visible to the listener. I would hypothesize that the visual cue will increase preference for Vinyl.

New to empicism

Reply #6 –
jj (aka Woodinville on this forum) has spent some time on another forum discussing what must be involved for the proper execution of such a test: http://thewombforums.com/showthread.php?t=14462

You should PM him at your earliest convenience.

New to empicism

Reply #7 –
Thanks for all your replies I really appreciate it.

Quote from: pdq on
Quote from: knucklehead on
I'm sure this must have been already done, but wouldn't ABXing a digital recording of a vinyl recording against the vinyl recording itself be worthwhile?

That's been tried, with an analog signal either going through ADC -> DAC or directly, and they were unable to ABX it.


Would you have any idea where I might find info on this?

Quite honestly, the results don't have to be that worthwhile - it's an assignment for my degree, although I'm already beginning to see the advantages of objective testing - it's my first test, and I really just have to come up with A result to support my hypotheses, null or otherwise. Having said that, the test has kind of mutated a bit...it's now looking like a double-blind between vinyl/CD/3 different types of MP3 bitrates, with CD as the reference, and the experimental question being something along the lines of "Have various format changes affected the perceived quality of audio, are we sacrificing quality, and what future implications could this have?"

Obviously articulated a little better in the end  I can't stress enough that I'm new to all this!

Thanks again

New to empicism

Reply #8 –
Quote from: Joefish on
Thanks for all your replies I really appreciate it.

Quote from: pdq on
Quote from: knucklehead on
I'm sure this must have been already done, but wouldn't ABXing a digital recording of a vinyl recording against the vinyl recording itself be worthwhile?

That's been tried, with an analog signal either going through ADC -> DAC or directly, and they were unable to ABX it.


Would you have any idea where I might find info on this?


The JAES paper about listening tests related to  high resolution audio

New to empicism

Reply #9 –
You should take note that when you compare CD to Vinyl, you're not only comparing the different mediums but also two completely different masters of the same material. This would render any conclusion regarding the effects of the medium invalid.

A solution would be what notat suggested: take the CD as source and simulate the vinyl's distortions. This would require a careful analysis plus the development of a DSP though.

New to empicism

Reply #10 –
Quote from: Northpack on
You should take note that when you compare CD to Vinyl, you're not only comparing the different mediums but also two completely different masters of the same material.

Are all CD's "completely different masters of the same material"?
Quis custodiet ipsos custodes?  ;~)

New to empicism

Reply #12 –
Quote from: 2tec on
Quote from: Northpack on
You should take note that when you compare CD to Vinyl, you're not only comparing the different mediums but also two completely different masters of the same material.

Are all CD's "completely different masters of the same material"?

I don't know if I understand your question... of course you could take the vinyl master and make a straight digital transfer from that, but noone would ever do that (exept for when the original master hasn't survived), because vinyl masters are designedly distorted to ideally "fit" on the medium. You don't know the exact(!) amount of distortion applied to them, so you can't restore the original tape Master's sound. On the side of the digital master, you have level differences and most probable some EQ (not to mention dynamic compression, digital cleanup etc.) - so you do most certainly come out with a completely different master compared to the vinyl.

New to empicism

Reply #13 –
Quote from: Arnold B. Krueger on
The JAES paper about listening tests related to high resolution audio
From this study;
Quote
... one trend became obvious very quickly and held up throughout our testing: virtually all of the SACD and DVD-A recordings sounded better than most CDs—sometimes much better. Had we not “degraded” the sound to CD quality and blind-tested for audible differences, we would have been tempted to ascribe this sonic superiority to the recording processes used to make them.
~ 4 A Note On High-Resolution Recordings

The authors found a difference in the manner in which recordings were engineered. Hi-resolution recordings received much more care and attention. If the analog record was based on a better recording than the digital CD / FLAC / MP3 version, I suspect I would prefer the analog to digital despite the advantages that digital medium has over the analog.
Quis custodiet ipsos custodes?  ;~)

New to empicism

Reply #14 –
Quote from: Joefish on
so was just wondering if anyone had any advice/constructive criticism for me please?

May I suggest looking up the scientific method and trying to write down your "scientific hypothesis" and particularly what is being measured. As others have said, if the signal on the record and CD are different then you are not only measuring the influence of more than one parameter you are also probably not in a position to say what they are. This effectively makes the experiment a write off due to poor design. Probably not a good idea if you want reasonable marks.

Recording the vinyl to 24/96 or 16/44 PCM, compressing to MP3 and then comparing record vs MP3 vs PCM would be a more reasonable experiment even though it has probably been done a zillion times before.

New to empicism

Reply #15 –
Quote from: Northpack on
A solution would be what notat suggested: take the CD as source and simulate the vinyl's distortions. This would require a careful analysis plus the development of a DSP though.

There are many off-the-shelf plugins that emulate various analog distortions - vinyl, tape, tubes.

New to empicism

Reply #16 –
Quote from: 2tec on
Quote from: Northpack on
You should take note that when you compare CD to Vinyl, you're not only comparing the different mediums but also two completely different masters of the same material.

Are all CD's "completely different masters of the same material"?


*All* is a big word. It's an open, uncontrolled industry. Enough different CDs have been made that every possible combination of mastering steps has been mixed and matched at least once.

If indistry scuttlebutt is true, then some commerical CDs were created from tapes that were intended to be directly cut into LP dies.

This is about as bad as it gets.

New to empicism

Reply #17 –
Quote from: 2tec on
Quote from: Arnold B. Krueger on
The JAES paper about listening tests related to  high resolution audio

Isn't this the JAES paper about ABXing a digital (SACD/DVD-A) signal going through a 16 / 44.1 ADA process? Is there a JAES study ABXing a CD versus a LP?


I don't see why anybody would bother ABXing a CD to a LP because so many variables are uncontrolled.

New to empicism

Reply #19 –
Quote from: Arnold B. Krueger on
Quote from: 2tec on
Quote from: Arnold B. Krueger on
The JAES paper about listening tests related to  high resolution audio

Isn't this the JAES paper about ABXing a digital (SACD/DVD-A) signal going through a 16 / 44.1 ADA process? Is there a JAES study ABXing a CD versus a LP?


I don't see why anybody would bother ABXing a CD to a LP because so many variables are uncontrolled.


Well, unless you make the CD from the LP.

Then you have to get time alignment right to under a millisecond or so, or you'll create cues that people WILL notice sometimes, and you'll get a false positive.
-----
J. D. (jj) Johnston

New to empicism

Reply #20 –
Subjective tests of minute differences tend to be large, time-consuming and difficult.

What about testing for the threshold when e.g. 50%*) of a random selection of listeners can hear any difference when degrading SACD/DVD-A/LP with a CD-like ADC/DAC using a variable number of bits and sampling rate?

If a practical treshold is e.g. 13 bits and 32kHz (using "good" dithering and resampling), then readers of your work can make their choices about how much headroom above that they want.

The fact that 3 persons out there may or may not be able to distinguish 16 bits from 15 bits using specially selected "killer samples" is interesting for some problems. For other problems, knowing that most of us, most of the time cannot reliably distinguish 14 from 13 bits may be most interesting.

If you can do the test PC-based, I really have interest in "ladder-tests", in which the test stimuli are served adaptively based on previous responses, meaning that the "psychometric function" is sampled more heavily close to the point where interesting things happen.

-k

*)I'drather not get into the statistics, but depending on the test, "a significantly larger number of positive outcomes than that suggested by chance"

New to empicism

Reply #21 –
Quote from: knutinh on
Subjective tests of minute differences tend to be large, time-consuming and difficult.


Depends on what you call large, time-consuming, and difficult. IME doing the tests is easy. Convincing the audiophiles of the world is another matter.

Quote
What about testing for the threshold when e.g. 50%*) of a random selection of listeners can hear any difference when degrading SACD/DVD-A/LP with a CD-like ADC/DAC using a variable number of bits and sampling rate?

If a practical treshold is e.g. 13 bits and 32kHz (using "good" dithering and resampling), then readers of your work can make their choices about how much headroom above that they want.


I don't think it was the first time it was done, but I demonstrated this at my old and now departed www.pcabx.com web site in or about 2002.

Quote
The fact that 3 persons out there may or may not be able to distinguish 16 bits from 15 bits using specially selected "killer samples" is interesting for some problems.


Reality is that you can easily do a non-represenative test that shows that most people can distinguish 15 bits from 16 bits, and you can also do any number of representative tests that show that nobody can distinguish 15 bits from 16 bits. 

Unfortunately, the world is full of people who *will* do something that amounts to being *no test at all* that  "shows" that anybody and everybody can "hear the difference" between 24 bits and 16 bits. Just do a sighted evaluation, prefereably non-level-matched. ;-)


New to empicism

Reply #22 –
Quote from: Arnold B. Krueger on
Depends on what you call large, time-consuming, and difficult. IME doing the tests is easy. Convincing the audiophiles of the world is another matter.

Doing a naiive, inconclusive test is often easy, but has limited value.

Doing a conclusive test of something that is allready proven (or easy to prove) may also be easy.

Doing a conclusive test of something that no-one has ever observed before, may be difficult (assuming knowledge Darwinism here).

Doing an inconclusive test that is solid enough to make believers question their beliefs is very hard.


I still believe that the more marginal a phenomenon is (1 in 1000 can hear it in 1 of 1000 sound-clips etc), the more time-consuming it is to prove its existence, and the less value the results will have in real-life (except used as arguments in audiophile debates).

-k

New to empicism

Reply #23 –
Quote from: knutinh on
Quote from: Arnold B. Krueger on
Depends on what you call large, time-consuming, and difficult. IME doing the tests is easy. Convincing the audiophiles of the world is another matter.

Doing a naiive, inconclusive test is often easy, but has limited value.


One would have to be pretty ignorant of the facts to say that the general run of DBT debunking golden ear mysticism has been naive or inconclusive.

Quote
Doing a conclusive test of something that is allready proven (or easy to prove) may also be easy.


What rational people find to be highly conclusive tests have no effect on people who are primarily motivated by raw emotion and irrational thinking.

Quote
Doing a conclusive test of something that no-one has ever observed before, may be difficult (assuming knowledge Darwinism here).


What is there about audio that has not already been observed?

Quote
Doing an inconclusive test that is solid enough to make believers question their beliefs is very hard.


What rational people find to be highly conclusive tests have no effect on people who are primarily motivated by raw emotion and irrational thinking.

There are a virtually unlimited number of examples of irrational behavoir in the world. Mao is dead, but there are still Maoists!

Quote
I still believe that the more marginal a phenomenon is (1 in 1000 can hear it in 1 of 1000 sound-clips etc), the more time-consuming it is to prove its existence, and the less value the results will have in real-life (except used as arguments in audiophile debates).


The 1 in 1000 audiophiles and the 1 in 1000 sound clips almost certainly don't exist. By now, 10,000s or 100,000s of each have been tested. It is like tested unplugged toasters. Out of a million conventional toasters, how many will toast bread without being plugged in? None!  How many runners can run a 2 minute mile? None! There are things that are simple to observe, even naively observe, and they simply never happen!

If you want to test 2 million toasters looking for that one toaster that will toast bread without being plugged in, please be my guest! But do so with your own time and money. Don't waste my time with your hopes and dreams of being successful at doing the impossible. No matter what your high school coach told you, there are some things that are simply impossible. Actually, a great many of them exist.

New to empicism

Reply #24 –
Quote from: Arnold B. Krueger on
What rational people find to be highly conclusive tests have no effect on people who are primarily motivated by raw emotion and irrational thinking.

No matter what my personal beliefs are, and what rules I follow in my day-to-day life, I still think that science should be conducted based on high standards. It is impossible to prove that God does not exist. We sceptics can use all of our brain-power, reasoning, arguments, etc, but proving that something does not exist is impossible.

What we can do is conduct tests that tries to prove Gods existance, and plead with believers that they do the same. I believe that the current state in the "does God exist?" question, is that science cannot prove either, and those that choose to believe should do so based on faith, not science. I have no problem with "faith" and "science" existing as two orthogonal cultural systems that can work independently ("I dont care about physics, using this expensive cable makes my experience more pleasant"). Problems do occur when people try to mix them in strange ways ("This cable sounds better than the other due to the skin-effect").
Quote
The 1 in 1000 audiophiles and the 1 in 1000 sound clips almost certainly don't exist.

I agree, many "audiophile phenomena" almost certainly dont exist as physical, audiological phenomena. But Sir Rayleigh was almost certain that Darwin was wrong based on physically sound theories of the earth being no more than 6000 years old. Physicists around 1900 was almost certain that all there was to discover in physics was allready discovered. Etc. I think that the sceptic making false claims about what science can disprove is making the same disservice to the public as the believer making false claims about what science has proven.

-k