This post is to discuss two issues I'm experiencing (or rather, hearing).
All test are listened on Foobar2000. MP3 are encoded with VFR slow mode.
1. @ 128Kbps VBR MP3 (lame 3.98.2) sounds better than 110Kbps (NeroAAC, q 0.4) AAC?!?! I thought AAC > MP3???
2. Lame's V5 (around 130Kbps) sounds pretty much transparent to me. I'm using a pair of head direct RE0, which is supposed to be an incredibly revealing pair of earphones.
I tested it with Kelly Clarkson's Breakaway, Train's Drops of Jupiter, and Tchaikovsky's 1812 overture. I didn't do a blind test, I knew which one is which and I switched back and forth.
As much as a want to think my lossless one sounds better, I couldn't. I honestly could not hear a difference.
That can't be right. People say that 128 is horrible, especially when I have revealing earphones like my RE0. There are people on head fi saying that they can easily hear the difference between 192Kbps MP3 and FLAC with their RE0.
Are my ears broken? I think my hearing isn't that bad. I listen at around 55dB all the time (even on the subway thanks to Shure's excellent tips). I don't think I have much hearing lost.
Without an ABX test, there's no way of telling if your described discrepancy is real or imagined.
Per the ToS #8, there is no such thing as "sounds better" (http://www.hydrogenaudio.org/forums/index.php?showtopic=3974#entry149481) until you can prove it.
That being said, I'm not at all surprised that a well-encoded MP3 is (apparently) transparent to you; that sounds about normal, in fact. The people telling you can hear the difference between a FLAC file and a well-encoded MP3 file are lying to you.
1. The mp3 files had a bitrate of at least 18kbps higher than the AAC ones. That is a rather large difference. When comparing lossy encoders, you need to use the same bitrate ranges. For example, comparing a 128kbps AAC file with a 130kbps mp3 file would be more than acceptable that testing files with an 18kbps difference.
2. And? That just means that the lossy encoder (Lame) is doing its job. That doesn't mean that your ears are borken or that you don't appreciate the music. It just means that Lame is performing as it should.
You really should conduct blind ABX tests though as sighted tests are flawed by the placebo affect. Additionally, dedicated users here will tell you to determine what lossy encoder and setting is right for you. Only you have the combination of your music, equipment, listening environment, and ears. Personally, I find Lame at -V 5 to be quite transparent for a bunch of my content when I am casually listening to the songs. Lastly, this definitely isn't the first (and it won't be the last) time people at head fi were spreading bad advice, making up audio quality stats, or formulating outrageous statements. Nothing is wrong with your ears. Enjoy your music at -V 5. At least you will be able to fill your portable player/computer with more songs than those people who worry about carrying around PCM WAV files (because they think FLAC and mp3/AAC/WMA/OGG sound bad) on their 2GB portables.
Without an ABX test, there's no way of telling if your described discrepancy is real or imagined.
Per the ToS #8, there is no such thing as "sounds better" (http://www.hydrogenaudio.org/forums/index.php?showtopic=3974#entry149481) until you can prove it.
That being said, I'm not at all surprised that a well-encoded MP3 is (apparently) transparent to you; that sounds about normal, in fact. The people telling you can hear the difference between a FLAC file and a well-encoded MP3 file are lying to you.
The MP3 vs AAC test is very obvious. The MP3 retained bass and treble far better than NeroAAC that telling which one is which is incredibly easy. Again, my bias is with AAC, and my mind is telling me that MP3 is better.
If anything is imagined, it should be me imagining the one I bias against sounding better, and in both of my case, I couldn't imagine that. My mind is telling me the opposite. I think blind test is unnecessary when under bias I still favor the other one.
I'm just shocked that such a low bitrate MP3 can make me think it is transparent, especially when my earphones are of very high quality.
The MP3 vs AAC test is very obvious.
We don't care.
Make an effort to demonstrate that you're willing to comply with TOS #8 or go away. It's that simple.
The MP3 vs AAC test is very obvious.
We don't care.
Make an effort to demonstrate that you're willing to comply with TOS #8 or go away. It's that simple.
Can anybody recommend a way to do a blind test without involving another person?
Install foobar2000 with ABX plug-in option.
http://www.foobar2000.org/components/view/foo_abx (http://www.foobar2000.org/components/view/foo_abx)
Install foobar2000 with ABX plug-in option.
http://www.foobar2000.org/components/view/foo_abx (http://www.foobar2000.org/components/view/foo_abx)
I got the plugin loaded. How do I use it?? I'm using Foobar 1 beta 4 portable version (like 0.9.4.5's interface a lot better).
The people telling you can hear the difference between a FLAC file and a well-encoded MP3 file are lying to you.
I used to think so too, and my dad told me there was a difference. I said, 'riiiiighhht' and one day I set up a little experiment with a whole crapload of formats (128, 192 and 256 mp3's, a ~500kb ogg and the original lossless). In a few double blind tests, my dad was able to put them in order of bitrate without even hesitating every time (it was pretty obvious to me also, listening with him). Anyways... his stereo is a little nuts and he even buys into some of that snake oil garbage, but an experiment is an experiment.
Now I say that anyone who says you can't hear the difference between lossy and lossless just needs better equipment.
So maybe your ears aren't the problem, chengbin. What's your source on those RE0's?
Now I say that anyone who says you can't hear the difference between lossy and lossless just needs better equipment.
...and what you say about needing better equipment is actually a myth, but whatever.
Do you have any other details to give us to go along with your anecdote (ABX logs, encoder(s) used and what settings, etc.)?
This may be obvious to those who do this regularly, but what double-blind protocol allows one to order multiple files by bitrate exactly?
I got the plugin loaded. How do I use it??
Load two files, select them and right-click. You'll see the option to ABX them under Utils.
Do you have any other details to give us to go along with your anecdote (ABX logs, encoder(s) used and what settings, etc.)?
Nope. Not logs or nothing. I wasn't trying to prove anything, just make my dad look like an idiot. I just renamed them randomly and then burned the randomly named files in a random order a few times over and then after the experiment checked the logs to see what was what.
Undoubtedly, you could find something wrong with the experiment, but it changed my views.
Feel free to dismiss my anecdote, greynol. I may have come on too strong with my statement; this is not an argument I care to start, just one man's opinion.
...and what you say about needing better equipment is actually a myth, but whatever.
Though this does make me curious as to what kind of system you do your tests on?
It doesn't matter what system I use since I don't claim to have golden ears. What I have read by those who do is that high end equipment doesn't help (this is an anecdote, of course). Lossy encoding models assume a flat frequency response upon playback (this is not an anecdote). On the flipside however, systems and/or environments with high levels of noise can help in masking some artifacts.
Here are a couple of links:
http://www.hydrogenaudio.org/forums/index....mp;#entry621426 (http://www.hydrogenaudio.org/forums/index.php?showtopic=70382&st=0&p=621426&#entry621426)
http://www.hydrogenaudio.org/forums/index....mp;#entry539257 (http://www.hydrogenaudio.org/forums/index.php?showtopic=60144&st=0&p=539257&#entry539257)
Qest, the "better equipment" gambit is a fallacy because the compression artifacts are reproducible with any decent equipment, even "cheap" ones.
AFAIK there are people here that have ABX'd correctly certain 320 kbps MP3 samples, and I don't think they use audiophile equipment (correct me if I'm wrong). They just know what to hear.
...and what you say about needing better equipment is actually a myth, but whatever.
If you have an equipment which is not able to reproduce above 16kHz without audible attenuation or FR distortion, then you can hardly tell a difference between two encodings which differ mostly in the 16kHz+ range (as is usually the case with low vs. hi bitrate lossy)...
If you had an equipment which does 20kHz within +-3dB tolerance, it might be, in this context, called a better equipment and it would be needed to tell the difference (provided you are able to hear above 16kHz).
I got the plugin loaded. How do I use it??
Load two files, select them and right-click. You'll see the option to ABX them under Utils.
I feel so stupid. I looked at every menu, but forgot right click...
The ABX is very interesting. The fact that I can switch the file right away is so useful.
Anyways, I did it for Kelly Clarkson's Breakaway 3 times. I got it right 2/3 times. I *think* I heard a difference, but it might have been my brain, which is why I think I guessed the first 2 right. But honestly, I couldn't hear a difference. They sounded 99.99% same for me.
...and what you say about needing better equipment is actually a myth, but whatever.
If you have an equipment which is not able to reproduce above 16kHz without audible attenuation or FR distortion, then you can hardly tell a difference between two encodings which differ mostly in the 16kHz+ range (as is usually the case with low vs. hi bitrate lossy)...
If you had an equipment which does 20kHz within +-3dB tolerance, it might be, in this context, called a better equipment and it would be needed to tell the difference (provided you are able to hear above 16kHz).
I don't think earphones can go up to 20kHz +-3dB. The RE0 is by far the treble earphone, extending higher than any other earphones out there, but I don't think it'll reach 16kHz without some dropping. I've seen many earphone FR graphs that are like -30dB at 20kHz.
Time to grab a tone generator to see how high I can hear.
EDIT: I'm surprised. I can only hear up to 18kHz square wave.
Time to grab a tone generator to see how high I can hear.
EDIT: I'm surprised. I can only hear up to 18kHz square wave.
This is not the proper way to test for this. What matters is the ability to hear those high frequencies in real material with other frequencies present at the same time.
What you should do is take a piece that you know has lots of high frequencies. Then apply lower and lower low-pass filters until you can hear the difference. You will probably find that this occurs several kHz below where you can hear a pure tone.
Time to grab a tone generator to see how high I can hear.
EDIT: I'm surprised. I can only hear up to 18kHz square wave.
This is not the proper way to test for this. What matters is the ability to hear those high frequencies in real material with other frequencies present at the same time.
What you should do is take a piece that you know has lots of high frequencies. Then apply lower and lower low-pass filters until you can hear the difference. You will probably find that this occurs several kHz below where you can hear a pure tone.
I can't edit my post my post for some reason. I don't know why I wrote square wave. I used sine waves.
I can hear perfectly at 18kHz. At 19kHz I couldn't hear anything (unless I turn my speakers way up).
I really don't think I know any songs with >18kHz material.
Lossy encoding models assume a flat frequency response upon playback (this is not an anecdote).
Exactly, and therefore one can imagine what boosting the treble does wonders when ABXing lossy codecs...
Of course you can cheat with an equalizer, or maybe the "better equipment" isn't so good after all and amplifies or distorts high frequencies.
Anyways, I did it for Kelly Clarkson's Breakaway 3 times. I got it right 2/3 times. I *think* I heard a difference, but it might have been my brain, which is why I think I guessed the first 2 right. But honestly, I couldn't hear a difference. They sounded 99.99% same for me.
Most proper blind ABX trials include about 10-15 "guesses" per song. Going a total of 3 times doesn't show anything. You could have correctly guessed or you could have actually been hearing a difference. Now, getting a 9/10 or 14/15 means that you likely heard a difference (assuming you are comparing the source lossless file with the lossy version) and the lossy file doesn't sound as good. Additionally, as previously said, you need to test your frequency hearing abilities with actual music and not just sound waves (whether they are square or not). People can often hear tones up in the ~19KHz range but they hear music up around the 16KHz range. That is a big difference.
I don't think earphones can go up to 20kHz +-3dB. The RE0 is by far the treble earphone, extending higher than any other earphones out there, but I don't think it'll reach 16kHz without some dropping. I've seen many earphone FR graphs that are like -30dB at 20kHz.
With headphones, you can't just compare to a flat frequency response but you need to measure it relative to some other curve whose name I forgot (but it is far from a flat line and is probably a function of the speaker's distance from the ear). Measuring this is really complicated for headphones.
Time to grab a tone generator to see how high I can hear.
EDIT: I'm surprised. I can only hear up to 18kHz square wave.
http://www.audiocheck.net/audiotests_frequencycheckhigh.php (http://www.audiocheck.net/audiotests_frequencycheckhigh.php)
Try this test instead of the square wave.
Anyways, I did it for Kelly Clarkson's Breakaway 3 times. I got it right 2/3 times. I *think* I heard a difference, but it might have been my brain, which is why I think I guessed the first 2 right. But honestly, I couldn't hear a difference. They sounded 99.99% same for me.
Most proper blind ABX trials include about 10-15 "guesses" per song. Going a total of 3 times doesn't show anything. You could have correctly guessed or you could have actually been hearing a difference. Now, getting a 9/10 or 14/15 means that you likely heard a difference (assuming you are comparing the source lossless file with the lossy version) and the lossy file doesn't sound as good. Additionally, as previously said, you need to test your frequency hearing abilities with actual music and not just sound waves (whether they are square or not). People can often hear tones up in the ~19KHz range but they hear music up around the 16KHz range. That is a big difference.
Can you recommend some songs that has music with that high of a frequency?
I'll do a 15 guess ABX test in the evening. I got to go somewhere later, and watch a movie after.
I don't think earphones can go up to 20kHz +-3dB. The RE0 is by far the treble earphone, extending higher than any other earphones out there, but I don't think it'll reach 16kHz without some dropping. I've seen many earphone FR graphs that are like -30dB at 20kHz.
With headphones, you can't just compare to a flat frequency response but you need to measure it relative to some other curve whose name I forgot (but it is far from a flat line and is probably a function of the speaker's distance from the ear). Measuring this is really complicated for headphones.
Time to grab a tone generator to see how high I can hear.
EDIT: I'm surprised. I can only hear up to 18kHz square wave.
http://www.audiocheck.net/audiotests_frequencycheckhigh.php (http://www.audiocheck.net/audiotests_frequencycheckhigh.php)
Try this test instead of the square wave.
I can start to hear 19kHz (tingle in my ear feeling). At 18kHz it is very "loud" (I can hear it very clearly)
Please read this:
http://www.hydrogenaudio.org/forums/index.php?showtopic=7516 (http://www.hydrogenaudio.org/forums/index.php?showtopic=7516)
It's a sticky at the top of the General Audio forum. The title says "Read before posting".
and this:
http://www.hydrogenaudio.org/forums/index....showtopic=16295 (http://www.hydrogenaudio.org/forums/index.php?showtopic=16295)
You see that an ABX run of only three trials is meaningless.
Here are some samples of real music with various low passes applied (mustang.flac):
http://ff123.net/samples.html (http://ff123.net/samples.html)
Point 1 looks like a TOS 8 violation, but the rest of the OP's post is saying "I can't hear a difference", and the 3-trial ABX was followed by "I can't hear a difference" - so I'm not sure we need to chase ABX results there.
@chengbin, your experience is normal. The audible differences are subtle - not usually night-and-day. People who always "hear" night-and-day differences between any two pieces of audio equipment (cables, amplifiers, CD tranports, and codecs at near-transparent bitrates) are usually imagining these differences.
People adopt this terminology from hi-fi magazines, which try to convince you that this year's must-have is night-and-day better than what was around last year. If things were really improving so quickly, they must have been truly awful ten years ago! (they weren't ) So, don't expect night-and-day differences. For many people saying they hear such things, it's a case of The Emperor's New Clothes.
There are general but subtle audible problems in the <128kbps region with mp3 - quite serious problems with some specific kinds of sounds. Once you give the format room to "breathe" by going above -V5 these problems become so subtle that, apart from those "problem" signals, it's mostly transparent to most people.
btw, equipment with a horrible irregular frequency response can reveal artefacts that equipment with an accurate flat frequency response will not reveal. If you boost a narrow frequency region by 10-20dB, it's going to reveal coding noise that was otherwise masked. This is a fault of the equipment, not the codec. Such equipment isn't "revealing" - it's trash.
One exception: Headphones aren't supposed to have a flat frequency response - they're supposed to simulate the frequency response of the part of the system that's missing (i.e. speakers > outer ear) which, due to the shape of the head and outer ear, isn't flat at all. That's OK - that won't break any codec - that just makes them as close as possible to normal listening.
Cheers,
David.
One exception: Headphones aren't supposed to have a flat frequency response - they're supposed to simulate the frequency response of the part of the system that's missing (i.e. speakers > outer ear) which, due to the shape of the head and outer ear, isn't flat at all. That's OK - that won't break any codec - that just makes them as close as possible to normal listening.
I've often wondered how much of an impact this has in helping people successfully ABX with headphones.
I've often wondered how much of an impact this has in helping people successfully ABX with headphones.
If the resulting overall frequency response is comparable with what you'd get from speakers (which is the idea) then it shouldn't make any.
What is hugely different is the impulse response (i.e. no reflections vs lots of room reflections), and the interaural separation (i.e. perfect vs only a few dB). I've always assumed this is what makes most of the difference.
More practically, good headphones are arguably available far more cheaply than good speakers - so more people get to hear what's actually in the recording with headphones than speakers.
There are things you can't hear on headphones though. Most obviously, if the recording has transaural processing, and the encoding damaged it, you wouldn't hear this so clearly (or at all) through headphones, as the original undamaged effect wouldn't work properly anyway.
A lot of audiophiles would claim that certain recordings (they consider them to be good recordings) have sound staging that is somewhat like that created by transaural processing - i.e. significant front/back depth, and sound sources that extend beyond the speakers.
Problem with this argument is that even low-ish bitrate mp3 preserves both binaural and transaural signals just fine! In fact it's one of the last properties to be destroyed as the bitrate is lowered - persisting long after transparency and even transient response have been audible compromised.
Cheers,
David.
Point 1 looks like a TOS 8 violation, but the rest of the OP's post is saying "I can't hear a difference", and the 3-trial ABX was followed by "I can't hear a difference" - so I'm not sure we need to chase ABX results there.
I think you're right David. That said, I also think it's important that the OP understands how to perform a proper double-blind test. Still, I probably should have been more gentle, especially because he appears to be opening to learning.
2Bdecided
Thank you for your very informative post.
You said that, the flatter the response of your earphones or speakers, the harder it is to tell difference. I have a question. My RE0 is lauded (and criticized) for being extremely neutral. I like this neutral sound. Does neutrality = flat FR?
Point 1 looks like a TOS 8 violation, but the rest of the OP's post is saying "I can't hear a difference", and the 3-trial ABX was followed by "I can't hear a difference" - so I'm not sure we need to chase ABX results there.
I think you're right David. That said, I also think it's important that the OP understands how to perform a proper double-blind test. Still, I probably should have been more gentle, especially because he appears to be opening to learning.
No, your response is normal after how I broke TOS #8 the second time. Sorry about that.
You said that, the flatter the response of your earphones or speakers, the harder it is to tell difference. I have a question. My RE0 is lauded (and criticized) for being extremely neutral. I like this neutral sound. Does neutrality = flat FR?
No, see the previous posts for explanation.
What I understand by neutrality is that properly recorded instruments (like acoustic guitar, drums, trumpet etc.) will sound as close as possible compared to a real world (live) audition. Sadly, this property is usually wasted by over-bassing the headphones (especially in the sub $100 range), treble roll-off etc.
Can you recommend some songs that has music with that high of a frequency?
greynol gave you some great links (regarding this and other topics). I listen to a lot of metal (Bleeding Through, Hatebreed, Dimmu Borgir, etc.) and it contains a bunch of information above 16KHz (mainly the drums). Most of the songs are still at 16KHz and below but there is plenty of information above that. Even then, the -Y switch (or an encoder/setting that applies a ~16KHz lowpass filter) doesn't have an audible affect on my music. That is just me though.
-Y is not a low pass filter, nor does it even remotely behave like one.
You said that, the flatter the response of your earphones or speakers, the harder it is to tell difference. I have a question. My RE0 is lauded (and criticized) for being extremely neutral. I like this neutral sound. Does neutrality = flat FR?
Flat FR for speakers and flat FR for headphones do not have the same effect. For example, look at
http://scitation.aip.org/getabs/servlet/Ge...ps&gifs=yes (http://scitation.aip.org/getabs/servlet/GetabsServlet?prog=normal&id=JASMAN0000870000S100S142000002&idtype=cvips&gifs=yes)
(I think this is freely available).
To cut a long story short, the experiment is as follows: The subject listens to a signal from two loudspeakers, then through headphones, and matches the loudness. This is repeated for different frequencies (roughly). The result is that one obtains an approximation of how the FR of a set of headphones should be to approximate the flat-FR loudspeaker. That is, if I play something through a pair of flat-FR loudspeakers, then, if I am now wearing headphones, what must the FR of these headphones be to hear the same thing (in terms of frequencies at least)?
The difference has to do with the way the acoustic waves from the loudspeakers are diffracted, reflected etc by the head, ears and so on, which differs from what happens to signals originating from headphones or earspeakers.
Most headphones are diffuse-field equalized, that is, try to achieve the FR that would be obtained by the experiment above in a reverbant room.
Point 1 looks like a TOS 8 violation
True, but it's not such an outrageous claim. I easily ABX'd a random track that was 103 kbps (Nero 1.3.3.0 -q 0.35), so doing it at 110 kbps isn't much of a stretch. And I regularly fail to ABX Lame at ~128 kbps, from the public listening tests I've done.
http://www.hydrogenaudio.org/forums/index....st&p=591802 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=66050&view=findpost&p=591802)
I've often wondered how much of an impact this has in helping people successfully ABX with headphones.
IIRC, guruboolez used a Beyerdynamic headphone that had exagerrated HF response. It was then speculated that this was a factor to his many positive ABX tests.
-Y is not a low pass filter, nor does it even remotely behave like one.
I am aware of that, I never said that -Y was a lopwass filter. I simply said that applying the -Y setting or a lowpass at ~16KHz has never affected audio quality for me.
Very interesting results.
I did the "mustang" hearing limit test, and my limit is 16kHz. Below 15kHz I can hear clear differences while ABXing. I can't tell a difference between 16kHz and 19kHz. I'm doing this on my earphones, which might have problems reproducing >16kHz. I'll test it with speakers that are -7dB at 20kHz later.
EDIT: Same thing with the speakers. 16kHz is my limit. 15kHz is noticeable.
Just curious, does a cold affect your hearing?
Just curious, does a cold affect your hearing?
Yes. First of all, having clogged ears (even if they are not totally clogged) is quite common when having a cold. Beyond that, being sick is probably going to diminish your concentration. You need to be focused during an ABX.
Of course, on a good day (cold free and enough mental space for good concentration) I still can't tell the difference between 192k MP3 and lossless
I listen to a lot of metal (Bleeding Through, Hatebreed, Dimmu Borgir
Practically every later Dimmu Borgir album has a lot of content above 16kHz, especially on the Enthrone Darkness Triumphant and Spiritual Black Dimensions albums which have really piercing hi-hat and cymbol sound (quite exaggerated, I can't imagine a real hi-hat sounding like that) .
These are also the albums which I used to bring the lossy encoders (around year 2000) to their knees.
After lots of ABX testing, I concluded that I'll be sticking to LAME for audio encoding. I'll use V5 setting because audio is transparent at this quality. I saved a lot of space by using this instead of Nero q 0.6.
Thanks everybody. I discovered a lot about my hearing, audio compression, and the "real" bitrate needed for transparency.
Now I wonder the point of the huge lossless tracks on Blu-rays. I think you can get very transparent results with around 1.5Mbps for any equipment source. I wonder if people can hear differences between the TrueHD soundtrack and lossless soundtrack on their Blu-rays.
Another question, the more I read here, the more it says that NeroAAC is better than LAME with low bitrates, like me (128Kbps). Why am I not hearing that? I think at 18Kbps less, NeroAAC should at least be on par with LAME. This time I ABXed it, getting it right every single time.
I wonder if people can hear differences between the TrueHD soundtrack and lossless soundtrack on their Blu-rays.
I often hear claims of people being able to discern "night and day" differences (a phrase which generally causes me to cringe upon hearing it), but I've never seen any ABX tests to substantiate any of these claims. My opinion is that the inclusion of lossless audio on Blu-rays (and HD DVDs, when they were around) is mere marketing, though it's not an entirely bad thing in my opinion. The discs themselves have an incredible amount of storage space, so video quality is rarely sacrificed for the inclusion of lossless audio.
Another question, the more I read here, the more it says that NeroAAC is better than LAME with low bitrates, like me (128Kbps). Why am I not hearing that?
These days, I would call 128 kbps a moderate bitrate. AAC tends to excel at much lower bitrates (particularly HE-AAC), whereas LAME performs admirably in the moderate and high bitrate ranges.
Another question, the more I read here, the more it says that NeroAAC is better than LAME with low bitrates, like me (128Kbps). Why am I not hearing that?
These days, I would call 128 kbps a moderate bitrate. AAC tends to excel at much lower bitrates (particularly HE-AAC), whereas LAME performs admirably in the moderate and high bitrate ranges.
I did some low bitrate NeroAAC test (80Kbps). It sounded surprisingly good. It is not until I ABX it until I hear obvious high frequency artifacts. It is like they used some trick to get the high frequencies.
If I didn't hear the source, I think I might be happy with the sound quality. I'm pretty surprised.
Another question, the more I read here, the more it says that NeroAAC is better than LAME with low bitrates, like me (128Kbps). Why am I not hearing that? I think at 18Kbps less, NeroAAC should at least be on par with LAME. This time I ABXed it, getting it right every single time.
Not all versions are the same. You could always try an earlier version of NeroAAC. That's what I did and I can't ABX the same files/bitrate that I could with 1.3.3.0.. IIRC Nero changed behavior of the lowpass filter in 1.3.3.0 and maybe that's what you (and me) are hearing with it.
I've learned alot here.
Thank you!
Now I say that anyone who says you can't hear the difference between lossy and lossless just needs better equipment.
...and what you say about needing better equipment is actually a myth, but whatever.
Do you have any other details to give us to go along with your anecdote (ABX logs, encoder(s) used and what settings, etc.)?
This may be obvious to those who do this regularly, but what double-blind protocol allows one to order multiple files by bitrate exactly?
I think you have to define what "better equipment" means based on meaningful performance specifications and technical measurements. Obviously if the equipment cannot accurately reproduce the entire bandwidth of the audio signals being tested, or it has high levels of distortion (usually the loudspeaker/headphone/room is the culprit here) then that could constitute a bias in the listening experiment.
Even the codec testing standards like ITU-R BS. 1116 have something to say about a minimum performance standard for the equipment and listening room used for conducting tests -- although one could argue they are not well-written.
Cheers
Sean
Audio Musings (http://www.seanolive/blogspot.com)
Obviously if the equipment cannot accurately reproduce the entire bandwidth of the audio signals being tested, or it has high levels of distortion (usually the loudspeaker/headphone/room is the culprit here) then that could constitute a bias in the listening experiment.
Both cases will usually make the ability to distinguish lossy from lossless easier, not harder.
One exception: Headphones aren't supposed to have a flat frequency response - they're supposed to simulate the frequency response of the part of the system that's missing (i.e. speakers > outer ear) which, due to the shape of the head and outer ear, isn't flat at all. That's OK - that won't break any codec - that just makes them as close as possible to normal listening.
I've often wondered how much of an impact this has in helping people successfully ABX with headphones.
Another problem with headphones - depending on the design - is you get a lot inter-intra listener varianc (http://www.aes.org/e-lib/browse.cfm?elib=11657)e in the frequency response measured in the ear due to how its fits and couples to the individual's ears. If different sounds are being delivered to the listeners at different times, then this could be a source of error variance in the listening test.
The biggest variance is at low frequencies from air leaks, and at high frequencies. This is certainly a factor when doing listening tests with binaural recordings/room scanning and measuring headphones. There are ways of calibrating for and removing these errors. The worse headphones types in this regard earbud-type phones that sit in your concha.
Cheers
Sean
Audio Musings (http://www.seanolive.blogspot.com)
Obviously if the equipment cannot accurately reproduce the entire bandwidth of the audio signals being tested, or it has high levels of distortion (usually the loudspeaker/headphone/room is the culprit here) then that could constitute a bias in the listening experiment.
Both cases will usually make the ability to distinguish lossy from lossless easier, not harder.
I don't understand how a poor loudspeaker can make it easier to distinguish lossy codecs from lossless ones.
For example, if the lossy codec applies a low pass at 15 kHz, and the loudspeaker dies at 12 kHz, wouldn't it be more difficult to hear the difference between the two codecs? You can't hear differences between the two signals if the loudspeaker can't reproduce the differences?
Cheers
Sean
Audio Musings (http://seanolive.blogspot.com)
That's easy. Frequencies not being reproduced between 12k and 15k will no longer provide necessary masking for the lossy encode to sound transparent.
Do I have to dig up the example of the guy with only an 8k response who was more able to distinguish lossless from lossy than the rest of the participants in the test?
Lossy codecs make assumptions about masking. For example, a loud 15 kHz tone may mask an immediately following tone (e.g. 1 kHz). Then the following tone is discarded to save space. In a playback chain with relatively flat frequency response this works beautifully and both versions are usually indistinguishable. If a system has a flawed FR and, for example, a severe dip at 15 kHz, the tone that should have masked the 1kHz tone isn't there anymore and so the missing 1 kHz becomes audible in comparison to the original.
That's pretty simplified, but that's about how it works. There are numerous other steps of the process, that ideally expect a playback environment with flat frequency response. Higher bitrates usually leave some headroom, but the basic principle is the same. You can easily verify this by comparing a strongly filtered lossy track against the original with the same filtering applied.
PS Greynol was faster and needed less words...
PPS It also works the other way around. Lossy encoders hide noise below masking components and if those are gone the noise becomes audible.
That's easy. Frequencies not being reproduced between 12k and 15k will no longer provide necessary masking for the lossy encode to sound transparent.
Do I have to dig up the example of the guy with only an 8k response who was more able to distinguish lossless from lossy than the rest of the participants in the test?
OK, I can understand how removing room reflections (via headphones) and part of the audio spectrum in the playback chain could enhance audibility of certain lossy codec artifacts through temporal/simultaneous unmasking. But at the same time, it could make other artifacts less audible as in my example.
So what criteria do you use in defining the performance of a standard playback system for testing codecs? To my knowledge, the objective measurement of audio codecs based on perceptual models, like PEAQ (http://en.wikipedia.org/wiki/PEAQ), are based on listening tests where the playback system was reasonably accurate using listeners with normal hearing -- test conditions that you suggest may over-estimate the transparency of the CODEC under less ideal playback conditions.
Cheers
Sean
Audio Musings (http://www.seanolive.blogspot.com)
So what criteria do you use in defining the performance of a standard playback system for testing codecs?
That's a good question, one for which I don't have an answer. I believe people should conduct DBTs on their own equipment set up in the most typical fashion for them. While I think public listening tests are useful and can provide a general baseline for people, I put more emphasis on personal listening tests. In the case of public listening tests, I think there should be some type of control over playback hardware and environments, but I don't think it's a make or break situation. If people really want to know what codec and settings to use, they should perform their own personal tests rather than rely on results from public tests.
My initial point to which you objected was to address the audiophile myth that audiophile grade components are required to most easily distinguish lossless from lossy. The people who perpetuate this myth are usually the very same people who think they can tell night and day differences between lossy and lossless but have never conducted a well-controlled double blind test.
So what criteria do you use in defining the performance of a standard playback system for testing codecs?
That's a good question, one for which I don't have an answer. I believe people should conduct DBTs on their own equipment set up in the most typical fashion for them. While I think public listening tests are useful and can provide a general baseline for people, I put more emphasis on personal listening tests. In the case of public listening tests, I think there should be some type of control over playback hardware and environments, but I don't think it's a make or break situation. If people really want to know what codec and settings to use, they should perform their own personal tests rather than rely on results from public tests.
My initial point to which you objected was to address the audiophile myth that audiophile grade components are required to most easily distinguish lossless from lossy. The people who perpetuate this myth are usually the very same people who think they can tell night and day differences between lossy and lossless but have never conducted a well-controlled double blind test.
Thanks. I don't disagree with you regarding the myth of needing audiophile grade component to hear codec artifacts. The problem I had with the term "audiophile component" is that as defined it's too vague and meaningless in terms of implied performance. My experience is that the term audiophile is often misused, and only implies "high price tag" which is not necessarily a good indicator of the component's sound quality and reliability.
I think we agree that frequency response aberrations, distortion\noise, room reflections, and listener training/hearing/aptitude are all nuisance variables that can influence the results of codec listening tests. High priced audio components don't necessarily help control these variables, and may in fact contribute to the problem.
If these nuisance variables are not well-controlled in public tests, then you may expect to get different results from different sites/people, and it may be difficult to reach consensus about the quality of the codec.
Cheers
Sean
Audio Musings (http://www.seanolive.blogspot.com)
If these nuisance variables are not well-controlled in public tests, then you may expect to get different results from different sites/people, and it may be difficult to reach consensus about the quality of the codec.
I hope by consensus you mean that one skilled listener can overrule your more mediocre participants. At higher bit rates you should expect results to diverge as people reach the limits of their hearing abilities at different points.
If these nuisance variables are not well-controlled in public tests, then you may expect to get different results from different sites/people, and it may be difficult to reach consensus about the quality of the codec.
I hope by consensus you mean that one skilled listener can overrule your more mediocre participants. At higher bit rates you should expect results to diverge as people reach the limits of their hearing abilities at different points.
By consensus, I meant that you should get better agreement in test results (based on the number of ABX positive responses, similar MOS or MUSHRA ratings) from different sites\listeners when all variables are well controlled.
If the nuisance variables are not well controlled, you would expect to get less agreement and more noise in the test results.
Cheers
Sean
Audio Musings (http://www.seanolive.blogspot.com)
By consensus, I meant that you should get better agreement in test results (based on the number of ABX positive responses, similar MOS or MUSHRA ratings) from different sites\listeners when all variables are well controlled.
If the nuisance variables are not well controlled, you would expect to get less agreement and more noise in the test results.
There are other things that can cause noise in test results. Don't you also expect to get more noise in the results as you approach transparency? For a codec that is nearly transparent, you'll have some listers that can reliably distinguish the difference and some that cannot. If you've removed every nuisance variable you can think of you can assume that noise is not due to those variables. You don't definitively know what variables you have not eliminated. When someone fails to distinguish a difference, you don't know if that is due to a limitation in the test or a limitation of the listener. When someone is able to reliably distinguish, in a properly calibrated and executed ABX, you
know that it is due to the codec. In your analysis of results, you need to ignore the noise of those who could not distinguish and concentrate on those who could. This is not a consensus process as you've described it. Building consensus typically involves persuasion. We know that listeners are easily persuaded. We don't want to persuade anyone. We want to know what they're hearing.
If you're testing for transparency, you might actually want to leave some of the "nuisance variables" in as they make or break distinguishibility for some listeners. Some examples: You will most likely degrade listening performance if you require subjects to listen on headphones, or prohibit them from adjusting their listening position to their liking. Audiophiles and recording engineers will tell you that they can't do their best listening on an unfamiliar system. There is some evidence indicating that acoustic reflections in the listening space enhances audibility of timebase errors.
So, in answer to a question you asked earlier, I'm going with the gist of greynol's answer: You want to do codec tests with a variety of listeners in a variety of realistic listening scenarios. Give your listeners latitude to attack the challenge creatively. Results may look noisier, and you'll have more work to do in back-end analysis, but you're casting a wider net and that's what you need to do to at this point to advance the art.
My point was that for any given individual, personal tests should take precedence over public tests and that those personal tests be conducted with the equipment and levels (volume, eq) that the listener typically uses. I do agree with all that you've said, though.
There are other things that can cause noise in test results. Don't you also expect to get more noise in the results as you approach transparency?...
Yes, agreed.
This is not a consensus process as you've described it. Building consensus typically involves persuasion. We know that listeners are easily persuaded. We don't want to persuade anyone. We want to know what they're hearing.
I'm not talking about persuading listeners but rather convincing ourselves (i.e. scientists) that the results from the CODEC tests are valid and can be explained by differences in the CODECS and not some uncontrolled variable.
If you're testing for transparency, you might actually want to leave some of the "nuisance variables" in as they make or break distinguishibility for some listeners. Some examples: You will most likely degrade listening performance if you require subjects to listen on headphones, or prohibit them from adjusting their listening position to their liking. Audiophiles and recording engineers will tell you that they can't do their best listening on an unfamiliar system. There is some evidence indicating that acoustic reflections in the listening space enhances audibility of timebase errors.
I agree with you 100% that we want to know what the listeners hearing perceptually, but let's make sure that the signals delivered to their ears are physically well-defined and controlled so we can a) learn something about the psychoacoustics of the codecs and b) ensure the experiment is repeatable.
So, in answer to a question you asked earlier, I'm going with the gist of greynol's answer: You want to do codec tests with a variety of listeners in a variety of realistic listening scenarios. Give your listeners latitude to attack the challenge creatively. Results may look noisier, and you'll have more work to do in back-end analysis, but you're casting a wider net and that's what you need to do to at this point to advance the art.
Again, I have no problem with this (note: I do similar experiments to study psychoacoustic interactions between different loudspeakers, room acoustics, trained vs naive listeners) as long as this is part of the experimental design and analysis. Otherwise you are missing a great opportunity to better understand how these variables (trained vs untrained listeners, loudspeakers versus headphones, room acoustics, different program,etc] influence the perception of CODECS. This is my main point.
If you are already doing this in the public tests, then please forgive me for stating the obvious.
Cheers
Sean
Audio Musings (http://www.seanolive.blogspot.com)
So I just did the audiocheck.net frequency test that was supplied earlier in this thread and I can hear the sweep right from the beginning, when it's at 22 khz. Also, as it goes down, it seems higher and louder to me and actually hurts my ears. I can see why it would hurt my ears ascending because as the frequency moves more into my audible range, it would be louder I think? How is this possible though? I'm aware that nobody can hear up to 22k and especially not me considering I did a test the other day with a pure sine wave and could only hear up to 16.5. What am I doing wrong? Sorry if I should've started another thread, this is my first time posting on this site btw.
So I just did the audiocheck.net frequency test that was supplied earlier in this thread and I can hear the sweep right from the beginning, when it's at 22 khz. Also, as it goes down, it seems higher and louder to me and actually hurts my ears. I can see why it would hurt my ears ascending because as the frequency moves more into my audible range, it would be louder I think? How is this possible though? I'm aware that nobody can hear up to 22k and especially not me considering I did a test the other day with a pure sine wave and could only hear up to 16.5. What am I doing wrong? Sorry if I should've started another thread, this is my first time posting on this site btw.
Okay, and now I can only hear up to 19. This is very confusing!
Edit : Nevermind, I just replicated what I heard the first time at 22 k with the Aliasing test. Looks like I need a new soundcard.
I'm not talking about persuading listeners but rather convincing ourselves (i.e. scientists) that the results from the CODEC tests are valid and can be explained by differences in the CODECS and not some uncontrolled variable.
Isn't this handled by the ABX? You can let listeners introduce any variables they like. Then they sit down and do the testing and any uncontrolled variable affects both A and B equally.
You would definitely want to tighten things if you need to move from determining
whether there's a transparency problem to determining
why there's a problem.
And, no, I do not do public testing. I don't consider my listening skills to be extraordinary. But, it is an interesting topic for me. Understanding the state of the art in listening motivates realistic design goals for systems I work on.
Isn't this handled by the ABX? You can let listeners introduce any variables they like. Then they sit down and do the testing and any uncontrolled variable affects both A and B equally.
"Affects both A and B" can be quite different from "affects both A and B equally". It may be that a particular setup emphasizes (or suppresses) one kind of artifact more than another, for example.
I would say that such transforms are a valid part of the test as long as it remains a realistic listening environment. It is not cheating to test the envelope of how people listen. Some people do crank the tone controls. Only spoilers listen to L-R.
And, no, I do not do public testing. I don't consider my listening skills to be extraordinary. But, it is an interesting topic for me. Understanding the state of the art in listening motivates realistic design goals for systems I work on.
I think Sean might mean private tests conducted during development, in which case he makes a very excellent point.
EDIT: I don't mean to give the wrong impression here, his points are excellent regardless.
I'm not talking about persuading listeners but rather convincing ourselves (i.e. scientists) that the results from the CODEC tests are valid and can be explained by differences in the CODECS and not some uncontrolled variable.
Isn't this handled by the ABX? You can let listeners introduce any variables they like. Then they sit down and do the testing and any uncontrolled variable affects both A and B equally.
You would definitely want to tighten things if you need to move from determining whether there's a transparency problem to determining why there's a problem.
And, no, I do not do public testing. I don't consider my listening skills to be extraordinary. But, it is an interesting topic for me. Understanding the state of the art in listening motivates realistic design goals for systems I work on.
Of course, I agree with you - that for any given ABX test setup, the nuisance variables are being held constant for both A and B.
Sorry, if I wasn't clear: my concern about controlling nuisance variable arises from pooling the results of public codec tests (using either ABC, ABC or MUSHRA methods) that are being conducted at multiple sites using different playback setups and listeners of unknown quality (e.g hearing, training, ability). When you start pooling the results together from these different tests, unless these different playback setups are well-defined and accounted for in the design and statistical analysis you risk getting increased, systematic errors/biases and possibly come to erroneous conclusions based on invalid results.
Cheers
Sean
Audio Musings (http://www.seanolive.blogspot.com)
And, no, I do not do public testing. I don't consider my listening skills to be extraordinary. But, it is an interesting topic for me. Understanding the state of the art in listening motivates realistic design goals for systems I work on.
I think Sean might mean private tests conducted during development, in which case he makes a very excellent point.
EDIT: I don't mean to give the wrong impression here, his points are excellent regardless.
Yes, I guess that's what I mean -- a private test - like the kind used for product development and validation, and the sorts of CODEC tests that are published by standards groups like ITU where they conduct CODEC tests at different sites throughout the world using a standard setup.
I'm quite ignorant about how Hydrogen audio organizes public listening tests, what their purpose is, and how they use the results. Is this strictly for hobbyists, or do companies actually use the results to help tweak the performance of their CODECS?
Perhaps you've published a document on recommended practices for public CODEC tests somewhere on HA that answers all these questions?
Thanks!
Cheers
Sean
You really should conduct blind ABX tests though as sighted tests are flawed by the placebo affect.
Though you're correct, it's not all that hard to identify what codec was used by listening to the artifacts, at least at lower bitrates.
You really should conduct blind ABX tests though as sighted tests are flawed by the placebo affect.
Though you're correct, it's not all that hard to identify what codec was used by listening to the artifacts, at least at lower bitrates.
This is not an issue in ABX tests, since the only thing being tested is whether the compressed file can be distinguished from the original.
In ABC/HR testing OTOH there certainly can be a bias based on recognising the codec, but then ABC/HR is asking for people's preferences, which can extend to factors beyond simple audible quality.