Skip to main content ↑ ↓

Notice

If you are using a Hotmail or Outlook email address, please change it now, as Microsoft is rejecting all email from our service outright.
Topic: Transparency: Public Listening Tests vs Anecdotes and My Experiences (Read 2315 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Transparency: Public Listening Tests vs Anecdotes and My Experiences

In my own listening tests (admittedly on only a few songs), I've found Opus 96 kbps reliably transparent.  Others' observations on HA seem to agree.  Yet, in the public 96 kbps listening test, most samples were not found to be transparent at this bitrate.  I also have found Vorbis near-transparent at this bitrate--I can sometimes pick out subtle artifacts in critical listening but never notice anything obvious.  Yet, Vorbis scored worse than Opus in the same listening test.

Similarly, 128 kbps AAC (Fhg) seems transparent to me even with CBR and even 96 is close.  Yet, the most recent listening test at this bitrate suggests various AAC codecs perform similarly to Opus@96 kbps.

Why do public listening tests seem so much more pessimistic than my experiences or, in the case of Opus, others' experiences on this forum?
  • Are exceptionally-hard samples typically selected for listening tests?
  • Do listening test participants typically have an exceptionally good ear for subtle artifacts?
  • Have codecs gradually improved with time such that the listening tests I cite are outdated?

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #1 –
Opus from 5 years ago and opus today are two different beasts. Optimizations were made, so sound quality is even better. There has been no, as far as I am aware, recent tests of codecs.
Error 404; signature server not available.

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #2 –
Quote from: itisljar on
Opus from 5 years ago and opus today are two different beasts. Optimizations were made, so sound quality is even better.
Original poster talks about 96 kbps. 

During the last 5 years of development of Opus:
5-48  kbps - big quality improvements
56-80 kbps - very small improvements
80-500 kbps - microscopic improvements which are hard to detect  + bugfixes

Quote from: dsimcha on
  • Are exceptionally-hard samples typically selected for listening tests?
It was a mixed bag.  Though hard samples were well represented.

Quote from: dsimcha on
  • Do listening test participants typically have an exceptionally good ear for subtle artifacts?
Yes,  more than half of the results had come from well trained listeners.  In real life scenario a such listeners represent low percentage of all people.

Quote from: dsimcha on
  • Have codecs gradually improved with time such that the listening tests I cite are outdated?
No. Last audible improvements were made in:

Apple AAC  - approx. 10 years ago or so
Vorbis - 2011 (last version of Aotuv Beta 6)
MP3 LAME  - 2011 (last version was 3.99 which contained quality improvement and not just bugfixes and misc stuff). 3.100, 3.100.1 are here just for maintenance and bugfixes.
Opus - last measurable improvements for bitrate higher than 80 kbps were made in version 1.1, December 2013. Since then only lower bitrates were improved.

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #3 –
Quote from: dsimcha on
In my own listening tests (admittedly on only a few songs), I've found Opus 96 kbps reliably transparent.  Others' observations on HA seem to agree.  Yet, in the public 96 kbps listening test, most samples were not found to be transparent at this bitrate.  I also have found Vorbis near-transparent at this bitrate--I can sometimes pick out subtle artifacts in critical listening but never notice anything obvious.  Yet, Vorbis scored worse than Opus in the same listening test.

The thing about such public tests (96kbps and higher) is that participants are mainly trained listeners and not quite average user.
In order to obtain more realistic results by including more of average listeners, test should be conducted at lower bitrates such as 48-64 kbps.

For example this one http://www.mp3-tech.org/tests/aac_48/results.html . Look how well MP3 LAME 128 kbps performed, because it was used as high anchor and because of low bitrate test  there were plenty of average users.

Considering this, everything that performs as well as LAME MP3 128k VBR will be in a  transparent zone for big mass of people.

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #4 –
Quote from: IgorC on

For example this one http://www.mp3-tech.org/tests/aac_48/results.html . Look how well MP3 LAME 128 kbps performed, because it was used as high anchor and because of low bitrate test  there were plenty of average users.

Considering this, everything that performs as well as LAME MP3 128k VBR will be in a  transparent zone for big mass of people.

Interesting!  I've definitely successfully ABX'd LAME 128 VBR (-V5) on some songs where Opus 96 is perfectly transparent to me.  I listen to a lot of rock with lots of cymbals.  At non-transparent bitrates, LAME can produce some very obvious non-linear/robotic artifacts there.  When Opus becomes non-transparent, the artifacts sound more like subtle additive noise and stereo image distortion.  They're harder to describe and harder to pick out by listening hard to a specific instrument. 

Also, Opus 96 scores ~0.4 MOS points higher than LAME 128 in the public 96 kbps listening test.  I guess the tl;dr is that Opus 96 is non-transparent on killer samples and/or to people who really know what subtle artifacts to look for, but probably transparent in most other cases.

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #5 –
Quote from: dsimcha on
I guess the tl;dr is that Opus 96 is non-transparent on killer samples and/or to people who really know what subtle artifacts to look for, but probably transparent in most other cases.
Agree

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #6 –
Pretty much how 144kbps Vorbis is enough for me 99% of the time. Yet AAC at same setting sounds horrid with Ambient music(some need 224kbps to fix it!), FHG is better but seems to really hate Doom metal like wut?. And both encoders seem to have killer samples that fail to respond to 256 ~ 500kbps unlike Vorbis/Opus.
Got locked out on a password i didn't remember. :/

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #7 –
Watch it!
Listen to the music, not the media it's on.

Musepack --quality 6
Wavpack -hb4.55x5cvm

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #9 –
Because you may learn a thing or two about e.g. the need for double-blind listening tests and the difference between a coding standard (e.g. AAC) and manufacturers of encoders generating files compliant with that standard (e.g. FhG). After that, I suggest you read your own sentence above again.
Quote from: ani_Jackal3 on
Yet AAC at same setting sounds horrid with Ambient music (some need 224kbps to fix it!), FHG is better but seems to really hate Doom metal like wut?. And both encoders seem to have killer samples that fail to respond to 256 ~ 500kbps unlike Vorbis/Opus.
Chris
If I don't reply to your reply, it means I agree with you.

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #10 –
Quote from: ani_Jackal3 on
Quote from: includemeout on
Watch it!
  
No idea why bother here. Lmao
 
 All that Chris said plus the fact that by ignoring such precepts, your statements - regardless of how well-meaning they are - incur exactly into the anecdotal evidence trap mentioned in this thread's title.
Listen to the music, not the media it's on.

Musepack --quality 6
Wavpack -hb4.55x5cvm

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #11 –
Quote from: includemeout on
Quote from: ani_Jackal3 on
Quote from: includemeout on
Watch it!
 
No idea why bother here. Lmao

 All that Chris said plus the fact that by ignoring such precepts, your statements - regardless of how well-meaning they are - incur exactly into the anecdotal evidence trap mentioned in this thread's title.

I posted a sample before here where Vorbis needs Q9.5 on merzbow. There old samples i made for AAC that suck at 144k but don't respond to 320k, Why should i care if your not even going to reply back when i spent 30mins wasting time making 30 sec samples with the same fucking replies?. Is this bait are you that self unaware on how this place just reeks of AAC fanboyism, Even GB 128k face off on problem sample page shows AAC struggling on some content i don't get with Vorbis/Opus(1.3)?.

I was going to make a thread about my views on multi codecs but might do it else where.

Got locked out on a password i didn't remember. :/

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #12 –
Quote from: ani_Jackal3 on
... old samples i made for AAC that suck at 144k but don't respond to 320k
I would be very helpful if you could point us to those samples so we don't have to search for them ourselves. And can you clarify what you mean by "don't respond to..."? As in "they become transparent at..."?
Quote
Even GB 128k face off on problem sample page shows AAC struggling on some content i don't get with Vorbis/Opus(1.3)?.
And Igor's recent test shows, that Vorbis at 192 kbps struggles on some content that AAC (iTunes encoder) at 192 kbps doesn't struggle on.

By the way, the test done by GB (I assume that's guruboolez) also shows the shockingly different quality different AAC encoders produce at the same bit-rate.

Chris
If I don't reply to your reply, it means I agree with you.

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #13 –
It doesn't help also the fact that all codecs seem to "break" in a very different way so it kind of makes it hard to rank them in the same kind of criteria because different listeners get "triggered" by different flaws. For example, for a long time I used to feel that 96kbps OPUS was my transparency threshold until I felt that something was off and after a lot of research I found out that alot of stuff I hear alot and I had trained my ears to expect elements to be heard in a certain way on the stereo field, they were being centered and I have not been able to "unhear" that since then. Now, other than that I d ont think any other codec can rival the quality of Opus at this bitrate, but still its a dealbreaker for me. So how do I quantify that in a test?

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #14 –
Quote from: Klimis on
It doesn't help also the fact that all codecs seem to "break" in a very different way so it kind of makes it hard to rank them in the same kind of criteria because different listeners get "triggered" by different flaws. For example, for a long time I used to feel that 96kbps OPUS was my transparency threshold until I felt that something was off and after a lot of research I found out that alot of stuff I hear alot and I had trained my ears to expect elements to be heard in a certain way on the stereo field, they were being centered and I have not been able to "unhear" that since then. Now, other than that I d ont think any other codec can rival the quality of Opus at this bitrate, but still its a dealbreaker for me. So how do I quantify that in a test?

Sounds like me i used to stick with musepack/ogg at 160kbps, But after a while i could tell it was adding noise and weird stuff to the stereo field. Stopped using them wen't back to LAME V2 ~ V0 just used Wavpack hybrid at 384kbps to squish any issues.
Got locked out on a password i didn't remember. :/

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #15 –
Quote from: Klimis on
It doesn't help also the fact that all codecs seem to "break" in a very different way so it kind of makes it hard to rank them in the same kind of criteria because different listeners get "triggered" by different flaws. ... So how do I quantify that in a test?
You're absolutely right, different listeners perceive different artifacts differently strong. Similar to how different people may react differently to some medication in medical tests. That's why it's called a subjective test, and that's why typically, those tests are conducted with at least 10 or 20 people.

How to quantify the distortions you hear is (mostly) up to you. The only thing that a typical blind listening test (with a reference = uncoded version you compare against) requires you to do is to consider any difference you hear, between a coded version and the uncoded reference, as a degradation. How severe that degradation is (perceptible, but not annoying, ... very annoying) is entirely up to you. But sometimes you get a "low quality anchor" as a reference, which makes it a bit easier for you to assign a score.

And yes, if you're doing a lot of audio codec comparisons, your perception of artifacts is likely to change over time. So the statements
Quote
and I have not been able to "unhear" that since then.
Quote from: ani_Jackal3 on
after a while i could tell it was adding noise and weird stuff to the stereo field.
are very reasonable. Being a codec developer, I've experienced such things a long time ago, and sometimes (when listening to Youtube music, for example) it's like a curse having to perceive all those coding artifacts (even more so in the video, by the way).

Chris
If I don't reply to your reply, it means I agree with you.

Re: Transparency: Public Listening Tests vs Anecdotes and My Experiences

Reply #16 –
Quote from: ani_Jackal3 on
I posted a sample before here where Vorbis needs Q9.5 on merzbow. There old samples i made for AAC that suck at 144k but don't respond to 320k, Why should i care if your not even going to reply back when i spent 30mins wasting time making 30 sec samples with the same fucking replies?.
 
Yeah, right. As if this was a social media platform and we were your followers, to be aware of your previous posts! If you want to be taken seriously here, and not just as another crackpot who happens to be fiercely defending his favourite codec, have at least the decency of referring to your onw posts and not expecting us to be able to quote you by heart!

Not replying back!? What the heck you're talking about?

Quote
Is this bait are you that self unaware on how this place just reeks of AAC fanboyism
 
 Man, you're definitely seeing things! What bait? That was just a reminder that, by now, you should've known better to keep an eye on the ball whenever making such apparently-unsuported claims over here - you even seemed to have taken that with a light heart at first, till I only complemented what someone else said and you started shit-throwing and using innuendos  - which didn't make your claims any favour, as they came out rather hollow  - and from what I learned hanging out here since 2001, such statement are actually a blatant tell-tale sign of said fanboyism - and TBH, these "mine is better than yours" statements are soo 2002, 2003, BTW!

Quote
I was going to make a thread about my views on multi codecs but might do it else where.
 
With that 'hollier-than-thou' attitude, you talk as if your chosing not to "enlighten us with your wisdom" would be a massive blow to this community! ::)
Listen to the music, not the media it's on.

Musepack --quality 6
Wavpack -hb4.55x5cvm
 
SimplePortal 1.0.0 RC1 © 2008-2020