Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Should we care about ABX test results? (Read 47016 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Should we care about ABX test results?

Reply #100 –
Quote
To answer my own questions, my best argument against changing the goal of codec development is that if codecs improved the sound, we could no longer have blind tests. The subject would always know that the "better" sounding sample is the encoded one and that information could bias him/her.

Suppose that the subject is a HA reader who is ideologically opposed to postprocessing and believes in the "original" sound.

Just apply the same postprocessing to lossless file -- and there will be no bias!

Quote
(2) Suppose it were possible to quantify listening pleasure by scanning the brain. Should we then redefine the goal of codec development and start measuring enjoyment instead of transparency?

What is pleasant for you, can be ugly for me, so please don't.

Should we care about ABX test results?

Reply #101 –
@tzsyn:

Way back in the old days of MP3 ( Xing days ), if you encoded an LP with too low bitrate, aside of getting artifacts (unwanted), you got most of the LP noise removed (improvement). There weren't enough bits, so the encoder kept the most prominent data (which excluded the noise).

If we modify today's encoders to not produce actifacts, but do filter the noise, the result could be preceived as an improvement for those cases, but.. what about a recorded classical concert? Most probably, the reverberation and ressonances would be taken as "noise" too.


Think about it. If there was an algorithm that improved all and every sample that could be encoded, don't you think that the first to use it would be the recording studios? And if so, wouldn't it be present in the original, before encoding it?


The only processing that has sense in a codec, if transparency is not a goal, is such that could improve the quality when using a lower bitrate. Some servers of internet radio stations apply DSP's to the input signal to modify certain characteristics of the sound, and ease the work of the encoder (or at least they used to say so). Intuitively that works like LossyWav, which modifies the original data so that FLAC and other lossless encoders can reduce the size further.

Should we care about ABX test results?

Reply #102 –
Quote from: tszyn on
... To answer my own questions, my best argument against changing the goal of codec development is that if codecs improved the sound, we could no longer have blind tests. ....

I beleive you don't have experience with codecs' artefacts. Codec artefacts usually sound like unpleasant distortion, and the only question is how annoying they are.
With bitrate high enough distortion is inaudible or very rarely audible in a more or less subtle way. Nobody is developing a codec with 'nice' distortion according to user's vote, and I guess this wouldn't be possible even if someone thought that this were a useful thing.

Sound improvement is a different area of development, and it's a good thing that these things are not mixed up. Purists (most of the HA members I guess) hate it, but that's not the necessary attitude for the way of listening. Whoever likes it can use the sound-changing machinery of his listening environment. As [JAZ] wrote internet radio stations do it. I personally was very pleased by the sound quality of radio paradise and asked them which codec they used. They couldn't tell me but they said that sound quality for the most part of it emerges from their sound preprocessing procedure.

So yes, there are sound-changing procedures that can be (individually or not) perceived as improvement, BUT: you really shouldn't mix up procedures targeting at sound improvement with codec development.
lame3995o -Q1.7 --lowpass 17

Should we care about ABX test results?

Reply #103 –
Quote from: tszyn on
  • That the lack of audible differences in an ABX test does not imply that two samples are equally good (ABX tests focus on conscious experience and so fail to take into account subconscious effects).


If subconscious effects (present or absent in the samples) are capable of affecting the listener  then they will influence the decision making. Consequently they will appear in the test results.

So the answer is obvious: YES. We should care about ABX results.


Should we care about ABX test results?

Reply #104 –
The subject has been raised whether a non-transparent codec can be more pleasant to the ear than a transparent codec.  In fact it can be, for some listeners.

In Australia, DAB+ radio started offically last month (July 2009).  The codec used is HE-AAC v2, which utilises the aac codec, plus optional spectral band replication (SBR), and optional parametric stereo (PS).  To date, even the lowest bitrate stations (32kbps) have chosen not to use PS; but all stations have chosen to use SBR.  The highest bitrate in common use is a nominal 88kbps, and many stations are using a nominal 64kbps. [I say "nominal" because there is overhead for error correction, and for text/images.]

If I prepare two samples, a flac file of music ripped from a CD and a flac file of the ripped music subjected to 64kbps HE-AAC v2 encoding (with SBR, and without PS) I can for some music ABX the result quite easily.  I personally dislike the processed version because for my ears the treble sounds less detailed, and  somewhat exaggerated and artificial, and there is a slight "hollowness" to the sound in the mid frequencies.  However ... the processed sound is elegant (rounded and smooth) and creates the impression of low distortion.  If I play two such versions to friends, some of the friends tell me sample "B" (the processed version) sounds "better".  They note it has "more treble".

For my ears, classic dolby 5.1 as used on DVDs also has an elegant smooth and pleasant sound.  It is probably a simpler sound to experience, as certain content deemed irrelevant and inaudible content has been discarded by the perceptual dolby codec. 

Getting back to the original topic, if a significant percentage of listeners prefer a perceptual codec that discards detail that is barely perceptible and proceeds to deliver a clear smooth sound, then for those listeners ABX testing will be of little relevance.  They are not pursuing the last nuance of detail and "fidelity".  They want something easy and pleasant to listen to. 

I continue to choose to listen to the classical music station of the Australian Broadcasting Corporation using FM radio.  Even though there is a slight amount of hiss, and an audible exent of distortion, at least I can hear separate instruments in the orchestra if I listen carefully; and in particular I can hear detail in the treble.  But many listeners would choose the nominal 80kbps HE-AAC v2 DAB+  broadcast, for its low noise, low apparent distortion, and neatness and elegance of sound.

I am hoping that in time the digital radio broadcasts in Australia of classical music will be at higher bitrates than they are currently.  But if many listeners find a non-transparent codec more pleasant to listen to than an unvarnished version from a CD, it may be an uphill battle.

[Reported tests carried out by the EBU suggest that for many listeners 64kbps HE-AACC v2 sounded better than FM.  I don't know what standard of FM transmitting and receiving equipment was used in the EBU tests, but that result on the face of it was very odd. 64kbps is well below transparency and for my ears artifacts are clearly audible and the sound not as good as FM reception [at good signal stength, and despite the 15KHz bandwidth limitation].  I suspect the listeners were impressed by the very low noise, the bursts of treble, and the overall elegance of the sound, and overlooked the loss of realism and detail.]

Should we care about ABX test results?

Reply #105 –
Quote from: tszyn on
(2) Suppose it were possible to quantify listening pleasure by scanning the brain. Should we then redefine the goal of codec development and start measuring enjoyment instead of transparency?

Isn't that what DSPs are for?

You know what codecs are for? Yes?
And you know what DSPs are for? Yes?

And so, why do you want codecs to also fill the role of DSPs again?

Audio codecs and DSPs are separate for the same reason video codecs and video processors/filters are separate. So everyone can select what they like. Beats the hell out of your "majority-benefit guaranteed!" filter effect, in my humble opinion

Circles circles circles, sorry if I'm not helping.

Should we care about ABX test results?

Reply #106 –
Quote from: halb27 on
So yes, there are sound-changing procedures that can be (individually or not) perceived as improvement, BUT: you really shouldn't mix up procedures targeting at sound improvement with codec development.


The ideas of changing and not changing the character of sound during the reproduction process  have been around as long as there have been means for reproduction, which is over 100 years. During that time there has been a clear and definate trend, which is towards sonic accuracy.

There are people who are vitally concerned with changes to the character of sounds, we call them actors, musicians, and recording engineers. If there was some "one size fits all" change that could be made to music that people would in general prefer, they are the people who are best equipped and best enabled to make that change.

Should we care about ABX test results?

Reply #107 –
Quote from: MLXXX on
[Reported tests carried out by the EBU suggest that for many listeners 64kbps HE-AACC v2 sounded better than FM.  I don't know what standard of FM transmitting and receiving equipment was used in the EBU tests, but that result on the face of it was very odd. 64kbps is well below transparency and for my ears artifacts are clearly audible and the sound not as good as FM reception [at good signal stength, and despite the 15KHz bandwidth limitation].  I suspect the listeners were impressed by the very low noise, the bursts of treble, and the overall elegance of the sound, and overlooked the loss of realism and detail.]
Do you have a link to this test? If it's by the EBU, it will have been published freely on the net, but I can't find it (easily) on their website.

Whether something is "better" than FM depends on the reception quality and the actual broadcast.

If you can't hear above 15kHz, then the only consumer format that consistently beats FM at its very best, is a CD.

However, if you're talking about typical FM broadcasts with heavy handed dynamic range compression, and typical FM reception with a bit of wire dangling from the back of a cheap stereo, then FM is easily beaten.


The important difference being, if you care sound quality, you can put an FM aerial on your roof and use a decent tuner. This will give good quality from the stations which don't use too much DRC. Whereas if you don't like the sound quality you get from digital radio, there's nothing you (as a user) can do to improve the situation!


I don't think those samples I've heard from Australian DAB+ really represent the best you can expect from DAB+ at those bitrates - it'll get better.

Cheers,
David.

Should we care about ABX test results?

Reply #108 –
David,
        I find  a comparison between HE-AAC and broadcast FM does not appear to have been published by the EBU (despite the impression I gained from other web discussion that it had).  However the firm Orban makes the following claims:

[blockquote]
Quote
AAC has an MPEG-standard extension called HE-AAC V2, which offers the best quality currently available at 96 kbps and below. HE-AAC can provide entertainment-quality stereo at bandwidths as low as 32 kbps and satisfying, high fidelity speech quality in mono at 20 kbps. Also known by its trademarked name of aacPlus, or eAAC+, HE-AAC's ability to work at these low network bandwidths means that HE-AAC files can pushed to the consumer several times faster than a 128 kbps MP3 or WMA file while using a fraction of the network bandwidth.

In double-blind tests, many listeners prefer the sound of 48 kbps HE-AAC to the sound of FM radio. 48 kbps is arguably the “sweet spot” for HE-AAC, where it provides a non-fatiguing and musical listening experience that makes consumers want to continue listening to the podcast.
at http://www.orban.com/products/streaming/op...020/podcasting/

Quote
Independent quality evaluations of aacPlus
The value of aacPlus v2 has been clearly demonstrated by independent tests. In careful double-blind listening tests conducted by 3GPP (3rd Generation Partnership Project), aacPlus v2 proved its superiority to its competitors even at bit rates as low as 18 kbps.

aacPlus v1 has been evaluated in multiple 3rd party tests by Digital Radio Mondiale, MPEG, the European Broadcasting Union, and the Institute für Rundfunktechnik IRT. aacPlus v1 outperformed all other codecs in the competition. Below is the results graph from the European Broadcasting Union testing at 48 kbps stereo. The full "EBU subjective listening test on low bit rate audio codecs" can be downloaded.
at  http://www.codingtechnologies.com/products/aacPlus.htm[/blockquote]

Quote from: 2Bdecided on
I don't think those samples I've heard from Australian DAB+ really represent the best you can expect from DAB+ at those bitrates - it'll get better.
The MUX upload of May 2009,  sounded dreadful and would not even play back at correct speed  [ http://groups.google.com.cy/group/alt.radi...947b840b794b34f ]. There have been more representative, better quality, uploads since.  A major issue is the tendency of some of the commercial broadcasters to maniplulate the sound (e.g. aggressively boost the treble).  The national, government funded broadcaster, the ABC,  is providing a fairly decent quality that avoids heavy pre-processing.  However even the nominal 64kbps the ABC uses for local talk/music contains noticeable SBR artifacts (for my ears), and the classic music station at 80kbps is elegant and tidy, but suppresses detail that can be heard in the simultaneous FM broadcast.  There's an example of that here.  Cheers.

--

The basic  point I'm trying to make is that although ABX testing is a wonderful technique for determining whether a codec is transparent, and will assist in testing for "fidelity" (accuracy), it is not a tool for evaluating whether a codec is satisfying to listen to.  Perceptual codecs do not merely act as tone controls.  Their "voodoo" is quite complex in removing that which will probably not be heard anyway by the average human ear.  What is left is a simpler, and [to my ears] often a "sweeter", sound. For me, this simpler sound is somewhat bland, and unsatisfying to listen to.  But others may prefer it.

Should we care about ABX test results?

Reply #109 –
MLXXX,

Thanks for the links. Even the FM version sounds like it came from a lossy source, but it's hard to tell with the DRC on there.

After nearly a decade complaining about UK digital radio, the BBC are now giving us excellent quality via the BBC iPlayer on-line. It's not available outside the UK, but I'll try to upload some samples at some point. There's still too much DRC on many stations, but the encoding is faultless.

I know it's an apple to oranges comparison, but people seeking quality in the UK have given up on UK DAB. There's still a hope that DAB+ in the UK might bring better results - but not if it sounds like DAB+ in Aus.

Cheers,
David.

P.S. On topic: on a trivial level, most people will think something sounds better if you increase the bass and treble a little. It doesn't mean a codec should do this.!I think in the Aus DAB samples you compare one set of problems on FM with another set on DAB+. Whereas if you had lossless vs DAB+ to compare, I think most people would prefer lossless, and the others would express no strong preference.

It's difficult - many people, having spent silly money on a new toy, can't bare to admit that it's not very good. We went through exactly the same thing with UK DAB. Strangely, now receiver prices are £20, people are more happy to admit than some stations sound attrocious - though they sounded just as atrocious (if not more so) on the £800 tuners!