Digital Radio listening test

Topic: Digital Radio listening test (Read 16222 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

Digital Radio listening test

2003-02-26 10:17:01

There has been endless discussion recently in the UK about the quality of our digital radio services (DAB). They typically use MPEG layer II (mp2) at 128kbps JS. The quality is interesting!

A campaign to get something done about this has been partially sucessful - many Radio stations are now available via the digital TV platforms at 192kbps mp2, which is an improvement.

What I, and others, would like to do is to run a listening test, comparing various bitrates, and (possibly) the two codecs in use worldwide in digital radio (mp2 and AAC). The results may or may not be a forgone conclusion, but the reason for doing this is to get some independent, unbiased information about what people actually hear, and, specifically, whether 128kbps mp2 is good enough.

This is a plea for help. I have followed listening tests here and elsewhere in the past, and carried them out myself. However, I need suggestions:

1. which tool is best for this test?
2. would anyone be willing to host this test?
3. what samples would you suggest?
4. would it be appropriate to use "processed" samples?
(in radio, you don't get the digital bits from the CD going to the encoder; you get A>D, mixing desk, dynamic range compression, D>A encoder at least)
5. do these sound sensible: mp2 128 160 192 256, AAC 96 128
6. which encoders should I use? Realtime hardware codecs are used in radio broadcast, but obviously software would be easier

I (and many other people!) would like to be able to take the results of this test to the regulator in the UK and say "look, 128kbps mp2 isn't good enough", and for them not to be able to pick any obvious holes in the test methodology.

Any help and suggestions greatly appreciated!

Cheers,
David.

P.S. for endless discussion over DAB audio quality, see alt.radio.digital
http://groups.google.com/groups?q=alt.radi...&oe=UTF-8&hl=en or your newsreader

Digital Radio listening test

Reply #1 – 2003-02-26 11:57:45

It would be nice doing a test like that. Here, in Italy, are sold some car stereo with DAB, but there are very few stations here.

I have another question to add: I read somewhere that DAB would use AAC+ giving very high quality at low bitrate, so why I read now that DAB is in MP2? I thought it was only for satellite radio station, but normal radio station would use AAC (or AAC+ when ready).

Digital Radio listening test

Reply #2 – 2003-02-26 12:22:24

DRM - Digital Radio Mondiale = digital broadcasts below 30MHz (that's AM to you and me)
This will use AAC + SBR at bitrates around 15-30kbps. It's a world wide standard.

DAB - Digital Audio Broadcasting = digital broadcast around 200 and 1500 MHz (that's above FM)
This uses mp2 at bitrates from 32-384kbps. It's a european standard, accepted most places except North America.

HD Radio - IBOC and PAC = In Band On Channel with Peceptual Audio Coding (that's in the FM band)
This is the North American standard. I would swear that this uses AAC, but there's no mention of it on their website.
It has been suggested that it uses 96kbps AAC during transition (it's in the same space as conventional FM, so piggy-backs on the existing broadcast)
and then 128kbps AAC + other services when the analogue FM station stops broadcasting.

In the UK, with a fixed home dish or aerial, you can use DSat (digital satelite) or DTT (digital terrestrial) TV to receive radio broadcasts from the BBC and many others. The quality is higher than on the actual digital radio (=DAB) broadcasts, due to the higher bandwidth (hence bitrates) available on the TV platforms. Strange but true!

Cheers,
David.
P.S. There are two satelite radio systems too, intended for mobile reception. This is in the USA, and I have no information on these.

Digital Radio listening test

Reply #3 – 2003-02-26 13:33:13

Imo the biggest problem would still be the credibility. Who says that certain software encoders act anywhere similar to certain hardware encoders? Or at least that's what can be said as a strong counter argument to this test.

Anyway, what comes to test samples, ff123 used nice variety of different music in his latest 64kbps test: http://ff123.net/64test/results.html

Imo the test, if started, should be done in similar style, using the ABC/HR Audio Comparison Tool.

Digital Radio listening test

Reply #4 – 2003-02-26 14:00:27

For Layer II encoders, I would use one from QDesign.

Digital Radio listening test

Reply #5 – 2003-02-26 14:06:48

Quote

Imo the biggest problem would still be the credibility. Who says that certain software encoders act anywhere similar to certain hardware encoders? Or at least that's what can be said as a strong counter argument to this test.

Yes, that's my worry too. In practice, some software encoders are better than the (now quite old) hardware encoders in use in the UK DAB industry, especially at the BBC. However, it would need another listening test to prove this!

I was also considering using actual DAB recordings as one section of the test. Then there's no argument. It's possible to get the same track broadcast on DAB (mp2 128), DTT or DSat (mp2 192), FM, and CD! Of course the CD hasn't been through all the processing of the radio station, but the others would make an interesting comparison.

Quote

Anyway, what comes to test samples, ff123 used nice variety of different music in his latest 64kbps test: http://ff123.net/64test/results.html

I had this in mind - I think "typical" rather than "stressful" would be fair - though I might put one stressful clip in there.

Quote

Imo the test, if started, should be done in similar style, using the ABC/HR Audio Comparison Tool.

Yes - totally agree.

Cheers,
David.

Digital Radio listening test

Reply #6 – 2003-02-26 14:27:54

To show differences between different encoder generations, you could do a quick comparison between ISO encoder (plain old musicam) and QDesign or TooLame.

Digital Radio listening test

Reply #7 – 2003-02-26 16:14:52

Thanks Gabriel - I'll try that - I have a purchased version of Qdesign mp2 somewhere - it's in one of the unpacked boxes in our new house! I understand tooLame is making good progress these days.

Where can I find a binary of the ISO mp2 code?

btw, I've been in touch with ff123, and he's offered to help, which is great.

Questions 4 and 5 are still open, as are the others if there are any more comments.

I'll be back when I've made more progress.

Cheers,
David.

EDIT: typo

Digital Radio listening test

Reply #8 – 2003-02-26 16:57:34

Quote

Where can I find a binary of the ISO mp2 code?

Get SoloH. It's a compile of the infamous dist10 sources with an usable frontend. It's slow, buggy and outputs terrible quality.

ftp://ftp.redcom.ru/pub/support/windows/u...egEnc_v007a.zip

Here's the old SoloH page:
http://www.euronet.nl/~soloh/mpegEnc/

(Binaries aren't available there anymore)

Digital Radio listening test

Reply #9 – 2003-02-26 17:44:12

Sounds interesting. Some possible concerns:

Probably the best possible outcome would be if there are clear faults (from a group perspective) in the low bitrates which go away somewhere around (say, for the sake of argument) 192 kbit/s. Then one could definitely say that 192 kbit/s is good enough, at least from a quality, if not from an economic standpoint.

However, if that doesn't happen, and the listening group is still finding fault at high bitrates, then interpreting the test results become more problematic. Or perhaps one or two samples out of a dozen shows that there's a problem at 192. How does one decide what's good enough? Perhaps the testers, being self-selected and highly focused on listening for quality differences, are "too sensitive" and don't represent real-world listeners. A possible counter-claim: "Yeah, there are small differences even at high bitrates if you play a short sample over and over and really listen for faults, but for most people most of the time it's good enough."

From the standpoint of gathering information on differences in codec quality, I think the test should be fine. It's the subsequent interpretation which I have some concerns with.

ff123

Digital Radio listening test

Reply #10 – 2003-02-26 18:48:15

David:

I'm still terrified by the quality of the DAB sample you once posted ("Eternal Flame"), so I think this test is a great idea. But where do you want to get the number of test listeners one would need to convince the regulators...? The 100-200 people regularly participating in HA wouldn't do, surely. (I hope I'm wrong!)

I'd also keep the number of test samples to a minimum - otherwise, you'll scare modem users like me away, and if you make this real, every listener counts.

I think the idea of using samples recorded directly off DAB isn't a good one - because of all the processing used to make the sound more "punchy", unexperienced listeners might rate the station's output higher than the samples encoded from unprocessed CD audio, despite it being more artifact-ridden.

Just my 0.02€.

Digital Radio listening test

Reply #11 – 2003-02-27 11:17:48

ff123,

You've hit the nail on the head - it's easy to do a "good" test that stands up for what it is - but avoiding later criticism is difficult.

I'm not going to pre-judge the result - and I'm not even going to be surprised if it shows almost the opposite of what I hope! It is probably true that some people here can find faults at most bitrates with an mp2 codec. However, many won't - there are some "normal" people here too, and the test won't just be open to HA members. It would be very interesting to find how many people think each bitrate is OK. For that matter, what is OK? Acceptable? Transparent? Not annoying? On the 1-5 scale it's anything above 4, I think. There are EBU definitions that I'll use, though, interestingly, only AAC meets them for all audio samples!

As for the argument that the testers simply "listened too carefully" - well, they probably will. Sometimes I listen too carefully to radio as well! I can't counter it, but I had thought about it. Any ideas?

Volcano,

I'll get as many people as possible - if as many as 100 people take part, I'll be amazed. "Real" listening tests are often done with less than 30 people, though they are pre-screened. I'm not too worried, so long as, say, more than 30 take part.

I will certainly keep modem users in mind - I am one myself!

I'd thought about the processing problem if I get samples from DAB directly. There's one argument that says "if you don't use any samples from DAB, then you're assessing an mp2 codec that could be very different from the ones used in DAB, so the test is pointless". There's your counter argument. In fact, the processing on some UK DAB stations is very mild, and once the levels are matched, it shouldn't be a problem. I guess I won't know until I've tried it.

Let's say I find an audio clip which I can get on CD, and can also record from FM, DAB, and DTT.
Possible options include:
1. Original CD
2. mp2 encoded directly from this CD at 112 128 160 192 256
3. AAC encoded directly from this CD at 96 128 192
4. "Processed" CD (A/D > mixer desk > D/A > some dynamics processing)
5. mp2 encoded from "Processed" CD at 112 128 160 192 256
6. AAC encoded from "Processed" CD at 96 128 192
7. "Transcoded" CD (via MD of mp2 256, because many many radio stations use these for playout!)
8. mp2 encoded from "Transcoded" CD at 112 128 160 192 256
9. AAC encoded from "Transcoded" CD at 96 128 192
10. recording from FM
11. recording from DAB
12. recording from DTT

Obviously 1,2,3 is one test, 4,5,6 is a separate test, 7,8,9 is yet another test, and 10,11,12 may fit in with any, or one, or none of the above!

Then there's the question of different AAC and mp2 codecs (I've found my Qdesign one btw, so now have at least 3 mp2 encoders)

Then there's the question of source material. If it's to include10-12 it's restricted to what I can find on the radio (The album chart show on BBC Radio 2 would be an easy start - the top 40 on Radio 1 is highly processed, making comparisons with CD difficult).

If it's not to include off-air recordings, the source material could be anything, but I would probably use:
1. piece of pop music
2. piece of classical music
3. extract of speech

For 3, I'll buy a recent BBC comedy show on CD, and use that. For 1, wayitis.wav is pretty middle-of-the-road, or maybe Dido, or whatever. I specifically don't want to use "codec killers".

I'm thinking allowed here, and I really don't know where to go with that 12 point list (and various codecs!) to get (ideally) 4-8 samples per source.

I want a fair test that represents what's heard on DAB, but I don't want any bias or undefensible method to be involved. This is harder than I expected, and I haven't even started yet!

Cheers,
David.

Digital Radio listening test

Reply #12 – 2003-03-03 06:45:38

Some questions and comments:

1. How many participants do you estimate there will be? The more, the better. Have you considered how you could drum up as much participation as possible?

2. How many different formats were you thinking of? As a participant, I probably wouldn't want to compare more than 5 or 6 formats at any given time. However, it might be possible to test many formats using a design in which different people compare different formats. I'd have to research how to do it properly, but almost certainly you need lots of people to make it work. And probably need to have some sort of automated software on the server side.

3. It's hard to say one format is better than another without listening to lots of different samples. A dozen is nice, 20 would be even better. Of course not everybody would want to listen to every sample (this worked out ok in the 64 kbit/s test).

4. If this turns out to be a large test, then automation is almost certainly necessary. When I did the 64 kbit/s test, I had people send me email. If I had to do it over, there's no way I would go that route again. Some sort of web form in which people paste their results would probably be the best way to go.

ff123

Digital Radio listening test

Reply #13 – 2003-03-03 08:52:32

Quite OT: hey, do you think that OpenDRM (based on Vorbis) could be useful ?

Digital Radio listening test

Reply #14 – 2003-03-03 11:56:33

Quote

What I, and others, would like to do is to run a listening test, comparing various bitrates, and (possibly) the two codecs in use worldwide in digital radio (mp2 and AAC).

Hi David, maybe you should have posted this in the AAC boards, too... These tests have already been done in a very thorough manner by the MPEG Audio Subgroup back in 1996 and 1998, and the full reports can be downloaded from their website. In my opinion they can also serve as a "role model" for other ambitious listening tests, because as usual the MPEG committee tried to cover every aspect of a valid comparison that could serve as a basis for important future decisions for radio broadcasters like the BBC and others.

The beginning was a comparison of MPEG-2 NBC (= AAC) with other codecs (also MPEG-2 Layer II) for multichannel purposes, so the bitrates in questions were quite high. The results have been published in a PDF file named "w1419.pdf" (72 pages). A shorter summary is available with "w1420.pdf" (14 pages).

The stereo setup of AAC at 96 kbps was compared to other codecs (again with MPEG-2 Layer II at 192 kbps) in "w2006.pdf". This is the official report that gets mentioned whenever the sound quality of AAC is described in comparison to MPEG-1/2 Layer III and Layer II. I don't remember if all these PDF files are also available in a HTML version from the MPEG Audio Subgroup, but you could have a look around for yourself maybe.

Some of these informations (links, excerpts, summaries) are also available from the Wiki of AudioCoding.com, e.g. the AAC page:
http://www.audiocoding.com/wiki/index.php?page=AAC

Digital Radio listening test

Reply #15 – 2003-03-03 13:24:45

Quote

Quote
What I, and others, would like to do is to run a listening test, comparing various bitrates, and (possibly) the two codecs in use worldwide in digital radio (mp2 and AAC).

Hi David, maybe you should have posted this in the AAC boards, too... These tests have already been done in a very thorough manner by the MPEG Audio Subgroup back in 1996 and 1998, and the full reports can be downloaded from their website.

Hi Hans,

You mean you don't read all the boards on the forum? Yes, yes, I know, things fly off the front page so quickly that it's easy to miss interesting threads these days.

I have all the official tests (I think - some were also published in the AES). I often recommend w2006 in the same way as you have just done - as an excellent example of how to run a listening test. (IIRC It's on Gabriel's mp3-tech.org as a word .doc)

Given the existance of all these tests, it's quite amazing that UK broadcasters are using 128kbps JS mp2 for almost all their digital radio broadcasts. Most wouldn't dream of using any higher bitrates - the commercial broadcasters are only doing what I would expect (due to the lack of regulation in this area) but I would have expected better from the BBC.

There's a saying about "knowing the cost of everything, and the value of nothing". They know the cost of broadcasting in terms of licenses and bandwidth and uplinks, but know nothing about the value of their own radio programmes, the listeners to which are being subjected to the most appauling sound quality!

There is one way in which the official tests fail to match reality: they all used "clean" audio. No hiss, no distortion, no dynamic range compression. And certainly no transcoding. So the results from those tests, though they do suggest that 128kbps mp2 doesn't sound very good, don't actually show how terrible it sounds in real radio broadcasts! Also, they used "golden ears" - and this fact alone is enough to make the UK radio industry completely ignore them. To quote someone from GWR (a large UK radio group) "If you want CD quality, go and buy the CD! As for the audiophiles - stuff them!" (paraphrased from http://www.radioacademy.org/techcon/4audio...ualityonmpg.doc discussed in http://groups.google.com/groups?threadm=b3...ol.co.uk&rnum=1 )

I don't expect this test to help that much, but it will be useful to have some new data, gathered using a wide range of listeners, with realistic, representative test material. And beside all this, the results will be very interesting. If I can include the right things in the test, it will answer:

1. How big is the effect of "unclean" sources (transcoding/D>A>D/etc)
2. What bitrate is transparent for people in the test (i.e. not just golden ears) using undemanding material

It may even show that, under test conditions, despite intentionally degrading the source material, the audio extracts STILL don't sound as bad as UK DAB broadcasts - which would beg the question "what on earth are the broadcasters doing?!". I suspect the answer is encoding/decoding/encoding/decoding... so many times that there's barely any signal left.

Cheers,
David.

Digital Radio listening test

Reply #16 – 2003-03-03 13:32:38

Quote

Some questions and comments:

1. How many participants do you estimate there will be? The more, the better. Have you considered how you could drum up as much participation as possible?

As many as possible. It sounded like question which you'd thought of an answer to? My answer would be publicise it here, and any other forum I could think of. What's yours?

Quote

2. How many different formats were you thinking of? As a participant, I probably wouldn't want to compare more than 5 or 6 formats at any given time. However, it might be possible to test many formats using a design in which different people compare different formats. I'd have to research how to do it properly, but almost certainly you need lots of people to make it work. And probably need to have some sort of automated software on the server side.

As you can see, it's not just formats, it's combinations of formats plus other factors. It's encouraging that this might be possible, while keeping the number of comparisons low per test. But as I couldn't guarrentee huge numbers of people, it may be best to keep it simple. A "small test that gets a result which leads to another test" is beter than a "large test that's never finished".

Quote

3. It's hard to say one format is better than another without listening to lots of different samples. A dozen is nice, 20 would be even better. Of course not everybody would want to listen to every sample (this worked out ok in the 64 kbit/s test).

You didn't have 20 samples in that test though! Again, I wonder how to balance lots of samples, each tested by a few people, or a few samples, each tested by lots of people. Maybe a few to start with. Or would that be unhelpful, because it would be difficult to get people back for a second round of "the same thing" with different samples?

Quote

4. If this turns out to be a large test, then automation is almost certainly necessary. When I did the 64 kbit/s test, I had people send me email. If I had to do it over, there's no way I would go that route again. Some sort of web form in which people paste their results would probably be the best way to go.

This is completely beyond me. Unless I could get some help with this, it'll be emails, PMs, and a big tables of result in Excel. Which doesn't appeal to me either!

Cheers,
David.

Digital Radio listening test

Reply #17 – 2003-03-03 15:55:40

Quote

As many as possible. It sounded like question which you'd thought of an answer to? My answer would be publicise it here, and any other forum I could think of. What's yours?

That was my thought too. Slashdot helped out a lot for the 64 kbit/s test. Possibly you could write an article for kuro5hin.org first, and then have somebody slashdot it a little bit later to keep up momentum.

Quote

You didn't have 20 samples in that test though! Again, I wonder how to balance lots of samples, each tested by a few people, or a few samples, each tested by lots of people. Maybe a few to start with. Or would that be unhelpful, because it would be difficult to get people back for a second round of "the same thing" with different samples?

Maybe a dozen might be ok. I don't know if you'd get enough people to fill out enough data for 20 samples. I think starting with the large number would be better than going with a few at first, for exactly the reason you mention.

Quote

This is completely beyond me. Unless I could get some help with this, it'll be emails, PMs, and a big tables of result in Excel. Which doesn't appeal to me either!

In this area, I probably can help. I can write a script to capture data to my server, which I can then zip up and send to you. There is a chance that people could "stuff the ballot," which is easier to catch if one uses email, but I can see no motivation for doing something like that.

I can also write programs or scripts to process the data, so you don't have to enter into Excel by hand. I hacked up several very ugly programs the last time -- I'm not going to make something like those publicly available. However, I could modify those and pass them along to you.

ff123

Digital Radio listening test

Reply #18 – 2003-03-03 23:02:14

Quote

You mean you don't read all the boards on the forum? Yes, yes, I know, things fly off the front page so quickly that it's easy to miss interesting threads these days.

I've bookmarked the "Active Postings" page of HA, and it got quite "foggy" recently.

Quote

Given the existance of all these tests, it's quite amazing that UK broadcasters are using 128kbps JS mp2 for almost all their digital radio broadcasts. Most wouldn't dream of using any higher bitrates - the commercial broadcasters are only doing what I would expect (due to the lack of regulation in this area) but I would have expected better from the BBC.

They probably are of the "never change a running system" kind...

Quote

I don't expect this test to help that much, but it will be useful to have some new data, gathered using a wide range of listeners, with realistic, representative test material. And beside all this, the results will be very interesting.

OK, but I'm not sure if it will reveal large differences to the MPEG tests that already targeted that bitrate range. So maybe you will invest a lot of time and effort with no significantly differing results that you could use to convince those DAB people in the U.K., if they won't believe the official test. But I can also understand the reasons you mention, e.g.:

Quote

1. How big is the effect of "unclean" sources (transcoding/D>A>D/etc)
2. What bitrate is transparent for people in the test (i.e. not just golden ears) using undemanding material

So all the best for your "MP2@128kbps stinks!" comparison...

Before I forget: if you want to use PsyTEL AACEnc for the 96 kbps AAC sample, don't forget to resample it to 32 kHz. This is also true for the -internet (~100 kbps) and -radio (~80 kbps) presets, if you decide to try them, too. And QuickTime 6 Pro can also deliver a good quality at this bitrate, see this thread on the Audiocoding.com forum:

http://www.audiocoding.com/phorum/read.php...1&i=2219&t=2219

By the way, maybe something like the Wiki could come in handy for gathering the user results? Since everyone can paste them into a Wiki page easily, it might serve as a quick platform. Of course it relies on the integrity of the people that participate in your test.

And last but not least: the recommended number of testers is at least 40 and samples at least 8 in the EBU listening tests (see e.g. http://www.audiocoding.com/wiki/index.php?...istinguishable)

Digital Radio listening test

Reply #19 – 2003-03-04 16:53:18

Thank you for the advice ff123 and Hans. All taken.

My thoughts are taking shape now. In this test, I want to compare factors which may improve DAB audio quality. These factors are:
a) bitrate
b) processing
c) transcoding
d) choice of encoder

It may turn out that the bitrate is the least of the problems, or it may turn out that, given a sufficiently high bitrate, some of the others are less of a problem.

Now, a question. If it turns out that, say, I end up with 8 bitrate/encoder combinations, I think that's too much. 10 would be even worse, but let's not go there.

You suggested ff123 that maybe they could be split across different tests, so no listener compared them all in one test, but overall, they were all tested against each other. Maybe that would work, but how? For example, it seems to me that people adapt the quality scale to fit the codecs they are presented. So, if in test A they happened to get all the good quality codecss, and in test B the happened to get all the poor quality codecs, then in test A they may mark lower than they should, and in test B they may mark higher than they should, simply due to the lack of high and low achor points in the tests.

However, if you ensure that each test includes a low and high anchor, then you're only left with 2-4 real codecs per test. If you choose 4 from 8 at random to present to each listener, then with enough listeners you'll get enough answers for the overall result to be meaningful. But doesn't it require many more listeners than if all codecs (e.g. 6) were auditioned by each listener? And do the statistics exist to check for significance in such a test?

I must include:

CD, resampled to 48kHz, encoded to ...

mp2 @ 256, 192, 160, 128, 112 kbps.
(There are rumours of plans to lower the minimum DAB bitrate to 112kbps, so it's worth including).
(There are several european stations at 256kbps, so it's worth including).

AAC @ 128, 96 kbps.
(No relevance here in Europe, but very relevant to US digital radio - and as many potential testers will probably be from the states, I think I need to include these to get broad interest)

I would also like to include a digital simulation of a good FM channel.

I also want to include at least one transcoded sample (probably mp2 128 twice or 192 twice; once at 44.1, once at 48) because many UK stations sound like they are doing this!

Finaly, for some samples, I want to do two separate versions:
1. CD, resampled to 48k, encoded to <whatever>
2. CD, dynamic range compression, resampled to 48kHz, encoded to <whatever>
These would be in separate tests - I wouldn't ask a listener to compare the two in a single test. For the "transcode" codec(S), one encode would be at 44.1kHZ before the DRC, the other would be at 48kHz after the DRC. Just like radio stations appear to do at present!

That's it. No actual DAB or FM recordings.

I make that 9 codecs (5 mp2, 2 AAC, 1 FM, 1 transcode). That's too many! Suggestions?

Cheers,
David.

(and I haven't even thought about comparing different mp2 and aac encoders - gulp!)

Digital Radio listening test

Reply #20 – 2003-03-04 20:45:12

Quote

You suggested ff123 that maybe they could be split across different tests, so no listener compared them all in one test, but overall, they were all tested against each other. Maybe that would work, but how?

Well, http://www.soundexpert.info works this way (interesting preliminary results, by the way... )

Quote

For example, it seems to me that people adapt the quality scale to fit the codecs they are presented. So, if in test A they happened to get all the good quality codecss, and in test B the happened to get all the poor quality codecs, then in test A they may mark lower than they should, and in test B they may mark higher than they should, simply due to the lack of high and low achor points in the tests.

Right, that's why the EBU chose to expand their standard listening tests to the new MUSHRA method (MUlti Stimulus test with Hidden Reference and Anchors), so that the listeners would always have a low quality sample and the original within each test sequence to "rest their ears" and/or normalize their hearing ability. See http://www.ebu.ch/trev_283-kozamernik.pdf (in case you don't know it already).

Quote

And do the statistics exist to check for significance in such a test?

Yes, see Chapter 4.9 of this Technical Review.

Quote

I make that 9 codecs (5 mp2, 2 AAC, 1 FM, 1 transcode). That's too many! Suggestions?

One or two less MP2 samples, maybe only 96 kbps AAC.

Digital Radio listening test

Reply #21 – 2003-03-04 21:16:17

Quote

It may turn out that the bitrate is the least of the problems, or it may turn out that, given a sufficiently high bitrate, some of the others are less of a problem.

You may wish to perform a small preliminary test so that you have an idea of what's going to happen, even if that means just you sitting with headphones going through some of the permutations. A small group of trusted listeners might work too.

Quote

However, if you ensure that each test includes a low and high anchor, then you're only left with 2-4 real codecs per test. If you choose 4 from 8 at random to present to each listener, then with enough listeners you'll get enough answers for the overall result to be meaningful. But doesn't it require many more listeners than if all codecs (e.g. 6) were auditioned by each listener? And do the statistics exist to check for significance in such a test?

Let me look through my book tonight to see how it works. I don't remember seing a statistical method for the exact way you propose (always include a high and low anchor). But now that you mention it, that does sound like the way to do it. The statistical method that EBU proposes is basically a mean and 95% confidence intervals. It doesn't adjust for the fact that many different samples might be under test, which would tend to yield higher type I errors (false positives). However, for what you're doing, it might be good enough.

ff123

Edit: BTW, the EBU paper recommends sticking with one type of transducer for all participants, and keeping volume within +/- 4 dB of a reference level. It is my opinion that not following these recommendations will tend to reduce the sensitivity of the test, but at the same time, it may make it more representative of real-world listeners.

Digital Radio listening test

Reply #22 – 2003-03-04 23:52:05

Just an Idea..

meybe you could contact a broadcasting station and ask them what kind of dymamics compression they use for their broadcasts and apply similiar filters to the original disc samples before encoding so the difference between your encodes and the FM and the Digital radio recording is less noticeable.

Digital Radio listening test

Reply #23 – 2003-03-05 08:15:41

I'd remove 112 and 160kbps for Layer II in order to reduce the number of samples.

Digital Radio listening test

Reply #24 – 2003-03-05 09:55:59

Quote

Just an Idea..

meybe you could contact a broadcasting station and ask them what kind of dymamics compression they use for their broadcasts and apply similiar filters to the original disc samples before encoding so the difference between your encodes and the FM and the Digital radio recording is less noticeable.

That's an excellent idea.

Unfortunately, they're using expensive Orban Optimod units, which I don't have access to. (It's a shame I've left uni - the radio station had one!). The other problem is that the "sound" of the station is created by the many settings which can be adjusted on an Optimod - most stations are very secretive about such things, and wouldn't let anyone know their settings.

However, if anyone is willing to send some samples through an optimod for me, I would happily take them up on their offer! It would be better than the mess I'm creating in Cool Edit Pro at the moment - though it's an authentic sounding "Independent Local Radio station with too much compression" mess!

Cheers,
David.

Notice