Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: the be all end all listening test (Read 4700 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

the be all end all listening test

if everyone is so concerened over which format is better

(or at least ogg vs mpc or aac vs mpc)

why dont we arrange a large scale listening test among the people on this site?

what we could do is have someone with no bias towards either codec take a sample known to be abx-able by some people and encode it in the different formats.

next this person names the samples original and 1.wav and 2.wav and 3.wav etc...

then a lot of people abx the 1, 2, 3.wav etc against the original.

they send their results to whomever was doind the original encoding and he (or she) determines the results.

i think the only way this could be biased would be for someone to take the original and encode it to the different codecs themselves then decode it back to wav then compare it to the other test samples...

perhaps to throw this off a bit send one or 2 extra samples that are just copies of the original and 1 or 2 of each codec.

just a thought...

the be all end all listening test

Reply #1 –
Seems a nice idea.

And I believe the best person to manage this experiment is ff123, given his experience in the field...

I, for instance, am too biased towards AAC... 

Regards;

Roberto.

the be all end all listening test

Reply #2 –
That's basically what ff123 has done in his second listening test. In addition, he gets you to randomize the names of the files you listen to, so that person A's 1.wav is not the same as person B's 1.wav.

The only difference is that you are focusing on ABXing rather than ff123's focus on quality measurements (on the 1-5 scale). All a positive ABX gives you is that you can distinguish the two samples. It doesn't give you any indication as to which is considered the 'better' of the two. How to you propose getting this information from the testers?

the be all end all listening test

Reply #3 –
well if you hear a difference then it isnt as good as not hearing a difference....

basically we would be searching for transparent settings for diference codecs

the be all end all listening test

Reply #4 –
Basically, all we really need to have a high bitrate listening test. Since most people agree mp3 isn't even a contender for "transparency" (I hate using that term with lossy formats) just use mpc, psytel aac, and rc3 ogg. BTW, Roberto is right! There is NO ONE on earth I would trust more than ff123 to arrrange and supervise a test such as this...

the be all end all listening test

Reply #5 –
Thanks for the vote of confidence.  I'd like to close the current 128 test, and investigate some of the anomalous data I've received.  One data point in particular is truly astounding to me and I'd like to understand what's going on to improve future tests.

The only way I'd organize a high bitrate test now would either place a heavy burden on the listener (ABX every sample using 16 trials or more) or would require the use of a tool which doesn't currently exist.

ff123

the be all end all listening test

Reply #6 –
Spase, do you mean a high bit rate test, eg:

MPC  Extreme;
OGG  -q6 or 7;
AAC  (equiv to above);

all averaging around 200 - 230 kbps?

If you do, then this is going to be very taxing on the ears and statistical methods. It is already a common belief on this board that MPC, for instance, is transparent to artefacts in 99.8% of music clips. Indeed the only recent clip that appeared to have a detectable artefact was Madonna – You Only See What You Want To See. I think Garf could just ABX this clip with MPC Extreme, but it was on the edge of perception.

I haven’t followed the threads on OGG and AAC, but are there many samples that are not transparent with these encoders at high bit rate?

Of course, if you are talking about 128 kbps, then ff123 hosts these comparisons regularly in any event.

Anyway, It would be nice to have a high bit rate showdown, but it's going to need a lot of planning re suitable clips. Perhaps we should start a thread devoted to voting for the best clips for this test, ie the encoder nightmares. Each camp can pick what they think are the worst clips for the other encoders.
Ruse
____________________________
Don't let the uncertainty turn you around,
Go out and make a joyful sound.

the be all end all listening test

Reply #7 –
My idea is to try to use wave clips used in MPEG listening tests, since MPEG group has especially chosen these to "break" codecs and to show various aspects of coding artifacts that one codec can produce.

I have these files, so maybe we can reduce number of test cases to some value acceptable for internet transmission.

the be all end all listening test

Reply #8 –
Wow, that's an interesting idea Ivan. Are you talking about non-music generated code busting type waves?

the be all end all listening test

Reply #10 –
More useful to me would be a *low* bitrate showdown - 64kpbs or lower. Audiophiles might be interested in the miniscule deviations from transparency at 200kpbs, but I know that *I* wouldn't be able to detect any problems at that bitrate. Wouldn't it be interesting to have an objective test to set against all the commercial claims of (for example) WMA or MP3Pro being great at 64kpbs?

the be all end all listening test

Reply #11 –
Cooool! And you have the clips they used?

the be all end all listening test

Reply #12 –
Quote
Wouldn't it be interesting to have an objective test to set against all the commercial claims of (for example) WMA or MP3Pro being great at 64kpbs?
Uhmmm... I think you're on the wrong forum for a test like that.  Maybe I'm mistaken but I believe that most of the people on here (like me) are looking for "transparency".

the be all end all listening test

Reply #14 –
Quote
Originally posted by layer3maniac
Uhmmm... I think you're on the wrong forum for a test like that.  Maybe I'm mistaken but I believe that most of the people on here (like me) are looking for "transparency".


I dunno about you, but I would be interested in a test of 64kbps MP3Pro vs WMA vs Ogg

--
GCP

the be all end all listening test

Reply #15 –
Same here...since current codec development is concentrating on low bitrate rather than on squeezing the last bit of transparancy out of a already highly transparent codec....

the be all end all listening test

Reply #16 –
Hey Ivan... When I was looking around the mpeg site: http://mpeg.telecomitalialab.com/
I saw references to audio Mpeg 7. Has anyone implemented that yet? Does it have any merits?
 

the be all end all listening test

Reply #17 –
MPEG-7 and MPEG-21 don't have new codecs, they are built on top of existing codec infrastructure.