## Topic: Multiformat Listening Test @ 64 kbps - FINISHED (Read 89060 times)previous topic - next topic

0 Members and 1 Guest are viewing this topic.
• Rio
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #25 – 16 August, 2007, 10:11:39 AM
This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.
"Listen to me...

• naylor83
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #26 – 16 August, 2007, 10:58:36 AM

If I ranked the reference, will the result text file say so? Or will it just not show a result for that file?
davidnaylor.org

Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #27 – 16 August, 2007, 11:00:11 AM
The decrypted result files will then contain the rating you gave for the reference.

Edit: It will look like this:

[...]
2L File: Sample08\Sample08.wav
2L Rating: 4.5
2L Comment: blah
[...]

• pdq
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #28 – 16 August, 2007, 11:07:31 AM

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.

I would say rather that A>C and the B is approximately equal to A and approximately equal to C, but is not necessarily A>B>C since there is a possibility that either B>A or B<C (but not both).

• benski
• Developer
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #29 – 16 August, 2007, 11:09:36 AM

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.

No.

There is a chance that A>B but also a chance that A<B.
There is a chance that B>C but also a chance that B<C.
A>C

To rank them, A and B are tied for first.  C is third.
Given the data set, the "true" rank has three possibilities.  ABC, BAC, ACB.  However, more samples would be necessary to determine this.

One thing I've always disliked about these tests is that, given the subjective nature of the ratings, the deviation in participants' rating style is likely larger than the standard deviation.

• naylor83
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #30 – 16 August, 2007, 11:13:19 AM

I'm trying to work out which samples are which contenders.

I realize number 3 is Vorbis, and that number 4 must be low anchor. But I'm confused about the others...
davidnaylor.org

• guruboolez
• Members (Donating)
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #31 – 16 August, 2007, 11:17:16 AM
You can use MrQuestionMan, foobar2000 or several other tools to check these files :

1: WMAPro (losslessly compressed due to the lack of WMA CLI decoder)
2: high anchor (iTunes LC-AAC at ~100 kbps)
3: vorbis (ogg fileformat)
4: low anchor (iTunes LC-AAC at 48 kbps)
5: HE-AAC (Nero Digital AAC at ~64 kbps).

• naylor83
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #32 – 16 August, 2007, 11:19:22 AM
1: WMAPro (losslessly compressed)
2: high anchor (LC-AAC at 96 kbps)
3: vorbis (ogg fileformat)
4: low anchor (LC-AAC at 48 kbps)
5: HE-AAC

Thanx.
davidnaylor.org

• ff123
• Developer (Donating)
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #33 – 16 August, 2007, 11:20:10 AM
One thing I've always disliked about these tests is that, given the subjective nature of the ratings, the deviation in participants' rating style is likely larger than the standard deviation.

In the analysis, each listener is treated as a separate "block", which takes into account the fact that different listeners have individual rating styles.

• Whelkman
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #34 – 16 August, 2007, 12:26:06 PM
In the analysis, each listener is treated as a separate "block", which takes into account the fact that different listeners have individual rating styles.

Thanks. I wondered about this. I doubt I applied consistent "objective" ratings across the board, but codecs were always ranked compared to each other.

Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #35 – 16 August, 2007, 04:39:05 PM
Does anyone know how to make Excel to refer to the current table when creating a plot? I have a document with 19 tables and I thought about plotting the results for the first sample and then copying and pasting this in the other 17 documents and then only changing the values. However, if I copy and paste a plot, the pasted plots still refer to the source table. Then if I change the data source, some of the plot formatting is gone, such as the margins, the vertical grid and the grid color.

Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #36 – 16 August, 2007, 06:31:31 PM
Uploaded the plots for each sample. The corresponding text is still missing, though, although there isn't much to say since all three were tied in almost every case.

Off-Topic: That listening test page needs rework badly. The design could be better and maybe offer some help for newbies.

• mezenga
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #37 – 16 August, 2007, 07:17:04 PM
Does anyone know how to make Excel to refer to the current table when creating a plot?
Maybe joining all 19 tables in a big one and making a single one for the plot. This single table should change its content among one of the 19 blocks from the big table. That would be my approach for a dynamic plot.

• ff123
• Developer (Donating)
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #38 – 16 August, 2007, 07:18:39 PM
Interesting.  he-aac had some clear winners over wmapro10, whereas there were none the other way around.  Poets of the fall and Bachpsichord are particularly striking.  Choice of samples is pretty critical in these tests.

• echo
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #39 – 16 August, 2007, 07:36:18 PM

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

not an expert, but at leas mathematically if A=B and B=C, A=C.

Mathematically yes, but this is not math, this is statistics.

To put it in simple terms, without any statistical talk, this means that A is probably equal to B, B is probably equal to C, while A is greater than C. Try to think "equal" like "roughly equal".

• rockcake
• Members (Donating)
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #40 – 17 August, 2007, 12:32:10 AM
I'd also like to give a big thankyou to Sebastian for organising another test (and publishing the results amazingly quickly!), especially under difficult circumstances e.g. HDD failure, widespread apathy, moving house etc. etc.  You're a legend!
Vorbis -q3 works for me.

• TechVsLife
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #41 – 17 August, 2007, 02:43:56 AM
I'd also like to give a big thankyou to Sebastian for organising another test (and publishing the results amazingly quickly!), especially under difficult circumstances e.g. HDD failure, widespread apathy, moving house etc. etc. You're a legend!

Or does such superhuman generosity border on insanity? Is his undying fame worth the terrible price he pays--with his very life etc. (Life itself is a 64 kbps lossy compression where you have to pick carefully what to carry to get to a half-decent harmony, but discerning ears will always be able to pick up the falseness, especially in critical passages.)

But seriously, thanks for the hard work, even if insane,
--and how about the next test! (128 kbps mp3?).

• IgorC
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #42 – 17 August, 2007, 03:05:48 AM
Thanks for test. Nero has done a good work.

• Alexxander
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #43 – 17 August, 2007, 04:30:42 AM
Congrat Nero!

I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR)

• halb27
• Developer
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #44 – 17 August, 2007, 05:21:59 AM
... I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR)

It's rather the other way around. The beleive in VBR's universal superiority has simply no good basis. Moreover there seems to be a common misconception that a constant frame bitrate (CBR) means constant audio data bitrate which is simply wrong. Maybe WMP10pro CBR offers a higher degree of audio data bitrate variation than for instance mp3 CBR. But even without it there's really no reason to think that constant bitrate automatically means reduced quality.
There's no contradiction to the fact that Vorbis, NeroAAC, MPC, Lame 3.98 are good at VBR.
Everything depends on codec principles and - may be to a larger extent - implementation details.
lame3995o -Q1.7
opus --bitrate 140

• Developer
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #45 – 17 August, 2007, 05:22:24 AM
Congrat Nero!
I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR)

Thanks!

Considering CBR: CBR is more dependent on choice of samples. It is expected that Nero would perform on this sample set a bit better when CBR 64kbps is used (most probably not enough to be statistically better than WMA). From this test it can also be concluded that VBR mode in Nero doesn't have big flaws.

• Alexxander
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #46 – 17 August, 2007, 06:29:09 AM
...Moreover there seems to be a common misconception that a constant frame bitrate (CBR) means constant audio data bitrate which is simply wrong. Maybe WMP10pro CBR offers a higher degree of audio data bitrate variation than for instance mp3 CBR...

So CBR actually means constant frame bitrate? I thought CBR referred to constant audio data bitrate, like plain old PCM: for example sampling 8000 times per second at fixed intervals with 8 bits per sample. Then, if frame bitrate is constant but audio bitrate varies within a frame it's actually VBR but only on a different timescale. It all depends on the exact definitions and the correct use of terms (as always).

Thanks for clearing up.

• Ivan Dimkovic
• Developer
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #47 – 17 August, 2007, 06:40:16 AM
CBR, in this context, means:  "Fixed bit rate within a fixed (predictable) period, or fixed amount of data"

Most "CBR" codecs are actually variable bitrate, but they have relatively small "bit buffer" which is constant in size and known a-priori, and that provides variations in frame bit rate.  Within those limits, codec has full freedom to allocate bits.

Even within a single frame, bits are allocated in the variable sense - depending on the psychoacoustic threshold, etc...

So, in a nutshell - "CBR" in the modern audio codec is way different than "CBR" in PCM sense - both frames and  individual samples are coded with different, variable, accuracies.

• Alex B
Multiformat Listening Test @ 64 kbps - FINISHED
##### Reply #48 – 17 August, 2007, 07:45:01 AM
Sebastian,

Could you possibly post the average results per sample as a table like this (in the original sample order):

Code: [Select]
`WMA    High    Vorbis    Low    Nero2.60    4.00    1.70    1.00    3.302.00    3.50    2.00    1.00    3.002.80    4.00    2.30    1.00    2.703.40    4.00    3.10    1.00    3.702.40    3.60    2.20    1.00    2.302.10    3.50    1.70    1.00    2.501.70    2.50    2.00    1.00    1.702.20    3.40    3.00    1.00    2.601.60    3.20    2.30    1.00    2.603.10    3.50    2.80    1.00    2.602.60    3.50    2.40    1.00    2.801.80    3.40    2.00    1.00    1.802.90    3.80    2.30    1.00    2.603.00    3.90    2.00    1.00    2.702.00    3.70    2.30    1.00    1.703.00    4.00    2.10    1.20    2.102.30    3.50    2.80    1.00    1.803.40    4.00    3.40    1.00    3.10`

I would like to draw a chart in the following format, but it would be quite laborious to grab the values from the result images.

Alex B's personal results:
some archived old tests: http://listening-tests.freetzi.com