Skip to main content

Topic: Multiformat Listening Test @ 64 kbps - FINISHED (Read 88246 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Rio
  • [*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #25
This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?


I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.
"Listen to me...
Never take unsolicited advice..."

  • naylor83
  • [*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #26
Stupid question alert:

If I ranked the reference, will the result text file say so? Or will it just not show a result for that file?
davidnaylor.org

Multiformat Listening Test @ 64 kbps - FINISHED
Reply #27
The decrypted result files will then contain the rating you gave for the reference.

Edit: It will look like this:

[...]
2L File: Sample08\Sample08.wav
2L Rating: 4.5
2L Comment: blah
[...]
  • Last Edit: 16 August, 2007, 11:00:52 AM by Sebastian Mares

  • pdq
  • [*][*][*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #28

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?


I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.

I would say rather that A>C and the B is approximately equal to A and approximately equal to C, but is not necessarily A>B>C since there is a possibility that either B>A or B<C (but not both).

  • benski
  • [*][*][*][*][*]
  • Developer
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #29

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?


I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.


No.

There is a chance that A>B but also a chance that A<B.
There is a chance that B>C but also a chance that B<C.
A>C


To rank them, A and B are tied for first.  C is third.
Given the data set, the "true" rank has three possibilities.  ABC, BAC, ACB.  However, more samples would be necessary to determine this.

One thing I've always disliked about these tests is that, given the subjective nature of the ratings, the deviation in participants' rating style is likely larger than the standard deviation.
  • Last Edit: 16 August, 2007, 11:15:25 AM by benski

  • naylor83
  • [*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #30
Stupid question alert (again):

I'm trying to work out which samples are which contenders.

I realize number 3 is Vorbis, and that number 4 must be low anchor. But I'm confused about the others...
davidnaylor.org

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #31
You can use MrQuestionMan, foobar2000 or several other tools to check these files :

1: WMAPro (losslessly compressed due to the lack of WMA CLI decoder)
2: high anchor (iTunes LC-AAC at ~100 kbps)
3: vorbis (ogg fileformat)
4: low anchor (iTunes LC-AAC at 48 kbps)
5: HE-AAC (Nero Digital AAC at ~64 kbps).
  • Last Edit: 16 August, 2007, 11:19:22 AM by guruboolez

  • naylor83
  • [*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #32
1: WMAPro (losslessly compressed)
2: high anchor (LC-AAC at 96 kbps)
3: vorbis (ogg fileformat)
4: low anchor (LC-AAC at 48 kbps)
5: HE-AAC


Thanx.
davidnaylor.org

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #33
One thing I've always disliked about these tests is that, given the subjective nature of the ratings, the deviation in participants' rating style is likely larger than the standard deviation.


In the analysis, each listener is treated as a separate "block", which takes into account the fact that different listeners have individual rating styles.

  • Whelkman
  • [*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #34
In the analysis, each listener is treated as a separate "block", which takes into account the fact that different listeners have individual rating styles.

Thanks. I wondered about this. I doubt I applied consistent "objective" ratings across the board, but codecs were always ranked compared to each other.

Multiformat Listening Test @ 64 kbps - FINISHED
Reply #35
Does anyone know how to make Excel to refer to the current table when creating a plot? I have a document with 19 tables and I thought about plotting the results for the first sample and then copying and pasting this in the other 17 documents and then only changing the values. However, if I copy and paste a plot, the pasted plots still refer to the source table. Then if I change the data source, some of the plot formatting is gone, such as the margins, the vertical grid and the grid color.

Multiformat Listening Test @ 64 kbps - FINISHED
Reply #36
Uploaded the plots for each sample. The corresponding text is still missing, though, although there isn't much to say since all three were tied in almost every case.

Off-Topic: That listening test page needs rework badly. The design could be better and maybe offer some help for newbies.

  • mezenga
  • [*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #37
Does anyone know how to make Excel to refer to the current table when creating a plot?
Maybe joining all 19 tables in a big one and making a single one for the plot. This single table should change its content among one of the 19 blocks from the big table. That would be my approach for a dynamic plot.
  • Last Edit: 16 August, 2007, 07:19:02 PM by mezenga

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #38
Interesting.  he-aac had some clear winners over wmapro10, whereas there were none the other way around.  Poets of the fall and Bachpsichord are particularly striking.  Choice of samples is pretty critical in these tests.

  • echo
  • [*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #39

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

not an expert, but at leas mathematically if A=B and B=C, A=C.

Mathematically yes, but this is not math, this is statistics. 

To put it in simple terms, without any statistical talk, this means that A is probably equal to B, B is probably equal to C, while A is greater than C. Try to think "equal" like "roughly equal".

  • rockcake
  • [*]
  • Members (Donating)
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #40
I'd also like to give a big thankyou to Sebastian for organising another test (and publishing the results amazingly quickly!), especially under difficult circumstances e.g. HDD failure, widespread apathy, moving house etc. etc.  You're a legend! 
Vorbis -q3 works for me.

  • TechVsLife
  • [*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #41
I'd also like to give a big thankyou to Sebastian for organising another test (and publishing the results amazingly quickly!), especially under difficult circumstances e.g. HDD failure, widespread apathy, moving house etc. etc. You're a legend!

Or does such superhuman generosity border on insanity? Is his undying fame worth the terrible price he pays--with his very life etc. (Life itself is a 64 kbps lossy compression where you have to pick carefully what to carry to get to a half-decent harmony, but discerning ears will always be able to pick up the falseness, especially in critical passages.)


But seriously, thanks for the hard work, even if insane,
--and how about the next test! (128 kbps mp3?).
  • Last Edit: 17 August, 2007, 02:44:30 AM by TechVsLife

  • IgorC
  • [*][*][*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #42
Thanks for test. Nero has done a good work.

  • Alexxander
  • [*][*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #43
Congrat Nero!

I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR)

  • halb27
  • [*][*][*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #44
... I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR)

It's rather the other way around. The beleive in VBR's universal superiority has simply no good basis. Moreover there seems to be a common misconception that a constant frame bitrate (CBR) means constant audio data bitrate which is simply wrong. Maybe WMP10pro CBR offers a higher degree of audio data bitrate variation than for instance mp3 CBR. But even without it there's really no reason to think that constant bitrate automatically means reduced quality.
There's no contradiction to the fact that Vorbis, NeroAAC, MPC, Lame 3.98 are good at VBR.
Everything depends on codec principles and - may be to a larger extent - implementation details.
  • Last Edit: 17 August, 2007, 05:28:27 AM by halb27
lame3995o -Q1

  • muaddib
  • [*][*][*][*]
  • Developer
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #45
Congrat Nero!
I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR)

Thanks!

Considering CBR: CBR is more dependent on choice of samples. It is expected that Nero would perform on this sample set a bit better when CBR 64kbps is used (most probably not enough to be statistically better than WMA). From this test it can also be concluded that VBR mode in Nero doesn't have big flaws.

  • Alexxander
  • [*][*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #46
...Moreover there seems to be a common misconception that a constant frame bitrate (CBR) means constant audio data bitrate which is simply wrong. Maybe WMP10pro CBR offers a higher degree of audio data bitrate variation than for instance mp3 CBR...

So CBR actually means constant frame bitrate? I thought CBR referred to constant audio data bitrate, like plain old PCM: for example sampling 8000 times per second at fixed intervals with 8 bits per sample. Then, if frame bitrate is constant but audio bitrate varies within a frame it's actually VBR but only on a different timescale. It all depends on the exact definitions and the correct use of terms (as always).

Thanks for clearing up.

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #47
CBR, in this context, means:  "Fixed bit rate within a fixed (predictable) period, or fixed amount of data"

Most "CBR" codecs are actually variable bitrate, but they have relatively small "bit buffer" which is constant in size and known a-priori, and that provides variations in frame bit rate.  Within those limits, codec has full freedom to allocate bits.

Even within a single frame, bits are allocated in the variable sense - depending on the psychoacoustic threshold, etc...

So, in a nutshell - "CBR" in the modern audio codec is way different than "CBR" in PCM sense - both frames and  individual samples are coded with different, variable, accuracies.

  • Alex B
  • [*][*][*][*][*]
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #48
Sebastian,

Could you possibly post the average results per sample as a table like this (in the original sample order):

Code: [Select]
WMA    High    Vorbis    Low    Nero
2.60    4.00    1.70    1.00    3.30
2.00    3.50    2.00    1.00    3.00
2.80    4.00    2.30    1.00    2.70
3.40    4.00    3.10    1.00    3.70
2.40    3.60    2.20    1.00    2.30
2.10    3.50    1.70    1.00    2.50
1.70    2.50    2.00    1.00    1.70
2.20    3.40    3.00    1.00    2.60
1.60    3.20    2.30    1.00    2.60
3.10    3.50    2.80    1.00    2.60
2.60    3.50    2.40    1.00    2.80
1.80    3.40    2.00    1.00    1.80
2.90    3.80    2.30    1.00    2.60
3.00    3.90    2.00    1.00    2.70
2.00    3.70    2.30    1.00    1.70
3.00    4.00    2.10    1.20    2.10
2.30    3.50    2.80    1.00    1.80
3.40    4.00    3.40    1.00    3.10


I would like to draw a chart in the following format, but it would be quite laborious to grab the values from the result images.

Alex B's personal results:

  • muaddib
  • [*][*][*][*]
  • Developer
Multiformat Listening Test @ 64 kbps - FINISHED
Reply #49
Could you possibly post the average results per sample as a table like this (in the original sample order):

It is possible to get that data using chunky on the complete test results which are available in .rar.