Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 64 kbit/s test ended (Read 16019 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

64 kbit/s test ended

Reply #1
moved to news section

64 kbit/s test ended

Reply #2
Thanks for setting up and managing such an interesting test. I listened to all 12 samples, was frequently surprised by the scores I gave, and look forward to your statistical analysis.

64 kbit/s test ended

Reply #3
Darryl, when do you expect the analysis is ready and you publish the results? Very soon I guess?

I just finished the last 3 samples and mailed you the results. All 12 done.
Juha Laaksonheimo

64 kbit/s test ended

Reply #4
Analysis is complete, and no further data is being added to the dataset.

http://ff123.net/64test/results.html

I need to add a link to all the comments after I write a small script to pipe them to files.

A little more info on the results I accepted:  no results which rated the original less than 5 was included in the dataset.

Ratings which were accompanied by ABX results having a pvalue > 0.05 *and* which had at least one incorrect trial were discarded as well, unless the addition of the correct choice in the ABC/HR section pushed the pvalue below 0.05.  For example, a couple people scored ABX runs of 8/10.  With the addition of a correct rating response, the total becomes 9/11, which is significant.  Another example.  One person ABX'd 4/4.  I accepted this result, even though the pvalue was > 0.05.

Lessons learned:  I need a general comments field.  Several people pulled down sliders just to tell me that they couldn't hear a difference!  One person (you know who you are!) modified the text file to include a comment attached to a rating of 5.0, which is an impossible result for my program to produce.

I also need a checkbox which allows people to tell me if I'm allowed to attach nicks to their comments when I publish them.  For this test, unfortunately, everybody is anonymous.

ff123

64 kbit/s test ended

Reply #5
Thanks for the quick analysis, and thanks for the great test.
The final results looks reasonable, altho I'm  still chocked by bot MP3Pro and WMA's results...
Hope to see another test like this soon.

64 kbit/s test ended

Reply #6
Not by WMA 
But mp3PRO surprised us all I suppose?

Anyhow, well done!!  @ everyone who helped realizing this test!

64 kbit/s test ended

Reply #7
Very interesting... I imagine there's a *lot* more statistical mangling that can be done to this data, but this provides us with a great summary.

Just for a little more information, could you consider telling us
a) how many results were considered for each sample?
b) the bitrate reached by Vorbis -q 0?

(b) may help explain why Vorbis -q 0 scored so low in comparison with -b 64 on the Liszt sample.

EDIT: on (a): am I going *completely blind*? This is the second time in as many days that I've missed something utterly obvious...

64 kbit/s test ended

Reply #8
Quote
Originally posted by Jon Ingram
a) how many results were considered for each sample?


I think this is N= on the top of each graph.

Is it possible to merge all samples together and see if you can make global conclusions?

--
GCP

64 kbit/s test ended

Reply #9
Quote
Originally posted by Jon Ingram
Very interesting... I imagine there's a *lot* more statistical mangling that can be done to this data, but this provides us with a great summary.

Just for a little more information, could you consider telling us
a) how many results were considered for each sample?
b) the bitrate reached by Vorbis -q 0?

(b) may help explain why Vorbis -q 0 scored so low in comparison with -b 64 on the Liszt sample.


Ogg had a very low bitrate (in the forties) on all the classical samples, which is the way it should have been (Classical solos with their deep noise floors and simple harmonics are relatively easy).  But the real reason Ogg scored so low in both (and Beauty Slept as well) was a) the tuning behind noise normalization is still not perfect.  This is the very first release of that feature, and the test found flaws  b)  also the first release of new, more aggressive stereo modes and I think that they too need more analysis infrastructure driving them.

I expect Ogg's performance on Liszt and Bach to be very subpar NN performance.  The poor performance on BeautySlept and Waiting was most likely insufficient stereo analysis.  Ogg had the infrastructure to win those four samples, but the encoder didn't know how to do it yet (because I didn't know it would be necessary).

This test will make Ogg stronger :-)

64 kbit/s test ended

Reply #10
We believe in the OGG
This was interesting for now, it would be interesting to do a retest when you feel OGG is ready for a new round to test it's strength.

64 kbit/s test ended

Reply #11
Quote
Originally posted by MTRH
We believe in the OGG
This was interesting for now, it would be interesting to do a retest when you feel OGG is ready for a new round to test it's strength.


Ogg was ready for serious testing at this point, really, I didn't mean to imply it wasn't.  It went toe to toe with the very best the commercial world had to offer and won the most samples of any codec (yes, it had two entries, but if you discard -b --managed, it still won the same number, six).  That's beyond respectable... that's a plausible basis for having won the test outright.

Regardless, it's still relatively untested and this was the very first exposure the new low bitrate techniques had to hardcore distributed testing outside of a very small group of people.  Naturally, this turned up some new things to deal with, but it was the only way to turn those things up.

64 kbit/s test ended

Reply #12
OT:

monty, some people from my forum noticed overemphasized high frequencies when using -q 4. Could you please take a look at that ?

64 kbit/s test ended

Reply #13
I assume you already know or have plans for what to tune in OGG to improve the results of those samples that proved worst with OGG. That's what I meant for retesting

[EDIT]
@ Benjamin - Yes, I concur, it's one of the things that makes OGG very easy to tell apart from other encoders IMHO

64 kbit/s test ended

Reply #14
Quote
Originally posted by Garf

Is it possible to merge all samples together and see if you can make global conclusions?


Here is my attempt at a global conclusion:

I eyeballed each graph and ranked the codecs from best to worst on a 5 point scale, where 5 is best and 1 is worst.  Codecs that tied received an average of the rankings that tied.

Code: [Select]
mp3pro    oggq0    ogg64    wma8    aac

2.5    4.5    4.5    2.5    1

4    1.5    4    4    1.5

4    2.5    2.5    5    1

5    3.5    3.5    1    2

4    5    2    2    2

4    5    3    1.5    1.5

3.5    5    3.5    1    2

4.5    1    2.5    2.5    4.5

4.5    4.5    3    1    2

5    3.5    3.5    1    2

3    4.5    4.5    1.5    1.5

5    3.5    3.5    1    2


Then I ran this through the Friedman calculator at:

http://ff123.net/friedman/stats.html

and got the following

Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url]http://ff123.net/[/url]

Friedman Analysis



Number of listeners: 12

Critical significance:  0.05

Significance of data: 3.76E-04 (highly significant)

Fisher's protected LSD for rank sums:  15.182



Ranksums:



mp3pro   oggq0    ogg64    wma8     aac      

49.00    44.00    40.00    24.00    23.00  



---------------------------- p-value Matrix ---------------------------



        oggq0    ogg64    wma8     aac      

mp3pro   0.519    0.245    0.001*   0.001*  

oggq0             0.606    0.010*   0.007*  

ogg64                      0.039*   0.028*  

wma8                                0.897    

-----------------------------------------------------------------------



mp3pro is better than wma8, aac

oggq0 is better than wma8, aac

ogg64 is better than wma8, aac


So for the samples chosen for this test, mp3pro seems to have won purely on ranking points, while wma8 and aac are the definite losers.  A different set of samples would have a different ranking.

ff123

64 kbit/s test ended

Reply #15
It's amazing how well mp3 pro performed considering how generally trashed the format has been on HA. I guess it's because the average HA enthusiast doesn't encode at these bitrates. What I am DYING to see is how well aac with sbr will sound (and perform in one of these tests).

BTW, nice job ff123

64 kbit/s test ended

Reply #16
Quote
Originally posted by ff123:
Ratings which were accompanied by ABX results having a pvalue > 0.05 *and* which had at least one incorrect trial were discarded as well, unless the addition of the correct choice in the ABC/HR section pushed the pvalue below 0.05. For example, a couple people scored ABX runs of 8/10. With the addition of a correct rating response, the total becomes 9/11, which is significant. Another example. One person ABX'd 4/4. I accepted this result, even though the pvalue was > 0.05.
What about ABX-tests between different coded samples (e.g. ogg 64 against ogg -q 0)?

I'm a little surprised of the good results of WMA at BachS1007 (3.96. I gave it 2! With MP3pro 4.2), the bad result of MP3pro with LifeShatters (I thought it nearly transparent: 4.9, by far the highest score I awarded).

I have no personal winner: MP3pro ranked slightly better with an average score of 3.69 compared to 3.58 with OGG -q0.

64 kbit/s test ended

Reply #17
Quote
Originally posted by Continuum
What about ABX-tests between different coded samples (e.g. ogg 64 against ogg -q 0)?


I didn't touch these.  The only ABX results I was concerned with wer those involving the original.

BTW, I added the overall graph to the results page.

ff123

64 kbit/s test ended

Reply #18
have anyone noticed that Frank Klemm submitted a Cradle Of Filth song  ?
Frank, i didn't know you're a metalist :listen: !

64 kbit/s test ended

Reply #19
Ha, so I wasn't nuts when I found MP3Pro to be the leader during my initial testing. I don't think the MP3Pro samples sound good necessarily, but they seemed to control annoying artifacts more often than other encoders, particularly WMA8 and AAC. Ogg showed promise but it's clear that it can stand a bit more tuning.

Will I use MP3Pro now? No, because 64kbps is still too low for pleasant listening. I've been using and will continue to use Ogg -q 2 (96kbps nominal) for music storage on my small hard drive (10GB) at work.

64 kbit/s test ended

Reply #20
1. maybe the lame group would be interested in adding SBR support
2. time to revisit plusv ?
3. isn't mp3pro codec has the highest cutoff frequency comparing all other codecs @ 64kbps ?
4. if 3 is true, wouldn't the test results are somewhat expected ?
5. if 4 is true, wouldn't it be better to setup a codec listening test @ higher bitrates (~100kbps, for instance) ?
6. ff123, would you be interested in running another test ?

Dg.

64 kbit/s test ended

Reply #21
Quote
Originally posted by layer3maniac
It's amazing how well mp3 pro performed considering how generally trashed the format has been on HA. I guess it's because the average HA enthusiast doesn't encode at these bitrates. What I am DYING to see is how well aac with sbr will sound (and perform in one of these tests).
BTW, nice job ff123
Yes, Mp3Pro is trashed here because of its high bitrate performance, but not at lower bitrates at least by me.
See for example:
http://www.hydrogenaudio.org/forums/showth...p3pro#post17157
http://www.hydrogenaudio.org/forums/showth...p3pro#post25566
http://www.hydrogenaudio.org/forums/showth...p3pro#post14987
http://www.hydrogenaudio.org/forums/showth...3pros#post26201
Juha Laaksonheimo

64 kbit/s test ended

Reply #22
Quote
Originally posted by DSPguru
1. maybe the lame group would be interested in adding SBR support
2. time to revisit plusv ?

Gabriel Bouvigne is interested on PlusV but he has said at PlusV forums that he doesn't have time to work on it.

It seems that there has been zero development on PlusV since source & specification were released, files on PlusV homepage haven't changed at all and there haven't been mention of otherwise on forums either despite Gabriels improvement suggestions.

PlusV is still technically on very early stages and it doesn't offer enough low-bitrate quality to make it usable on Lame (or other codec).


Monty, when PlusV was announced i recall you said that you have something similiar in mind for low-bitrate Vorbis. Do i remember right? If yes, can you tell us more about it?

If no, do you think PlusV would be useful on Vorbis if it could be made patent-free for Vorbis use?

64 kbit/s test ended

Reply #23
Quote
Originally posted by JohnV
Yes, Mp3Pro is trashed here because of its high bitrate performance, but not at lower bitrates at least by me.


Now, now, John. Did I single you out? Dibrom also had some good quotes:

Dibrom: What could be the best solution overall in regards to quality, might be to just use mp3pro however. At these bitrates you will probably get better quality with mp3pro than you would with most other codecs even.

However, check out these quotes:

Cygnus X1: For the last time, MP3 PRO IS SHIT! Avoid it like the plague--same goes for VQF and WMA!

Phobos: it sux
mp3 pro uses a named SBR system that builds up part of the music on the fly, it really distorts the original music a lot, not acceptable for most users...
USE VORBIS INSTEAD OF MP3PRO!!!
vorbis quality itself beats mp3PRO with hands down...

Benjamin Lebsanft: did you try the Pre RC4 Releases of vorbis? I my opinion it beats mp3pro quite easily!


64 kbit/s test ended

Reply #24
Quote
Originally posted by layer3maniac


Now, now, John. Did I single you out? Dibrom also had some good quotes:

Dibrom: What could be the best solution overall in regards to quality, might be to just use mp3pro however. At these bitrates you will probably get better quality with mp3pro than you would with most other codecs even. 

However, check out these quotes:

Cygnus X1: For the last time, MP3 PRO IS SHIT! Avoid it like the plague--same goes for VQF and WMA!

Phobos: it sux 
mp3 pro uses a named SBR system that builds up part of the music on the fly, it really distorts the original music a lot, not acceptable for most users...
USE VORBIS INSTEAD OF MP3PRO!!!
vorbis quality itself beats mp3PRO with hands down...

Benjamin Lebsanft: did you try the Pre RC4 Releases of vorbis? I my opinion it beats mp3pro quite easily!
Yes, but you said that MP3pro is trashed at HA, although me and Dib clearly didn't trash it at all quality-wise at low bitrate.. 
I guess it's a question whose opinion you value more. Mine and Dib's, or these "it sux"-messages... I wouldnt like to see any of these "it sux" messages here anyway.

But now after the blind group listening test, everybody can make some real conclusions about the low bitrate quality, not based only on single person opinions..
Juha Laaksonheimo