Skip to main content

Topic: Multiformat Listening Test @ 128 kbps - FINISHED (Read 141424 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Pio2001
  • [*][*][*][*][*]
  • Global Moderator
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #50
Quote
Pio, have you automated somehow the process of compiling this personal result table of yours?
[a href="index.php?act=findpost&pid=357231"][{POST_SNAPBACK}][/a]


No, I typed it manually using the results published in the rar file, and the tables of encoder IDs.

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #51
I received a total of 467 results. Since only 403 are valid, I had to delete 64 results.
  • Last Edit: 15 January, 2006, 06:45:35 AM by Sebastian Mares

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #52
BTW, could someone post the results to Doom9?

  • halb27
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #53
Wonderful results!
lame3995o -Q1

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #54
It may be interesting to observe the performance's evolution of various formats according to the collective tests already performed. A rigorous comparison wouldn't make sense (samples and listeners aren't the same), but some tendency may appear.

[span style='font-size:14pt;line-height:100%']Summer 2003:[/span]

) and for more and more people. When HA.org was found in 2001, such quality at this bitrate was only a dream; four years later it becomes our reality. I take my hat off to all developers
  • Last Edit: 15 January, 2006, 10:50:29 AM by guruboolez

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #55
Managed to decode my own results:

Code: [Select]
                               iTunes   LAME    Nero    Shine   AuTuV  WMA pro                    
01 BigYellow                    4.9     4.9     5.0     1.0     5.0     5.0
02 bodyheat                     5.0     5.0     5.0     1.0     5.1x    5.2x
03 Carbonelli                   5.7x    3.0     4.7     4.0     5.9x    4.9     Focus on 0.00 - 5.55
04 Coladito                     5.5x    5.2x    4.9     1.0     4.7     5.4x    Focus on wosh - wosh 5.63 - 10.88
05 DontLetMeBeMisunderstood     5.2x    5.0     5.0     9.0y    5.0     5.0     Focus on 4.16 - 6.46
06 yello                        5.0     5.0     5.0     1.0     5.0     5.1x
07 Elizabeth                    5.0     5.0     5.0     5.0     5.0     5.0     Focus 0.00 - 3.71
08 eric_clapton                 5.2x    5.1x    4.5     1.0     5.8x    6.0x    Focussed on 9.96 - 14.69
09 ReunionBlues                 5.7x    4.1     4.0     1.0     5.3x    5.2x    Focussed on 24.26 - 27.06
10 LesJoursHeureux              5.2x    4.4     4.6     2.0     5.8x    4.0     Focus on 0.10 - 5.18
11 macabre                      4.9     4.9     5.1x    1.0     4.9     5.0
12 MysteriousTimes              5.0     5.0     5.0     3.0     5.0     5.0     Focus on 0.00 - 3.13
13 ravel                        4.6     5.2x    4.8     1.0     5.4x    5.2x    Focus on 4.08 - 7.93
14 School                       4.7     5.2x    4.6     2.1     4.3     4.0     Focused on 8.18 - 13.49
15 Senor                        5.0     5.0     5.0     1.0     5.0     5.0
16 SongForGuy                   4.8     2.0     5.0     5.3x    3.0     5.4x    Focus 2.20 - 4.97
17 TheDraperyFalls              4.8     5.0     4.9     1.0     5.2x    4.8     Focus on 17.21 - 26.31
18 WhiteAmerica                 5.2x    4.2     4.6     1.0     5.6x    4.8     Focus 0.00 - 7.74

Average                         4.93    4.58    4.81    1.84    4.83    4.86
Ranked reference                 7x      4x      1x      1x      8x      7x

x = Ranked reference. Rating calculated as 10 - rate of reference in the above table to indicate rank given to reference.
y = Pulled the wrong slider on the low anchor

Only managed to avoid ranking the reference for all encoders in samples 1, 7, 12, and 15. 

Funny observation: The ranking of references doesn't seem to be random as the encoders most often having ranked references are the top codecs of this test. 

I know my results wouldn't change anything with respect to the overall result the multiformat listening test. However next time around please specify the criteria for discarding result files. 

I could then have used a different approach in my testing that would not result in ranked references. 

Any suggestions for improving my approach to ABC/HR are welcome.

Edit: Added average. Removed direct question for Sebastion about which of my results were used.
  • Last Edit: 15 January, 2006, 08:49:39 AM by sehested

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #56
What was the mail address you used for testing (you can send me a PM if you don't want it to be public)?

Nevermind found it - decrypting results...
  • Last Edit: 15 January, 2006, 07:51:21 AM by Sebastian Mares

  • SirGrey
  • [*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #57
Thanks for working on the test, Sebastian.
As for me, I can hardly ABX most modern codecs at this bitrate from the original.  And can not ABX one from another.
So it really seems that it is time to lower a bitrate a bit in a such public tests 

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #58
sehested, you are anon10.

I used your results for BigYellow, Elizabeth, Mysterious Times and Senor.
  • Last Edit: 15 January, 2006, 07:56:55 AM by Sebastian Mares

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #59
By the way... People who did not enter their (nick)name in ABC/HR (like sehested) are going to have anonXX in front of their result.
Please don't ask me who has which number since I don't know it by heart.

One more thing - IIRC, a tester entered his name for all results except one. The respective result is also marked as anonXX since the tester might've had a reason for not disclosing his name for that single result.
  • Last Edit: 15 January, 2006, 08:03:16 AM by Sebastian Mares

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #60
Quote
sehested, you are anon10.

I used your results for BigYellow, Elizabeth, Mysterious Times and Senor.
[a href="index.php?act=findpost&pid=357280"][{POST_SNAPBACK}][/a]

Thanks! I better include my name with the test results next time around.

  • bond
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #61
thx again for this interesting test, sebastian!

btw can you create a final zoomed in plot without the anchor and without the nero results plz, its nicer to point people to (most newbies will propably not read or understand the whole nero explanation)
I know, that I know nothing (Socrates)

  • minisu
  • [*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #62
Quote
thx again for this interesting test, sebastian!

btw can you create a final zoomed in plot without the anchor and without the nero results plz, its nicer to point people to (most newbies will propably not read or understand the whole nero explanation)
[a href="index.php?act=findpost&pid=357289"][{POST_SNAPBACK}][/a]

Wouldn't that mislead beginners even more since the point that none of the tested encoders is proved to be better than another?

(I don't mind if you do, just wanted to point out that there might be a risc...)
  • Last Edit: 15 January, 2006, 09:06:55 AM by minisu
Opera bookmark synchronizer: http://osync.sf.net

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #63
Quote
thx again for this interesting test, sebastian!

btw can you create a final zoomed in plot without the anchor and without the nero results plz, its nicer to point people to (most newbies will propably not read or understand the whole nero explanation)
[a href="index.php?act=findpost&pid=357289"][{POST_SNAPBACK}][/a]


The final zoomed plot does not contain the anchor and I am not going to remove Nero since I see no point in doing so. If people want to read about the Nero problem, they can follow the link from the plot.

  • bond
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #64
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable

sebastian wrote
Quote
Because of the mentioned problems (unfairness, no real-life relevance...) and after discussing the issue with Francis, Roberto Amorim (rjamorim on Hydrogenaudio Forums) and Darryl Miyaguchi (ff123 on Hydrogenaudio Forums) thoroughly, I decided, against Ivan's and Juha's suggestion, to exclude Nero from the test.
because of this exclusion i think there should be also a final plot provided that doesnt mention nero

edit:

Quote
Quote
thx again for this interesting test, sebastian!

btw can you create a final zoomed in plot without the anchor and without the nero results plz, its nicer to point people to (most newbies will propably not read or understand the whole nero explanation)
[a href="index.php?act=findpost&pid=357289"][{POST_SNAPBACK}][/a]


The final zoomed plot does not contain the anchor and I am not going to remove Nero since I see no point in doing so. If people want to read about the Nero problem, they can follow the link from the plot.
[a href="index.php?act=findpost&pid=357291"][{POST_SNAPBACK}][/a]
so the exclusion of nero from the test is not reason enough to provide a final plot without the excluded nero?

well ok, if this makes sense for you... 
  • Last Edit: 15 January, 2006, 09:15:36 AM by bond
I know, that I know nothing (Socrates)

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #65
Quote
Quote
On the total results: how much % of the gradings gave a "transparent" mark, if we exclude shine?

The same question but adding the graded references as 5.0 for the codec in question?
[a href="index.php?act=findpost&pid=357241"][{POST_SNAPBACK}][/a]


Quote
Is there a way to see your own results, including results for discarded result files?
[a href="index.php?act=findpost&pid=357242"][{POST_SNAPBACK}][/a]


Sorry, I didn't understand both of you. 
[a href="index.php?act=findpost&pid=357246"][{POST_SNAPBACK}][/a]


Let me rephrase: How many of the grades given were 5.0? And how much if you re-add the ranked references as meaning that codec got a 5.0 for that sample?

So, there's 403 valid test results times 5 codecs (Shine doesn't count), or about 2015 grades. How many of those are 5.0, i.e. perfectly transparent?
  • Last Edit: 15 January, 2006, 09:31:10 AM by Garf

  • yulyo!
  • [*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #66
When can we see the final version of Nero's new AAC encoder? (5)
I am curoius how Nero would compete if there was not bug 

  • bond
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #67
Quote
I am curoius how Nero would compete if there was not bug 
[a href="index.php?act=findpost&pid=357298"][{POST_SNAPBACK}][/a]

nero always performs better in a not yet available version ™

/cynism
I know, that I know nothing (Socrates)

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #68
Quote
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
[a href="index.php?act=findpost&pid=357292"][{POST_SNAPBACK}][/a]

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.

  • bond
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #69
Quote
Quote
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
[a href="index.php?act=findpost&pid=357292"][{POST_SNAPBACK}][/a]

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.
[a href="index.php?act=findpost&pid=357303"][{POST_SNAPBACK}][/a]

yeah, thats why i meant there should be both, a final plot with nero (as currently available) and one without, that can be thrown on newbies without making them think nero performed as its shown on the plot (even if its red and with the link to the explanation)
I know, that I know nothing (Socrates)

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #70
Quote
well ok, if this makes sense for you... 
[a href="index.php?act=findpost&pid=357292"][{POST_SNAPBACK}][/a]


You can always create the plot yourself. 

Quote
Let me rephrase: How many of the grades given were 5.0? And how much if you re-add the ranked references as meaning that codec got a 5.0 for that sample?

So, there's 403 valid test results times 5 codecs (Shine doesn't count), or about 2015 grades. How many of those are 5.0, i.e. perfectly transparent?
[a href="index.php?act=findpost&pid=357297"][{POST_SNAPBACK}][/a]


Geez, no idea. That would take too much time - time that I don't have right now. I could send you all results if you really want to do it yourself.

  • bond
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #71
Quote
Quote
well ok, if this makes sense for you... 
[a href="index.php?act=findpost&pid=357292"][{POST_SNAPBACK}][/a]


You can always create the plot yourself. 

you wish we play:
removed

everyone feel free to link to it
  • Last Edit: 15 January, 2006, 10:12:23 AM by bond
I know, that I know nothing (Socrates)

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #72
Quote
Quote
Quote
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
[a href="index.php?act=findpost&pid=357292"][{POST_SNAPBACK}][/a]

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.
[a href="index.php?act=findpost&pid=357303"][{POST_SNAPBACK}][/a]

yeah, thats why i meant there should be both, a final plot with nero (as currently available) and one without, that can be thrown on newbies without making them think nero performed as its shown on the plot (even if its red and with the link to the explanation)
[a href="index.php?act=findpost&pid=357305"][{POST_SNAPBACK}][/a]




Damn, you were faster!
  • Last Edit: 15 January, 2006, 09:58:32 AM by Sebastian Mares

  • bond
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #73
Quote
Quote
Quote
Quote
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
[a href="index.php?act=findpost&pid=357292"][{POST_SNAPBACK}][/a]

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.
[a href="index.php?act=findpost&pid=357303"][{POST_SNAPBACK}][/a]

yeah, thats why i meant there should be both, a final plot with nero (as currently available) and one without, that can be thrown on newbies without making them think nero performed as its shown on the plot (even if its red and with the link to the explanation)
[a href="index.php?act=findpost&pid=357305"][{POST_SNAPBACK}][/a]




Damn, you were faster!
[a href="index.php?act=findpost&pid=357309"][{POST_SNAPBACK}][/a]

lol yeah happy now 

tough i would be even more happy if it would be shown on the results page too
I know, that I know nothing (Socrates)

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #74
OK, results page now contains 3 final graphs: non-zoomed with Nero, zoomed with Nero and zoomed without Nero.

BTW, could you please remove your plot, bond? I would like people to link to my graphs in case I change something.
  • Last Edit: 15 January, 2006, 10:11:56 AM by Sebastian Mares