Skip to main content

Topic: Multiformat Listening Test @ 128 kbps - FINISHED (Read 141650 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • kwanbis
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #25
could a ranking based on rating and bitrate be made? also as somebody already pointed, it would be good to include the version of the encoders used on the graphs, as many people would be linking to them.

edit:by the way, great job guys.
  • Last Edit: 14 January, 2006, 06:13:49 PM by kwanbis

  • mdmuir
  • [*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #26
When are we reaching the point that further tests in the future become superfluous? Just from looking at the results of this test, not one codec is significally better than another one. Can we safely say "stick to lame for universal compatibilty or take your pick for whatever your hardware device will suport"

Choice for compatibilty:

1. lame

2. apple aac/nero aac

3. vorbis

4.wma pro-good codec, nothing but computer software plays it
you will make mp3's for compatibility reasons.

  • kwanbis
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #27
Quote
When are we reaching the point that further tests in the future become superfluous? Just from looking at the results of this test, not one codec is significally better than another one. Can we safely say "stick to lame for universal compatibilty or take your pick for whatever your hardware device will suport"

i prety much agree ... ony thing is gapless option on some of them ...

  • kwanbis
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #28
Quote
Quote
Quote from: Sebastian Mares,Jan 14 2006, 03:41 Can you also add more complete info of the encoder in the graph? e.g.: AoTuv -> Vorbis AoTuV 4.51, Nero -> AAC Nero 3.1.0.2, etc... I prefer first the format, then vendor and version. Many sites link only to the image and can give a confusing idea of which encoder was used. Also on that image should be a link to the full explanation of the results.[/quote


Well, if someone posts the image, he should also post to results page.
Adding the full encoder version / information is useless IMHO - it's stated already on the presentation page (which can be accessed if you are on the results page, that is supposed to be posted together with the plot).

i agree with fpi, i would add encoder, version, and probably command line. Even if i link to your page, people most of the time just look at the graph.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #29
Quote
When are we reaching the point that further tests in the future become superfluous?
[a href="index.php?act=findpost&pid=357164"][{POST_SNAPBACK}][/a]

At this bitrate, further tests are indeed questionable. Quality of the tested encoders is apparently too high for most listeners at ~130 kbps - at least for those interested by participating in such tests. The 192 kbps syndrom has now reached the 128 kbps area: it's beyond most listeners abilities, including HA.org members' one. At this stage, all people who can't differenciate MP3 from Vorbis or AAC and interested by these formats should try to lower the bitrate (I guess that it's already the case for many of them).
It's maybe the last 128 kbps multiformat collective test organized here. The next "mid/high" collective test should maybe lower the pretension and be limited to 100...112 kbps.

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #30
Quote
i agree with fpi, i would add encoder, version, and probably command line. Even if i link to your page, people most of the time just look at the graph.


You know, space is limited. Writing "VBR/Stereo - Streaming, 100-120 kbps [LC AAC]" in the graph is pretty much overkill. Anyways, I will see what I can do tomorrow.
  • Last Edit: 14 January, 2006, 06:27:06 PM by Sebastian Mares

  • Halcyon
  • [*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #31
Thanks for the results and work everyone!

I could only offer 6 results, not only because I fell ill, but because trying to spot the differences (assumed anchor excluded) it was really hard work!

It also showed me how some samples truly are more useful (at least for me).

The louder, more compressed and "busy" samples (like metal, rap and pop) were not nearly as useful for me as the acoustic and classical tracks. Not that they were easy either!

c. 128 kbps level has gone a long way in the past few years. I'm almost afraid to think when we'll reach the similar performance with 96kbps  Not that I'm complaining, it's all for good.
  • Last Edit: 14 January, 2006, 06:45:09 PM by Halcyon

  • kwanbis
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #32
Quote
You know, space is limited. Writing "VBR/Stereo - Streaming, 100-120 kbps [LC AAC]" in the graph is pretty much overkill. Anyways, I will see what I can do tomorrow.
[{POST_SNAPBACK}][/a]

you can do better than this paint shop pro hack for sure, but its the idea:

[a href="http://imageshack.us][/url]
  • Last Edit: 14 January, 2006, 06:54:57 PM by kwanbis

  • vinnie97
  • [*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #33
Vorbis' place at the top correlates with it being the highest bitrate.

Good results from everyone all around.  Thanks for the test.

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #34
Quote
And BTW, this was a funny result:

Code: [Select]
 ABC/HR for Java, Version 0.5b, 06 december 2005 
 Testname: DontLetMeBeMisunderstood
 
 Tester: 
 
 1R = Sample05\DontLetMeBeMisunderstood_1.wav
 2L = Sample05\DontLetMeBeMisunderstood_2.wav
 3L = Sample05\DontLetMeBeMisunderstood_3.wav
 4L = Sample05\DontLetMeBeMisunderstood_6.wav
 5R = Sample05\DontLetMeBeMisunderstood_4.wav
 6R = Sample05\DontLetMeBeMisunderstood_5.wav
 
 ---------------------------------------
 General Comments: Focus on 4.16 - 6.46
 ---------------------------------------
 1L File: Sample05\DontLetMeBeMisunderstood.wav
 1L Rating: 4.8
 1L Comment: 
 ---------------------------------------
 5L File: Sample05\DontLetMeBeMisunderstood.wav
 5L Rating: 1.0
 5L Comment: 
 ---------------------------------------
 
 ABX Results:
[a href=\"index.php?act=findpost&pid=357129\"][{POST_SNAPBACK}][/a]
I'm the culpit here 

I started by listening to all encoders to locate the low anchor.
Apparently I pulled the wrong slider on sample 5, although I had it nailed.

Next time I better ABX the samples to lock the reference slider, in order to eliminate this kind of stupid "pressing the wrong button" mistakes.

I hope your are able to use my other test results, even though I'm positive that I on several occasions have rated the reference.

Anyway great work Sebastian.

  • richard123
  • [*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #35
Do these results help answer the question: Are these encoders transparent to most of those participating in the tests?

  • westgroveg
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #36
Does the results page state what settings where used for each encoder?

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #37
SoundExpert preliminary results on the same contenders are here. Alternative testing will end on 22 Jan. Details and discussion are in this thread.
keeping audio clear together - soundexpert.org

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #38
Quote
By the way, results that were invalid (didn't meet ABX minimums) were not uploaded. Since I posted the encryption key, you can decrypt the results yourself if you are wondering why your result is not counted.
[{POST_SNAPBACK}][/a]
Sebastian,

Could you please elaborate on the criteria used for invalidating a result file?

I'm also interested in knowing how many result files have been discarded and for what reasons.

Previously discussions have taken place on criterias for [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=18474&view=findpost&p=189044]discarding result files[/url].

However I seem to have missed any discussions on the subject prior to this listening test.

How would the consolidated result look if you would have used results were the reference where rated slightly below 5.0?

In the AAC @ 128 kbps listening test only using "clean" files did not change the outcome of the test. Can the same be said for this test?

  • Pio2001
  • [*][*][*][*][*]
  • Global Moderator
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #39
Thank you for the results. I was Anon08.
Here are my individual results. Every mark different from 5.0 has been validated with a successful ABX test.

Code: [Select]
Sample  AoTuV AAC-Itunes AAC-Nero Lame Shine WMApro
01      5.0   4.5        5.0      2.0  2.0   5.0  
03      5.0   5.0        5.0      2.0  2.0   5.0
04      5.0   5.0        5.0      5.0  2.0   5.0
05      5.0   5.0        3.0      5.0  1.0   5.0
06      5.0   5.0        5.0      5.0  1.0   5.0
07      5.0   5.0        5.0      5.0  3.0   5.0
08      5.0   4.0        4.0      3.0  1.5   4.5
10      5.0   5.0        5.0      5.0  2.0   5.0


I can tell that I dislike MP3
  • Last Edit: 14 January, 2006, 11:48:20 PM by Pio2001

  • plonk420
  • [*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #40
i only got thru 10 of the tracks.... but i could only ABX the diff on maybe 1 or 2 of the samples out of the 4 or 5 X 10 tracks i tried.....!! i blame my noisy DLP projector >_> (where i have my nice speakers setup) ;-\

  • Raptus
  • [*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #41
Pio, have you automated somehow the process of compiling this personal result table of yours?

  • Raptus
  • [*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #42
As I stated in the other topic, I was not very happy with the sample selection. Anyway I managed to descriminate more than average:
Code: [Select]
Sample iTunes LAME Nero Shine AoTuv WMA Pro
1      3,2    3,2  4    1     5     3,5
2      5      4,5  4    1     5     4,5
3      5      2,2  5    1,5   4     3,5
4      4      3,2  3,5  1     5     2,8
5      5      4    5    1,5   4     3,5
6      4,4    2,8  4    1,5   5     3
7      5      4    4,5  2,5   5     4,5
8      5      3,5  4    1     5     3,5
9      5      4    3,5  1     5     4
10     5      4    4    1,5   5     3,5
11     3,5    4    4    1,5   5     3,5
12     4,5    3,5  4    1,5   4,5   3,5
13     5      3,5  5    1     5     4
14     5      4    4,5  1     5     4
15     4,5    3,5  4    1     5     3,5
16     5      4    4    1     5     4
17     5      3    4    1     5     3,5
18     5      4    5    1     5     5

AVG    4,67   3,61 4,22 1,25  4,86  3,74

Congrats to the conducer and everyone who participated 

EDIT: Corrected a number
  • Last Edit: 15 January, 2006, 03:49:26 AM by Raptus

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #43
Quote
Does the results page state what settings where used for each encoder?
[a href="index.php?act=findpost&pid=357197"][{POST_SNAPBACK}][/a]


No, but on the presentation page.

Quote
Quote
By the way, results that were invalid (didn't meet ABX minimums) were not uploaded. Since I posted the encryption key, you can decrypt the results yourself if you are wondering why your result is not counted.
[a href="index.php?act=findpost&pid=357124"][{POST_SNAPBACK}][/a]
Sebastian,

Could you please elaborate on the criteria used for invalidating a result file?[a href="index.php?act=findpost&pid=357204"][{POST_SNAPBACK}][/a]


If the results contain ranked references and no ABX logs, the results are invalid.

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #44
Quote
If the results contain ranked references and no ABX logs, the results are invalid.
[a href="index.php?act=findpost&pid=357238"][{POST_SNAPBACK}][/a]
Well that is new to me. I thought that slightly ranking a reference would result in a 5.0 rating, not invalidating the result file for that sample.

BTW: Were was this mentioned?

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #45
On the total results: how much % of the gradings gave a "transparent" mark, if we exclude shine?

The same question but adding the graded references as 5.0 for the codec in question?

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #46
Is there a way to see your own results, including results for discarded result files?

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #47
Quote
On the total results: how much % of the gradings gave a "transparent" mark, if we exclude shine?

The same question but adding the graded references as 5.0 for the codec in question?
[a href="index.php?act=findpost&pid=357241"][{POST_SNAPBACK}][/a]


Quote
Is there a way to see your own results, including results for discarded result files?
[a href="index.php?act=findpost&pid=357242"][{POST_SNAPBACK}][/a]


Sorry, I didn't understand both of you. 

Anyways, I edited all graphs to show the encoder and its version. Additionally, I included a link to the full results page in the overall rankings and the zoomed plot. The encoder settings can be seen on the presentation page (which as I said can be reached from the results page).

Edit: sehested, do you want to see the results you submitted or what? If you still have the encrypted files, you can decrypt them using ABC/HR and the encryption key I linked to on the results page.
  • Last Edit: 15 January, 2006, 05:04:12 AM by Sebastian Mares

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #48
Quote
sehested, do you want to see the results you submitted or what? If you still have the encrypted files, you can decrypt them using ABC/HR and the encryption key I linked to on the results page.
[a href="index.php?act=findpost&pid=357246"][{POST_SNAPBACK}][/a]
Thanks, I will check it right away.

The other question I have is about the amount of result files that where discarded.

Can you give any numbers?

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #49
Quote
Quote
sehested, do you want to see the results you submitted or what? If you still have the encrypted files, you can decrypt them using ABC/HR and the encryption key I linked to on the results page.
[a href="index.php?act=findpost&pid=357246"][{POST_SNAPBACK}][/a]
Thanks, I will check it right away.

The other question I have is about the amount of result files that where discarded.

Can you give any numbers?
[a href="index.php?act=findpost&pid=357249"][{POST_SNAPBACK}][/a]


I am going to check ASAP.