Skip to main content
Topic: Multiformat@128 public listening test - CANCELLED (Read 36702 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multiformat@128 public listening test - CANCELLED

Hello.

I'd like to announce the opening of my Multiformat at 128kbps public listening test.

The test consists of comparing how 6 encoders: iTunes AAC, Ogg Vorbis, Lame MP3, Musepack, WMA Std. and Atrac3 behave compressing 18 different samples representing a wide spectrum of musical styles.

People interested in participating are invited to visit this page:
http://www.rjamorim.com/test/multiformat12...esentation.html

The test will end on May 16th and results will be posted soon after.

Best regards;
Roberto.

Multiformat@128 public listening test - CANCELLED

Reply #1
i'm in

downloading from bittorrent

first time, will try to do my best... 

+

Multiformat@128 public listening test - CANCELLED

Reply #2
I'd like to help but, unbelievable as it might seem, the last time I checked I couldn't distinguish a l3enc 1.0 encode from MPC Q8 :/ It's quite sad...

Anyway rjamorim thanks and congrats, I'm looking forward to the results

rgd


Multiformat@128 public listening test - CANCELLED

Reply #4
Let's download!

Multiformat@128 public listening test - CANCELLED

Reply #5
Just for the next test:

Would be nice to include a shell script/Makefile for decoding on the non-windows platforms. Wouldn't be too much work and sure is pretty easy.

Doing the decoding by hand is error-prone too.

Multiformat@128 public listening test - CANCELLED

Reply #6
Is saving/loading sessions disabled for this test?

If it's not, consider this a bug report:

using MacOS X 10.3.3
java version "1.4.2_03" JRE Standard Edition (build 1.4.2_03-117.1)

apple.awt.EventQueueExceptionHandler Caught Throwable :
java.lang.NullPointerException
        at abchr.gui.modulecontrols.m.a(Unknown Source)
        at abchr.gui.cb.c(Unknown Source)
        at abchr.gui.bq.actionPerformed(Unknown Source)
[...]

Multiformat@128 public listening test - CANCELLED

Reply #7
Quote
Just for the next test:

Would be nice to include a shell script/Makefile for decoding on the non-windows platforms. Wouldn't be too much work and sure is pretty easy.

Doing the decoding by hand is error-prone too.

http://www.comp.nus.edu.sg/~s0300325/decodeall.txt

Just don't take this as an example of how one should make shell scripts... It worked for me...

Multiformat@128 public listening test - CANCELLED

Reply #8
great!

btw roberto you write "Ogg Vorbis auTuV tuning", it should say aoTuV
I know, that I know nothing (Socrates)

Multiformat@128 public listening test - CANCELLED

Reply #9
Don't be too sensitive on my reply, I just want to get an impression of peoples usage/utilisation of 128kbps-files.
Are you using 128kbps as it is a passable weighting of quality and diskusage with the focus on using transportable players or to have results which are comparable with older listening tests or just because it's too hard to rate files with higher bitrates (for example 192 kbps).
To go lossy I only use LAME's aps (sometimes musepack, but mp3 is still more usable with (cheap) everyday-life-hardware like Standalones (SA) or discmen, also my friends are more happy to get stuff in a format they get along with  ) and this mostly on my Desktop-PC, so I guess (I don't really know, that's why I am asking) I am not in the target-group this test is started for or ?

...and now I have to laugh, just read your words at your page (stupid as I am I first started writing as my brain gave me a *ping* '128kbps? - so where's the need if you want to have fun with your music?'). Oh man, things become better, at least this is my hope.
As I spend time with writing this stuff I spend these lyrics to outta space... don't argue about useless, spacewasting words.

Cheers

Multiformat@128 public listening test - CANCELLED

Reply #10
Roberto,

Regarding bit torrents.
I have a suggestion to make, I'm not saying you should change it now but it might be something to think about next time.

People may find it easier to keep torrents open if you could bundle all the samples together in one archive for the bit torrent release.

Personally I use the default windows bit torrent client which opens one window for each torrent. I have 7 of them open for upload now and will keep them open for the next couple of days. They are swamping my start bar as it is, to open 18 would make it a little tricky to use my computer normally.


Thanks,

Luke

Multiformat@128 public listening test - CANCELLED

Reply #11
Is it permitted to discuss here which areas of samples to inspect for weaknesses?

Sometimes I find it can take quite a bit of listening to pick up a flaw which then become obvious in most of the sample as soon as you have detected it.

It might save people listening time and to share these points.

I also understand if you think it is best to have no discussion during the test.

Luke

Multiformat@128 public listening test - CANCELLED

Reply #12
Quote
To go lossy I only use LAME's aps (sometimes musepack, but mp3 is still more usable with (cheap) everyday-life-hardware like Standalones (SA) or discmen, also my friends are more happy to get stuff in a format they get along with ) and this mostly on my Desktop-PC, so I guess (I don't really know, that's why I am asking) I am not in the target-group this test is started for or ?

I think you are missing the point of this test. 128kbps is a good bitrange to listen to for two main reasons (as far as I can see):

1. People can actually do it. If we had a test of lame standard and it's equivalents then we would struggle to get meaningful results with real world samples. I am sure I am in the majority when I say that at 190kbps LAME, Musepack, Vorbis and the rest sound identical to me (except for some known 'evil' samples). What this means is that except for a small number of talented listeners, all we would get is 5.0 ratings for everything.

2. It's a useful bitrate. For streaming. For small (solid state) portables. For short samples on download sites. 128kpbs is a nice medium between 'listenable' and 'huge'.

Anyways, I am busy downloading. This is the first test I will post results for, so hopefully I do ok.

Multiformat@128 public listening test - CANCELLED

Reply #13
Quote
Personally I use the default windows bit torrent client which opens one window for each torrent. I have 7 of them open for upload now and will keep them open for the next couple of days. They are swamping my start bar as it is, to open 18 would make it a little tricky to use my computer normally.

Have you considered using ABC yet? This client is very convenient when you want to download several torrents simultaneously. And it has a global upload limit setting, so the uploads won't slow down your downloads too much.
Over thinking, over analyzing separates the body from the mind.

Multiformat@128 public listening test - CANCELLED

Reply #14
PoisonDan, that is a nice client, I have set it up now, thanks.

I've never had any need for anything apart from the standard client before now but this does give you more control.

(I think I would still prefere everything to be in one torrent in the future but it is a minor point.)

Luke

Multiformat@128 public listening test - CANCELLED

Reply #15
Ok for the first time I take part in a listening test.
I've begun with "hongroise", and at fisrt I couldnt make any difference between files. But then .... I noticed the length of some tracks slightly differ from others : it's then obvious to chose the original file, even if there is no hearable difference.

But there is something more serious : at the beginig of the original track, there is a VERY DISTURBING "clic".... And this click disappears in some tracks !

What must I do ? Put a bad mark (2-3) to those files because I can easily distinguish them from the original, or put a good mark (4-5) because they are even more pleasant than the original? 

Edit : to be sure I made an ABX test... Here are the results

ABX log
ABC/HR for Java Version 0.4b4
May 6, 2004 2:59:50 PM

Sample A: Hongroise.wav
Sample B: Hongroise_XXX.wav
Playback Range: 00.000 to 30.000
    2:55:17 PM p 1/1 pval = 0.5
    2:55:21 PM p 2/2 pval = 0.25
    2:55:25 PM p 3/3 pval = 0.125
    2:55:29 PM p 4/4 pval = 0.062
    2:55:31 PM p 5/5 pval = 0.031
    2:55:34 PM p 6/6 pval = 0.015
    2:55:36 PM p 7/7 pval = 0.0070
    2:55:39 PM p 8/8 pval = 0.0030
    2:55:41 PM p 9/9 pval = 0.0010
    2:55:43 PM p 10/10 pval < 0.001
    2:55:45 PM p 11/11 pval < 0.001
    2:55:55 PM p 12/12 pval < 0.001
    2:57:48 PM p 13/13 pval < 0.001
    2:57:52 PM p 14/14 pval < 0.001
    2:57:54 PM p 15/15 pval < 0.001
    2:58:00 PM p 16/16 pval < 0.001
    2:58:03 PM p 17/17 pval < 0.001
    2:58:07 PM p 18/18 pval < 0.001
    2:58:09 PM p 19/19 pval < 0.001
    2:58:11 PM p 20/20 pval < 0.001
    2:58:21 PM p 21/21 pval < 0.001
    2:58:24 PM p 22/22 pval < 0.001
    2:58:25 PM p 23/23 pval < 0.001
    2:58:27 PM p 24/24 pval < 0.001
    2:58:31 PM p 25/25 pval < 0.001
    2:58:33 PM p 26/26 pval < 0.001
    2:58:35 PM p 27/27 pval < 0.001
    2:58:41 PM p 28/28 pval < 0.001
    2:58:44 PM p 29/29 pval < 0.001
    2:58:47 PM p 30/30 pval < 0.001
    2:58:49 PM p 31/31 pval < 0.001
    2:58:59 PM p 32/32 pval < 0.001
    2:59:01 PM p 33/33 pval < 0.001
    2:59:03 PM p 34/34 pval < 0.001
    2:59:06 PM p 35/35 pval < 0.001
    2:59:10 PM p 36/36 pval < 0.001
    2:59:13 PM p 37/37 pval < 0.001
    2:59:15 PM p 38/38 pval < 0.001
    2:59:17 PM p 39/39 pval < 0.001
    2:59:18 PM p 40/40 pval < 0.001
    2:59:20 PM p 41/41 pval < 0.001
    2:59:22 PM p 42/42 pval < 0.001
    2:59:24 PM p 43/43 pval < 0.001
    2:59:26 PM p 44/44 pval < 0.001
    2:59:30 PM p 45/45 pval < 0.001
    2:59:32 PM p 46/46 pval < 0.001
    2:59:36 PM p 47/47 pval < 0.001
    2:59:38 PM p 48/48 pval < 0.001
    2:59:40 PM p 49/49 pval < 0.001
    2:59:42 PM p 50/50 pval < 0.001

---------
Total: 50 out of 50, p < 0.001

O fault, and each time less than 4 seconds to find the good track 
(I've hidden the track)

Multiformat@128 public listening test - CANCELLED

Reply #16
Quote
Is it permitted to discuss here which areas of samples to inspect for weaknesses?


I think this is somewhat similar to "is practise allowed"?

I hope so. Otherwise the test would have to be carried out with completely naive (as in domain knowledge/skills naive) testers.

I suspect that is not the purpose of this test, considering the forum in which it was announced

Multiformat@128 public listening test - CANCELLED

Reply #17
I found the problem.
The version of vorbis used by the test was old. The ogg file was not created by aoTuV beta2. (It's aoTuV experiment [20040402])  If aoTuV test page to "aoTuV beta2 Win32 OggEnc-locale fix" (file name is oggenc_aotuv_b2m.zip) downloads and a suitable file is encoded, it will output the file which is clearly different.

vendor name is
aoTuV experiment [20040402] = AO; aoTuV b2 test [20040402] (based on Xiph.Org's 1.0.1)
aoTuV beta2 [20040420] = AO; aoTuV b2 [20040420] (based on Xiph.Org's 1.0.1)
...

[EDIT]
Correction of an expression

Multiformat@128 public listening test - CANCELLED

Reply #18
I have the same problem as Zurman - this time with the Kraftwork sample

The .wavs are fine, but when played in ABCHR, there's a click at the start of the original, but not all the coded versions (2 out of six have clicks).

Maybe ABCHR is reading some data in the wav file as audio?

This is a very serious problem.

I'm using WinXP, audiophile 2496, just installed Java.


Also, are the files being ReplayGained in ABCHR? If so, why?! If not, why is playback so quiet?

Cheers,
David.

Multiformat@128 public listening test - CANCELLED

Reply #19
Quote
Ok for the first time I take part in a listening test.
I've begun with "hongroise", and at fisrt I couldnt make any difference between files. But then .... I noticed the length of some tracks slightly differ from others : it's then obvious to chose the original file, even if there is no hearable difference.

But there is something more serious : at the beginig of the original track, there is a VERY DISTURBING "clic".... And this click disappears in some tracks !

Possibly the clicking noises you're hearing are the same type that were reported for the last test, and unfortunately are associated with the abchr-java program.  You can try increasing the buffer size under Options->Settings->Playback (default is 500 ms).  Also sometimes just moving the starting position of the playback range using the selection bar down at the bottom will eliminate the clicking.

The short answer is that you should not downrate quality because of this clicking noise.  Also, you should not downrate quality because you can tell the difference in files because of a difference in length.

ff123

Multiformat@128 public listening test - CANCELLED

Reply #20
Quote
I have the same problem as Zurman - this time with the Kraftwork sample

The .wavs are fine, but when played in ABCHR, there's a click at the start of the original, but not all the coded versions (2 out of six have clicks).

Maybe ABCHR is reading some data in the wav file as audio?

This is a very serious problem.

I'm using WinXP, audiophile 2496, just installed Java.


Also, are the files being ReplayGained in ABCHR? If so, why?! If not, why is playback so quiet?

The java library lowers the volume, for some unknown reason.  Hopefully this is not too serious of a problem, since this occurs for all samples equally.

The clicking is serious, though, I agree.  But there is not much one can do about it.

ff123

Multiformat@128 public listening test - CANCELLED

Reply #21
Quote
The version of vorbis used by the test was old. The ogg file was not created by aoTuV beta2. (It's aoTuV experiment [20040402])  If aoTuV test page to "aoTuV beta2 Win32 OggEnc-locale fix" (file name is oggenc_aotuv_b2m.zip) downloads and a suitable file is encoded, it will output the file which is clearly different.

Oh no!

This is bad...
Over thinking, over analyzing separates the body from the mind.

Multiformat@128 public listening test - CANCELLED

Reply #22
Quote
Is it permitted to discuss here which areas of samples to inspect for weaknesses?

Sometimes I find it can take quite a bit of listening to pick up a flaw which then become obvious in most of the sample as soon as you have detected it.

It might save people listening time and to share these points.

I also understand if you think it is best to have no discussion during the test.

Luke

I think the answer to this would have to be no.

Discussion would bias the results.

ff123

Multiformat@128 public listening test - CANCELLED

Reply #23
Quote
Quote
The version of vorbis used by the test was old. The ogg file was not created by aoTuV beta2. (It's aoTuV experiment [20040402])  If aoTuV test page to "aoTuV beta2 Win32 OggEnc-locale fix" (file name is oggenc_aotuv_b2m.zip) downloads and a suitable file is encoded, it will output the file which is clearly different.

Oh no!

This is bad... 

Oh my god

Multiformat@128 public listening test - CANCELLED

Reply #24
Quote
Quote
Ok for the first time I take part in a listening test.
I've begun with "hongroise", and at fisrt I couldnt make any difference between files. But then .... I noticed the length of some tracks slightly differ from others : it's then obvious to chose the original file, even if there is no hearable difference.

But there is something more serious : at the beginig of the original track, there is a VERY DISTURBING "clic".... And this click disappears in some tracks !

Possibly the clicking noises you're hearing are the same type that were reported for the last test, and unfortunately are associated with the abchr-java program.  You can try increasing the buffer size under Options->Settings->Playback (default is 500 ms).  Also sometimes just moving the starting position of the playback range using the selection bar down at the bottom will eliminate the clicking.

The short answer is that you should not downrate quality because of this clicking noise.  Also, you should not downrate quality because you can tell the difference in files because of a difference in length.

ff123

I agree, I shouldn't.

But I will for sure be influenced by this clic or even the length by the tracks 

By the way even with the buffer @ 2000 ms, the noise is still here 

 
SimplePortal 1.0.0 RC1 © 2008-2019