Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multiformat 64 kbps Listening Test (Read 132944 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multiformat 64 kbps Listening Test

Reply #100 –
Quote from: Gabriel on
Quote from: haregoo on
I'd like to exclude sample10 bibilolo and 13 aquatisme and change the length of sample 1 (43sec).
IMHO those samples are interesting because they seem to be hard to encode. Why removing them?
In the 48 kbps test I spent a lot more time with bibilolo than with any other sample. The bibilolo files were obviously very bad, so hearing various problems was not a problem. The problem was to rate the contenders fairly. On each listening time different problems seemed to be more annoying.

After spending an hour or so with bibilolo I ended up with this judgement:

Code: [Select]
ABC/HR for Java, Version 0.52b, December 07, 2006
Testname: Sample10: bibilolo

Tester: Alex B

1R = Sample10\Sample10_3.wav
2L = Sample10\Sample10_4.wav
3L = Sample10\Sample10_1.wav
4L = Sample10\Sample10_6.wav
5R = Sample10\Sample10_5.wav
6L = Sample10\Sample10_2.wav

Ratings on a scale from 1.0 to 5.0

---------------------------------------
General Comments: 
---------------------------------------
1R File: Sample10\Sample10_3.wav  (WMA Std)
1R Rating: 1.8
1R Comment: heavily low passed, but less distorded than #2
---------------------------------------
2L File: Sample10\Sample10_4.wav  (WMA Pro)
2L Rating: 1.5
2L Comment: highs have bad distortion
---------------------------------------
3L File: Sample10\Sample10_1.wav  (Vorbis)
3L Rating: 2.5
3L Comment: preecho, lowpass
---------------------------------------
4L File: Sample10\Sample10_6.wav  (low anchor)
4L Rating: 1.0
4L Comment: most of the sample is missing because of the lowpass
---------------------------------------
5R File: Sample10\Sample10_5.wav  (high anchor)
5R Rating: 3.5
5R Comment: 
---------------------------------------
6L File: Sample10\Sample10_2.wav  (Nero HE-AAC)
6L Rating: 2.1
6L Comment: Not so lowpassed, but the highest frequencies are "wrong" HE-AAC?
---------------------------------------

ABX Results:
[!--sizeo:1--][span style=\"font-size:8pt;line-height:100%\"][!--/sizeo--](I added the encoder names.)[/size]

However, I am not sure if I could repeatably end up with the same results.


Aquatisme was a killer sample too. There was no doubt about the huge amount of problems. In addition it was even more difficult to rate. I really couldn't put the contenders in any order. After trying for a little while I decided to pass the sample and move on.

Multiformat 64 kbps Listening Test

Reply #101 –
Quote from: Sebastian Mares on
Quote from: haregoo on

But I think there are too much sampels from unusual music in recent test.


Like what? I think the three ones mentioned are the only unusual samples at all.

3 of 18 are too many IMO. Your last 128kbps test didn't have such a sample at all.

ff123 explains listener psychology.
Quote
sample01
A note on listener psychology: they will tend to download and listen to sample01 first, and then decide whether they want to continue based on their experience on that first sample. I know that's what I do ;-) Ideally, there would be some sort of randomizer which assigns different music to each of the samples dynamically, but that would require some way to sort things out in the end. Barring that, I would try to make sample01 as friendly as possible.

I think this applies to overall samples. The preference of music highly depends on personal taste. But I don't think people feel like listening those samples time after time. I believe test itself has to motivate a listener since this is a collective test.

(spmg54_1 is a boring sample too. But spoken voice sample is interesting at this bitrate. 48 or 64kbps would be a good compromise between quality and bitrate for long speech encoding.)

Multiformat 64 kbps Listening Test

Reply #102 –
So you are proposing to exclude Aquatisme and Bibilolo because they are:
A - too hard to encode
B - do not suit your musical taste

Regarding A, 64kbps has 30% more bits than 48kbps, which should improve results. But you got an interesting point regarding potential difficulty to rank contenders.

Regarding B, at least there won't be complains about bias toward a specific musical genre
;-)

Multiformat 64 kbps Listening Test

Reply #103 –
My little musings on the sample-selection issue: I think the best ones to remove would be some of the ones that are of a similar style & therefore not adding anything major to the process e.g. if there are three (or maybe even two) samples of guitar-based pop, get rid of one of them.  I'd tend to favour keeping a wide range of sample styles to reflect the wide range of music that the HA community (or even the entire listening public) doubtless listens to.  I'd also tend to favour keeping 'difficult-to-encode' samples for research/intellectual purposes, but these samples are perhaps less important if very few people actually listen to that style of music recreationally - after all, that's the end purpose of music encoding, isn't it?
As regards the difficult-ranking issue raised by Alex B, I don't think one should get too hung up about it - if a given set of encodings sound about the same, then they should be ranked about the same.  (Conversely, if a sample sounds better to you, rate it higher!)  Each person is only one 'data point' as it were, and the end result will be a statistical one based on many people's responses - this is the point of having 'public' or 'community-based' listening tests i.e. to get multiple responses & thereby increase statistical validity, but also to see how things sound to the folks who are going to use these things.  I'm not saying that we shouldn't spend a little time trying to rank contenders, just that we shouldn't stress out about it.
Of course, this is all just "my 2 cents".
Regards,
rc
Edit: I feel a bit hypocritical reading this now, 'cos I vividly remember spending inordinate amounts of time trying to rank contenders in the last listening test I participated in! Oh, well...
Vorbis -q3 works for me.

Multiformat 64 kbps Listening Test

Reply #104 –
Quote from: rockcake on
... As regards the difficult-ranking issue raised by Alex B, I don't think one should get too hung up about it - if a given set of encodings sound about the same, then they should be ranked about the same.  (Conversely, if a sample sounds better to you, rate it higher!)  Each person is only one 'data point' as it were, and the end result will be a statistical one based on many people's responses - this is the point of having 'public' or 'community-based' listening tests i.e. to get multiple responses & thereby increase statistical validity, but also to see how things sound to the folks who are going to use these things.  I'm not saying that we shouldn't spend a little time trying to rank contenders, just that we shouldn't stress out about it. ...

Besides being generally very different from the reference all aquatisme and bibilolo samples were also clearly different from each other, like each sample was encoded from a different version of the source file. It was difficult to put the various obvious artifacts in an annoyance order because these samples were not any kind of familiar music.

Multiformat 64 kbps Listening Test

Reply #105 –
When is the test starting? Not so much discussion anymore, it seems.

Multiformat 64 kbps Listening Test

Reply #107 –
I think I will remove Locomotive Breath since it's way too long and also not very special. At 48 kbps, the modern contenders were tied and received a pretty good rating.

What else to exclude?

Multiformat 64 kbps Listening Test

Reply #108 –
OK, no feedback - how lovely. Locomotive Breath and Debusy are the samples that I am going to remove. Debusy also got a very good rating at 48 kbps, so I expect it to be (near)transparent at 64 kbps. Any new samples you guys want to propose? I would like to prepare the test this week.

Multiformat 64 kbps Listening Test

Reply #109 –
So after all, the same samples as in previous 48 kbps test, will be used?
I think it would be good to cut a bit from beginning and end of Paganini_Allegro_spirituoso, because sample is long and there are lot of repetitions. Also WhiteAmerica can be shorter.

Multiformat 64 kbps Listening Test

Reply #111 –
Quote from: Sebastian Mares on
OK, I'll see what I can do. As for the overall samples selection - I have no problem including new ones, but nobody suggested anything yet.


Is an 8 second sample long enough?

/Kef

Multiformat 64 kbps Listening Test

Reply #114 –
I just send in our final encoder for the test. Still fixed at one q value, because we still need to do some tuning on some bitrates. Release will be pretty soon though.

Maybe I can already tell some of the changes that have been made for this new release:
* Linux version
* New q value mappings (now better matches http://www.hydrogenaudio.org/forums/index....howtopic=44310)
* Retuning of almost all bitrates
* Minor bugfixes in the psychoacoustic model
* Fix in PS encoder/decoder delay
* Fixed incorrect streamlength written in MP4 files
* Speed ups

Multiformat 64 kbps Listening Test

Reply #118 –
How is the preparation for the test going?

Multiformat 64 kbps Listening Test

Reply #120 –
Crap, spent the whole Saturday at Mann Mobilia looking for furniture - thought it would take only a few hours. 

I also haven't seen Roberto online for several days and he's the person who is usually hosting the samples and taking care of the torrents.

Anyways, what I wanted to ask - is Java 1.5 now finally available for all major operating systems like Windows, Linux and Mac OS?

Edit: BTW, I removed aquatisme but kept bibilolo. Hope nobody is mad at me because of this...

Multiformat 64 kbps Listening Test

Reply #121 –
OK, regarding the Java thingy, it seems that all users can download and run the latest ABC/HR. If you're on a system which does not support JRE 1.5, please use a Linux live CD like Ubuntu, SuSE, Fedora, Knoppix, whatever.

Roberto, can you host my samples again, pleeeeeze? *makes pretty eyes*

I have to get the test up and running since I sold my old WD 250 GB on eBay and the 500 GB Seagate that I ordered is no longer in stock. Will have to use Ubuntu live CD myself the next few days...  Yeah, I was dumb, I know...

Multiformat 64 kbps Listening Test

Reply #122 –
OK, here's the bitrate table

Code: [Select]
High Anchor     Low Anchor     Ogg Vorbis     Nero     WMA          Max Min Delta
97              48             75             57       64           75  57  18
97              48             73             60       64           73  60  13
109             48             72             73       64           73  64  9
96              48             65             52       64           65  52  13
104             48             61             64       64           64  61  3
103             48             61             63       64           64  61  3
105             48             85             56       64           85  56  29
104             48             65             60       64           65  60  5
101             48             75             65       64           75  64  11
96              48             65             68       64           68  64  4
103             48             71             65       64           71  64  7
106             48             89             71       64           89  64  25
108             48             64             76       64           76  64  12
106             48             72             73       64           73  64  9
110             48             69             69       64           69  64  5
96              48             68             43       64           68  43  25
107             48             72             64       64           72  64  8
103             48             66             65       64           66  64  2
----------------------------------------------------------
103             48             70             64       64

Anything critical?

Edit: Added average bitrate at the bottom of the table. High anchor is within 10% tolerance (105.6 kbps), so is Vorbis (70.4 kbps). Not worth mentioning Nero which hits exactly 64 kbps as well as low anchor and WMA which both use CBR coding.

Edit 2: Encoder settings:

Vorbis: -q 0
WMA Pro: Codec: WMA9PRO (it's actually 10, but the VBS file I use is outdated) / Mode: 0 (1-pass CBR) / Setting: 64_44_2_16
Nero: -q 0.24 but hardcoded anyways

High Anchor: iTunes LC-AAC 96 kbps VBR
Low Anchor: iTunes LC-AAC 48 kbps CBR

Multiformat 64 kbps Listening Test

Reply #124 –
OK, everything is set. Roberto is hosting the samples (thanks mate!) and I've got everything ready to go. I am waiting for Menno to send me the whole Nero Digital Audio package with tagger, decoder and encoder tomorrow morning. This package will be hosted on the listening test site or on some Nero server, I don't know yet.

So, the test will start tomorrow. Thanks to everyone who participated in the pre-test discussion.