Hydrogenaudio Forums

Hydrogenaudio Forum => Validated News => Topic started by: Sebastian Mares on 2006-12-09 22:17:12

Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Sebastian Mares on 2006-12-09 22:17:12
The much awaited results of the Public, Multiformat Listening Test @ 48 kbps are ready.

Here is the results page: http://www.maresweb.de/listening-tests/mf-48-1/results.htm (http://www.maresweb.de/listening-tests/mf-48-1/results.htm)

(http://www.maresweb.de/listening-tests/mf-48-1/resultsz2.png)

Nero is first, followed by Vorbis and WMA Pro. which are tied on second place, WMA Standard is third and loses.

I think this test shows that with modern encoders, the quality at 48 kbps is acceptable and should be good enough for Internet streaming or portable use with cell phones for example. It's also interesting to see that WMA Professional perfomed quite well although it was the only contender that used CBR.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: rjamorim on 2006-12-09 22:43:08
Fantastic! Thank-you very much for your efforts, Camil
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Sebastian Mares on 2006-12-09 22:54:23
Don't mention it.

Anyways, congrats to Nero and its devs for a nice low-bitrate codec. Another interesting thing is that iTunes at 96 kbps seems to be transparent to most users.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Gambit on 2006-12-09 23:05:10
Thanks to Sebastian and everybody who contributed.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Junon on 2006-12-09 23:58:05
Anyways, congrats to Nero and its devs for a nice low-bitrate codec.


Though I'm even more impressed about how the new Windows Media Audio 10 Professional codec performed. As we all know, VBR can drastically increase the overall sound quality compared to plain CBR. But although the codec was in a clear disadvantage compared to its antagonists, it was still able to rank as 2nd, with its result not being too far from Nero. I wonder how that's possible - does it feature some special techniques for low-bitrate encoding, comparable to Nero's SBR?
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: rjamorim on 2006-12-10 00:02:21
As we all know, VBR can drastically increase the overall sound quality compared to plain CBR.


That's arguable. Indeed, a good VBR model can improve things a lot. But a bad model can ruin everything. Check the iTunes MP3 encoder for an example.

Quote
does it feature some special techniques for low-bitrate encoding, comparable to Nero's SBR?


Yes
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: guruboolez on 2006-12-10 00:02:31
Great test, thanks!
Two interesting points:

• iTunes AAC CBR at 96 kbps as high anchor get the same score than iTunes AAC at VBR 128 kbps when tested as competitor.

• quality varies a lot with samples. It means that at such low bitrate VBR doesn't imply constant quality.
eig, aquatisme, spmg54 and bibilolo were all ranked under 3.0/5 with all competitors (with one exception for bibilolo = 3.16 with vorbis). On the other side other samples like locomotive breath, symphony metal, bebussy, white america, are close to transparency.
In other words, current encoders could sound pretty well but also poorly. Quality is simply unstable - and VBR doesn't really help.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Redmond on 2006-12-10 00:03:29
WMA Standard came out significantly better than iTunes AAC-LC at this bit rate...
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: rjamorim on 2006-12-10 00:54:28
WMA Standard came out significantly better than iTunes AAC-LC at this bit rate...


That's to be expected. iTunes AAC doesn't implement intensity stereo.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Junon on 2006-12-10 01:06:50
WMA Standard came out significantly better than iTunes AAC-LC at this bit rate...


That's also a bit surprising to me, starting from the fact that people keep complaining about WMA Standard's bad quality here on HA, although Apple's AAC implementation performs even worse. Might be possible that it's the other way round on higher quality settings. But nonetheless, that doesn't change the fact that they're both quite impractical in the 48 kbps bitrate region. During the past few years I've been using WMA 9 STD Q50 on two portable devices which supported nothing but MP3 and WMA, and that was the first bearable compromise between filesize and quality.

Thanks to Roberto for the quick reply about WMA 10 Pro. At the first moment I wondered a bit about my lossy codec's of choice result, Vorbis, which is just on par with that CBR competitor made by Microsoft. But since both Nero and Microsoft added low-bitrate techniques to their codecs (SBR in Nero's case, don't know what Microsoft exactly uses in WMA Pro) in contrast to Vorbis, I'd say aoTuV beta 5 did a good job in this test.

Edit: Ah, the answer about WMA STD vs. iTunes AAC can be found above. Should've reloaded this topic before replying.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Firon on 2006-12-10 01:35:45
Boy, I don't remember giving such high scores to a lot of those samples...

I'm impressed with iTunes at 96kbps though. I'm definitely impressed with Nero, it does an excellent job at 48kbps (well, considering the bitrate anyway...).

I also expected Vorbis to do worse than it did.

TomsDiner was awful on all encoders though (my highest rating was a 2), save for iTunes at 96; I don't know how it got such a high average.


Sebastian Mares: are you planning to do any other listening tests? Maybe one at 96kbps, or god forbid, one at 24?  96kbps might be a little difficult since it's already apparently transparent for a lot of people, but it would be nice to test, since I think with Vorbis and AAC, that's the new minimum bar (instead of 128kbps) for transparency.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Mangix on 2006-12-10 02:05:30
I also expected Vorbis to do worse than it did.

opposite for me. at ~96kbps, Vorbis does a better job than HE-AAC IMO. i've only tested it with 2 songs but Vorbis still outperformed HE-AAC
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: IgorC on 2006-12-10 02:15:13
Thank you for result. I found my results in anonumous37xxx.txt

But some *.txt files haven't any ABX results. Only marks. It's seems to be normal. Why?

I waıted somethıng more from WMA 10 pro at 48. More close to Nero than to Vorbıs.

there is a gram. error in name of second sample. It's sample was uploaded by me and I taped it bad. Sample's name should be symphony_metal (my paranoia doesn't admit spaces either). More correctly saying the song is from S&M Metallıca live album  called  " Wherever I May Roam".
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Jillian on 2006-12-10 02:47:03
Could you give me any hints of the next listening test?
Please  (This test doesn't surprise me at all)
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Redmond on 2006-12-10 03:33:01
I wa?ted someth?ng more from WMA 10 pro at 48. More close to Nero than to Vorb?s.


Does Nero use PS (HE-AAC v2) at this quality?
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Mangix on 2006-12-10 03:47:06

I wa?ted someth?ng more from WMA 10 pro at 48. More close to Nero than to Vorb?s.


Does Nero use PS (HE-AAC v2) at this quality?

no
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: singaiya on 2006-12-10 03:55:57
edit: Nevermind, I figured it out. Thank you for running the test, Sebastian.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: singaiya on 2006-12-10 04:57:59
In a quick comparison, I found that my results more or less match those of the cumulative (imagine that). The only variance was choosing 2nd place. On some samples I ranked Vorbis higher where the group total ranked WMA Pro higher, and vice versa on others. Of course, it's so close that the <95% explains that.

One interesting find was on the debussy sample I ranked the low anchor 3.8, and WMA Std 1.0. Also on Kraftwerk I ranked vorbis 1.0 and low anchor 2.7.

No encoder handled biblio very well.

I'll share this discovery I had on sample 18 (the wizard). From my results log: "At this sample I notice that some encoders are doing well at certain parts of the sample and worse at others. If I focus on a part in the beginning, quickly put the sliders to a rough area, then focus on some other part (in order to fine tune the ratings), the ratings from the first pass often don't apply. Then we must decide which failure is more important.

For example, here encoder 3 was better than the others in the first few seconds. But much worse once the vocals came in at the end."

Did anyone else notice this at any point?
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: ff123 on 2006-12-10 05:01:39
Thank you for result. I found my results in anonumous37xxx.txt

But some *.txt files haven't any ABX results. Only marks. It's seems to be normal. Why?


ABX testing is optional in these tests.  Perhaps that should be made more clear in the instructions, if it isn't already.

Thanks for the interesting test, Sebastian.  Looks like lots of people were interested.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: IgorC on 2006-12-10 05:53:16
ABX testing is optional in these tests.  Perhaps that should be made more clear in the instructions, if it isn't already.

Can you g?ve, please, d?rect l?nk where it says that.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: ff123 on 2006-12-10 06:30:00

ABX testing is optional in these tests.  Perhaps that should be made more clear in the instructions, if it isn't already.

Can you g?ve, please, d?rect l?nk where it says that.


It's not directly stated what is required and what is optional.

Sebastian's readme file says to consult this page:

http://ff123.net/64test/practice.html (http://ff123.net/64test/practice.html)

That tutorial should probably be updated to at least show screenshots of abchr-java.  ABX is emphasized so much on HA that it may come as a surprise to some people that not all double-blind tests require this type of repeated result.

ff123
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Sebastian Mares on 2006-12-10 07:41:04
This was stated in the listening test thread: as long as there are no ranked references, a simple score is enough and ABX test can be omitted.

By the way, for people who want to modify plots: http://www.maresweb.de/listening-tests/mf-48-1/pandts.xls (http://www.maresweb.de/listening-tests/mf-48-1/pandts.xls)
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: jido on 2006-12-10 10:29:33
The much awaited results of the Public, Multiformat Listening Test @ 48 kbps are ready.

Here is the results page: http://www.maresweb.de/listening-tests/mf-48-1/results.htm (http://www.maresweb.de/listening-tests/mf-48-1/results.htm)

(http://www.maresweb.de/listening-tests/mf-48-1/resultsz2.png)

Thanks for the test and all participants for the results!

The link (http://www.maresweb.de/mf-48-1-results) printed in the graph is broken.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Ivan Dimkovic on 2006-12-10 11:06:40
Anyways, congrats to Nero and its devs for a nice low-bitrate codec.


Though I'm even more impressed about how the new Windows Media Audio 10 Professional codec performed. As we all know, VBR can drastically increase the overall sound quality compared to plain CBR. But although the codec was in a clear disadvantage compared to its antagonists, it was still able to rank as 2nd, with its result not being too far from Nero.


I wouldn't be so sure about almost mythical powers of VBR at such low bit rate.

For example, VBR does not do any wonder for Nero HE-AAC @48 kbps:

http://www.hydrogenaudio.org/forums/index....showtopic=41191 (http://www.hydrogenaudio.org/forums/index.php?showtopic=41191)

As you can see, for Nero HE-AAC 48 kbps, CBR and VBR were pretty much the same, VBR being marginally better.  And, Nero's VBR implementation is pretty good as it could be seen from this test (German Speech sample - average bit rate 31 kbps - and the codec was still performing best on average - even compared to much higher bit rate contenders)

So, I strongly doubt that usage of VBR in WMA codec, even if it could be forced to give average 48 kbps (which was not possible) would change the score so much, that it would make statistical difference.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Raptus on 2006-12-10 11:19:36
Very interesting results. Not that I have use for such low bitrates, but very impressive performance by some.

Ah, found myself as anonymous12 
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Alex B on 2006-12-10 12:09:05
... German Speech sample - average bit rate 31 kbps - and the codec was still performing best on average - even compared to much higher bit rate contenders ...
Interesting that you mentioned this sample. Surprisingly I found all contenders to be unusable for encoding it even though it is "only" a speech sample. The strong echo effect made the speech unpleasant and very different from the original - like the person speaking was moved to a cave. I actually found the low anchor better than the contenders with this sample.

My results:

Code: [Select]
ABC/HR for Java, Version 0.52b, 08 joulukuu 2006
Testname: Sample16: spmg54_1

Tester: Alex B

1R = Sample16\Sample16_1.wav
2L = Sample16\Sample16_5.wav
3L = Sample16\Sample16_4.wav
4L = Sample16\Sample16_6.wav
5L = Sample16\Sample16_3.wav
6R = Sample16\Sample16_2.wav

Ratings on a scale from 1.0 to 5.0

---------------------------------------
General Comments:
---------------------------------------
1R File: Sample16\Sample16_1.wav
1R Rating: 1.0
1R Comment: strong echo
---------------------------------------
2L File: Sample16\Sample16_5.wav
2L Rating: 4.5
2L Comment:
---------------------------------------
3L File: Sample16\Sample16_4.wav
3L Rating: 1.4
3L Comment: some echo
---------------------------------------
4L File: Sample16\Sample16_6.wav
4L Rating: 1.8
4L Comment: low anchor - most lowpassed, but more pleasant than the echo effect that the other contenders produce
---------------------------------------
5L File: Sample16\Sample16_3.wav
5L Rating: 1.3
5L Comment: distorded, some echo
---------------------------------------
6R File: Sample16\Sample16_2.wav
6R Rating: 1.3
6R Comment: echo
---------------------------------------

ABX Results:
Original vs Sample16\Sample16_5.wav
    8 out of 8, pval = 0.0030


---- Detailed ABX results ----
Original vs Sample16\Sample16_5.wav
Playback Range: 06.662 to 09.154
    2:15:23 AM p 1/1 pval = 0.5
    2:15:47 AM p 2/2 pval = 0.25
    2:15:57 AM p 3/3 pval = 0.125
    2:16:02 AM p 4/4 pval = 0.062
    2:16:06 AM p 5/5 pval = 0.031
    2:16:10 AM p 6/6 pval = 0.015
    2:16:16 AM p 7/7 pval = 0.0070
    2:16:30 AM p 8/8 pval = 0.0030
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Alex B on 2006-12-10 12:33:05
Great test, thanks!
Two interesting points:

• iTunes AAC CBR at 96 kbps as high anchor get the same score than iTunes AAC at VBR 128 kbps when tested as competitor.

The obvious reason for this is the low overall quality. With most samples I found iTunes 96 kbps to be in the range of 3-4 instead of 4.5 or over. I had no difficulties in distinguishing it. I ABXed the high anchor a few times, but that only confirmed what I already knew after the initial listening. In my opinion the codecs in the 128 kbps test were clearly better and almost transparent with many samples.

Quote
• quality varies a lot with samples. It means that at such low bitrate VBR doesn't imply constant quality.
eig, aquatisme, spmg54 and bibilolo were all ranked under 3.0/5 with all competitors (with one exception for bibilolo = 3.16 with vorbis). On the other side other samples like locomotive breath, symphony metal, bebussy, white america, are close to transparency.
In other words, current encoders could sound pretty well but also poorly. Quality is simply unstable - and VBR doesn't really help.

I agree, the quality is quite unstable and in my opinion generally too low for pleasant listening experience even with a typical portable device. I quess that e.g. 64 kbps would already be much better for these encoders.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: guruboolez on 2006-12-10 13:36:40
@Alex B> You're not the only one to dislike the coding effects of all competitors with the German spoken voice. The average mark for this sample is very low (maybe the lowest one of all 20 samples). That's really interesting: voice is often considered as "non-complex", easy to encode stuff. As a consequence ultra-low bitrate is often used for DVD ripping and de facto considered as "good enough" for this kind of material. This sample is probably not representative of DVD soundtrack; noneless it should warn people about the possible damage such low bitrate can cause on video DVD. From my experience I must say that I often heard such very disturbing artefacts on soundtrack encoded at this bitrate.

> about iTunes AAC@96: I was tempted myself to lower the mark in order to make it match with the ABC/HR scale (i.e. below "perceptible but not annoying"). But I didn't, in order to gain some headroom to rank the other competitors. Therefore I didn't bother with the high anchor and I give it a high score each time. I wouldn't give the same score to this encoder in a different situation (like a 96 kbps listening test or, worse, a 180 kbps with iTunes@96 as low anchor). It's a limit to cross-comparison of different listening test.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Firon on 2006-12-10 14:17:30
I also ranked that one very low (highest was 1.5), it was pretty unbearable. Even with ABX I had such a hard time picking out iTunes at 96 that I just stopped trying so hard on it. I generally ranked it 4-4.5 (when I bothered). Guess my ears aren't as golden as guruboolez's.

How did you rank TomsDiner though, guru?

opposite for me. at ~96kbps, Vorbis does a better job than HE-AAC IMO. i've only tested it with 2 songs but Vorbis still outperformed HE-AAC


You do realize that Vorbis was tested at 48kbps, don't you? HE-AAC has been shown to not really be worth it above about 80kbps.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: guruboolez on 2006-12-10 14:47:08
How did you rank TomsDiner though, guru?


Code: [Select]
ABC/HR for Java, Version 0.52b, 07 décembre 2006
Testname: Sample05: TomsDiner

Tester:

1L = Sample05\Sample05_5.wav
2R = Sample05\Sample05_6.wav
3R = Sample05\Sample05_4.wav
4L = Sample05\Sample05_1.wav
5L = Sample05\Sample05_2.wav
6L = Sample05\Sample05_3.wav

Ratings on a scale from 1.0 to 5.0

---------------------------------------
General Comments:
---------------------------------------
2R File: Sample05\Sample05_6.wav
2R Rating: 2.5
2R Comment: very smooth sound; less agressive to my ears than the reference. Nonetheless not acceptable as coding tool...
---------------------------------------
3R File: Sample05\Sample05_4.wav
3R Rating: 3.5
3R Comment:
---------------------------------------
4L File: Sample05\Sample05_1.wav
4L Rating: 3.5
4L Comment:
---------------------------------------
5L File: Sample05\Sample05_2.wav
5L Rating: 3.0
5L Comment:
---------------------------------------
6L File: Sample05\Sample05_3.wav
6L Rating: 2.5
6L Comment:
---------------------------------------

ABX Results:


Do we know which number correspond to each encoder?
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Gabriel on 2006-12-10 14:50:26
Wma std has a very good performance, compared to iTunes, especially considering that theorically, there is nothing in the wma std format that would provide better efficiency than AAC-LC.

Stanley, if you are reading this: no SBR and no intensity stereo? Obviously you're going to be ranked quite lower than competitors, and I'm wondering if there is anything planned regarding this.
(yes, Lame is in the exact same situation, with no code to handle low bitrates...)
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Junon on 2006-12-10 14:59:36
I agree, the quality is quite unstable and in my opinion generally too low for pleasant listening experience even with a typical portable device. I quess that e.g. 64 kbps would already be much better for these encoders.


I agree to that guess, especially in the case of aoTuV beta 5. After the codec's release I ABXed a few samples to find a bitrate that suits my portable player. Although my hearing's untrained (haven't done much ABXing so far) it wasn't too hard to distinguish the 48 kbps samples from the original ones. But I already was in serious trouble hearing differences between most 64 kbps samples and the .wav source files, which led me to the decision to choose this bitrate for the portable player. Too bad I don't have the logs anymore, they would have been good examples to see the improvements in quality from the -q-1 to -q0 step.


Besides, thanks @everyone for the comments to my statement about CBR vs. VBR in my first reply. I was under the apparently false impression that VBR's always an indicator for better quality, therefore I didn't even bother carrying out any tests to arrive at an objective conclusion.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Sebastian Mares on 2006-12-10 15:12:33
The link (http://www.maresweb.de/mf-48-1-results) printed in the graph is broken.


No it's not.  Apache is set to redirect to the correct document and it works fine for me.

Do we know which number correspond to each encoder?


Damn, deleted my list already and I forgot to post. IIRC, it should be:

1, Vorbis
2, Nero
3, WMA Std.
4, WMA Pro.
5, High Anchor
6, Low Anchor
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: robert on 2006-12-10 15:48:41
Do we know which number correspond to each encoder?

? Was it supposed to be not obvious??
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: guruboolez on 2006-12-10 15:59:47
 
How could it be obvious? Testers only discovered their own results since the publication of the encryption key (i.e. few hours ago).
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Alex B on 2006-12-10 16:13:31
For me, the contents of a sample zip file made the order obvious:

Code: [Select]
#  Archive U:\48test\Sample01.zip
2006-11-19 23:54        Folder        Folder  Sample01
2006-11-19 23:49       4880852       4881597  Sample01\Sample01.wv
2006-11-19 23:17        258820        256617  Sample01\Sample01_1.ogg
2006-11-19 23:19        275938        269579  Sample01\Sample01_2.mp4
2006-11-19 23:48       3869678       3870273  Sample01\Sample01_3.wv
2006-11-19 23:48       4865678       4866423  Sample01\Sample01_4.wv
2006-11-19 23:33        570634        520673  Sample01\Sample01_5.m4a
2006-11-19 23:39        310726        260186  Sample01\Sample01_6.m4a
#
# Total                   Size        Packed  Files
#                     15032326      14925348  8

The numbers 5 (high anchor) and 6 (low anchor) are obvious and also 3 and 4 (WMA Standard & Pro)

Also, the test results are presented in the same order (1-6).
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: jido on 2006-12-10 16:14:51

The link (http://www.maresweb.de/mf-48-1-results) printed in the graph is broken.


No it's not.  Apache is set to redirect to the correct document and it works fine for me.

Thanks, working now 
The testers should not know which number is which encoder, otherwise they are at risk of bias.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: haregoo on 2006-12-10 16:28:10
The testers should not know which number is which encoder, otherwise they are at risk of bias.

No risk for that because of well-conceived ABC/HR methodology.

Anyway it's good that over 20 results per sample were submitted only in 2 weeks. We can expect that next test goes well.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: guruboolez on 2006-12-10 16:50:47
For me, the contents of a sample zip file made the order obvious:

Indeed, it can't be more obvious... I'm a nut 
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Alex B on 2006-12-10 16:53:29

The testers should not know which number is which encoder, otherwise they are at risk of bias.

No risk for that because of well-conceived ABC/HR methodology.

As haregoo said, there's no risk. In the actual test the order is scrambled and cannot be determined by examining the sample files.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: IgorC on 2006-12-10 16:54:49
It's not directly stated what is required and what is optional.

That tutorial should probably be updated to at least show screenshots of abchr-java.  ABX is emphasized so much on HA that it may come as a surprise to some people that not all double-blind tests require this type of repeated result.


It would be nice if next time all conditions will be described _clearly_ in readme.txt because it's not possible for all people to follow all tips from that were disucced in topics.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Sebastian Mares on 2006-12-10 17:06:12


The link (http://www.maresweb.de/mf-48-1-results) printed in the graph is broken.


No it's not.  Apache is set to redirect to the correct document and it works fine for me.

Thanks, working now 


I did not change anything so I am wondering how it didn't work before while now it is.

It would be nice if next time all conditions will be described _clearly_ in readme.txt because it's not possible for all people to follow all tips from that were disucced in topics.


At no place I did write that ABX tests are mandatory in order to participate.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: IgorC on 2006-12-10 17:10:14
At no place I did write that ABX tests are mandatory in order to participate.

the same valid way to say:
At no place you did write that ABX test can be omited.....
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Sebastian Mares on 2006-12-10 17:14:43
Nobody asked (actually, somebody did and I explained in what situations an ABX test is needed). Anyways, I will write that more clearly next time.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: IgorC on 2006-12-10 17:30:49
Anyways, I will write that more clearly next time.

Thank you.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Ivan Dimkovic on 2006-12-10 18:12:39
@Alex B> You're not the only one to dislike the coding effects of all competitors with the German spoken voice. The average mark for this sample is very low (maybe the lowest one of all 20 samples). That's really interesting: voice is often considered as "non-complex", easy to encode stuff. As a consequence ultra-low bitrate is often used for DVD ripping and de facto considered as "good enough" for this kind of material. This sample is probably not representative of DVD soundtrack; noneless it should warn people about the possible damage such low bitrate can cause on video DVD. From my experience I must say that I often heard such very disturbing artefacts on soundtrack encoded at this bitrate.


Agreed.

German Speech sample is a typical example of a sound sample which generates "double-speak" type of artifacts, which are actually one type of general pre-echo artifacts.

TNS was actually invented with this problem mainly in mind - so, by no means, speech samples like this are easy to be encoded.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: muaddib on 2006-12-11 10:35:21
> about iTunes AAC@96: I was tempted myself to lower the mark in order to make it match with the ABC/HR scale (i.e. below "perceptible but not annoying"). But I didn't, in order to gain some headroom to rank the other competitors. Therefore I didn't bother with the high anchor and I give it a high score each time. I wouldn't give the same score to this encoder in a different situation (like a 96 kbps listening test or, worse, a 180 kbps with iTunes@96 as low anchor). It's a limit to cross-comparison of different listening test.


I am wondering if most of the participants did it like this and didn't bother about giving correct score to high anchor and just gave it 5?
And was it also the case that most people didn't bother and just gave 1 to low anchor?
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: pepoluan on 2006-12-11 13:40:26
Reflected here:

http://wiki.hydrogenaudio.org/index.php?title=Current_events (http://wiki.hydrogenaudio.org/index.php?title=Current_events)
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: t.g.deck on 2006-12-11 14:40:40
I am wondering if most of the participants did it like this and didn't bother about giving correct score to high anchor and just gave it 5?
And was it also the case that most people didn't bother and just gave 1 to low anchor?

Not me, I certainly didn't - my poor ears had a hard time distinguishing even the HE AAC contender from the reference in most cases.    The low anchor sounded much crappier than the contenders IMHO, except perhaps for 1 sample.

Thanks for this great test, Sebastian!
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Woodinville on 2006-12-11 18:46:06
Wma std has a very good performance, compared to iTunes, especially considering that theorically, there is nothing in the wma std format that would provide better efficiency than AAC-LC.



Out of curiousity, how do you know that, and could you direct me to the documentation?
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Garf on 2006-12-11 21:06:33

Wma std has a very good performance, compared to iTunes, especially considering that theorically, there is nothing in the wma std format that would provide better efficiency than AAC-LC.



Out of curiousity, how do you know that, and could you direct me to the documentation?


There's a reverse-engineerd decoder in the ffdshow project.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Gabriel on 2006-12-12 08:47:16

Wma std has a very good performance, compared to iTunes, especially considering that theorically, there is nothing in the wma std format that would provide better efficiency than AAC-LC.

Out of curiousity, how do you know that, and could you direct me to the documentation?

Don't worry, there is no leaked wmaV2 documentation (to my knowledge). However, we have a decoder (as mentionned by Garf) source code available in ffmpeg, which is enough to know about the format features.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: rjamorim on 2006-12-12 10:47:11
There's a reverse-engineerd decoder in the ffdshow project.


ffmpeg, actually.
http://svn.mplayerhq.hu/ffmpeg/trunk/libav...pe=text%2Fplain (http://svn.mplayerhq.hu/ffmpeg/trunk/libavcodec/wmadec.c?revision=6577&content-type=text%2Fplain)

ffdshow is the directshow filter mostly based on ffmpeg
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Aoyumi on 2006-12-12 13:13:10
I agree to that guess, especially in the case of aoTuV beta 5. After the codec's release I ABXed a few samples to find a bitrate that suits my portable player. Although my hearing's untrained (haven't done much ABXing so far) it wasn't too hard to distinguish the 48 kbps samples from the original ones. But I already was in serious trouble hearing differences between most 64 kbps samples and the .wav source files, which led me to the decision to choose this bitrate for the portable player. Too bad I don't have the logs anymore, they would have been good examples to see the improvements in quality from the -q-1 to -q0 step.

If "Quality0" is compared with "Quality-1", there is a very big difference in impulse sound.
It influences the quantity of pre/post echo.

The greatest cause of the difference is in block(window) size.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Firon on 2006-12-12 16:25:37
Aoyumi, do you have plans to continue tuning the low bitrate range? I would especially like to see improvements to q0 and q-1, so it can be even more competitive with HE-AAC. Or is it not possible for Vorbis to really compete with SBR at this bitrate range?
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: PatchWorKs on 2006-12-13 08:34:21
My conclusion are that Voirbis has good performances (that borders M$-WMA), it's open source and royalty-free: finally the codec to choice !
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: fpi on 2006-12-13 09:23:45
Vorbis has another advantage: while other codecs (see HE-AAC and LC-AAC) are good only in a certain bitrate range, Vorbis work very well from low to high bitrate. And has also a good support on portable player.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Ivan Dimkovic on 2006-12-13 09:35:26
SBR is merely a tool in AAC, so I wouldn't call HE-AAC a different codec than LC-AAC for example.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Aoyumi on 2006-12-13 14:38:05
do you have plans to continue tuning the low bitrate range?
For the moment, I am continuing it.
Quote
I would especially like to see improvements to q0 and q-1, so it can be even more competitive with HE-AAC. Or is it not possible for Vorbis to really compete with SBR at this bitrate range?
There is no clear answer.
I like Vorbis most generally in 64kbps range at present. Therefore, for me, Vorbis is competitive enough.  I don't know whether Vorbis will become competitive for you.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Junon on 2006-12-13 19:10:26
Aoyumi, do  you have plans to continue tuning the low bitrate range? I would  especially like to see improvements to q0 and q-1, so it can be even  more competitive with HE-AAC. Or is it not possible for Vorbis to  really compete with SBR at this bitrate range?

 
  The question is in what way you want it to be more competitive with  HE-AAC. If it's only quality-wise, then HE-AAC will most certainly keep  being supreme in the, let's say, < 80 kbps bitrate range. That's  mainly due to SBR's ability to reproduce high frequencies which have  been cut off during the encoding process. Since Vorbis doesn't feature  any comparable technique it's the clear loser in this case. Hence I  deem it quite impossible that Vorbis will ever be able to compete with  HE-AAC in low-bitrate listening tests like this one.
 
  But nonetheless, in my opinion Vorbis is very competitive to HE-AAC  concerning both codecs' use in practice. Low bitrates like 48/64/80  kbps are common to be used on hardware players equipped with memory  sticks. And these usually rely on rechargable batteries which heavily  suffer from SBR's power appetite. The question is whether the few extra  songs, that can be squeezed on the device using extra-low bitrates in  conjunction with SBR, are worth the additional draining of the  expensive battery's life. For me it's a clear "no", I prefer encoding  to LC-AAC/Vorbis at slightly higher bitrates instead. Therefore I  wouldn't make use of an SBR-like implementation for Vorbis at all.
 
  But of course, HE-AAC's a real help concerning streaming in conjunction  with low-bandwidth as well as volume-restricted internet connections.  For this kind of use there's no single codec that can compete with it.
 
  Besides, are there any reliable data sheets about SBR's actual power  hunger? Using Google I only stumble across some mostly outdated press  releases and Gabriel's almost  ancient article about MP3Pro (http://www.mp3-tech.org/sbr.html) which aren't really suitable to be used as sources for my claims  about SBR's power consumption.

PS: At least there's a SBR-LP (Low Power) mode available, as mentioned in this article. (http://www.chiariglione.org/mpeg/technologies/mp04-sbr/index.htm) Too bad there aren't any actual figures to be found.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Garf on 2006-12-13 20:14:51
The question is whether the few extra  songs, that can be squeezed on the device using extra-low bitrates in  conjunction with SBR, are worth the additional draining of the  expensive battery's life. For me it's a clear "no",


only if you ignore the fact that reading more data eats batteries, too.

Quote
Besides, are there any reliable data sheets about SBR's actual power  hunger? Using Google I only stumble across some mostly outdated press  releases and Gabriel's almost  ancient article about MP3Pro (http://www.mp3-tech.org/sbr.html) which aren't really suitable to be used as sources for my claims  about SBR's power consumption.

PS: At least there's a SBR-LP (Low Power) mode available, as mentioned in this article. (http://www.chiariglione.org/mpeg/technologies/mp04-sbr/index.htm) Too bad there aren't any actual figures to be found.


https://datatype.helixcommunity.org/2005/aacfixptdec (https://datatype.helixcommunity.org/2005/aacfixptdec)

These are older figures, claimed by Real. They claim 52Mhz ARM9E to decode stereo HE-AAC (using full, not using low power, decoding). I know that 128kbps Vorbis takes 40Mhz on the same chip (using Tremor).

Note that most devices which actually have HE-AAC support have hardware, or at least DSP assitance for the decoding.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Junon on 2006-12-13 21:09:30
https://datatype.helixcommunity.org/2005/aacfixptdec (https://datatype.helixcommunity.org/2005/aacfixptdec)

These are older figures, claimed by Real. They claim 52Mhz ARM9E to decode stereo HE-AAC (using full, not using low power, decoding). I know that 128kbps Vorbis takes 40Mhz on the same chip (using Tremor).

Note that most devices which actually have HE-AAC support have hardware, or at least DSP assitance for the decoding.


A lot more hungry than LC-AAC, but also way more bearable in comparison to Vorbis than I imagined it to be. Thanks for the input, it changed my opinion about HE-AAC quite a bit.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: fpi on 2006-12-15 16:40:27
Aoyumi, do you have plans to continue tuning the low bitrate range? I would especially like to see improvements to q0 and q-1, so it can be even more competitive with HE-AAC. Or is it not possible for Vorbis to really compete with SBR at this bitrate range?


A way to improve Vorbis at all bitrate would be to use rehuff. Rehuff is a tool to losslessly recompress data of a Vorbis file. Unfortunately the last version was buggy: was able to lossless reduce file size, but generated wrong packet giving problem with seek.

More info here:
http://lists.xiph.org/pipermail/vorbis-dev...ust/018522.html (http://lists.xiph.org/pipermail/vorbis-dev/2006-August/018522.html)
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: gameplaya15143 on 2006-12-16 16:26:27
The test results don't suprise me.  But with everything I've read about wma10, I would have half-expected it to be closer to HE-AAC.

I would especially like to see improvements to q0 and q-1, so it can be even more competitive with HE-AAC. Or is it not possible for Vorbis to really compete with SBR at this bitrate range?
IMO q0 has been at least on par with HE-AAC at 64kbps for quite a few years.. just have to move up the lowpass a little bit  .. not so much with any negative quality value though.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Firon on 2006-12-16 17:09:19
q0 is not quite on par with HE-AAC. It's clearly worse on most songs, even with a raised lowpass. It's not that bad, but it's certainly not as good either.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: SebastianG on 2006-12-18 22:18:47
Edit:
@mods: This entry can be deleted.

I commented on
Wma std has a very good performance, compared to iTunes, especially considering that theorically, there is nothing in the wma std format that would provide better efficiency than AAC-LC.

saying that WMA Std uses VQ (http://www.data-compression.com/vq.shtml) but verified later that it does not (wmadec.c, wmadata.h). I guess I mistook the tables for LSP values for VQ tables.
Title: Multiformat Listening Test @ 48 kbps - FINISHED
Post by: Hollunder on 2006-12-18 23:35:55
Thanks for the test!
My ego got beaten another time as it was hard for my untrained ears to distinguish most codecs at half of the samples.
I wasn't even able to distinguish the low anchor at one sample.
Some other, pre-echo prone samples were quite obvious while I could barely hear other artifacts.

Guess I should get a fresh pair of ears for the next test
SimplePortal 1.0.0 RC1 © 2008-2020