Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multi-Codec Listening Test: 96-128-192-256Kbps (Read 61729 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #75 –
Sorry but in all honesty I have no interest in tuning CELT specially on mono, it was just a one shot test done by curiosity to see what it has in the stomach. Now I know that CELT requires much tuning which will take a long time. I am a lossless user so I will not spend my time helping the development of all lossy codecs in the world, I already tried to help lossywav & to a lesser extend aotuv & I noticed how much time greedy this is compared to the little gain. I have spend hours for almost no gain on lossywav (only discarding useless parameters) & a very little gain on aotuv (only fixing the Rush sample). I have high hope for CELT but I am short of time. Also I feel that it is a natural processus that after some ABXing you finally get bored & lost your interest. I will re-test CELT for sure, but after several releases, in something like a year & for my selfish personnel need, not for tuning. The ability to split CELT is a priority for me because I use my music in a particular way, I want only CDImage+cue, so I want to use splitable lossy CDImage+cue, that's why my personnal interest lies in lossywav & vorbis actually. But I am not specially a free software defender, if nero AAC would be easily splitable with a cue, I would most likely switch to aac/mp4 as I already use avc for video. Good luck with CELT, it's nice too see that Monty is not travelling alone anymore, even if it seems faster. It's gonna be a looong way to the top

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #76 –
Quote from: sauvage78 on
Note: The artefact noted as medium for Celt are often severe medium, I could have noted them as M+ instead of just M, but I marked them as medium because if I wouldn't have done so, almost everything would be in the red zone. Actually Celt 256Kbps is clearly worst than Vorbis 128Kbps, I am not even sure that it can compete with Vorbis 96Kbps.


Thanks a lot for spending time doing this test with CELT. From what I understand, CELT does very well on "normal samples" and very bad on "problem samples". There's probably a few explanations for that. The first (and obvious one) is that the other codecs are VBR (and can bump the rate during transients), while CELT is CBR for now (VBR support planned, but not that useful anyway). Now, that can't explain why it does so badly on the problem samples, even at very high bit-rates. Of course, there's a tuning issue, but there's also some features like folding and short blocks that need to be tuned. Another thing I hope will eventually help at high rate is to actually include a real psy model. As surprising as it may sounds, CELT currently does the bit allocation without even considering the signal or computing any masking curve. I'm hoping to fix all that (and possibly other problems I don't know about yet) for 1.0.

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #77 –
I don't know how CELT perform on "normal samples" I didn't even try to encode a full song to see how it sounds on average input, but I wouldn't expect it to perform "very well", I would rather say that it should sound "acceptable" at high bitrate considering that it is experimental code. I wouldn't use it on anything for backup actually even on non-problems samples. I consider it a toy for enthousiast actually, nothing more. I know quite well the quality of Vorbis at various development stage so if I would compare to old vorbis I would say that it's likely that even old vorbis RC beat the ass of CELT

It's not that CELT is bad, I am confident that the improvement margin is huge. It's just that as a codec, actually, it's a baby.

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #78 –
Quote from: sauvage78 on
I don't know how CELT perform on "normal samples" I didn't even try to encode a full song to see how it sounds on average input, but I wouldn't expect it to perform "very well", I would rather say that it should sound "acceptable" at high bitrate considering that it is experimental code. I wouldn't use it on anything for backup actually even on non-problems samples. I consider it a toy for enthousiast actually, nothing more. I know quite well the quality of Vorbis at various development stage so if I would compare to old vorbis I would say that it's likely that even old vorbis RC beat the ass of CELT

It's not that CELT is bad, I am confident that the improvement margin is huge. It's just that as a codec, actually, it's a baby.


I'm getting the impression that you missed something about what CELT is trying (and especially *not* trying) to be. CELT is all about real-time communication with really, really low delay. It's designed so you can play music remotely through a DSL connection or to have perfect quality videoconference again using little bandwidth. This is something that neither AAC nor Vorbis can do because they have >100 ms delay. Even AAC-LD and G.722.1x have too much delay for network music performances. So if you're waiting for CELT to beat Vorbis or AAC, that will likely never happen. However, if you compare it to low delay codecs (even things like AAC-LD and G.722.1C that have a lot more delay), then it does already does quite well (see the paper that I mentioned in the other thread). I'm already surprised that CELT beats MP3 (LAME) in CBR mode in the tests we did. That is far better than I hoped to achieved when starting CELT.

That being said, I'm sure CELT can do a lot better, especially on the problem samples you tried. Now that I know what to test on, I'm hoping to be able to do better. So thanks very much again for doing these tests.

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #79 –
I know very well that CELT is supposed to beat all the other codecs on latency, but I have no interest at all in things like low latency, mono or voice. I have read the Xiph website on CELT, if you don't intend to compete with non-streaming oriented codecs you'd better not say that CELT is supposed to be able to handle music. Because you advertise on your website that CELT is supposed to be something between vorbis & speex & when you read that too quickly you can understand that CELT will swallow both speex & vorbis, obviously in your mind it is closer to speex than to vorbis. I have nothing against codec feeding the needs of webmasters rather than the use of audiophiles. I am used to Xiph, I know you guys from a long time, I used to stay in your IRC channels back in the days when I had some irrationnal faith in Monty & vorbis & I know that your priorities are not mine. That's why I quit using vorbis & I will most likely never use CELT if it doesn't target transparency: I have learned from my past misstakes. But before I knew what to think of CELT I had to give it a try. I am sorry if CELT doesn't sound as good as you expected, I cannot honnestly recommend it to anyone in its actual state & anyway as I said several time in this topic I invite anyone to test for himself. Some people are very happy with vorbis -q2, what can I say ? I will not fight the whole world, if people are happy with vorbis -q2 I am happy for them, the same goes for CELT. If CELT improves be sure I'll be fair with it. I am already happy that my short test helped you realized that CELT wasn't perfect, it's a good start.

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #80 –
Quote from: sauvage78 on
Thks too Alex B, I have quickly tried these two (plus Track02cut from the same topic) on aotuv -q4 & lossywav -q 1.5:

I can ABX amnesia on aotuv -q4, I failed to catch anything on all the other samples/parameters. Will retry once later today then I'll give up...

Yeah, my sample appears to be easier than I thought for Vorbis and also for Nero. I was barely able to ABX Vorbis at -q4 (I heard a small tonal difference in the highest frequencies in the quieter part of the sample. I tried also Nero -q 0.45 and failed to quickly ABX it.

However, just for kicks you could try LAME -V2. The problem I noticed with LAME3.97 is now somewhat less pronounced, but still there with LAME 3.98. It produces clear artifacts. An ABX test is not needed to hear them. -V0 is still easily ABXable. Also Musepack --quality 5  produces a funny and obvious artifact. Musepack --quality 6 is better and I couldn't easily ABX it. (My easily is a quick test that produces 8/8 without doubt. I didn't feel like trying to seriously test a possibly transparent sample. I need to be in a very good shape to try that.)

EDIT

The Wab5s sample is available here: http://www.hydrogenaudio.org/forums/index....showtopic=54752

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #81 –
Quote from: sauvage78 on
I know very well that CELT is supposed to beat all the other codecs on latency, but I have no interest at all in things like low latency, mono or voice. I have read the Xiph website on CELT, if you don't intend to compete with non-streaming oriented codecs you'd better not say that CELT is supposed to be able to handle music. Because you advertise on your website that CELT is supposed to be something between vorbis & speex & when you read that too quickly you can understand that CELT will swallow both speex & vorbis, obviously in your mind it is closer to speex than to vorbis. I have nothing against codec feeding the needs of webmasters rather than the use of audiophiles. I am used to Xiph, I know you guys from a long time, I used to stay in your IRC channels back in the days when I had some irrationnal faith in Monty & vorbis & I know that your priorities are not mine. That's why I quit using vorbis & I will most likely never use CELT if it doesn't target transparency: I have learned from my past misstakes. But before I knew what to think of CELT I had to give it a try. I am sorry if CELT doesn't sound as good as you expected, I cannot honnestly recommend it to anyone in its actual state & anyway as I said several time in this topic I invite anyone to test for himself. Some people are very happy with vorbis -q2, what can I say ? I will not fight the whole world, if people are happy with vorbis -q2 I am happy for them, the same goes for CELT. If CELT improves be sure I'll be fair with it. I am already happy that my short test helped you realized that CELT wasn't perfect, it's a good start.


Still missing the point. CELT *does* handle music. The fact that it doesn't handle it as well as Vorbis is not the point. If you want to hear a codec that really doesn't handle music, try Speex. Also, the simple fact that you do not need real-time behaviour means you don't need CELT. It's of no use for you, even if it improves a lot. OTOH, for the people who do need very low delay, CELT is really what they need. The only codec besides CELT that can give you a delay below 10 ms is FhG's ULD codec and in the samples we got, CELT has better quality. I think you misread my previous post. CELT has already *exceeded* my original expectations. The low-delay constraint that CELT has really comes with a heavy price, but I always though that price was even higher than it is. To give you an idea of what we're talking about. Imagine taking AAC or Vorbis, forcing it to CBR, forcing it to always use short blocks, preventing it from using any look-ahead for the psychoacoustics (has to be based on the current frame only), and preventing it from using a bit reservoir. That's what CELT has to do to achieve low delay. Of course, it could have a lot higher quality if it didn't have those constraints, but that's not what I'm trying to do.

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #82 –
Well I had some hope that CELT would come in different flavors targetting different uses. I'll put this hopes in the delete bin. It's not a big problem for me  Sorry about the missundertanding about CELT handling music, but you marketed it like this ... just sell it as high quality VoIP if you don't want such missundertanding, because reading the website I thought it was both high quality VoIP & maybe music via a VBR mode (I mean CD quality). You seem to say that VBR will not help much, so I wouldn't call it a codec suited for music, even if it's more than just voice. Obviously handling music doesn't mean the same thing for an audio developper & for an end user.

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #83 –
Quote from: sauvage78 on
Well I had some hope that CELT would come in different flavors targetting different uses. I'll put this hopes in the delete bin. It's not a big problem for me  Sorry about the missundertanding about CELT handling music, but you marketed it like this ... just sell it as high quality VoIP if you don't want such missundertanding, because reading the website I thought it was both high quality VoIP & maybe music via a VBR mode (I mean CD quality). You seem to say that VBR will not help much, so I wouldn't call it a codec suited for music, even if it's more than just voice. Obviously handling music doesn't mean the same thing for an audio developper & for an end user.


You seen to have a weird definition of "handling". According to your definition, Speex doesn't handle speech (it's not 100% transparent) and neither does any other speech codec, nor any phone -- much less cell phone. You have the right to care only about lossless audio, but what most people actually want is decent quality with decent bit-rate. Of course, "decent" is different depending on whether we're talking about music players, web streaming, cell phones, ...

Multi-Codec Listening Test: 96-128-192-256Kbps

Reply #84 –
Quote from: sauvage78 on
I know very well that CELT is supposed to beat all the other codecs on latency, but I have no interest at all in things like low latency, mono or voice. I have read the Xiph website on CELT, if you don't intend to compete with non-streaming oriented codecs you'd better not say that CELT is supposed to be able to handle music.


Only lossless is unconditionally guaranteed to be indistinguishable to all listeners on all samples in all circumstances. Of course, the price you pay for that is higher bitrates.

Of course— we would like to have CELT as transparent as possible at sufficiently high bitrates, and I'll reiterate my thanks for testing and pointing out test cases where it is not… but if you define failing ABX with carefully selected torture samples at high rate to be 'not handling' music then you really have unreasonable expectations for lossy codecs. You might think that lossy codec X is unconditionally transparent at Y bits per second, but it's highly likely that with a different sample, different listening conditions, or a better listener you'd be able to ABX them. 

Quote
Because you advertise on your website that CELT is supposed to be something between vorbis & speex & when you read that too quickly you can understand that CELT will swallow both speex & vorbis,


The webpage states "The CELT codec is meant to bridge the gap between Vorbis and Speex for applications where both high quality audio and low delay are desired".  Speex can't produce music quality high enough to fool even the most tin-eared layman, but it offers fairly low latency. Vorbis offers decent quality but requires high latency. CELT is intended to bridge the gap or, in other words, to fill in the space *between* Vorbis and Speex.  This is almost the opposite of how you read the text on the webpage.  If you can suggest language which is less likely to confuse people, it could be included.

Cheers.