Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multiformat 48 kbps Listening Test (Read 46510 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multiformat 48 kbps Listening Test

So, as most of you know, I am planning to conduct a multiformat listening test at 48 kbps in early to mid June. Since 48 kbps is not really the optimal bitrate for low-speed Internet streams, I guess we can use VBR over CBR since that is supposed to give a better quality. The codec decision should be made considering that this test should include popular formats and encoders that were tuned or at least perform fair at 48 kbps (so definitely not plain MP3 or Musepack). However, because I would really like to test the new WMA encoder, I cannot really set "portability" as criterium since I don't know any portable players that support WMA 10.

So, first thing we should clear up is which encoders to test? I was thinking about testing not more than 6 encoders. 2 of them will be anchors, which leaves room for 4 contenders. Since mp3PRO was "replaced" by HE-AAC, I think we should leave it out. My encoder selection so far is:

High anchor (maybe LAME MP3 or LC-AAC at a higher bitrate like 128 kbps+)
Nero HE-AAC
AoTuV Vorbis (current version, or latest beta?)
WMA10+
Suggestions please
Low anchor (either lowpass or MP3?)

Multiformat 48 kbps Listening Test

Reply #1 –
Yay, a new test.

This is really a performance test for me, so I want to see latest WMA, Nero and Vorbis. Lame at 48kbps for low anchor. For the high anchor I'd like to see V5 Lame.

Looking forward to it. This should be much easier to test, so hopefully lots of people will participate.

Good luck Sebastian.

Multiformat 48 kbps Listening Test

Reply #2 –
Quote from: Sebastian Mares on
High anchor (maybe LAME MP3 or LC-AAC at a higher bitrate like 128 kbps+)


I'd prefer to see a high anchor around 96kbps. That would be test bitrate * 2, and 128 would be "almost transparent" for most people anyway.

But if you go with MP3, I'd indeed prefer 128, as sooner or later Microsoft and Nero will start claiming their codecs at 48kbps are better than MP3 at 128 (if they didn't start claiming that already)

Then again, if you go with MUSHRA instead of ABC/HR, you should use a lowpass, so to stay within the spirit of the specification.

Quote
AoTuV Vorbis (current version, or latest beta?)


Ask aoyumi.

Quote
WMA10+


What if Microsoft doesn't explicitly authorize you to conduce the test?

Quote
Low anchor (either lowpass or MP3?)


Using MP3 is just being mean. Lowpass is highly uninteresting (again, unless you go with MUSHRA)

I'd prefer to see some oddball format. But then again, that's me...


I'd also vote for including CT AAC plus, as implemented by Winamp.

Multiformat 48 kbps Listening Test

Reply #3 –
Quote
Low anchor (either lowpass or MP3?)


Why you dont use one of top contenders nero or aotuv at even lower bitrates?
(I m mostly questioning, not proposing)

Multiformat 48 kbps Listening Test

Reply #4 –
Quote from: rjamorim on
What if Microsoft doesn't explicitly authorize you to conduce the test?

Testing is allowed, only publishing results isn't. If Microsoft doesn't give me the permission to publish WMA results, blame Microsoft.

Quote from: jazzymelody on
Why you dont use one of top contenders nero or aotuv at even lower bitrates?
(I m mostly questioning, not proposing)

That's also an option.

Multiformat 48 kbps Listening Test

Reply #5 –
Not publishing results would mean editing out the raw data for wma (assuming you would want to publish the rest of the raw data).  Better hone your scripting skills.

I almost have to laugh because the no-benchmark clause sounds so ludicrous on the face of it, even though I understand its intent.

ff123

Edit:  Also, you will personally know the results, and if you don't want me to put a hex on you, you'll share the results with me :-)

But then what happens if I decide to leak the results?  Who gets (hypothetically) sued?

Edit2:  If you were really paranoid, you'd also hold back the decrypting key, so that each tester couldn't know their own results, otherwise any anonymous Joe could decide to post his own log, and who would get blamed for that?

Multiformat 48 kbps Listening Test

Reply #6 –
Doing the same as I did for Nero wouldn't work? I mean, simply leaving out WMA in the result graphs.

Forget it, I also have to release the results as TXT.

Quote from: ff123 on
Edit2:  If you were really paranoid, you'd also hold back the decrypting key, so that each tester couldn't know their own results, otherwise any anonymous Joe could decide to post his own log, and who would get blamed for that?

I am not publishing any results, so I am not doing anything against the EULA.

Multiformat 48 kbps Listening Test

Reply #7 –
As a low anchor I'd suggest something like AAC-LC.

Low anchor is most usefull if it's related to the test (ie similar artifacts/degradations).
Mp3@48 is definitively a lot lower in quality (using current available encoders) than the test contenders. To me it would be too much degraded to be usefull.
I am also against using lowpasses as anchors (even in a mushra context). They are not providing degradations that are related to the kind of degradations that will be found in contenders.
Even CRC people think that the lowpass anchors are bad for a mushra test of modern codecs, as they mostly disturb listeners by providing totally unrelated artifacts, thus only introducing noise in the test.

Regarding contenders, I'd also like to see regular wma, in order to see of it fares against other codecs. After all, regular wma is still claimed to be cd-quality@48.

Multiformat 48 kbps Listening Test

Reply #8 –
I can't agree more with Gabriel. Lowpass (especially the 3.5 KHz one specified by MUSHRA) is useless and disturbing at the same time.
I'd also like to see standard WMA. I'd prefer a newest encoder, but if Microsoft doesn't allow any publication of listening test, we don't have any other choice.
Wavpack Hybrid: one encoder, one encoding for all scenarios
WavPack -c4.5hx6 (44100Hz & 48000Hz) ≈ 390 kbps + correction file
WavPack -c4hx6 (96000Hz) ≈ 768 kbps + correction file
WavPack -h (SACD & DSD) ≈ 2400 kbps at 2.8224 MHz

Multiformat 48 kbps Listening Test

Reply #9 –
Especially with unknown samples, the low anchor is a good indicator, where to expect flaws. LC-AAC at 48 kbps should be a good choice, but HE-AAC at 24 kbps too.

I would also like to see WMA10 completely included or discarded.

My suggestion:

High anchor (LAME MP3 -V5 as in the 48 kbps AAC-Test)
Nero HE-AAC
CT aacPlus
AoTuV Vorbis
WMA10 or WMA9 Standard
Low anchor LC-AAC at 48 kbps or HE-AAC at 24 kbps

Multiformat 48 kbps Listening Test

Reply #11 –
Quote from: rjamorim on
I'd prefer to see some oddball format. But then again, that's me...
what about MV3? (plusV), just using the refrence encoder with lame would be sufficient
lamepv --abr 48
I would love to see plusV in this test.

For low anchors I would use both MP3 and LC-AAC.. but definately at least mp3 because just about everyone would probably have some idea of how a 48kbps mp3 sounds already, and would be a good reference point.

and what about including WMA pro (or std) in this test as well (wma10+ is the new thing, so people aren't using it yet)?  Since WMA is used by some people, it should be in the test.. then there would be a clear answer for any CD quality @48kbps claims.
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune

Multiformat 48 kbps Listening Test

Reply #13 –
If MS gave me the permission to publish WMA results, I could also conduct a WMA test in July or August since there are several WMA encoders now (WMA 9.1 Standard and Pro, 9.2 Standard, 10 Professional and 10 Professional + AFAIK). That's why I am not sure if I should include two WMAs.

Roberto, I thought you mean oddball format for anchors, not competitors.

Multiformat 48 kbps Listening Test

Reply #14 –
If you won't get the permisson, maybe it would be alright to test only Vorbis and the two known HE-AAC v2 implementations?

48 kbps is still a tough nut for encoders that are not optimized for bitrates this low… There's not much sense to test any of the old encoders we're already familiar with.
Infrasonic Quartet + Sennheiser HD650 + Microlab Solo 2 mk3. 

Multiformat 48 kbps Listening Test

Reply #15 –
The older encoders would be used as anchor, not as candidate.

I'm afraid that for a multiformat test the amount of formats would be a bit low if the new WMA is not allowed to participate. 

I guess as alternative the older WMA Pro could be used? Or would it be too obsolete for that?
"We cannot win against obsession. They care, we don't. They win."

Multiformat 48 kbps Listening Test

Reply #16 –
As I said, I am against testing two AAC encoders again. While Nero AAC didn't win the AAC test, it had a slightly better ranking which is why I am going to use it. That's it. No CT. We just had an AAC test that included CT.

WMA10+ should be tested because this is the last 48 kbps test at least this year, unless AAC or Vorbis experience any massive changes, which I doubt. Maybe if we already have test results we can use it to make some pressure on MS. If I still don't get permission, I think that is going to harm MS more than just letting me publish the results.

Multiformat 48 kbps Listening Test

Reply #20 –
I suggest using iTunes 128 kbps AAC as high anchor (about the same quality as they offer in their iTunes Music store).
Do not taunt audiophiles; they may bite.

Multiformat 48 kbps Listening Test

Reply #21 –
I think you should include the Real codec.
BBC uses Real at about 48k.

Multiformat 48 kbps Listening Test

Reply #24 –
If a big site like BBC actually use Real, why not?