HydrogenAudio

Hi!

So now that Gabriel's HE-AAC test at 48 kbps is over, it's time for a new multiformat test which I would love to conduct. I think we all agree that an extension test at 128 kbps isn't going to be very productive (feedback is welcome, though) and therefore I would say that the new test should also target low bitrates (32 or 48 kbps).

If we target low bitrates, which codecs do you think we should test and why? IMO, low bitrates are optimal for streaming, so the codec decision should be based on this. If this is the case, should we therefore test CBR only?

The test is scheduled to begin late April or early May so that the HE-AAC testers can rest for at least one month.

Regards,
Sebastian

My own opinion is also to test codecs in CBR (but "big" buffer is possible), however in the aac test several people disagreed that 48kbps was streamable, so CBR was finally not mandatory.

Regarding bitrate, I'd vote for 48kbps.

Codecs:
*Vorbis
*wma (pro?)
*he-aac
*mp3 ??? (not sure about this one)

Discarded codecs:
*mp3pro: totally proprietary and sparsely supported. Support is unlikely to grow
*Atrac3: really not its target bitrate, no real use at this bitrate

• I'd like to see atrac3plus, but it's so boring to evaluate (Sonic Stage + acquisition [no CLI decoder nor diskwriter AFAIK] + editing and sample-exact cut) that I won't request it.
• I agree with Gabriel: mp3pro is pretty useless to test nowadays.
• WMApro is impossible (VBR 10) at this bitrate.

So, first thing that we should clear up: which is the target bitrate of this test?

In order to answer this question, we should also define the goal for this test, but when comparing codecs at low bitrates, the only things that come into my mind are streaming or use in devices with limited storage capacity and where Hi-Fi is not really required (mobile phones with AAC support for example - I doubt you need Hi-Fi when sitting in the tram or when driving a car).

Since you say that 48 kbps is not streamable all the time, the only thing left is 32 kbps. If we ignore streaming and focus on portability, we can go up to 64 kbps. However, portability would mean using WMA Standard.

I think it would not be too good to change the bit-rates as we did AAC 48-kbps pre-test to pick-up best 48 kbps AAC encoder.  If we do it, people might complain that wrong / suboptimal HE-AAC encoders are used, etc... etc...

I give my vote to 48 kbps because this is the bit-rate where modern state-of-the-art low bitrate codecs should have a "sweet spot"

- 32 kbps was already tested in Roberto's multiformat test
- 64 kbps was tested, too

48 kbps was never tested in multiformat conditions, and it would be very good to verify how codecs rank in this area.

Regarding compatibility - It makes sense to use compatible WMA codec, but I would also vote for inclusion of the latest WMA Pro codec, to see how this technology progressed.

So, my wishlist is:

- HE-AAC
- Ogg Vorbis
- WMA Standard
- Latest WMA Pro (preferably Vista codec that Microsoft claims is found to be better than HE-AAC)
- mp3
- ???  (maybe RealAudio ?)

Hmm

It's time to find encoder for DVD backup. How about 6ch test? Proposed bitrate 128 or 160 kbps

Nero AAC (HE and LC)
CT AAC (HE and LC)
OggVorbis
Aud-X

What you suggested is almost impossible because you cannot take the test using headphones. Also, 128 kbps is too high.

What you suggested is almost impossible because you cannot take the test using headphones.

Does it mean headphones is test requerment?

Also, 128 kbps is too high.
[a href="index.php?act=findpost&pid=373368"][{POST_SNAPBACK}][/a]

It's normal for DVD-backup puposes

It's time to find encoder for DVD backup. How about 6ch test? Proposed bitrate 128 or 160 kbps
[a href="index.php?act=findpost&pid=373364"][{POST_SNAPBACK}][/a]

The term "backup" always makes me think of "perceptually transparent". Without wanting to start a religious war about this, I gotta say that I'd consider the bitrates you suggest extremely insufficient.

What you suggested is almost impossible because you cannot take the test using headphones. Also, 128 kbps is too high.
[a href="index.php?act=findpost&pid=373368"][{POST_SNAPBACK}][/a]

...128kbps too high for six-channel audio? 

Does it mean headphones is test requerment?
[a href="index.php?act=findpost&pid=373369"][{POST_SNAPBACK}][/a]

Headphones is test requirement - sort of. It's much easier to spot artifacts using headphones.

It's normal for DVD-backup puposes
[a href="index.php?act=findpost&pid=373369"][{POST_SNAPBACK}][/a]

Did you look at my last multiformat test at 128 kbps? All encoders were tied on 1st place, so testing again at 128 kbps or even 160 kbps won't be of any use. Maybe it's more difficult for encoders because they have to share the 128 kbps with 6 channels instead of 2, but still - I doubt I am going to find enough testers in order to have some rankings.

...128kbps too high for six-channel audio?  
[a href="index.php?act=findpost&pid=373373"][{POST_SNAPBACK}][/a]

Well, to be honest, I don't know for sure how bits get distributed across the channels. Is it like Dual Channel for MP3 where each channel gets 1/6 of the 128 kbps or is it like Stereo or Joint Stereo where some sort of middle channel is created and only the difference between the channels is stored in some side channels?

Supposing that 48kbps is the bitrate for listening test.

As for OggVorbis, aoTuV pre-beta 5 released today.
This snapshot aim at improvement @48kbps and this is almost all final(at this bitrate).

Do we test aoTuV beta5?

I am supportive for testing latest and greatest of what all competitors could offer

Dito.

I am supportive for testing latest and greatest of what all competitors could offer
[a href="index.php?act=findpost&pid=373383"][{POST_SNAPBACK}][/a]

Does this mean we need to pre-test OggVorbis too(like AAC),
or blindly use the latest encoder?

As you know, aoTuV has developed since beta 4.0 below -q3.
But no one confirm this AFAIK.

Without wanting to start a religious war about this, I gotta say that I'd consider the bitrates you suggest extremely insufficient.
[a href="index.php?act=findpost&pid=373373"][{POST_SNAPBACK}][/a]

Actually we need ABX test to say it. That's why I propose such test.
Some peoples from Doom9 is satisfied by CT HE-AAC @ 112kbps! So I really ask for such test - nobody berore made such test but huge community (at least from doom9.org ) really need it!

But as I said, it's more difficult to spot artifacts when using loudspeakers instead of headphones.

But as I said, it's more difficult to spot artifacts when using loudspeakers instead of headphones.
[a href="index.php?act=findpost&pid=373391"][{POST_SNAPBACK}][/a]

I understand it. Anycase it's impossible to listen 5.1 into headphones so if You doesn't spot artefacts it's mean encoder is sutable for backup, isn't it?

I'd like to see RealAudio (as in cook) in this, seen as BBC use .RM as the codec for their listen again feature.

Why would we need MP3 in this? You only have to encode an MP3 at 48kbits to know it's not meant for this bitrate. If it's meant as a low-anchor, do we really need one for this low bitrate?


[span style='font-size:8pt;line-height:100%']EDIT: Spelling[/span]

I'd say we make a full stop here regarding 5.1 because it is going to make things worse. I have no idea how many people have decent surround sound cards that are hooked up to decent surround speakers to allow such testing. A low-bitrate multiformat test also seems natural to me now that we have a finished HE-AAC test.

BTW, does ABC/HR even support playback of 6 channel WAVs?

Edit: We could use MP3 as low anchor. Maybe something like Shine...

I am supportive for testing latest and greatest of what all competitors could offer

I am supportive for testing something that can really be used by end-users, more than testing prototypal encoders.

Well, if the new version gets more "mature" until the test starts, I have no problem testing it. I have to agree that testing pre-betas or some nightly builds isn't very productive and can be pretty dangerous (look at the Nero problem in my last multi-format test).

96kbps could be also interesting ...

96kbps could be also interesting ...
[a href="index.php?act=findpost&pid=373482"][{POST_SNAPBACK}][/a]

Without wanting to start a revolt here, I agree :-) But so is 48 kbps, I guess.

I'd also vote for ~90 kbps (leaving headspace for 112kbps ISDN and DSL, while being a good enough bitrate, I think.)

for testing?  The Classics : nero LC-AAC HE-AACv1, vorbis (prebeta5), LAME (latest alpha? Recommended version?) Windows media from wmp10 (std) and itunes mp3, itunes aac, wmp10 mp3.

Another vote for 48kbps. 96kbps is too close to 128kbps and doesn't make much sense to test right now IMO.

I would also like that we try the complete MUSHRA methodology with appropriate ranking scale

- 32 kbps was already tested in Roberto's multiformat test
- 64 kbps was tested, too

48 kbps was never tested in multiformat conditions, and it would be very good to verify how codecs rank in this area.
[a href="index.php?act=findpost&pid=373354"][{POST_SNAPBACK}][/a]

I do not agree with your reasoning. It doesn't matter whether that bitrate was tested before, what we want to know is whether the bitrate is widely used (or if it should be widely used, e.g. the new encoders allow 48 kbps where 64 kbps was required before)

Goal:

Radio Streams: More and more, stations are moving to reduce the bitrate in order to have more listeners/less bandwidth used. We know that HE-AAC sounds quite acceptable for "high" quality (remember 128kbps MP3 is the "high" quality streams). We would verify how much this is true and if there are any other contenders.

Ripping/Webcasting : 48kbps for stereo audio is interesting for videos, when wanting to squeze the bitrate, but not the quality. Not really needed for today's 1-CD rips, but probably interesting for web-downloads/live streams.


Probably there are others, I can't think of more just now.



Encoders:  Those that fit the quality, and how much, will depend on the bitrate, between 5 and 8.

It is always exciting to look forward to a new listening test, however I can't quite figure out what all the enthusiasm would be for a test at 48kbps. 

Nero and CT have just been tested at this bitrate.  Are there other contenders that have made significant leaps here as well?

There has been no testing of 64kbps since long before the great advances made in the past few months with He-aac and Vorbis.  I would love to see how the claims of "close to mp3 at 128" will actually ring true. 

Once the latest aoTuv is released, it would be very interesting to test 64kbps - Nero, CT, Helix, Vorbis ect. 

It would be nice to test Vorbis and the new WMA codec. Ivan wanted to clear some things up regarding the Vista EULA. Also, it is true that Nero and CT have just been tested, but this is something good because now the best HE-AAC encoder can be tested against Vorbis, WMA and other possible competitors.

(BTW, I know that there is no clear HE-AAC winner at 48 kbps, but some tendency can be recognized.)

I vote for 48kbps.
For broadband video streaming, I notice that 48kbps is usually used for audio.

IMHO
I really doesn't believe any non-SBR algoritm can beat or be close to SBR-based for such low bitrate.
And keep in mind AAC-LC (used as carrier) is really very good...
So this proposed multiformat test is mostly predictable - AAC+ wins (I will eat my hat otherwise)  with a big gap (at least 15%)...

,Mar 22 2006, 06:09 AM]Ripping/Webcasting : 48kbps for stereo audio is interesting for videos, when wanting to squeze the bitrate, but not the quality. Not really needed for today's 1-CD rips, but probably interesting for web-downloads/live streams.

Well audio 48 kbit/s is normal for 1 cd-rip.  Imagine you have video 2-2,5 hours -> total bitrate is approx. 700 kbit/s.

700 kbit/s :
1. 652 video + 48 audio .  - good balance between audio/video q.
2.  644 vid  + 56 audio - most balanced for me
3.  636 + 64 audio -  good balance

Today 1-cd rip is used not for backups but something as for mobile needs (traveling, send family video via internet etc. )

@dimzon,

Microsoft announced that according to some tests latest WMA is quite competetive, even better than HE-AAC. 

Since  3G AAC (HE-AAC / HE-AAC v2) and WMA  would be the most important codecs in the years to come for mobile/portable delivery, it would be very nice if this claim could be publically tested under a multiformat listening test.

Also it would be very good to evaluate latest Vorbis - there have been also claims on this forum by some users that Vorbis internet radio stations are better sounding than HE-AAC,  why not testing that, too?

So. Most people are tending to be agree that 48 kbit/s is interesting for internet stream purpose.

What codec will be used as HE-AAC. I  would suppose that will be Nero's one.
But what version? 4.9.9.5  ( SBR only)  or Nero will provide other version for this test?
Yes, that would be interesting to see performance of SBR comparing it with the last prebeta Aotuv Vorbis 5.0 and wma 9.1 .
However HE-AAC is N1 pretender to win test at 48 kbit/s

Microsoft announced that according to some tests latest WMA is quite competetive, even better than HE-AAC.
[a href="index.php?act=findpost&pid=373933"][{POST_SNAPBACK}][/a]

They have only one chance to be competetive with AAC+ - if they use SBR and/or PS

Actually I don't understand at all - why there are no OggVorbis+SBR implementation... It will be awesome IMHO

WMA :

std 9.1: not really a contender.

pro 9.1: difficult to set a value that down, but not impossible.

pro 10: Coming with vista? (but Vista comes next year!). This is the one Ivan mentions, afaik...


Vorbis :  q -1 is precisely what is being tuned the most in aotuv codec and precisely aims to 48kbps.
(Note: Vorbis wouldn't get as much gain from SBR as the MPEG family get, because Vorbis tries to squeeze the bitrate requirements of the highs with other methods. SBR is not a magic keyword for everything.)

HE-AAC : From the pretest, i would get the new Nero codec, with HE-AACv1.

atrac3plus : it aims to 64 kbps... it would be unfair in a 48kbps test.

pro 10: Coming with vista? (but Vista comes next year!). This is the one Ivan mentions, afaik...
[a href="index.php?act=findpost&pid=373974"][{POST_SNAPBACK}][/a]

What do you mean? Both Ivan and I have access to the February release of Vista (build 5308).

,Mar 22 2006, 10:34 PM](Note: Vorbis wouldn't get as much gain from SBR as the MPEG family get, because Vorbis tries to squeeze the bitrate requirements of the highs with other methods. SBR is not a magic keyword for everything.)
[a href="index.php?act=findpost&pid=373974"][{POST_SNAPBACK}][/a]

talking about this i really want to ask Ivan Dimkovic to provide us standalone SBR encoder/decoder CLI utility

SBR_encoder must obtain original wav as input and produce 2 files - file with SBR info and downsampled wav file

SBR_decoder must  obtain 2 files - file with SBR info and downsampled wav file as input and decode it back to single wav

So we can emulate AnyEncoder+SBR

SBR_Encode.exe -input original.wav -out_sbr my.sbr -out_wav my.wav
oggenc.exe my.wav my.ogg -q -1
oggdec.exe my.ogg temp.wav
SBR_Decode.exe -input_wav temp.wav -input_sbr my.sbr -output oggPlus.wav  

@Ivan Dimkovic
Can You provide us such tool? It's REALLY VERY interesting!

I kinda wanna hear a test at 64kbps.

there was just a 48kbps HE-AAC test that to me wasnt really conclusive. So maybe there needs to have more headroom to be conclusive.

that's just my opinion.

I kinda wanna hear a test at 64kbps.

there was just a 48kbps HE-AAC test that to me wasnt really conclusive. So maybe there needs to have more headroom to be conclusive.

that's just my opinion.
[a href="index.php?act=findpost&pid=374010"][{POST_SNAPBACK}][/a]

What do you mean with conclusive?

I would like to see a 64 kbps listening test with the following formats:

- WMA Pro (Vista)
- attrac3plus
- Nero HEv1
- Vorbis

The old WMA std is being toothed as equal in quality to mp3 @ 128 kbps. We all know that an overstatement. It would be interesting to see if the newest and best WMA makes the myth come true...

Sony is toothing Attrac3plus @ 64 kbps and base their battery estimates on this bit rate. Again I would like this test to reveal the quality of the newest attrac3plus.

AAC should be represented and well as Vorbis.

For high anchor I would consider either iTunes/lame VBR @ 128.

For low anchor iTunes/lame VBR @ 64 kbps.

What do you mean? Both Ivan and I have access to the February release of Vista (build 5308).
[a href="index.php?act=findpost&pid=374006"][{POST_SNAPBACK}][/a]

I mean that Microsoft has officially said that Vista won't be released until next year to consumers ( although it expects a november release to businesses/partners).

I'm not sure how much can the codec change (if at all) in these 9 months, so maybe using the beta release you have could be enough.

What do you mean with conclusive?
[a href="index.php?act=findpost&pid=374013"][{POST_SNAPBACK}][/a]

Well i mean the scale is out of 5 right? It seemed the best codec NeroHEv1 only go 3.3 out of 5. That's like what 66%? So from what im getting out of it is that roughly these 48kbps samples sound 2/3 as a good as the original right?

Thus i was suggesting that maybe a 64kbps would allow these lower bitrate codecs to sound better and possibly make the public more informed about them?

Maybe im just reading/interpreting these results wrong?

I would like to see a 64 kbps listening test with the following formats:

- WMA Pro (Vista)
- attrac3plus
- Nero HEv1
- Vorbis

The old WMA std is being toothed as equal in quality to mp3 @ 128 kbps. We all know that an overstatement. It would be interesting to see if the newest and best WMA makes the myth come true...

Sony is toothing Attrac3plus @ 64 kbps and base their battery estimates on this bit rate. Again I would like this test to reveal the quality of the newest attrac3plus.

AAC should be represented and well as Vorbis.

For high anchor I would consider either iTunes/lame VBR @ 128.

For low anchor iTunes/lame VBR @ 64 kbps.
[a href="index.php?act=findpost&pid=374014"][{POST_SNAPBACK}][/a]

I agree. Would be interresting to know how good atrac at 64 is.

I really want to see a (multi-pass) 5.1 test with both audio from music and movie sources @ 128 kbps.

We can test it with our headphones for audio artefacts and with our speakers for the surround problems. Not much 5.1 testing has been done so far and I'm sure the folks at Doom9 would be very interested to participate. With the previous 2.0 tests participation was a bit on the low site. This might rekindle interest. I also want to limit the test to codecs that are already shipping.

Possible formats :

Aud-X (matrix)
Dolby Digital 5.1 (discrete)
Dolby Prologic II (matrix)
Nero AAC 5.1 (discrete)
Windows Media Audio 9 Professional 5.1 (discrete)

I am against doing a 5.1 listening test now, because of:

- Lack of free ABX software able of doing it  (just tried 6ch file with ABC-HR, can't work)

- Problems with the setup, too many possibilities that something going badly wrong

- Probably lack of enough number of listeners (even stereo tests are on the edge of statistical relevance)

So therefore I'd wait for 5.1 test for couple of months, until proper 5.1 listening test software is deployed.

Also, new MPEG surround codecs are coming, that will get requirements for a good 5.1 performance down to 48 kbps, so they should be tested too

Why Nero HE-AAC at 48 kbps? As far as I know, only CT aacPlus is can be considered as popular (Winamp/shoutCAST DSP) an it is tied with Nero AAC in the latest test.

No more arguing about 5.1 testing! Only stereo encoders will be tested.

Why Nero HE-AAC at 48 kbps? As far as I know, only CT aacPlus is can be considered as popular (Winamp/shoutCAST DSP) an it is tied with Nero AAC in the latest test.
[a href="index.php?act=findpost&pid=374865"][{POST_SNAPBACK}][/a]

Well, they are tied, yes, but Nero has a slightly better ranking and that's why I would say we should test Nero.

any new update on what bitrate or codecs?

I would also like that we try the complete MUSHRA methodology with appropriate ranking scale

But is there a good tool for MUSHRA testing? (and by good, I don't mean your buggy comparator (http://faac.sourceforge.net/oldsite/files/wavrate.zip)  )

Besides, it should be a tool that outputs test logs in a format recognizable by Chunky, otherwise it would be a major pain for Sebastian to work on the results.

5.1 test will be cool especialy now. Doing test @ 96kbs will be great. CT now have excelent encoder and 5.1 sounds great at all bitrates and is god for 1cd rips, comparing to Nero 6/7 and lastest beta which is terible.

Encoders in test

Aud-X 5.1 (mp3 5.1)
Nero 7 (lastest beta)
CT (from winamp 5.21)
oog vorbis

Well...read the previous messages and you'll see that a blind test for multichannel is not planned for the moment.


In addition, vorbis multichannel is not well tuned.

Quote from: Ivan Dimkovic on 2006-03-22 00:12:10

I would also like that we try the complete MUSHRA methodology with appropriate ranking scale

But is there a good tool for MUSHRA testing? (and by good, I don't mean your buggy comparator (http://faac.sourceforge.net/oldsite/files/wavrate.zip)  )

Besides, it should be a tool that outputs test logs in a format recognizable by Chunky, otherwise it would be a major pain for Sebastian to work on the results.

Hell no, not comparator (where did you dig THAT from)

Maybe developers of ABC/HR and Chunky would be interested to extend their tools to support ITU-R BS.1534/MUSHRA arrangement.

So far as I could see three changes need to be made:

- Scoring from 0 to 100, istead of 1 to 5

- Rearranging the panel so there are just test samples and hidden reference,  not a hidden reference paired with each sample (useless IMO for 48 kbps, as differences are clearly bigger than a need to blindly identify the orignal against each sample - it just puts too much efforts for the testers without a particular need)

- Modifying Chunky to accept new values (is that needed?)

Maybe there is something else, too?  I would personally like to see this change for low bitrates,  as it would make tests more similar to ones done within EBU, etc...  and in compliance with latest ITU-R recommendations.

Well...read the previous messages and you'll see that a blind test for multichannel is not planned for the moment.

Yes i know, to wait until Nero relase new encoder, why, why test @48 isn't worked wit old nero encoder than new optimized for low bitrate, and in test use old CT encoder.

New Nero encoder will be released much before than any 96 kbps multichannel test could be arranged, so I don't think this is an issue or the reason why there is no 5.1 test.

Sebastian clearly pointed out the reasons why 5.1 multichannel test (regardless the bitrate/contenders) will not be held at this point - please check them out.

Yes i know, to wait until Nero relase new encoder, why, why test @48 isn't worked wit old nero encoder than new optimized for low bitrate, and in test use old CT encoder.

AFAIK, I think there were no significant improvements of 48 kbps between last and current CT encoders  in subsequent Winamp builds (and anyway build revisions were quite minor),  I did tests with at least 20 samples and I did not find any diff - they are probably focusing on something else.  On the other hand, Nero encoder underwent a major update (complete code rewrite).

I think there were no significant improvements of 48 kbps between last and current CT encoders in subsequent Winamp builds

I don't think about CT improvements i think about nero lastest beta which is be on test. This binaries is have big optimisation for low bitrates especilay for 48kbs. I do some test with this binaries and lastest nero 7 ofical binaries and quality is more better, so next time when doing test's use lastest encoders from both CT and Nero.

Sebastian clearly pointed out the reasons why 5.1 multichannel test (regardless the bitrate/contenders) will not be held at this point - please check them out.

Yes it true but test must work on cbr not abr/vbr becouse Aud-X and CT don't have abr/vbr modes.

I'm voting for a 48 kb/s test.  The codecs that ought to be included:
Vorbis (aoTuV pb5), HE-AAC (Nero), LAME latest, and some variant of WMA (but I'm not very educated about WMA, so I won't suggest a version).

Those should be the important ones.  48kb/s should be a good streaming bitrate, and besides, I'm considering it for portable versions of my lossless material, as to me, HE-AAC does sound good at that bitrate.

... Yes it true but test must work on cbr not abr/vbr becouse Aud-X and CT don't have abr/vbr modes.

That's a moot point imho.

I don't think about CT improvements i think about nero lastest beta which is be on test. This binaries is have big optimisation for low bitrates especilay for 48kbs. I do some test with this binaries and lastest nero 7 ofical binaries and quality is more better, so next time when doing test's use lastest encoders from both CT and Nero.

I think that was a general conclusion - Gabriel made some requirements for the tests, and I think they should be usually considered (test latest encoders, if developers recommend them + they become available within next 3 months)  unless there is some extraorinary demand to test something that could not be released on time.

So, I think this is quite safe

Yes it true but test must work on cbr not abr/vbr becouse Aud-X and CT don't have abr/vbr modes.

HA-wise, I do not think that makes too much sense. Goal of the tests is to find best possible solution for encoding at a certain bit rate.  The fact that some of the competitors do not have higher quality modes is absolutely not relevant for the test itself - doing otherwise would mean that you intentionally cripple the quality of the other encoders in order to make "fair" test.

Fair to whom? 

- To participants of the test with less options?  Maybe
- To users, wanting to see what they could get at a certain bit rate (within limits) - I don't think so

Besides, many other encoders (e.g. Vorbis, WMA) probably do not have CBR models similar to MP3/AAC - so I don't see any particular reason to cripple AAC/MP3 encoders.

Only valid reason for test AAC/MP3 CBR would be for fixed-channel streaming (e.g. ISDN, digital satellite / microwave, etc...) over sync. channels - that is where deterministic buffer of AAC/MP3 CBR makes only reasonable sense.  However, I don't think HA community would be interested too much in these kind of tests - but I could be easily mistaken.

I think that was a general conclusion - Gabriel made some requirements for the tests, and I think they should be usually considered (test latest encoders, if developers recommend them + they become available within next 3 months) unless there is some extraorinary demand to test something that could not be released on time.

So, I think this is quite safe

You probably right, but when test started winamp is came out 5.2 which have big improvments, on test is used i think encoder from 5.111 which is very old CT encoder.

Goal of the tests is to find best possible solution for encoding at a certain bit rate.

That is true but you can say Nero HE-AAC is best encoder becouse isn't aslo can't say that for CT but CT work correctly, I see that you doing some test about differences in abr/cbr modes but in practice that is not true, i see all samples from test and most have averge bitrate of 64kbs and more. Aslo did you use Nero BIG oversize like good thing.

Quote

I think that was a general conclusion - Gabriel made some requirements for the tests, and I think they should be usually considered (test latest encoders, if developers recommend them + they become available within next 3 months) unless there is some extraorinary demand to test something that could not be released on time.

So, I think this is quite safe

You probably right, but when test started winamp is came out 5.2 which have big improvments, on test is used i think encoder from 5.111 which is very old CT encoder.

First of all, encoder used in the latest test was from Winamp 5.2 Beta 393

http://www.mp3-tech.org/content/?48kbps%20...20public%20test (http://www.mp3-tech.org/content/?48kbps%20AAC%20public%20test)  (see the encoder IDs)

Do you have any proof of audible differences among Winamp 5.2 Beta 393 and 5.2 retail when it comes to 48 kbps AAC encoding?  I don't - I actually couldn't find any at 48 kbps.

Quote

Goal of the tests is to find best possible solution for encoding at a certain bit rate.

That is true but you can say Nero HE-AAC is best encoder becouse isn't aslo can't say that for CT but CT work correctly, I see that you doing some test about differences in abr/cbr modes but in practice that is not true, i see all samples from test and most have averge bitrate of 64kbs and more. Aslo did you use Nero BIG oversize like good thing.

Nero AAC scored with highest average score among all codecs in the test - that is a fact drawn after statistical analysis of the test results.

No, it wasn't statistically better than CT's encoder, but it did scored better on average, and it was statistically better than CT on one of the tested samples, sample1 - I will not draw any conclusions from that, I leave that to people interpreting the test results to judge what is better then.

Second, it would be appreciated a lot if you could please pinpoint me to which samples had average bitrate of 64 kbps and more?  And which Nero "big oversize" are you talking about?

http://www.mp3-tech.org/tests/aac_48/results.html (http://www.mp3-tech.org/tests/aac_48/results.html)

Average bit rate of Nero AAC was 47.4 kbps  - well within the usual 10% limits of tests conducted here.  It was within 10% of the limits for 17 of 18 samples, and within 15% on only one sample (where it was 54 kbps)

I would follow by a third listening test at 48 kbps. We did pre-tests for Nero AAC then for AAC; the logical new step should be a multiformat test (AAC vs others).
What I'd like to see is the reproduction of the same testing conditions as previous test:
• same anchors
• same samples

Why?
— for curiosity first: it would be nice to see if the same results are reproducible when testing conditions are the same.
— because the previous conditions were apparently very-well balanced. The test ended with both anchors on the extreme position in the scale and all contenders were located on the middle. HE-AAC is, in one sense, like a third anchor: a mid-anchor. Not bad, not good: something exactly in the middle. I'd like to see the position of different contenders in a scale filled by these three references/anchors.
I'm not sure that I'm clear... 

Nevertheless, a new ABC/HR software using MUSHRA arrangement is something I really like to experiment first.

Nevertheless, a new ABC/HR software using MUSHRA arrangement is something I really like to experiment first.

I agree - would be good to test it first, but it seems that full MUSHRA won't be available for this test, either

As far as sample selection goes - I would personally like to avoid using the same samples, as this might lead to assumptions (probably non-justified) that some of the contenders used them to "tune" their encoders.  I don't believe this will happen - but we all know how much flames results usually cause

Do you have any proof of audible differences among Winamp 5.2 Beta 393 and 5.2 retail when it comes to 48 kbps AAC encoding? I don't - I actually couldn't find any at 48 kbps.

Yes there are small improvment which resulting more better quality becouse i founded on winamp forums about build 393 have some bug in enc_aacplus which is correct in next buld i think 440, but pro version of winamp 5.2 have totaly new encoder which support LC-AAC and HE-AAC up to 320kbs.

No, it wasn't statistically better than CT's encoder

I wanna to hear this. , i now the facts but ppl now talking about CT encoder like his most whrose encoder.

Second, it would be appreciated a lot if you could please pinpoint me to which samples had average bitrate of 64 kbps and more? And which Nero "big oversize" are you talking about?

When i found this sampes i post to you. Now i don't have much time to do this. btw about oversize, I do some rip movie and sound will be AC3 stereo, i try to convert using lastest beta @48kbs, the file should be about 35mb and i get about 50mb, finaly i rip sound with CT and i get right size whit amaizing quailty. 48 is good for 1CD rips. Bye

Yes there are small improvment which resulting more better quality becouse i founded on winamp forums about build 393 have some bug in enc_aacplus which is correct in next buld i think 440, but pro version of winamp 5.2 have totaly new encoder which support LC-AAC and HE-AAC up to 320kbs.

I think it is still quite unrelated to the 48 kbps SBR performance of the CT encoder. Unless someone from CT comes and claims that Gabriel did a huge mistake by testing Winamp build 393.

Would also help if people from Winamp could tell which version of CT encoder is used in the Winamp 393 and which one in retail, if it is not a secret - e.g. latest CT encoder version known to me is 7.2.1  - but I am not in position to publish the changelog.

I wanna to hear this. biggrin.gif, i now the facts but ppl now talking about CT encoder like his most whrose encoder.

Now I am confused, who is talking about what?  If you are refering to the listening test results  - they are quite clear: all encoders were tied so there was no winner in the statistical sense, Nero was almost statistically better overall than 3GPP encoder (missed by a very small margin), Nero  also had the highest average score - and, it was significantly better than #2 encoder (CT v1) on one sample.

That's it - calling any encoder "much worse" is a mistake IMO, and people making this mistake are just illiterate when it comes to reading of the test results. 

It could be said that 3GPP HE-AAC v1 is probably worse than Nero HE-AAC v1, but still missing just a little bit in order to be statistically relevant.

When i found this sampes i post to you. Now i don't have much time to do this. btw about oversize, I do some rip movie and sound will be AC3 stereo, i try to convert using lastest beta @48kbs, the file should be about 35mb and i get about 50mb, finaly i rip sound with CT and i get right size whit amaizing quailty. 48 is good for 1CD rips. Bye

What we are talking are the samples from the listening test , where the bit-rate was measured in order to be compliant. And all encoders complied.

Now, if you have a sample which generates "64 kbps" than it is clearly an encoder bug - it would be great if you could send that sample to me, but it indeed sounds pretty weird.

Quote

Yes there are small improvment which resulting more better quality becouse i founded on winamp forums about build 393 have some bug in enc_aacplus which is correct in next buld i think 440, but pro version of winamp 5.2 have totaly new encoder which support LC-AAC and HE-AAC up to 320kbs.

I think it is still quite unrelated to the 48 kbps SBR performance of the CT encoder. Unless someone from CT comes and claims that Gabriel did a huge mistake by testing Winamp build 393.

Would also help if people from Winamp could tell which version of CT encoder is used in the Winamp 393 and which one in retail, if it is not a secret - e.g. latest CT encoder version known to me is 7.2.1  - but I am not in position to publish the changelog.

Build 393 has the same CT encoder as 5.2 and 5.21.  It is the 7.2.0a version of the aacPlus encoder.  All changes made to enc_aacplus between Build 393 and 5.21 were in the "front-end" (GUI, file handling, MP4 handling, etc) only.  Winamp Pro unlocks options for the oversampled SBR mode (which, incidentally, faired well on http://www.soundexpert.info (http://www.soundexpert.info), if their methodology is to be trusted).

Thanks Benski!

Hopefully this clears out the confusion and unnecessary panic concerning the Winamp builds and CT's encoders being used.

It could be said that 3GPP HE-AAC v1 is probably worse than Nero HE-AAC v1, but still missing just a little bit in order to be statistically relevant.

Yes but everyone now has no wish to use 3GPP HE-AAC v1 or CT becouse test prove that is not good encoders. Especialy ppls who first time use AAC. In practice everything is different.

latest CT encoder version known to me is 7.2.1 - but I am not in position to publish the changelog.

I don't know which version is becouse dll don't have version info, but enc_aacplus.dll for 393 have size about 400kb and lastest have about 550-600kb i am not sure.

What we are talking are the samples from the listening test , where the bit-rate was measured in order to be compliant. And all encoders complied.

Yes i am talking about this samples and i said when i have time i download samples again and found this sampes.

Yes but everyone now has no wish to use 3GPP HE-AAC v1 or CT becouse test prove that is not good encoders. Especialy ppls who first time use AAC. In practice everything is different.

Eek... in this case let's just stop doing listening tests, because uneducated people would make wild claims ;-)  Actually I wouldn't use 3GPP HE-AAC v1 anyway as its performance is definitely worse as it could be seen from the graphs.

But it was not statistically worse - (that is, it was almost for Nero, but still not there)

I don't know which version is becouse dll don't have version info, but enc_aacplus.dll for 393 have size about 400kb and lastest have about 550-600kb i am not sure.

Please check out Benski's post - hopefully it will stop completely unfounded speculations about encoders.

Actually I wouldn't use 3GPP HE-AAC v1 anyway as its performance is definitely worse as it could be seen from the graphs

I am just talkig about it, of course me and anybody now don't want use 3GPP.

Eek... in this case let's just stop doing listening tests, because uneducated people would make wild claims ;-)

Of course we have to do some test but not in that way.

Please check out Benski's post - hopefully it will stop completely unfounded speculations about encoders.

Yes i check and i don't know why size of encoder grow up. If version same then is my mistake. Aslo i have real pro version and is not same encoder in pro and free.

Of course we have to do some test but not in that way.

And which way would you propose?

And which way would you propose?

Will see i don't have now much time to explain whole procedure but simply CBR/ABR/VBR vs CBR/ABR/VBR

This approach would make sense if and only if all encoders used same bit-allocation methods, but it is not true.

So, therefore - if you compare Vorbis and WMA to, say, AAC - only measure you would have is average bit rate, total - since bit rate allocations and strategies between these algorithms differ too much.

Regarding AAC vs. AAC - I dunno, only implementation capable of doing CBR, VBR and ABR is Nero - so, you either have to cripple Nero's  encoder - or you don't have too much to test anyway.

Jesus, I wasn't at home one day and look at this monster!

I would follow by a third listening test at 48 kbps. We did pre-tests for Nero AAC then for AAC; the logical new step should be a multiformat test (AAC vs others).
What I'd like to see is the reproduction of the same testing conditions as previous test:
• same anchors
• same samples

Why?
— for curiosity first: it would be nice to see if the same results are reproducible when testing conditions are the same.
— because the previous conditions were apparently very-well balanced. The test ended with both anchors on the extreme position in the scale and all contenders were located on the middle. HE-AAC is, in one sense, like a third anchor: a mid-anchor. Not bad, not good: something exactly in the middle. I'd like to see the position of different contenders in a scale filled by these three references/anchors.
I'm not sure that I'm clear... 

Nevertheless, a new ABC/HR software using MUSHRA arrangement is something I really like to experiment first.

I see one problem - it is possible that some encoders are now tuned to work well with these samples. Choosing (some) new samples can rule out the possibility of testing samples that were used for tuning.

Both 96 and 48 kbps would be great for separate reasons:
• 96 kbps seems like a logical step since we are now pretty sure that 128 kbps are already hard enough to discern;
• 48 kbps is a good streaming bitrate, and it'll be nice to test the performance of the new versions of HE-AAC (from any developer) and Vorbis.
Moreover, neither variant was tested recently.

96 kbps is not too far from 128 where a test was done earlier this year, I think 80's a better choice.

New test will be at 48 kbps, Stereo.

We should start discussing contenders and anchors now, as well as settings. No doubt, Nero HE-AAC, WMA and Vorbis. What else?
What Vorbis version do you guys recommend? Pre-Betas aren't so great to test, so should use use the latest AoTuV Beta? Also, since I didn't receive any reply from Microsoft staff regarding the new WMA codec, I guess I am forced to test the "old" WMA codec - Pro or Standard?

This approach would make sense if and only if all encoders used same bit-allocation methods, but it is not true.

So, therefore - if you compare Vorbis and WMA to, say, AAC - only measure you would have is average bit rate, total - since bit rate allocations and strategies between these algorithms differ too much.

Regarding AAC vs. AAC - I dunno, only implementation capable of doing CBR, VBR and ABR is Nero - so, you either have to cripple Nero's  encoder - or you don't have too much to test anyway.

In multiformat test everything is legal becouse test more different algorithms. Like you see iTunes CBR is better than any lame/vorbis cbr/abr/vbr. But when comparing same tehologies (AAC vs AAC) then make sub testings like CBR vs CBR, if there is not encoders which support same features like test at 48 then no winner, only show results and encoder scores but in different diagrams. So don't make test when not time. Same thing like Video testing, when testing tehologies like ASP,AVC,Theora,VP7 use max setting which encoder have, but testing ASP codecs example XviD/DivX/ND ASP XviD uses Qpel, than DivX and ND aslo use Qpel etc.

Ivan can you provide app for SBR like a dimzon says http://www.hydrogenaudio.org/forums/index....ic=42696&st=25# (http://www.hydrogenaudio.org/forums/index.php?showtopic=42696&st=25#) i wanna see how vorbis or similar codec work with sbr. Thanks

Nero HE-AAC, WMA and Vorbis. What else?

And maybe mp3PRO. I think to wait WMA 10 becouse the winner without test is Nero Right. vorbis don't have SBR and wait until vorbis got real SBR or similar algorithm.

Purpose of the last test was to determine the best codec for the multiformat test.  I sense no point in forcing all AAC encoders to be "CBR" in that case, neither I would find any useful purpose for CBR when there is VBR/ABR available, unless you want to stream your content over narrowband networks.

If you are interested in evaluation of CBR performance of modern AAC encoders, you could still schedule a listening test

Ivan can you provide app for SBR like a dimzon says http://www.hydrogenaudio.org/forums/index....ic=42696&st=25# (http://www.hydrogenaudio.org/forums/index....ic=42696&st=25#) i wanna see how vorbis or similar codec work with sbr. Thanks

Unfortunately I cannot. 

I must point out that SBR is backed up with a set of international patents.  Their inventors agreed to provide these patents under fair and nondiscriminatory terms for the purpose of MPEG and 3GPP/ETSI standardization, as this is one of the requirements of both bodies (ISO and 3GPP)

So, using SBR technology on anything outside of MPEG/3G AAC is a violation of their patents, and it is illegal.  Therefore I am in no legal position to provide anything regarding SBR, which is not in connection with 3GPP and/or MPEG.

And maybe mp3PRO. I think to wait WMA 10 becouse the winner without test is Nero Right. vorbis don't have SBR and wait until vorbis got real SBR or similar algorithm.

Vorbis certainly will not get SBR in any time soon - even I could recall that Monty himself pointed out few times that using of these technologies is not so effective in Vorbis.

I think that if CBR is going to be tested, it should be alongside the VBR settings.  Shouldn't each codec be given a chance to work in it's best conditions (provided that is how they are used).

Do we really need to test mp3PRO? There hasn't been any development during the past years and I doubt anyone still uses it.

Purpose of the last test was to determine the best codec for the multiformat test

Yes nero is winner and go to the next round he have a higest score. My point is to show in diagram that is 3GPP and CT are CBR and Nero is ABR, so Nero is more effective as ABR then, Nero is the better overall but is one step up than other AAC encoders.

So, using SBR technology on anything outside of MPEG/3G AAC is a violation of their patents, and it is illegal.

does it mean even proposed simulation (actually not embeding, just emulation in order to evaluate how good/sutable is SBR in Vorbis) is prohibited? In this case can You provide here e-mail of copyright holders, maybe we(community) can write petition to them. Dont get me wrong - I'm just want to know how good will be Vorbis+SBR in theory - nothing more

Vorbis certainly will not get SBR in any time soon - even I could recall that Monty himself pointed out few times that using of these technologies is not so effective in Vorbis.

There is no sense to work any test now, only will be if WMA 10 came out then Nero HE-AAC vs WMA 10 Pro, it is true that microsoft found some magic to beat sbr

Do we really need to test mp3PRO?

My mistake you right, but i think now there is not nothing to test until WMA 10 not came out or vorbis make something like sbr. I think is better to do a 5.1 test.

Vorbis is not going to make "something like SBR". AFAIK, Vorbis already has something similar to SBR. And how many more times should I write that 5.1 listening tests are impossible right now because of lack of software...

AFAIK, Vorbis already has something similar to SBR.

Hardly, as explained in one of the other threads.

But the idea is vaporware anyway. Current Vorbis research is going into an entirely different direction.

Vorbis already has something similar to SBR

Is this work good or is developers only testing, becouse doing tests with broken encoder is not good. Why just wait until encoders come out in full versions and than make big test with multiforamat and multi bitrates, and all of these encoders is multichannel so we then can make and multichannel test but later.

What Sebastian was thinking of has already been deployed for a few years in Vorbis...

lame @48kbps (low anchor)
lc-aac @48kbps (another low anchor, but a good one that can show that it is better/worse than plain mp3, and would show how much sbr actually does)
he-aac " (probably with nero, since it came out slightly on top in the 48kbps he-aac test)
mp3pro (should be included as a reference since it hasn't changed)
wma 9 pro
mp3+V (just for kicks, I'm curious how bad/good it might do against the rest)
vorbis aotuv b5 (that is if b5 is released when the test gets going.. otherwise b4.51)

Please remember: more codecs = harder on listeners = harder to get useful results.

I'd say MP3 at 48kbps as a low anchor. Then LC-AAC, HE-AAC, and either the aotuv pre-beta 5 or 4.51 at ~48kbps. Maybe WMA std to see how it compares at low bitrates.

I'd say no more than 4 contenders. Do we need both a high and a low anchor?

As far as sample selection goes - I would personally like to avoid using the same samples, as this might lead to assumptions (probably non-justified) that some of the contenders used them to "tune" their encoders.

I see one problem - it is possible that some encoders are now tuned to work well with these samples. Choosing (some) new samples can rule out the possibility of testing samples that were used for tuning.

I didn't thought about this possible issue. Too bad... 

I'd say no more than 4 contenders. Do we need both a high and a low anchor?

I'd say YES without hesitation. They're both important to put results into perspective (see as good example the last listening test).

*I think that once the test goal/conditions (ie 48kbps stereo) is decided, you should start a new discussion thread focusing on it.

*I think that you should have both high an low anchors

*Contenders:
#HE-AAC
#Vorbis
#Wma
#Wma pro
#AAC-LC

*anchors:
mp3?

There is a possibility that mp3 (at least Lame) would be too bad as a low anchor. In this case, you could use AAC-LC as a good low anchor.

I think that inclusing both wma and wmaPro would be interesting. It would clearly establish the quality difference between both.

I see no point in testing mp3Pro, Atrac3 as they are not really used.

Why do you think we should test AAC-LC as contender?

Because of its use in portable players?
Right now there are several portable devices able to play AAC-LC: some Nokia phones, iPods, some Expaniums,...
while I know zero portable player able to play HE-AAC (except in pure software mode)

Aren't Sony Ericsson phones able to play HE-AAC? AFAIK, W800i can, while I don't know about K750i.

Recent Siemens phones also have HE-AAC support.

So, using SBR technology on anything outside of MPEG/3G AAC is a violation of their patents, and it is illegal.

does it mean even proposed simulation (actually not embeding, just emulation in order to evaluate how good/sutable is SBR in Vorbis) is prohibited? In this case can You provide here e-mail of copyright holders, maybe we(community) can write petition to them. Dont get me wrong - I'm just want to know how good will be Vorbis+SBR in theory - nothing more

@Sebastian Mares i agree with you not only Sony Ericsson and may others support HE-AAC

@Gabriel why to work with mp3 and LC-AAC when they can be good at that bitrate. And why doing this test when we know the winner. Vorbis or WMA 9 can't beat Nero HE-AAC now. Wait for WMA 10

And why doing this test when we know the winner.

It may be interesting to see how big the different between he-aac and other competitors is.

I would personally like to see WMA and WMA pro tested.  At the very least, WMA Pro.  I think that around here, there might be a lot of anti-microsoft and anti-WMA sentiment, and I think that it needs to be given the fair chance that other codecs get.

That way, when it turns out to suck, at least there is some quantitative and objective proof.

It may be interesting to see how big the different between he-aac and other competitors is.

That is not interesting that is silly, I want real test with with good enemies where a little thing can change all war LOL

@Gabriel why to work with mp3 and LC-AAC when they can be good at that bitrate. And why doing this test when we know the winner. Vorbis or WMA 9 can't beat Nero HE-AAC now. Wait for WMA 10

Because those tests are not done just to know who is the winner. Relative rankings between codecs, changes of ranking according to samples and the likes are also very interesting. (at least top me)

In this case can You provide here e-mail of copyright holders, maybe we(community) can write petition to them.

You need to get back into reality...

Because those tests are not done just to know who is the winner

It is so hard to everyone see that mp3/LC-AAC is not good @ 48kbs. Better that do a test @ 80kbs becouse every codec in that bitrate can prove something.

Aren't Sony Ericsson phones able to play HE-AAC? AFAIK, W800i can, while I don't know about K750i.

They are playing it in hardware and not in software mode? Then I stand corrected.

Quote

Because those tests are not done just to know who is the winner

It is so hard to everyone see that mp3/LC-AAC is not good @ 48kbs.

Answer is the same: those tests are not done just to know who is the winner.

Quote from: Sebastian Mares on 2006-04-11 09:51:50

Aren't Sony Ericsson phones able to play HE-AAC? AFAIK, W800i can, while I don't know about K750i.

They are playing it in hardware and not in software mode? Then I stand corrected.

Regardless of what they do, does the difference matter if support is shipped by default?

Answer is the same: those tests are not done just to know who is the winner.

So when WMA 10 or similar codec came out you will open new test @ 48kbs right.

They are playing it in hardware and not in software mode? Then I stand corrected.

What is the difference anyway?  There is no "hardware" HE-AAC chip as far as I know of (except Nero's AAC and Digital Radio Mondiale IP core offerings)

So most of the decoders built in todays phones are just using standard ARM or TIC54x AAC decoders (software binary code, that is) running on typical host CPUs  (ARM9, ARM11, Freescale mx21/31, TI-OMAP, Intel XScale, etc...)  -  where exactly that decoder code reside;  in phone ROM, or somewhere on the memory storage has nothing to do with the "nature" of the codec - all these codecs are software codecs in their essenc.e and performance

Only difference is, as Garf pointed out - is this particular software package coming integrated on the phone by OxM / Carrier,  or do you need to download/install the module/application afterwards (e.g. after-market software)

So when WMA 10 or similar codec came out you will open new test @ 48kbs right.

I think there is an ongoing effort by Sebastian to try to get permission for testing of the WMA10 Pro codec, which is shipping with Vista CTP.

If that goes through and Sebastian manages to test  HE-AAC, Vorbis, WMA9 and WMA10  I seriously don't think that the proponents of stereo 48 kbps will change in the next 2 years(*) - this test will be a landmark.


(*) Unless we see Vorbis-II

What is the difference anyway?  There is no "hardware" HE-AAC chip as far as I know of (except Nero's AAC and Digital Radio Mondiale IP core offerings)

I was thinking about usability because of the battery life.

I think there is an ongoing effort by Sebastian to try to get permission for testing of the WMA10 Pro codec, which is shipping with Vista CTP

I have lastest beta of vista installed and i have WMA 10 but i can test becouse is only MP10 support him but nothing under 96kbs.

I seriously don't think that the proponents of stereo 48 kbps will change in the next 2 years

You think that any codec can defeat Nero until next 2 years. I newer work with vorbis so where i can download that vorbis 2, i wanna se how vorbis can beat Nero.

Quote from: Ivan Dimkovic on 2006-04-11 11:55:51

What is the difference anyway?  There is no "hardware" HE-AAC chip as far as I know of (except Nero's AAC and Digital Radio Mondiale IP core offerings)

I was thinking about usability because of the battery life.

Hmm - but if we have 2 decoders:

a) One shipped in the Phone ROM (firmware) running on, say ARM9

b) One shipped as software application (on, e.g. SD card), also running on ARM9

If these two codecs are equally optimized - what is the difference to the battery life then?

Quote

I think there is an ongoing effort by Sebastian to try to get permission for testing of the WMA10 Pro codec, which is shipping with Vista CTP

I have lastest beta of vista installed and i have WMA 10 but i can test becouse is only MP10 support him but nothing under 96kbs.

This is not true - you are doing something wrong then.  I am quite sure you can get 48 kbps

Quote

I seriously don't think that the proponents of stereo 48 kbps will change in the next 2 years

You think that any codec can defeat Nero until next 2 years.

I think we don't have a clear communication - no I don't think "any codec can defeat Nero until next 2 years" but I have a sense that you wanted to say something else?

I newer work with vorbis so where i can download that vorbis 2, i wanna se how vorbis can beat Nero.

I wanna see that too at low bitrates

I am quite sure you can get 48 kbps

Maybe i do some hack into registry i will see. It is not wrong becouse mp3 have range 128-256kbs.

I think we don't have a clear communication - no I don't think "any codec can defeat Nero until next 2 years" but I have a sense that you wanted to say something else?

You understand me but i don't understand you so my mistake

I wanna see that too at low bitrates

What is name of that algorithm which use vorbis to beat SBR. So where to find that vorbis 2

When this test will be openend?

No offense, but whenever I read one of Dzamburu's posts, I get very confused.

There is no Vorbis 2. As far as I understood Ivan, he pointed out that there won't be any major changes at 48 kbps unless something like a Vorbis 2 comes out and therefore there is no need to do any additional 48 kbps tests in the near future (after this one).

Having just got a k750i I can say that it supports playback of he-aac via the inbuilt media player, and are treated exactly the same as any other sound file (mp3, midi, etc).  It also supports mp4 video, but I haven't investigated this feature fully.

I assume that the addition of mp4 support, is because the phone networks will be using it to deliver content via GPRS or 3G.

Hmm - but if we have 2 decoders:

a) One shipped in the Phone ROM (firmware) running on, say ARM9

b) One shipped as software application (on, e.g. SD card), also running on ARM9

If these two codecs are equally optimized - what is the difference to the battery life then?

In this case, there is no difference.
I just think that on devices supporting AAC-LC using decoding hardware, there is a battery advantage against the same device decoding HE-AAC in software, and so AAC-LC might have some advantages in some cases (due to the lack of HE decoding chip).
Let's consider iPods or some old Nokia phones: they have AAC-LC decoding chips, but if you want to add support for HE-AAC to them, it must be done using a software player. In this case, AAC-LC (using chip) will provider better battery life.

All this just to explain that to me there is an interest in including AAC-LC in the test (could be the low anchor)

Just a small clarification - there is no LC-AAC decoding chip, either (unless one I mentioned, in IP-core form)

So, even a LC-AAC is done in software in any case.  Of course, this still means what you claimed - that HE-AAC will consume more battery, but purely to the algorithm complexity itself.

For example, one state-of-the-art decoder known to me works like this:

LC-AAC, Stereo 44.1 kHz:  24 MHz
HE-AAC, Stereo 44.1 kHz:  78 MHz

This is the test done on the real ARM9 powered device, with real memory (no marketing-tuned benchmarks done on ARMulator with simulation of zero-wait-state memory, etc...) and real AAC content (not some "benchmark signal")

So, it is to be expected that HE-AAC will consume more battery than LC-AAC, but this could be tested only in the real-world conditions - it could also be possible that this MHz impact does not play significant rule (some other parts of the DAP consume more battery - e.g. display + sound amplifier + in case of mobile phone RF chipset, etc..)

Just a small clarification - there is no LC-AAC decoding chip, either (unless one I mentioned, in IP-core form)

So, even a LC-AAC is done in software in any case.

So I was totally wrong then. I thought that the aac/mp3 decoding done on the N-Gage was using a specific decoding chip, not a "generic" DSP. (for sure it's not done on the main ARM processor, although it would be fast enough))

Most likely, if the multimedia applications processor is TI-OMAP, as it is in many phones (dunno for Nokia)  MP3/AAC code is usually executed on the C54x/C55x part of the Chip. 

As you migh know, OMAP chips have "dual" core - one is ARM9/11 and other is DSP C54x/C55x - so it is possible to shift decoding to whatever is more suitable for the particular use case of the product - and leave the other unit to do something else (e.g. render video

- Scoring from 0 to 100, istead of 1 to 5

Damn, I find it hard already to decide if a codec gets 4.3 or 4.4. Having a range between 0 and 100 is overkill IMHO.

- Rearranging the panel so there are just test samples and hidden reference,  not a hidden reference paired with each sample (useless IMO for 48 kbps, as differences are clearly bigger than a need to blindly identify the orignal against each sample - it just puts too much efforts for the testers without a particular need)

My hearing is not the best (it sucks actually), but there were several encodes in Gabriel's listening test that I barely distinguished from the reference file. Also, having the reference grouped with the encoded sample makes it easier for me to see if someone didn't actually move the sliders randomly.

Damn, I find it hard already to decide if a codec gets 4.3 or 4.4. Having a range between 0 and 100 is overkill IMHO.

I agree with you, range 0-100 is silly.

So what gona be

Nero HE-AAC
Vorbis 2 this Aotuv beta 5
WMA 9 is not good oppontent so can we use WMA 10. i can force vista and MP10 to use 48kbs

I agree with you, range 0-100 is silly.

It is the range standardized in ITU for the ITU-R BS.1534 recommendation - testing of audio signals with large impairment, such as codecs at 48 kbps.

I will comment later on that subject.

I think 0-100 is a good choice -- if no fractional increments are allowed.  The low anchor would probably be around 10-20, generally, and transparency is at 100.  Different levels of annoyance are in between, and if it has NOTHING to do with the original signal, it's 0.

It is the range standardized in ITU for the ITU-R BS.1534 recommendation

But if that recommendation valid should be then used on test.

Aren't 40 possible rankings enough already? And Dzamburu, there is no such thing as Vorbis 2.

And Dzamburu, there is no such thing as Vorbis 2.

I mean on this Aotuv vorbis, is amazing @ 48, did you try WMA 10, becouse i can force MP10 to 48kbs i have lastest vista 5342 or something like that.

If there will be 2 samples. Very bad quality 1 point and other even worse 0.5 point but min.  mark is 1.

So the mark scale 0-100 will be  more appropriate.

If there will be 2 samples. Very bad quality 1 point and other even worse 0.5 point but min.  mark is 1.

So the mark scale 0-100 will be  more appropriate.

Well, isn't this the reason why we have a low anchor? The low anchor gets the worst score and the other samples are ranked according to the low anchor. So, if low anchor gets 1, the others get 1.3 for example. If one sample sounds even worse than the low anchor, the low anchor can get 1.5 and the sample 1.

I agree with Sebastian that a scale of 0-100 is not really justifiable and seems too granular. When a person ranks a sample as 73, he should be able to explain why he didn't rank it 72 or 74 instead. I can only speak for myself of course, but I certainly wouldn't be able to do this.

I think that a 0-100 range would be appropriate if we expect big impairements.
In this context you are likely to only use the lower part of the scale for most contenders.
We will perhaps encounter 1 good contender, that would be ranked at 70% of the scale, and several "less good" ones that would be ranked between 25 and 50% of the scale, the low anchor beein ranked at about 10%.
In such case, you only have about 25% of dynamic range to rank most contenders. To me, a 0-100 scale makes sense.

0-100 scale and tools:
Right now we have a 40 points scale (1-5 with a .1 resolution). The 0-100 scale is more a matter of visual presentation to the user than a real granularity matter.
I think that a modified testing tool could display a 0-100 scale to the user, but output results in the 1-5 range in results file, allowing to keep tools modifications to a minimum.

When a person ranks a sample as 73, he should be able to explain why he didn't rank it 72 or 74 instead.

The same for mark scale 1-5 . Why to choose 3.7 and not 3.6 or 3.8?  Sometimes I wanted  to put somethig like 3.75 .

The idea of previous post is  100% scale has higher "definition" (range) . Personaly I will feel myself more comfortable/free  to mark samples this way.

Sebastian Mares hm. I didn't think about low anchor at that moment. But 100% scale has error 0.01 , and 1-5 40 steps - error 0.025.
From previous 48 he-aac test CT and Nero were on par. With error 0.025  CT aac+  would have 3.18*102.5% = 3.25  , with error 0.01 - 3.18*101% = 3.21

OK, just so the discussion is not needlessly simplified by lack of suitable software, I just put support for custom rating scales into ABC/HR for Java: Binaries (http://user.uni-frankfurt.de/~bkuckuck/abchr-java-0.52b.zip), Sources (http://user.uni-frankfurt.de/~bkuckuck/abchr-java-0.52b-src.zip)

You do quite the kick-azz job, schnofler.  Way to go!

OK, just so the discussion is not needlessly simplified by lack of suitable software, I just put support for custom rating scales into ABC/HR for Java: Binaries (http://user.uni-frankfurt.de/~bkuckuck/abchr-java-0.52b.zip), Sources (http://user.uni-frankfurt.de/~bkuckuck/abchr-java-0.52b-src.zip)

Thank you very much.

Any chance you could add an option for incrementing 2000 to the calculated offsets? Currently, you have to calculate the offsets and then go through each sample and manually type in the offset.

Any chance you could add an option for incrementing 2000 to the calculated offsets? Currently, you have to calculate the offsets and then go through each sample and manually type in the offset.

Here you go: Binaries (http://user.uni-frankfurt.de/~bkuckuck/abchr-java-0.52b.zip), Sources (http://user.uni-frankfurt.de/~bkuckuck/abchr-java-0.52b-src.zip)

Thanks. Do I have to enter the additional offset before calculating the offsets, after, or doesn't it even matter?

Thanks. Do I have to enter the additional offset before calculating the offsets, after, or doesn't it even matter?

Doesn't matter. The additional offset is simply added to the individual sample offsets.

OK, thanks schnofler!

As for the new WMA codec, after several e-mails, and newgroups / forums posts I was told that I have to check with the licensing department of Microsoft. However, I doubt that they are going to give me a permission in time (or at all). There are two things that are possible now:

• Wait until the public beta test of Windows Vista begins with the hope that Microsoft will also change the EULA allowing results to be posted
• Test the old WMA codec

Personally, I would go with 2. However, I am not sure if the test is going to be so interesting.

The public beta test is scheduled to start in May.

Damn, I don't know becouse wma9 don't belong in group of low bitrate encoders and have terrible quality at 48kbs. If you make second decision then let's test begin,  personally, i would to see that new Microsoft beast

If you intend for this to be the last 48kbps test for a while then I'd say wait for the public beta.

Well, I don't know what to intend. Unless the big players HE-AAC and Vorbis don't face major improvements, it's going to be the last at 48 kbps for at least a year.

Small update: Microsoft is discussing my request internally and someone will be back in contact with me soon.

Small update: Microsoft is discussing my request internally and someone will be back in contact with me soon.

was MS going to present open beta test for WMA 10 already in may?

Nobody ever mentioned an open beta test of WMA 10. The only open beta test that is planned for May is for Windows Vista, but I am not sure if they are also going to change the EULA to allow publishing of benchmark results and that sort of things.

Threre is a small chance that Microsoft will provide you with a stand-alone app, then Vista's EULA does not matter.
Let's wait until MS decision, then discuss 

Yeah, I hope so too.

Today's new major update to Nero's AAC codec should be relevant to this listening test.

Also, Sebastian, any love from Microsoft?

Nope.

I also have problems getting the new April build to run under VMware - it keeps crashing with a BSOD. I have to squeeze my Ubuntu partition a bit.

Quote

I agree with you, range 0-100 is silly.

It is the range standardized in ITU for the ITU-R BS.1534 recommendation - testing of audio signals with large impairment, such as codecs at 48 kbps.

I don't see the point in the MUSHRA test, to be honest. It seems to be more of an excuse to allow mediocre-performing codec/bit rate combinations to get high scores. If a codec/bit rate combination does cause a large impairment, then I think it should be scored as a large impairment on the original BS.1116 test. At the end of the day, reduced bit rate audio coding should be about trying to get as close to the original as possible, IMO, not letting codecs off just because they're using a low bit rate.

And if you change the scale from 0-5 to 0-100 you run the risk of people confusing results from a BS.1116 test with a MUSHRA test, which has a totally different scale:

MUSHRA (EBU Tech Review article about MUSHRA (http://www.ebu.ch/en/technical/trev/trev_283-kozamernik.pdf)):

80 - 100 Excellent
60 - 80 Good
40 - 60 Fair
20 - 40 Poor
0  - 20 Bad

which is not the same as the BS.1116 scale. And according to the EBU article it says:

"MUSHRA uses an unprocessed original programme material of full bandwidth as the reference signal. In addition, at least one additional signal (anchor) – being a low-pass filtered version of the unprocessed signal – should be used. The bandwidth of this additional signal should be 3.5 kHz."

So it's not really surprising that you get very high scores for mediocre performance when you've got such an incredibly low quality anchor.

For example, CT 48kbps HE AAC scored 88 in a MUSHRA test (if I remember correctly), whereas in the recent test on HA it only got somewhere in the region of 3.5 out of 5, and such a big difference has got to be down to the difference in the testing process.

I think you need to stick with the 0-5 scale using the BS.1116 test, as you've always used, or if you're going to change the scale to 0-100 you should fully implement the MUSHRA testing methodology, but not mix the two tests together.

Quote from: Ivan Dimkovic on 2006-04-12 16:55:38

Quote

I agree with you, range 0-100 is silly.

It is the range standardized in ITU for the ITU-R BS.1534 recommendation - testing of audio signals with large impairment, such as codecs at 48 kbps.

I don't see the point in the MUSHRA test, to be honest. It seems to be more of an excuse to allow mediocre-performing codec/bit rate combinations to get high scores. If a codec/bit rate combination does cause a large impairment, then I think it should be scored as a large impairment on the original BS.1116 test, and the MUSHRA test just makes it more difficult to compare the performance of codecs/bit rates because it creates a totally new scale and confuses lay people, because they see a high score (e.g. CT 48kbps HE AAC getting a score of 88 on a MUSHRA test) and think it means that the codec provides high quality at that bit rate, when the recent HE AAC test showed that it's not all that good in absolute terms. I think it's best to stick with using BS.1116.

So, your only argument not to use MUSHRA is that you suppose that it could be an "excuse" to get high scores that might not be warranted in absolute terms? Do you know what these listening tests are used for in the ITU? First and foremoest, to find the best encoding solution, with the best possible accuracy and discriminating power.

Did you even read Gabriel's post?

http://www.hydrogenaudio.org/forums/index....ndpost&p=381952 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=42696&view=findpost&p=381952)

I edited my post while you were replying, so the edited version might cast more light on why I think changing to the 0-100 scale isn't a good idea. Basically, it's to avoid confusing the results from BS.1116 and MUSHRA tests, because they do use very different testing methodologies.

Maybe I was being a bit harsh when I described 48kbps HE AAC as "mediocre", but do you think the CT 48kbps HE AAC deserved a score as high as 88? In absolute terms I don't think it deserves 88.

As I said in my edited post, I think it would be better to either stick with the 0-5 scale or change completely to MUSHRA, because you can just imagine the confusion you'd get when a MUSHRA test gives a score of 88 and then a BS.1116 test with a scale of 0-100 only gives it 70.

And fair enough, I can see the reasoning behind MUSHRA, and saying that it's just an excuse to allow very high scores for mediocre-performing codecs/bit rate combinations is a bit harsh, but given some of the claims made about 48kbps HE AAC, such as "CD-quality stereo down to 48 kbps" (http://www.codingtechnologies.com/products/aacPlus.htm), I think a 48kbps test should use BS.1116.

Another update folks: MS didn't reply - not even after having sent a reminder. Seems that the public beta of Windows Vista is going to be available later this month. I hope, they also changed the EULA.

OK, here is what I thought...
I sent MS a last reminder and told them to reply by the end of next week (21.05 the latest). If not, I told them I am going to test 9.1.
Now, the Vista EULA says I am not allowed to disclose the results of benchmarks - benchmarking itself is allowed. If MS doesn't give me the permission to publish the results yet, I could simply include the codec in the test, but exclude it from the results until the EULA changes. In that way, I am not breaking EULA and I also tested it against HE-AAC and Vorbis. I am doing this because I am not sure when the next multiformat test at 48 kbps is going to take place. Also, only testing WMA 10 against WMA 9.1 later in order be able to compare the results will not work since the testing environment changed between the upcoming multiformat test and the WMA vs. WMA test (even if the same samples are used, maybe other listeners will participate and even if I also have the same testers, they might be in a different mood or whatever). I hope you understand what I mean.

Edit: What I mean is this... Let's say we run the multiformat test and the following results are available:

WMA 9.1: 3
HE-AAC: 3.5
Vorbis: 4

If we later test WMA 9.1 against WMA 10 and have the following results:

WMA 9.1: 2.5
WMA 10: 4

we cannot say that WMA 10 is as good as Vorbis, because the testing environment changed between the two tests. Even if WMA 9.1 would still get 3 in the second test, it wouldn't mean anything.

Now, the Vista EULA says I am not allowed to disclose the results of benchmarks - benchmarking itself is allowed. If MS doesn't give me the permission to publish the results yet, I could simply include the codec in the test, but exclude it from the results until the EULA changes.

That would still be illegal.

The EULA applies to the program you are using. If you encode samples with a version which EULA forbids benchmark disclosure, you can never disclose said benchmarks, even if a later version of the same program changes EULA, since that new EULA will only apply to the later version, and not the old version.

Microsoft says that WMA10Pro+ will give better quality @64 Kb/s than aacPlus v2 so take WMA10Pro+ to test  is it true that new WMP11 (with new WMA10 codecs) will be to download on 17 mai? When you install WMP11 then in XP system in Windows Media Encoder you will see new codecs to converting (WMA10Voice, WMA10, WMA10Pro and WMA10Pro+)

Quote from: digitalradiotech on 2006-05-02 14:14:24

Quote from: Ivan Dimkovic on 2006-04-12 16:55:38

Quote

I agree with you, range 0-100 is silly.

It is the range standardized in ITU for the ITU-R BS.1534 recommendation - testing of audio signals with large impairment, such as codecs at 48 kbps.

I don't see the point in the MUSHRA test, to be honest. It seems to be more of an excuse to allow mediocre-performing codec/bit rate combinations to get high scores. If a codec/bit rate combination does cause a large impairment, then I think it should be scored as a large impairment on the original BS.1116 test, and the MUSHRA test just makes it more difficult to compare the performance of codecs/bit rates because it creates a totally new scale and confuses lay people, because they see a high score (e.g. CT 48kbps HE AAC getting a score of 88 on a MUSHRA test) and think it means that the codec provides high quality at that bit rate, when the recent HE AAC test showed that it's not all that good in absolute terms. I think it's best to stick with using BS.1116.

So, your only argument not to use MUSHRA is that you suppose that it could be an "excuse" to get high scores that might not be warranted in absolute terms? Do you know what these listening tests are used for in the ITU? First and foremoest, to find the best encoding solution, with the best possible accuracy and discriminating power.

Did you even read Gabriel's post?

http://www.hydrogenaudio.org/forums/index....ndpost&p=381952 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=42696&view=findpost&p=381952)

I think there isn't even a point discussing wether to use MUSHRA or not, since we have no access to a working comparator.

Maybe somebody interested in it could contact Schnofler?

Microsoft says that WMA10Pro+ will give better quality @64 Kb/s than aacPlus v2 so take WMA10Pro+ to test  is it true that new WMP11 (with new WMA10 codecs) will be to download on 17 mai? When you install WMP11 then in XP system in Windows Media Encoder you will see new codecs to converting (WMA10Voice, WMA10, WMA10Pro and WMA10Pro+)

I don't have any official information about the release date of Windows Media Player 11.

I think there isn't even a point discussing wether to use MUSHRA or not, since we have no access to a working comparator.

Maybe somebody interested in it could contact Schnofler?

http://www.hydrogenaudio.org/forums/index....ndpost&p=382210 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=42696&view=findpost&p=382210)

Microsoft WMP 11 released, and can be downloaded here:
http://www.9down.com/story.php?sid=6574 (http://www.9down.com/story.php?sid=6574)

Don't know if this package include the new codec or not.

Don't know if it's already been mentioned, but does the Microsoft EULA even apply in Germany? Wikipedia says it does not: http://de.wikipedia.org/wiki/EULA (http://de.wikipedia.org/wiki/EULA) (german)

Edit: Should have read the Wikipedia article to the end. It seems, Microsoft EULA does indeed apply here, because the software needs to be downloaded from the internet. It it was bought in a store it would not apply, since the contract about the use of the software is made while buying it, and as you cannot read the EULA before buying it would not be valid.
So...

Microsoft WMP 11 released, and can be downloaded here:
http://www.9down.com/story.php?sid=6574 (http://www.9down.com/story.php?sid=6574)

Don't know if this package include the new codec or not.

I don't see the official Microsoft page mentioning the availability of a Windows XP version of WMP 11.

Microsoft WMP 11 released, and can be downloaded here:
http://www.9down.com/story.php?sid=6574 (http://www.9down.com/story.php?sid=6574)

Don't know if this package include the new codec or not.

Can also download from http://www.shooter468.com/ccount/click.php?id=32 (http://www.shooter468.com/ccount/click.php?id=32)

Don't know if this package include the new codec or not.

WMP11:

(http://img88.imageshack.us/img88/8917/wmp11wma10pro2sw.th.jpg) (http://img88.imageshack.us/my.php?image=wmp11wma10pro2sw.jpg)

The Good
Writes ID3 tags to the physical file regardless where the rip folder is
Windows Media Audio 10 Professional codec
Windows Media Audio 9.2 codec
Expanded tile view
Live search
Very responsive media library

The Bad
Aggressively replaces my high quality Folder.jpg with a low res Folder.jpg
Does not notify the user when ID3 tag writing fails
Does not let remove monitor folders (only ignore)


WME9:

(http://img135.imageshack.us/img135/2291/wmp11codecsxp7yo.th.jpg) (http://img135.imageshack.us/my.php?image=wmp11codecsxp7yo.jpg)

Two new codecs:

Windows Media Audio 9.2
Windows Media Audio 10 Professional

Cheers,
McoreD

Interesting, now we could run a WMA test: 9.1 Standard vs. 9.1 Professional vs. 9.2 Whatever vs. 10 Professional.

http://www.hydrogenaudio.org/forums/index....ndpost&p=382210 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=42696&view=findpost&p=382210)

Hrm... OK, but custom scale is not enough to be a MUSHRA test. There must also be only one hidden reference for the whole test, and not one for each sample.

Also to be true MUSHRA, there must be at least one anchor, and the recommendation for that first anchor is 3.5 kHz lowpassed.  After all, the procedure is termed MUlti Stimulus test with Hidden Reference and Anchors.  I think this part is probably more important than the 100-interval scale or the labels or maybe even eliminating the side-by-side reference for each codec, because the anchors are what standardizes the ratings.

I wonder if a current 48 kHz test isn't becoming too good for the original intent of MUSHRA?

ff123

Microsoft WMP 11 released, and can be downloaded here:
http://www.9down.com/story.php?sid=6574 (http://www.9down.com/story.php?sid=6574)

Don't know if this package include the new codec or not.

It works. WMAStd 9.2 & WMAPro 10 (or 11) is automatically working with different audio tools (like dBpoweramp). WMAPro is now working with very low bitrate (dbPowerAmp crashed with bitrate < 48 kbps). It shows interesting things like very high lowpass value, similar to SBR encoders but without SBR artefacts (but with other ones, clearly more irritating IMO).
The decoding side is less complex than SBR encoders: WMAPro at low bitrate is twice faster on decoding. Encoding speed has been clearly lowered with WMAPro.
I noticed strong regressions with WMA 9.2 Std ~128 kbps with harpsichord.

I didn't played that much this week-end with this encoder.

Both new WMA work fine for me.

I feel compelled to point out that there are 2 versions of WMA 10 Pro included with WMP11.  The first version--"WMA10Pro+"--is available only within WMP (under "ripping options") as Windows Media Audio Pro using SBR and PS (or direct clones of these technologies), which is quite obvious not because of the artifacts and similar sound (that would be TOS #8 ) but becuase when played on my pocket pc, it plays as mono 22khz. 

The second version of WMA 10 Pro (no plus) is usable via WME and external applications, does not use SBR or PS and is playable on my pocket pc flawlessly.

Just thought I'd point out that piece of important info

edit: Also, I'd like to refer your attention to the following post from Doom9's forum: http://forum.doom9.org/showpost.php?p=827660&postcount=2 (http://forum.doom9.org/showpost.php?p=827660&postcount=2), which states:

That's a leaked build 11.0.5358.4826 from April 19th. I'm pretty sure that wasn't the final beta build...

The Sheep of DEATH, thanks for that info. Unfortunately "WMA10Pro+" version still displays as "Windows Media Audio 10 Professional". Hope they make the difference clear.

WMA ripped from WMP11:

Windows Media Audio 10 Professional
128 kbps, 44 kHz, 2 channel 16 bit 1-pass CBR

WMA converted using WME9 while having WMP11:

Windows Media Audio 10 Professional
128 kbps, 44 kHz, 2 channel 16 bit 1-pass CBR

FYI, the new version of WMP 11 Beta is officially out. See new thread about it at:
http://www.hydrogenaudio.org/forums/index....showtopic=44719 (http://www.hydrogenaudio.org/forums/index.php?showtopic=44719)

Thanks, I just read about it on heise.

Did you guys read point 6 of the EULA?

6. SCOPE OF LICENSE.  The software is licensed, not sold. This agreement only gives you some rights to use the software.  Microsoft reserves all other rights.  Unless applicable law gives you more rights despite this limitation, you may use the software only as expressly permitted in this agreement.  In doing so, you must comply with any technical limitations in the software that only allow you to use it in certain ways.    You may not

* disclose the results of any benchmark tests of the software to any third party without Microsoft’s prior written approval;
* work around any technical limitations in the software;
* reverse engineer, decompile or disassemble the software, except and only to the extent that applicable law expressly permits, despite this limitation;
* make more copies of the software than specified in this agreement or allowed by applicable law, despite this limitation;
* publish the software for others to copy;
* rent, lease or lend the software;
* transfer the software or this agreement to any third party; or
* use the software for commercial software hosting services.

Did this version of wma 10 use SBR+PS because sound's very good and suppored by and hardware player.

As far as everyone here knows - there is no public information that Microsoft is using SBR and PS technologies in their codecs, so I don't think it is a good idea anyway to call these technologies with those names.

For sure, it seems from the evidence on this forum, that Microsoft is indeed using some frequency-enhancing technology, maybe similar to HE-AAC's SBR - but what kind of technology is that exactly it is still unknown.

Did you guys read point 6 of the EULA?

Quote

You may not disclose the results of any benchmark tests of the software to any third party without Microsoft’s prior written approval

Wait a minute, does this mean no more listening tests with WM codecs?

This wasn't in the WMP10 EULA, was it?  Does an ABC/HR test count as a "benchmark test"?

Did this version of wma 10 use SBR+PS because sound's very good and suppored by and hardware player.

WMA10Pro+, which is usable in WMP11 ONLY and not in WME and 3rd-party apps, does use it (or something close, if not necessarily it, but judging by the SBR+PS artifacts and the fact that its 32kbps 44.1khz stereo files play back as 22khz mono on my Pocket PC, yeah).
WMA10Pro standard, available in WME and 3rd-pary apps and which is compatible with previous hardware devices and WMA9Pro decoder, doesn't use SBR+PS-type technology as far as I can tell, but simply uses "other" high-end preservation tools which don't act quite like SBR+PS and retain compatibility with previous decoders.

Wait a minute, does this mean no more listening tests with WM codecs?

No, it means no listening test with WMP 11 Beta or Vista Beta. I guess the reason is that the software is still beta and MS doesn't want to get a bad image if something doesn't work right or in a suboptimal way in these versions.

Did you guys read point 6 of the EULA?
...
You may not

* disclose the results of any benchmark tests of the software to any third party without Microsoft’s prior written approval;

Then results of benchmark have to be released by someone who did not accepted WMP11 EULA.

It says "to any third party". So the person doing the benchmark should not have accepted the WMP11 EULA either. (IOW - someone else has to encode the WMA files and then hand them over).

Although, I don't agree with using beta software in a listening test if the developers do not give their consent.

Thing is,

Some people on this forum that, I believe were sounding like "insiders" or maybe even more than that  recommended waiting for this very codec.  So, it would be very unfortunate if Microsoft does not grant Sebastian a right to test the WMA10Pro+ codec.

It would be a very valuable codec for the 2006 low-bitrate codec shootout.

I agree it would be unfortunate if WMA could not be tested. But I would find it even more unfortunate if a codec was tested that had an apparent bug in it. It would be bad for both sides, as it is unneccesary bad advertising for the codec (ok, if it's not good, it's not good, but each codec should at least have a fair chance), reduces the value of the test and would be a waste of time.

I remember Woodinville asking for waiting for WMA Vista Codec for the Sebastian's  128 kbps test - dunno if that has any weight but for me it sounded that the codec seems to be mature enough

Also, Microsoft anyway did a PR where the quoted independent testing lab results of quite competetive quality of the WMA10+ codec.

It still does not mean, of course, that the codec is finished - but for me personally it sounded like it really was, if not completely finished, then at least already reached the state of a good stability

Sorry to ask this but what do you mean by PR ?


@Sebastian : have a look at this (http://forum.doom9.org/showthread.php?p=828840#post828840)...

Public Relations  MS issued marketing material claiming the independent test lab found new WMA codec to be better than HE-AAC.

I think this was discussed here at HA.

Thanks kurtnoise, Ivan already told me about that thread. I lost my Doom9 password (I always use 32 random character PWs) and the bloody forum doesn't want to send me the reset mail.

Anyways, I am not sure how much of help it would be to have the e-mail address of those right people, since I already mailed the WMA and licensing division and they told me that they are discussing the issue internally. Didn't receive an update ever since. So, I am afraid if I write another e-mail, I might get the same reply and then that's it.

download new Windows Media Player 11 and install then Rip from CD to WMAPro at 64 kb/s - sound of new codec Windows Media Audio 10 Professional at 64 Kb/s it's GREAT!

Please do not make illegal posts:

http://www.hydrogenaudio.org/forums/index....ndpost&p=393261 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=42696&view=findpost&p=393261)

You might find that line stays in the EULA after release of WMP 11 full, ie if you have done a test that MS does not agree with then you cannot publish.

Even this beta version, I doubt such a post is illegal, yes if the user had signed an NDA it might hold true, but not a click through EULA (of a public beta) - the majority of items in a EULA if they take rights away which are normal for a country will not be enforcable.

The question from my side would then be if "hosting" the result of a listening test which is a violation of that license, is in itself illegal (I for sure never agreed to the EULA on WMP11). Any pointers or precedents? I know Oracle pulls the same sh*t with their database.

The thing is that I know that:

• The MS audio codec developers read HydrogenAudio
• They are well-aware that we run tests
• Sebastian asked and did not get premission to publish test results
• It's expressly forbidden in the license
• MS might or might not like the test results
• Lawyers have harassed us in the middle of the night before and threatened legal action against HA for things which were only arguably license violations (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=44105&view=findpost&p=388921)

So my desire to stick out my neck for this is entirely nonexistant.

Hosting test results can not be a violation of a license that you did not accepted, IMO.
To me the workaround is simple: the person conducing the test should not accept this EULA, and so should ask someone else to provide the wma encoded/decoded samples.

Well, Ivan managed to get in touch with someone on D9 who is trying to help out now by talking to some MS people who are responsible for this. Anyways, I will include WMA in the test and then hope to get a permission to publish the results. If not, shit happens.

I would really like to conduct the test myself, but I agreed to the EULA already. Also, even if someone else would conduct and gathers the encoded samples from someone else, what if that person who did the encoding screwed up something? I really don't think it's a good idea if two or more people do the preparation job.

Do you guys think I should start a new thread for discussing the encoder selection and settings, or can we stick here?

Also to be true MUSHRA, there must be at least one anchor, and the recommendation for that first anchor is 3.5 kHz lowpassed.  After all, the procedure is termed MUlti Stimulus test with Hidden Reference and Anchors.  I think this part is probably more important than the 100-interval scale or the labels or maybe even eliminating the side-by-side reference for each codec, because the anchors are what standardizes the ratings.

I wonder if a current 48 kHz test isn't becoming too good for the original intent of MUSHRA?

ff123

My original intent was to get a better scale than the current one. I already explained why I dislike the usual 1...5 one (in short: the scale is too small and the corresponding description isn't well balanced for my own taste). In that perspective MUSHRA was interesting but I didn't mean that we have to religiously follow all MUSHRA recommendations. Anyway, previous experiences are proving that MUSHRA can't pertinently be used « as it » with listening tests involving HE-AAC. A 3.5 KHz lowpass as obligatory low anchor with contenders like HE-AAC (where lowpass is often = or > 16 Khz) is close to be nonsensical. And the high-anchors (7.5 lowpassed) were also rated as significantly inferior to HE-AAC encoding; it defeats the purpose of high anchors!
Exemples:
http://www.ebu.ch/en/technical/trev/trev_283-kozamernik.pdf (http://www.ebu.ch/en/technical/trev/trev_283-kozamernik.pdf)
http://www.telos-systems.com/techtalk/host..._AAC_paper).pdf (http://www.telos-systems.com/techtalk/hosted/m4-in-30100%20(M4IF_HE_AAC_paper).pdf)
and also one previous test from Roberto using the same anchors


My idea would be to think about and finally decide about a new, well-balanced and universal scale for the upcoming listening tests rather than follow existing recommendations.
A quick conversion should explain why the current one (1-5) is a problem. If you convert it to a 100% scale, it will look exactly like this:

			   CURRENT SCALE	100% SCALE

transparent			5.0		  100%
perceptible but not annoying	4.0	 	   75%	
slightly annoying		3.0	 	   50%
annoying			2.0	 	   25%
very annoying			1.0		    0%

75% for something that is not annoying?! A med-iocre [50%: mid of the scale] ranking for something that is just slightly annoying? Does it sound balanced to you?

1/ I would personaly say that « perceptible but not annoying » is a state, not a level, and  that it barely accepts nuance. And for harsh, demanding or intolerent people, the sole existence of such state may even be nonsensical: every audible difference starts to be annoying (this point of vue is defendible).
In my own perception, the « perceptible but not annoying » state would correspond to very subtle distortions I can hear with high lossy encodings (160 kbps or more) during specific conditions (like meticulous ABX trials) and should be at 95% of a 100% scale. Maybe 90% for rounding purpose, but certainly not 75% like in our current 1-5 scale.

2/ The « slightly annoying » state would correspond to artefacts or distortions I can hear without too much effort but that don't make my hackles rise. To give a concrete example it may correspond to LAME -V5 encodings: they're not completely transparent; I don't need two hours to succesfully ABX it; but quality is pretty good despite the few audible differences I can perceive. On a 100% scale it would correspond to 75...80% (with some nuances: 85% for distortions that are near non-irritating; 70% for artefacts that are just slightly more than slightly annoying.
Would it be a scandal if collective listening tests would end with LAME VBR 130 kbps at 80...85% of a 100 full quality? I don't think so.

=> my own vision of the top of a balanced scale would shift our current scale from one step (4.0 = slightly annoying and 4.8 = perceptible but not annoying).

If everybody agrees with it, we could follow:
100% = transparent /unABXable encoding
90% = perceptible but not annoying
80% = slightly annoying
70% = slightly annoying
60% = annoying but decent
50% = annoying but decent
40% = unpleasant
30% = very unpleasant
20% = bad/very bad
10% = very bad
0% = chaos, total mess

or something like that (someone with a better english lexical knowledge may refine it).

That way, I could imagine as possible something like that:
MP3 ~192 kbps = 95...100%
MP3 ~160 kbps = 90%
MP3 ~130 kbps = 80%
OGG ~100 kbps =70%
OGG ~80 kbps = 60%
HE-AAC 64 kbps = 50-55%
HE-AAC 32 kbps = 40-45%
WMA 64 kbps = 25-30%
etc...

It's of course my own vision (or imagination). But it gives me a coherent position and coherent distance among different coding solutions at different bitrate - something I can't get with the current scale (HE-AAC would obtain 2.0 "annoying" but then I don't have room enough to coherently rate a low anchor (3.5Khz) and WMA 48 kbps: the low anchor should get 0 but the minimum is 1/5).

...
MP3 ~130 kbps = 80%
OGG ~100 kbps =70%
...

Definitely classic samples  not modern.

Definitely classic samples  not modern.

Mmm, doesn't Vorbis handle classical samples better than the other codecs? I have an impression that Vorbis tends to be very effective in encoding clear tones, but fail to handle noise correctly (at least at <q5).

Can we please keep the discussion about the listening test and not discuss opinions that require one?

@Sebastian

Perhaps for clarity it is better to start a new thread with an appropiate title (e.g. upcoming 48 kbps listening test - discussion). I think it would draw refreshed attention to it.

OK, will start a new thread today or tomorrow. Just have to get a project done.

Hosting test results can not be a violation of a license that you did not accepted, IMO.
To me the workaround is simple: the person conducing the test should not accept this EULA, and so should ask someone else to provide the wma encoded/decoded samples.

Sounds good, unless the beta eula also forbids distributing encoded music.

Alternate ending: compile results.  If Msoft doesn't allow showing them, put it that way:

"Microsoft has seen how WMA stacks up against the others, and doesn't want it revealed."

I am definitely not going to use such tricks and stuff like person X is conducting the test, Y is encoding the samples and Z is publishing the results. WMA is going to be included and results will be disclosed if MS agrees and if not, shit happens.

What I think it sucks is that they still have the WMA vs. HE-AAC page up where they tell the people how great WMA performed, but now they don't let someone proof this. Also, someone from MS said in a newsgroup that these beta versions are work-in-progress products and cannot be compared to final versions - yeah, but using these betas to show off against HE-AAC is perfectly fine. How fair is that?

What I think it sucks is that they still have the WMA vs. HE-AAC page up where they tell the people how great WMA performed, but now they don't let someone proof this.

Are you really that surprised?

Why would they want people to prove it? Marketing isn't made of hard proofs, it's made of embellishment.

It's been a good while since this thread has been updated.  Any news?  Are we waiting for the release of Vista or something so that we can test WMA Pro properly?

edit: I guess M$ could be a problem here huh?  They probably don't want proof of how bad their codec really is    It'd be nice to see if it's improved though.

OK, will start a new thread today or tomorrow. Just have to get a project done.

Was there another thread that carries this subject on, that I don't know about?

I think at least until Aoyumi finished tuning aoTuV5 there...

It's been a good while since this thread has been updated.  Any news?  Are we waiting for the release of Vista or something so that we can test WMA Pro properly?

edit: I guess M$ could be a problem here huh?  They probably don't want proof of how bad their codec really is    It'd be nice to see if it's improved though.

Quote from: Sebastian Mares on 2006-05-21 19:17:30

OK, will start a new thread today or tomorrow. Just have to get a project done.

Was there another thread that carries this subject on, that I don't know about?

Thread: http://www.hydrogenaudio.org/forums/index....showtopic=44881 (http://www.hydrogenaudio.org/forums/index.php?showtopic=44881)
Poll: http://www.hydrogenaudio.org/forums/index....showtopic=45840 (http://www.hydrogenaudio.org/forums/index.php?showtopic=45840)

However, the poll made no sense at all and I am sorry I started it. The test was delayed until 2007 when either Vista or WMP 11 is available as final version.

I would appreciate it if a moderator could lock this topic and the two aforemoentioned ones, since the 48 kbps test was delayed until next year.

Dicussion about the next test(s) will follow shortly.