Skip to main content

Topic: NSTL WMA Pro listening test (Read 21678 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Woodinville
  • [*][*][*][*][*]
NSTL WMA Pro listening test
Quote
Quote

i think they are 'supposed' to represent a % quality.. but they are just high-to-low (or should it say low-to-lower... uh oh.. i better watch out for tos8)
[{POST_SNAPBACK}][/a]



Have you heard problems with a ~240Kbps VBR (Q98) wma 9.1 file to say it is low quality?
[a href="index.php?act=findpost&pid=356416"][{POST_SNAPBACK}][/a]


How about this new report (DBT and all) on WMA Pro quality?

[a href="http://www.microsoft.com/windows/windowsmedia/9series/codecs/Comparison.aspx]http://www.microsoft.com/windows/windowsme...Comparison.aspx[/url]
-----
J. D. (jj) Johnston

  • Cyaneyes
  • [*][*][*][*]
NSTL WMA Pro listening test
Reply #1
Quote
How about this new report (DBT and all) on WMA Pro quality?

http://www.microsoft.com/windows/windowsme...Comparison.aspx
[a href="index.php?act=findpost&pid=356444"][{POST_SNAPBACK}][/a]



It would be nice if they stated the bitrate of the HE-AAC used in the test.

NSTL WMA Pro listening test
Reply #2
Quote
It would be nice if they stated the bitrate of the HE-AAC used in the test.[a href="index.php?act=findpost&pid=356450"][{POST_SNAPBACK}][/a]
one would asume that the bitrate was close to that of the wma it was being compared to...  but that IS just an asumption

i liked this line: "71% of all listeners indicated that WMA Pro was equal to or better than HE AAC."

there is a 48kbps multiformat test coming up isnt there? I'll wait for those results 

@[JAZ] no problems at that kind of a bitrate.. but then again i dont know, i don't really use wma, i just keep the dbpoweramp encoder around for occational tests
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune

  • stephanV
  • [*][*][*][*]
NSTL WMA Pro listening test
Reply #3
Quote
It would be nice if they stated the bitrate of the HE-AAC used in the test.
[a href="index.php?act=findpost&pid=356450"][{POST_SNAPBACK}][/a]

Bitrate is in the pdf and is indeed 64 kbps. Even precise steps to how the clips were encoded are given in there.

Of course using the same rethoric you could say that 63% of the participants rated HE-AACv2 equal or better than WMA.
"We cannot win against obsession. They care, we don't. They win."

  • kritip
  • [*][*][*][*][*]
NSTL WMA Pro listening test
Reply #4
Anyone notive the MS codec used?

Quote
15. Set the “Codec” to Windows Media Audio 10 Professional+


I don't keep up on the WMA codecs, is this a new version?? I thought 9.1 Pro wsa the latest avaliable??

Kristian

  • [JAZ]
  • [*][*][*][*][*]
NSTL WMA Pro listening test
Reply #5
** Test codecs:

WMA Pro at 64Kbps? Interesting...It might be the codec they are making for Vista.
Let's see if now they ditch WMA Standard (or start to do so, even though "PlaysForSure")

OTOH, wouldn't HE-AAC V1 achieve better results than HE-AAC v2 at 64kbps?

**Test methodology:
Faults ( IMO ):

From the .pdf, it seems that nor ABX neither ABC-HR was done when electing the samples(1), plus the randomness in which the samples where played is questionable(2).

Overall, most of the results seem random(3), except for a few samples(4) where there could really be a difference.


(1) "The first clip to be played for every song was the reference clip. After listening to the reference clip, the participant would hear two additional clips, Clip-A and Clip-B. The participant was asked to compare both Clip-A and Clip-B to the reference file, and determine which sounded more like the reference file. NSTL would replay any of the three clips for the participant upon request, but once a decision was made on a song, the test would advance to the next song"

(2) BBABBBAAABAA  ( B->HE-AAC first, A-> WMA first. ). They did this for 2/3rd of the participants, and inverted the order for the other 1/3rd.


(3) For most of the samples, the listeners elected "Identical". The percentage in which they elected wma or he-aac is similar. No big differences in prefference (only ~5% although generally is in favour of WMA...)

(4) The results, from my POV (point of  view):
BobMarley,ColdPlay,Greenday,si02 -> Favour WMA

SimpleMinds,BlankBaby,Floyd -> Seem to Favour WMA, but with a small margin.

LimpBizkid,DireStraits -> There really is a difference, but prefference between one or the other is random (or averaged)

GuanoApes,Maroon5, -> There could be a difference, but the distribution is much averaged.

TheVerve -> Seem to favour HE-AAC, but with a small margin.


**Conclusion:

It could be true that WMA Pro 10 at 64Kbps is better than Nero 7, HE-AAC v2 at 64Kbps from the results shown. It definitely shows some problematic samples (especifically BobMarley,Greenday,si02) for HE-AAC, and that WMA Pro 10 is now competitive at this bitrate.
Let's see if it can be included in the multiformat low-bitrate test that we will do soon.



[Edit -> Forgot to say a few things]
  • Last Edit: 12 January, 2006, 01:23:10 PM by [JAZ]

  • spoon
  • [*][*][*][*][*]
  • Administrator
NSTL WMA Pro listening test
Reply #6
> don't keep up on the WMA codecs, is this a new version?? I thought 9.1 Pro wsa the latest avaliable??

9.1 is the latest, the confusion comes from WMP 10.

This report, to tell you the truth I wouldn't believe 1 word of it, even if it appears to be from a separate testing agency (guess who pay their bills...). The same goes for any company, Nero, FhG, any self-tests or sponsored tests should not be given a second thought, it is all advertising spiel. That is why listening tests by non-affiliated groups / people are worth their weight in gold.

  • singaiya
  • [*][*][*][*]
NSTL WMA Pro listening test
Reply #7
I found it interesting that at the very bottom of the pdf report, they write "NSTL does not guarantee the accuracy, adequacy or completeness of the services provided to Microsoft or the data included herein." I know that disclaimers are common, but in this case it seems ironic especially for a company that on the same page claims "NSTL is the leading independent information technology testing organization for the computer industry, dedicated to providing high quality services and test tools to hardware developers, software publishers, government agencies and corporations."

Also for such an large testing group I think they should have included more types of music such as classical and jazz. It's nearly all rock/pop oriented.

NSTL WMA Pro listening test
Reply #8
Quote
"NSTL is the leading independent information technology testing organization for the computer industry, dedicated to providing high quality services and test tools...."[a href="index.php?act=findpost&pid=356623"][{POST_SNAPBACK}][/a]
.... they forgot to add "maybe" 
I didn't know they were using some new wma 10+ codec... just have to wait till it's available to see how it really fares
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune

  • Woodinville
  • [*][*][*][*][*]
NSTL WMA Pro listening test
Reply #9
I must admit that I am both offended and annoyed by the various insinuations in this thread. Hence, I shall say nothing here, and await evidence of the various insinuations and suggestions.
-----
J. D. (jj) Johnston

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
NSTL WMA Pro listening test
Reply #10
@Woodinville,  is there any way for the public to obtain the codec used in the test - so some kind of public verification could be done in the future, or this codec is to be released with Vista?

There is a new 48 kbps multiformat listening test scheduled for, most likely, March (after 48 kbps AAC listening test) - and testing the latest WMA codec would be a very very nice thing, especially after latest claims from the NSTL.
  • Last Edit: 14 January, 2006, 03:42:52 AM by Ivan Dimkovic

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
NSTL WMA Pro listening test
Reply #11
Quote
I must admit that I am both offended and annoyed by the various insinuations in this thread. Hence, I shall say nothing here, and await evidence of the various insinuations and suggestions.
[a href="index.php?act=findpost&pid=356931"][{POST_SNAPBACK}][/a]


I'll make 2 factual observations:

1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

Edit: Since this behaviour seems to have been fixed/may not have existed in the version used, you can ignore this for the sake of argument. I'll replace it by: 1) It seems impossible to verify the test results and setup.

2) They consulted with MS how to set up WMA and got new tools that are apparently not publicly available. They didn't consult with us, and used an old version of the Nero AAC encoder.
  • Last Edit: 14 January, 2006, 10:02:59 AM by Garf

  • Gabriel
  • [*][*][*][*][*]
  • Developer
NSTL WMA Pro listening test
Reply #12
Quote
1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

You mean that when selecting “Nero Digital Audio (HE-AAC v2) (*.mp4)” only AAC-LC is used by default? If this is the case, you probably have a user interface design problem.

  • Latexxx
  • [*][*][*][*][*]
NSTL WMA Pro listening test
Reply #13
Quote
1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).
[a href="index.php?act=findpost&pid=356950"][{POST_SNAPBACK}][/a]

If this is the case, somebody should contact Microsoft because they can't keep that comparison online if they know that it is completely false. Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.

And doesn't that mean that Microsoft's hyper super ultra new and amazing WMA 10 Pro + is only tied with lc aac instead of he aac.
  • Last Edit: 14 January, 2006, 08:03:15 AM by Latexxx

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
NSTL WMA Pro listening test
Reply #14
Quote
Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.
[a href="index.php?act=findpost&pid=356998"][{POST_SNAPBACK}][/a]

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?

  • Latexxx
  • [*][*][*][*][*]
NSTL WMA Pro listening test
Reply #15
Quote
Quote
Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.
[a href="index.php?act=findpost&pid=356998"][{POST_SNAPBACK}][/a]

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?
[a href="index.php?act=findpost&pid=357000"][{POST_SNAPBACK}][/a]

No. But if they really have tested lc aac and know it, they can't keep lying by keeping that document publically available.

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
NSTL WMA Pro listening test
Reply #16
Quote
Quote
1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

You mean that when selecting “Nero Digital Audio (HE-AAC v2) (*.mp4)” only AAC-LC is used by default? If this is the case, you probably have a user interface design problem.
[a href="index.php?act=findpost&pid=356991"][{POST_SNAPBACK}][/a]


You select that "Nero Digital Audio (HE-AAC)" out of a list of MP3Pro, TwinVQ, WMA, AIFF, WAV etc.

By default, it will encode to 128kbps LC-AAC after selecting that.

Selecting the CBR/Stereo - 64kbps [HE-AAC] preset works correctly.

You have to go inside the encoder settings, force it to 64kbps CBR, and ignore the explicit combobox that says "LC (Low Complexity) AAC", to get it to fail this way. And that is exactly what their description says they did.

PS. I do agree this part of the GUI could be done better, but it can only fail this way by tinkering with some "advanced" settings and ignoring others. I don't think removing the LC-AAC options is good, either.
  • Last Edit: 14 January, 2006, 08:35:37 AM by Garf

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
NSTL WMA Pro listening test
Reply #17
Quote
Quote
Quote
Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.
[a href="index.php?act=findpost&pid=356998"][{POST_SNAPBACK}][/a]

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?
[a href="index.php?act=findpost&pid=357000"][{POST_SNAPBACK}][/a]

No. But if they really have tested lc aac and know it, they can't keep lying by keeping that document publically available.
[a href="index.php?act=findpost&pid=357005"][{POST_SNAPBACK}][/a]

I agree with you. It would be a lie. But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).

EDIT: emphasis in the quote
  • Last Edit: 14 January, 2006, 08:33:03 AM by guruboolez

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
NSTL WMA Pro listening test
Reply #18
Quote
But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).
[a href="index.php?act=findpost&pid=357011"][{POST_SNAPBACK}][/a]


If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???

  • sehested
  • [*][*][*][*]
  • Members (Donating)
NSTL WMA Pro listening test
Reply #19
Quote
Quote
But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).
[a href="index.php?act=findpost&pid=357011"][{POST_SNAPBACK}][/a]


If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???
[a href="index.php?act=findpost&pid=357014"][{POST_SNAPBACK}][/a]
You could contact NTSL to obtain the .wav files used in the test - both reference, A, and B samples.
You could then encode the reference using the same software and settings as described by NTSL and make one LC-AAC and one HC-ACC.
This would allow you to find out what settings where in fact used.

  • sehested
  • [*][*][*][*]
  • Members (Donating)
NSTL WMA Pro listening test
Reply #20
What about volume leveling?

With the test procedure described by NTSL the encoder raising the volume the most would normally win such a comparison.

Not that I suspect neither WMA PRO nor AAC of boosting the volume, however software is software and IMHO the testing metholody should have taken this issue into account. At least provide an analysis of the gain of the resulting wav files.

  • Alex B
  • [*][*][*][*][*]
NSTL WMA Pro listening test
Reply #21
Hmm...

I followed the procedure described in the test report and got this Nero encoding setting:





Isn't this a HE setting?

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
NSTL WMA Pro listening test
Reply #22
Quote
Quote
But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).
[a href="index.php?act=findpost&pid=357011"][{POST_SNAPBACK}][/a]


If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???
[a href="index.php?act=findpost&pid=357014"][{POST_SNAPBACK}][/a]

You should read the testing procedure again. Step#6:
Select "Nero Digital Audio (HE-AAC v.2)....
The person has set Nero to use HE-AAC v.2 as described in the frontend. If the result is LC-AAC, then it would rather look to a bug than a feature.

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
NSTL WMA Pro listening test
Reply #23
Quote
Isn't this a HE setting?
[a href="index.php?act=findpost&pid=357025"][{POST_SNAPBACK}][/a]


Yes. Does it automatically switch to HE-AAC for you? Older versions of Nero 7 don't seem to, and one newer one I have here doesn't either.
  • Last Edit: 14 January, 2006, 09:25:55 AM by Garf

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
NSTL WMA Pro listening test
Reply #24
Quote
You should read the testing procedure again. Step#6:
Select "Nero Digital Audio (HE-AAC v.2)....
The person has set Nero to use HE-AAC v.2 as described in the frontend. If the result is LC-AAC, then it would rather look to a bug than a feature.
[a href="index.php?act=findpost&pid=357027"][{POST_SNAPBACK}][/a]


I read the testing procedure and it says that person also went on to the detailed encoder settings and changed them. As I already said, using the defaults, or using a preset, all would have produced a good result.
  • Last Edit: 14 January, 2006, 09:28:20 AM by Garf