Skip to main content

Topic: Multiformat 64 kbps Listening Test (Read 115163 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
Multiformat 64 kbps Listening Test
Based on the results of my last poll, the majority would like a 64 kbps multiformat test. This is also a great chance to see if Microsoft's test results (Nero HE-AAC vs. WMA Professional 10) were / are still valid.

I was thinking about using the same codecs as in the 48 kbps test with the exception of WMA Standard which can either be left out entirely or replaced by Winamp's HE-AAC for example. What do you guys think?
For those of you who don't know the last codecs tested:

Vorbis AoTuV 5 Beta
Nero HE-AAC
WMA Professional 10

iTunes LC-AAC @ 48 and @ 96 kbps for low and high anchor

I was now wondering about the settings to use for WMA Professional. If I recall correctly, it was possible to encode to 64 kbps VBR last time I checked, but looking at the test results Microsoft obtained, it seems that they used 1-pass CBR like in my 48 kbps test. So, if I have the option to encode to VBR, should I go with it or stick to what Microsoft recommended (CBR)? Personally, I would go with option #2. Maybe use both CBR and VBR in case nobody really wants Winamp HE-AAC.

The last test had 20 samples. In order to make things a bit easier for both testers and me, I would like to have a maximum of 18 samples this time. Therefore, I am also open for suggestions on what samples to remove or if you have any sample that is better than one used already, please tell.
  • Last Edit: 02 March, 2007, 11:50:29 AM by Sebastian Mares

Multiformat 64 kbps Listening Test
Reply #1
Wow, look at the countless replies I've received!

  • rjamorim
  • [*][*][*][*][*]
Multiformat 64 kbps Listening Test
Reply #2


Welcome to the wonderful world of listening test feedback
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • Hollunder
  • [*][*][*]
Multiformat 64 kbps Listening Test
Reply #3
Your post isn't a day old yet and it's weekend, are you in a hurry, did you steal something? ;P

Seriously, I'd guess the knowledgeable people simply haven't seen the post yet.

I'm looking forward to the actual test, hope I get some stuff done until then, as it barely makes sense to take part with my current setup.

Multiformat 64 kbps Listening Test
Reply #4
Well, I was expecting some feedback because it is weekend. Anyways, let's see how this turns out in a few days.

  • Junon
  • [*][*][*][*][*]
Multiformat 64 kbps Listening Test
Reply #5
I was thinking about using the same codecs as in the 48 kbps test with the exception of WMA Standard which can either be left out entirely or replaced by Winamp's HE-AAC for example. What do you guys think?

I'd prefer going for WMA Standard @Q25, because my personal experience tells me that this codec isn't uncommon being used on cheap low-memory flash players at this bitrate, at least those which don't support anything but MP3 and WMA. Besides, in a multiformat test I'd generally prefer testing entirely different codecs instead of just different implementations of the same format, like the mentioned HE-AAC ones. If we wanted to test the AAC codecs versus each other, then we could carry out a complete AAC test instead of a multiformat one, which was already done sometime in the last year.
Quote
So, if I have the option to encode to VBR, should I go with it or stick  to what Microsoft recommended (CBR)? Personally, I would go with option  #2. Maybe use both CBR and VBR in case nobody really wants Winamp  HE-AAC.

For practical reasons I'd go for VBR. To achieve the best possible quality people are usually recommended not to encode to CBR if there's no special utilisation like streaming via internet, though the VBR algorithm doesn't always implicitly deliver better results, as we already discussed in the 48 kbps test results topic.

  • jorsol
  • [*][*]
Multiformat 64 kbps Listening Test
Reply #6
Well, I don't like to test too many codecs because is long and somewhat complicated.... said that, I would like to test only:

Vorbis AoTuV Beta5
Nero HE-AAC
WMA Professional 10

Something that I noted with WMA Pro 10 in Winamp encoder, is that in CBR the Sample Format is 16 bits using 44100 Hz and in VBR use 24 bits for 44100 Hz, because of that I think that VBR may have problems to convert 16 bits to 24 bits or something like that... and that the quality will be worse than CBR when used with 16 bits inputs (prety much all Audio CDs). Well this is with the Winamp encoder and with Windows Media Player I think is the same (it use CBR), not sure about any other WMA encoder...
  • Last Edit: 03 March, 2007, 12:08:15 PM by jorsol
JorSol
aoTuVb5 -q4

Multiformat 64 kbps Listening Test
Reply #7
Well, there also has to be a low and a high anchor, so we have at least 5 codecs.

As for the WMA CBR vs. VBR question, I think I will go with CBR since that is what Microsoft seems to recommend and it would be unfair to choose a different setting for the respective codec, while use recommended settings (by developers) for all other contenders. Also, like Junon stated, it might be wiser to either use a different format as fourth contender or leave it out entirely instead of testing another implementation of the same format. I don't know how interesting WMA Standard would be since it performed quite poorly in the 48 kbps.

  • muaddib
  • [*][*][*][*]
  • Developer
Multiformat 64 kbps Listening Test
Reply #8
The last test had 20 samples. In order to make things a bit easier for both testers and me, I would like to have a maximum of 18 samples this time. Therefore, I am also open for suggestions on what samples to remove or if you have any sample that is better than one used already, please tell.


Maybe remove samples for which listeners have least interest, by checking number of results received for each sample in previous tests.
Maybe aditional criteria is to remove samples which are to hard to encode at low bitrates (like fatboy). It seems to me that there are some very hard to encode samples which were used in previous listening test at 48 kbps (those that got very low score).

  • sketchy_c
  • [*][*][*]
Multiformat 64 kbps Listening Test
Reply #9
I'm pretty new to formal listening tests, so take my with a grain of salt, please.

- Fourth test subject: Agreed it should be a different format, but I don't have a preference of which one to add. 

- CBR/ABR/VBR: use developer recommendations.  That said, I'm curious if Microsoft recommends CBR on the assumption that people will choose this bitrate specifically for streaming.  I haven't read their literature on it.

- Samples to remove: The ones that most closely match the overall results from your 48kbps test. EDIT: In a sense, they have the least impact.
  • Last Edit: 05 March, 2007, 09:00:05 AM by sketchy_c

  • muaddib
  • [*][*][*][*]
  • Developer
Multiformat 64 kbps Listening Test
Reply #10
- Samples to remove: The ones that most closely match the overall results from your 48kbps test. In a sense, they have the least impact.


I don't agree with this one since you can not predict if those samples will have the same effect at 64 kbps. But if there is a sample which in many tests don't reveal differences among encoders, then it should be removed.

  • pepoluan
  • [*][*][*][*][*]
Multiformat 64 kbps Listening Test
Reply #11
Well, I was expecting some feedback because it is weekend. Anyways, let's see how this turns out in a few days.
Which shows that most people accessed HA from their work... don't you guys have jobs to do?
Nobody is Perfect.
I am Nobody.

http://pandu.poluan.info

  • benski
  • [*][*][*][*][*]
  • Developer
Multiformat 64 kbps Listening Test
Reply #12
Well, I was expecting some feedback because it is weekend. Anyways, let's see how this turns out in a few days.
Which shows that most people accessed HA from their work... don't you guys have jobs to do?


Some of us are doing our job when we read HA

Quote
Something that I noted with WMA Pro 10 in Winamp encoder, is that in CBR the Sample Format is 16 bits using 44100 Hz and in VBR use 24 bits for 44100 Hz, because of that I think that VBR may have problems to convert 16 bits to 24 bits or something like that... and that the quality will be worse than CBR when used with 16 bits inputs (prety much all Audio CDs). Well this is with the Winamp encoder and with Windows Media Player I think is the same (it use CBR), not sure about any other WMA encoder...


I believe this has to do with the internal precision of the decoder.  It should not effect coding quality at all.
  • Last Edit: 05 March, 2007, 12:14:08 PM by benski

  • IgorC
  • [*][*][*][*][*]
Multiformat 64 kbps Listening Test
Reply #13
Maybe it will be more fair to use a totaly new samples cause new encoders maye be highestly optimizied for the  samples from previous tests. Especially that ones from 48/80 kbps test and hard lame samples 

Multiformat 64 kbps Listening Test
Reply #14
I also feared this several times, but I don't think you can optimize several samples only. I mean, if you optimize for those samples, chances are good that those optimizations will affect other samples as well.
  • Last Edit: 06 March, 2007, 01:05:44 AM by Sebastian Mares

  • muaddib
  • [*][*][*][*]
  • Developer
Multiformat 64 kbps Listening Test
Reply #15
I also feared this several times, but I don't think you can optimize several samples only. I mean, if you optimize for those samples, chances are good that those optimizations will affect other samples as well.


This is true if samples chosen for listening test are representative subset of all existing samples. And for listening test to be good, chosen samples should be representative subset.

Multiformat 64 kbps Listening Test
Reply #16
I don't get your point. The last samples used are a mixture of all important music genres and also some difficult samples.

  • muaddib
  • [*][*][*][*]
  • Developer
Multiformat 64 kbps Listening Test
Reply #17
Let A represent set of all possible samples and B and C represent samples used in a listening test.
If B and C are representative subset of A (meaning that each of them cover all possible distortions that might appear in A due to encoding and that these distortions are evenly distributed) then it does not matter whether you choose B or C for a listening test.
And if some encoder is optimized for B, then it will also give good results for C.
But if you choose subset D which is not representative subset of A (it does not contain all distortions or some distortions are represented more than others) then encoder optimized for B might produce worse results for D.

I hope this did not introduce more confusion 

Basicaly diverse samples should be used for a listening test and some distortion should not be present much more than other distortions (i.e. not to many preecho problematic samples).

Multiformat 64 kbps Listening Test
Reply #18
Vorbis AoTuV 5 Beta
Nero HE-AAC
WMA Professional 10

My $0.02...
Also mp3 and lc-aac at 64kbps would be nice.
Personally, I couldn't care less about high/low anchors.

[offtopic]
work? 
[/offtopic]
  • Last Edit: 08 March, 2007, 01:04:49 PM by gameplaya15143
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune

Multiformat 64 kbps Listening Test
Reply #19
Anchors have to be there regardless if you care about that or not.

  • kanak
  • [*][*][*][*][*]
Multiformat 64 kbps Listening Test
Reply #20
Would it be possible to include LC AAC (nero encoder) as well? Since LC seems to be supported by more hardware devices, it might be more beneficial?

Multiformat 64 kbps Listening Test
Reply #21
At 64? I would rather use LC for the next 80 kbps test.

  • naylor83
  • [*][*][*]
Multiformat 64 kbps Listening Test
Reply #22
I think the three mentioned in your first post sound like a great selection.
davidnaylor.org

Multiformat 64 kbps Listening Test
Reply #23
I think the three mentioned in your first post sound like a great selection.

I second that. 

Maybe a quick preliminary test to determine whether vbr or cbr is better for wmaPro is in order (much like the Nero AAC pre-test conducted before the 48kbps AAC test).
Copy Restriction, Annulment, & Protection = C.R.A.P. -Supacon

Multiformat 64 kbps Listening Test
Reply #24
I will go with CBR because it seems that CBR is recommended by Microsoft. They instructed NTSL to use CBR and not VBR, so I guess they have their reasons. Choosing VBR would go against the decision of using the recommended settings by the developers.

So you think a fourth contender would be too much?