Skip to main content

Topic: Personal multiformat listening test at ~130 kbps (Read 36753 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Personal multiformat listening test at ~130 kbps
[span style='font-size:14pt;line-height:100%']INTRODUCTION[/span]

During a sleepless night, I decided to perform my own listening test, with samples based on my musical tastes. I chose a friendly bitrate encoding (~130 kbps) : in my opinion, it’s a good compromise for limited hardware space (notebook) and good quality (my preliminary experience with lower bitrate was catastrophic). Of course, testing encoding at this bitrate range is much easier. I also had in mind the idea to build a comfortable library for my notebook, and the idea to clarify some feelings I had about different encoders, on my music only. Among them :
- I suspected lame mp3 to be a good audio format on some (quiet) situation, and to not suffering as much as with pop, metal or electronic music.
- I suspected Nero AAC to be simply worse than Lame MP3 ; I did some encodings on common music the last times, and I was disappointed by the poor results I obtained for a VBR setting (-streaming). I couldn’t conclude on Nero worse quality based on three or four samples. This extend test will be more conclusive.
- I suspected QuickTime AAC and WMA 9 PRO to reach a near-transparency quality at this bitrate.



[span style='font-size:14pt;line-height:100%']SAMPLES[/span]

Unfortunately, I worked with a limited choice of samples. With the start of the new university year, I had to leave my summer house. My discs are not for the moment in my new apartment : only the few one I bought since September. I choose samples from them, and completed the gallery with samples I burned on CD-R some times ago. Most of them are music from baroque period. It’s my favorite one. Therefore, I couldn’t materially built a coherent sample library for this test with romantic or contemporary music. That’s why there’s no piano, no Beethoven, no electronic samples here. That’s why, too, so many harpsichord (hard-to-encode instrument) is present, during singing, sacred, chamber or orchestral music. I did my best to find some samples without this instrument : more of the half.
Therefore, this test only had a limited impact: results are useful for baroque music only ; they may be different for other kind of “classical” ; and they must be totally different for jazz, country, R&B or anything else. Of course, notation is mine, dependant on my own subjectivity, my on hardware setting, and on the mood I had during the test.



[span style='font-size:14pt;line-height:100%']ENCODERS[/span]


I encoded them with foobar2000 (except for WMA9 and PsyTEL AAC). All encodings were replaygained, original too. Decoding stage include the calculated gain (without dithering – maybe not the best choice). I didn’t make any offset correction (needed for some AAC encoding).
Challengers are :
• AAC Faac (don’t know the exact version : I used Case diskwriter component bundled with foobar2000 0.7.1 beta 4), quantizer 128, LC, m/s coding and with TNS.
• AAC Nero (aac.dll 2.5.6.3 & aacenc32.dll 2.5.5.8) : -streaming, LC, High quality
• AAC PsyTEL 2.15 –streaming [ADDED ONE WEEK LATER: I did a big mistake, and placed the good aacenc32.dll in the wrong location Therefore, I tested an old encoder : 2.5.1.6. Some odd artifacts (Dorilla, Passacaglia) are not present in 2.5.5.8. See here for complete report and additional test]
• AAC QuickTime 6.3, CBR 128, High Quality
All AAC files were decoded with faad2 (foobar2000)
• MP3 Lame 3.90.3 –alt-preset 134
• MPC 1.14 –quality 4 –xlevel
• OGG vorbis GT3b1 –q 4.25
• WMA 9 PRO VBR-2pass 128 kbps

(I chose abr 134 in order to be close to 128 kbps, because ABR mode isn’t reliable with most classical stuff. I used –b 4.25 for Vorbis, according to the average size collected some times ago, before Roberto’s test: it was a small mistake).


[span style='font-size:14pt;line-height:100%']DECODING & PLAYBACK[/span]


All files (including reference) were replaygained with foobar2000, and then decoded with calculated gain and without dithering, No offset correction was applied (needed for correcting some AAC encoding offset) : I didn’t have the correct value.
Note than one sample (« Laudate pueri ») wasn’t properly encode/decode with PsyTEL : 15 seconds are missing, due to a bug (« negative scalfactor… »). I rated the quality for the first part only.

Comparisons were made with ff123 tool (ABC/HR). I didn’t take the time for ABXing (except for one sample).

I worked with my notebook, an its weak AC Audio hardware. My desktop computer is with all my CDs : away. Headphone is a Philips SBC-HP910.



[span style='font-size:14pt;line-height:100%']RESULTS[/span]





[span style='font-size:14pt;line-height:100%'](few) COMMENTS[/span]


• The most shocking results are Nero AAC poor performances. The quality is terribly poor. Faac isn’t for the moment a good encoder ; Nero, on my 18 samples, isn’t really better, but is slower and with a slightest higher average weight… I suspect (and hope) that a bug in current version is responsible. “Dorilla” sample should be an interesting one; “cello” too, but in a more common way. “Passacaglia” sample was never annoying, except with Nero encoder: maybe a problem with background noise, noticed last week with the “biniou” sample.
Interesting thing to note: PsyTel is better for my ears than Nero. Only one sample encoded with Nero ancestor was considered worse than with latest Nero encoder (“hornpipe” sample).
ADDED ONE WEEK LATER: The biggest and unexpected flaws I heard, and badly rated, with Nero AAC, are removed with the latest stable and official AAC encoder, 2.5.5.8. See
here for complete report and additional test]. Nevertheless, the more usual distortions are still present
.


• Lame mp3 isn’t a bad choice for this music, and for my ears. Better than Faac, Nero and PsyTEL AAC, not too far from overall MPC and Vorbis performances.

• Vorbis surprised me : on Roberto’s multiformat 128 test, vorbis wasn’t too competitive, due to hiss/HF boost and dirty feeling it gives to me. Here, problem wasn’t always perceptible. Nevertheless, on some samples, this added noise gives the impressions of coarse manufactured instruments, rough tones, etc… Good ranking for vorbis ; excessive bitrate consumption will be partially corrected with a conventional setting (-b 4). Few pre-echo problems : I used GT3b1 encoder (but it was maybe not the best choice – I forgot that foobar2000 component was based on this particular –and particularly good– library)

• MPC was very good during Roberto’s test; with classical (baroque) only, performances are not so exciting. Nevertheless, mpc stays competitive in quality area. Unfortunately, and as expected, average bitrate isn’t competitive at all: strings, harpsichord, and ‘tonal music’ in general are bitrate greedy.

• AAC QuickTime is very pleasant. I must confess that I’m a bit disappointed: I hoped better results. Some distortions (flanging) are still noticeable with this very good encoder. Best encoder on 6 samples (33%).
With classical, QuickTime is in my opinion the very best AAC encoder at mid-bitrate.

• WMA9 PRO : I expected very good results. They are excellent. Very few flaws (more annoying was the unstable noise with “Requiem” ultra-quiet part). I couldn’t distinguish the encoded from the reference on 4 samples. Best format for 10 samples (55%).
This new encoder fits to my ears :-)




Note : I’ve got a very limited internet connection. I will try to upload the 18 samples, but I need time for it. I will begin this night (i.e. in 6 hours), with the most important one (useful, in my opinion, for Ivan). Then, I will try to finish the next week on a DSL connection. Samples will be hosted on HA server (Upload Forum). Thanks for comprehension.

Note.2 : bitrate table will follow. Total size of all files are at the bottom of the previous table.

Note.3 : I posted this result in AAC forum, just because 4 different AAC encoders were tested, though it wasn't the goal of the test.

http://membres.lycos.fr/guruboolez/AUDIO/t...early/early.htm

EDIT : total size for PsyTEL and QuickTime are changed (html table only) ; I forgot to include one sample.
  • Last Edit: 18 October, 2003, 06:48:04 AM by guruboolez

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Personal multiformat listening test at ~130 kbps
Reply #1
Nero's performance on tonal instruments is due to tonality estimation algorithm in Nero AAC - it has been on my replacement schedule for a long time, but SBR development was taking all of my research time.

Now I am working on completely new tonality estimation method, and it is under in-depth tests.  I hope it will manage to be completed until the next web release (actually, it is a work of couple of months, it is finally been debugged and now the parameters are being calibrated for each bit rate) - some initial feedback resulted in very positive ranking of older problem clips.

guruboolez: plese PM me if you want to test new tonality algorithm.
  • Last Edit: 11 October, 2003, 11:45:46 AM by Ivan Dimkovic

  • mdmuir
  • [*][*][*]
  • Members (Donating)
Personal multiformat listening test at ~130 kbps
Reply #2
Once again Guru, you grace us with your amazing ears! Thank you for your hard work.

Very interesting that overall, WMAPro9 does so well.  Considering the source of the test, it speaks volumes. People complain about DRM with WMA, but it seems to me there is an option to turn it off in the settings of the player.  Since down the road more portables will support the newer MS codec, this is important.

I also find it interesting you find the Nero encoder to be such a poor performer, considering how it has obtained "darling" status around here as of late. With so many people finding it to be great, it demonstrates how much can be ignored during casual listening. You listen to nero files while surfing the web or answering email, your brain is not on an artifact "hunt" 
you will make mp3's for compatibility reasons.

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Personal multiformat listening test at ~130 kbps
Reply #3
Quote
I also find it interesting you find the Nero encoder to be such a poor performer, considering how it has obtained "darling" status around here as of late. With so many people finding it to be great, it demonstrates how much can be ignored during casual listening. You listen to nero files while surfing the web or answering email, your brain is not on an artifact "hunt"  


If you read carefully, the results are here based on one particular music genre - classic music, and Nero still has some issues with tonality estimation (which are being worked out now) that directly affect classic music.

I am quite sure that if you include other genres that you will get different result
  • Last Edit: 11 October, 2003, 11:54:54 AM by Ivan Dimkovic

  • ScorLibran
  • [*][*][*][*][*]
  • Banned
Personal multiformat listening test at ~130 kbps
Reply #4
Fascinating results.  Especially interesting to me since you used the same Vorbis -q setting I use for normal listening (though classical is in the minority of my listening, and I use Post 1.0 CVS rather than GT3b1, so my nominal bitrate's a little lower).  Psy-models are different between the two versions, so I can't extrapolate too much to how Post 1.0 CVS could compare in this test, but these results are still important to know.

I'm very surprised to see the WMA-Pro results.  Not that you rated it the highest overall, but how dramatic of a difference you found between it and most of the other formats (only 0.28 between it and second place QT AAC, but 0.71 between it and tied-for-third Vorbis and MPC).  If/when WMA-Pro gets decent hardware support, it'll be a serious contender with this kind of performance (if it's anything close in other genre's as well, that is).

Thanks for the work you did on this, guruboolez!


Edit: Clarification.
  • Last Edit: 11 October, 2003, 12:04:57 PM by ScorLibran

  • JohnV
  • [*][*][*][*][*]
  • Developer
Personal multiformat listening test at ~130 kbps
Reply #5
Quote
I also find it interesting you find the Nero encoder to be such a poor performer, considering how it has obtained "darling" status around here as of late.
I think the "darling" status comes from the fact that Nero is currently one of the very few encoder which are rapidly evolving, and the developer (Ivan) is actively participating here.

What comes to Guru's results, they are really no surprise to me (and shouldn't be to Guruboolez either imo, he has sent quite a few harpsichord samples here). One of the biggest weaknesses of Nero AAC has been samples like harpsichord, pipes, and in general music with only a single very defined instrument which need very precise tonality estimation especially with VBR coding. Many of these samples have been posted to HA also, so the problem has been known. In that regard there's nothing new here.

Too bad for Ivan, only one very specific genre of music, like baroc, is Guru's favorite and is tested by him.
But, it indeed doesn't show the whole picture at all. 

The new development versions have already had very remarkable improvement in this area, so again too bad for Ivan that Guru decided to do his big test already.
Nero AAC is going through rapid development phases, and it is already better in this area of music than the test here indicates. I think Guru will notice this too when he gets the chance to test the new versions.
Juha Laaksonheimo

  • JohnV
  • [*][*][*][*][*]
  • Developer
Personal multiformat listening test at ~130 kbps
Reply #6
Another thing. With this kind of music CBR/ABR mode has the advantage, because VBR quality and bit allocation depends heavily on whether tonality estimation is working quite flawlessly or not (this is emphasized with the tested low-medium bitrate vbr). But CBR/ABR allocates pretty much always so much bits that possible deficiencies in tonality estimation don't appear so easily. It would have been interesting to see how wma9 pro does with VBR, would it still be on top? That's another reason why people shouldn't make too conclusive conclusions about this test. It's no surprise that Quicktime does good. It's excellent with CBR.

Obviously I'm not surprised that the top two codecs in this tests use CBR/ABR.

And I also wonder why Guru chose to use ABR with WMA, but VBR with Nero. I'm pretty sure that the tested Nero codec would have performed clearly better in CBR mode with several of those samples... Not to mention that Nero vbr streaming is rated 110-120kbps average (although because of not well working TE in the tested codec version, it surely is often much lower), while Vorbis for example gets over 128kbps nominal vbr mode, although this is not the deciding factor here.
  • Last Edit: 11 October, 2003, 04:02:26 PM by JohnV
Juha Laaksonheimo

  • mdmuir
  • [*][*][*]
  • Members (Donating)
Personal multiformat listening test at ~130 kbps
Reply #7
Maybe by the time I am ready to upgrade to Nero 6.0, the improvements Ivan is working on will be fully implemented. At this juncture, I am in no hurry to upgrade.
Having the new version of foobar2000 burn audio cds from just about any codec source, in dao mode, without worrying about a passle of plugins, has not inspired me to seek out Nero 6.0 yet  Maybe after Christmas or so.
you will make mp3's for compatibility reasons.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Personal multiformat listening test at ~130 kbps
Reply #8
Quote
It would have been interesting to see how wma9 pro does with vbr, would it still be on top? That's another reason why people shouldn't make too conclusive conclusions about this test.


18 samples encoded with WMA VBR-2 PASS 128 : 7,31 Mo (7 673 606 octets)
18 samples encoded with WMA VBR QUAL 90 : 7,32 Mo (7 677 178 octets)

Bitrate are fortunately very close with one of the few offered VBR setting of wma pro. A fair comparison is possible. I'll probably made another one the next week


Quote
And I also wonder why Guru chose to use ABR with WMA, but VBR with Nero. I'm pretty sure that the tested Nero codec would have performed clearly better in CBR mode with several of those samples...


I didn't choose any single VBR pass setting, for practical reason. I didn't have too much experience with WMA PRO. Therefore, it's hard for me to estimate average bitrate of the samples I progressively chose during the test. The only data I have are these :
http://membres.lycos.fr/guruboolez/AUDIO/a...assical_VBR.txt
http://membres.lycos.fr/guruboolez/AUDIO/a...etailed_VBR.txt

VBR Q90 is nice for something near 130 kbps. But for classical music only, if I remember correctly the discussions around the call for data of Roberto's for the test he launched some months ago. ABR 128 was the more universal setting. And the safest too.



I didn't chose ABR/CBR with Nero, because I had bad experiences with it, compared to VBR -streaming. Lowpassing, for exemple, is really noticeable with CBR 128. This opinion is based on the few tests I made to defend Roberto's choice of QuickTime against Ahead -streaming.

I've no scientific knowledge about audio encoding. I'm a good HA pupil : I learned than VBR is the best way to obtain a good quality. If I had opt for CBR with Nero, many people will certainly asked me why, by a polite or less polite way (ask Roberto : he have good experience now of this kinf of controversy )



If you want, I can perform a new test during this week, by comparing WMAPRO ABR/VBR and Nero AAC ABR/VBR. I probably have time for this.
  • Last Edit: 11 October, 2003, 04:17:51 PM by guruboolez

Personal multiformat listening test at ~130 kbps
Reply #9
 

I have my revenge...thanks to guruboolez.
Thank you very much guruboolez !   

One made fun of me because I used the WMA with the DRM...and now one finally proves  the superiority of the WMA.

God heard me: victory! 

WMA Rules.
That's it 

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Personal multiformat listening test at ~130 kbps
Reply #10
Quote
One made fun of me because I used the WMA with the DRM...and now one finally proves the superiority of the WMA


Well, on some other music genres WMA is not that superior    But for classical music it surely is not bad

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Personal multiformat listening test at ~130 kbps
Reply #11
Nick Jr III> Don't forget that WMA PRO is not traditionnal WMA. It's a different audio format.
WMA PRO is very competitive - WMA standard isn't. And WMA PRO is far from perfection. Try with sharp electronic music for exemple (death2.wav sample on Roberto's test was a good alarm). On a test based on this kind of music, I would bet that WMA PRO will be last, behind gogo mp3, on some interesting samples.

Nevertheless, you're right : WMA PRO need consideration. We can't admit primitive M$ bashing anymore.
  • Last Edit: 11 October, 2003, 05:05:03 PM by guruboolez

Personal multiformat listening test at ~130 kbps
Reply #12
Quote
Well, on some other music genres WMA is not that superior    But for classical music it surely is not bad





                                               
I would like to know for which type of music the WMA is worse?
isn't this rather a mistrust towards WMA ? 
  • Last Edit: 11 October, 2003, 05:17:06 PM by Nick Jr III

  • JohnV
  • [*][*][*][*][*]
  • Developer
Personal multiformat listening test at ~130 kbps
Reply #13
Quote
God heard me: victory! 

WMA Rules.
Right, here we have 1 test by 1 person with specific settings and quite homogenous samples. And people start this. 

I could set up a test which gives completely different results, except that I probably couldn't do much to save FAAC.. 
  • Last Edit: 11 October, 2003, 05:41:00 PM by JohnV
Juha Laaksonheimo

Personal multiformat listening test at ~130 kbps
Reply #14
Quote
Try with sharp electronic music for exemple (death2.wav sample on Roberto's test was a good alarm). On a test based on this kind of music, I would bet that WMA PRO will be last, behind gogo mp3, on some interesting samples.



too bad... 
  • Last Edit: 11 October, 2003, 05:27:48 PM by Nick Jr III

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Personal multiformat listening test at ~130 kbps
Reply #15
Quote
I would like to know for which type of music the WMA is worse?
isn't this rather a mistrust towards WMA ? 


WMA has problems with percussions, music with attacks and strong stereo separation, etc...

Personal multiformat listening test at ~130 kbps
Reply #16
Quote
Quote

I would like to know for which type of music the WMA is worse?
isn't this rather a mistrust towards WMA ? 


WMA has problems with percussions, music with attacks and strong stereo separation, etc...

thank you for this reply, Ivan.

Problems ? even with WMA Pro ? 

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Personal multiformat listening test at ~130 kbps
Reply #17
@128 - http://audio.ciara.us/test/128extension/results.html

Samples: 41_30sec, Death2, waiting

@64 - http://audio.ciara.us/test/64test/results.html

Samples: experiencia, gone, mybloodrusts, polonaise, riteofspring, waiting

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Personal multiformat listening test at ~130 kbps
Reply #18
During the 64 kbps listening test of Roberto, WMA9 standard was used, not WMA PRO. This last format is really better at low bitrate than common WMA (in VBR mode, only available for low bitrate). I'm not sure that we can extrapolate results from one format to another.

P.S. I forgot to upload some samples last night. Is someone interested by the 18 clips ? 35 MB.

Personal multiformat listening test at ~130 kbps
Reply #19
Quote
@128 - http://audio.ciara.us/test/128extension/results.html

Samples: 41_30sec, Death2, waiting

Thank you Ivan ! 

perhaps WMA 10 could... 

regards,
Nick
  • Last Edit: 12 October, 2003, 02:53:04 PM by Nick Jr III

  • Joe Bloggs
  • [*][*][*][*]
Personal multiformat listening test at ~130 kbps
Reply #20
In the meantime, Microsoft is still teh evil™.

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
Personal multiformat listening test at ~130 kbps
Reply #21
Quote
Quote
I also find it interesting you find the Nero encoder to be such a poor performer, considering how it has obtained "darling" status around here as of late.
I think the "darling" status comes from the fact that Nero is currently one of the very few encoder which are rapidly evolving, and the developer (Ivan) is actively participating here.

I don't know about other persons, but I like it because it gives (IMHO) clearly the best results for rock, metal and electronic music. WMA does quite bad here. in comparison. So I guess the music genre matters a lot, and I'm not a 'classical' person

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Personal multiformat listening test at ~130 kbps
Reply #22
I finally had time enough for a specific WMA comparison: WMA PRO ABR against WMA PRO VBR. Size are near the same (see above).
Some difference heard on ABA were not significant on ABX direct comparison between two encodings.
Nevertheless, slight but significative differences appeared :
- additional noise with VBR
+ reduced distortions with VBR
+ less artifact on specific case (drum roll) with VBR

http://membres.lycos.fr/guruboolez/AUDIO/t...WMA_ABR_VBR.htm
  • Last Edit: 13 October, 2003, 07:47:29 AM by guruboolez

  • JohnV
  • [*][*][*][*][*]
  • Developer
Personal multiformat listening test at ~130 kbps
Reply #23
Quote
I finally had time enough for a specific WMA comparison: WMA PRO ABR against WMA PRO VBR. Size are near the same (see above).
Some difference heard on ABA were not significant on ABX direct comparison between two encodings.
Nevertheless, slight but significative differences appeared :
- additional noise with VBR
+ reduced distortions with VBR
+ less artifact on specific case (drum roll) with VBR

http://membres.lycos.fr/guruboolez/AUDIO/t...WMA_ABR_VBR.htm

WMA 9pro q90 gives me quite high bitrate on average, with other than classical baroc music. I'm getting pretty high bitrate on average, like closer to 170kbps. Did you choose q90 based only on the bitrates from the baroc music? If so, you probably gave it relatively quite high quality level...
Juha Laaksonheimo

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Personal multiformat listening test at ~130 kbps
Reply #24
Yes. WMA9 PRO VBR Q90 is bitrate friendly with classical. I had, of course, to encode many discs in order to have a more precise idea of average bitrate with classical, in general (not only baroque instruments).

I know that with other musical genre, Q90 gives higher bitrate. That's mainly why I choose ABR 128 for the whole listening test: no bitrate deviation. Roberto made the same choice for his 128 public test: Q75 is closer to 128 kbps with most music, but too low with classical samples.



PS : I forgot to say that the WMA test was made with another soundcard and another headphone (Terratec & Beyer, and not AC & Philips).
  • Last Edit: 13 October, 2003, 08:44:20 AM by guruboolez