Skip to main content

Topic: Ahead AAC VBR vs Vorbis aoTuV beta 3 (Read 48655 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
The excellent performances of newer Nero AAC encoder (2.9.998 “fast”) lead me to compare it with other strong competitors. Unfortunately, there are not so many potential challengers.

  • lame MP3: I’m using it every day, and though I like his new VBR mode (-V 5), I’m sure it’s not strong enough with my music to fully compete with the new AAC VBR encoder.

  • wma9 PRO: I’m fond of this encoder, working very well with classical music. I don’t know how perform the newest 9.1 encoder. Unfortunately, two years after the first release of WMA9PRO, there’s still apparently no dedicated portable player supporting it (except maybe telephone or PDA). The industry is still manufacturing electronic components and players for standard WMA, which is not a very competitive format.

  • ogg vorbis: since Aoyumi works, vorbis improved a lot (last year, with classical music, quality was behind lame ABR for my taste). aoTuV encoders (and now official 1.1) have drastically reduced the hiss/coarseness problems, which annoys me a lot with classical music (maybe other genres are affected too, I can’t really say). Unfortunately, the problem is only reduced, and not fully corrected. Nevertheless, I consider vorbis as one of the most robust format. Latest aoTuV beta 3, announcing an even lower coarseness level and a better pre-echo performance, could be a potential champion. I’d personally consider it a priori as the most dangerous challenger for strong AAC encoders.



I didn’t include wma9PRO in the following test. I don’t know that much this encoder: Is there one VBR profile close to ~130 kbps? Which one? Is it reliable, or is CBR/VBR2-pass a better solution? Answers are needed before including this format in a comparison test, and I don’t have time enough to answer them by myself.

Therefore, I’ve restricted the comparison to vorbis and Nero AAC. As encoder, I had some choice (official 1.1, megamix, aoTuV beta 2/3). I’ve privileged aoTuV encoder, because Aoyumi’s works are mainly focused on the most important vorbis problem (coarseness). I hesitated a moment between beta 2 and beta 3. I’ve finally opted for beta 3 without preliminary investigations: this test is a good occasion to see how reliable the new beta is, and if potential problems will handicap the format during this test, they will at last be detected, and probably quickly corrected.
I don’t know which vorbis setting would be the fairest to compete with Nero –internet. –q 4,00 is the roundest, and probably one of the most popular. In order to prevent (or at least to limit) some excessive difference, I’ve used –q 4,25. With my computer, I can’t obtain quickly a full and reliable bitrate table. But the comparison based on the tested samples is very satisfying:
- Nero encodings: 136,2 kbps
- aoTuV –q 4,25 encodings: 136,5 kbps
…hard to find closest settings :·)


The testing conditions (hardware, software, methodology…) are the same as those described in the global AAC comparison. Samples are also the same. One difference: I didn’t searched to ABX the difference between the encoded files and the reference ones (already done with all AAC encodings in previous test), but focused my efforts to a direct comparison between the two encoded files. Again, I’ve fixed a dual limit to the number of trials: eight if easy, sixteen trials if less easy. I have respected these limits, without any exception. First second of each sample was systematically discarded from listening.
Now, results:



vorbis aoTuV beta 3: Nice quality, but still coarseness issue. It’s fortunately limited, and with some samples, I simply can’t detect it (problem was systematically audible with older vorbis encoder). Very limited pre-echo problems: Aoyumi has apparently solved most issues in this area. However there is a deep ringing artefact with the organ (Bruhns) sample. I’ve in addition noticed a very strange problem, occurring with solo violin on two different samples (Avison first, and then Mozart-Sonata): it sounded like a noise-reduction processed recording, but additional hiss was audible on the same time. This paradox also happened to the old experimental Quantum Knot’s encoder (v. 3.5 aka vorbis “classical edition”). I don’t know: maybe Aoyumi did similar tuning to aoTuV beta 3? I had to investigate further…

compared to Nero AAC: better with harpsichord (note that aoTuV allocates even more bits for this instrument than Nero AAC!) on Couperin and Bach (excellent encoding) samples. I’m badly surprised by poor performances on Bayle sample: I’d expect from a VBR encoder to detect the difficulty (micro-attack). With aoTuV, both bitrate and quality are low… Noise/hiss/coarseness are like a signature with most samples, and often betray vorbis presence.


STATISTICS

You could try with the following table:
Code: [Select]
NERO    AOTUV
4,0    2,0
4,5    4,5
5,0    3,5
4,5    3,5
4,5    1,5
5,0    5,0
3,5    2,5
3,5    4,5
4,0    4,5
4,5    4,0
4,0    1,5
4,0    3,0
3,0    3,5
3,0    3,5
3,0    2,0


Conclusions are the same with ANOVA and Tukey parametric:

« NERO is better than AOTUV ».


• ABX logs are available here
• SAMPLES (11 MB) are available here. Keep them: I can’t keep them very long time online.
  • Last Edit: 29 December, 2005, 04:20:14 PM by guruboolez

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #1
In addition to this test, I’ve compared the performance of aoTuV beta3 to the previous beta 2 on two problem samples: Avison and Mozart-Sonata. To complete the investigation, I’ve also add QK 3.5 “classical edition” at –q 3,4 (this encoder need to lower the –q value for matching bitrate values).

Here are the results:

Code: [Select]
ABC/HR Version 0.9b, 30 August 2002
Testname:

1R = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - quantum knot classical edition q 3,4.wav
2L = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - aoTuV beta 3 q 4,25.wav
3R = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - aoTuV beta 2 q 4,25.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - quantum knot classical edition q 3,4.wav
1R Rating: 1.0
1R Comment: Highly distorted violin. Hollow and empty.
---------------------------------------
2L File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - aoTuV beta 3 q 4,25.wav
2L Rating: 2.0
2L Comment: sound is hollow, like previous (1R) file, but impact is a bit lowered.
---------------------------------------
3R File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - aoTuV beta 2 q 4,25.wav
3R Rating: 3.5
3R Comment: a bit distorted.
---------------------------------------
ABX Results:


Code: [Select]
ABC/HR Version 0.9b, 30 August 2002
Testname:

1L = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - aoTuV beta 3 q 4,25.wav
2L = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - quantum knot classical edition q 3,4.wav
3R = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - aoTuV beta 2 q 4,25.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - aoTuV beta 3 q 4,25.wav
1L Rating: 2.0
1L Comment: Empty sound. Musical matter is missing.
---------------------------------------
2L File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - quantum knot classical edition q 3,4.wav
2L Rating: 1.0
2L Comment: clearly the most affected by this hollow sounding.
---------------------------------------
3R File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - aoTuV beta 2 q 4,25.wav
3R Rating: 2.5
3R Comment: very flat violin. Slightly better than 1L.
---------------------------------------
ABX Results:


Conclusion: aoTuV beta3 introduces similar artefacts on these two samples when compared to beta 2. The experimental “classical edition” have similar issue, but at higher level. Of course, it doesn’t necessary mean that beta 2 would be a better competitor than beta 3 against Nero AAC, but just that recent changes to aoTuV encoder have some negative consequences. I’ve already experimented regression of coarseness with beta 3 compared to beta 2.

  • JohnV
  • [*][*][*][*][*]
  • Developer
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #2
Quote
The excellent performances of newer Nero AAC encoder (2.9.998 “fast”) lead me to compare it with other strong competitors.

The fast mode uses so called Scalefactor estimation mode, which hopefully will become the default very high quality mode after it has been refined.
After SFE and some other tweaks are working at full power, well, the LC-core should be extremely good, that is our goal.. 
  • Last Edit: 09 December, 2004, 05:14:24 AM by JohnV
Juha Laaksonheimo

  • john33
  • [*][*][*][*][*]
  • Developer
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #3
Juha Laaksonheimo alias JohnV
Ahead Engineer


Congratulations, JohnV, HA seems to have been a very fertile recruiting ground for Ahead!!!
John
----------------------------------------------------------------
My compiles and utilities are at http://www.rarewares.org/

  • QuantumKnot
  • [*][*][*][*][*]
  • Developer
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #4
Again, thanks for the insightful tests, guruboolez.  They give Vorbis developers the perfect 'reality check' on their work.

It seems classical music continues to be a weak point for Vorbis.  *sigh*

Just as a sidenote, I found some free-time this week playing with some wavelet-based transient analysis (for block switching and dynamic impulse adjustment) for Vorbis 1.1.  Nothing concrete to release yet, but the results look quite interesting.  Expect an alpha of QKTune 2 sometime in the new year. 
  • Last Edit: 09 December, 2004, 07:22:39 AM by QuantumKnot

  • Aoyumi
  • [*][*][*]
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #5
@guruboolez,
Thank you for a test. I checked the clear problem with some samples.
I already recognize the cause. And I will release an experimental version in the near future, in order to solve or reduce this problem.
  • Last Edit: 09 December, 2004, 07:56:04 AM by Aoyumi

  • HotshotGG
  • [*][*][*][*][*]
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #6
Quote
Just as a sidenote, I found some free-time this week playing with some wavelet-based transient analysis (for block switching and dynamic impulse adjustment) for Vorbis 1.1.


Care to eleborate? ;-D
budding I.T professional

  • Redmond
  • [*]
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #7
"I didn’t include wma9PRO in the following test. I don’t know that much this encoder: Is there one VBR profile close to ~130 kbps? Which one? Is it reliable, or is CBR/VBR2-pass a better solution? Answers are needed before including this format in a comparison test, and I don’t have time enough to answer them by myself."

1P CBR, 2P CBR, and 2P VBR have 128kbps modes. For this test, you should use either 2P CBR or 2P VBR.
  • Last Edit: 09 December, 2004, 12:06:47 PM by Redmond

  • rjamorim
  • [*][*][*][*][*]
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #8
Very interesting test. Thanks Guru.

Now we need a test comparing all musical styles. Who will be brave enough to stand up and conduce such test? (anyone who answers rjamorim will get socked in the eyes by yours truly  )
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #9
Quote
Very interesting test. Thanks Guru.

Now we need a test comparing all musical styles. Who will be brave enough to stand up and conduce such test? (anyone who answers rjamorim will get socked in the eyes by yours truly  )
[a href="index.php?act=findpost&pid=258744"][{POST_SNAPBACK}][/a]


Roberto Amorim should do it. (I did NOT say "rjamorim"! )

Nah, joking... I would love to, but I don't have that much experience.

  • Dologan
  • [*][*][*][*]
  • Members (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #10
Quote
Quote
The excellent performances of newer Nero AAC encoder (2.9.998 “fast”) lead me to compare it with other strong competitors.

The fast mode uses so called Scalefactor estimation mode, which hopefully will become the default very high quality mode after it has been refined.
After SFE and some other tweaks are working at full power, well, the LC-core should be extremely good, that is our goal.. 
[a href="index.php?act=findpost&pid=258676"][{POST_SNAPBACK}][/a]

Well, unrefined as it may be, it seems its quality is already higher than high. 

  • JohnV
  • [*][*][*][*][*]
  • Developer
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #11
Quote
Quote
Quote
The excellent performances of newer Nero AAC encoder (2.9.998 “fast”) lead me to compare it with other strong competitors.

The fast mode uses so called Scalefactor estimation mode, which hopefully will become the default very high quality mode after it has been refined.
After SFE and some other tweaks are working at full power, well, the LC-core should be extremely good, that is our goal.. 
[a href="index.php?act=findpost&pid=258676"][{POST_SNAPBACK}][/a]

Well, unrefined as it may be, it seems its quality is already higher than high. 
[a href="index.php?act=findpost&pid=258774"][{POST_SNAPBACK}][/a]

There are still issues which Guru's test don't reveal, that's why it's not default yet.
Juha Laaksonheimo

  • QuantumKnot
  • [*][*][*][*][*]
  • Developer
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #12
Quote
Quote
Just as a sidenote, I found some free-time this week playing with some wavelet-based transient analysis (for block switching and dynamic impulse adjustment) for Vorbis 1.1.


Care to eleborate? ;-D
[a href="index.php?act=findpost&pid=258703"][{POST_SNAPBACK}][/a]


Still early times but the results are interesting.  That's all I'll say for now.  Note that I am not working on a quasi-Vorbis II.  It is 100% pure Vorbis I.

btw. who else at HA has the experience (and time) to hold a listening test?

  • bond
  • [*][*][*][*][*]
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #13
Quote
After SFE and some other tweaks are working at full power, well, the LC-core should be extremely good, that is our goal..

great stuff, neros aac encoder really needed this
I know, that I know nothing (Socrates)

  • DreamTactix291
  • [*][*][*][*][*]
  • Members (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #14
Nero's AAC did really really well here.  This was also a good test for aoTuV b3 as it has for the most part gone untested so far (but it is reall really new of course).  I'm actually really impressed with Nero's AAC here.
Nero AAC 1.5.1.0: -q0.45

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #15
Quote
There are still issues which Guru's test don't reveal, that's why it's not default yet.
[a href="index.php?act=findpost&pid=258778"][{POST_SNAPBACK}][/a]


I've just discovered the great performance of newer Nero AAC. I've encoded some additionnal tracks in order to perceive additional issues. Is it worth to send samples (to Garf, Ivan, Menno or you)? Or is it better (for you) to wait for the final release?

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #16
Hello,

First of all, guruboolez, many thanks for your tests - as always they are extremely useful to all of us developing codecs (please send any clips that you think that might be useful to us)

Also, I would like to clear out some unknowns about the "fast" mode -

"Fast" mode is using direct estimation of the scale-factor values - and it turned out that this method gives better results than traditional "two pass iteration loop" algorithm found in older generation of encoders.

This mode is still "beta" as many improvements are currently being done to both LC and SBR algorithms that should, we hope, result in significant improvement in quality of our codec.

I am also sorry for delay of the planned "V3.0" release of Nero AAC encoder - as we are all extremenly busy with the Nero Digital development - I can't say much now, but we are planning couple of big surprises that will be announced soon

Once again, thank you all for the big support!
  • Last Edit: 10 December, 2004, 07:19:07 AM by Ivan Dimkovic

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #17
Hello Ivan,
first: BRAAAVOOO! You (and other people) did an impressing job with Nero AAC. Honestly, I didn't think that such progress could be possible in less than one year. And thanks for the explanation.

second: here is a sample, probably the worse I heard for Nero VBR 'fast' internet profile with regular music. The artifact is very common (noise is missing and fluctuant), and occurs mainly on the second part of the sample. The problem is audible on regular listening volume. I've found some samples showing this behaviour, but this one is the worse I've heard.

P.S. There's also (small but real) problems with pre-echo. But you probably know this.
  • Last Edit: 10 December, 2004, 07:53:43 AM by guruboolez

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #18
Quote
Just as a sidenote, I found some free-time this week playing with some wavelet-based transient analysis (for block switching and dynamic impulse adjustment) for Vorbis 1.1.  Nothing concrete to release yet, but the results look quite interesting.  Expect an alpha of QKTune 2 sometime in the new year. 
[a href="index.php?act=findpost&pid=258691"][{POST_SNAPBACK}][/a]

If you need some help for tests or opinions about it, count on me (if I'm not too busy).

  • QuantumKnot
  • [*][*][*][*][*]
  • Developer
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #19
Quote
If you need some help for tests or opinions about it, count on me (if I'm not too busy).
[a href="index.php?act=findpost&pid=258900"][{POST_SNAPBACK}][/a]


Many thanks.

  • alter4
  • [*][*][*]
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #20
I made some ABX test with guruboolez's samples compressed  ~128 bitrate and...
for my ears NERO AAC not better than Aotuv beta3 аnd can even worse??
tested on Avison,  Mozart  - sonate and Bruhns - prelude.

Avison  AAC        8/10
Avison Ao Beta3  7/10

Mozart  AAC        7/10
Mozart Ao Beta3    7/10

Bruhns    AAС      6/10
Bruhns  Ao Beta3  7/10

I thank guruboolez for interesting sample.
 
  • Last Edit: 13 December, 2004, 05:00:21 AM by alter4

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #21
Strange results for Bruhns.wav: it's one of the most wounded IMO by aoTuV.
Anyway, thanks for additionnal tests
  • Last Edit: 13 December, 2004, 04:59:36 AM by guruboolez

  • alter4
  • [*][*][*]
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #22
Quote
Strange results for Bruhns.wav: it's one of the most wounded IMO by aoTuV.

sorry, typing error
I have corrected
  • Last Edit: 13 December, 2004, 05:29:11 AM by alter4

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #23
I've just noticed:
results below 8/10 are not significant.
8/10 -> pval = 0.055
7/10 -< pval = 0.172

http://ff123.net/abx/abx.php

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Ahead AAC VBR vs Vorbis aoTuV beta 3
Reply #24
Quote
I've just noticed:
results below 8/10 are not significant.
8/10 -> pval = 0.055
7/10 -< pval = 0.172

http://ff123.net/abx/abx.php
[a href="index.php?act=findpost&pid=259393"][{POST_SNAPBACK}][/a]


Yeah, I would only choose 8 trials if you're reasonably sure that you'll be able to detect a difference.

ff123