HydrogenAudio

Lossy Audio Compression => Ogg Vorbis => Ogg Vorbis - Tech => Topic started by: guruboolez on 2004-02-05 08:10:22

Title: Vorbis quality – wrong direction?
Post by: guruboolez on 2004-02-05 08:10:22
I promised to Quantum Knot to test his new tuning. I finally found the time, and tested QK2, 1.01 - compared to RC3 library. Yes, the two-years old RC3.


I. CHALLENGERS

The challengers are of course questionable. I had to explain my choices.
First, I didn't include Nyaochi and Aoyumi tunings. I really apologize to both persons. It might appear as disrespectful, but in my opinion, comparing different tunings in alpha stage each others is not something fair. I guess that alpha tunings are released for report, not for competition. Comparison will probably follow later, when all tunings will be advanced enough.

Second thing: why old vorbis encoders?
I've tested vorbis 1.00 and 1.01 many times with natural instruments, and it always sounded approximative and disapointing, generally worse than lame mp3. To my ears, there are serious problems: not only hiss or high frequencies boost (reported many times by other users), but imprecise or coarse sound, aggressive rendering. With classical music, or natural instruments well recorded, all these problems are clearly audible – probably more than with loud and (over-)compressed music.

These problems are like a vorbis “signature” (in blind test, it’s easy to find vorbis among different challengers). I never heard something comparable with other formats. But I’m sure that these flaws are not consubstantial to vorbis format, but that they are a consequence not only of a lack of tuning, but from contestable choices made in the past. I'm still convinced that something wrong happened between RC3 and final "1.00" version from July 2002, though I never really tested this difference. My suspicions are based on the following things.


Important changes were made between RC3 and Final library. This was particularly noticeable on encoding speed (final is much faster), and on sound quality at low bitrate, even for untrained people. Before “final” encoder, Vorbis wasn’t very competitive at low bitrate (especially the popular and symbolic 64 kbps). The reference two years ago was mp3pro, and vorbis was clearly behind. At low bitrate, Vorbis sounded like other non-SBR format (wma, real, aac): metallic, heavily distorted, etc… But final encoder sounded totally different: not metallic anymore, less distorted, but simply noisier (and some stereo issues). For people used to find traditional flaws at this bitrate, vorbis was simply amazing. With some habits, noise and stereo problems would be more noticeable, but even here, vorbis is a very good solution at 64 kbps (see Roberto’s recent test) compared to all present non-SBR format. Noise and stereo reduction are generally more acceptable and less ugly than metallic coloration a la wma or DivX audio...
But this incontestable victory (for an open-source and non-patented project) had some bad reverse. In my opinion, pre-final encoders were more transparent at mid-bitrate[/b] (especially 128 kbps or –q4 setting), at least on non-killer samples (90% or 99.9% of [my?] music, I can’t say) and at least with natural instruments recorded on hi-fi principles. At this bitrate, encodings are now sharing the same flaw as low bitrate: hiss, imprecise rendering… On cool samples, where even mp3 sounds flawlessly, vorbis has audible problems. I suspect that the secret of vorbis good quality at low bitrate is the cause of wrong things that are happening above.



I never seriously tried to confirm or infirm my suspicions with blind test. I was more interested by high bitrate & transparent encodings, and because of pre-echo serious problems, vorbis wasn’t really interesting (except with Garf tuning). Now than Vorbis begins to infiltrate industrial manufacturers, the audio format looks for me more interesting at lower bitrate (I expect from modern format a good quality at 130-150 kbps]. Testing Quantum Knot tuning encoder appears like a good opportunity for me to oppose post-1.00 encoders to old RC3 library, in order to see if my suspicions are justified or simple rubbish.


I looked on old CD-R, and founded three RC3 builds, dated on: February 2002, March 2002 and April 2002. I was tempted to test two different builds, and not only one RC3 encoder. Why? In my souvenirs, a vorbis developer talked about changes happened within RC3 (can't remember or understand what - maybe something related to stereo model). I think it happened in springs (April or May). February and March encoders produce same output result (I did a bit to bit comparison, and files were the same, except tiny difference on the first samples). April encoder is a different beast. Encoding speed had improved (twice faster, if not more!), and output is different too. If problems occur with final library, the cause is maybe in this crucial moment of vorbis history. I'm maybe completely wrong in my suspicions, and maybe RC3 sounds worse than 1.00 encoder. The best thing to be sure about it is to test the different encoders.


The following test confronts four vorbis encoders:
• 2002.03.07
• 2002.04.06
• Official 1.01 oggenc – JohnV compile (2003.09.09)
• GordianKnot tuning QK2 (January 2004, based on 1.01)

Setting for all encoder is the same: -q4 (VBR, 128 kbps nominal)

II. SAMPLES

In order to make the 1.01 & QK2 comparison (primary goal of the test) useful, I’ve selected sample with transients. I didn’t include well-known killer samples: positive reports for QK2 were done with this kind of samples, and for 128 kbps encodings, I guess that common musical samples are difficult enough for all encoders and more representative of real usage too. All samples are coming from my own library. I’ve deliberately chosen very short ones, so upload will be easier.

Short description:
harpsichord.wav: solo harpsichord, maybe too reverberated but very sharp and nice recording. Encoders usually suffer from pre-echo and from heavy distortions
erhu10.wav: erhu (Chinese string instrument) with percussive instruments in accompaniment.
Arche I.wav: orchestral extract from Penderecki first Symphony. Very sharp and loud attack, followed by something like a rattle [not Simon]
Mandolins.wav: extract from a famous Vivaldi concerto. This sample (reduced to 5 seconds) is one of my favorite, because many encoders failed to encode this properly at ~130 kbps.
Transfiguration.wav: part of an orchestral work of Olivier Messiaen. Two different problems should occur: distortions with cymbals (and with Vorbis 1.00 family: flattened noisy sound) and pre-echo/blurred brass instruments.
La Spagna.wav: percussions & wind instruments from Renaissance playing concertedly. Pre-echo reduction is expected from GK2, and maybe noise problems may occurs with wind instruments.
Mars.wav: beginning of the first Planet[/U], from Gustav Holst. Naturally quiet (violin pizzicatos and threatening winds, this sample is very quiet due to weird mastering of the CD layer [rip from a SACD]. This sample is comparable to orchestral lace, and I listened it without harm at very high volume in order to magnify all possible problems (pre-echo, ATH issues, background changes...)
Brahms6.wav: piano is a percussive instruments, potentially affected by pre-echo. This recording from the 6th Hungarian Dance of Johannes Brahms is really sharp and well-recorded.


In other words, the samples tested are probably not favorable to pre-Final libraries. For measuring vorbis noise problems at ~128 kbps, tonal sample are more interesting. But even here, interesting things happened during the test.



III. RESULTS


(http://audiotests.free.fr/tests/2004/vorbisoldnew.png)

QK2 modifications have a positive effect on 5 samples. The benefits of this encoder concern pre-echo and sharpness only. On low-volume transients, GK2 is inoperative (Mars.wav). On sharp and detailed micro-attacks (Arche I.wav second part, and Transfiguration.wav), QK2 sounds identical to official 1.01. But when attacks are clearly defined (Arche I.wav first part and Harpsichord.wav), the difference is appreciable. Benefits are audible too when transients are not excessively strong (piano, percussions on La Spagna.wav), but difference is then mildly audible. I’m disappointed by the few differences with Mandolins.wav, and surprised by the identical scores I obtained with Erhu10.wav.
Vorbis library needs tweaking for mid bitrate encodings, and QK2 is a good answer for the pre-echo problem.

• The official post final “1.01” encoder sounded worse than RC3 libraries to my ears on 7 samples. Harpsichord is the only exception, with less distortion (vibrating effect). But most often, the noise issues (louder, unstable, affecting micro-dynamic of some instruments and definition/contours of others) was more discriminating than pre-echo variations. With Erhu10.wav, Transfiguration.wav and Arche I.wav (second part), difference between RC3 family and 1.01 is really big (whereas progress made with QK2 from 1.01 had a much more limited impact).
Interesting thing to note: official 1.01 never reached the 3.0 (= “slightly annoying”) notation. Notation is of course a very imprecise thing, but this ranking isn’t totally meaningless. 1.01 produces non transparent and barely acceptable sound with natural instruments, whereas the RC3 branch, as all different audio format, is able to reach near-transparency with non-difficult samples. To the 8 samples of this test I could add the results of the ~30 other samples I’ve tested in the past with vorbis: final vorbis results were each time around 3/5. In other words, the changes introduced with 1.00 are negative at –q 4 with a consequent part of the editorial production (at least for my ears).

• Other thing to note: there wasn’t any rupture between March 2002 [m2k2] and April 2002 [a2k2]. Library from m2k2 is not the crystal clear one, and a2k2 the Hoover© sounding library. M2k2 RC3 was better than a2k2 four time; a2k2 was better (but by a slightly margin) the four other time. Nevertheless, if April 2002 wasn’t an historic moment for Vorbis (except for encoding speed), it was surely the beginning of tendency: noise increased with a2k2 and reached an alarming level. The accident occurred few months later…
Other point: pre-echo issues were reduced with a2k2 library (compared to the previous one). But comparing RC3s to Finals libraries on pre-echo is not easy. Pre-echo is sometimes lower with RC3 (a2k2 or m2k2) : erhu10, Arche I, Mandolins and Brahms - and sometimes higher than with 1.01 output : Spagna, Holst.


IV. CONCLUSIONS


Hard to conclude anything with only eight samples. I nevertheless reached the following one: my suspicions weren’t unfounded, and there’s obviously something rotten in 1.00 – and following libraries. I haven’t test Vorbis with non-classical/over-compressed/electronic music, and therefore I can’t evaluate the possible benefits of 1.00 modifications. But with classical (or more generally, natural instruments recorded with fidelity principles), these changes have clearly a negative effect – with some local exceptions (like harpsichord).

I guess that without these changes, Vorbis will sound poorly at low bitrate (a quick try with RC3 at 64 kbps convinced me that 1.00 is much better). But isn’t RC3 a better basis for mid-bitrate tuning than 1.00? Or is it possible to disconnect Vorbis mid settings from low settings, in order to avoid all characteristic introduced by “final” version to contaminate all encoding area (at least, up to 5.99]?


Note: the eight samples (FLAC – 4.5 MB] are available on the uploading forum – or will follow very soon.
Title: Vorbis quality – wrong direction?
Post by: westgroveg on 2004-02-05 08:32:17
I think you mean QuantumKnot. This (http://gordianknot.sourceforge.net/) is Gordian Knot.
Title: Vorbis quality – wrong direction?
Post by: guruboolez on 2004-02-05 08:32:27
Files are uploaded.
I've just realized that I forgot to post the exact results.


EDIT > replaced Gordian Knot by Quantum Knot    Thanks
EDIT2 > replaced GK2 by QK2 
Title: Vorbis quality – wrong direction?
Post by: PoisonDan on 2004-02-05 12:16:07
Very interesting test, as usual.

Thank you, guruboolez.
Title: Vorbis quality – wrong direction?
Post by: PoisonDan on 2004-02-05 12:36:02
I'm wondering: now that we have several different encoders (RC3, 1.0.1, GT3b1, QK2, aoTuV), it may be worthwhile to perform a full listening test comparing these encoders.

The MP3 encoder listening test has recently been completed, the AAC encoder test will soon begin, so it may make sense to do a Vorbis encoder test as well.

It probably won't be a good idea right now, since some of the tunings are still a work in progress, but I think it could be something to consider later this year.

I wonder if Roberto would be interested in setting up this listening test...

Edit: Technically, they're not really different "encoders" of course, just differently tuned revisions of the Vorbis encoder. So the terminology I used is not 100% correct.
Title: Vorbis quality – wrong direction?
Post by: MGuti on 2004-02-05 12:36:56
what bitrate was the test run at? and i am also running a test and have noticed similar problems with vorbis. i would like to try an RC3 test and if it yields better results, it would be nice to have someone investigate the changes (Quantum, if you could).

im sure it would be a nice addition to the QK-tune.

good job with the test.
Title: Vorbis quality – wrong direction?
Post by: guruboolez on 2004-02-05 12:42:50
I maybe forgot to precise: all tests were performed with -q4 setting (128 kbps nominal).
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-06 00:33:34
Many thanks to guruboolez for being able to perform this detailed listening test.  It is essentially the feedback I was hoping for and it has brought about some fairly interesting points, esp. about RC3, which I haven't played with before.

I'll download the RC3 sources and have a look at the changes that were made.
Title: Vorbis quality – wrong direction?
Post by: guruboolez on 2004-02-06 07:03:15
(http://audiotests.free.fr/tests/2004/vorbisoldnew2.png)

uncoupled -q4 was tested here. Bitrate is higher, but problem is considerably lowered (few hiss).
I did this test yesterday. Very quickly: I was very tired. I didn't test lower setting, except with few sample.
Apparently, coupling is THE problem.
Title: Vorbis quality – wrong direction?
Post by: ChangFest on 2004-02-06 18:03:25
guruboolez, I'm somewhat confused as to what your graphs are explaining.  Considering I'm somewhat of a n00b, could you or someone else explain the graphic results for me?
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-07 00:50:12
Quote
uncoupled -q4 was tested here. Bitrate is higher, but problem is considerably lowered (few hiss).
I did this test yesterday. Very quickly: I was very tired. I didn't test lower setting, except with few sample.
Apparently, coupling is THE problem.

Was this the uncoupled one I posted or the one nyaochi posted?  His one uses uncoupled stereo while mine used lossless stereo coupling (though at the time, I was under the impression it was uncoupled).

Looks like this hiss problem is affected by various factors, and stereo coupling is one of them.

On the HF boost front, Segher on vorbis-dev believes he knows a way of fixing this problem, which he has done in Lame.  Let us wait for the good news.

EDIT:  Gosh, I had a listen of "Arche I" in one of guruboolez' samples and Vorbis has a tonality problem on the first attack. 
Title: Vorbis quality – wrong direction?
Post by: [proxima] on 2004-02-07 14:56:16
This morning i’ve quickly done the same test and can confirm what guruboolez has noticed.
It seems that channel coupling is the problem, moreover with totally uncoupled stereo mode the results are even more better. Don’t give too much attention to average, the scores are really significant only if compared each other.
Here is the table of results:
http://www.hydrogenaudio.org/forums/index....ic=18360&st=0&# (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360&st=0&#)

[span style='font-size:8pt;line-height:100%']edit: i've problems uploading with with my ftp.[/span]
Title: Vorbis quality – wrong direction?
Post by: Sebastian Mares on 2004-02-07 16:04:27
Quote
guruboolez, I'm somewhat confused as to what your graphs are explaining.  Considering I'm somewhat of a n00b, could you or someone else explain the graphic results for me?

The numbers in the table show the score. Green cells indicate that the specified encoder won, red cells indicate the opposite.

The names you see on the left are samples (pieces of music), the names at the top are the encoders.
Title: Vorbis quality – wrong direction?
Post by: guruboolez on 2004-02-10 14:52:33
Some clarifications. I've used QK vorbis_uc for my tests. I didn't test Nyaochi uncoupled recently. Is it the encoder known as "ModestTuning"? The same encoder released in december? [proxima] notations are really interesting - and for me, surprising. I've tested "Modest Tuning" when it was released, and though I heard real progress in noise reduction, I wasn't totally impressed. Was -q4 used for comparison? I've tested it at -q2 or -q3 (can't remember), following Nyaochi recommandations for a fair comparison.

Anyway, I'd like to thank you, [proxima], for your tests (and similar results)
Title: Vorbis quality – wrong direction?
Post by: nyaochi on 2004-02-10 16:03:00
Wow!  I also find [proxima]'s listening result very interesting. If QuantumKnot and I change stereo mode correctly, the result shows lossless stereo is worse than uncoupled (dual) stereo. Where does the difference come from? Parameter difference in setup_44.h and setup_44u.h?

My uncooupled (dual stereo) oggenc is available in this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=18263). It is not my "Modest Tuning". 
Title: Vorbis quality – wrong direction?
Post by: guruboolez on 2004-02-10 16:13:58
nyaochi> thanks for the clarification. I missed this encoder you provided some days ago. I'll test it as soon as I can.

Question: what about bitrate? Is it possible to lower the bitrate in order to have something similar between uncoupled and coupled stereo? Or does coupling save a lot of bits?
Title: Vorbis quality – wrong direction?
Post by: nyaochi on 2004-02-10 16:38:39
Quote
Is it possible to lower the bitrate in order to have something similar between uncoupled and coupled stereo?

Unfortunately or fortunately, no. We must lower the quality of uncoupled one in order to get similar bitrates. Channel coupling actually saves alot of bits (with some expense of quality according to the listening results?)

Uncoupled stereo mode requires [number of channels] x [monoral bitrate]. Quality 4 requres around 80[kbps] for monoral samples, so dual-stereo quality 4 does around 160[kbps].
Title: Vorbis quality – wrong direction?
Post by: [proxima] on 2004-02-10 17:24:02
Quote
Wow!  I also find [proxima]'s listening result very interesting. If QuantumKnot and I change stereo mode correctly, the result shows lossless stereo is worse than uncoupled (dual) stereo.

I 'm very surprised too !
Before this test, i was quite sceptical about the possibility of using an uncoupled stereo mode for a mid/low bitrate (128 kbps). Vorbis (in the current state) have to be an exception.

I want to advert you that i manly focused my attention with HF boost/hiss and not with other possible artifacts, moreover i'm not very familiar with classical samples so take my findings with care. Maybe Guruboolez, that certainly is more familiar with his samples, can test your encoder and eventually confirm the hiss reduction.
Title: Vorbis quality – wrong direction?
Post by: PoisonDan on 2004-02-10 17:42:58
On a slightly off-topic note, I'm delighted to see that there has been so much Ogg Vorbis-related activity lately.

It feels to me like there the tuning of the format is finally starting to take off...

Title: Vorbis quality – wrong direction?
Post by: kjoonlee on 2004-02-10 17:54:05
Here are some relevant links from Xiph.org's mailing list archive.

http://www.xiph.org/archives/vorbis-dev/200402/0009.html (http://www.xiph.org/archives/vorbis-dev/200402/0009.html)
Quote
You could test this effect on monoaural files.
The "HF boost" will still be present.
It's because of heavy quantization. The SNR is *so*
low at high frequencies in low bitrate modes, that the
quantization error increases energy noticably.


Test it yourself:
- generate some random numbers p_i (around -1..1)
- round them to the nearest int q_i:=round(p_i);
- compare the "energy":
  \sum{i} p_i^2
  \sum{i} q_i^2


The bad news:
I don't know how to avoid this in general.


Ghis!
Sebastian


http://www.xiph.org/archives/vorbis-dev/200402/0011.html (http://www.xiph.org/archives/vorbis-dev/200402/0011.html)
Quote
> You could test this effect on monoaural files.
> The "HF boost" will still be present.
> It's because of heavy quantization. The SNR is *so*
> low at high frequencies in low bitrate modes, that the
> quantization error increases energy noticably.


Ooh, if *this* causes that "HF boost" problem, I know how
to fix it (I fixed it for Lame a few years ago ;-) )


> The bad news:
> I don't know how to avoid this in general.


But I do :-)


Stay tuned...



Segher
Title: Vorbis quality – wrong direction?
Post by: maikmerten on 2004-02-10 18:27:38
Quote
On a slightly off-topic note, I'm delighted to see that there has been so much Ogg Vorbis-related activity lately.

I´m delighted as well and would like to say "thank you" to all those working on vorbis.
Title: Vorbis quality – wrong direction?
Post by: MGuti on 2004-02-10 18:50:27
me as well, its good to know that vorbis may not be dead from this point forward!!

anyway. encoding a mono file still has HF boost i suppose. so it must be a combination of things. if the HF boost in vorbis was removed, it would do much better in listening tests i think. maybe if the problem can be pinned down by the multiformat test vorbis will score better than last time.

i am willing to do just about any amount of listening to fix this problem as i would love for vorbis to be the real competitor for AAC.
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-11 00:16:45
Quote
Wow!  I also find [proxima]'s listening result very interesting. If QuantumKnot and I change stereo mode correctly, the result shows lossless stereo is worse than uncoupled (dual) stereo. Where does the difference come from? Parameter difference in setup_44.h and setup_44u.h?

My uncooupled (dual stereo) oggenc is available in this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=18263). It is not my "Modest Tuning". 

I have a feeling my lossless coupled Vorbis wasn't implemented correctly or optimally so its too early to make assumptions on it

Essentially what I did was replicate the adj_stereo of q 10 down to 5, 4, 3 and 2.  I'm not sure whether there are other parts that I need to change. 

So I wouldn't put that much attention in my rather quick hack
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-11 00:18:50
Quote
On a slightly off-topic note, I'm delighted to see that there has been so much Ogg Vorbis-related activity lately.

It feels to me like there the tuning of the format is finally starting to take off...


As long as there are plenty of people willing to stick around, test our tunings, and give us feedback, then that's enough drive for us to continue tuning Vorbis.
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-11 13:15:33
I encoded guruboolez' violin sample with the standard 1.0.1 coder and nyaochi's uncoupled stereo coder.  Then in CoolEdit, I did a mix paste and saved the differences in order to (informally) highlight the differences.

http://www.hydrogenaudio.org/forums/index....ndpost&p=183492 (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360&view=findpost&p=183492)

There seems to be a high pitched noise in the middle of the sample.  Is this the dreaded HF boost/hiss of Vorbis?


EDIT:  I've also done the same process with guruboolez' harpsichord sample (uploaded to the same thread). Looking at the spectrum (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360&view=findpost&p=183502) (I know, its naughty  ), the HF difference is mostly above 8 kHz.  The spectrum of the violin sample also has most of this high frequency difference above 8 kHz.

Hopefully something useful can be gathered from this.


EDIT 2:  Did the same thing with mandolins sample where 1.0.1 got a low score and uncoupled got the highest in [proxima]'s test.  Same kind of noise.  Spectrum also mostly above 8 kHz.
Title: Vorbis quality – wrong direction?
Post by: tigre on 2004-02-11 14:26:47
Quote
EDIT:  I've also done the same process with guruboolez' harpsichord sample (uploaded to the same thread). Looking at the spectrum (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360&view=findpost&p=183502) (I know, its naughty  ), the HF difference is mostly above 8 kHz.  The spectrum of the violin sample also has most of this high frequency difference above 8 kHz.

Hopefully something useful can be gathered from this.

Not much, I think. If the substraction shows content in > 8kHz range only this only tells you that both encodes are different in this range, while they're very similar/identical below 8kHz, but you don't know if one encoder contains more energy in >8kHz band than the other. The difference could simply be caused by phase shift (probably oversimplified, but to get the idea)

You could gather more useful information from  (CEP/Audition) frequency analysis of identical positions.
Title: Vorbis quality – wrong direction?
Post by: music_man_mpc on 2004-02-11 14:27:40
Quote
I encoded guruboolez' violin sample with the standard 1.0.1 coder and nyaochi's uncoupled stereo coder.  Then in CoolEdit, I did a mix paste and saved the differences in order to (informally) highlight the differences.

I don't think Dibrom would be too impressed with this stuff.  Maybe you should back it up with a listening test of you own?  I just don't want to see you get flamed. 
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-11 14:34:37
Quote
I don't think Dibrom would be too impressed with this stuff.  Maybe you should back it up with a listening test of you own?  I just don't want to see you get flamed.  

The point of the exercise is to pinpoint the differences that uncoupled stereo has over coupled stereo rather than to compare coders subjectively (ie. this coder sounds better than that one).  I'm not making any claims here, but just doing a bit of detective work and seeing if stereo coupling really is the cause of HF boost in Vorbis.

I know it's bad (I even stated it that this is bad) but I don't have good ears and have trouble hearing these distortions so I need an empirical way of observing the differences that guruboolez and [proxima] are noticing in their listening tests.  Hopefully Dibrom will pardon me on this. 
Title: Vorbis quality – wrong direction?
Post by: kjoonlee on 2004-02-11 14:38:03
Will it be possible to "isolate" the HF boost?

Will it be possible to "add" the isolated HF boost to the uncompressed original?

Will the "spiked" original be ABXable against the original?

Pardon me if these are stupid questions, please..
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-11 14:41:14
Quote
Will the "spiked" original be ABXable against the original?

Pardon me if these are stupid questions, please..


It's quite a good idea actually.  What I might do is take these differences, add them back to the original, and get some people with good ears to ABX them.

EDIT:  I've uploaded three files where I have added the differences back to the original.  The expectation is these will be ABXable with similar distortions.
Title: Vorbis quality – wrong direction?
Post by: music_man_mpc on 2004-02-11 14:52:23
Quote
The point of the exercise is to pinpoint the differences that uncoupled stereo has over coupled stereo rather than to compare coders subjectively (ie. this coder sounds better than that one).  I'm not making any claims here, but just doing a bit of detective work and seeing if stereo coupling really is the cause of HF boost in Vorbis.

I know it's bad (I even stated it that this is bad) but I don't have good ears and have trouble hearing these distortions so I need an empirical way of observing the differences that guruboolez and [proxima] are noticing in their listening tests.  Hopefully Dibrom will pardon me on this. 

Fair enough.  I suppose Guruboolez will provide the conclusive evidence in the form of a listening test soon anyway, I guess its nice to have some idea of what's going on in the mean time.  Although, it still might not go over too well.
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-11 15:01:39
Quote
Fair enough.  I suppose Guruboolez will provide the conclusive evidence in the form of a listening test soon anyway, I guess its nice to have some idea of what's going on in the mean time.  Although, it still might not go over too well.

My choices are rather limited unfortunately and I'm unable to go into the Vorbis source code and try to tackle this HF boost problem by relying on subjective comments on quality only.

But we'll try kjoonlee's suggestion and see how it goes.  If these corrupted samples are ABXable and distortions are similar in nature to the HF boost, then we've at least verified experimentally that stereo coupling has problems 
Title: Vorbis quality – wrong direction?
Post by: [proxima] on 2004-02-11 15:02:29
Quote
It's quite a good idea actually.  What I might do is take these differences, add them back to the original, and get some people with good ears to ABX them.

I think this is not useful because, in this manner, we are completely ignoring masking. I see no reason to abx a such sample. Maybe someone more expert can confirm my assumption.
I remember some Musepack spectral analysis where original and encoded differ a lot visually, but all the quantization noise is perfectly masked and the files are not ABXable.
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-11 15:10:15
Quote
,Feb 12 2004, 01:02 AM] I think this is not useful because, in this manner, we are completely ignoring masking. I see no reason to abx a such sample. Maybe someone more expert can confirm my assumption.
I remember some Musepack spectral analysis where original and encoded differ a lot visually, but all the quantization noise is perfectly masked and the files are not ABXable.

I sort of guessed it wouldn't be so simple. 

Oh well, in that case, I give up on the HF boost problem.  I guess there is little point in me trying to fix something I cannot hear that well.

I'll focus on the tonality problem in the Arche I sample which I can actually hear as well as pre-echo issues.
Title: Vorbis quality – wrong direction?
Post by: kjoonlee on 2004-02-11 15:12:49
Indeed, I was a little hesitant to ask my questions because I don't understand noise masking very well, if at all.

Also, on second thought, I doubt if my suggestion would be very useful.

I'm sorry for the confusion.
Title: Vorbis quality – wrong direction?
Post by: tigre on 2004-02-11 15:38:48
Quote
I sort of guessed it wouldn't be so simple.  

Oh well, in that case, I give up on the HF boost problem.  I guess there is little point in me trying to fix something I cannot hear that well.

Before you give up - have you tried using frequency analysis for comparing orignal vs. different encodes hf boost-wise, as I suggested in my last post?
Title: Vorbis quality – wrong direction?
Post by: nyaochi on 2004-02-11 17:38:14
Let me add two samples available here (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360). You can hear boosted close hi-hat cymbal in the 8823 sample with 1.0.1 -q4. The second sample is white noise created by CoolEdit. I suppose some of you disagree with the use of artificially created sample, but the problem with 1.0.1 -q4 is too obvious for me. I found the sample encoded by 1.0.1 -q4 totally different from the original. I'm not sure this sample exposes HF boost problem (or another problem?), but I believe it's a good material to improve Vorbis.

Actually, I created my "Modest Tuning" to improve these samples. Here's a part of my personal listening result (in Japanese only) (http://nyaochi.cocolog-nifty.com/audio/2004/01/aotuv_alpha3_.html) of 8823 sample (I focused on only HF boost) with Modest Tuning beta2 (mtb2), aoTuV alpha2 (at2), GarfTuned3 beta1 (gt), QKTune beta2 (qk2).

Code: [Select]
ABC/HR Version 0.9b, 30 August 2002
Testname: 8823

1L = C:\users\musics\test\compare\8823-q5qk2.wav  (171kbps)
2L = C:\users\musics\test\compare\8823-q5at2.wav  (158kbps)
3L = C:\users\musics\test\compare\8823-q5gt.wav   (178kbps)
4L = C:\users\musics\test\compare\8823-q4mtb2.wav (169kbps)

---------------------------------------
General Comments:
---------------------------------------
1L File: C:\users\musics\test\compare\8823-q5qk2.wav
1L Rating: 3.0
1L Comment:
---------------------------------------
2L File: C:\users\musics\test\compare\8823-q5at2.wav
2L Rating: 3.5
2L Comment:
---------------------------------------
3L File: C:\users\musics\test\compare\8823-q5gt.wav
3L Rating: 2.5
3L Comment:
---------------------------------------
4L File: C:\users\musics\test\compare\8823-q4mtb2.wav
4L Rating: 4.5
4L Comment:
---------------------------------------
ABX Results:
Original vs C:\users\musics\test\compare\8823-q5qk2.wav
   13 out of 14, pval < 0.001
Original vs C:\users\musics\test\compare\8823-q5at2.wav
   13 out of 14, pval < 0.001
Original vs C:\users\musics\test\compare\8823-q5gt.wav
   13 out of 14, pval < 0.001
Original vs C:\users\musics\test\compare\8823-q4mtb2.wav
   12 out of 21, pval = 0.332


I haven't test these samples with uncoupled stereo but I'll later. 
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-12 00:07:33
Quote
Quote
I sort of guessed it wouldn't be so simple.  

Oh well, in that case, I give up on the HF boost problem.  I guess there is little point in me trying to fix something I cannot hear that well.

Before you give up - have you tried using frequency analysis for comparing orignal vs. different encodes hf boost-wise, as I suggested in my last post?

No.  It was 1 am in the morning and I was getting really tired so I gave up trying.
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-12 00:51:54
Quote
Let me add two samples available here (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360). You can hear boosted close hi-hat cymbal in the 8823 sample with 1.0.1 -q4.

My ABXing of the uncoupled stereo hi-hat cymbal at q 4 though I thought the differences were so subtle, I often doubted myself in most cases.

Code: [Select]
-------------------------------------
WinABX v0.23 test report
02/12/2004 10:42:23

A file: E:\vsamples\8823.wav
B file: E:\vsamples\uncoupled.wav (155.27 kbps)

10:43:57    1/1  p=50.0%
10:44:11    2/2  p=25.0%
10:44:25    3/3  p=12.5%
10:44:34    3/4  p=31.2%
10:44:47    4/5  p=18.8%
10:45:33    5/6  p=10.9%
10:45:45    5/7  p=22.7%
10:46:06    6/8  p=14.5%
10:46:34    7/9  p= 9.0%
10:46:40   8/10  p= 5.5%
10:46:57   9/11  p= 3.3%
10:47:05  10/12  p= 1.9%
10:47:11  11/13  p= 1.1%
10:47:17  12/14  p= 0.6%
10:47:24  13/15  p= 0.4%
10:47:40  14/16  p= 0.2%
10:47:47  test finished


That is quite a nice sample for HF boost.
Title: Vorbis quality – wrong direction?
Post by: kwanbis on 2004-02-12 03:24:14
i hope you all get together an make vorbis the great encoder it was meant to be ... even if forking grom xiph ... the world NEEDS a free encoder ... thank you guys
Title: Vorbis quality – wrong direction?
Post by: ErikS on 2004-02-12 04:44:45
Quote
,Feb 11 2004, 11:02 PM] I think this is not useful because, in this manner, we are completely ignoring masking. I see no reason to abx a such sample. Maybe someone more expert can confirm my assumption.
I remember some Musepack spectral analysis where original and encoded differ a lot visually, but all the quantization noise is perfectly masked and the files are not ABXable.

That's why you add in the original signal to the difference. That way you have the masker signals present at those time the difference signal would be masked. Just listening to the difference signal however would have the problem you mention. Right? But why not compare the uncoupled stereo with the "joint stereo" one directly. All other variables are constant right?
Title: Vorbis quality – wrong direction?
Post by: knik on 2004-02-13 11:25:11
A listening may not help to fix the HF boost. I'm affraid one would need to dig deeply into the source until finds where the energy difference comes from.
Title: Vorbis quality – wrong direction?
Post by: [proxima] on 2004-02-13 17:26:05
Quote
A listening may not help to fix the HF boost. I'm affraid one would need to dig deeply into the source until finds where the energy difference comes from.

Maybe in this case listening cannot help to resolve codec problems, we need a capable person who can understand the code and correct this problem. Anyway, due to the availability of hardware Vorbis players, i think that testing is still important because :
- maybe we can suggest the newbies a best, "HA approved" version/128 kbps settings.
- there is the chance to reduce HF boost with "small hacks" such as disabling stereo coupling.
Title: Vorbis quality – wrong direction?
Post by: guruboolez on 2004-02-14 17:18:22
I’m late, but I didn’t have much free-time this week (and internet access is a second problem too). I have performed the test yesterday, comparing:
- RC3 (march 2002)
- 1.01 (october 2003)
- lossless coupling (Quantum Knot)
- uncoupled encoder (Nyaochi)

According to the fact that uncoupled encodings at –q4 are a lot bigger than traditionnal vorbis encoder at the same setting, I also add an uncoupled encoder at lower setting. Nyaochi uncoupled at –q2 is close to average bitrate of 1.01/RC3 –q4, but slightly inferior. I’ve therefore tested fully uncoupled encoder at –q2,3 in order to measure the negative impact of a sub-efficient (uncoupled) channel coupling at ~128 kbps, compared to the same encoder, helped with channel coupling.



Results – /|\ Don’t care about notation absolute signification: it isn’t ITU compliant


(http://audiotests.free.fr/tests/2004/vorbisoldnew3.png)


• uncoupled encoders are winning again, without any doubts.
• full uncoupled encoder [FU] (Nyaochi) is clearly superior to lossless coupled [LC] (provided by Quantum Knot) vorbis encoder. FU was always better than LC, except on one sample  – Mars: same notation. These results confirm [proxima]’s conclusions.
• both uncoupled encoders are still suffering from noise or tonality difference. The problem is not fully corrected: there are still differences. It’s not awfully annoying, but it’s difficult to forgive at ~160 kbps...

• at ~128 kbps, 1.01 appears again as a complete loser, compared to RC3 (harpsichord is the exception again) and to FU –q 2,3. There are so much noise and imprecision with 1.01... In other words, 1.01 without coupling was always better than 1.01 with coupling!!!
• But FU –q 2,3 is far from perfection: there are much more noise than with FU at –q 4. Noise isn’t the only difference. Other artifacts appeared (for example, acidity on Spagna, distortions on Brahms). And pre-echo reached an annoying level.
• Therefore, there are no clear winner between RC3 –q4 and FU –q 2,3. On four samples, RC3 sounded better; on the four others, it was FU. Overall notation is    in favor of RC3, but the difference is distorted by an approximate notation (with deliberately exaggerating contrasts).
Title: Vorbis quality – wrong direction?
Post by: rjamorim on 2004-02-14 17:48:50
I can only imagine the discussions that will arise when choosing codecs for the 128kbps multiformat test

Vorbis 1.0.1 vs. GT3 vs. QK vs. Nyaochi vs. CVS...
Title: Vorbis quality – wrong direction?
Post by: guruboolez on 2004-02-14 18:03:09
Quote
I can only imagine the discussions that will arise when choosing codecs for the 128kbps multiformat test

Vorbis 1.0.1 vs. GT3 vs. QK vs. Nyaochi vs. CVS...

Yes, probably.
But you could request to people demanding for x or y encoder a blind test: nobody will bother you anymore

I guess that uncoupled encoder can't be considered as real competitors. There were released in order to isolate a problem (hiss), and probably need tuning in order to increase their efficiency.

P.S. Thank you for your private messages (I'll answer you later)
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-15 00:05:55
Again, thanks to guruboolez for his testing.

The aim of testing lossless and uncoupled stereo is to probe the behaviour of the HF boost.  I think from guruboolez and [proxima]'s tests, we can conclude that the lossy coupling (point) is having problems but is not the sole cause. We also see that uncoupled stereo is not perfect and still suffers from noise. 

So perhaps, the uncoupling has only reduced the effect of the hiss but not totally removed it.  That is, it hasn't made the problem go away or solved it, but only made it harder to detect.  Hence I'm beginning to believe that we can rule out stereo coupling as the cause of HF boost.  Digging into the source code of rc3 may provide new insight as well as nyaochi's focus on the noise companding in his 'Modest Tuning'.

I know there may be some other people on the Vorbis-dev mailing list who are saying 'I told you so', but hey, it never hurts to do some comprehensive listening tests to verify it.
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-22 05:32:15
I just came across results from these listening tests, performed by nyaochi.  The big table in the middle of the page should be self explanatory.

http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html (http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html)
Title: Vorbis quality – wrong direction?
Post by: harashin on 2004-02-22 05:59:55
Quote
I just came across results from these listening tests, performed by nyaochi.  The big table in the middle of the page should be self explanatory.

http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html (http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html)

Interesting. He also releases the 1.0.1 Modest Tuning beta 2 (http://homepage3.nifty.com/nyaochi/soft/dist/oggencmtb2.zip) used in these tests.
Title: Vorbis quality – wrong direction?
Post by: nyaochi on 2004-02-22 07:36:42
Quote
I just came across results from these listening tests, performed by nyaochi.  The big table in the middle of the page should be self explanatory.

http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html (http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html)

Some notes for the test. I ranked my favorite 3 samples as well as 8 classical samples presented by guruboolez. The main purpose of this test is to evaluate aoTuV b1 -q4 compared with 1.0.1 official -q4, RC3 (March 2002) -q4, lossless coupling -q4, uncoupled stereo -q4/-q2.3, and my modest tuning -q4. Note that bitrates are not similar between these samples (LC -q4, UC -q4 and MTb2 -q4 is much higher than 128kbps.)

I failed to abx 2 samples because I was not familiar with classical samples and got tired.   

Results:
At a grance, the total score shows that MTb2 was the first. However, it was not so good because the average bitrate of MTb2 is too high (around 160-170kbps). On the other hand, aoTuV has a good cost-performance without bitrate inflation. Preecho problem (e.g., Harpsichord and Mandolins) seems to lower the score (preecho is not his main target of the tuning).

Just as the listening results of guruboolez and [proxima], 1.0.1 official ends up the last place. UC was better than LC.
Title: Vorbis quality – wrong direction?
Post by: harashin on 2004-02-22 07:54:15
The Modest Tuning Beta2 won my own test (http://cyberquebec.ca/harashin/florida_seq_Vorbis_test.html).  Apparently it's superior to other Vorbis encoders on this kind of sample. (sharp attack, symbal)
Although aoTuV b1 is also thinkable as a good choice for its reasonable bitrate.
Title: Vorbis quality – wrong direction?
Post by: music_man_mpc on 2004-02-22 16:45:08
Quote
However, it was not so good because the average bitrate of MTb2 is too high (around 160-170kbps).

Perhaps next time you could compare it to 1.01 -q5 as well?  Results would be interesting, IMO.
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-22 23:39:56
It certainly is surprising that lossless coupling is doing so badly compared with uncoupled.  Theoretically, you'd expect they should be the same! 
Title: Vorbis quality – wrong direction?
Post by: harashin on 2004-02-23 06:40:05
Quote
It certainly is surprising that lossless coupling is doing so badly compared with uncoupled.  Theoretically, you'd expect they should be the same! 

Indeed, it's very odd. However, I was able to ABX between LC and UC with score at 15/20.  Also, they don't actually seem to be same.

lc_uc.png (http://cyberquebec.ca/harashin/lc_uc.png)
Title: Vorbis quality – wrong direction?
Post by: QuantumKnot on 2004-02-24 02:18:54
Quote
Quote
It certainly is surprising that lossless coupling is doing so badly compared with uncoupled.  Theoretically, you'd expect they should be the same! 

Indeed, it's very odd. However, I was able to ABX between LC and UC with score at 15/20.  Also, they don't actually seem to be same.

lc_uc.png (http://cyberquebec.ca/harashin/lc_uc.png)

Monty made a good point in IRC that this is to be expected since compilers do random things to FP computations
Title: Vorbis quality – wrong direction?
Post by: guruboolez on 2005-02-14 15:59:10
[span style='font-size:15pt;line-height:100%']...One year later[/font][/span]

Just for curiosity, I've tried another comparison, using the same samples, in order to check vorbis's progress during three years, and especially during the last one.
I kept vorbis RC3 (march 2002), vorbis 1.01 (march 2003), and add the most advanced vorbis encoder: aoTuV beta 3. I'd like to check the amount of noise/coarseness audible with aoTuV compared to an old encoder which wasn't too affected by this problem.

results:

(http://audiotests.free.fr/tests/2005.02/results.png)

ABX log/scores are here (http://audiotests.free.fr/tests/2005.02/hybrid_log.zip).

There are still audible issues with aoTuV at this bitrate, but they are much lower than vorbis RC3. One exception: the last sample, which RC3 performed differently (and to my ears: better) on brass (micro-attacks sample).
Bravo to Aoyumi!
Title: Vorbis quality – wrong direction?
Post by: DreamTactix291 on 2005-02-15 06:26:20
Wow.  That is some nice improvement.

It looks like aoTuV is pushing Vorbis in the right direction instead of regressing it
Title: Vorbis quality – wrong direction?
Post by: Aoyumi on 2005-02-16 14:01:23
Quote
There are still audible issues with aoTuV at this bitrate, but they are much lower than vorbis RC3. One exception: the last sample, which RC3 performed differently (and to my ears: better) on brass (micro-attacks sample).
[a href="index.php?act=findpost&pid=273435"][{POST_SNAPBACK}][/a]


The handling of a micro attack is one of the big subjects truly.

I know that QuantumKnot is performing the trial for solving this problem.
Probably, I am looking forward to it.