I promised to Quantum Knot to test his new tuning. I finally found the time, and tested QK2, 1.01 - compared to RC3 library. Yes, the two-years old RC3.
I. CHALLENGERS
The challengers are of course questionable. I had to explain my choices.
First, I didn't include Nyaochi and Aoyumi tunings. I really apologize to both persons. It might appear as disrespectful, but in my opinion, comparing different tunings in alpha stage each others is not something fair. I guess that alpha tunings are released for report, not for competition. Comparison will probably follow later, when all tunings will be advanced enough.
Second thing: why old vorbis encoders?
I've tested vorbis 1.00 and 1.01 many times with natural instruments, and it always sounded approximative and disapointing, generally worse than lame mp3. To my ears, there are serious problems: not only hiss or high frequencies boost (reported many times by other users), but imprecise or coarse sound, aggressive rendering. With classical music, or natural instruments well recorded, all these problems are clearly audible – probably more than with loud and (over-)compressed music.
These problems are like a vorbis “signature” (in blind test, it’s easy to find vorbis among different challengers). I never heard something comparable with other formats. But I’m sure that these flaws are not consubstantial to vorbis format, but that they are a consequence not only of a lack of tuning, but from contestable choices made in the past. I'm still convinced that something wrong happened between RC3 and final "1.00" version from July 2002, though I never really tested this difference. My suspicions are based on the following things.
Important changes were made between RC3 and Final library. This was particularly noticeable on encoding speed (final is much faster), and on sound quality at low bitrate, even for untrained people. Before “final” encoder, Vorbis wasn’t very competitive at low bitrate (especially the popular and symbolic 64 kbps). The reference two years ago was mp3pro, and vorbis was clearly behind. At low bitrate, Vorbis sounded like other non-SBR format (wma, real, aac): metallic, heavily distorted, etc… But final encoder sounded totally different: not metallic anymore, less distorted, but simply noisier (and some stereo issues). For people used to find traditional flaws at this bitrate, vorbis was simply amazing. With some habits, noise and stereo problems would be more noticeable, but even here, vorbis is a very good solution at 64 kbps (see Roberto’s recent test) compared to all present non-SBR format. Noise and stereo reduction are generally more acceptable and less ugly than metallic coloration a la wma or DivX audio...
But this incontestable victory (for an open-source and non-patented project) had some bad reverse. In my opinion, pre-final encoders were more transparent at mid-bitrate[/b] (especially 128 kbps or –q4 setting), at least on non-killer samples (90% or 99.9% of [my?] music, I can’t say) and at least with natural instruments recorded on hi-fi principles. At this bitrate, encodings are now sharing the same flaw as low bitrate: hiss, imprecise rendering… On cool samples, where even mp3 sounds flawlessly, vorbis has audible problems. I suspect that the secret of vorbis good quality at low bitrate is the cause of wrong things that are happening above.
I never seriously tried to confirm or infirm my suspicions with blind test. I was more interested by high bitrate & transparent encodings, and because of pre-echo serious problems, vorbis wasn’t really interesting (except with Garf tuning). Now than Vorbis begins to infiltrate industrial manufacturers, the audio format looks for me more interesting at lower bitrate (I expect from modern format a good quality at 130-150 kbps]. Testing Quantum Knot tuning encoder appears like a good opportunity for me to oppose post-1.00 encoders to old RC3 library, in order to see if my suspicions are justified or simple rubbish.
I looked on old CD-R, and founded three RC3 builds, dated on: February 2002, March 2002 and April 2002. I was tempted to test two different builds, and not only one RC3 encoder. Why? In my souvenirs, a vorbis developer talked about changes happened within RC3 (can't remember or understand what - maybe something related to stereo model). I think it happened in springs (April or May). February and March encoders produce same output result (I did a bit to bit comparison, and files were the same, except tiny difference on the first samples). April encoder is a different beast. Encoding speed had improved (twice faster, if not more!), and output is different too. If problems occur with final library, the cause is maybe in this crucial moment of vorbis history. I'm maybe completely wrong in my suspicions, and maybe RC3 sounds worse than 1.00 encoder. The best thing to be sure about it is to test the different encoders.
The following test confronts four vorbis encoders:
• 2002.03.07
• 2002.04.06
• Official 1.01 oggenc – JohnV compile (2003.09.09)
• GordianKnot tuning QK2 (January 2004, based on 1.01)
Setting for all encoder is the same: -q4 (VBR, 128 kbps nominal)
II. SAMPLES
In order to make the 1.01 & QK2 comparison (primary goal of the test) useful, I’ve selected sample with transients. I didn’t include well-known killer samples: positive reports for QK2 were done with this kind of samples, and for 128 kbps encodings, I guess that common musical samples are difficult enough for all encoders and more representative of real usage too. All samples are coming from my own library. I’ve deliberately chosen very short ones, so upload will be easier.
Short description:
• harpsichord.wav: solo harpsichord, maybe too reverberated but very sharp and nice recording. Encoders usually suffer from pre-echo and from heavy distortions
• erhu10.wav: erhu (Chinese string instrument) with percussive instruments in accompaniment.
• Arche I.wav: orchestral extract from Penderecki first Symphony. Very sharp and loud attack, followed by something like a rattle [not Simon]
• Mandolins.wav: extract from a famous Vivaldi concerto. This sample (reduced to 5 seconds) is one of my favorite, because many encoders failed to encode this properly at ~130 kbps.
• Transfiguration.wav: part of an orchestral work of Olivier Messiaen. Two different problems should occur: distortions with cymbals (and with Vorbis 1.00 family: flattened noisy sound) and pre-echo/blurred brass instruments.
• La Spagna.wav: percussions & wind instruments from Renaissance playing concertedly. Pre-echo reduction is expected from GK2, and maybe noise problems may occurs with wind instruments.
• Mars.wav: beginning of the first Planet[/U], from Gustav Holst. Naturally quiet (violin pizzicatos and threatening winds, this sample is very quiet due to weird mastering of the CD layer [rip from a SACD]. This sample is comparable to orchestral lace, and I listened it without harm at very high volume in order to magnify all possible problems (pre-echo, ATH issues, background changes...)
• Brahms6.wav: piano is a percussive instruments, potentially affected by pre-echo. This recording from the 6th Hungarian Dance of Johannes Brahms is really sharp and well-recorded.
In other words, the samples tested are probably not favorable to pre-Final libraries. For measuring vorbis noise problems at ~128 kbps, tonal sample are more interesting. But even here, interesting things happened during the test.
III. RESULTS
(http://audiotests.free.fr/tests/2004/vorbisoldnew.png)
• QK2 modifications have a positive effect on 5 samples. The benefits of this encoder concern pre-echo and sharpness only. On low-volume transients, GK2 is inoperative (Mars.wav). On sharp and detailed micro-attacks (Arche I.wav second part, and Transfiguration.wav), QK2 sounds identical to official 1.01. But when attacks are clearly defined (Arche I.wav first part and Harpsichord.wav), the difference is appreciable. Benefits are audible too when transients are not excessively strong (piano, percussions on La Spagna.wav), but difference is then mildly audible. I’m disappointed by the few differences with Mandolins.wav, and surprised by the identical scores I obtained with Erhu10.wav.
Vorbis library needs tweaking for mid bitrate encodings, and QK2 is a good answer for the pre-echo problem.
• The official post final “1.01” encoder sounded worse than RC3 libraries to my ears on 7 samples. Harpsichord is the only exception, with less distortion (vibrating effect). But most often, the noise issues (louder, unstable, affecting micro-dynamic of some instruments and definition/contours of others) was more discriminating than pre-echo variations. With Erhu10.wav, Transfiguration.wav and Arche I.wav (second part), difference between RC3 family and 1.01 is really big (whereas progress made with QK2 from 1.01 had a much more limited impact).
Interesting thing to note: official 1.01 never reached the 3.0 (= “slightly annoying”) notation. Notation is of course a very imprecise thing, but this ranking isn’t totally meaningless. 1.01 produces non transparent and barely acceptable sound with natural instruments, whereas the RC3 branch, as all different audio format, is able to reach near-transparency with non-difficult samples. To the 8 samples of this test I could add the results of the ~30 other samples I’ve tested in the past with vorbis: final vorbis results were each time around 3/5. In other words, the changes introduced with 1.00 are negative at –q 4 with a consequent part of the editorial production (at least for my ears).
• Other thing to note: there wasn’t any rupture between March 2002 [m2k2] and April 2002 [a2k2]. Library from m2k2 is not the crystal clear one, and a2k2 the Hoover© sounding library. M2k2 RC3 was better than a2k2 four time; a2k2 was better (but by a slightly margin) the four other time. Nevertheless, if April 2002 wasn’t an historic moment for Vorbis (except for encoding speed), it was surely the beginning of tendency: noise increased with a2k2 and reached an alarming level. The accident occurred few months later…
Other point: pre-echo issues were reduced with a2k2 library (compared to the previous one). But comparing RC3s to Finals libraries on pre-echo is not easy. Pre-echo is sometimes lower with RC3 (a2k2 or m2k2) : erhu10, Arche I, Mandolins and Brahms - and sometimes higher than with 1.01 output : Spagna, Holst.
IV. CONCLUSIONS
Hard to conclude anything with only eight samples. I nevertheless reached the following one: my suspicions weren’t unfounded, and there’s obviously something rotten in 1.00 – and following libraries. I haven’t test Vorbis with non-classical/over-compressed/electronic music, and therefore I can’t evaluate the possible benefits of 1.00 modifications. But with classical (or more generally, natural instruments recorded with fidelity principles), these changes have clearly a negative effect – with some local exceptions (like harpsichord).
I guess that without these changes, Vorbis will sound poorly at low bitrate (a quick try with RC3 at 64 kbps convinced me that 1.00 is much better). But isn’t RC3 a better basis for mid-bitrate tuning than 1.00? Or is it possible to disconnect Vorbis mid settings from low settings, in order to avoid all characteristic introduced by “final” version to contaminate all encoding area (at least, up to 5.99]?
Note: the eight samples (FLAC – 4.5 MB] are available on the uploading forum – or will follow very soon.
I think you mean QuantumKnot. This (http://gordianknot.sourceforge.net/) is Gordian Knot.
Files are uploaded.
I've just realized that I forgot to post the exact results.
EDIT > replaced Gordian Knot by Quantum Knot Thanks
EDIT2 > replaced GK2 by QK2
Very interesting test, as usual.
Thank you, guruboolez.
I'm wondering: now that we have several different encoders (RC3, 1.0.1, GT3b1, QK2, aoTuV), it may be worthwhile to perform a full listening test comparing these encoders.
The MP3 encoder listening test has recently been completed, the AAC encoder test will soon begin, so it may make sense to do a Vorbis encoder test as well.
It probably won't be a good idea right now, since some of the tunings are still a work in progress, but I think it could be something to consider later this year.
I wonder if Roberto would be interested in setting up this listening test...
Edit: Technically, they're not really different "encoders" of course, just differently tuned revisions of the Vorbis encoder. So the terminology I used is not 100% correct.
what bitrate was the test run at? and i am also running a test and have noticed similar problems with vorbis. i would like to try an RC3 test and if it yields better results, it would be nice to have someone investigate the changes (Quantum, if you could).
im sure it would be a nice addition to the QK-tune.
good job with the test.
I maybe forgot to precise: all tests were performed with -q4 setting (128 kbps nominal).
Many thanks to guruboolez for being able to perform this detailed listening test. It is essentially the feedback I was hoping for and it has brought about some fairly interesting points, esp. about RC3, which I haven't played with before.
I'll download the RC3 sources and have a look at the changes that were made.
(http://audiotests.free.fr/tests/2004/vorbisoldnew2.png)
uncoupled -q4 was tested here. Bitrate is higher, but problem is considerably lowered (few hiss).
I did this test yesterday. Very quickly: I was very tired. I didn't test lower setting, except with few sample.
Apparently, coupling is THE problem.
guruboolez, I'm somewhat confused as to what your graphs are explaining. Considering I'm somewhat of a n00b, could you or someone else explain the graphic results for me?
uncoupled -q4 was tested here. Bitrate is higher, but problem is considerably lowered (few hiss).
I did this test yesterday. Very quickly: I was very tired. I didn't test lower setting, except with few sample.
Apparently, coupling is THE problem.
Was this the uncoupled one I posted or the one nyaochi posted? His one uses uncoupled stereo while mine used lossless stereo coupling (though at the time, I was under the impression it was uncoupled).
Looks like this hiss problem is affected by various factors, and stereo coupling is one of them.
On the HF boost front, Segher on vorbis-dev believes he knows a way of fixing this problem, which he has done in Lame. Let us wait for the good news.
EDIT: Gosh, I had a listen of "Arche I" in one of guruboolez' samples and Vorbis has a tonality problem on the first attack.
This morning i’ve quickly done the same test and can confirm what guruboolez has noticed.
It seems that channel coupling is the problem, moreover with totally uncoupled stereo mode the results are even more better. Don’t give too much attention to average, the scores are really significant only if compared each other.
Here is the table of results:
http://www.hydrogenaudio.org/forums/index....ic=18360&st=0 (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360&st=0&#)
[span style='font-size:8pt;line-height:100%']edit: i've problems uploading with with my ftp.[/span]
guruboolez, I'm somewhat confused as to what your graphs are explaining. Considering I'm somewhat of a n00b, could you or someone else explain the graphic results for me?
The numbers in the table show the score. Green cells indicate that the specified encoder won, red cells indicate the opposite.
The names you see on the left are samples (pieces of music), the names at the top are the encoders.
Some clarifications. I've used QK vorbis_uc for my tests. I didn't test Nyaochi uncoupled recently. Is it the encoder known as "ModestTuning"? The same encoder released in december? [proxima] notations are really interesting - and for me, surprising. I've tested "Modest Tuning" when it was released, and though I heard real progress in noise reduction, I wasn't totally impressed. Was -q4 used for comparison? I've tested it at -q2 or -q3 (can't remember), following Nyaochi recommandations for a fair comparison.
Anyway, I'd like to thank you, [proxima], for your tests (and similar results)
Wow! I also find [proxima]'s listening result very interesting. If QuantumKnot and I change stereo mode correctly, the result shows lossless stereo is worse than uncoupled (dual) stereo. Where does the difference come from? Parameter difference in setup_44.h and setup_44u.h?
My uncooupled (dual stereo) oggenc is available in this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=18263). It is not my "Modest Tuning".
nyaochi> thanks for the clarification. I missed this encoder you provided some days ago. I'll test it as soon as I can.
Question: what about bitrate? Is it possible to lower the bitrate in order to have something similar between uncoupled and coupled stereo? Or does coupling save a lot of bits?
Is it possible to lower the bitrate in order to have something similar between uncoupled and coupled stereo?
Unfortunately or fortunately, no. We must lower the quality of uncoupled one in order to get similar bitrates. Channel coupling actually saves alot of bits (with some expense of quality according to the listening results?)
Uncoupled stereo mode requires [number of channels] x [monoral bitrate]. Quality 4 requres around 80[kbps] for monoral samples, so dual-stereo quality 4 does around 160[kbps].
Wow! I also find [proxima]'s listening result very interesting. If QuantumKnot and I change stereo mode correctly, the result shows lossless stereo is worse than uncoupled (dual) stereo.
I 'm very surprised too !
Before this test, i was quite sceptical about the possibility of using an uncoupled stereo mode for a mid/low bitrate (128 kbps). Vorbis (in the current state) have to be an exception.
I want to advert you that i manly focused my attention with HF boost/hiss and not with other possible artifacts, moreover i'm not very familiar with classical samples so take my findings with care. Maybe Guruboolez, that certainly is more familiar with his samples, can test your encoder and eventually confirm the hiss reduction.
On a slightly off-topic note, I'm delighted to see that there has been so much Ogg Vorbis-related activity lately.
It feels to me like there the tuning of the format is finally starting to take off...
Here are some relevant links from Xiph.org's mailing list archive.
http://www.xiph.org/archives/vorbis-dev/200402/0009.html (http://www.xiph.org/archives/vorbis-dev/200402/0009.html)
You could test this effect on monoaural files.
The "HF boost" will still be present.
It's because of heavy quantization. The SNR is *so*
low at high frequencies in low bitrate modes, that the
quantization error increases energy noticably.
Test it yourself:
- generate some random numbers p_i (around -1..1)
- round them to the nearest int q_i:=round(p_i);
- compare the "energy":
\sum{i} p_i^2
\sum{i} q_i^2
The bad news:
I don't know how to avoid this in general.
Ghis!
Sebastian
http://www.xiph.org/archives/vorbis-dev/200402/0011.html (http://www.xiph.org/archives/vorbis-dev/200402/0011.html)
> You could test this effect on monoaural files.
> The "HF boost" will still be present.
> It's because of heavy quantization. The SNR is *so*
> low at high frequencies in low bitrate modes, that the
> quantization error increases energy noticably.
Ooh, if *this* causes that "HF boost" problem, I know how
to fix it (I fixed it for Lame a few years ago ;-) )
> The bad news:
> I don't know how to avoid this in general.
But I do :-)
Stay tuned...
Segher
On a slightly off-topic note, I'm delighted to see that there has been so much Ogg Vorbis-related activity lately.
I´m delighted as well and would like to say "thank you" to all those working on vorbis.
me as well, its good to know that vorbis may not be dead from this point forward!!
anyway. encoding a mono file still has HF boost i suppose. so it must be a combination of things. if the HF boost in vorbis was removed, it would do much better in listening tests i think. maybe if the problem can be pinned down by the multiformat test vorbis will score better than last time.
i am willing to do just about any amount of listening to fix this problem as i would love for vorbis to be the real competitor for AAC.
Wow! I also find [proxima]'s listening result very interesting. If QuantumKnot and I change stereo mode correctly, the result shows lossless stereo is worse than uncoupled (dual) stereo. Where does the difference come from? Parameter difference in setup_44.h and setup_44u.h?
My uncooupled (dual stereo) oggenc is available in this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=18263). It is not my "Modest Tuning".
I have a feeling my lossless coupled Vorbis wasn't implemented correctly or optimally so its too early to make assumptions on it
Essentially what I did was replicate the adj_stereo of q 10 down to 5, 4, 3 and 2. I'm not sure whether there are other parts that I need to change.
So I wouldn't put that much attention in my rather quick hack
On a slightly off-topic note, I'm delighted to see that there has been so much Ogg Vorbis-related activity lately.
It feels to me like there the tuning of the format is finally starting to take off...
As long as there are plenty of people willing to stick around, test our tunings, and give us feedback, then that's enough drive for us to continue tuning Vorbis.
I encoded guruboolez' violin sample with the standard 1.0.1 coder and nyaochi's uncoupled stereo coder. Then in CoolEdit, I did a mix paste and saved the differences in order to (informally) highlight the differences.
http://www.hydrogenaudio.org/forums/index....ndpost&p=183492 (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360&view=findpost&p=183492)
There seems to be a high pitched noise in the middle of the sample. Is this the dreaded HF boost/hiss of Vorbis?
EDIT: I've also done the same process with guruboolez' harpsichord sample (uploaded to the same thread). Looking at the spectrum (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360&view=findpost&p=183502) (I know, its naughty ), the HF difference is mostly above 8 kHz. The spectrum of the violin sample also has most of this high frequency difference above 8 kHz.
Hopefully something useful can be gathered from this.
EDIT 2: Did the same thing with mandolins sample where 1.0.1 got a low score and uncoupled got the highest in [proxima]'s test. Same kind of noise. Spectrum also mostly above 8 kHz.
EDIT: I've also done the same process with guruboolez' harpsichord sample (uploaded to the same thread). Looking at the spectrum (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360&view=findpost&p=183502) (I know, its naughty ), the HF difference is mostly above 8 kHz. The spectrum of the violin sample also has most of this high frequency difference above 8 kHz.
Hopefully something useful can be gathered from this.
Not much, I think. If the substraction shows content in > 8kHz range only this only tells you that both encodes are different in this range, while they're very similar/identical below 8kHz, but you don't know if one encoder contains more energy in >8kHz band than the other. The difference could simply be caused by phase shift (probably oversimplified, but to get the idea)
You could gather more useful information from (CEP/Audition) frequency analysis of identical positions.
I encoded guruboolez' violin sample with the standard 1.0.1 coder and nyaochi's uncoupled stereo coder. Then in CoolEdit, I did a mix paste and saved the differences in order to (informally) highlight the differences.
I don't think Dibrom would be too impressed with this stuff. Maybe you should back it up with a listening test of you own? I just don't want to see you get flamed.
I don't think Dibrom would be too impressed with this stuff. Maybe you should back it up with a listening test of you own? I just don't want to see you get flamed.
The point of the exercise is to pinpoint the differences that uncoupled stereo has over coupled stereo rather than to compare coders subjectively (ie. this coder sounds better than that one). I'm not making any claims here, but just doing a bit of detective work and seeing if stereo coupling really is the cause of HF boost in Vorbis.
I know it's bad (I even stated it that this is bad) but I don't have good ears and have trouble hearing these distortions so I need an empirical way of observing the differences that guruboolez and [proxima] are noticing in their listening tests. Hopefully Dibrom will pardon me on this.
Will it be possible to "isolate" the HF boost?
Will it be possible to "add" the isolated HF boost to the uncompressed original?
Will the "spiked" original be ABXable against the original?
Pardon me if these are stupid questions, please..
Will the "spiked" original be ABXable against the original?
Pardon me if these are stupid questions, please..
It's quite a good idea actually. What I might do is take these differences, add them back to the original, and get some people with good ears to ABX them.
EDIT: I've uploaded three files where I have added the differences back to the original. The expectation is these will be ABXable with similar distortions.
The point of the exercise is to pinpoint the differences that uncoupled stereo has over coupled stereo rather than to compare coders subjectively (ie. this coder sounds better than that one). I'm not making any claims here, but just doing a bit of detective work and seeing if stereo coupling really is the cause of HF boost in Vorbis.
I know it's bad (I even stated it that this is bad) but I don't have good ears and have trouble hearing these distortions so I need an empirical way of observing the differences that guruboolez and [proxima] are noticing in their listening tests. Hopefully Dibrom will pardon me on this.
Fair enough. I suppose Guruboolez will provide the conclusive evidence in the form of a listening test soon anyway, I guess its nice to have some idea of what's going on in the mean time. Although, it still might not go over too well.
Fair enough. I suppose Guruboolez will provide the conclusive evidence in the form of a listening test soon anyway, I guess its nice to have some idea of what's going on in the mean time. Although, it still might not go over too well.
My choices are rather limited unfortunately and I'm unable to go into the Vorbis source code and try to tackle this HF boost problem by relying on subjective comments on quality only.
But we'll try kjoonlee's suggestion and see how it goes. If these corrupted samples are ABXable and distortions are similar in nature to the HF boost, then we've at least verified experimentally that stereo coupling has problems
It's quite a good idea actually. What I might do is take these differences, add them back to the original, and get some people with good ears to ABX them.
I think this is not useful because, in this manner, we are completely ignoring masking. I see no reason to abx a such sample. Maybe someone more expert can confirm my assumption.
I remember some Musepack spectral analysis where original and encoded differ a lot visually, but all the quantization noise is perfectly masked and the files are not ABXable.
,Feb 12 2004, 01:02 AM] I think this is not useful because, in this manner, we are completely ignoring masking. I see no reason to abx a such sample. Maybe someone more expert can confirm my assumption.
I remember some Musepack spectral analysis where original and encoded differ a lot visually, but all the quantization noise is perfectly masked and the files are not ABXable.
I sort of guessed it wouldn't be so simple.
Oh well, in that case, I give up on the HF boost problem. I guess there is little point in me trying to fix something I cannot hear that well.
I'll focus on the tonality problem in the Arche I sample which I can actually hear as well as pre-echo issues.
Indeed, I was a little hesitant to ask my questions because I don't understand noise masking very well, if at all.
Also, on second thought, I doubt if my suggestion would be very useful.
I'm sorry for the confusion.
I sort of guessed it wouldn't be so simple.
Oh well, in that case, I give up on the HF boost problem. I guess there is little point in me trying to fix something I cannot hear that well.
Before you give up - have you tried using frequency
analysis for comparing orignal vs. different encodes hf boost-wise, as I suggested in my last post?
Let me add two samples available here (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360). You can hear boosted close hi-hat cymbal in the 8823 sample with 1.0.1 -q4. The second sample is white noise created by CoolEdit. I suppose some of you disagree with the use of artificially created sample, but the problem with 1.0.1 -q4 is too obvious for me. I found the sample encoded by 1.0.1 -q4 totally different from the original. I'm not sure this sample exposes HF boost problem (or another problem?), but I believe it's a good material to improve Vorbis.
Actually, I created my "Modest Tuning" to improve these samples. Here's a part of my personal listening result (in Japanese only) (http://nyaochi.cocolog-nifty.com/audio/2004/01/aotuv_alpha3_.html) of 8823 sample (I focused on only HF boost) with Modest Tuning beta2 (mtb2), aoTuV alpha2 (at2), GarfTuned3 beta1 (gt), QKTune beta2 (qk2).
ABC/HR Version 0.9b, 30 August 2002
Testname: 8823
1L = C:\users\musics\test\compare\8823-q5qk2.wav (171kbps)
2L = C:\users\musics\test\compare\8823-q5at2.wav (158kbps)
3L = C:\users\musics\test\compare\8823-q5gt.wav (178kbps)
4L = C:\users\musics\test\compare\8823-q4mtb2.wav (169kbps)
---------------------------------------
General Comments:
---------------------------------------
1L File: C:\users\musics\test\compare\8823-q5qk2.wav
1L Rating: 3.0
1L Comment:
---------------------------------------
2L File: C:\users\musics\test\compare\8823-q5at2.wav
2L Rating: 3.5
2L Comment:
---------------------------------------
3L File: C:\users\musics\test\compare\8823-q5gt.wav
3L Rating: 2.5
3L Comment:
---------------------------------------
4L File: C:\users\musics\test\compare\8823-q4mtb2.wav
4L Rating: 4.5
4L Comment:
---------------------------------------
ABX Results:
Original vs C:\users\musics\test\compare\8823-q5qk2.wav
13 out of 14, pval < 0.001
Original vs C:\users\musics\test\compare\8823-q5at2.wav
13 out of 14, pval < 0.001
Original vs C:\users\musics\test\compare\8823-q5gt.wav
13 out of 14, pval < 0.001
Original vs C:\users\musics\test\compare\8823-q4mtb2.wav
12 out of 21, pval = 0.332
I haven't test these samples with uncoupled stereo but I'll later.
I sort of guessed it wouldn't be so simple.
Oh well, in that case, I give up on the HF boost problem. I guess there is little point in me trying to fix something I cannot hear that well.
Before you give up - have you tried using frequency analysis for comparing orignal vs. different encodes hf boost-wise, as I suggested in my last post?
No. It was 1 am in the morning and I was getting really tired so I gave up trying.
Let me add two samples available here (http://www.hydrogenaudio.org/forums/index.php?showtopic=18360). You can hear boosted close hi-hat cymbal in the 8823 sample with 1.0.1 -q4.
My ABXing of the uncoupled stereo hi-hat cymbal at q 4 though I thought the differences were so subtle, I often doubted myself in most cases.
-------------------------------------
WinABX v0.23 test report
02/12/2004 10:42:23
A file: E:\vsamples\8823.wav
B file: E:\vsamples\uncoupled.wav (155.27 kbps)
10:43:57 1/1 p=50.0%
10:44:11 2/2 p=25.0%
10:44:25 3/3 p=12.5%
10:44:34 3/4 p=31.2%
10:44:47 4/5 p=18.8%
10:45:33 5/6 p=10.9%
10:45:45 5/7 p=22.7%
10:46:06 6/8 p=14.5%
10:46:34 7/9 p= 9.0%
10:46:40 8/10 p= 5.5%
10:46:57 9/11 p= 3.3%
10:47:05 10/12 p= 1.9%
10:47:11 11/13 p= 1.1%
10:47:17 12/14 p= 0.6%
10:47:24 13/15 p= 0.4%
10:47:40 14/16 p= 0.2%
10:47:47 test finished
That is quite a nice sample for HF boost.
i hope you all get together an make vorbis the great encoder it was meant to be ... even if forking grom xiph ... the world NEEDS a free encoder ... thank you guys
,Feb 11 2004, 11:02 PM] I think this is not useful because, in this manner, we are completely ignoring masking. I see no reason to abx a such sample. Maybe someone more expert can confirm my assumption.
I remember some Musepack spectral analysis where original and encoded differ a lot visually, but all the quantization noise is perfectly masked and the files are not ABXable.
That's why you add in the original signal to the difference. That way you have the masker signals present at those time the difference signal would be masked. Just listening to the difference signal however would have the problem you mention. Right? But why not compare the uncoupled stereo with the "joint stereo" one directly. All other variables are constant right?
A listening may not help to fix the HF boost. I'm affraid one would need to dig deeply into the source until finds where the energy difference comes from.
A listening may not help to fix the HF boost. I'm affraid one would need to dig deeply into the source until finds where the energy difference comes from.
Maybe in this case listening cannot help to resolve codec problems, we need a capable person who can understand the code and correct this problem. Anyway, due to the availability of hardware Vorbis players, i think that testing is still important because :
- maybe we can suggest the newbies a best, "HA approved" version/128 kbps settings.
- there is the chance to reduce HF boost with "small hacks" such as disabling stereo coupling.
I’m late, but I didn’t have much free-time this week (and internet access is a second problem too). I have performed the test yesterday, comparing:
- RC3 (march 2002)
- 1.01 (october 2003)
- lossless coupling (Quantum Knot)
- uncoupled encoder (Nyaochi)
According to the fact that uncoupled encodings at –q4 are a lot bigger than traditionnal vorbis encoder at the same setting, I also add an uncoupled encoder at lower setting. Nyaochi uncoupled at –q2 is close to average bitrate of 1.01/RC3 –q4, but slightly inferior. I’ve therefore tested fully uncoupled encoder at –q2,3 in order to measure the negative impact of a sub-efficient (uncoupled) channel coupling at ~128 kbps, compared to the same encoder, helped with channel coupling.
Results – /|\ Don’t care about notation absolute signification: it isn’t ITU compliant
(http://audiotests.free.fr/tests/2004/vorbisoldnew3.png)
• uncoupled encoders are winning again, without any doubts.
• full uncoupled encoder [FU] (Nyaochi) is clearly superior to lossless coupled [LC] (provided by Quantum Knot) vorbis encoder. FU was always better than LC, except on one sample – Mars: same notation. These results confirm [proxima]’s conclusions.
• both uncoupled encoders are still suffering from noise or tonality difference. The problem is not fully corrected: there are still differences. It’s not awfully annoying, but it’s difficult to forgive at ~160 kbps...
• at ~128 kbps, 1.01 appears again as a complete loser, compared to RC3 (harpsichord is the exception again) and to FU –q 2,3. There are so much noise and imprecision with 1.01... In other words, 1.01 without coupling was always better than 1.01 with coupling!!!
• But FU –q 2,3 is far from perfection: there are much more noise than with FU at –q 4. Noise isn’t the only difference. Other artifacts appeared (for example, acidity on Spagna, distortions on Brahms). And pre-echo reached an annoying level.
• Therefore, there are no clear winner between RC3 –q4 and FU –q 2,3. On four samples, RC3 sounded better; on the four others, it was FU. Overall notation is in favor of RC3, but the difference is distorted by an approximate notation (with deliberately exaggerating contrasts).
I can only imagine the discussions that will arise when choosing codecs for the 128kbps multiformat test
Vorbis 1.0.1 vs. GT3 vs. QK vs. Nyaochi vs. CVS...
I can only imagine the discussions that will arise when choosing codecs for the 128kbps multiformat test
Vorbis 1.0.1 vs. GT3 vs. QK vs. Nyaochi vs. CVS...
Yes, probably.
But you could request to people demanding for x or y encoder a blind test: nobody will bother you anymore
I guess that uncoupled encoder can't be considered as real competitors. There were released in order to isolate a problem (hiss), and probably need tuning in order to increase their efficiency.
P.S. Thank you for your private messages (I'll answer you later)
Again, thanks to guruboolez for his testing.
The aim of testing lossless and uncoupled stereo is to probe the behaviour of the HF boost. I think from guruboolez and [proxima]'s tests, we can conclude that the lossy coupling (point) is having problems but is not the sole cause. We also see that uncoupled stereo is not perfect and still suffers from noise.
So perhaps, the uncoupling has only reduced the effect of the hiss but not totally removed it. That is, it hasn't made the problem go away or solved it, but only made it harder to detect. Hence I'm beginning to believe that we can rule out stereo coupling as the cause of HF boost. Digging into the source code of rc3 may provide new insight as well as nyaochi's focus on the noise companding in his 'Modest Tuning'.
I know there may be some other people on the Vorbis-dev mailing list who are saying 'I told you so', but hey, it never hurts to do some comprehensive listening tests to verify it.
I just came across results from these listening tests, performed by nyaochi. The big table in the middle of the page should be self explanatory.
http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html (http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html)
I just came across results from these listening tests, performed by nyaochi. The big table in the middle of the page should be self explanatory.
http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html (http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html)
Interesting. He also releases the 1.0.1 Modest Tuning beta 2 (http://homepage3.nifty.com/nyaochi/soft/dist/oggencmtb2.zip) used in these tests.
I just came across results from these listening tests, performed by nyaochi. The big table in the middle of the page should be self explanatory.
http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html (http://nyaochi.cocolog-nifty.com/audio/2004/02/128kbps.html)
Some notes for the test. I ranked my favorite 3 samples as well as 8 classical samples presented by guruboolez. The main purpose of this test is to evaluate aoTuV b1 -q4 compared with 1.0.1 official -q4, RC3 (March 2002) -q4, lossless coupling -q4, uncoupled stereo -q4/-q2.3, and my modest tuning -q4. Note that bitrates are not similar between these samples (LC -q4, UC -q4 and MTb2 -q4 is much higher than 128kbps.)
I failed to abx 2 samples because I was not familiar with classical samples and got tired.
Results:At a grance, the total score shows that MTb2 was the first. However, it was not so good because the average bitrate of MTb2 is too high (around 160-170kbps). On the other hand, aoTuV has a good cost-performance without bitrate inflation. Preecho problem (e.g., Harpsichord and Mandolins) seems to lower the score (preecho is not his main target of the tuning).
Just as the listening results of guruboolez and [proxima], 1.0.1 official ends up the last place. UC was better than LC.
The Modest Tuning Beta2 won my own test (http://cyberquebec.ca/harashin/florida_seq_Vorbis_test.html). Apparently it's superior to other Vorbis encoders on this kind of sample. (sharp attack, symbal)
Although aoTuV b1 is also thinkable as a good choice for its reasonable bitrate.
However, it was not so good because the average bitrate of MTb2 is too high (around 160-170kbps).
Perhaps next time you could compare it to 1.01 -q5 as well? Results would be interesting, IMO.
It certainly is surprising that lossless coupling is doing so badly compared with uncoupled. Theoretically, you'd expect they should be the same!
It certainly is surprising that lossless coupling is doing so badly compared with uncoupled. Theoretically, you'd expect they should be the same!
Indeed, it's very odd. However, I was able to ABX between LC and UC with score at 15/20. Also, they don't actually seem to be same.
lc_uc.png (http://cyberquebec.ca/harashin/lc_uc.png)
It certainly is surprising that lossless coupling is doing so badly compared with uncoupled. Theoretically, you'd expect they should be the same!
Indeed, it's very odd. However, I was able to ABX between LC and UC with score at 15/20. Also, they don't actually seem to be same.
lc_uc.png (http://cyberquebec.ca/harashin/lc_uc.png)
Monty made a good point in IRC that this is to be expected since compilers do random things to FP computations
[span style='font-size:15pt;line-height:100%']...One year later[/font][/span]
Just for curiosity, I've tried another comparison, using the same samples, in order to check vorbis's progress during three years, and especially during the last one.
I kept vorbis RC3 (march 2002), vorbis 1.01 (march 2003), and add the most advanced vorbis encoder: aoTuV beta 3. I'd like to check the amount of noise/coarseness audible with aoTuV compared to an old encoder which wasn't too affected by this problem.
results:
(http://audiotests.free.fr/tests/2005.02/results.png)
ABX log/scores are here (http://audiotests.free.fr/tests/2005.02/hybrid_log.zip).
There are still audible issues with aoTuV at this bitrate, but they are much lower than vorbis RC3. One exception: the last sample, which RC3 performed differently (and to my ears: better) on brass (micro-attacks sample).
Bravo to Aoyumi!
Wow. That is some nice improvement.
It looks like aoTuV is pushing Vorbis in the right direction instead of regressing it
There are still audible issues with aoTuV at this bitrate, but they are much lower than vorbis RC3. One exception: the last sample, which RC3 performed differently (and to my ears: better) on brass (micro-attacks sample).
[a href="index.php?act=findpost&pid=273435"][{POST_SNAPBACK}][/a]
The handling of a micro attack is one of the big subjects truly.
I know that QuantumKnot is performing the trial for solving this problem.
Probably, I am looking forward to it.