Skip to main content
Topic: Opus 1.3-beta is here (Read 25456 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Opus 1.3-beta is here

Reply #125
 You have done a lot of interesting tests. Thank you.

The thing is that developers are interested in comparison between oldtf vs newtf2.
It’s already know that newtf2 or any other lossy files won’t be transparent at those rates but still can have high quality.

ABC/HR should be used in order to compare “old tf” vs “newtf2”. Not ABX comparator.
Here is some info on ABC/HR https://web.archive.org/web/20110611013208/http://ff123.net/64test/practice.html

And other thing. 5/5 (probability of guessing, p=3%) is already enough for ABX log in testing lossy codecs. It’s even more true for ABC/HR. Nobody pretends to think that you fake a results or try multiple times in order to get “valid” results.

You can probably download some ABC/HR app for your Linux distro. 
Here is one http://listening-tests.hydrogenaud.io/igorc/aac-96-a/ABC-HR_bin.zip  
Some documentation http://listening-tests.hydrogenaud.io/igorc/aac-96-a/readme.txt
(Well, Java can give some headache)

Testing Opus newtf2 vs oldtf is hard because differences between them are rather small at least in this moment.

 :)

Re: Opus 1.3-beta is here

Reply #126
The important thing isn't the exact collection itself, it's having some calibration. Otherwise, I could simply "improve" Opus by just increasing the bitrate for the same command-line arguments. With the calibration, I ensure that any increase in bitrate for a particular file is compensated by a decrease in other files. The result is that different versions should give about the same overall bitrate on your collection, even if it differs from mine (and even if your average bitrate differs from mine).

I would appreciate if I got (more or less) constant quality for different encoder versions no matter what the bitrate is :-(

Re: Opus 1.3-beta is here

Reply #127
Focussing on the default setting, unconstrained VBR --vbr, and ignoring the little-used constrained CBR --cvbr or strict CBR --hard-cbr

The developers stuck to the default philosophy of unconstrained VBR in letting the bitrate fluctuate greatly and fall where it may for a specific track or section of audio, depending on the difficulty of encoding with the same target quality.

However, they differ in philosophy from many popular encoders like Ogg Vorbis, LAME, etc. as the Opus developers calibrate the quality scale to achieve a certain bitrate specified by the user, averaged over a wide and diverse collection of audio. Most encoders using unconstrained VBR have some arbitrary quality metric, though I get the impression that iTunes TVBR AAC quality setting is calibrated to the equivalent bitrate per audio channel so would be closer to the Opus approach.  Any independently developed Opus encoders could take the approach of their choice.

Then when a new version comes out with no regressions (and this thread indicates how careful they are to avoid regressions), the bitrate at the same setting will be the same but the quality will be at least the same or higher. This might be the same in iTunes TVBR AAC if improved versions were released. In other codecs such as LAME or Vorbis with unconstrained VBR, the same setting in an improved version is likely to have the same quality but at the same or slightly lower bitrate overall.

The exception to this can be problem samples, where for that sample alone (but not the diverse collection of audio), the fix may recognise that a higher bitrate is needed for the affected frames, whereas before the fix it didn't allocate enough bits to the right features to encode adequately and caused an artifact.

For me, this is just as valid and bitrate is easier to understand than some arbitrary quality scale, and I appreciate the significant improvements made in the sub-64kbps area and the artifact fixes in the 64kbps+ area, but I can understand the other point of view too.

If you typically use 64 or 96 kbps or higher, I'd say the quality is only marginally changed except that more problem samples have been fixed. If you typically use bitrates of 48kbps or lower, I'd say bitrate is more important than quality and you'd accept some differences from the original that can be noticed but aren't annoying, but you could choose to lower bitrate significantly compared to v1.0 or v1.1 and still achieve music that's good to listen to with unannoying differences from the original.
Dynamic – the artist formerly known as DickD


Re: Opus 1.3-beta is here

Reply #129
I rarely sit on Windows so if you want to enable more of us to give you opinions then start with supplying at least some Linux compiles, thanks.
I also develop on Linux. In fact, I even build these Windows binaries from Linux. If you'd like to test, the easiest is probably to grab commits from Git. The "old tf" commit is edaafe3 and the "newtf2" commit is 195ea42.

 

Re: Opus 1.3-beta is here

Reply #130
However, they differ in philosophy from many popular encoders like Ogg Vorbis, LAME, etc. as the Opus developers calibrate the quality scale to achieve a certain bitrate specified by the user, averaged over a wide and diverse collection of audio. Most encoders using unconstrained VBR have some arbitrary quality metric, though I get the impression that iTunes TVBR AAC quality setting is calibrated to the equivalent bitrate per audio channel so would be closer to the Opus approach.  Any independently developed Opus encoders could take the approach of their choice.
Actually, the Vorbis -q values are calibrated to give specific bitrates on a large collection of music. You can see them on the Vorbis Wikipedia page.

Then when a new version comes out with no regressions (and this thread indicates how careful they are to avoid regressions), the bitrate at the same setting will be the same but the quality will be at least the same or higher. This might be the same in iTunes TVBR AAC if improved versions were released. In other codecs such as LAME or Vorbis with unconstrained VBR, the same setting in an improved version is likely to have the same quality but at the same or slightly lower bitrate overall.
So keep in mind here that all of the improvements between 1.0 and 1.2 probably (my opinion, not measured) amount to ~10% reduction in bitrate for the same quality over a period of ~5 years. To keep some kind of equal-quality option properly calibrated across version, we would need very accurate listening tests at multiple bitrates. The effort involved in conducting those tests would probably be greater than the effort involved in making the quality improvements in the first place. It would also delay any new release by a year or so. That's why nobody does that.

Re: Opus 1.3-beta is here

Reply #131
So keep in mind here that all of the improvements between 1.0 and 1.2 probably (my opinion, not measured) amount to ~10% reduction in bitrate for the same quality over a period of ~5 years.
I'd prefer Opus 1.2.1@96 kbps over  Opus 1.0@112 kbps.  That's ~15%
1.2.1@64 kbps over 1.0@72-75 kbps.  That's ~15%
1.2.1@32 kbps over 1.0@40-42.  That's 25-30%+!


Re: Opus 1.3-beta is here

Reply #132
  • Mixa04
  • 1.3b (libopus 1.3-beta, libopusenc version 0.1.1-dirty)
  • 160kbps
  • Focusrite Scarlett 2i2 Gen2 headphone output, AKG K240 Studio 55 ohms
I just want to let you know that I was able to ABX at 160kbps opus 1.3b (not the newtf's). In this file I was detected that a ride cymbal that was played with the tip and mostly in the left channel and was delay to the right to create Haas effect had the transients on the right channel weakened compare to the unaltered left channel. Imagine that you would make some of the hits pianossimo (pp) instead of them all being forte.
Code: [Select]
foo_abx 2.0.4 report
foobar2000 v1.4 beta 2
2018-02-06 14:54:33

File A: mixa04.flac
SHA1: 3409e6a76cf6fc66bc64a7fbb8fa9cb8b182bb9c
Gain adjustment: +1.51 dB
File B: mixa04_160kbps_cvbr.opus
SHA1: f2d5313ece1132f54ccb97771291678ba97f5863
Gain adjustment: +1.51 dB

Output:
ASIO : Focusrite USB ASIO
Crossfading: NO

14:54:33 : Test started.
14:55:21 : 00/01
14:55:35 : 01/02
14:56:02 : 02/03
14:56:18 : 03/04
14:56:32 : 04/05
14:56:58 : 05/06
14:57:18 : 06/07
14:57:35 : 07/08
14:58:12 : 08/09
14:58:37 : 09/10
14:58:37 : Test finished.

 ----------
Total: 9/10
Probability that you were guessing: 1.1%

 -- signature --
05b99a0810f8db11ca5ae89782960e0354826703

I attached a sample of the problematic ride cymbal. Encode to opus and listen between 00:01 and 00:05. Some hits always sound let their initial attacks have been removed. I tried mp3 at V0 and ogg at high bitrates and they didn't manage it either. I could not successfully ABX the sample when I used opus and 192kbps.

Re: Opus 1.3-beta is here

Reply #133
Until now I could ABX a difference between newTF2 vs oldTF only on one sample.

Right now I'm at the point to start to listen white noise and hear some human voices .... from hell 
Yes, differences are that small.

Files

Code: [Select]
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname:

1R = E:\DETODO\Audio\When The Levee Breaks\When The Levee Breaks Opus 96k old TF.wav
2L = E:\DETODO\Audio\When The Levee Breaks\When The Levee Breaks Opus 96k NEW TF2.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: E:\DETODO\Audio\When The Levee Breaks\When The Levee Breaks Opus 96k old TF.wav
1R Rating: 4.0
1R Comment:
---------------------------------------
2L File: E:\DETODO\Audio\When The Levee Breaks\When The Levee Breaks Opus 96k NEW TF2.wav
2L Rating: 4.1
2L Comment:
---------------------------------------
ABX Results:


---------------------------------------
MY NOTES. ABX training log of "old TF.wav" vs "NEW TF2.wav"
---------------------------------------
1 of   1, p = 0.500
  2 of   2, p = 0.250
  3 of   3, p = 0.125
  3 of   4, p = 0.313
  4 of   5, p = 0.188
  5 of   6, p = 0.109
  6 of   7, p = 0.063
  7 of   8, p = 0.035
TRAINING MODE -- not written to file

P.S. As for this sample I think newTF2 is just a little bit more constant on backround, less splashy (less wavy).

Re: Opus 1.3-beta is here

Reply #134
Igorc, the couple of seconds in the beginning I mentioned when it sounds like it is tuning the high frequencies. What is that? It goes from very audible in the beginning during to noise calming down after a few seconds.

Re: Opus 1.3-beta is here

Reply #135
@ punkrockdude

Bro, could you please test this sample against Apple AAC @160 kbps as well please.

Re: Opus 1.3-beta is here

Reply #136
@ Leo 69
No problem at all but do you mean the sample with the ride having a Haas effect or the voice sample containing noise? I don't have Apple AAC Encoder and rarely use Windows except for when I record or mix and now that is rare since Reaper has unofficial versions for Linux.

Wait, since you said 160kbps I suspect you mean the ride sample since I could almost ABX it using opus. If you encode it and attach it in a message then I will ABX it for you.

Re: Opus 1.3-beta is here

Reply #137
Igorc, the couple of seconds in the beginning I mentioned when it sounds like it is tuning the high frequencies. What is that? It goes from very audible in the beginning during to noise calming down after a few seconds.
Provide a link to your original message and/or sample which You are talking about because honestly I can't follow every word in 6-page disscussion.

Code: [Select]
foo_abx 2.0.4 report
foobar2000 v1.4 beta 2
2018-02-06 14:54:33

File A: mixa04.flac
SHA1: 3409e6a76cf6fc66bc64a7fbb8fa9cb8b182bb9c
Gain adjustment: +1.51 dB
File B: mixa04_160kbps_cvbr.opus
SHA1: f2d5313ece1132f54ccb97771291678ba97f5863
Gain adjustment: +1.51 dB

Output:
ASIO : Focusrite USB ASIO
Crossfading: NO

14:54:33 : Test started.
14:55:21 : 00/01
14:55:35 : 01/02
14:56:02 : 02/03
14:56:18 : 03/04
14:56:32 : 04/05
14:56:58 : 05/06
14:57:18 : 06/07
14:57:35 : 07/08
14:58:12 : 08/09
14:58:37 : 09/10
14:58:37 : Test finished.

 ----------
Total: 9/10
Probability that you were guessing: 1.1%

 -- signature --
05b99a0810f8db11ca5ae89782960e0354826703

mixa04_160kbps_cvbr.opus

Any particular reason for CVBR instead of VBR?

Opus CVBR is equivalent to AAC CBR (ABR with very little variation/window)

Re: Opus 1.3-beta is here

Reply #138
Quote
Wait, since you said 160kbps I suspect you mean the ride sample since I could almost ABX it using opus. If you encode it and attach it in a message then I will ABX it for you.

It's worth noting that the output of libopus 1.2.1 and 1.3 beta(with fixes) is identical when encoding at --bitrate 160.

Apple AAC encoding at a slightly higher bitrate.


Re: Opus 1.3-beta is here

Reply #140
It's worth noting that the output of libopus 1.2.1 and 1.3 beta(with fixes) is identical when encoding at --bitrate 160.
I ABXed lossless vs opus and not one version of opus to another version.

I know. I just wanted to make it clear that there is no beta regressions here.

I didn't notice your opus encode is CVBR though (why?). Luckily, the output is also identical between 1.2.1 and 1.3-beta with --cvbr --bitrate 160.

Here is the sample encoded with qaac -v 160 (actual bitrate is 176kbps). Although, I agree with Igor that this is not a very useful comparison.

Side note: You can use qaac in GNU/Linux via wine. That's what I do.

Re: Opus 1.3-beta is here

Reply #141
Why I used CVBR in this case was just to try out the different options and see if it made any audible reason. I didn't detect any. I will ABX the AAC file asap.


Re: Opus 1.3-beta is here

Reply #143
@jmvalin the main problems with the opus codec at bitrates of 48kbps is with violins and certain other strings (real & electronic) . Would it be possible for your team to fix that issue? Every other song sounds transparent at that bitrate except those ones

Re: Opus 1.3-beta is here

Reply #144
@jmvalin the main problems with the opus codec at bitrates of 48kbps is with violins and certain other strings (real & electronic) . Would it be possible for your team to fix that issue? Every other song sounds transparent at that bitrate except those ones
Yes, I'm aware of the problem. So far I haven't found any way to improve that, but I'm still looking. I already have a few samples, but if you have a short clip that makes the problem particularly obvious, it's always useful.

Re: Opus 1.3-beta is here

Reply #145
Some results for 1.3 beta
Opus 1.2.1 - 1.3 beta vs HE-AAC  ( Test at 32 kbps)
Files
There are some improvements ( "Can't wait" and "Fatboy" samples) but also some regressions. Most notable regression is on sample 4º.

OK, I'm back tracking this regression with two builds:
These will only affect the speech samples, so no need to test the music ones. In fact, the "regression" is due to an improvement in the speech/music detector. Previously the speech samples would get mostly coded with CELT and in 1.3-beta they're being coded with hybrid. It just turns out that hybrid could do better, which I'm trying to fix here. I couldn't really hear much artefacts on sample 4, but the ones on sample 7 (Korean speech) were pretty obvious. Last detail, make sure you decode with the opusdec I'm providing (either is fine) because they have the "recent" decoder update (RFC 8251) enabled, which may not be the case with all players. Let me know if either or both these builds address what you were hearing.
(For non-Win32 users, just checkout the Git hash in the file name and build that)

Re: Opus 1.3-beta is here

Reply #146
Hi everybody,

I'm Julio. I've been lurking HA for a looong time (back from the "Project Mayhem" era in the early 2000s, with the orange board). I finally created an account today because I like Opus and recently have found some free time to volunteer with listening tests. I think I can help with the low-bitrate tuning effort @jmvalin has been working on.

As somebody who has never done any ABC/HR or ABX tests (ironically!): can you help me with recommended resources on how to set up everything, in terms of installing software and preparing the encoded samples for the listening tests? I use Windows 10. Also, let me know if there are any samples you want me to focus on first, and which Opus encoders should I use to encode and compare the samples. I'll give it a shot over the long weekend.

Re: Opus 1.3-beta is here

Reply #147
Hi everybody,

I'm Julio. I've been lurking HA for a looong time (back from the "Project Mayhem" era in the early 2000s, with the orange board). I finally created an account today because I like Opus and recently have found some free time to volunteer with listening tests. I think I can help with the low-bitrate tuning effort @jmvalin has been working on.

As somebody who has never done any ABC/HR or ABX tests (ironically!): can you help me with recommended resources on how to set up everything, in terms of installing software and preparing the encoded samples for the listening tests? I use Windows 10. Also, let me know if there are any samples you want me to focus on first, and which Opus encoders should I use to encode and compare the samples. I'll give it a shot over the long weekend.

Hi Julio! And, welcome to the club. What's your audio gear?

@punkrockdude

Hi man, did you have a chance to test that AAC sample?

Re: Opus 1.3-beta is here

Reply #148
Leo '69'

What part of TOS5 You don't understand
Quote
Submission of a thread that is not related to any of the topics covered by the Hydrogenaudio forums and subforums, or posts which are not related to the thread that they are submitted to, is against the rules. If you wish to discuss Off-Topic issues, please use our Off-Topic forum. Repeated violation of this rule after one notification may lead to administrative action including banishment.

Re: Opus 1.3-beta is here

Reply #149
Leo '69'

What part of TOS5 You don't understand
Quote
Submission of a thread that is not related to any of the topics covered by the Hydrogenaudio forums and subforums, or posts which are not related to the thread that they are submitted to, is against the rules. If you wish to discuss Off-Topic issues, please use our Off-Topic forum. Repeated violation of this rule after one notification may lead to administrative action including banishment.

So not even being a moderator here, You decided to publicly charge me with some imaginary violations? While I DO respect your reputation on these forums and couple of others, you should stop going overboard.

You'd better answer the guy, who sincerely wants to help improve the already great codec and devote a lot of time to this. Instead of being a douche. Sorry for being so straightforward, but that's me.

 
SimplePortal 1.0.0 RC1 © 2008-2018