Skip to main content

Topic: Personal Listening Test of 2 Opus encoders (Read 28457 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • jmvalin
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of 2 Opus encoders
Reply #50
If You ask me I think it's a slight bitrate bump/increase rather than quality improvement.

P.S. I didn't notice any audible regressions.

Did you test other bitrates? The new code actually impacts all bitrates because it changes the way we compute the tonality (by using a larger window).

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #51
JM,

Only 32 kbps. Sorry, lack of time. I'll try another bitrate if will get some time.

  • jmvalin
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of 2 Opus encoders
Reply #52
Only 32 kbps. Sorry, lack of time. I'll try another bitrate if will get some time.

Thanks. 64 kb/s is probably the logical one to try since it's far from 32 kb/s but the effect of transient detection is still obvious.

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #53
I don't hear any improvement at 64 kbps either. And  bitrate tiny bit higher for "new tonality" build.   :-\

  • Kamedo2
  • [*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #54
I will be able to start testing Opus from 2016 Dec 31.

  • jmvalin
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of 2 Opus encoders
Reply #55
I don't hear any improvement at 64 kbps either. And  bitrate tiny bit higher for "new tonality" build.  :-\

As long as it's not worse, I'm OK since the new analysis code is nearly 2x cheaper to compute. Normally, it should also help on some of the bachpsichord notes that were too low for the other code to pick up.

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #56
@JM
Alright then.

BTW I was listening different songs/styles encoded by 1.2a at 32 kbps VBR and the quality was pretty good.
Except some songs with a lot of energy in HF (mostly cymbals). A styles like loud metal are affected most of all.
Quality breaks the same manner and often even worse than on a tonal stuff. A lot of smearing, distortion. Pretty unlistenable. :(  Maybe something can be done with rate distribution for LF/HF (just my guess).
1.2a doesn't detect it as something hard to encode and codes it at pretty flat bitrate 30-31 kbps.

Here sample: Metallica - Atlas, Rise!

P.S. 64 kbps was a different story. Tonal stuff is the only  most vulnerable (mostly). 

I would rather like to get rid of my remaining mp3 listening test sessions first. Now 24 sessions remaining and 56% done.
Lets try Opus. MP3 won't go anyway. :)


  • Last Edit: 11 December, 2016, 10:33:55 PM by IgorC

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #57
There is also an old issue.
Opus tends to increase bitrate on quiet passages where it's not justified. 

I've tried two outros of Pink Floyd "The Wall" (MFSL): "Goodbye Cruel World" and "Outside The Wall". Both are quiet songs.  Opus -32 kbps encodes them at 36 and 39 kbps respectively.  And there is no need for such bitrates as there aren't very audible (far from that) artifacts here.

  • jmvalin
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of 2 Opus encoders
Reply #58
BTW I was listening different songs/styles encoded by 1.2a at 32 kbps VBR and the quality was pretty good.
Except some songs with a lot of energy in HF (mostly cymbals). A styles like loud metal are affected most of all.
Quality breaks the same manner and often even worse than on a tonal stuff. A lot of smearing, distortion. Pretty unlistenable. :(  Maybe something can be done with rate distribution for LF/HF (just my guess).
1.2a doesn't detect it as something hard to encode and codes it at pretty flat bitrate 30-31 kbps.
Interestingly, I don't find these artefacts particularly objectionable. They're of course obvious, but I find them less annoying than many other artefacts at 32 kb/s. The main thing I'm noticing with the cymbals isn't so much pre-echo, but a tendency to sound tonal. Is that what you're hearing. In any case, I hacked two different changes to the encoder (not something immediately mergeable) to get your opinion. Can you listen to these files (A is the unmodified encoder):
https://jmvalin.ca/misc_stuff/Atlas32_A.wav
https://jmvalin.ca/misc_stuff/Atlas32_B.wav
https://jmvalin.ca/misc_stuff/Atlas32_C.wav

Quote
Opus tends to increase bitrate on quiet passages where it's not justified. 
That's a known issue, but it's hard to do anything about it because the encoder does not know what volume you're listening at. It could be that your volume knob is turned all the way up and it's no longer a quiet passage.

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #59
The main thing I'm noticing with the cymbals isn't so much pre-echo, but a tendency to sound tonal.Is that what you're hearing
Yes, it's right description.

I'll try samples.  Thanks.

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #60
https://jmvalin.ca/misc_stuff/Atlas32_A.wav
https://jmvalin.ca/misc_stuff/Atlas32_B.wav
https://jmvalin.ca/misc_stuff/Atlas32_C.wav

MOS of a blind test:

C  - 3.3
B  - 3.1
A - 3.0

MOS are the same for both tests (studio speakers/monitors and headphones)
Code: [Select]
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname:

1L = E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_B.wav
2L = E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_A.wav
3L = E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_C.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_B.wav
1L Rating: 3.1
1L Comment:
---------------------------------------
2L File: E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_A.wav
2L Rating: 3.0
2L Comment:
---------------------------------------
3L File: E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_C.wav
3L Rating: 3.3
3L Comment: stereo loss
---------------------------------------
ABX Results:

My comments (not actual log):

"Sample 3 vs Sample 2
  1 of   1, p = 0.500
  2 of   2, p = 0.250
  3 of   3, p = 0.125
  4 of   4, p = 0.063
  5 of   5, p = 0.031
TRAINING MODE -- not written to file"

I don't know why I perceive the C as stereo loss or energy change (?) . Anyway it sounds better.
  • Last Edit: 27 December, 2016, 11:37:13 AM by IgorC

  • Kamedo2
  • [*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #61
I just uploaded another experimental build that changes the tonality analysis to (hopefully) be more sensitive, especially when there's lots of closely-spaced harmonics. This change affects pretty much all rates and I'd be curious to have feedback on any improvements or regression it causes before I merge it in for 1.2. You can compare it to 1.2-alpha. Note that this new build also forces the encoder to consider everything as music.
I am planning a new listening test. Which version and bitrate should I test? Is this the version I should start testing now?

  • jmvalin
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of 2 Opus encoders
Reply #62
I just uploaded another experimental build that changes the tonality analysis to (hopefully) be more sensitive, especially when there's lots of closely-spaced harmonics. This change affects pretty much all rates and I'd be curious to have feedback on any improvements or regression it causes before I merge it in for 1.2. You can compare it to 1.2-alpha. Note that this new build also forces the encoder to consider everything as music.
I am planning a new listening test. Which version and bitrate should I test? Is this the version I should start testing now?
Yes, I think that's a good build to test. In terms of bit-rate, I'd say 32 to 48 kb/s is where it would be most interesting.

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #63
hi, Kamedo2,

There is also more recent build https://jmvalin.ca/misc_stuff/opus-tools-buildC.zip
It addresses the issue with Atlas sample here https://hydrogenaud.io/index.php/topic,111798.msg933126.html#msg933126

I'll try to make some time and test it on a different samples at 32-48-64 kbps. If You or someone else can test it even better.
As I understand an idea is to find some pattern how quality changes among different type of samples.
  • Last Edit: 07 January, 2017, 06:59:09 PM by IgorC

  • jmvalin
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of 2 Opus encoders
Reply #64
There is also more recent build https://jmvalin.ca/misc_stuff/opus-tools-buildC.zip
It addresses the issue with Atlas sample here https://hydrogenaud.io/index.php/topic,111798.msg933126.html#msg933126

I'll try to make some time and test it on a different samples at 32-48-64 kbps. If You or someone else can test it even better.
As I understand an idea is to find some pattern how quality changes among different type of samples.
I just uploaded a new build:
https://jmvalin.ca/misc_stuff/opus-tools-buildD.zip
It's playing with the same "allocation trim" parameter as build C, but instead of giving a slight boost to HF on all frames, it's trying to do it adaptively. Can you see how it compares to build C and 1.2-alpha?.

  • Kamedo2
  • [*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #65
It's playing with the same "allocation trim" parameter as build C, but instead of giving a slight boost to HF on all frames, it's trying to do it adaptively. Can you see how it compares to build C and 1.2-alpha?.
OK. Then I will test

https://jmvalin.ca/misc_stuff/opus-tools-exp-df24252.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildC.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildD.zip

at 36k and 48kbps.

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #66
I think it has more sense to test https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip.
I remember exp-df24252.zip uses hybrid mode at low bitrates while "analysis24k" does only CELT.
https://hydrogenaud.io/index.php/topic,111798.msg931347.html#msg931347


Don't know whether builds C and D use hybrid or just CELT.

The mode selection might interfere into test results.

  • jmvalin
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of 2 Opus encoders
Reply #67
I think it has more sense to test https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip.
I remember exp-df24252.zip uses hybrid mode at low bitrates while "analysis24k" does only CELT.
https://hydrogenaud.io/index.php/topic,111798.msg931347.html#msg931347
I guess analysis24k would be safer, but otherwise it's going to be pretty obvious if there's a mode change because the builds should normally sound relatively similar and just weight the noise slightly differently.
Quote
Don't know whether builds C and D use hybrid or just CELT.
Yes, builds C and D force CELT all the time.

BTW, just uploaded a new build:
https://jmvalin.ca/misc_stuff/opus-tools-buildE.zip
If you haven't started yet, please add that too. Otherwise, I'll use the feedback on what you have to adjust thing further.

Note that the only thing these different builds change is the bit allocation of LF vs HF, aka allocation trim. The rest is the same. Basically, I'm trying to dynamically adjust that trim depending on the audio.

  • Kamedo2
  • [*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #68
BTW, just uploaded a new build:
https://jmvalin.ca/misc_stuff/opus-tools-buildE.zip
If you haven't started yet, please add that too. Otherwise, I'll use the feedback on what you have to adjust thing further.
I tried.
These are the bitrate distributions of "opusenc --bitrate 36" and "opusenc --bitrate 48". There are almost no bitrate changes between builds. Only Bachpsichord at low bitrates, analysis24k uses fewer bits.
Are there any builds I can omit? Reducing encoders from 8 to 6 will significantly shorten test times.

https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildC.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildD.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildE.zip



BTW,
Code: [Select]
opusenc --version
is seemingly not working.
Code: [Select]
opus-tools-analysis24k\opusenc --version 
opusenc opus-tools 0.1.9-35-g49c5f27-dirty (using libopus 1.2-alpha)
Copyright (C) 2008-2013 Xiph.Org Foundation

opus-tools-buildC\opusenc --version
opusenc opus-tools 0.1.9-35-g49c5f27-dirty (using libopus 1.2-alpha)
Copyright (C) 2008-2013 Xiph.Org Foundation

opus-tools-buildD\opusenc --version
opusenc opus-tools 0.1.9-35-g49c5f27-dirty (using libopus 1.2-alpha)
Copyright (C) 2008-2013 Xiph.Org Foundation

opus-tools-buildE\opusenc --version
opusenc opus-tools 0.1.9-35-g49c5f27-dirty (using libopus 1.2-alpha)
Copyright (C) 2008-2013 Xiph.Org Foundation

  • kode54
  • [*][*][*][*][*]
  • Administrator
Re: Personal Listening Test of 2 Opus encoders
Reply #69
Version appears to be working, as those three versions appear to be "dirty" working trees, meaning they have uncommitted changes.

  • Kamedo2
  • [*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #70
It may be interesting to add mp3 at 2x the bitrate. (but I am running out of slots)

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #71
Or HE-AAC at the same rate.
The closest competitor to Opus at 32-48 kbps will be HE-AACv2/v1 (at the same bitrate). MP3 and Vorbis simply can't compete with it at those rates.

  • Kamedo2
  • [*][*][*][*]
Re: Personal Listening Test of 2 Opus encoders
Reply #72
https://jmvalin.ca/misc_stuff/opus-tools-buildE.zip
If you haven't started yet, please add that too.

BTW, is this build E slower than the other builds?
build E is about 40x realtime on the standard settings, while other builds reach nearly 50x .

Code: [Select]
bitrate	analysis24k	buildC	buildD	buildE
12 29.6 56.3 56.5 46.4
24 47.3 48.9 49.2 42.4
36 48.9 48.8 48.8 41.3
48 48.0 48.1 47.9 40.5
60 46.5 46.2 45.9 39.6

  • butrus
  • [*]
Re: Personal Listening Test of 2 Opus encoders
Reply #73
Quote
Opus tends to increase bitrate on quiet passages where it's not justified. 
That's a known issue, but it's hard to do anything about it because the encoder does not know what volume you're listening at. It could be that your volume knob is turned all the way up and it's no longer a quiet passage.

Maybe there could be an optional switch in the encoder where you can set the anticipated absolute loudness of the playback (or the anticipated absolute level of the ambient noise)?

E.g. if you know you're encoding files to be heard in the car this would enable the encoder to optimize the encoding process for this purpose and spare some bits...

  • jmvalin
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of 2 Opus encoders
Reply #74
Maybe there could be an optional switch in the encoder where you can set the anticipated absolute loudness of the playback (or the anticipated absolute level of the ambient noise)?

E.g. if you know you're encoding files to be heard in the car this would enable the encoder to optimize the encoding process for this purpose and spare some bits...

This is something I've considered in the past and eventually decided against simply because the gain are minuscules (over a large and varied corpus), while the potential for misuse (or just wrong encoder decisions) is huge.