Hydrogenaudio Forums

Hydrogenaudio Forum => Listening Tests => Topic started by: Kamedo2 on 2016-04-30 18:31:30

Title: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-04-30 18:31:30
Abstract:
Blind sound quality comparison between experimental and released Opus encoders, at 32 kbps, 48 kbps, 64 kbps and 80kbps.

Encoders:
Opus 1.1, with opus-tools 0.1.9 https://archive.mozilla.org/pub/opus/win32/opus-tools-0.1.9-win32.zip
Opus 1.1.1-rc-49-g5db9e14 branch exp_lbr_tune, with opus-tools 0.1.9.

Settings:
opusenc --bitrate 32 in.wav out.opus
opusenc --bitrate 48 in.wav out.opus
opusenc --bitrate 64 in.wav out.opus
opusenc --bitrate 80 in.wav out.opus

opusenc --bitrate 32 in.wav out.opus
opusenc --bitrate 48 in.wav out.opus
opusenc --bitrate 64 in.wav out.opus
opusenc --bitrate 80 in.wav out.opus

Samples:
Total 15 samples from my corpus (https://hydrogenaud.io/index.php/topic,98003.msg815018.html), tested twice per sample.

Hardwares:
Sony PSP-3000 + RP-HT560(1st), RP-HJE150(2nd).

Results:
(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_exp_compare_en.png)
(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_exp_compare_table_en.png)
The change of scores in exp_lbr_tune. They are mostly positive.
(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_exp_compare_diff_en.png)
The bitrate vs quality graph.
(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_exp_compare_graph_en.png)

Conclusions & Observations:
The Opus development branch exp_lbr_tune, has solid quality improvements in 32kbps and 48 kbps.
In 64 kbps and 80kbps, there are no big difference in quality between the exp_lbr_tune and the trunk Opus 1.1.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 1.11E-016 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              119          92.40
Testers (blocks)    14           6.66
Codecs eval'd        7          81.87   11.70   296.65  1.11E-016
Error               98           3.86    0.04
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.144

Means:

exp80k   org80k   org64k   exp64k   exp48k   org48k   exp32k   org32k  
  4.14     4.13     3.90     3.87     3.23     3.05     2.24     1.82  

---------------------------- p-value Matrix ---------------------------

         org80k   org64k   exp64k   exp48k   org48k   exp32k   org32k  
exp80k   0.854    0.001*   0.000*   0.000*   0.000*   0.000*   0.000*  
org80k            0.002*   0.001*   0.000*   0.000*   0.000*   0.000*  
org64k                     0.714    0.000*   0.000*   0.000*   0.000*  
exp64k                              0.000*   0.000*   0.000*   0.000*  
exp48k                                       0.015*   0.000*   0.000*  
org48k                                                0.000*   0.000*  
exp32k                                                         0.000*  
-----------------------------------------------------------------------

exp80k is better than org64k, exp64k, exp48k, org48k, exp32k, org32k
org80k is better than org64k, exp64k, exp48k, org48k, exp32k, org32k
org64k is better than exp48k, org48k, exp32k, org32k
exp64k is better than exp48k, org48k, exp32k, org32k
exp48k is better than org48k, exp32k, org32k
org48k is better than exp32k, org32k
exp32k is better than org32k


Raw data:
Code: [Select]
exp32k	exp48k	exp64k	exp80k	org32k	org48k	org64k	org80k	
1.650 3.100 3.500 3.950 1.800 2.800 3.650 4.000
2.600 3.650 4.100 4.350 1.950 3.200 4.150 4.350
2.350 3.400 4.000 4.250 1.550 3.100 3.900 4.200
1.700 2.900 3.750 4.050 1.700 2.850 3.950 4.000
3.050 3.700 4.200 4.300 2.000 3.400 4.100 4.250
1.750 2.350 3.450 3.700 1.500 2.400 3.550 3.550
2.650 3.250 4.150 4.250 2.000 3.200 4.150 4.250
2.100 3.450 3.900 4.200 1.750 3.150 3.950 4.200
2.350 3.100 3.450 3.700 1.900 2.850 3.500 3.700
1.650 2.450 3.800 4.250 1.700 2.300 3.750 4.250
2.200 3.200 3.850 4.300 1.800 3.050 3.950 4.300
2.150 3.350 4.050 4.100 1.650 3.050 4.000 4.150
2.650 3.500 4.050 4.400 1.900 3.600 4.000 4.400
1.950 3.450 3.950 4.150 1.650 3.200 4.000 4.150
2.750 3.600 3.850 4.150 2.450 3.600 3.850 4.150
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples HearttoHeart Pops
%samples Tom'sDiner Acappella


Bitrates:
The exp_lbr_tune branch has more bitrate deviations down to 32 kbps.
(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_exp_compare_rate_en.png)
Code: [Select]
%bitrate
exp32k exp48k exp64k exp80k org32k org48k org64k org80k
36749 54937 73371 92363 33033 52076 73438 92430
48355 68543 87084 106794 34026 59325 87161 106872
37576 56210 74723 93189 33190 52377 74764 93229
35411 53122 70453 87718 33374 51285 70523 87786
42211 62890 82978 103395 33742 56865 83108 103540
34296 50768 67697 84163 33250 50278 67966 84437
31638 47318 61532 75075 33409 48583 61552 75096
38539 57412 76608 95888 33395 53455 76761 96040
41265 60008 78435 97111 34648 55392 78772 97397
36911 54502 72279 90022 33483 51861 72311 90054
34338 50941 67863 85263 33064 49980 68072 85471
38476 56844 74928 92831 34037 53583 75361 93264
36575 54928 72355 89386 33628 52318 72495 89527
33863 51076 67929 85114 33186 50203 67987 85174
33287 50391 75053 88353 33507 51445 75864 89179

%album bitrate
33564 50005 66320 82681 32480 49071 66477 82842

Title: Re: Personal Listening Test of 2 Opus encoders
Post by: saratoga on 2016-04-30 18:34:54
Are you blinded to which encoder/bitrate is used for which file?
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-04-30 18:50:49
Are you blinded to which encoder/bitrate is used for which file?
Yes. This test is fully blinded, with 4*2 = 8 samples shuffled in a single ABC/HR session. 30 sessions were performed in this test.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: eahm on 2016-04-30 21:20:22
Damn nice work! Thank you Kamedo2.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-05-01 03:03:28
Thanks very much for this test. It not only validates my work in progress, but it also provides me with some clues for further improvements. I'm also interested in any other (qualitative) information you might have. For example, do you think the improvement at 32 kbps is mostly due to reduced noise/roughness or to the wider stereo image (or something else)?
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-05-01 06:09:29
For example, do you think the improvement at 32 kbps is mostly due to reduced noise/roughness or to the wider stereo image (or something else)?
They are of about equal contribution. I've got the good stereo image representation and reduced noise in the "better 32k" version. They are both great!
On 48 kbps, only the reduced noise/roughness was significant.
(Although the test is blinded, I could easily tell the 32k and 48k groups.)
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: scharfis_brain on 2016-05-02 20:00:57
Where can I obtain a binary of this experimental Encoder from?
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-05-07 02:19:09
I've been doing more (highly experimental) work on very low bitrates. Here's three files on which I'd be interested in getting feedback:
http://jmvalin.ca/misc_stuff/collapse0.wav
http://jmvalin.ca/misc_stuff/collapse1.wav
http://jmvalin.ca/misc_stuff/collapse2.wav
The original is at http://jmvalin.ca/misc_stuff/comp_stereo48.wav but it shouldn't be needed for such low rate.
I'm interested both in which one you think sounds best and the qualitative aspect. Trying to see if I'm on the right track here...
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-05-09 16:01:03
Jmvalin, I tested them but the results were not significant.
Code: [Select]
ABC/HR for Java, Version 0.53a, 09 5月 2016
Testname: jmvalin_opus_111798_20160506

Tester: Kamedo2

1L = C:\Users\PCC\Documents\ABC-HR\collapse1.wav
2R = C:\Users\PCC\Documents\ABC-HR\collapse2.wav
3R = C:\Users\PCC\Documents\ABC-HR\collapse0.wav

Ratings on a scale from 1.0 to 5.0

---------------------------------------
General Comments: Hard to say which is better. Not significant.
---------------------------------------
1L File: C:\Users\PCC\Documents\ABC-HR\collapse1.wav
1L Rating: 2.7
1L Comment: (4)Mid and LF are muddy.
---------------------------------------
2R File: C:\Users\PCC\Documents\ABC-HR\collapse2.wav
2R Rating: 2.7
2R Comment: (2) Less noise (6)More HF noise
---------------------------------------
3R File: C:\Users\PCC\Documents\ABC-HR\collapse0.wav
3R Rating: 2.6
3R Comment: (1) Mid Freq. is muddy.(3) Pre and Post echo problem. (6)HF noise and post-echo
---------------------------------------

ABX Results:
score per each track.
Code: [Select]
collapse1	collapse2	collapse0
2.8 2.9 2.7
3 3.1 3
2.2 2.2 2
2.3 2.5 2.6
2.8 2.8 2.8
3 2.6 2.7
=================
2.683333333 2.683333333 2.633333333
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-05-09 21:42:29
While a differences are subtle probably Collapse1 is somewhat better (less noisy):

Scores:
Collapse0 - 2,9833
Collapse1 - 3,0333
Collapse2 - 2,9833

Code: [Select]
	
Collapse0 Collapse1 Collapse2 Observation
01 Guitar intro 3,3 3,4 3,4 Coll 0 has a bit more distortion/ hoarse noise
02 Drum into 2,6 2,7 2,7 Coll 0 has a bit more distortion/ hoarse noise
03 hihat 3,1 3,2 3
04 sp. Guitar 3 3,1 3
05 wood drums 3,1 2,9 3
06  female voice 2,8 2,9 2,8

Average Score 2,983333333 3,033333333 2,983333333
Geometric Mean 2,974506988 3,024765597 2,975505423
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-05-10 05:33:59
Kamedo2, IgorC, thanks for the feedback. It looks like the effect I'm trying to produce is still a bit too subtle. I uploaded a variant on collapse1:
http://jmvalin.ca/misc_stuff/collapse1b.wav
Can you guys compare to collapse0 and see if you can hear more of an improvement?
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-05-10 12:52:04
Can you guys compare to collapse0 and....
Did you mean collapse1? collapse0 seems to be the worst candidate.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-05-10 17:41:03
This is the average of the two results above.
Code: [Select]
		collapse0	collapse1	collapse2
01 Guitar intro 3.00 3.10 3.15
02 Drum into 2.80 2.85 2.90
03 hihat 2.55 2.70 2.60
04 sp. Guitar 2.80 2.70 2.75
05 wood drums 2.95 2.85 2.90
06 female voice 2.75 2.95 2.70
(https://ss1.coressl.jp/listening-test.coresv.net/img2/111798_opus_collapse.png)
I like the behavior of the yellow(collapse1) because it is the best in the worst performing sample(03.hihat).
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-05-10 17:56:01
Can you guys compare to collapse0 and....
Did you mean collapse1? collapse0 seems to be the worst candidate.

I meant comparing to collapse0 since that's the "baseline" (i.e. what you tested in exp_lbr_tune). Though I guess you could also compare to collapse1 too.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-05-11 13:02:15
Where can I obtain a binary of this experimental Encoder from?
Sorry for the late reply,
http://listening-test.coresv.net/opus-tools0.1.9usinglibopus1.1.1-rc-49-g5db9e14.zip
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: lithoc on 2016-05-14 11:19:14
Can you guys compare to collapse0 and....
Did you mean collapse1? collapse0 seems to be the worst candidate.

I meant comparing to collapse0 since that's the "baseline" (i.e. what you tested in exp_lbr_tune). Though I guess you could also compare to collapse1 too.

The binaries have dependancy libopus-0.dll missing
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: 2012 on 2016-05-16 21:48:53
Opus-tools 0.1.9 Windows binaries (win32 + win64).
Linked against the experimental branch exp_lbr_tune.

Statically linked against:
* libopus (branch: exp_lbr_tune, commit: 4ed368e)
* libflac  1.3.1
* libogg  1.3.2

Built in a safe GNU/Linux environment with:
mingw-w64 4.0.6 / GCC 6.1.1

https://archive.org/download/opus-tools-exp_lbr_tune-g4ed368e/opus-tools-exp_lbr_tune-g4ed368e.zip
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-05-22 16:26:07
Jmvalin, I had the time to test collapse1b.
Collapse1 seems to be the best.

Code: [Select]
collapse0	collapse1	collapse1b
3.0 2.9 3.2
2.8 3.1 3.0
2.7 2.8 2.6
2.9 2.8 2.7
3.0 3.0 2.9
2.8 2.9 3.0

Average: 2.866 2.916 2.9
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-05-23 06:12:54
Collapse1b  - 2,78
Collapse1 - 2,77
Collapse0 - 2,74

Code: [Select]
	Collapse0	Collapse1	Collapse1b
01 Guitar intro 3,2 3,4 3,3
02 Drum into 2,4 2,3 2,4
03 hihat 2,55 2,6 2,5
04 sp. Guitar 2,7 2,8 2,75
05 wood drums 3 2,9 3
06  female voice 2,6 2,6 2,7

Average 2,742 2,767 2,775
Geometric Mean 2,728 2,747 2,759
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-05-23 11:31:12
This is the average of the two results above.
Code: [Select]
		collapse0	collapse1	collapse1b
01 Guitar intro 3.10 3.15 3.25
02 Drum into 2.60 2.70 2.70
03 hihat 2.63 2.70 2.55
04 sp. Guitar 2.80 2.80 2.73
05 wood drums 3.00 2.95 2.95
06 female voice 2.70 2.75 2.85
(https://ss1.coressl.jp/listening-test.coresv.net/img2/111798_opus_collapse2.png)
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-05-28 18:27:12
Kamedo2, IgorC, thanks for the feedback. It looks like the effect I'm trying to produce is still a bit too subtle. I uploaded a variant on collapse1:
http://jmvalin.ca/misc_stuff/collapse1b.wav
Can you guys compare to collapse0 and see if you can hear more of an improvement?
BTW was it 48 kb?

Your set of samples has different kind of samples. That's excellent. But if it's possible also to try my set of samples for future tests as well, I will be glad.  Samples (https://drive.google.com/folderview?id=0ByvUr-pp6BuURktPc255Wi1mYlk&usp=sharing)

Thank You.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-05-30 20:51:21
BTW was it 48 kb?

Your set of samples has different kind of samples. That's excellent. But if it's possible also to try my set of samples for future tests as well, I will be glad.  Samples (https://drive.google.com/folderview?id=0ByvUr-pp6BuURktPc255Wi1mYlk&usp=sharing)

The test was at 32 kb/s VBR, which for these hard samples averaged about 38 kb/s. If you have a way to build an Opus encoder from source, I can point you the exact version I was testing.

About the results, thanks very much for these tests. It does look like I can produce some improvement, but it's still a bit marginal. The idea in the collapse* series (except for collapse0 which was the control) is to do a sort of "gradual" intensity stereo rather than a binary decision.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-06-02 08:35:55
If you have a way to build an Opus encoder from source, I can point you the exact version I was testing.
Sincerely, I have no anymore. Lack of time. Barely a hour or so to do a blind test. If someone here can provide binaries and
If You tell which snapshots + settings to pay attention then I'll submit some results from time to time.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: DonP on 2016-06-02 17:19:39
Thanks for both the testing and intuitive graphics.  I've been using low bitrate opus for on-phone storage, so this of interest!
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: eahm on 2016-06-02 17:33:38
Thanks for testing again.

Does 1.1.2 change anything?
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-06-10 20:01:48
According to official site 1.1.2 fixes a bugs. All quality changes go to 1.2 ( exp_lbr_tune branch and others)

Opus 1.2 looks promising.  8)

Some time ago I have tried exp_lbr_tune build (see post #16) and Opus was already at least on par (if not better) than top-quality HE-AAC encoders at 48 kbps. And much better than Vorbis. 

Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-09-20 18:50:33
Took some time, but the exp_lbr_tune has now landed in git master and I have done some more work to "generalize" the stereo collapse work I uploaded here earlier. Here's three samples on which I'd be interested in feedback:
https://jmvalin.ca/misc_stuff/weighting0.wav
https://jmvalin.ca/misc_stuff/weighting1.wav
https://jmvalin.ca/misc_stuff/weighting2.wav
These are again 32 kb/s VBR (final rate 38 kb/s). Please let me know what you think of them.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-09-24 18:24:11
Please let me know what you think of them.
OK, I'll do this tomorrow.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-09-24 20:36:49
Great. I will try to listen them too during these days.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-09-26 01:14:18
Code: [Select]
w0 w1 w2
3.3 3.2 3.0
3.1 2.9 3.3
3.3 3.3 3.2
3.0 3.2 3.1
3.6 3.8 3.8
3.3 3.1 3.0
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-09-26 04:13:17
I wouldn't take seriously the following results:

Code: [Select]
	W0	W1	W2
3,475 3,5 3,35
2,15 2,125 2,125
2,35 2,275 2,325
2,45 2,4 2,4
2,8 2,9 2,925
2,775 2,7 2,65

Average 2,667 2,650 2,629
Those are the average scores per sample. Because scores change per each pass (weird). The differences are very small, near to nonexistent
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-10-03 03:20:15
Kamedo2, IgorC, thanks very much for the testing. It seems like my idea didn't work. What I was trying to do was slightly "shift" the stereo masking so that the loudest channel in a band gets a little higher SMR than the weakest one. Doesn't seem like it's had much effect.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-10-03 06:49:42
Probably it's worth to mention that an amount of noise is high at 32 kbps. It might be that all improvement from this new approach was completely masked by distortion from, i don't know, frequency leakage, high quantization noise, an effect of some tools like bandwidth extension or a combination of all of them. Just my guess.

During this test I didn't even get into stereo part because the general distortion of both channels was making way bigger picture. Either I didn't noticed any meaningfull (not sure if any) difference in a stereo image.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-10-03 18:10:16
Probably we ought to change the set of sound samples in the next test.
We can use the "Stochastic gradient descent" in this manual optimization process in here.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-10-28 00:49:56
OK, I've got a new experiment I'd like feedback on. This time I think I've managed to produce Windows builds, so it might be easier to try all kinds of samples. Here's the two binaries:
https://jmvalin.ca/misc_stuff/opus-tools-master-2af92cd.zip (https://jmvalin.ca/misc_stuff/opus-tools-master-2af92cd.zip)
https://jmvalin.ca/misc_stuff/opus-tools-exp-df24252.zip (https://jmvalin.ca/misc_stuff/opus-tools-exp-df24252.zip)
This is mostly meant for very low bitrates again (32-48 kbps), though I'd be curious what difference it makes at higher bitrates. Let me know if you have any issues with the binaries.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-11-02 02:48:11
~48 kbps VBR, stereo, 48 KHz
MOS:
exp-df24252  - 3.33
master-2af92cd - 3.35

12 samples. https://drive.google.com/file/d/0ByvUr-pp6BuUSnlwUG1WNkZzeDA/view?usp=sharing

Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-11-03 05:04:08
Thank you, IgorC. I visualized it.

(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_exp_compare_161103_2.png)
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-11-03 05:06:36
OK, I've got a new experiment I'd like feedback on.
Added to my TODO list. I'm going to test after finishing my mp3 192kbps test (Now 44% done).
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-11-03 20:28:22
It looks from IgorC's results that what I did wasn't really effective at 48 kbit/s. Digging further, I think I understand why I was seeing an improvement at 32 kbit/s but there's none at even 48 kbit/s. My change was effectively just trying to not use folding below 8 kHz, which I'm now doing explicitly. I just released 1.2-alpha, which includes that change. An example or clip where it provides an improvement (at least for me) is EnolaGay at 32 kb/s. I wish the improvement went all the way to 48 kbit/s, but unfortunately it doesn't.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-11-06 18:46:40
~32 kbps, VBR, stereo, 48 KHz
(https://s14.postimg.org/qhxltgb75/results_Opus_32_kbps.png)

Files: https://drive.google.com/file/d/0ByvUr-pp6BuUR3VMekNSa1NQU2c/view?usp=sharing

exp-df24252 produces considerably better results at ~32 kbps. The only issue is speech-music detector which interprets Fatboy sample as speech (well, it's actually kind of speech) and quality isn't homogeneous there.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-11-06 20:55:16
Unbelievable  :o
Opus 1.2 alpha outperforms HE-AACv2 at ~32 kbps VBR for the first time.

Scores of blind listening test at 32 kbps VBR:
1.  Opus 1.2 alpha (exp-df24252) - 3.22
2.  CELT (music) Mode - 3.28
3. HEAACv2 (FhG Winamp, VBR 1) - 2.92

(https://s11.postimg.org/4tnxhb5mb/Opus_HEAACv2_32_kbps_VBR.png)

Files https://drive.google.com/file/d/0ByvUr-pp6BuUMkRlS3RGWXI1MmM/view?usp=sharing

(http://www.christianguitar.org/forums/attachments/100984d1431451978-slow-clap.gif)
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: kode54 on 2016-11-06 21:50:17
Everybody always forgets that Citizen Kane scene was representative of undeserved applause.

Back on topic, congratulations on this, and I hope the voice/music detection can be trained to pick up the attributes of that fatboy sample without degrading anything else noticeably.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: smok3 on 2016-11-07 23:20:43
So basically if you remove the fatboy then auto-Opus wins (but HEAACv2 is still a clear looser).
Code: [Select]
                        32 kbps               
                   Opus exp-df24252 CELT mode HEAACv2  
01 castanets                     3,2       2,9       2,6
                                             
03 eig                           3,4       3,8       2,3
04 Bachpsichord                    3       2,9       3,9
5 Enola                          2,8       2,7       3,4
6 Trumpet                        3,4       3,5       3,6
7 Applaud                        2,7       2,7       2,4
8 Velvet                         2,9       2,9       2,7
9 Linchpin                         3         3         2
10 Spill the blood                 3         3         3
11 Female speech                 4,5       4,2       4,3
12 French ad                       3       2,9       2,2
13 german sppech                 4,2       3,7       3,8
                                             
Average                         3,26      3,18      3,02
                                             
Min.                             2,7       2,7         2

Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-11-08 02:05:03

smok3, Yes,  the speech-music detector works quite good on all tested samples except this 'musical-speech' sample Fatboy.
It will be great if folks will try some mixed samples to see how it performs.

Everybody always forgets that Citizen Kane scene was representative of undeserved applause.
I haven't seen that movie yet. Thank you for clarifying.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: kode54 on 2016-11-08 04:57:12
I need to see the movie for real. I was off base. It wasn't undeserved applause.

This topic (http://www.neogaf.com/forum/showthread.php?t=889337) explains it quite well.

Kane, who is building up his mistress-turned-second-wife, Susan, to be an opera singer. She gives a "bad" performance. He feels obligated to clap for her anyway, to build her up to be the success that he trained her to be, to support his own need to see her succeed. So, "This thing I am clapping for is terrible, but I am clapping for it anyway, because my sanity depends on it being the rousing success I hyped myself up for it to be."

Still not "generic clapping gif", but I was still off with the exact reason.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-11-09 22:15:21
Unbelievable  :o
Opus 1.2 alpha outperforms HE-AACv2 at ~32 kbps VBR for the first time.

Scores of blind listening test at 32 kbps VBR:
1.  Opus 1.2 alpha (exp-df24252) - 3.22
2.  CELT (music) Mode - 3.28
3. HEAACv2 (FhG Winamp, VBR 1) - 2.92

Really cool! I again did not expect Opus to do so well against HE-AAC v2 at 32 kb/s. I'll see what I can do about the speech/music detection on fatboy. At the very least, we're looking at an "official" option for forcing the encoder to consider everything as speech or music.

One other thing I'm a bit curious about is the difference on the other samples (e.g. castanets, eig). If they're real and not due to the speech/music detector, then it's worth investigating since the two builds you have differ only by a tiny detail in the bit allocation.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-11-13 05:58:17
Really cool! I again did not expect Opus to do so well against HE-AAC v2 at 32 kb/s.
The test was done in headphones and parametric stereo is known to perform not great there.  Results can be different with speakers/studio monitors. (I will try later). Anyway Opus alpha 1.2 is at least in a same league now with HE-AAC v2 at such bitrate.

 


I'll see what I can do about the speech/music detection on fatboy. At the very least, we're looking at an "official" option for forcing the encoder to consider everything as speech or music.
Great.
Another sample where the quality (especially stereo) is intermittent due to SILK/CELT mode changes.
https://drive.google.com/file/d/0ByvUr-pp6BuUbXF5MUMxYW5VM2s/view?usp=sharing
I've performed more rigorous test to be sure about it. Anyway an audible differences are pretty obvious there. Also the CELT+SILK(hybrid) sample hasn't the same loudness (the sound goes down and high) across the whole sample while CELT-only hasn't this issue.
Original - -2.39 dB
CELT      -  -2.31 dB
Hybrid   -  -1.74 dB

One other thing I'm a bit curious about is the difference on the other samples (e.g. castanets, eig). If they're real and not due to the speech/music detector, then it's worth investigating since the two builds you have differ only by a tiny detail in the bit allocation.
Don't pay attention. I've performed an ABX test on castanets,eig samples and I saw those were personal and momental preferences.  I could ABX between two lossy files but wasn't sure what to prefer this time. Somewhere bettter somewhere worse. Not sure. It will be better if someone else can perform the blind test on them.  As for me both versions (CELT and exp) made good there.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-11-25 20:24:12
I just uploaded another experimental build (https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip) that changes the tonality analysis to (hopefully) be more sensitive, especially when there's lots of closely-spaced harmonics. This change affects pretty much all rates and I'd be curious to have feedback on any improvements or regression it causes before I merge it in for 1.2. You can compare it to 1.2-alpha. Note that this new build also forces the encoder to consider everything as music.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-11-28 17:57:07
I just uploaded another experimental build (https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip) that changes the tonality analysis to (hopefully) be more sensitive, especially when there's lots of closely-spaced harmonics. This change affects pretty much all rates and I'd be curious to have feedback on any improvements or regression it causes before I merge it in for 1.2. You can compare it to 1.2-alpha. Note that this new build also forces the encoder to consider everything as music.
Sorry for not reacting recently. I would rather like to get rid of my remaining mp3 listening test sessions first. Now 24 sessions remaining and 56% done.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-12-09 16:00:09
Quote
   1.2a (celt mode)   New tonality
32 kbps
01 Castanets   2,8   3
04 Bachpsichord   2   2
05 Enola   2,4   2,4
06 Tumpet   3,6   3,6
10 Spill the blood   3,2   3,3
Take your fingers from my head   2   2
      
http://listening-test.coresv.net/tracks/take_your_finger_from_my_hair.wav from a  test http://listening-test.coresv.net/results.htm

Average:   2,67   2,72


The latest experimental build (new tonality) has higher average bitrate at 32 kbps. It's 0.4-0.9 kbps more on my bitrate calibration set of 44 songs and slightly more on tested samples. Not that much of a bitrate increase but it could be a reason of slightly higher average score.

I notice some very tiny improvements only on guitar intro of "Castanets"  and on "spill the blood" samples.

If You ask me I think it's a slight bitrate bump/increase rather than quality improvement.

P.S. I didn't notice any audible regressions.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-12-09 19:21:34
If You ask me I think it's a slight bitrate bump/increase rather than quality improvement.

P.S. I didn't notice any audible regressions.

Did you test other bitrates? The new code actually impacts all bitrates because it changes the way we compute the tonality (by using a larger window).
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-12-09 20:17:32
JM,

Only 32 kbps. Sorry, lack of time. I'll try another bitrate if will get some time.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-12-10 03:44:04
Only 32 kbps. Sorry, lack of time. I'll try another bitrate if will get some time.

Thanks. 64 kb/s is probably the logical one to try since it's far from 32 kb/s but the effect of transient detection is still obvious.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-12-11 07:27:02
I don't hear any improvement at 64 kbps either. And  bitrate tiny bit higher for "new tonality" build.   :-\
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2016-12-11 09:56:52
I will be able to start testing Opus from 2016 Dec 31.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-12-11 23:29:36
I don't hear any improvement at 64 kbps either. And  bitrate tiny bit higher for "new tonality" build.  :-\

As long as it's not worse, I'm OK since the new analysis code is nearly 2x cheaper to compute. Normally, it should also help on some of the bachpsichord notes that were too low for the other code to pick up.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-12-12 03:05:44
@JM
Alright then.

BTW I was listening different songs/styles encoded by 1.2a at 32 kbps VBR and the quality was pretty good.
Except some songs with a lot of energy in HF (mostly cymbals). A styles like loud metal are affected most of all.
Quality breaks the same manner and often even worse than on a tonal stuff. A lot of smearing, distortion. Pretty unlistenable. :(  Maybe something can be done with rate distribution for LF/HF (just my guess).
1.2a doesn't detect it as something hard to encode and codes it at pretty flat bitrate 30-31 kbps.

Here sample: Metallica - Atlas, Rise! (https://drive.google.com/drive/folders/0ByvUr-pp6BuURUNsd1NhOXdpLTg)

P.S. 64 kbps was a different story. Tonal stuff is the only  most vulnerable (mostly). 

I would rather like to get rid of my remaining mp3 listening test sessions first. Now 24 sessions remaining and 56% done.
Lets try Opus. MP3 won't go anyway. :)


Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-12-21 01:10:17
There is also an old issue.
Opus tends to increase bitrate on quiet passages where it's not justified. 

I've tried two outros of Pink Floyd "The Wall" (MFSL): "Goodbye Cruel World" and "Outside The Wall". Both are quiet songs.  Opus -32 kbps encodes them at 36 and 39 kbps respectively.  And there is no need for such bitrates as there aren't very audible (far from that) artifacts here.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2016-12-21 17:47:45
BTW I was listening different songs/styles encoded by 1.2a at 32 kbps VBR and the quality was pretty good.
Except some songs with a lot of energy in HF (mostly cymbals). A styles like loud metal are affected most of all.
Quality breaks the same manner and often even worse than on a tonal stuff. A lot of smearing, distortion. Pretty unlistenable. :(  Maybe something can be done with rate distribution for LF/HF (just my guess).
1.2a doesn't detect it as something hard to encode and codes it at pretty flat bitrate 30-31 kbps.
Interestingly, I don't find these artefacts particularly objectionable. They're of course obvious, but I find them less annoying than many other artefacts at 32 kb/s. The main thing I'm noticing with the cymbals isn't so much pre-echo, but a tendency to sound tonal. Is that what you're hearing. In any case, I hacked two different changes to the encoder (not something immediately mergeable) to get your opinion. Can you listen to these files (A is the unmodified encoder):
https://jmvalin.ca/misc_stuff/Atlas32_A.wav
https://jmvalin.ca/misc_stuff/Atlas32_B.wav
https://jmvalin.ca/misc_stuff/Atlas32_C.wav

Quote
Opus tends to increase bitrate on quiet passages where it's not justified. 
That's a known issue, but it's hard to do anything about it because the encoder does not know what volume you're listening at. It could be that your volume knob is turned all the way up and it's no longer a quiet passage.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-12-21 22:35:32
The main thing I'm noticing with the cymbals isn't so much pre-echo, but a tendency to sound tonal.Is that what you're hearing
Yes, it's right description.

I'll try samples.  Thanks.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2016-12-27 16:28:59
https://jmvalin.ca/misc_stuff/Atlas32_A.wav
https://jmvalin.ca/misc_stuff/Atlas32_B.wav
https://jmvalin.ca/misc_stuff/Atlas32_C.wav

MOS of a blind test:

C  - 3.3
B  - 3.1
A - 3.0

MOS are the same for both tests (studio speakers/monitors and headphones)
Code: [Select]
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname:

1L = E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_B.wav
2L = E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_A.wav
3L = E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_C.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_B.wav
1L Rating: 3.1
1L Comment:
---------------------------------------
2L File: E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_A.wav
2L Rating: 3.0
2L Comment:
---------------------------------------
3L File: E:\DETODO\Audio\Opus new tonality\Atlas rise\Atlas32_C.wav
3L Rating: 3.3
3L Comment: stereo loss
---------------------------------------
ABX Results:

My comments (not actual log):

"Sample 3 vs Sample 2
  1 of   1, p = 0.500
  2 of   2, p = 0.250
  3 of   3, p = 0.125
  4 of   4, p = 0.063
  5 of   5, p = 0.031
TRAINING MODE -- not written to file"

I don't know why I perceive the C as stereo loss or energy change (?) . Anyway it sounds better.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2017-01-07 17:34:18
I just uploaded another experimental build (https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip) that changes the tonality analysis to (hopefully) be more sensitive, especially when there's lots of closely-spaced harmonics. This change affects pretty much all rates and I'd be curious to have feedback on any improvements or regression it causes before I merge it in for 1.2. You can compare it to 1.2-alpha. Note that this new build also forces the encoder to consider everything as music.
I am planning a new listening test. Which version and bitrate should I test? Is this the version I should start testing now?
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2017-01-07 21:46:35
I just uploaded another experimental build (https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip) that changes the tonality analysis to (hopefully) be more sensitive, especially when there's lots of closely-spaced harmonics. This change affects pretty much all rates and I'd be curious to have feedback on any improvements or regression it causes before I merge it in for 1.2. You can compare it to 1.2-alpha. Note that this new build also forces the encoder to consider everything as music.
I am planning a new listening test. Which version and bitrate should I test? Is this the version I should start testing now?
Yes, I think that's a good build to test. In terms of bit-rate, I'd say 32 to 48 kb/s is where it would be most interesting.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2017-01-07 23:47:12
hi, Kamedo2,

There is also more recent build https://jmvalin.ca/misc_stuff/opus-tools-buildC.zip
It addresses the issue with Atlas sample here https://hydrogenaud.io/index.php/topic,111798.msg933126.html#msg933126

I'll try to make some time and test it on a different samples at 32-48-64 kbps. If You or someone else can test it even better.
As I understand an idea is to find some pattern how quality changes among different type of samples.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2017-01-10 03:30:34
There is also more recent build https://jmvalin.ca/misc_stuff/opus-tools-buildC.zip
It addresses the issue with Atlas sample here https://hydrogenaud.io/index.php/topic,111798.msg933126.html#msg933126

I'll try to make some time and test it on a different samples at 32-48-64 kbps. If You or someone else can test it even better.
As I understand an idea is to find some pattern how quality changes among different type of samples.
I just uploaded a new build:
https://jmvalin.ca/misc_stuff/opus-tools-buildD.zip
It's playing with the same "allocation trim" parameter as build C, but instead of giving a slight boost to HF on all frames, it's trying to do it adaptively. Can you see how it compares to build C and 1.2-alpha?.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2017-01-11 16:50:27
It's playing with the same "allocation trim" parameter as build C, but instead of giving a slight boost to HF on all frames, it's trying to do it adaptively. Can you see how it compares to build C and 1.2-alpha?.
OK. Then I will test

https://jmvalin.ca/misc_stuff/opus-tools-exp-df24252.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildC.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildD.zip

at 36k and 48kbps.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2017-01-11 21:17:39
I think it has more sense to test https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip.
I remember exp-df24252.zip uses hybrid mode at low bitrates while "analysis24k" does only CELT.
https://hydrogenaud.io/index.php/topic,111798.msg931347.html#msg931347


Don't know whether builds C and D use hybrid or just CELT.

The mode selection might interfere into test results.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2017-01-12 22:13:59
I think it has more sense to test https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip.
I remember exp-df24252.zip uses hybrid mode at low bitrates while "analysis24k" does only CELT.
https://hydrogenaud.io/index.php/topic,111798.msg931347.html#msg931347
I guess analysis24k would be safer, but otherwise it's going to be pretty obvious if there's a mode change because the builds should normally sound relatively similar and just weight the noise slightly differently.
Quote
Don't know whether builds C and D use hybrid or just CELT.
Yes, builds C and D force CELT all the time.

BTW, just uploaded a new build:
https://jmvalin.ca/misc_stuff/opus-tools-buildE.zip
If you haven't started yet, please add that too. Otherwise, I'll use the feedback on what you have to adjust thing further.

Note that the only thing these different builds change is the bit allocation of LF vs HF, aka allocation trim. The rest is the same. Basically, I'm trying to dynamically adjust that trim depending on the audio.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2017-01-21 21:56:51
BTW, just uploaded a new build:
https://jmvalin.ca/misc_stuff/opus-tools-buildE.zip
If you haven't started yet, please add that too. Otherwise, I'll use the feedback on what you have to adjust thing further.
I tried.
These are the bitrate distributions of "opusenc --bitrate 36" and "opusenc --bitrate 48". There are almost no bitrate changes between builds. Only Bachpsichord at low bitrates, analysis24k uses fewer bits.
Are there any builds I can omit? Reducing encoders from 8 to 6 will significantly shorten test times.

https://jmvalin.ca/misc_stuff/opus-tools-analysis24k.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildC.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildD.zip
https://jmvalin.ca/misc_stuff/opus-tools-buildE.zip

(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_build_acde_bitrates.png)

BTW,
Code: [Select]
opusenc --version
is seemingly not working.
Code: [Select]
opus-tools-analysis24k\opusenc --version 
opusenc opus-tools 0.1.9-35-g49c5f27-dirty (using libopus 1.2-alpha)
Copyright (C) 2008-2013 Xiph.Org Foundation

opus-tools-buildC\opusenc --version
opusenc opus-tools 0.1.9-35-g49c5f27-dirty (using libopus 1.2-alpha)
Copyright (C) 2008-2013 Xiph.Org Foundation

opus-tools-buildD\opusenc --version
opusenc opus-tools 0.1.9-35-g49c5f27-dirty (using libopus 1.2-alpha)
Copyright (C) 2008-2013 Xiph.Org Foundation

opus-tools-buildE\opusenc --version
opusenc opus-tools 0.1.9-35-g49c5f27-dirty (using libopus 1.2-alpha)
Copyright (C) 2008-2013 Xiph.Org Foundation
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: kode54 on 2017-01-21 22:12:13
Version appears to be working, as those three versions appear to be "dirty" working trees, meaning they have uncommitted changes.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2017-01-26 05:09:25
It may be interesting to add mp3 at 2x the bitrate. (but I am running out of slots)
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2017-01-26 12:58:14
Or HE-AAC at the same rate.
The closest competitor to Opus at 32-48 kbps will be HE-AACv2/v1 (at the same bitrate). MP3 and Vorbis simply can't compete with it at those rates.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2017-01-26 16:02:14
https://jmvalin.ca/misc_stuff/opus-tools-buildE.zip
If you haven't started yet, please add that too.

BTW, is this build E slower than the other builds?
build E is about 40x realtime on the standard settings, while other builds reach nearly 50x .

Code: [Select]
bitrate	analysis24k	buildC	buildD	buildE
12 29.6 56.3 56.5 46.4
24 47.3 48.9 49.2 42.4
36 48.9 48.8 48.8 41.3
48 48.0 48.1 47.9 40.5
60 46.5 46.2 45.9 39.6
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: butrus on 2017-01-26 22:28:23
Quote
Opus tends to increase bitrate on quiet passages where it's not justified. 
That's a known issue, but it's hard to do anything about it because the encoder does not know what volume you're listening at. It could be that your volume knob is turned all the way up and it's no longer a quiet passage.

Maybe there could be an optional switch in the encoder where you can set the anticipated absolute loudness of the playback (or the anticipated absolute level of the ambient noise)?

E.g. if you know you're encoding files to be heard in the car this would enable the encoder to optimize the encoding process for this purpose and spare some bits...
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2017-01-26 22:34:20
Maybe there could be an optional switch in the encoder where you can set the anticipated absolute loudness of the playback (or the anticipated absolute level of the ambient noise)?

E.g. if you know you're encoding files to be heard in the car this would enable the encoder to optimize the encoding process for this purpose and spare some bits...

This is something I've considered in the past and eventually decided against simply because the gain are minuscules (over a large and varied corpus), while the potential for misuse (or just wrong encoder decisions) is huge.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2017-01-26 22:36:45
BTW, is this build E slower than the other builds?
build E is about 40x realtime on the standard settings, while other builds reach nearly 50x .
Hmm, that's strange, but maybe I just used slightly different optimizations in the build. In any case, it seems like the differences between all these builds is very small and potentially "in the noise". I'll see if I can extract useful information from your data that would allow me to at least use the best settings for each of the files.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2017-01-27 04:37:25
jmvalin, is there any builds I can omit testing?
I would like to deliver the results quickly.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: jmvalin on 2017-01-27 15:41:36
jmvalin, is there any builds I can omit testing?
I would like to deliver the results quickly.
I think you can skip D.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2017-04-11 01:04:40
I am testing build A, build C, build E and now it's 63%(17/27) done.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: atomnuker on 2017-04-12 23:34:24
I should probably mention I'm writing an Opus encoder from scratch for FFmpeg. The base encoder already got merged in git master 2 months ago, though I'm still working on a psychoacoustic system. If anyone's interested in taking a look I've sent a patch (https://ffmpeg.org/pipermail/ffmpeg-devel/2017-April/210093.html) to the mailing list. This currently only implements transient detection (untweaked), dual stereo (kinda pointless) and intensity stereo (slow) searches, so it still has ways to go to sound better. At least it sounds halfway decent because you can't really write a bad Opus encoder. The psychoacoustic system will probably replace the current filter-based system used by AAC once its good enough (not really difficult considering it isn't very good).
I'll keep this thread updated if I make any significant progress that might warrant a careful listen, since I'm not really good at that.
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Lumitopia on 2017-04-17 13:48:21
I should probably mention I'm writing an Opus encoder from scratch for FFmpeg. The base encoder already got merged in git master 2 months ago, though I'm still working on a psychoacoustic system. If anyone's interested in taking a look I've sent a patch (https://ffmpeg.org/pipermail/ffmpeg-devel/2017-April/210093.html) to the mailing list. This currently only implements transient detection (untweaked), dual stereo (kinda pointless) and intensity stereo (slow) searches, so it still has ways to go to sound better. At least it sounds halfway decent because you can't really write a bad Opus encoder. The psychoacoustic system will probably replace the current filter-based system used by AAC once its good enough (not really difficult considering it isn't very good).
I'll keep this thread updated if I make any significant progress that might warrant a careful listen, since I'm not really good at that.
Hi atomnuker,
I'm interested in testing out your new patch, but don't have the technical expertise to apply it to FFmpeg myself. I've already tested your base encoder, but that's only because it comes integrated with the official Windows build. Would you recommend just waiting until the new changes get merged into the git master? Thanks and best of luck with the improvements.

As for the main 'libopus' encoder - I have been converting podcasts and similar content to VBR 18kbps, and the results are remarkably clear - even musical interludes sound listenable. It actually sounds better than many of the FM radio stations here in the UK! I'm using the latest experimental build (1.2-alpha2-31-g8e19536b) - very impressive. A huge improvement on whatever version of Opus is bundled with "Switch Audio File Converter v5.19".
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: Kamedo2 on 2017-04-20 20:12:48
I finished the test of the analysis24k, buildC, buildE.
The buildC seems to be marginally better.
I am going to post the details later.
(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_ace36k48k_compare1.png)
(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_ace36k48k_compare2.png)
(https://ss1.coressl.jp/listening-test.coresv.net/img2/opus_ace36k48k_compare3.png)
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: IgorC on 2017-04-22 02:25:54
Thank You, Kamedo2. :)

Great to see that BuildC has some potential.

I think BuildC is a start point on what can be done with Opus'es bitrate distribution amongst different frequency bands.
Sorry, couldn't do tests.
 
Title: Re: Personal Listening Test of 2 Opus encoders
Post by: RiCON on 2017-05-24 21:48:09
@Lumitopia here's a x64 build of FFmpeg (https://i.fsbn.eu/pub/ffmpeg_opus_psyacoustics.7z) N-86248-gcfec0d6475 (http://git.videolan.org/?p=ffmpeg.git;a=commit;h=cfec0d64752509f8ac798acca6225df630fa5284) with @atomnuker psyacoustics patch (https://i.fsbn.eu/pub/opuspsy_v3.patch) included along with libopus git master (https://github.com/xiph/opus/commit/f17991727e94230af16ec8f11c9a3fb247794fba). It's built using media-autobuild_suite (https://github.com/jb-alvarado/media-autobuild_suite) (MSYS2 + MinGW + GCC 6.3)
SimplePortal 1.0.0 RC1 © 2008-2018