RESULTS
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 60
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 299 455.02
Testers (blocks) 59 57.64
Codecs eval'd 4 299.02 74.75 179.36 0.00E+000
Error 236 98.36 0.42
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.232
Means:
HIGH USAC OPUS HE-AAC LOW
4.74 3.08 3.07 2.69 1.64
---------------------------- p-value Matrix ---------------------------
USAC OPUS HE-AAC LOW
HIGH 0.000* 0.000* 0.000* 0.000*
USAC 0.910 0.001* 0.000*
OPUS 0.001* 0.000*
HE-AAC 0.000*
-----------------------------------------------------------------------
HIGH is better than USAC, OPUS, HE-AAC, LOW
USAC is better than HE-AAC, LOW
OPUS is better than HE-AAC, LOW
HE-AAC is better than LOW
• HE-AAC : Statistically inferior to both OPUS and USAC (Exhale). Generally speaking, it suffers from SBR artifacts (sizzling noise on sibilants, tonal or high pitched sound) and from smearing (pre-echo). Good stero image. Sound could be very pleasing but SBR is the cure and the poison. It saves a lot of bit which can be spent elsewhere but introduce those SBR artifacts.
• OPUS : Best tied with USAC. Suffers from a puffy sound: coarse, hiss, messy. Also have a narrowed stereo audible many times. It also introduces some artifacts. It performed very well with audiobooks and really poorly with classical music (issue I also noticed with 64 kbps test few months ago).
• USAC: Exhale encoder (32000 Hz) doesn't support SBR. It seems a bit naked to handle such strong bitrate starvation but surprisingly it performs very well and also competes with OPUS. It also sounded better than HE-AAC on every six group of samples (Billboard charts, classical music, HA.org, problem samples, audiobooks and movies). IgorC got few weeks ago the same conclusion. USAC has some issues at this bitrate: slightly narrowed stereo image on some samples, metallic sound which was more irritating, smearing and some other kind of common lossy artifacts.
• LOW ANCHOR: I was looking for something that wasn't too lowpassed. HE-AACv2 (with parametric stereo) at 24 kbps has a 15 KHz lowpass and seemed to be a good choice. I chose FDK encoder, which is CBR only at this mode and bitrate. I made a mistake and used it at 16 kbps for the 20 speech files. The main quality issues were narrow and sometimes fanciful stereo imaging. It also add a metallic sound and the effect increased a lot for 16 kbps encoding (lowpass at 12 KHz).
• HIGH ANCHOR: I used MP3 LAME VBR ~130kbps for my last comparison at 64 kbps and it wasn't strong enough to play the high anchor role. This time I chose something more robust and used the winner of my AAC test: Apple's encoder, set as Constraint VBR (CVBR) to target ~128kbps. Overall quality is very good, not always fully transparent but rarely annoying.
Now, let's see in details how encoders are performing accordingly on samples' genre.
MUSIC ONLY (40 samples)
SPEECH ONLY (20 samples)
••••••••••••••••••••••••
•MUSIC ONLY (40 samples•
••••••••••••••••••••••••
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 40
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 199 294.66
Testers (blocks) 39 50.23
Codecs eval'd 4 177.09 44.27 102.55 0.00E+000
Error 156 67.34 0.43
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.290
Means:
HIGH USAC OPUS HE-AAC LOW
4.65 3.06 2.88 2.56 1.78
---------------------------- p-value Matrix ---------------------------
USAC OPUS HE-AAC LOW
HIGH 0.000* 0.000* 0.000* 0.000*
USAC 0.229 0.001* 0.000*
OPUS 0.031* 0.000*
HE-AAC 0.000*
-----------------------------------------------------------------------
HIGH is better than USAC, OPUS, HE-AAC, LOW
USAC is better than HE-AAC, LOW
OPUS is better than HE-AAC, LOW
HE-AAC is better than LOW
•••••••••••••••••••••••••••
•AUDIOBOOKS & MOVIES GROUP•
•••••••••••••••••••••••••••
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 20
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 99 158.45
Testers (blocks) 19 5.50
Codecs eval'd 4 129.51 32.38 105.00 0.00E+000
Error 76 23.44 0.31
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.350
Means:
HIGH OPUS USAC HE-AAC LOW
4.91 3.45 3.13 2.93 1.36
---------------------------- p-value Matrix ---------------------------
OPUS USAC HE-AAC LOW
HIGH 0.000* 0.000* 0.000* 0.000*
OPUS 0.077 0.004* 0.000*
USAC 0.247 0.000*
HE-AAC 0.000*
-----------------------------------------------------------------------
HIGH is better than OPUS, USAC, HE-AAC, LOW
OPUS is better than HE-AAC, LOW
USAC is better than LOW
HE-AAC is better than LOW
• MUSIC: OPUS and USAC are tied, both are better than HE-AAC which is better than LOW ANCHOR (HE-AACv2 at 24 kbps).
• SPEECH: OPUS and USAC are tied, but only OPUS is better than HE-AAC. HE-AAC and USAC are tied on this group. I recall thtat the LOW ANCHOR is here HE-AACv2 at 16 kbps only (mistake of mine).
DETAILED RESULTS FOR MUSIC
TEST FAQ
Are these samples difficult ones?
Some of them. The 10 classical samples are musical parts I really enjoy. There was no selection based on difficulty. The 10 “Billboard” samples were made indiscriminately: 30 seconds coming from an exact range of each track (1 min 00 sec to 1 min 30 sec). But HA.org samples may be selected by members for their possible difficulty. And a final group of 10 samples is intentionally a possible “killer-samples” group. These samples and selection was already used for previous tests I made this year.
For this test I add 20 more samples. 12 for audiobooks (6 female, 6 male: german, italian, spanish, english, french, korean, dutch) and 8 for movies. Audiobooks are lossless sourced (no MP3 CD). Movies are Blu-Ray source (48000 Hz, 24 bit, lossless), downsampled to stereo. They were converted to 16 bit for blind test (ABC/HR refused to open 24 bit WAV/FLAC).
All samples can be downloaded at: https://www.dropbox.com/sh/zjephy3g54j4gur/AACjGhM9tabl26n7s4ihYl2Ra?dl=0
Methodology
Java ABC/HR is my software of choice. Volume was normalized and delay removed within foobar2000. Everything was decoded and converted as WAV file using foobar2000 and resampler at 48000 Hz. It’s a blind test but with no ABX sessions. I’d rather test non-extensively a wide set of samples and let small ranking errors be statistically vanished than spending a lot of time on a smaller set of samples. My hardware setup is very basic: laptop headphone output (no DAC), AKG Q701 headphone, moderate listening volume playback.
EDIT: I ranked 265 files (score < 5.0) and I made no mistakes (ranking the reference instead of the encoded one). Final score is therefore 265/265 and probability of guessing is very close to zero 
Encoders setup
• HE-AAC: encoded through foobar2000's converter (AAC Apple Graphical Interface): CVBR 48 in HE profile. Metadata: qaac 2.67, CoreAudioToolbox 7.10.9.0, AAC-HE Encoder, CVBR 48kbps, Quality 96
• HIGH ANCHOR: encoded through foobar2000's converter (AAC Apple Graphical Interface): CVBR 48 in HE profile. Metadata: qaac 2.67, CoreAudioToolbox 7.10.9.0, AAC-LC Encoder, CVBR 128kbps, Quality 96
• LOW ANCHOR: encoded through foobar2000's converter (AAC FDK Graphical Interface): CBR 24 and accidentaly 16. Metadata: fdkaac 1.0.0, libfdk-aac 4.0.0, CBR 24kbps (and 16kbps)
• OPUS: encoded through foobar2000 command line encoder: VBR 48 (--quiet --bitrate 48 --vbr --ignorelength - %d). Metadata: libopus 1.3-26-ge85ed772, libopusenc 0.2.1-2-g9cb17c6
• USAC: encoded through foobar2000 command line encoder: mode 0 (0 %d). Metadata: none. Encoder is exhale-1.0.7-9323a9d0 from 2020.09.28
TABLE
(with the nice help of kamedo2 and his fantastic tool!!! thanks again )
Topic: Personal listening test at 48 kbps: Opus, Exhale, HE-AAC (Read 8320 times)
previous topic - next topic