Skip to main content
Topic: Personal Listening Test of bad/experimental AAC/MP3 Encoders (Read 19129 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of bad/experimental AAC/MP3 Encoders

Abstract:
Blind Comparison of AAC/MP3 files encoded by FAAC, vo-aacenc, FFmpeg's native experimental aac encoder, and BladeEnc(primitive MP3 encoder from 2001).
Note that these encoders are experimental/bad and do not offer the quality offered by best encoders such as Apple AAC, FDK-AAC, and LAME.

Encoders and settings:
FAAC 1.28 mod from rarewares, VBR -q 30
faac -q 30 -o out.mp4 in.wav
FAAC 1.28 mod from rarewares, ABR 96kbps
faac -b 96 -o out.mp4 in.wav
vo-aacenc(VisualOn AAC encoder from Android) 0.1.3 via FFmpeg r.63227
ffmpeg -y -i in.wav -c:a libvo_aacenc -b:a 96k out.mp4
BladeEnc 0.94.2 (latest, 2001)
bladeenc -quit -nocfg in.wav out.mp3 -96
FFmpeg's native AAC encoder (r.62757), no patch
ffmpeg -y -i in.wav -c:a aac -strict experimental -b:a 96k out.mp4
FFmpeg's native AAC encoder (r.63227), with v8g patch applied
ffmpeg -y -i in.wav -c:a aac -strict experimental -b:a 96k out.mp4

The v8g and other patches are developed and distributed in https://trac.ffmpeg.org/ticket/2686
Due to my mistake, FFmpeg's native mp2 encoder was not tested and instead, FAAC VBR q30(~48kbps) were tested twice.

Samples:
15 Sounds of various genres from the 2011 Multiformat Public Listening Test.
http://listening-tests.hydrogenaud.io/igor...and_samples.zip

Hardwares:
Sony PSP-3000 + RP-HJE150.

Results:



Conclusions & Observations:
FFmpeg's native AAC encoders had the best score among the 7 settings tested. The v8g patch offered better quality.
vo-aacenc and FAAC(ABR) came second. BladeEnc was worse than vo-aacenc and FAAC(ABR).
Understandably, the ~48kbps FAAC(-q 30 VBR) was the worst settings.
If we were to use FAAC ABR 96k as a mid-low anchor, use of FAAC VBR -q30 seems to be a reasonable choice.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of        Degrees    Sum of    Mean
variation        of Freedom  squares  Square    F      p

Total              104          39.21
Testers (blocks)    14          4.91
Codecs eval'd        6          28.01    4.67  62.39  0.00E+000
Error              84          6.28    0.07
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:  0.199

Means:

ffaacv8g ffaac    vo-aac  faac96  bladeenc faac48  faac48
  3.24    2.93    2.70    2.51    2.28    1.78    1.77

---------------------------- p-value Matrix ---------------------------

        ffaac    vo-aac  faac96  bladeenc faac48  faac48
ffaacv8g 0.002*  0.000*  0.000*  0.000*  0.000*  0.000*
ffaac            0.026*  0.000*  0.000*  0.000*  0.000*
vo-aac                    0.056    0.000*  0.000*  0.000*
faac96                              0.024*  0.000*  0.000*
bladeenc                                    0.000*  0.000*
faac48                                                0.973
-----------------------------------------------------------------------

ffaacv8g is better than ffaac, vo-aac, faac96, bladeenc, faac48, faac48
ffaac is better than vo-aac, faac96, bladeenc, faac48, faac48
vo-aac is better than bladeenc, faac48, faac48
faac96 is better than bladeenc, faac48, faac48
bladeenc is better than faac48, faac48
Raw data:
It's not strange that some scores have 0.05 scale because I tested them twice per sample and took the average.
Code: [Select]
faac(q30)	faac(q30)	faac(abr)	vo-aacenc	FFmpeg_aac	FFmpeg_aac+v8g	BladeEnc
%feature 0 48kbps 48kbps 96kbps 96kbps 96kbps 96kbps 96kbps
%feature 8 AAC_LC AAC_LC AAC_LC AAC_LC AAC_LC AAC_LC MP3
1.7 1.75 2.25 2.85 3.15 2.9 2.25
1.95 1.95 2.35 3.2 3.25 3.65 2.25
1.55 1.55 2.55 2.65 2.55 3.15 2
1.55 1.55 2.35 2.4 2.45 2.7 1.85
1.75 1.75 2.8 2.3 3.25 3.05 1.85
1.7 1.7 2.25 2.1 2.65 3.1 2.25
1.85 1.85 2.65 3.6 3.55 3.75 3.25
1.85 1.85 2.45 2.4 2.95 3.15 3.35
1.9 1.9 2.4 2.65 3 3.45 2.55
2.35 2.35 3.2 2.55 3.15 3.3 2
1.8 1.8 2.6 3.3 3 3.75 2.7
1.75 1.75 2.6 1.9 2.65 2.95 1.6
1.7 1.7 2.35 3 2.9 3.25 2
1.7 1.7 2.2 2.8 2.7 3.15 1.95
1.5 1.5 2.65 2.85 2.75 3.35 2.35
%samples poets_of_the_fall guitar
%samples bachpsichord classic
%samples spmg54_1_german speech
%samples Tom's_diner a_cappella
%samples kraftwerk techno
%samples macabre classic
%samples bibilolo techno
%samples Stravinskii Capriccio strings
%samples fly_to_fly pops
%samples fatboy techno
%samples les_voleurs pops
%samples eig techno
%samples enter_sandman rock
%samples big_yellow pops
%samples davinci speech
Bitrates and distribution:
Code: [Select]
faac(q30)	faac(q30)	faac(abr)	vo-aacenc	FFmpeg_aac	FFmpeg_aac+v8g	BladeEnc
%bitrate
56688 56688 97539 99328 98685 98313 96196
59350 59350 97699 97866 97924 98083 96097
44685 44685 97441 98016 100108 98700 96008
52863 52863 97786 98091 98610 98076 96101
44351 44351 97888 97849 98430 99474 96041
55378 55378 97394 98025 98057 97842 96043
42647 42647 97621 97806 98373 100887 96086
54271 54271 97463 97997 97998 97812 96118
53065 53065 97737 97869 98128 99656 96001
49583 49583 97794 97829 99065 99196 96080
51266 51266 98197 98007 98193 99486 96092
47838 47838 97984 98318 98593 98609 96126
49015 49015 97649 97782 98000 99697 96040
54271 54271 97777 97859 98023 99586 96035
53578 53578 97732 97772 98194 99886 96005
Encoding and Decoding Logs:
Code: [Select]
===================Processing Wav File "public_multi64\sample01.wav"===================
===================Encode Section Start===================

bin\sox "public_multi64\sample01.wav" -b 16 "sound_raw\sample01.int16.wav"

bin\faac-1.28-mod\faac -b 96 -o "sound_out\sample01.faac.abr96k.mp4" "sound_raw\sample01.int16.wav"
Freeware Advanced Audio Coder
FAAC 1.28

Average bitrate: 96 kbps
Quantization quality: 100
Bandwidth: 10325 Hz
Object type: Low Complexity(MPEG-4) + M/S
Container format: MPEG-4 File Format (MP4)
Encoding sound_raw\sample01.int16.wav to sound_out\sample01.faac.abr96k.mp4
  frame          | bitrate | elapsed/estim | play/CPU | ETA
  347/347  (100%)|  95.4  |    0.3/0.3    |  28.67x | 0.0


bin\faad -b 2 -q -o "sound_temp\sample01.faac.abr96k.mp4.int44k.wav" "sound_out\sample01.faac.abr96k.mp4"

bin\sox "sound_temp\sample01.faac.abr96k.mp4.int44k.wav" -b 32 "sound_raw\sample01.faac.abr96k.mp4.int32.wav" trim 0s gain 0

bin\faac-1.28-mod\faac -q 30 -o "sound_out\sample01.faac.vbrq30.mp4" "sound_raw\sample01.int16.wav"
Freeware Advanced Audio Coder
FAAC 1.28

Quantization quality: 30
Bandwidth: 7600 Hz
Object type: Low Complexity(MPEG-4) + M/S
Container format: MPEG-4 File Format (MP4)
Encoding sound_raw\sample01.int16.wav to sound_out\sample01.faac.vbrq30.mp4
  frame          | bitrate | elapsed/estim | play/CPU | ETA
  347/347  (100%)|  54.5  |    0.2/0.2    |  32.36x | 0.0


bin\faad -b 2 -q -o "sound_temp\sample01.faac.vbrq30.mp4.int44k.wav" "sound_out\sample01.faac.vbrq30.mp4"

bin\sox "sound_temp\sample01.faac.vbrq30.mp4.int44k.wav" -b 32 "sound_raw\sample01.faac.vbrq30.mp4.int32.wav" trim 0s gain 0

bin\ffmpeg63227 -y -i "sound_raw\sample01.int16.wav" -c:a mp2 -b:a 96k "sound_out\sample01.ffmpeg.96k.mp2"
ffmpeg version N-63227-g43c18fe Copyright © 2000-2014 the FFmpeg developers
  built on May 18 2014 04:21:13 with gcc 4.8.1 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-libfdk-aac --enable-libfaac --enable-libmp3lame --enable-libvo-aacenc --extra-ldflags=-static --extra-cflags='-march=native -m
  libavutil      52. 84.100 / 52. 84.100
  libavcodec    55. 62.100 / 55. 62.100
  libavformat    55. 38.100 / 55. 38.100
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter    4.  5.100 /  4.  5.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample  0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Guessed Channel Layout for  Input Stream #0.0 : stereo
Input #0, wav, from 'sound_raw\sample01.int16.wav':
  Duration: 00:00:08.03, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Output #0, mp2, to 'sound_out\sample01.ffmpeg.96k.mp2':
  Metadata:
    encoder        : Lavf55.38.100
    Stream #0:0: Audio: mp2, 44100 Hz, stereo, s16, 96 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s16le -> mp2)
Press [q] to stop, [?] for help
size=      94kB time=00:00:08.03 bitrate=  96.1kbits/s   
video:0kB audio:94kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

bin\madplay -q -b 24 -o "sound_temp\sample01.ffmpeg.96k.mp2.int44k.wav" "sound_out\sample01.ffmpeg.96k.mp2"

bin\sox "sound_temp\sample01.ffmpeg.96k.mp2.int44k.wav" -b 32 "sound_raw\sample01.ffmpeg.96k.mp2.int32.wav" trim 481s gain 0

bin\ffmpeg63227 -y -i "sound_raw\sample01.int16.wav" -c:a libvo_aacenc -b:a 96k "sound_out\sample01.vo-aacenc013.abr96k.mp4"
ffmpeg version N-63227-g43c18fe Copyright © 2000-2014 the FFmpeg developers
  built on May 18 2014 04:21:13 with gcc 4.8.1 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-libfdk-aac --enable-libfaac --enable-libmp3lame --enable-libvo-aacenc --extra-ldflags=-static --extra-cflags='-march=native -m
  libavutil      52. 84.100 / 52. 84.100
  libavcodec    55. 62.100 / 55. 62.100
  libavformat    55. 38.100 / 55. 38.100
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter    4.  5.100 /  4.  5.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample  0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Guessed Channel Layout for  Input Stream #0.0 : stereo
Input #0, wav, from 'sound_raw\sample01.int16.wav':
  Duration: 00:00:08.03, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Output #0, mp4, to 'sound_out\sample01.vo-aacenc013.abr96k.mp4':
  Metadata:
    encoder        : Lavf55.38.100
    Stream #0:0: Audio: aac (libvo_aacenc) ([64][0][0][0] / 0x0040), 44100 Hz, stereo, s16, 96 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s16le -> libvo_aacenc)
Press [q] to stop, [?] for help
size=      97kB time=00:00:08.04 bitrate=  99.1kbits/s   
video:0kB audio:95kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.195797%

bin\faad -b 2 -q -o "sound_temp\sample01.vo-aacenc013.abr96k.mp4.int44k.wav" "sound_out\sample01.vo-aacenc013.abr96k.mp4"

bin\sox "sound_temp\sample01.vo-aacenc013.abr96k.mp4.int44k.wav" -b 32 "sound_raw\sample01.vo-aacenc013.abr96k.mp4.int32.wav" trim 576s gain 0

bin\bladeenc -quit -nocfg "sound_raw\sample01.int16.wav" "sound_out\sample01.bladeenc.96k.mp3" -96"

BladeEnc 0.94.2    © Tord Jansson            Homepage: [url=http://bladeenc.mp3.no]http://bladeenc.mp3.no[/url]
===============================================================================
BladeEnc is free software, distributed under the Lesser General Public License.
See the file COPYING, BladeEnc's homepage or www.fsf.org for more details.

Files to encode: 1

Encoding:  sound_raw\sample01.int16.wav
Input:    44.1 kHz, 16 bit, stereo.
Output:    96 kBit, stereo.

Completed. Encoding time: 00:00:00 (1.#JX)                                 

All operations completed. Total encoding time: 00:00:00

bin\madplay -q -b 24 -o "sound_temp\sample01.bladeenc.96k.mp3.int44k.wav" "sound_out\sample01.bladeenc.96k.mp3"

bin\sox "sound_temp\sample01.bladeenc.96k.mp3.int44k.wav" -b 32 "sound_raw\sample01.bladeenc.96k.mp3.int32.wav" trim 1057s gain 0

bin\ffmpeg62757 -y -i "sound_raw\sample01.int16.wav" -c:a aac -strict experimental -b:a 96k "sound_out\sample01.ffmpeg.96k.mp4"
ffmpeg version N-62757-g18af0ce Copyright © 2000-2014 the FFmpeg developers
  built on Apr 28 2014 12:43:24 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-avisynth --enable-fontconfig --enable-libass --enable-libbluray --enable-libcaca --enable-libcdio --enable-libfreetype --enable-libgme --enable
  libavutil      52. 79.100 / 52. 79.100
  libavcodec    55. 59.100 / 55. 59.100
  libavformat    55. 37.101 / 55. 37.101
  libavdevice    55. 13.100 / 55. 13.100
  libavfilter    4.  4.100 /  4.  4.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample  0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Guessed Channel Layout for  Input Stream #0.0 : stereo
Input #0, wav, from 'sound_raw\sample01.int16.wav':
  Duration: 00:00:08.03, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Output #0, mp4, to 'sound_out\sample01.ffmpeg.96k.mp4':
  Metadata:
    encoder        : Lavf55.37.101
    Stream #0:0: Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s16le -> aac)
Press [q] to stop, [?] for help
size=      97kB time=8.03 bitrate=  98.6kbits/s   
video:0kB audio:95kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.209358%

bin\faad -b 2 -q -o "sound_temp\sample01.ffmpeg.96k.mp4.int44k.wav" "sound_out\sample01.ffmpeg.96k.mp4"

bin\sox "sound_temp\sample01.ffmpeg.96k.mp4.int44k.wav" -b 32 "sound_raw\sample01.ffmpeg.96k.mp4.int32.wav" trim 1s gain 0

bin\ffmpeg63227 -y -i "sound_raw\sample01.int16.wav" -c:a aac -strict experimental -b:a 96k "sound_out\sample01.ffmpeg_v8g.96k.mp4"
ffmpeg version N-63227-g43c18fe Copyright © 2000-2014 the FFmpeg developers
  built on May 18 2014 04:21:13 with gcc 4.8.1 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-libfdk-aac --enable-libfaac --enable-libmp3lame --enable-libvo-aacenc --extra-ldflags=-static --extra-cflags='-march=native -m
  libavutil      52. 84.100 / 52. 84.100
  libavcodec    55. 62.100 / 55. 62.100
  libavformat    55. 38.100 / 55. 38.100
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter    4.  5.100 /  4.  5.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample  0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Guessed Channel Layout for  Input Stream #0.0 : stereo
Input #0, wav, from 'sound_raw\sample01.int16.wav':
  Duration: 00:00:08.03, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Output #0, mp4, to 'sound_out\sample01.ffmpeg_v8g.96k.mp4':
  Metadata:
    encoder        : Lavf55.38.100
    Stream #0:0: Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s16le -> aac)
Press [q] to stop, [?] for help
size=      96kB time=00:00:08.03 bitrate=  98.3kbits/s   
video:0kB audio:94kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.217894%

bin\faad -b 2 -q -o "sound_temp\sample01.ffmpeg_v8g.96k.mp4.int44k.wav" "sound_out\sample01.ffmpeg_v8g.96k.mp4"

bin\sox "sound_temp\sample01.ffmpeg_v8g.96k.mp4.int44k.wav" -b 32 "sound_raw\sample01.ffmpeg_v8g.96k.mp4.int32.wav" trim 0s gain 0
===================Calc_bitrate Section Start===================
Sample Length:8.029342 seconds(354094 samples).
sound_out\sample01.faac.abr96k.mp4 File: 97897 Bytes. Bitrate: 97539 bps.
sound_out\sample01.faac.vbrq30.mp4 File: 56896 Bytes. Bitrate: 56688 bps.
sound_out\sample01.ffmpeg.96k.mp2 File: 96548 Bytes. Bitrate: 96195 bps.
sound_out\sample01.vo-aacenc013.abr96k.mp4 File: 99692 Bytes. Bitrate: 99328 bps.
sound_out\sample01.bladeenc.96k.mp3 File: 96549 Bytes. Bitrate: 96196 bps.
sound_out\sample01.ffmpeg.96k.mp4 File: 99047 Bytes. Bitrate: 98685 bps.
sound_out\sample01.ffmpeg_v8g.96k.mp4 File: 98674 Bytes. Bitrate: 98313 bps.
8029 97897 56896 96548 99692 96549 99047 98674 97539 56688 96195 99328 96196 98685 98313 642944
===================Rawgene5 Section Start===================
rawgene v.5.0.2  /  original reference file:sound_raw\sample01.int16.wav
                    Offset[sample]  Gain  Gain[dB]  Score  MaxAmplitude
INX 1>sample01.faac.>        0  0.995208 -0.041721  0.103592  0.386957
INX 2>sample01.faac.>        0  0.997336 -0.023173  0.237759  0.405624
INX 3>sample01.ffmpe>        0  0.998974 -0.008913  0.079806  0.377329
INX 4>sample01.vo-aa>        0  1.001523  0.013216  0.074541  0.372431
INX 5>sample01.blade>        0  1.008047  0.069618  0.186137  0.364100
INX 6>sample01.ffmpe>        0  1.007290  0.063088  0.080842  0.366759
INX 7>sample01.ffmpe>        0  1.008907  0.077025  0.097769  0.393529
biggest delay    : 0
smallest delay  : 0
original max amp : 0.377045
global gain      : 1.000000
usable length    : 353759
original rate    : 44100



Personal Listening Test of bad/experimental AAC/MP3 Encoders

Reply #2
 Thank You very much, kamedo2. Now it's clear which anchors to use.

 

Personal Listening Test of bad/experimental AAC/MP3 Encoders

Reply #3
Quote
FFmpeg's native AAC encoders had the best score among the 7 settings tested.


This got me curious, but the first sample I encoded with ffmpeg (N-64919-ga613257 built on Jul 23 2014) completely failed (crackle), too bad too because other parts of the song are a lot less warbly than vo-accenc.

 
SimplePortal 1.0.0 RC1 © 2008-2019