Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal Listening Test of LAME, iTunes and Helix MP3 encoders (Read 18155 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Abstract:
Blind sound quality comparison between MP3 encoders at 192 kbps.

Encoders:
Helix mp3 encoder v5.1 -V111 -HF2
LAME 3.99.5 -V2.3
LAME 3.100.alpha2 -V2.75
iTunes 12.4.2.4 high quality default, vbr enabled

Samples:
Total 15 samples from my corpus, tested twice per sample.
Total 12 samples from IgorC's test tracks, tested twice per sample.

Hardwares:
Sony PSP-3000 + RP-HT560(1st), RP-HJE150(2nd).

Results:

I will post the bitrate of those settings over diverse albums in 2016 Dec 31. The average bitrate of the album and sample bitrate is very close to 192kbps and equal.



Conclusions & Observations:
The superiority of the Helix mp3 encoder is effectively proved in this test. Helix encodes diverse music tracks at better audio quality using the same bitrate compared to the LAME.
The experimental version LAME 3.100.alpha2 does not offer significantly better average quality, compared to the current stable version of the LAME 3.99.5.
The iTunes has not caught up to the quality of LAME encoders in the mp3 encoding, as of August 2016.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 27
Critical significance:  0.05
Significance of data: 1.42E-009 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              107           8.62
Testers (blocks)    26           4.02
Codecs eval'd        3           1.98    0.66   19.62  1.42E-009
Error               78           2.62    0.03
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.099

Means:

helixv   l3995v   l100a2   itunes   
  4.46     4.35     4.32     4.09   

---------------------------- p-value Matrix ---------------------------

         l3995v   l100a2   itunes   
helixv   0.029*   0.005*   0.000*   
l3995v            0.530    0.000*   
l100a2                     0.000*   
-----------------------------------------------------------------------

helixv is better than l3995v, l100a2, itunes
l3995v is better than itunes
l100a2 is better than itunes

Raw data:
Code: [Select]
LAME3.100α2	LAME3.99.5	HelixV5.1	iTunes12.4.2.4	
%feature	10	-V2.75	-V2.3	-HF2 -V111	High Quality VBR
%feature	11	201k 	201k	197k	194k
4.400	4.450	4.350	3.900	
4.300	4.200	4.500	4.150	
4.400	4.100	5.000	4.400	
4.450	4.250	4.800	4.400	
4.350	4.250	4.050	3.850	
4.400	4.150	4.300	3.900	
4.250	4.300	4.550	4.200	
4.500	4.350	4.700	4.200	
4.200	4.050	3.950	3.700	
4.250	4.500	4.750	3.950	
4.250	4.250	4.150	4.200	
3.800	3.850	4.200	3.950	
4.100	4.400	4.500	4.150	
4.000	4.300	4.400	3.750	
4.400	4.400	4.700	3.950	
4.100	4.150	4.350	3.750	
4.350	4.450	4.750	3.750	
4.100	4.250	4.350	3.650	
4.350	4.450	4.450	4.300	
4.300	4.250	4.400	4.300	
4.350	4.600	4.450	4.050	
4.100	4.150	4.250	4.100	
4.800	5.000	4.850	4.300	
4.600	4.600	4.050	4.300	
4.100	4.350	4.500	4.550	
5.000	4.900	4.750	4.600	
4.450	4.550	4.450	4.150	
%samples	10.41_30sec	hihats
%samples	11.finalfantasy	cemb.
%samples	12.ATrain	Jazz
%samples	13.BigYellow	Pops
%samples	14.FloorEssence	Techno
%samples	15.macabre	orch
%samples	16.mybloodrusts	guitar
%samples	17.Quizas	Latin
%samples	18.VelvetRealm	Techno
%samples	19.Amefuribana	Pops
%samples	20.Trust	Gospel
%samples	21.Waiting	Rock
%samples	22.Experiencia	Latin
%samples	23.Heart to Heart	Pops
%samples	24.Tom's Diner	Vocal
%samples	01 castanets	inst.
%samples	02 fatboy_30sec	Techno
%samples	03 eig	Techno
%samples	04 Bachpsichord	inst.
%samples	05 Enola	Techno
%samples	06 trumpet	inst.
%samples	07 applaud	Live
%samples	08 velvet	perc.
%samples	09 Linchpin	Rock
%samples	10 spill_the_blood	guitar
%samples	11 female_speech	Speech
%samples	12 French_Ad	Speech

Bitrates:
Code: [Select]
%bitrate
212892	213747	237940	192993
168438	155646	165261	192912
192156	192103	209717	193442
204127	205068	203986	193170
232176	233635	188200	193399
194632	186036	195050	193630
190862	202640	184752	193518
202882	200424	205121	193007
234691	231216	207721	194882
180358	180686	181454	193118
203168	206754	201816	192950
186724	189212	210432	193448
197553	195467	192937	193377
194530	194412	205535	193053
163321	166355	189490	193560
203531	209582	201625	196098
272590	277249	283337	193087
212430	217041	216048	194049
192217	176450	186486	193275
198941	190717	199853	193050
236340	228336	202575	194692
234272	230913	216725	195333
240073	247948	240665	194388
193818	198111	167236	193980
180981	180072	180092	192959
139441	145012	112302	193536
177329	180439	146029	194422

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #1 –
Thanks for the test, Kamedo!
I wonder what the authors of Helix did to make it so damn good.
How about encoding speed? As far as I remember, Helix was also significantly faster than Lame.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #2 –
great Kamedo2.

Interesting test. Oldies never die (as of Helix)

I wonder how LAME and Helix scale up to  256 kbps VBR.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #3 –
I added the album bitrates of all encoder settings to the graph image.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #4 –
Hi Kamedo2,

thank you for testing these mature codecs. The results are still surprisingly - at least to me.
Could you please tell me why you have choosen the -HF2 switch in case of the Helix codec?
I just did a short test and found out that -HF2 emphasizes the coding of signals >16 kHz more than the alternative -HF switch?
Doesn't this automatically lead to a lower coding quality in the "important" region <16 kHz where other codecs often apply
their low-pass filters? Have you testet -HF2 vs. -HF switch?

... and a Happy New Year!

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #5 –
For anyone who would like to try Helix, I have found these parameters work fine in foobar2000 converter:
Code: [Select]
- %d -V150 -HF2 -X2 -U2
-Vn is the VBR quality: 0-150 (150 is about 235 kpbs)
-HF2 is optional; I used it to be consistent with what OP did
-X2 I found necessary otherwise you get a bad VBR header; foobar2000 reports wrong track length and kbps
-U2 tells the encoder to use SSE (Pentium II and above)
-h will give you the detailed help page including all available switches

You can find the encoder at RareWares: http://www.rarewares.org/mp3-others.php
Development appears to have stopped in 2005.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #6 –
Quote from: 145dBSPL on
Could you please tell me why you have choosen the -HF2 switch in case of the Helix codec?
Actually, I tested the Helix without the -HF2 option on 224kbps, and it was not significantly better than the Lame.
https://hydrogenaud.io/index.php/topic,100896.0.html
So I needed something different to get some clearer results. Desteroid's comment sounds like using -HF1 or -HF2 on high bitrates is a good idea.
https://hydrogenaud.io/index.php/topic,100896.msg834601.html#msg834601

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #7 –
For those who need or prefer to use constant bit rate with the Helix mp3 encoder, these parameters work fine in foobar2000:
Code: [Select]
-X2 -U2 -B128 -HF2 -F18500 - %d

B128 corresponds to 256kbps CBR.  Use B160 if 320kbps is desired.  The -F command sets the lowpass filter.  Adjust or omit as preferred. 

The Helix encoder is lightning fast on my older laptop.  It will encode 500 songs from FLAC in about 10 minutes.  Perfect for a quick playlist switch on a micro SD card I use in an old flip phone with buggy software that doesn't like VBR and doesn't play gapless.  Lack of gapless encoding is the only drawback I know of in the Helix encoder.

Best regards.  LedHed8 

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #8 –
Quote from: LedHed8 on
Lack of gapless encoding is the only drawback I know of in the Helix encoder.
I also noticed that Helix introduces sector boundary errors; LAME does not.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #9 –
Lame 3.99 / 3.100 is very fast with -f and can be gapless with certain players. The quality of -f is fine and some problematic samples are  better strangely.  One could assume using --noreplaygain -f -V2 ..1.5..1.0  will bring it on-par.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #10 –
The score improvement of each encoders, relative to the LAME 3.99.5 VBR.
Unfortunately, LAME 3.100 alpha2 is more likely to degrade quality than to improve quality.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #11 –
Quote from: Apesbrain on
Quote from: LedHed8 on
Lack of gapless encoding is the only drawback I know of in the Helix encoder.
I also noticed that Helix introduces sector boundary errors; LAME does not.

RareWares hosts a newer version of Helix than you used back in 2017, namely 5.2.1 2022-12-19.
@Kamedo2 @LedHed8 @Apesbrain Could you please check if there has been any improvement since you commented?

• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #12 –
Quote from: Kraeved on
Could you please check if there has been any improvement since you commented?

I am aware of the thread Resurrecting/Preserving the Helix MP3 encoder and the maikmerten's Helix v5.2.1, 2022-12-19, but this thread is about the ancient Helix v5.1, and the modernized Helix version is out of the scope of this thread, sorry!

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #13 –
Quote from: Kraeved on
Quote from: Apesbrain on
Quote from: LedHed8 on
Lack of gapless encoding is the only drawback I know of in the Helix encoder.
I also noticed that Helix introduces sector boundary errors; LAME does not.

RareWares hosts a newer version of Helix than you used back in 2017, namely 5.2.1 2022-12-19.
@Kamedo2 @LedHed8 @Apesbrain Could you please check if there has been any improvement since you commented?


To the best of my recollection @maikmerten's work was about making both the encoded and decoded files more standards compliant and had nothing to do with tuning of the encoder. I stand to be corrected, but I believe that is the case.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #14 –
Quote from: Kraeved on
Could you please check if there has been any improvement since you commented?
No change. Still creates SBEs.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #15 –
@Apesbrain, could you explain what is a 'sector boundary error' and how to detect one?
Also, do you encounter such errors when encoding via pipe or when calling directly too (hmp3 in.wav out.mp3)?
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #16 –
I don't think Sector Boundary Error is correct term, that is related to incorrect writing of audio data to a CD.
This MP3 encoder just fails at being gapless. The way audio is sent to the encoder is irrelevant.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #17 –
Quote from: Kraeved on
@ApesbrainAlso, do you encounter such errors when encoding via pipe or when calling directly too (hmp3 in.wav out.mp3)?
Encoding via pipe. I guess it doesn't matter that SBEs are created. I was just curious since LAME did not exhibit this behavior. TBH, I'd need to check that again.

EDIT: Confirming LAME (32bits version 3.100) does not create SBEs.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #18 –
Quote from: Kraeved on
@Case, I hope you'll understand why the source storm.wv¹ is compressed with WavPack. My trust in FLAC has been shaken recently due to its inability to reliably preserve non-audio data.
No idea if WavPack always keeps all metadata. Better use zip.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #19 –
This is getting off-topic, but:

Quote from: Wombat on
No idea if WavPack always keeps all metadata. Better use zip.
WavPack does, it is designed as a "file compressor": Being "zip for wave" was WavPack's concept from day one as far as I know.  Storing headers and footers bit by bit (uncompressed!), and restoring files bit by bit. This makes it easier to use it as plug-in for audio editors, because the WAVE headers and footers - and metadata that might be there - are available.

That also goes for Monkey's, TAK, OptimFROG and MPEG-4 ALS. You can ask WavPack and TAK to discard non-audio, but AFAIK Monkey's doesn't have any such switch. (WavPack and Monkey's also support other formats than WAVE, and do the same thing there: store headers and footers.)

In FLAC, "foreign metadata" is a retro-fit that you must invoke both upon encoding and upon decoding. FLAC is - like Shorten - basically an audio compressor - for audio signals, not for audio files; yeah sure the signals typically come wrapped in a file, but FLAC wasn't designed to preserve files.


As for the "file compressor" formats, there have been ways ways to "fool" some of the encoders by feeding them non-compliant files, but most often they will give you your broken file back because well, they inherently process files. There are a couple of counterexamples where WavPack will fix a file and warn you about it (DSD terminating prematurely at odd bit or something), and in old days there was a counterexample where WavPack would fix a file without telling you.

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #20 –
Quote from: Wombat on
No idea if WavPack always keeps all metadata. Better use zip.

Quote from: Porcus on
This is getting off-topic, but:
WavPack does, it is designed as a "file compressor": Being "zip for wave" was WavPack's concept from day one as far as I know. Storing headers and footers bit by bit (uncompressed!), and restoring files bit by bit. This makes it easier to use it as plug-in for audio editors, because the WAVE headers and footers - and metadata that might be there - are available.

And if the historical reference, kindly provided by @Porcus, does not sound convincing enough and you want to play it extra safe by using a more conservative approach to compression, then it makes more sense to consider alternatives to ZIP such as Nanozip, Bzip3, Zpaq, Zstandard, and 7-zip. Nanozip stands out here because, even on vintage hardware, it squeezes WAV files with comparable efficiency to La and OptimFROG in a period of TAK time, i.e. in a matter of seconds.

Code: [Select]
 3 647 040 = storm.v150hf2.mp3
11 618 112 = storm.la
11 874 364 = storm.ofr
11 961 887 = storm.nz
12 052 258 = storm.tak
12 651 934 = storm.hhx3.wv
17 960 850 = storm.bz3
18 025 854 = storm.m3.zpaq
18 678 360 = storm.mx.7z
20 997 602 = storm.10.zst
22 556 344 = storm.zip
30 203 996 = storm.wav
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #21 –
I guess nobody being in this audio forum longer as a week considers zip for anything. I only wondered why someone consider flac fails for him because some weird wav files metadata.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Personal Listening Test of LAME, iTunes and Helix MP3 encoders

Reply #22 –
Helix was one of the best MP3 encoder in 2008, also. (What I did in this image below is to redo the visualization, that's all.)
https://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm
Spoiler (click to show/hide)