Resurrecting/Preserving the Helix MP3 encoder

Topic: Resurrecting/Preserving the Helix MP3 encoder (Read 47012 times) previous topic - next topic

0 Members and 6 Guests are viewing this topic.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #275 – 2024-03-25 23:08:04

@maikmerten I got the impression that the development of Helix stalled just at the moment when they reached 16 kHz, and frequencies above this threshold remained something like “here be dragons”, where you are allowed at your own risk with HF flag. For example, see and listen to what happens in the following case of a noisy HF range.

Spoiler (click to show/hide)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #276 – 2024-03-26 01:38:19

Somehow choosing AVX optimized gcc options do almost nothing here as do all other optimizations. The code compiles in a few seconds.
I'd be cautious with fastmath since the ouput of lossy codecs may suffer. At least the resulting files differ and may result in degration.
I guess the -fno-math-errno -fno-trapping-math already in the makefile is all.
I attached a simple generic x64 compile that seems not to be faster as former maikemertens x64 build but smaller.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #277 – 2024-03-26 02:06:34

Quote from: Kraeved on 2024-03-25 23:08:04

@maikmerten I got the impression that the development of Helix stalled just at the moment when they reached 16 kHz, and frequencies above this threshold remained something like

Is there a reason why this encoder doesn't put a lowpass filter after -V80?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #278 – 2024-03-26 06:18:53

Quote from: Kraeved on 2024-03-25 23:08:04

@maikmerten I got the impression that the development of Helix stalled just at the moment when they reached 16 kHz, and frequencies above this threshold remained something like “here be dragons”, where you are allowed at your own risk with HF flag. For example, see and listen to what happens in the following case of a noisy HF range.

Spoiler (click to show/hide)
Input.

Output. hmp3 -V150

Output. hmp3 -V150 -HF2

Output. lame3395o -Q1

I think the funny-looking problems in the HF-range may stem from difficulties in the MP3 format to efficiently encode content beyond 16 kHz itself. Due to how the MP3 bitstream format is designed, encoding high frequencies with high precision comes at the expense of being wasteful (and thus inefficient) in the lower frequencies, due to the high-frequency band not having a separate scale factor ("sfb21 defect"). Helix appears to favor giving precision to the lower frequency bands, which in most cases should be the "sane" choice.

Your test signal doesn't just happen to have high-frequency content, its HF-energy is completely blasting over the more relevant content in the lower frequency ranges. This is certainly not something the psychoacoustic model would be expecting. I also hope that you added the high-frequency high-energy noise there artificially and isn't part of an actual recording.

For comparison, I enclosed another LAME encoding (3.100 -q0 -V0). It manages to preserve the high frequency content much better in terms of how the spectrogram looks, but sounds worse to me, adding a more pronounced metallic warbling noise. It's clear the encoder is struggling, too, using 100% 320 kbps frames, which isn't typical for a VBR encoding.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #279 – 2024-03-26 06:26:28

Quote from: vina54 on 2024-03-26 02:06:34

Is there a reason why this encoder doesn't put a lowpass filter after -V80?

Without the -HF switch (which is only available starting with -V80), the encoder will not encode anything beyond 16 kHz anyways, no matter what, so there's an implicit lowpass by not coding >16 kHz signals. With the -HF switch and starting with -V80, it's expected the encoder will make somewhat sane decisions on how to approximate the >16 kHz band, not expecting a lot of energy there anyways.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #280 – 2024-03-26 08:47:27

Quote from: maikmerten on 2024-03-26 06:18:53

For comparison, I enclosed another LAME encoding (3.100 -q0 -V0).

I replaced LAME 3.100 (there is even some 3.101 beta 3 out there) with tweaked 3995o, because the new psychoacoustic model (vbr-new vs vbr-old) left much to be desired. Perhaps an earlier version would have been even better, but I had to stop the exploration somewhere and settle down to stay sane. How do you like the sound of the 3995o encoded sample I attached above?

Quote from: vina54 on 2024-03-26 02:06:34

Is there a reason why this encoder doesn't put a lowpass filter after -V80?

If we run hmp3 with -EC flag, we will see more information about the encoding process:
V0-52 uses freq_limit 11713-15847, V53-150 uses freq_limit 16536, -HF2 raises freq_limit to 24000.
We can use -F flag to modify that threshold, e.g. -HF2 -F18000 makes freq_limit 18000.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #281 – 2024-03-26 09:58:55

Quote from: Kraeved on 2024-03-26 08:47:27

How do you like the sound of the 3995o encoded sample I attached above?

Your LAME 3.995 encode did overall a better job, certainly better than LAME 3.100 for this sample. However, as one can see in the spectrogram, it somewhat avoided most of the sbf21 trouble by effectively lowpassing the high frequency band and thus getting rid of the excessive energy there, which might be the best tradeoff in this case.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #282 – 2024-03-26 12:29:31

@maikmerten Forgive my intuitive observations and vernacular language, but it looks as if LAME 3995o -Q1 sets the upper threshold around 18.5 kHz and processes the entire material, while Helix splits the material into two parts (0-16 kHz and 16-18.5 kHz with -HF2 -F18500) and the issue arises in the very upper part, which is separated from the rest of the material by a certain (unexpected black) gap on the spectrogram and that discontinuity brings metallic vibes. What does the Helix code say: is this approach to high frequencies hardwired into the psychoacoustic model and it is difficult to change it without damaging the rest, or is it some parameter that can be adjusted?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #283 – 2024-03-26 12:56:55

As far as I can tell, Helix chooses encoding parameters for the < 16 kHz bands (e.g., global and per-band scale factors) and then encodes > 16 kHz with whatever global scale has been selected. Helix won't trade off quality in the lower bands to accommodate the high frequency band (as might be necessary to increase >16 kHz resolution due to the sfb21 MP3 format problem). This is actually a pretty sane approach (perhaps somewhat resembling LAME's -Y parameter) and should be pretty robust apart from (hopefully rare) edge cases. It would explain why >16 kHz looks disconnected from the other frequency bands.

The code to achieve all of this is rather obtuse, with poor naming conventions regarding variable names etc., so I might misread things. I certainly am not sure of my findings. The relevant logic is in bitallo3.cpp, though.

There doesn't appear to be a parameter to influence that behavior.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #284 – 2024-03-26 14:09:16

@maikmerten Hence, the serious question arises: should we use -HF2 flag at all, by itself and in conjunction with -F to limit the upper threshold? If I remove frequencies below 16 kHz and listen to what's left above, then I hear them clearly, so it seems that I will lose not some barely audible scratching, but something significant. On the other hand, how correct is it to estimate losses in this way, since we listen to the material in its entirety (think AAC-compressed music on Youtube that is limited to 16 kHz and yet is capable of pumping up your mood) and agree in advance that it is lossy. In particular, @halb27, who tweaked the mentioned Lame 3995o during his lifetime, was inclined to believe that we should pay less attention to -HF2 and rather enjoy a high-quality and quickly encoded range from 0 to 16 kHz, containing a musical basis. Probably, samples that sound better with -HF2 would help answer this question.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #285 – 2024-03-26 14:38:55

Surely you are now venturing into areas of personal preference? The Wiki entry should offer guidance and nothing more.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #286 – 2024-03-26 14:58:00

@john33 We don't have access to the original developers who could give us the certainty that we would put on the Wiki. Therefore, there is this topic where, like archaeologists, we try to reconstruct the picture (of the psychoacoustic model, in particular) from fragments as much as possible. It's a form of brainstorming that consists of researching the code, processing samples, asking uncomfortable questions and searching for answers. It may well turn out that Helix was never designed to process the entire spectrum of frequencies and -HF2 was just one of many experiments, and if so, then people will need to be warned about this. We can put it in the section “Usage guide”, or “Recommended usage”.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #287 – 2024-03-26 15:22:47

Didn't this encoder come out during the days of dialup? Wasn't the main focus back then is to make the encoding as efficient as possible at lower bitrates? (similar to FDK)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #288 – 2024-03-26 15:50:37

Yeah, sure, it's valid to ask how to properly operate the tools at our disposal.

Regarding -HF2, I'm not very worried. As was for the MP3 format itself, >16 kHz frequencies may have been are more of an afterthought for the Helix encoder, but the approach chosen appears sane. Also, there are listening tests with -HF2 that show that Helix is doing plenty fine, e.g., https://hydrogenaud.io/index.php/topic,113324.0.html - so if the HF implementation is an afterthought, it's a pretty solid afterthought.

As for the lullaby sample: It's a case where the actually significant audio is drowned in near-inaudible HF-noise. I attached an image with a 16 kHz lowpassed version showing the "actual" content (top tracks), compared to the sea of HF noise (bottom tracks). In the killer-sample thread, it has been demonstrated that this might have been due to noise-shapened dither dumping all that energy into the HF range. I'd strongly advise not to generalize findings from this sample, it's apparently already processed in ways that are detrimental to psychoacoustic compression.

So I wouldn't recommend always lowpassing at 16 kHz (or just leaving the -HF switch to off) for now. In my younger years, I certainly could distinguish between files with a 16 kHz lowpass and the original sample. Now it's two decades later, so I most likely don't need to worry one way or another anymore, but apparently there are still young folks with good ears out there.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #289 – 2024-03-26 18:52:27

Mr. @Porcus, over the years you have been painstakingly processing your diverse library of samples with various audio tools. Perhaps you have a few samples that are rich enough above 16 kHz so they could potentially sound better with -HF2 flag, which obliges Helix MP3 encoder not to cut frequencies above the mentioned threshold?

Also, I compared this x64 build by @Wombat (577 024 B) with the current x86 favorite by @maikmerten (1 240 909 B).

Code: [Select]

-------------------------------------------------------------------------------------
  Command                                         Mean [s]        Min [s]   Max [s] 
-------------------------------------------------------------------------------------
  hmp3-i686-sse.v2.exe in.wav -V150 -HF2 -U2 -D   5.906 ± 0.048   5.821     5.954   
  hmp3-wombat-x64.exe in.wav -V150 -HF2 -U2 -D    5.381 ± 0.042   5.306     5.431   
-------------------------------------------------------------------------------------

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #290 – 2024-03-26 22:41:28

I don't have "problem samples" for lossy encoding. (Of course I "have", but I have not catalogued them as such - I'm sticking to the lossless department.)

Anyway, 30 seconds of harpsichord which needs quite some bitrate for lossless compression: https://filebin.net/ivsp7ep0jk1gav68
See the spectrogram ... but it has been known for quite a while that harpsichord is "difficult".

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #291 – 2024-03-27 01:09:39

@maikmerten
I saw you removed "-fno-math-errno -fno-trapping-math" from the makefile because of being careful about FP-math optimizations.
I guess because i mentioned concerns about math optimizations may change lossy encoders behaviour.
The 2 flags you choosed didn't change the output here but also didn't help with speed on a 5900x.
It is more obvious when i enable fast-math or AVX2.
I really may be overcautious but i think we once had problems with a lame compile because of such things.
Sorry if i have confused anyone.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #292 – 2024-03-27 05:18:57

Quote from: Wombat on 2024-03-27 01:09:39

@maikmerten
I saw you removed "-fno-math-errno -fno-trapping-math" from the makefile because of being careful about FP-math optimizations.
I guess because i mentioned concerns about math optimizations may change lossy encoders behaviour.
The 2 flags you choosed didn't change the output here but also didn't help with speed on a 5900x.
It is more obvious when i enable fast-math or AVX2.
I really may be overcautious but i think we once had problems with a lame compile because of such things.
Sorry if i have confused anyone.

I reconsidered including those flags because I didn't find a speed improvement on x64 (Zen2-based Ryzen here), but these optimizations might perhaps make searching for problems more difficult (no errno, no traps).

So if these options don't gain speed, but may make detecting problems (which may arise on yet-untested platforms) more difficult, I felt they were superfluous.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #293 – 2024-03-27 15:46:28

Current git x64 generic.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #294 – 2024-03-27 16:31:49

Thanks guys, anyone and everyone who contributed to this project's revival. The encoding speed is phenomenal.

For the frequencies above 16 kHz, though they may largely be lowpassed out by default, wouldn't the lower frequencies still generate some harmonics in that range during playback?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #295 – 2024-03-27 19:53:47

Quote from: sld on 2024-03-27 16:31:49

wouldn't the lower frequencies still generate some harmonics in that range during playback?

They shouldn't be powerful enough to replace the cutoff! Even if the distortions in your playback chain are higher than normal

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #296 – 2024-03-27 20:01:28

Quote from: Wombat on 2024-03-27 15:46:28

Current git x64 generic.

That's what I was afraid of. As soon as the changes began to be committed to Github, enthusiasts started publishing all sorts of binaries daily without explaining what's new. Like dig that out of the nerdy depths of the repository yourself. @Case's approach was more user-oriented: he attached own binaries here, describing the changes they contained, then processed the feedback. Nothing remained shrouded in darkness, and now the darkness is thickening again.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #297 – 2024-03-27 20:08:27

Do you bother to check the github before you make these comments? Nothing has changed quality-wise.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #298 – 2024-03-27 20:14:59

Quote from: Kraeved on 2024-03-27 20:01:28

That's what I was afraid of. As soon as the changes began to be committed to Github, enthusiasts started publishing all sorts of binaries daily without explaining what's new. Dig that out of the nerdy depths of the repository yourself. @Case's approach was more user-oriented: he described changes, attached binaries and processed the feedback here.

Nobody forces you to download. Besides that above i mentioned the changes to makefile.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #299 – 2024-03-27 21:58:04

Quote from: john33 on 2024-03-27 20:08:27

Do you bother to check the github before you make these comments? Nothing has changed quality-wise.

What is it? @john33, are you ready to put your reputation on the line and walk out the window if this change does affect the quality to speak about it so confidently? Or will we let the author of the edit have his say? Helix's legacy is already fraught with obscure code and mysterious options such as -TX and -SBT, so it is not surprising there is increased attention to changes, especially unannounced.

Notice