Text only | Text with Images

HydrogenAudio

Lossy Audio Compression => MP3 => MP3 - General => Topic started by: maikmerten on 2022-11-12 13:40:13

Title: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-12 13:40:13
Hello there,

given that the original Helix MP3 Encoder thread (https://hydrogenaud.io/index.php/topic,35531.0.html) has been inactive for years and I'm not sure if resurrecting threads *that* old is welcome, I'm starting a fresh one. If this is in itself welcome, I hope mods can merge threads to keep things tidy. Sorry for any inconvenience.

Given my interest in Text-to-Speech, I've been looking into efficiently coding speech with MP3 as "works everywhere" fallback. Lame certainly is the obvious choice for a MP3 encoder to investigate, but I was eager to also look into the Helix MP3 encoder for its reputation of being fast and robust at lowish bitrates.

The Helix community project was closed many years ago, but an archive with apparently the newest (?)  source code was recently posted over at https://hydrogenaud.io/index.php/topic,122473.0.html thankfully. I'm on Linux and am interested in doing server-side MP3 encodes. With really minor source code modifications (about three lines of code) and a custom Makefile (attached to this post) I could get this old code to compile with only some minor warnings (which in the future I might be able to rectify).

Some observations and opinions:

I'm somewhat tempted to put the code on GitHub for preservation, future compile fixes, and code maintenance. However, while the code is open source, it comes with a custom RealNetworks license that reeks of "open source yes, but let's keep all control". I believe it is possible to publish the source code, but the license reads a bit awkward. Is there somebody here with insights into the actual intent there?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-24 16:06:43
I now published a version of the Helix MP3 Encoder, with some minor code maintenance having been done: https://github.com/maikmerten/hmp3

My main platform is Linux, but the original project files for Windows are still in there. I fixed some compiler warnings, fixed some typos in help texts and also enabled writing Xing TOC headers by default, so VBR files now seek properly. The Makefile also includes targets for debugging and profiling.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-24 18:39:59
I have also successfully compiled this, 32 and 64 bit, but using the C codepath. "sbt.asm" and "xsbt.asm" generate errors when compiling within VS2019 and trying to correct those is above my pay grade, I'm afraid! The other .asm modules appear to compile OK but, obviously, I can't know for sure without these other two compiling. Even without the asm code, the speed is very respectable. I can upload these if anyone is particularly interested. They are, of course, Windows compiles.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-24 19:00:26
Thank you, John, for looking into the Windows side of things!

I looked over the ASM files - those are good, old, plain 32-bit x86 and won't integrate into 64-bit compiles. However, compilers got a lot better, and I'm seeing GCC do some pretty neat stuff with SSE by its own. I'm profiling things here and there and my current approach to speed is to see if one can do faster approaches in pure C.

For instance, rounding floats to int in the original C code path was done as such:

Code: [Select]
    int ix;

    if ( x >= 0.0 )
        ix = ( int ) ( x + 0.5f );
    else
        ix = ( int ) ( x - 0.5f );
    return ix;

However, this introduces hard to predict branches. A faster way to get to the same result is

Code: [Select]
return (int)(x + copysignf(0.5f, x));

(copying over the sign from the float to be rounded onto the 0.5 constant - yielding -0.5 for negative values, then adding that offset before truncating to int.)

GCC generates some pretty neat, branch-free code from that, operating with SSE bit operations to get the job done. Fast!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-25 18:24:44
Well, the asm code compiles perfectly well in VS2015 for 32 bit. Not tried to compile for 64 bit, but would expect a fail.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KrajzegaX on 2022-11-25 18:55:34
@john33 Can you provide a compiled Windows release for testing?
It's nice that there is some development on this codec. 😀
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-25 19:45:27
For any distribution of binaries, please include information on the license and where to get the source code. I guess having LICENSE.txt, RPSL.txt and RCSL.txt and a link to https://github.com/maikmerten/hmp3 in the archive with the binary should help being compliant with the licensing terms. Or just ZIP up the complete source code, license texts included.

Quote
(e) if You Deploy Covered Code in object code, executable form only, You must
include a prominent notice, in the code itself as well as in related
documentation, stating that Source Code of the Covered Code is available under
the terms of this License with information on how and where to obtain such
Source Code. You must also include the Object Code Notice set forth in Exhibit A
in the "about" box or other appropriate place where other copyright notices are
placed, including any packaging materials.

(That's not me trying to annoy anybody, but those are the terms RealNetworks released the code under.)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-11-25 20:01:27
Edit: err not so fast, Swine.
This RPSL.txt is a newer version than the one described at https://en.wikipedia.org/wiki/RealNetworks_Public_Source_License and that is classified as "free" by FSF/OSI.
And the differences are quite significant - there is a ton about "Research" purposes.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-25 21:07:31
Quote from: KrajzegaX on
@john33 Can you provide a compiled Windows release for testing?
It's nice that there is some development on this codec. 😀
Here you go:

https://www.rarewares.org/files/mp3/hmp3-main.zip (https://www.rarewares.org/files/mp3/hmp3-main.zip)

Slightly larger than required as source code, etc, is included with updated VS2015 project files. Executable is in the hmp3\Release folder.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-26 09:48:39
Quote from: Porcus on
Edit: err not so fast, Swine.
This RPSL.txt is a newer version than the one described at https://en.wikipedia.org/wiki/RealNetworks_Public_Source_License and that is classified as "free" by FSF/OSI.
And the differences are quite significant - there is a ton about "Research" purposes.

Hi Porcus, I think you may have confused RPSL.txt and RCSL.txt here.

RPSL.txt is the "proper" open-source license and, as far as I can see, matches https://opensource.org/licenses/RPSL-1.0 . It happens to contain the word "research" only once.

However, the RCSL.txt is a completely different beast and is not a "proper" open-source (in the sense of "libre") license:
Quote
(i) Source. Covered Code may be distributed in Source Code form only to
another Licensee (except for students as provided below).

Its main purpose appears to be "research use", with 24 instances of the word "research". Another use case of that license appears to be for commercial usage, for which additional terms may be agreed upon on top of the RCSL, as explained in the preamble.

Thankfully, LICENSE.txt makes clear that the awkward RCSL license only applies to those who actively sought to license the code in that fashion:

Quote
The contents of this directory, and (except where otherwise
 indicated) the directories included within this directory, are
 subject to the current version of the RealNetworks Public Source
 License (the "RPSL") available at RPSL.txt in this directory, unless
 you have licensed the directory under the current version of the
 RealNetworks Community Source License (the "RCSL") available at
 RCSL.txt in this directory, in which case the RCSL will apply.
(emphasis mine)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-11-26 10:09:46
Ah good. I just hurried to edit when I saw different version numbers and discrepancies, and I didn't to the checking properly. Sorry to anyone I have confused (that, hopefully, amounts to "myself").

Anyway, this means Helix is free software, just with a slightly more awkward "freedom comes with this note attached".

Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-26 10:22:07
Quote from: Porcus on
Anyway, this means Helix is free software, just with a slightly more awkward "freedom comes with this note attached".

Yeah, it's free software as far as I can tell, but also a clear example of needless "license proliferation" (inflating the number of open source licenses in existence). I speculate that releasing the code under a more traditional license, such as the LGPL, may have made it easier to actually get a community started, instead of Helix "fizzling out".
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-11-26 10:57:59
I guess it has to do with what Real thought of themselves? If they are important enough, they set the terms - and Real was everywhere on the 'net twenty-five years ago.
Not excusing, explaining ... but from what I can read at https://en.wikipedia.org/wiki/Comparison_of_free_and_open-source_software_licenses , most FOSS licenses are actually newer.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-26 13:22:35
Yeah, I guess that's a realistic assessment. To be fair, albeit I wished they would have gone with a more popular/known license, getting a company of that size to open the sources for key products is an achievement in itself. Their software, their terms - and that's perfectly fine.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2022-11-26 14:23:56
Everytime i stumbled across the helix encoder i wondered if it is possible to hack in the gapless effort lame uses?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-26 15:02:19
Quote from: Wombat on
Everytime i stumbled across the helix encoder i wondered if it is possible to hack in the gapless effort lame uses?

Well, should be pretty possible, I guess. Is gapless MP3 done using some tags expressing how many samples at the beginning and how many samples at the end shall be cut? Is there documentation somewhere?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2022-11-26 15:08:40
Quote from: maikmerten on
Well, should be pretty possible, I guess. Is gapless MP3 done using some tags expressing how many samples at the beginning and how many samples at the end shall be cut? Is there documentation somewhere?
I think that really is how it works. No idea where a complete documentation is.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-26 16:08:35
The values for encoder delay and padding are included in the 'lametag' added to the lame encoded mp3. The Helix encoder does not currently write such a tag.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-26 16:37:55
Quote from: john33 on
The values for encoder delay and padding are included in the 'lametag' added to the lame encoded mp3. The Helix encoder does not currently write such a tag.

Yeah, for Lame there's https://lame.sourceforge.io/tech-FAQ.txt

edit: even more relevant may be http://gabriel.mp3-tech.org/mp3infotag.html
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-27 11:56:03
There seems to be a bigger problem here, at least when using lame and foobar2000 as decoders, as both decode to fewer wave samples than in the original wave file on the file tested, although both to the same size. This suggests that the decoders are making delay/padding assumptions that don't hold true which is no great surprise but is a real problem for playback. Without a resolution, this makes the encoder unusable so far as I am concerned.

Edit: Actually, more worrying is that the encoder seems to 'throw samples away', or foobar properties is not reporting the correct number of samples. The encoded mp3 files show fewer samples that the original wave files and decoding to wav shows a further reduction in samples!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-11-27 12:26:11
Quote from: john33 on
both decode to fewer wave samples than in the original wave file on the file tested
Dumb question, but have you checked the actual wave files, not just fb2k's user output?
There have been instances where fb2k versions have reported wrong sample counts.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-27 12:38:19
Quote from: Porcus on
Quote from: john33 on
both decode to fewer wave samples than in the original wave file on the file tested
Dumb question, but have you checked the actual wave files, not just fb2k's user output?
There have been instances where fb2k versions have reported wrong sample counts.

No, not a dumb question at all. I hadn't appreciated that fb2k had that issue. Thanks for the heads-up. I will check but the issue appears consistent over all the files tested so far.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-27 14:03:38
Yeah, I'm seeing samples getting lost at the end as well. This might be an "off-by-one" error, where after reaching the end of the input file the last frame is not properly flushed to the output file or something of that sort. I also think that I'm seeing a difference depending on whether the Xing headers are written (-X2, Xing headers with TOC, the new default) or not (-X0 - no Xing headers).
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-27 15:49:47
I've just uploaded a fresh compile. It's here:

https://www.rarewares.org/files/mp3/hmp3-main.zip (https://www.rarewares.org/files/mp3/hmp3-main.zip)

The VS2015 project files have changed and there are a couple of preprocessor directives added which were in the old builds but somehow didn't get carried over.

The net result is that files encoded with this version seem to play in fb2k in gapless mode without any issue. I am currently listening to an album that runs continuously without gaps and all appears fine.

Let me know if there are any issues, but, otherwise, have fun. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-27 17:15:20
Hi John,

I guess the preprocess definitions happen to be the following?

Quote
<PreprocessorDefinitions>NDEBUG;WIN32;_M_IX86;IEEE_FLOAT;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>

(I don't see how those would lead to different behavior regarding sample counts, but perhaps things are pretty well hidden...)

Under Linux, with Wine, I think I'm still seeing samples being cut off at the end with your provided build. To test, I prepared a 1 second, 44.1 kHz (so exactly 44100 samples) mono WAV file, with a 441 Hz tone. To mark the beginning and the end, I reduced the volume for short sections (exactly three sine periods) at the beginning and the end ("sine441-notches.wav").

As can be seen in the enclosed screenshot, the Lame encode perfectly lines up with the original input file.

The Helix CBR encode is a bit longer then the original, but there's a considerable gap at the beginning. For VBR, the gap at the start appears to be identical, but there are fewer samples than in the input file. It doesn't seem to matter if I include a full Xing header (-X2) or none whatsoever (-X0).

Is there some kind of MP3 frame analyzer available somewhere? I very faintly remember Lame having had some kind of graphical analysis tool, but that's like 20 years ago, most likely written with GTK1 and most likely not operable nowadays if it hasn't seen updates. I'd love to see if the cut off samples are in the MP3 and decoders being a bit too eager to remove padding.


Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-27 17:48:27
It was actually "_M_IX86;IEEE_FLOAT" that were missing. I agree with all that you say, and without a 'lametag' equivalent, the problem will remain. I also seem to remember that lame had a frame analyzer but I'm not sure whether it compiles these days. I'll look into this further over the next few days.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-27 18:50:07
Cool, thanks! I'm looking into the Lame info tag specification and it's quite clear where it would have to be implemented. Looks pretty doable, but is of course a "once time permits" thing.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-27 21:39:55
mp3x: https://www.rarewares.org/files/mp3/mp3x.zip (https://www.rarewares.org/files/mp3/mp3x.zip)
Includes the required dlls.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-11-27 22:39:41
Quote from: john33 on
Quote from: Porcus on
Quote from: john33 on
both decode to fewer wave samples than in the original wave file on the file tested
Dumb question, but have you checked the actual wave files, not just fb2k's user output?
There have been instances where fb2k versions have reported wrong sample counts.

No, not a dumb question at all. I hadn't appreciated that fb2k had that issue.

I should have been more precise, but I didn't recall precisely the findings - but now I do, as the issue was noted, along with a fix, in the v2 beta 13 changelog (https://wiki.hydrogenaud.io/index.php?title=Foobar2000:Version_2.0_Beta_Change_Log).

The issue was detected in the binary compare "component" (in v2 it isn't anymore a separate "component") and then only when comparing pairs with at least one file malformed one way or another. I think it even was WAVE files. I have no idea if it would even manifest itself outside binary compare.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-28 07:21:04
Noted, thanks. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-28 08:15:37
Hi John, thanks for providing a working build for mp3x. It somewhat works on Linux with Wine (I cannot click anything - but I can operate it with the keyboard. Some weird issue with super-old GTK on Wine, I guess).

I could confirm that there sometimes actually are too few frames being emitted - it's not decoder trimming away samples. The encoding application works under the assumption that every call to the encoder will emit one frame of audio, but this isn't actually true: Very small frames seem not be emitted, they're only flushed out with additional audio frames being submitted. The lower the VBR quality, the fewer frames actually are being written out.

I now added logic to shove additional silent audio into the encoder until the expected number of frames is emitted and written to file. Now I'm getting a) the complete audio and b) consistent padding at the start and the end, no matter if CBR or VBR and no matter the VBR quality.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-28 08:55:12
Great work. As I was using the '-V150' option, that may explain why I didn't perceive that problem at the same level.

I'll download and recompile your modified code, then re-post it here for testing.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-28 09:28:10
Updated compile:

https://www.rarewares.org/files/mp3/hmp3-20221128.zip (https://www.rarewares.org/files/mp3/hmp3-20221128.zip)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: itisljar on 2022-11-28 12:33:51
Is it possible to leave tool recognition tag, like lame does when you check which tool encoded audio? It doesn't leave anything that foobar2000 recognizes.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-28 16:36:58
Quote from: itisljar on
Is it possible to leave tool recognition tag, like lame does when you check which tool encoded audio? It doesn't leave anything that foobar2000 recognizes.

Yeah, I'm currently reading into the LAME info tag specification with plans to implement that. It's needed for gapless playback anyways. No promises on when this will be ready, though, as I will have to implement this as time is available.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KrajzegaX on 2022-11-28 17:29:35
I tested latest build and found a bug. When I convert mp3 from stdin with a dash in foobar2000 there is a truncation at beginning of the sample. With a wav file it's fine.

One more thing, when I published a post with "Helix MP3 Encoder 5.1.1" then in tomp3.cpp I changed the values from 16 to 4 to shorten the gap at the end of the tracks.
It's here and below in parentheses:
https://github.com/maikmerten/hmp3/blob/6edb64282e29b92029c3723bfbdbc913b43da301/hmp3/src/test/tomp3.cpp#L644 (https://github.com/maikmerten/hmp3/blob/6edb64282e29b92029c3723bfbdbc913b43da301/hmp3/src/test/tomp3.cpp#L644)
Now that there are no such problems with different lengths of gap, these lines of code are unnecessary and cause a longer gap. What do you think?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-28 17:47:48
Hi KrajzegaX, I'll happily check out what reverting the 4 back to 16 will accomplish, thanks for the hint (I assumed those values were all original RealNetworks).

As for stdin-input: Thanks for testing. This might be a platform-specific thing, on Linux things seem to behave properly:

Code: [Select]
hmp3 - - < sine441-notches.wav > test.mp3

gives me a non-truncated output file, as enclosed screenshot illustrates. However, it's certainly good to know of that defect. That whole input code looks somewhat wonky to me anyways and I don't trust it in particular.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-28 17:56:13
Changing 4 to 16 in https://github.com/maikmerten/hmp3/blob/6edb64282e29b92029c3723bfbdbc913b43da301/hmp3/src/test/tomp3.cpp#L644 doesn't appear to have an effect on the gaps. (See enclosed image)

I consider this a good thing, as it may indicate that the new frame flushing logic works irrespective of how the input chunks were processed.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-28 19:12:16
Okay, got some initial LAME info tag going on there.

Code: [Select]
maik@maik-desktop:/tmp$ eyeD3 -P lameinfo test.mp3
/tmp/test.mp3                                                                                                                   [ 4.83 KB ]
--------------------------------------------------------------------------------------------------------------------------------------------
Encoder Version     : LAME-H511
LAME Tag Revision   : 2
VBR Method          : Unknown
Lowpass Filter      : 0
Encoding Flags      : 
ATH Type            : 0
Bitrate (Constant)  : 0
Encoder Delay       : 1675 samples
Encoder Padding     : 2304 samples
Noise Shaping       : 0
Stereo Mode         : Undefined
Unwise Settings     : False
Sample Frequency    : 48 kHz
MP3 Gain            : 0 (+0.0 dB)
Preset              : Unknown
Surround Info       : None
Music Length        : 0.00 Bytes
Music CRC-16        : 0000
LAME Tag CRC-16     : 0000

The padding values are placeholders, but at least I'm getting the values back that I wrote into the headers. Audacity also reads things and crops away the padding. (see screenshot)

Now, I planned to write "HELIX.511" into the encoder version field, but then tools won't recognize the info tag. For now I'm going for "LAME-H511" for compatibility, but I wonder if there's a clever way to avoid using the "LAME"-prefix and still be recognized...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KrajzegaX on 2022-11-28 19:22:15
Quote from: maikmerten on
As for stdin-input: Thanks for testing. This might be a platform-specific thing, on Linux things seem to behave properly:

Code: [Select]
hmp3 - - < sine441-notches.wav > test.mp3
I tried this line on the console "cmd" and indeed it doesn't truncate at the beginning. Is there a bug in foobar2000? I don't know. ¯\_(ツ)_/¯
Quote from: maikmerten on
Changing 4 to 16 in https://github.com/maikmerten/hmp3/blob/6edb64282e29b92029c3723bfbdbc913b43da301/hmp3/src/test/tomp3.cpp#L644 doesn't appear to have an effect on the gaps. (See enclosed image)

I consider this a good thing, as it may indicate that the new frame flushing logic works irrespective of how the input chunks were processed.
Nice that it ignores that "PCM buffer" code. Good job.  :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-29 15:10:52
Okay, I have the LAME Info tag now implemented somewhat complete.

Code: [Select]
maik@maik-desktop:/tmp$ eyeD3 -P lameinfo lunarsanctum.mp3 
/tmp/lunarsanctum.mp3                                               [ 5.33 MB ]
--------------------------------------------------------------------------------
Encoder Version     : LAME-H511
LAME Tag Revision   : 2
VBR Method          : Unknown
Lowpass Filter      : 16500
Encoding Flags      : 
ATH Type            : 0
Bitrate (Constant)  : 0
Encoder Delay       : 1680 samples
Encoder Padding     : 2028 samples
Noise Shaping       : 0
Stereo Mode         : Undefined
Unwise Settings     : False
Sample Frequency    : 44.1 kHz
MP3 Gain            : 0 (+0.0 dB)
Preset              : Unknown
Surround Info       : None
Music Length        : 5.33 MB
Music CRC-16        : 0000
LAME Tag CRC-16     : 0000

According to my limited testing, MPEG-1 (32 kHz, 44.1 kHz, 48 kHz) files now have a sample-accurate length when decoded with  a decoder supporting the LAME info tag.

(For MPEG-2 I disabled the info tag for now. This needs more testing, due to a different frame size.)

THIS IS EXPERIMENTAL. Don't mass-encode for long-term storage. Should be fine for portable "throw away" use.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-29 16:05:35
For reasons I don't quite understand, lame decodes an hmp3 encoded file accurately, as you describe, but fb2k ignores the tag info and decodes to the same length as the mp3. Similarly, fb2k reports the mp3 file length including the delay/padding whereas a lame encoded file is reported as the same length as the wave file. Interestingly, the little lametag program reports that the encoded mp3 file tag info only includes the version.
Code: [Select]
F:\Test-lame>lame --decode 101.mp3
input:  101.mp3  (44.1 kHz, 2 channels, MPEG-1 Layer III)
output: 101.wav  (16 bit, Microsoft WAVE)
skipping initial 2209 samples (encoder+decoder delay)
skipping final 2243 samples (encoder padding-decoder delay)
Frame#  9784/9785    80 kbps   MS

F:\Test-lame>lametag 101.mp3
LameTag - Reads the LAME tag from an mp3 file
Copyright (c) 2005 phwip
Release 0.4.1, compiled 2005-09-09

F:\Test-lame\101.mp3

LAME tags were introduced with LAME version 3.90, but
this file was encoded by an earlier version of LAME and
the header only contains a version string.

Encoder string:     LAME
Version string:     -H511?ð


F:\Test-lame>lametag 10.mp3
LameTag - Reads the LAME tag from an mp3 file
Copyright (c) 2005 phwip
Release 0.4.1, compiled 2005-09-09

F:\Test-lame\10.mp3
Tag revision:       0
Encoder string:     LAME
Version string:     3.101
Quality:            93 (V0 and q7)
Encoding method:    vbr new / vbr mtrh
Lowpass:            22,100Hz
RG track peak:      <not stored>
RG track gain:      -6.6dB (determined automatically)
RG album gain:      <not stored>
nspsytune:          yes
nssafejoint:        yes
nogap continued:    no
nogap continuation: no
ATH type:           5
Bitrate:            minimal (-b) bitrate 32
Encoder delay:      576 samples
Padded at end:      1,572 samples
Noise shaping:      0
Stereo mode:        joint
Unwise settings:    no
Source sample freq: 44.1kHz
MP3Gain change:     <none>
Preset:             V0: preset extreme (fast mode)
Surround info:      none
Music length:       8,352,387 bytes
Music CRC:          7BBD
Actual Music CRC:   7BBD
Info tag CRC:       E8D6
Actual InfoTag CRC: E8D6


F:\Test-lame>
10.mp3 (lame encoded) and 101.mp3 (hmp3 encoded) are both encoded from 10.wav. On reflection, it looks like the lametag program also checks the 'Version String' and it may be that fb2k does the same and ignores the info if the version string is not greater than 3.90. But I am only guessing. ;)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-29 16:41:57
Oh my, thanks for testing. Guess I'm running into some implementation-specific behavior there.

Possible source for problems:


I guess I'll have to install FB2K in Wine and see what's going on. Is that nifty little "lametag" program available somewhere?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-29 16:46:35
Here you go: https://www.rarewares.org/files/LameTag_0.4.1.zip (https://www.rarewares.org/files/LameTag_0.4.1.zip)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-29 16:59:33
Quote from: maikmerten on
Oh my, thanks for testing. Guess I'm running into some implementation-specific behavior there.

Possible source for problems:

  • The encoder name and version. I'm trying to make Helix encodes distinguishable from LAME encodes, but it might be that I need to "borrow" a LAME version string
  • I don't compute CRCs for the info tag or the music frames. Perhaps info tags with mismatching CRCs are ignored.

I guess I'll have to install FB2K in Wine and see what's going on. Is that nifty little "lametag" program available somewhere?
I think you're right. I just changed the version string to 'LAME39111' and fb2k picked up the delay and padding and decoded it correctly. 'Lametag', however, fails to see the tag at all so it may not be as useful/more restrictive.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-29 17:15:20
'LAME5.1.1' works in fb2k but not lametag. So, I think it's safe to assume that lametag is more restrictive and perhaps less useful.

Edit: I spoke prematurely!! While fb2k sees the info correctly for the mp3, it ignores it on decode!!!

Edit 2: LAME still decodes this correctly. It would seem, therefore, that while LAME knows what it's doing with the lametag, other programs seem to be imposing restrictions that LAME doesn't. Irritating and slightly annoying!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: doccolinni on 2022-11-29 17:32:35
Quote from: john33 on
It would seem, therefore, that while LAME knows what it's doing with the lametag, other programs seem to be imposing restrictions that LAME doesn't.

Sounds like something to file a bug report to those programs for.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-29 17:58:51
Okay, yeah, confusingly fb2k seems to read the padding info just right, but doesn't actually seem to take that information into account correctly.

Going for "LAMEH5.11" for now. The "." makes it look better parseable as version number. The "H" behind LAME denotes "Helix" (I really really don't want to steal an actual LAME version string). Also, the string now contains "MEH", which is neat.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-29 18:32:32
Actually, LAME works with 'HELIX5.11', and decodes correctly. However, fb2k recognises 'HELIX5.11' and quotes delay and padding numbers which differ from what LAME reports but sum to the same number. In Audacity, the original and decoded wavefiles seem to be sample accurate timewise. fb2k does not decode correctly.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-30 07:51:01
Yeah, LAME still decodes fine with HELIX5.11. However, my version of Audacity (2.4.2) doesn't seem to want to read the info tag if the encoder string doesn't start with LAME...

As for fb2k, I guess I might just ask for guidance in the fb2k forum.

And all of this started because I want to encode MP3 fallbacks for voice content on the web. ;-)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-30 12:38:43
As an aside, on my 8500T based test micro system, a 64 bit build of the C code using LLVM 15.0.5, optimised for AVX2 is roughly 10% faster than the 32 bit ASM build.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-11-30 12:44:42
Quote from: john33 on
As an aside, on my 8500T based test micro system, a 64 bit build of the C code using LLVM 15.0.5, optimised for AVX2 is roughly 10% faster than the 32 bit ASM build.

Did you specify the "-U2"-flag to use the Pentium III (=SSE) code path for 32-bit?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-11-30 13:57:57
Quote from: maikmerten on
Did you specify the "-U2"-flag to use the Pentium III (=SSE) code path for 32-bit?
Actually, no. Using -U2, results in more or less identical times.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-01 06:38:07
Thanks to Peter in the fb2k forums it became clear that the Helix encoder writes a slightly off (by +1) number of total frames into the Xing VBR header. fb2k doesn't decode until the last frame (then discards surplus samples), but uses the number of frames to determine the total number of samples, then subtracts the padding.

Long story short: I pushed changes to GitHub so that encodes should now be sample-accurate-length in fb2k.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-01 08:39:56
I can confirm all is good. Many thanks for your efforts in solving all the issues here. Great work! :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-01 09:28:53
@maikmerten, I hope you don't mind but I have taken the liberty of providing this on rarewares, attributed to you, in place of the old compile which obviously had issues.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-01 09:45:04
Quote from: john33 on
@maikmerten, I hope you don't mind but I have taken the liberty of providing this on rarewares, attributed to you, in place of the old compile which obviously had issues.

Thanks for your vote of confidence, your testing and providing Windows builds! :-)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-01 12:51:00
You might be amused to know that LameTag 0.4.1 still fails to read the tag. However, I discovered I had version 0.3.1, including the pascal/Delphi source, and that reads the tag perfectly!!

If you're interested, you can d/l from here: https://www.rarewares.org/files/LameTag-0.3.1.zip (https://www.rarewares.org/files/LameTag-0.3.1.zip)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2022-12-01 14:52:30
Wow! Never thought the gapless implementation will come at all, leave alone that fast. I asked for that several years back already.
Thanks for your time folks :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-01 16:17:41
Quote from: john33 on
You might be amused to know that LameTag 0.4.1 still fails to read the tag. However, I discovered I had version 0.3.1, including the pascal/Delphi source, and that reads the tag perfectly!!

If you're interested, you can d/l from here: https://www.rarewares.org/files/LameTag-0.3.1.zip (https://www.rarewares.org/files/LameTag-0.3.1.zip)

Well, I'm pretty sure that many MP3 tools have some built-in assumptions that are based on what LAME in particular is doing.... and who can blame them, with no other implementations choosing to implement the info tag?

Quote from: Wombat on
Wow! Never thought the gapless implementation will come at all, leave alone that fast. I asked for that several years back already.
Thanks for your time folks :)

You're welcome, it's a fun side-project!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-01 18:26:01
A slight conundrum. According to LameTag 0.4.1, the latest 'lame-svn-r6507-trunk' writes Revision 0 to the lametag. Strangely, LameTag 0.3.1 fails to read the lametag beyond indicating the lame version. Changing the Revision number to 0 in hmp3 results, in my tesfile, in 196 fewer bytes being written which is reflected in the padding being reduced by the number. fb2k appears to be Revision agnostic as it displays the correct information for both files and decodes correctly. Probably of no consequence but just struck me as odd.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-01 19:18:12
Quote from: john33 on
Changing the Revision number to 0 in hmp3 results, in my tesfile, in 196 fewer bytes being written which is reflected in the padding being reduced by the number.

I'm not sure I follow. Did you edit xhead.c to simply write "version 0" into the info tag version field and suddenly fewer bytes are emitted?

I used http://gabriel.mp3-tech.org/mp3infotag.html as specification. Sadly, I don't see information on what version that spec describes, so I just assumed it would be the presumably more recent one ("version 1").
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-01 20:12:24
Quote from: maikmerten on
Quote from: john33 on
Changing the Revision number to 0 in hmp3 results, in my tesfile, in 196 fewer bytes being written which is reflected in the padding being reduced by the number.

I'm not sure I follow. Did you edit xhead.c to simply write "version 0" into the info tag version field and suddenly fewer bytes are emitted?

I used http://gabriel.mp3-tech.org/mp3infotag.html as specification. Sadly, I don't see information on what version that spec describes, so I just assumed it would be the presumably more recent one ("version 1").
Yes, that's all I did. Bizarre, I know, but I can't follow the lame code well enough to understand what's going on.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-01 20:27:50
Quote from: john33 on
Yes, that's all I did. Bizarre, I know, but I can't follow the lame code well enough to understand what's going on.

Ah, so you played around with LAME, not the Helix encoder?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-01 20:38:24
No, I changed the Revision number in xhead.c. I wasn't trying to be contentious, I was just trying to understand the difference between the revisions. ;)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-02 06:11:14
Quote from: john33 on
No, I changed the Revision number in xhead.c. I wasn't trying to be contentious, I was just trying to understand the difference between the revisions. ;)

Ah, okay, thanks for confirming. Then I don't see a reason why just changing the version of the info tag in xhead.c would have any effect on the number of bytes being written. The length of the header is determined before the actual encoding loop (line 607 in tomp3.cpp) and there's no code there to switch between version 0 and 1. The actual info header contents (including the version number) are written after the encoding loop (line 777 in tomp3.cpp). At that point the file size should already be final...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-02 08:17:13
Apologies, I think I managed to confuse myself and mixed up what was encoded using the asm build and that which was encoded with the 64 bit C build. I have just confirmed that you are, of course, correct. Sorry about that!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-02 09:23:49
No problem, at least this confirms that the ASM builds follow a different codepath ;-)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KrajzegaX on 2022-12-04 16:02:20
I managed to get gapless in CBR by modifying the header in the mp3 file and it works just like in VBR. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: iwod on 2022-12-04 17:02:59
Interesting, Did they ever release their raac encoder on Helix? Or was it MP3 only?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-04 18:14:39
Quote from: KrajzegaX on
I managed to get gapless in CBR by modifying the header in the mp3 file and it works just like in VBR. :)

Hi KrajzegaX, yeah, there's no proper reason why it shouldn't work for CBR - its just that RealNetworks didn't hook up the logic to submit the bitrate to their Xing-Header code. However, I know quite exactly where to hook things in and opened a ticket for that: https://github.com/maikmerten/hmp3/issues/1

(Long story short: For VBR, the header logic can just choose the smallest frame bitrate for the info frame that is big enough to fit all that information - it's VBR, after all, so frames can be of different size. For CBR, the header logic needs to pick exactly the same frame  bitrate as for the audio frames, it wouldn't be CBR otherwise. However, RealNetworks just submits "0" to the header code as bitrate, which means the information will never fit, thus the info frame is skipped)

I hope to get to that pretty soon.

Quote from: iwod on
Interesting, Did they ever release their raac encoder on Helix? Or was it MP3 only?

To my knowledge, they never opened their proprietary format encoders.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-04 19:19:37
Okay, took the liberty of enabling the Xing/Info-Header for CBR files now. Also, since yesterday, MPEG-2 files get the Xing/Info-Header as well.

edit: Note that for very low CBR bitrates, the info header may not fit and is then disabled.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-04 21:34:52
Fresh compiles at rarewares. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kamedo2 on 2022-12-05 12:42:45
I believe the version number should be bumped to Helix 6.
Thanks to the renovation effort of maikmerten, the new Helix is now very modern, compared to the ancient 2005 version, Helix v5.1.
Easier to recognize if the major version number is higher.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-12-05 13:26:53
I second that. Gapped playback is such a shortcoming that resurrecting it into a proper gapless-enabled encoder warrants a major-version-bump.

Quote from: maikmerten on
Going for "LAMEH5.11" for now. The "." makes it look better parseable as version number. The "H" behind LAME denotes "Helix" (I really really don't want to steal an actual LAME version string).
LAMELIX6.0 starts with "LAME", is sufficiently close to "HELIX" and ... well is the string too long? To test that, call it LAMELIX6.beta and see how many applications choke on it.

Quote from: maikmerten on
Also, the string now contains "MEH", which is neat.
If testing indicates that this works everywhere, I'd say you have de-meh'ed it successfully.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-05 14:11:02
A version bump is certainly a good idea, if only as a differentiator. Can I also just add that the lametag version should be set to '0', the same as is set in the latest lame svn.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: NetRanger on 2022-12-05 15:44:15
A version bump to v6 would be a great idea. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-05 16:31:03
Well, thanks for the kind suggestion of the changes being enough to justify a major version. Personally, I think that while a version bump can be well justified, I think it should suffice to bump the version to 5.2.0. My reasoning: The encoding algorithms were not changed even a tiny bit, and thus the quality is unchanged. This is merely an update with bugfixes, fewer compiler warnings and one actually new feature (gapless playback).

Just keep in mind how long LAME is on version 3.xx, with much, much bigger changes and how much tuning went into that!

Given that the encoder name string can only consist of nine characters, but for good compatibility I need it to start with "LAME", I want a major/minor/tiny-version number in there and a decimal point to have it look like a parse-able number, I guess there is not much room for major cleverness in there. For me, this looks like "LAMEH5.20"...

Is this something people can live with?

I'm testing setting the lametag version field to "0", to follow LAME's example. edit: Interestingly, with the header set to version "0", eyeD3 won't find a LAME info header... john33, did you find a case where version "0" works, but version "1" gives trouble?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-05 17:07:08
Quote from: maikmerten on
....
I'm testing setting the lametag version field to "0", to follow LAME's example. edit: Interestingly, with the header set to version "0", eyeD3 won't find a LAME info header... john33, did you find a case where version "0" works, but version "1" gives trouble?
Well, no, but Gabriel's Mp3 Info Tag revision 1 spec provides for 20 bytes between the Xing/VBR tag and the lametag which, unless I am much mistaken, is not currently allocated/used in the tag written by lame. I can see what looks like a provision for it in xhead.c but the conditions are never met and nothing gets written. I don't believe that Revision 1 has ever been implemented.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-05 18:33:10
Quote from: john33 on
Quote from: maikmerten on
....
I'm testing setting the lametag version field to "0", to follow LAME's example. edit: Interestingly, with the header set to version "0", eyeD3 won't find a LAME info header... john33, did you find a case where version "0" works, but version "1" gives trouble?
Well, no, but Gabriel's Mp3 Info Tag revision 1 spec provides for 20 bytes between the Xing/VBR tag and the lametag which, unless I am much mistaken, is not currently allocated/used in the tag written by lame. I can see what looks like a provision for it in xhead.c but the conditions are never met and nothing gets written. I don't believe that Revision 1 has ever been implemented.

Yeah, I think you're right. Version 0 it is, then.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-05 18:36:29
Quote from: maikmerten on
...
Yeah, I think you're right. Version 0 it is, then.
In truth I don't think anything out there pays any attention to it, but I do think '0' is correct. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-05 21:27:55
Rarewares compile updated. :) Huge thanks to @maikmerten for excellent work in bringing this project up-to-date.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-06 07:19:21
Thanks for testing and making builds available :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: jaybeee on 2022-12-06 09:34:26
I updated the Xing wiki page (https://wiki.hydrogenaud.io/index.php?title=Xing).

I also searched the wiki for Helix entries and there were a few in the Private Listening Test page (https://wiki.hydrogenaud.io/index.php?title=Private_Listening_Tests), so I updated the links and then decided to update all the url links as they were all borked.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: [JAZ] on 2022-12-06 12:36:05
Quote from: jaybeee on
I updated the Xing wiki page (https://wiki.hydrogenaud.io/index.php?title=Xing).

Mmm.. Shouldn't AudioCatalyst (https://www.rarewares.org/rrw/audiocat.php) be mentioned too there?  (and at RRW, there is also the Xing encoder)

Quote from: maikmerten on
Given that the encoder name string can only consist of nine characters, but for good compatibility I need it to start with "LAME", I want a major/minor/tiny-version number in there and a decimal point to have it look like a parse-able number, I guess there is not much room for major cleverness in there. For me, this looks like "LAMEH5.20"...

I don't remember it exactly now, but I remember that when lame went to lame 3.100, there were some software that choked with the version string change. Not sure if that's what happens with LAMEX5.11.

Also, I see this in the lame code ( VbrTag.c)
Code: [Select]
    /*
    NOTE:
            Even though the specification for the LAME VBR tag
            did explicitly mention other encoders than LAME,
            many SW/HW decoder seem to be able to make use of
            this tag only, if the encoder version starts with LAME.
            To be compatible with such decoders, ANY encoder will
            be forced to write a fake LAME version string!
            As a result, the encoder version info becomes worthless.
    */
    const char *szVersion = get_lame_tag_encoder_short_version();
Code: [Select]
/*! Get the _very_ short LAME version string. */
/*!
  It's used in the LAME VBR tag only, limited to 9 characters max.
  Due to some 3rd party HW/SW decoders, it has to start with LAME.

  \param void   
  \return a pointer to the short version of the LAME version string.
 */
const char*
get_lame_tag_encoder_short_version(void)
{
    static /*@observer@ */ const char *const str =
            /* FIXME: new scheme / new version counting / drop versioning here ? */
    "LAME" STR(LAME_MAJOR_VERSION) "." STR(LAME_MINOR_VERSION) P
    ;
    return str;
}
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: jaybeee on 2022-12-06 13:45:46
Quote from: [JAZ] on
Quote from: jaybeee on
I updated the Xing wiki page (https://wiki.hydrogenaud.io/index.php?title=Xing).

Mmm.. Shouldn't AudioCatalyst (https://www.rarewares.org/rrw/audiocat.php) be mentioned too there?  (and at RRW, there is also the Xing encoder)
Possibly.
Feel free to update the wiki as appropriate :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-06 14:15:03
Thanks for the Wiki updates!

And yeah, the code snippets seem to confirm that for maximum compatibility, one has to prefix the encoder name with "LAME"... :-/
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KrajzegaX on 2022-12-06 16:49:42
I found a problem with CBR files, it shows that there is a variable instead of constant bit rate. The reason is for VBR there should be an "Xing" header and for CBR there should be an "Info" header.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-06 17:11:50
Quote from: KrajzegaX on
I found a problem with CBR files, it shows that there is a variable instead of constant bit rate. The reason is for VBR there should be an "Xing" header and for CBR there should be an "Info" header.

I think the proper solution here is to use the "Info Tag revision + VBR method" field, as specified in http://gabriel.mp3-tech.org/mp3infotag.html#rev-method

I'll implement that.

edit: Okay, I now implemented both the "Xing" (VBR) vs "Info" (CBR) header marking and the "Info Tag revision + VBR method" to denote CBR encodings. Thanks for testing!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-06 21:33:27
Just tested this and while fb2k interprets it as expected, Windows Explorer doesn't see the correct bitrate and therefore shows an incorrect length. My test was at 320kbps and Windows showed 312kbps, so 4 mins 15 secs became 4 min 22 secs. Incidently, neither version of lametag could read the tag, although the newer version can read a lame encoded CBR tag.

Edit: I think the issue is that lame fills the Xing/Info header to the standard length and the current hmp3 'Info' tag doesn't.

Edit 2: Padding out the Xing/Info header results in the earlier lametag reader reading the tag correctly with correct delay/padding, but fb2k fails to read the delay/padding and Windows Explorer still shows the wrong length!! Seems to be something weird going on here!!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: radorn on 2022-12-06 22:45:35
Question from a noob: I never used it, although I remember seeing passing mentions of the Helix open source media project long ago. But now that it was long dead, what's the point of resurrecting the Helix MP3 codec? Does it do anything that LAME can't? Is it maybe just so that there's another free alternative to it?

I'm not being dismissive. It's just curiosity from an admitted ignorance standpoint.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: rutra80 on 2022-12-06 23:21:54
There are listening tests showing that Helix is as good as LAME quality-wise, if not better...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2022-12-06 23:24:37
helix was better on some music as lame. A very robust encoder imho. It never had to much attention and to this day i still don't know why -TX0 did best with some samples :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-12-06 23:34:28
Helix was known to be very fast - what does it take to encode on-the-fly in realtime nowadays?

And yes a good pair of ears could distinguish Helix from LAME even at a bitrate of 193: https://hydrogenaud.io/index.php/topic,113324.0.html
(The bar diagrams overlap, but the ANOVA test concludes Helix scores significantly better.)

Edit: my memory is just not reliable here. Surely I didn't remember Helix coming out on top in any test, but it was surprising that it even was competitive, given its reputation as a fast one ... but then, LAME had speed-ups at some stage too?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Eurobeat_fan on 2022-12-07 03:09:11
I remember that Helix didn't have any noticeable chirping at VBR ~128kbps while LAME V5 with current VBR model did. My memory may screwing around with me though.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-07 06:05:03
Quote from: john33 on
Windows Explorer doesn't see the correct bitrate and therefore shows an incorrect length.

Okay, that's a bit problematic to debug, given that I don't have the source code for Windows flying around. Does the length read correctly when writing "Xing" into the header instead of "Info" (which, apparently, some players need to detect CBR files)?

Quote from: radorn on
But now that it was long dead, what's the point of resurrecting the Helix MP3 codec? Does it do anything that LAME can't? Is it maybe just so that there's another free alternative to it?

Well, actually I went into this journey because I'm interested in low-bitrate 24 kHz "super-wideband" MP3 files to encode speech content in a "plays everywhere" format for the Web and I find (and I may have to be a bit cautious here to not infringe on TOS 8 here) Helix to fare better in low-bitrate situations. This *might* have something to do with LAME's psychoacoustic being mostly tuned for 44.1 kHz CD rips (the source code has a few "only tuned for 44.1 kHz" remarks), while Xing/RealNetworks/Helix presumably cared a bit more for low-bitrate (dial-up) applications.

For low bitrates, where transparency isn't expected to be achieved anyways, Helix does, e.g., support some stereo trickery ("intensity stereo" - basically just encode gentle hints on the stereo image, instead of "proper" mid/side stereo) that LAME doesn't support.

For instance, compare

Code: [Select]
lame -q 0 --lowpass 16000 --resample 44100 -b 96 <input.wav> <lame.mp3>

with

Code: [Select]
hmp3 <input.wav> <hmp3.mp3> -F16000 -B48 -N8

(96 kbps, 44.1 kHz, stereo for both encoders)

on a track such as Michael Jackson's "Beat it". For me, LAME collapses in to a warbly smeary mess, while Helix remains quite listenable for streaming-radio applications.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: synclagz on 2022-12-07 07:37:26
Quote from: rutra80 on
There are listening tests showing that Helix is as good as LAME quality-wise, if not better...

Yes. there is Kamedo2 listening test. Helix came on top.

https://hydrogenaud.io/index.php/topic,113324.0.html

Considering Helix quality (and taking it's speed into account, I really can't see why is not popular as lame - is much faster and better quality - Am I missing something here...? )
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-07 08:40:29
Quote from: maikmerten on
Okay, that's a bit problematic to debug, given that I don't have the source code for Windows flying around. Does the length read correctly when writing "Xing" into the header instead of "Info" (which, apparently, some players need to detect CBR files)?
Yes, that fixes it except that fb2k displays it as VBR but is otherwise correct.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-12-07 08:46:44
Quote from: synclagz on
I really can't see why is not popular as lame
Uh, well, ... a couple of shortcomings weren't fixed until ... like, a couple of weeks ago?  :D

(Besides, the "Helix" product range was originally associated with some DRM-infestation that Real Networks was peddling.)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-07 08:55:02
Quote from: synclagz on
Considering Helix quality (and taking it's speed into account, I really can't see why is not popular as lame - is much faster and better quality - Am I missing something here...? )

Well, LAME is a proper community project, was developed in the open and comes with a well-known and somewhat well-understood license (LGPL, IIRC). Helix comes with an exotic open-source license that isn't as well known and was developed in a CVS repository you had to register access for (and I think that was read-access only, with actual commits being done RealNetworks employees). Also, Helix was open-sourced back ca. 2004 ~ 2005, when LAME was already well-established and momentum switched to MP3's successors.

As for "better quality": I guess one would need a proper blind test with several participants, just like in the good old days here on HA, to collect more data on this. I really like what hmp3 produces, though. Perhaps its also a case of "the artifacts sound *different*", which sounds refreshing ;-)

Quote from: john33 on
Quote from: maikmerten on
Okay, that's a bit problematic to debug, given that I don't have the source code for Windows flying around. Does the length read correctly when writing "Xing" into the header instead of "Info" (which, apparently, some players need to detect CBR files)?
Yes, that fixes it except that fb2k displays it as VBR but is otherwise correct.

This might actually warrant rolling back to writing "Xing" in the header, even for CBR files. fb2k might be fixable, but Windows isn't.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-07 09:43:20
Quote from: maikmerten on
This might actually warrant rolling back to writing "Xing" in the header, even for CBR files. fb2k might be fixable, but Windows isn't.
That would be the quicker fix, I guess. Lame writes a full size Xing/Info tag even for CBR files and everything reads it correctly. I've been trying to figure out where the hmp3 tag writing differs that causes the issue. Clearly the lame written tag includes more info as it was designed for lame and the hmp3 code either doesn't generate the same information, or doesn't use the same functions/variables, but I believe there must be some other fundamental point that I'm missing! I will continue to try to find it!!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-07 10:11:23
Quote from: john33 on
That would be the quicker fix, I guess. Lame writes a full size Xing/Info tag even for CBR files and everything reads it correctly. I've been trying to figure out where the hmp3 tag writing differs that causes the issue. Clearly the lame written tag includes more info as it was designed for lame and the hmp3 code either doesn't generate the same information, or doesn't use the same functions/variables, but I believe there must be some other fundamental point that I'm missing! I will continue to try to find it!!

Hmmm... one thing I did change: If the file is CBR, I do not include the the Xing VBR Table of Contents (TOC). This makes the header smaller and thus fits into more CBR frame sizes. What happens if you comment out lines 241 and 242 in xhead.c?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: radorn on 2022-12-07 12:25:11
Thanks for all the explanations.
Please carry on  :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-07 13:46:04
Quote from: maikmerten on
Hmmm... one thing I did change: If the file is CBR, I do not include the the Xing VBR Table of Contents (TOC). This makes the header smaller and thus fits into more CBR frame sizes. What happens if you comment out lines 241 and 242 in xhead.c?
If I leave the header as 'Xing' and re-enable the TOC, all is OK except that fb2k calls it VBR. Lametag 0.3.1 recogises the tag as expected. If I then change the header to 'Info', all is good except that Windows now ignores the delay/padding, but Lametag 0.3.1 still reads the tag correctly. I know Windows is a law unto itself but since the lame 'Info' tag is read correctly by Windows, there must be something included in the lame written tag that is missing from the hmp3 written tag that is important to Windows. The question is, what?!?!

Edit: On reflection, I think it likely that Windows is probably expecting one, or more, positions in the tag to be filled with data that hmp3 doesn't generate, or can't fill because the encoder internals are different and the data isn't available in a suitable format. Unless someone has a bright idea, it's probably simpler to use the 'Xing' tag + TOC and perhaps Peter can tweak fb2k to see it as CBR, or otherwise just live with it.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-07 17:58:59
Okay, I think I got this fixed.

Turns out that for CBR the size of the first frame (where the info tag is stored) was only half of what it was supposed to be - because in the Helix MP3 encoder, the bitrate is set *per channel*, but I forgot to multiply the bitrate by the number of channels.

Thus "CBR" files were not _actually_ CBR, because the first frame was smaller than the rest. This most likely lead fb2k to say "VBR" (not all frames are of equal size), and I guess Windows got confused as well.

I now also write the TOC into CBR files, unless the bitrate is very low.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-07 18:13:33
Ladies and Gentlemen, I believe we have a winner! This now appears to meet all demands. Should we bump to 5.2.1 as this is not an insignificant fix?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-07 18:59:49
I don't think we should burn through the version numbers quite that quickly, as I don't assume anyone really mass-encoded with the daily builds ;-)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-07 20:34:20
Rarewares compile updated. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-07 20:58:09
Thanks!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: 145dBSPL on 2022-12-10 20:18:55
Thank you, experts, for further improving such a legacy MP3 encoder!

Over the past years i used mp3guessenc (https://mp3guessenc.sourceforge.io/) for analyzing MP3s. This tool also verifies the LAME tag in some way. This "Tag verification" check fails with the hmp3 encoder.
Could this also be relevant, referring to the prior compatibility discussions? I'm not a developer, so I'm not that experienced in reading source codes to find an answer myself.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-10 21:04:52
lametag 0.3.1 reads the tag fine, but 0.4.1 does not. I can't speak for mp3guessenc, but I suspect it depends on how specific the software is in needing lame specific information. It could even be that it fails because the version characters of the lametag are not 3.xxx, but I am only guessing. Without knowing the specific determing factors that mp3guessenc uses, further comment is irrelevant.

However, lame decodes hmp3 encoded files correctly, so I would believe that it's a software problem rather than a 'tagging' problem.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: AiZ on 2022-12-10 21:09:22
Hello,

I don't know if it has already been reported: tag part of tagged WAV files is encoded as if it was PCM.

    AiZ
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-10 21:24:42
Quote from: AiZ on
Hello,

I don't know if it has already been reported: tag part of tagged WAV files is encoded as if it was PCM.

    AiZ
I presume you are taking about id3 tagged wav files which may be so, but then wav files shouldn't be id3 tagged in the first place.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: AiZ on 2022-12-10 21:53:15
Hi John,

Quote from: john33 on
I presume you are taking about id3 tagged wav files which may be so, but then wav files shouldn't be id3 tagged in the first place.

You're right, problems come with id3 tags, RIFF tags are Ok.

    AiZ
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-11 10:13:12
Quote from: 145dBSPL on
Over the past years i used mp3guessenc (https://mp3guessenc.sourceforge.io/) for analyzing MP3s. This tool also verifies the LAME tag in some way. This "Tag verification" check fails with the hmp3 encoder.

Oh, that's perfectly expected. The info tag has fields for 16-bit checksums (tag checksum, MP3 checksum). Those fields are currently left "empty" (filled with zeros), so there's no checksum information to check against.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-11 10:13:44
I opened an issue regarding the checksums: https://github.com/maikmerten/hmp3/issues/2
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: 145dBSPL on 2022-12-11 20:00:47
Thank you for your explanations, John and maikmerten.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Aleron Ives on 2022-12-11 23:36:29
Maybe it's just me, but it seems like the correct tagging approach should be to write HELIX instead of LAME, and then have other programs like foobar2000 update to support the existence of multiple MP3 encoders, rather than writing "LAMEH" or similar by default to trick other programs into thinking LAME was used.

There ought to be a switch to enable this behaviour if you need your MP3s to work on some old software that will never get updated, but the fact that other software developers have required the string "LAME" in the tag seems to me like their problem, not yours, and it would be better if they updated their programs, rather than you being forced to use hacks by default in yours.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-12 08:53:41
fb2k plays and decodes correctly with a 'HELIX5.20' header, as do some other players, and lame decodes such a file correctly. The versions of lametag that I have fail to read the tag with the Helix header and the chances are that there are many pieces of software in use that will fail at some level where the chances of the necessary changes being implemented are negligable. So, although I agree with you philosophically, pragmatically it is probably better to stick with what it is currently.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-12 09:25:00
The problem really is that there is little development interest in MP3 these days. I don't expect a lot of playback software to get updated to read something like "HELIX5.20" properly (which I'd prefer, of course). This might lead to some user frustration with, e.g., gapless playback not working properly and updates needing months or years to tickle down to users.

Also note that "LAMEH5.20" includes all the information necessary to identify it as a Helix file - if there's a "H", it's Helix.

This is very much like LAME having to write "Xing" into the info header for players to recognize the Xing-VBR tag, despite not being Xing, obviously. We've now come full-circle, with LAME using "Xing" and Xing/Helix using a "LAME"-prefix.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-12-12 21:39:26
Quote from: maikmerten on
We've now come full-circle, with LAME using "Xing" and Xing/Helix using a "LAME"-prefix.
Mathie: "Two wrongs don't make a right!"
Engineer: "If it makes a workable ...?"
Patching up lossy codecs' old sins falls into the engineering category. Go ahead with it if it works.

There is an aside here: Back in the day when Xing was called Xing, did other encoders write headers such that (later?) tool would (mis-)identify them as "Xing" encodes? Anyone remembering them early days of the industrial revolution? :)


Reason why I am asking is that - although I don't remember twenty-five year old artifacts and that kind of thing, I remember that "Xing-identified" files could be quite crappy. But that impression may also have stuck out of all the things people did back then, like for example "merging MP3s by concatenating the files". Hey if there were no tags, it wouldn't be too bad, at least not when MP3s wouldn't be completely gapless anyway. And if there only was an ID3v1 ...
But if there were other "worse" (audio quality or header implementation!) codecs that would identify or be mis-identified as "Xing", and would later die out ... would explain something.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-13 04:26:34
Well, Xing had a VBR header SDK for implementation in other encoders: http://www.mp3-tech.org/programmer/sources/vbrheadersdk.zip

That one also includes writing "Xing" to the header frame, so I guess many tools of the era would mis-identify VBR encodings with that header as being "Xing".
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: 145dBSPL on 2022-12-13 08:51:30
One more thing - the code compiles smoothly and without errors using MinGW/GCC. Thumbs up.

The ASM code path is only used when compiling with Visual C++/VS - is that correct?
But never mind - the optimization via CFLAGS provides a pretty fast GCC binary as well.  ;D
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-13 09:10:16
Yeah, the ASM path should be Visual C++/VS only. And then only for 32 bit compiles.

With modern compilers and instruction sets, the C compiles may not fare any worse. I think John did some speed testing and the 64 bit C-compiles fared very well.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Aleron Ives on 2022-12-13 21:49:30
Quote from: maikmerten on
The problem really is that there is little development interest in MP3 these days.
Sadly, that seems to apply to all lossy audio these days. I guess it's reached "good enough" status, and nobody has any practical ideas on how to improve things beyond the capabilities of AAC/Opus.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: iwod on 2022-12-18 04:40:49
Damn it really wish they had Open Source their AAC encoder as well.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: itisljar on 2022-12-18 09:28:50
Is it any good, or do they use licenced Fraunhoffer?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-18 09:49:01
Yeah, the AAC encoder would have been very interesting, albeit I'm not sure it would be as mature compared to other AAC encoders as the Helix encoder is compared to other MP3 encoders. The Helix MP3 encoder has been around for a long time before being open-sourced. While I trust LAME more for (near-)transparent encodes (on grounds of being well-tested), at non-transparent, low bitrates (such as 96 kbps) it appears that Helix is pretty well tuned to keep the typical low-bitrate-MP3-artifacts somewhat less annoying.

For instance, check https://lame.sourceforge.io/download/samples/iron.wv (this is a sample file losslessly encoded with wavpack), decoded to iron.wav:

Code: [Select]
lame -q0 -b96 --lowpass 16000 --resample 44100 iron.wav lame.mp3

and compare to

Code: [Select]
hmp3 iron.wav hmp3.mp -b48 -F16000

(96 kbps stereo, lowpass 16000 Hz, 44100 Hz samplerate - well outside the usual comfort zone)

I'm aware that we do not listen with our eyes and that at HA, sound quality is not to be judged via spectograms, but this is an example where LAME has very audible problems with "bald patches"/"holes" in the spectogram, leading to the familiar "MP3 at too low a bitrate" artifacts, while Helix seems to be able to keep the frequency bands stocked much better, with obvious audible differences (looking nervously at TOS 8 ).

(I'd love to attach the encoded samples, but the copyright situation might not permit it. I'd love some freely available lossless test samples, e.g. CC-licensed)

I wonder if there's something for LAME to learn here.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-19 12:10:49
I now updated the encoder so that proper CRCs are included in the info header. The very, very useful tool "mp3guessenc" is now happy regarding the tag CRC and music CRC.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-19 14:11:36
Marvelous! :) Rarewares compile updated.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-19 14:28:53
Thanks!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KrajzegaX on 2022-12-19 17:50:16
I noticed one thing about the encoder version. In Mediainfo with CBR it shows garbage at the end of the version, where with VBR it is correct. Looking at the hex editor there is a difference of one byte.
CBR:
Code: [Select]
0000:0080 |                                           4C 41 4D |              LAM
0000:0090 | 45 48 35 2E  32 31 01 F0                           | EH5.21.ð       
VBR:
Code: [Select]
0000:0080 |                                           4C 41 4D |              LAM
0000:0090 | 45 48 35 2E  32 31 00 F0                           | EH5.21.ð       
A workaround for this is to change byte from 01 to 00 and after this operation correctly appears without strange characters. I don't know if this can be corrected in code, @maikmerten?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-20 10:38:27
Quote from: KrajzegaX on
I noticed one thing about the encoder version. In Mediainfo with CBR it shows garbage at the end of the version, where with VBR it is correct. Looking at the hex editor there is a difference of one byte.
CBR:
Code: [Select]
0000:0080 |                                           4C 41 4D |              LAM
0000:0090 | 45 48 35 2E  32 31 01 F0                           | EH5.21.ð       
VBR:
Code: [Select]
0000:0080 |                                           4C 41 4D |              LAM
0000:0090 | 45 48 35 2E  32 31 00 F0                           | EH5.21.ð       
A workaround for this is to change byte from 01 to 00 and after this operation correctly appears without strange characters. I don't know if this can be corrected in code, @maikmerten?

Hi, thanks for reporting, but to me it looks like that's a bug in Mediainfo.

The short version string in the info tag is 9 characters long. In C, strings are "zero terminated", meaning the end of the string is denoted by a zero value after the last character, to avoid having to store a string length somewhere else. The short version string, however, is not zero-terminated, it has a fixed length of 9 bytes, and directly following the short version string is the "Info Tag revision + VBR method" byte: http://gabriel.mp3-tech.org/mp3infotag.html#rev-method

For VBR, I write 0x00, to denote "rev.0 info tag, and unknown VRB type" ("unknown" because whatever Helix is doing with VBR, it does not match any specific LAME VBR method). This by chance happens to zero-terminate the short version string.

For CBR, I write 0x01, to denote "rev.0 info tag, and constant bitrate", which doesn't zero-terminate the short version string by chance anymore.

Does this lead to any further trouble, other than Mediainfo showing trailing garbage? I feel that hmp3 is doing the right thing here, but if it happens to format hard drives and set buildings ablaze, it might be better to compromise here.

(This, btw, demonstrates why standards need several implementations and clear documentation to achieve some robustness.)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-12-20 11:13:39
MediaInfo is open source, if anyone feels like scrutinizing it.  https://mediaarea.net/en/MediaInfo/Download/Source
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: VEG on 2022-12-20 12:14:59
Maybe it is worth just to use a shorter string like LAMEH5.2 or LAMEH521 for better compatibility. I guess some other programs may also expect a null terminator there since it is usually there in existing files.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KrajzegaX on 2022-12-20 17:25:25
I've been fiddling with the encoder name and confirm that it's a bug in MediaInfo. When I changed from "LAMEH5.21" to "LAME3.100" characters at the end disappear and "Encoder settings" appears underneath.
Quote from: VEG on
Maybe it is worth just to use a shorter string like LAMEH5.2 or LAMEH521 for better compatibility. I guess some other programs may also expect a null terminator there since it is usually there in existing files.
I tested these shortened encoder names and characters at the end do not appear. This is a good idea to solve this problem.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-12-20 18:06:44
Seems that MediaInfo expects  LAME<singlechar><dot><something>

If there is a risk that other applications behave as MediaInfo, then the following strings could be considered:
LAMEH.520
LAMEH.5.2
LAMEH.52b
... the latter could be used for beta versions. But to flag that it is a beta, you could also replace "H" by "h" - at least as far as MediaInfo goes - so that 5.2 beta is LAMEh.5.2 and a "proper" release is LAMEH.5.2
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-20 19:21:57
The problem with having a shorter version string and then using the 9-th character to zero-terminate the 8-char string is that applications that actually work properly (not relying on zero-termination) might misbehave, as "0" is not a displayable ASCII character.

So, avenues to "fix" things: a) fix Mediainfo. If nobody steps in, I might file a bug report or b) always write 0x00 behind the short version string, also for CBR files, as initially suggested. I attached a representation of my currently preferred approach.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2022-12-20 20:31:05
"9 characters" isn't enough for MediaInfo:
LAME3.100 is OK
LAME3.99r is OK
LAME03.99 is not, despite being 9 characters.
LAME3.99 is not OK either, so it does require the 9th character ... did ancient LAME write NUL or something else?

So I think it needs the dot in the right place. If it is only MediaInfo, then ask them to fix it, but ...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KrajzegaX on 2022-12-20 20:52:25
The easiest way I would choose option b, because decoder recognizes at Xing and Info header what kind of bitrate mode it is anyway.
This is my opinion. :D
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: VEG on 2022-12-20 23:05:34
This is how it is implemented there:
https://github.com/MediaArea/MediaInfoLib/blob/master/Source/MediaInfo/Audio/File_Mpega.cpp#L1432
And the most recent fix that was added because of LAME 3.100:
https://github.com/MediaArea/MediaInfoLib/commit/b9dc0b8de3a228fceadefd34744d2a073ecbb3ff
So it expects "LAME3.90" and newer, and the dot is expected at the fixed position. As an alternative, it supports "L3.99" prefix also.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2022-12-21 08:24:05
Clearly the problem lies in MediaInfo imposing constraints upon the content of the LameTag that are not present in its specification. While the LAME developers anticipated other software developers imposing such restrictions, I don't believe that they expected such tight constraints. It should be filed as a MediaInfo bug.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-21 10:25:46
Yeah, that parsing code is very specific - that's not very flexible. Thanks for finding the specific code locations!

I'll file a bug for MediaInfo and perhaps some insights will be generated how to handle things in a general matter.

edit: MediaInfo issue opened over at https://github.com/MediaArea/MediaInfoLib/issues/1637
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2022-12-21 16:45:58
Okay, MediaInfo now is on track to receive support for hmp3: https://github.com/MediaArea/MediaInfoLib/pull/1638
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: capma on 2022-12-23 15:24:55
MediaInfo change log:

Version 22.12, 2022-12-22
-------------
...
x I1637, MPEG-Audio: proper support of Helix MP3 encoder detection and encoder settings
...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KrajzegaX on 2022-12-23 15:29:56
Quote from: maikmerten on
Okay, MediaInfo now is on track to receive support for hmp3: https://github.com/MediaArea/MediaInfoLib/pull/1638
Fix for MediaInfo works. Thank you @maikmerten for all you do for Helix.  :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Eurobeat_fan on 2023-01-03 07:41:39
Hey guys, could you please help me solve my confusion? :)

There's a setting called nsbstereo - "Applies to mode-1 stereo mode only. Number of subbands to encode in independent stereo.  Valid values are 4, 8, 12, and 16.". What does it mean? I thought you can switch the whole block to mid-side or true stereo but you can't switch specific bands to ms/ss in MP3. Was I wrong?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2023-01-03 11:18:41
Hi, Helix, unlike LAME, supports intensity stereo. As far as I can tell, with nsbstereo one selects how many bands are coded with intensity stereo, with the rest being coded with mid-side-stereo.

This, however, only works for a) CBR files and b) means that no short blocks will be used. The Helix encoder has two separate bit allocation engines. Intensity stereo is only supported with the old engine (CBR only, no short blocks) apparently directly inherited from Xing. The new bitrate allocation engine supports VBR and short blocks, but has no support for intensity stereo.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: rutra80 on 2023-01-03 12:57:28
Would it be possible for Helix to switch automatically between mono/stereo encoding? Currently it throws an error when I want to encode a mono file and won't change the commandline.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2023-01-03 18:52:30
Quote from: rutra80 on
Would it be possible for Helix to switch automatically between mono/stereo encoding? Currently it throws an error when I want to encode a mono file and won't change the commandline.

Hi, can you provide a command-line example that fails for mono files? Usually Helix should auto-detect mono/stereo...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: rutra80 on 2023-01-03 21:05:10
In fb2k I use -V35 - %d
I need to add -M3 for mono files...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2023-01-04 12:41:42
Quote from: rutra80 on
In fb2k I use -V35 - %d
I need to add -M3 for mono files...

Can you try passing an intermediate WAV file to hmp3 (instead of using stdin)? I've seen funky things happen with input from stdin on Windows with hmp3. Under normal circumstances, the -M switch should never be needed.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: rutra80 on 2023-01-04 23:37:51
Apparently the problem is bit-depth higher than 16bit - mono files I tried were lossy or sequenced (internally 32bit float in fb2k), while the stereo ones that worked were 16bit lossless. Setting highest bps mode supported to 16bit in fb2k makes it work good even with pipe.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: whepper on 2023-02-14 13:24:18
I compiled hmp3 successfully on MacOS on a M1 (arm64). It encodes in a flash. Amazing encoder with similar quality.

What are you favourite parameters? Is -V120 (or -V150) -X2 -SBT450 -TX0 -HF2 still recommended? Anyone ABXed some settings? Thanks!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Eurobeat_fan on 2023-05-01 14:44:10
Hmmm Helix seems to struggle with higher bitrates as far as I can see. It just doesn't know how to utilize more bitrate. Not a good choice for 320kbps CBR.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Nani786 on 2023-05-04 18:56:54
Quote from: Eurobeat_fan on
Hmmm Helix seems to struggle with higher bitrates as far as I can see. It just doesn't know how to utilize more bitrate. Not a good choice for 320kbps CBR.

So would you favor LAME over Helix?.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Nani786 on 2023-06-08 21:26:28
Quote from: Eurobeat_fan on
Hmmm Helix seems to struggle with higher bitrates as far as I can see. It just doesn't know how to utilize more bitrate. Not a good choice for 320kbps CBR.

Not to stress TOS#8 but I never had any Issues with V120 ~ V150 with Noise/electronic/extreme metal. Feels like some here are trying to justify LAME when Helix MP3 does a lot better on way more samples mainly pre-echo rich ones. I gave up using AAC at 256kbps VBR when my Sony DAP's AAC decoder will ignore flies that reach 480 ~ 510kbps???.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2023-06-09 07:43:42
Quote from: Nani786 on
Helix MP3 does a lot better on way more samples mainly pre-echo rich ones.

I certainly have a soft spot for the Helix MP3 encoder (no surprises there, I guess), but I feel that this is a statement that deserves supporting data, just like Eurobeat_fan's "Not a good choice for 320kbps CBR" (at such bitrates I expect "usually transparent" results). Gathering data is good - learning what the encoders handle differently might highlight venues for improvement.

Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: The Irish Man on 2023-06-09 08:26:24
Quote from: Nani786 on
Not to stress TOS#8 but I never had any Issues with V120 ~ V150 with Noise/electronic/extreme metal. Feels like some here are trying to justify LAME when Helix MP3 does a lot better on way more samples mainly pre-echo rich ones.

Sorry you just can't say stuff like that & walk away, without some ABX testing, or bare minimum some sample tracks.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Eurobeat_fan on 2023-06-09 14:07:51
Quote from: maikmerten on
just like Eurobeat_fan's "Not a good choice for 320kbps CBR" (at such bitrates I expect "usually transparent" results)
What I meant about calling it an inferior choice is that it doesn't really utilize the bitrate it has in CBR 320 kbps. It continues to show its typical differences from LAME (sticking to mid-side stereo instead of true stereo most of the time, using sfb21 less often), and it leads to the fact that it almost never uses bit reservoir to its full potential. I have only seen Helix using 150 bytes out of 511 for any block possible but never the whole 511 byte, not for a single transient. This means that at 320 kbps we just waste bits for nothing, the bitrate is excessive to the psy-model that Helix utilizes and it can do just fine with, let's say, 224 kbps CBR. It may be not a problem in general but I'm pretty sure that Helix will fail on more samples than LAME does at 320 kbps simply because LAME has more ideas about what information to save into a 320 kbps file.

I don't have any complains about VBR mode in that regard (obviously).
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2023-06-09 14:25:55
Ah, thanks for elaborating, that's very useful context.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Nani786 on 2023-06-11 08:01:32
Quote from: Eurobeat_fan on
Quote from: maikmerten on
just like Eurobeat_fan's "Not a good choice for 320kbps CBR" (at such bitrates I expect "usually transparent" results)
What I meant about calling it an inferior choice is that it doesn't really utilize the bitrate it has in CBR 320 kbps. It continues to show its typical differences from LAME (sticking to mid-side stereo instead of true stereo most of the time, using sfb21 less often), and it leads to the fact that it almost never uses bit reservoir to its full potential. I have only seen Helix using 150 bytes out of 511 for any block possible but never the whole 511 byte, not for a single transient. This means that at 320 kbps we just waste bits for nothing, the bitrate is excessive to the psy-model that Helix utilizes and it can do just fine with, let's say, 224 kbps CBR. It may be not a problem in general but I'm pretty sure that Helix will fail on more samples than LAME does at 320 kbps simply because LAME has more ideas about what information to save into a 320 kbps file.

I don't have any complains about VBR mode in that regard (obviously).

Sounds like the Helix Dev's so no reason to care about CBR since 98% of devices can handle VBR MP3.

Quote from: The Irish Man on
Quote from: Nani786 on
Not to stress TOS#8 but I never had any Issues with V120 ~ V150 with Noise/electronic/extreme metal. Feels like some here are trying to justify LAME when Helix MP3 does a lot better on way more samples mainly pre-echo rich ones.

Sorry you just can't say stuff like that & walk away, without some ABX testing, or bare minimum some sample tracks.


I'm Sorry does the Listening Tests sub-forum become Invisible when there public data showing Helix doing better than LAME like Kamedo2's at 192kbps?. I've never seen a public DBT with LibVorbis 1.3.7 vs aoTuV 6.03 yet the libvorbis encoder is the default encoder for Foobar2000.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: shadowking on 2023-06-11 13:55:57
Quote from: maikmerten on
Yeah, the AAC encoder would have been very interesting, albeit I'm not sure it would be as mature compared to other AAC encoders as the Helix encoder is compared to other MP3 encoders. The Helix MP3 encoder has been around for a long time before being open-sourced. While I trust LAME more for (near-)transparent encodes (on grounds of being well-tested), at non-transparent, low bitrates (such as 96 kbps) it appears that Helix is pretty well tuned to keep the typical low-bitrate-MP3-artifacts somewhat less annoying.

For instance, check https://lame.sourceforge.io/download/samples/iron.wv (this is a sample file losslessly encoded with wavpack), decoded to iron.wav:

Code: [Select]
lame -q0 -b96 --lowpass 16000 --resample 44100 iron.wav lame.mp3

and compare to

Code: [Select]
hmp3 iron.wav hmp3.mp -b48 -F16000

(96 kbps stereo, lowpass 16000 Hz, 44100 Hz samplerate - well outside the usual comfort zone)

I'm aware that we do not listen with our eyes and that at HA, sound quality is not to be judged via spectograms, but this is an example where LAME has very audible problems with "bald patches"/"holes" in the spectogram, leading to the familiar "MP3 at too low a bitrate" artifacts, while Helix seems to be able to keep the frequency bands stocked much better, with obvious audible differences (looking nervously at TOS 8 ).

(I'd love to attach the encoded samples, but the copyright situation might not permit it. I'd love some freely available lossless test samples, e.g. CC-licensed)

I wonder if there's something for LAME to learn here.

Lame adds a ringing artifact in CBR / ABR mode + lower bitrate; perhaps up to 128 or 160k. 
Its easy to isolate and usually adding -q level < 3 can yield a significant improvement.  I use -q5 , perhaps
even -q7 for 128 or lower. For VBR it doesn't seem to be as effective.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: The Irish Man on 2023-06-11 19:23:12
Quote from: Nani786 on


I'm Sorry does the Listening Tests sub-forum become Invisible when there public data showing Helix doing better than LAME like Kamedo2's at 192kbps?. I've never seen a public DBT with LibVorbis 1.3.7 vs aoTuV 6.03 yet the libvorbis encoder is the default encoder for Foobar2000.

You did not say your info, was from a listening test. even so if it's this one (https://hydrogenaud.io/index.php/topic,113324.0.html) it does not show "Helix MP3 does a lot better" as you say, so maybe you can post some of your own ABX tests.
You never know, you may be the NEW Kamedo2.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Eurobeat_fan on 2023-06-14 11:02:59
Quote from: Nani786 on
Sounds like the Helix Dev's so no reason to care about CBR since 98% of devices can handle VBR MP3.
No, they just didn't do extra tuning for high bitrates. Their psy model does incredible job at bitrates like 128-192 kbps but it doesn't use any strategy for scaling up aside from activating sfb21 (iirc HF2 switch does not work at bitrates lower than 192 kbps). This is actually a good approach - I even see parallels with Opus here (hell, even VBR decisions are pretty similar to what Opus does), the only problem is that MP3 as a codec is too bad for this approach, so to squeeze more potential out of MP3 at high bitrates like 320 kbps you need more complex solutions (and that's exactly what LAME does - more true stereo blocks, densier sfb21 frequencies, heavier bit reservoir usage).
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: vina54 on 2023-08-06 04:38:43
I noticed that this encoder is very efficient when it comes to space and sound quality. I think it encodes MP3s in a very similar way to what the "-Y" switch does on LAME and if you are using the "V80-150" setting, then it would apply a hard lowpass filter at 20khz (kinda like what FDK/OPUS does). I think the reason why is because it forces the encoder to focus more on the lower frequencies (below 16khz) and not waste bits on the higher non-audible range
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2023-11-11 16:11:09
Comrades, could we draw an intermediate conclusion by agreeing that Helix is ​​definitely a good choice, for example, when encoding audio books and radio plays with low bitrate, when non-MP3 codecs for some reason (e.g. old hardware compatibility) cannot be used? If so, what encoding parameters do you consider appropriate in this case?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: vina54 on 2023-11-11 18:58:34
I think the original poster on here recommended the parameter -F16000 -B48 -N8 for low bitrate/radio streaming application and for streaming quality music is -V75 -X2 -SBT450 -TX0 -HF2 -F16000
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2023-11-12 10:55:12
For low bitrate audiobooks (with occasional music in between), I guess -F16000 -B48 -N8 might indeed work as a starting point if you need CBR.

The latter command (-V75 -X2 -SBT450 -TX0 -HF2 -F16000) is a bit strange. Why enable high-frequency encoding (> 16 kHz), but put a 16 kHz lowpass there? Also not quite sure tweaking the short block threshold has clear benefits.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: vina54 on 2023-11-12 12:48:29
Hmmm well I guess discard the -SBT450 one.  :D
I think I saw in the manual that you can use -HF-1 (that's I would use) and let the encode decide the high frequencies or not. Otherwise just disregard that as well. Therefore it'll probably be:

-V75 -X2 -TX0 -HF-1   ;)

If one wants some high frequencies then
-V80 -X2 -TX0 -HF-1 -F18000
would work as well


Title: Re: Helix MP3 Encoder v5.1.1
Post by: Nani786 on 2024-02-05 22:53:43
This is the encoder I use to prove that MP3 can still hold up.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-02-25 14:22:27
Aggrrhh, I've come across this limitation. Helix wants 16 bit.

Code: [Select]
$ hmp3.exe in.wav out.mp3 -V150 -HF2 -U2

 hmp3 MPEG Layer III audio encoder 5.2.1, 2022-12-19
 Utilizing the Helix MP3 encoder, Copyright 1995-2005 RealNetworks, Inc.

 PCM input file: in.wav
 channels = 2  bits = 24,  rate = 48000  type = 1
 UNSUPPORTED PCM FILE TYPE
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: vina54 on 2024-02-25 15:34:49
Ah yes! One of the biggest limitations of this MP3 encoder is that it's only 16-bits only. No 32-bit floating unlike LAME and other modern encoders
Title: Re: Re: Helix MP3 Encoder v5.1.1
Post by: Kraeved on 2024-03-12 09:07:49
@KrajzegaX, have you examined the code of this Helix encoder?
Perhaps you could lift the veil of secrecy over -SBT450 -TX0 settings then.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-13 15:40:58
Quote from: Case on
I don't think Sector Boundary Error is correct term, that is related to incorrect writing of audio data to a CD.
This MP3 encoder just fails at being gapless. The way audio is sent to the encoder is irrelevant.

Quote from: Apesbrain on
Quote from: Kraeved on
@ApesbrainAlso, do you encounter such errors when encoding via pipe or when calling directly too (hmp3 in.wav out.mp3)?
Encoding via pipe. I guess it doesn't matter that SBEs are created. I was just curious since LAME did not exhibit this behavior. TBH, I'd need to check that again.

EDIT: Confirming LAME (32bits version 3.100) does not create SBEs.

I should learn to shut up sometimes. I just tested the Helix encoder and the latest version from Rarewares does write proper gapless info to the headers. But it seems to have some limitations or bugs as it doesn't do it when the data is fed through a pipe. Not even when the piped data is a full WAV with perfectly valid length fields.

Detecting this type of error is super simple. Just look at file length as samples and you'll see the original and the encode have different lengths. Such files can never be gapless. Easy workaround for this is to use temp file in foobar2000 converter. Replace the single dash '-' with '%s'.
And how this can relate to Audio CD sector boundaries, audio CDs require track lengths to be multiples of 588 samples. If a track length doesn't match that the burning software needs to pad it with silence. The files aren't gapless outside CD but when burned to a CD as-they-are, the tracks would get extra gaps between them.




MOD edit: added quotes before splitting post to a different topic
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-13 19:16:42
Quote from: Case on
I should learn to shut up sometimes. I just tested the Helix encoder and the latest version from Rarewares does write proper gapless info to the headers. But it seems to have some limitations or bugs as it doesn't do it when the data is fed through a pipe.

That's why I asked @Apesbrain exactly how he encodes. Yesterday I was looking for the optimal way to compress an audiobook and came across that Helix-encoded files sound different when switching from one to another if we abandon the pipe (-) in favor of temporary files (%s). I was inspired to try that by @KrajzegaX comment (https://hydrogenaud.io/index.php/topic,122473.msg1041009.html), which dates back to version 5.1.1. But I am afraid your Fake Gapless DSP (https://hydrogenaud.io/index.php/topic,115009.msg1040572.html#msg1040572) is still required.

Quote
Detecting this type of error is super simple. Just look at file length as samples and you'll see the original and the encode have different lengths. Such files can never be gapless.

Hmm.
FLAC 01: 0:10.117 (485 603 samples), Musepack 01: 0:10.117 (446 148 samples)
FLAC 02: 0:05.447 (261 445 samples), Musepack 02: 0:05.447 (240 203 samples)
Yet gapless playback without DSPs. Perhaps you meant “never” applicable to MP3 format.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-13 19:51:30
Quote from: john33 on
Quote from: Kraeved on
Quote from: Apesbrain on
Quote from: LedHed8 on
Lack of gapless encoding is the only drawback I know of in the Helix encoder.
I also noticed that Helix introduces sector boundary errors; LAME does not.

RareWares hosts a newer version of Helix than you used back in 2017, namely 5.2.1 2022-12-19.
@Kamedo2 @LedHed8 @Apesbrain Could you please check if there has been any improvement since you commented?


To the best of my recollection @maikmerten's work was about making both the encoded and decoded files more standards compliant and had nothing to do with tuning of the encoder. I stand to be corrected, but I believe that is the case.

That is indeed correct. I only added compile support for modern Linux and headers for (hopefully) gapless playback. No work on audio quality.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-13 20:22:57
Quote from: Kraeved on
Quote from: Case on
Detecting this type of error is super simple. Just look at file length as samples and you'll see the original and the encode have different lengths. Such files can never be gapless.

Hmm.
FLAC 01: 0:10.117 (485 603 samples), Musepack 01: 0:10.117 (446 148 samples)
FLAC 02: 0:05.447 (261 445 samples), Musepack 02: 0:05.447 (240 203 samples)
Yet gapless playback without DSPs. Perhaps you meant “never” applicable to MP3 format.
Ok I should have been more specific with my wording. Since this was about "Sector Boundary Error", thus about audio CDs, I kind of only thought about 44.1 kHz material. The sample count matching rule only applies when no resampling is involved.

Your example files show 48 kHz FLACs and 44.1 kHz Musepacks. Resampling has a possibility to introduce gapless issues of its own, though there are now nice resamplers for foobar2000 that extrapolate the signal to prevent that problem.

This is getting way off topic, but those gapless files from your example would need some adjustments to burn on audio CD. The FLACs of course are in wrong sample rate and would need to be resampled to 44100 Hz. And since the lengths don't align to sector boundaries, the track endings and beginnings would need to be shifted so that they align on sector boundaries. I believe foobar2000 Audio CD Writer component does this automatically for you.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-14 15:27:30
Here's a patch to hmp3 encoder (sources from Rarewares) to fix the pipe encoding bug. I also fixed one undefined behavior bug and initialized some variables so they aren't used with random data - these issues caused the encoder to freeze at least with newer compilers.
Unfortunately the asm code didn't want to compile, perhaps it needs an older MASM version, so I can't post an asm-enabled binary.
But I added a basic MSVC compile so any interested party can give it a try.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-14 15:45:14
@Case, thank you for trying to improve Helix MP3 encoder.
I believe the ASM part is crucial for speed, we need a hero to integrate it.

Code: [Select]
$ lptimer105.exe hmp3.exe storm.wav -V96

 PCM input file: storm.wav
MPEG ouput file: TEST.MP3
 pcm file:  channels = 2  bits = 16,  rate = 48000  type = 0
 Layer III   mode 1 STEREO   48000Hz  VBR-96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553984 |   100%   | 120.52 / 129.81  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455782%

Kernel Time  =     0.109 = 00:00:00.109 =    5%
User Time    =     1.341 = 00:00:01.341 =   71%
Process Time =     1.450 = 00:00:01.450 =   77%
Global Time  =     1.873 = 00:00:01.873 =  100%

$ lptimer105.exe hmp3case20240314a.exe storm.wav -V96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553984 |   100%   | 120.57 / 129.81  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455782%

Kernel Time  =     0.093 = 00:00:00.093 =    3%
User Time    =     1.965 = 00:00:01.965 =   75%
Process Time =     2.059 = 00:00:02.059 =   78%
Global Time  =     2.608 = 00:00:02.608 =  100%
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-14 16:16:51
And here you are: https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240314-Case.zip (https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240314-Case.zip)

Visual Studio 2015 with MASM. :)

Edit: If we can confirm all is well (it works fine on the quick test that I did), I'll post it to Rarewares officially. And, I guess, @maikmerten would then probably like to update his git repository as well. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-14 16:48:00
Thank you. The encoder seems to be just as fast as previous build from you and in my testing I get bit-identical output from piped encoding in foobar and from the same source wav using old binary on command line. But you'll probably want a confirmation from an unbiased third party.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-14 20:03:06
I see the ASM part is back, as is the speed.
Kudos to @john33, there's plenty of pep in you yet.

Code: [Select]
 $ lptimer105.exe hmp3case20240314b.exe storm.wav -V96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553984 |   100%   | 120.52 / 129.81  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455782%

Kernel Time  =     0.093 = 00:00:00.093 =    4%
User Time    =     1.419 = 00:00:01.419 =   75%
Process Time =     1.513 = 00:00:01.513 =   80%
Global Time  =     1.888 = 00:00:01.888 =  100%

Now let me invoke hmp3case20240314b.exe in three ways:
a) via direct call in the command-line shell
b) via Foobar2000 with a temporary file (%s)
c) via Foobar2000 with a pipe (-)

$ for %# in (*.mp3) do @echo %~z# = %#
2553984 = storm-direct.mp3
2553984 = storm-pipe.mp3
2553984 = storm-tempfile.mp3

$ b3sum *.mp3
424f76a649f35de484c8980a50fd13675dd17a702f0eaa60bed8ba5e5b48bc93  storm-direct.mp3
fe0dd60dcf3e5341631850f4b6a3a2444b5d34a9b19fda1022c287598366f5ec  storm-tempfile.mp3
fe0dd60dcf3e5341631850f4b6a3a2444b5d34a9b19fda1022c287598366f5ec  storm-pipe.mp3

What is the difference between direct and the other two?
Since MediaInfo and ExifTool reported nothing special, let's dig a bit deeper.

$ for %# in (*.mp3) do mp3packer -i %# | rg -oA999 INFO: > %~n#.txt
$ fc /N storm-direct.txt storm-pipe.txt

***** storm-direct.txt
    9:   18516192 bits of payload data (117.625858 kbps)
   10:   2317405 bytes of payload data (117.772272 kbps)
   11:   23048 bits wasted from partially-full bytes (0.146415 kbps)
   12:   2553529 bytes of MP3 data (129.772272 kbps) = minimum bitrate possible
   13:   455 bytes of padding (0.023123 kbps)
***** storm-pipe.txt
    9:   18516087 bits of payload data (117.625191 kbps)
   10:   2317392 bytes of payload data (117.771612 kbps)
   11:   23049 bits wasted from partially-full bytes (0.146421 kbps)
   12:   2553516 bytes of MP3 data (129.771612 kbps) = minimum bitrate possible
   13:   468 bytes of padding (0.023784 kbps)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-15 06:36:32
I assume there is something special about the WAV file that causes the difference. When you encode with foobar2000, the encoder doesn't see the original wav but a new fresh temporary WAV created only from the audio bits. If you shared the source track I could take a look at some point.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-15 09:01:34
@Case, I hope you'll understand why the source storm.wv (https://reactfiles.com/en/J5PSHH3mZmwXofB/file)¹ is compressed with WavPack. My trust in FLAC has been shaken recently due to its inability to reliably preserve non-audio data (https://hydrogenaud.io/index.php/topic,123176.msg1040976.html#msg1040976).

By the way, since you took a peek at the Helix code, is it difficult to raise the bit depth threshold of incoming data from the current 16-bit to 32-bit floating-point, as LAME team did?

¹ SHA256: 6516c87b4d81851b59b5d0ca54944f135bf04a2b3014a58862a8fb815588a6ad
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-15 09:37:45
The hmp3 frontend handles the wav incorrectly. The encoded file has gained 14 extra samples that should not be there. I haven't yet looked at why, busy with actual work.

Just to clarify, that issue has nothing to do with my pipe encoding fix. The original encoder produces bit-identical output when not using pipes.

My intention was to see if the frontend is worth improving by making it at least support basic input bit depths (24 and 32-bit float) and fix file accesses to use 64-bit interfaces. Now it can't handle source files over 2 GB correctly. But that is something potential stuff for the future, if this thing has any users.

PS. I can decode WavPack, thanks. Though as I'm not a musician I have zero need to keep non-audio information in audio files, other than tags.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-15 15:04:34
I think I must be missing something here.

I have decoded 'storm.wv' to wav and encoded it, on the command line, using hmp3 at default VBR;
I have encoded the 'wv' file, in foobar, using the same default, above;
I have done the same with the wav file;
I have decoded the file encoded from the .wv file and ALL have exactly the same duration and number of samples!

What am I missing?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-15 15:37:01
Quote from: john33 on
I have decoded 'storm.wv' to wav and encoded it, on the command line, using hmp3 at default VBR;
Did you decode it with foobar2000? If you did, it should have created a lovely fresh WAV without the issue. Use wvunpack.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-15 15:43:11
Quote from: Case on
Quote from: john33 on
I have decoded 'storm.wv' to wav and encoded it, on the command line, using hmp3 at default VBR;
Did you decode it with foobar2000? If you did, it should have created a lovely fresh WAV without the issue. Use wvunpack.
Yes, I did. Obviously what I was missing! I'll try again. ;)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-15 16:54:31
Examined the situation. The file has 56 bytes of extra at the end which the encoder happily took in. I changed the encoder to only encode as much data as data chunk's size indicates.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-15 17:01:20
I just looked at the files in a hex editor and came to the same conclusion, which is great! Have you amended the patch file linked to above?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-15 17:44:51
Here's an updated patch to the original Rarewares sources that fixes the pipe encoding + unitialized variables + prevents encoding beyond indicated data section size.

And it looks like support for higher bitdepths could be added. It appears this encoder converts the data to 32-bit floating point for processing anyway.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-15 20:38:26
Updated compile: https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240315-Case.zip (https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240315-Case.zip) :)

Many thanks to Case for his efforts.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-15 22:05:58
Code: [Select]
$ hmp3case20240315a.exe storm.wav storm-direct.mp3 -V96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553984 |   100%   | 120.52 / 129.81  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455810%

$ b3sum *.mp3
d92f158207015b10480f35f6c08a8e2dff550a5c6adc320588f1880687a534a1  storm-direct.mp3
dbbd7bec1b2275315e0e7222f09724e09523bd5f9f40141c2dac4f0050d85380  storm-pipe.mp3
dbbd7bec1b2275315e0e7222f09724e09523bd5f9f40141c2dac4f0050d85380  storm-tempfile.mp3

Sorry, but storm-direct.mp3 checksum is still different on my end.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-15 22:24:06
So it seems. Some patches are not in. Here's my compile, I dug up old version of Microsoft's assembler.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-15 22:55:59
Code: [Select]
$ hmp3case20240316a.exe storm.wav storm-direct.mp3 -V96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553936 |   100%   | 120.44 / 129.80  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455651%

$ b3sum *.mp3
3090e261c6eb89ab813029e9f0756b01d7fabfb50aba513dc56fcb56d1199aba  storm-direct.mp3
3090e261c6eb89ab813029e9f0756b01d7fabfb50aba513dc56fcb56d1199aba  storm-pipe.mp3
3090e261c6eb89ab813029e9f0756b01d7fabfb50aba513dc56fcb56d1199aba  storm-tempfile.mp3

Well done, @Case.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-16 09:41:49
OK, sorry about the earlier hiccup, but this one works; https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240316-Case.zip (https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240316-Case.zip)

I've incremented the date attached to avoid confusion and removed the earlier compile.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-16 18:22:36
I added support for handling WAV files in WAVE_FORMAT_EXTENSIBLE format, 24-bit and 32-bit input support. 32-bit handles both integer and float. And I added unicode file name support. I performed basic tests, like confirmed that 16-bit input produces bit-perfect output with standard version. The tool can now encode @Kraeved 's dolle.wav from command line.

I'll upload source patch soon if no bugs are found.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-17 01:52:27
Progress and bitrate counters have gone crazy: -6116.64, Kbpsss, 101%.

Sample: dolle.wav (https://hydrogenaud.io/index.php/topic,125651.msg1041278.html)

Code: [Select]
$ hmp3 dolle.wav -V96
 pcm file:  channels = 1  bits = 24,  rate = 44100  type = 1
 Layer III   MONO   44100Hz  VBR-96

-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
     883  |    3048192 /     233881 |   100%   | -6116.64 /  81.15  Kbpsss
-------------------------------------------------------------------------------
 Compress Ratio 7.702456%

Sample: bestia.wav

Code: [Select]
$ hmp3 bestia.wav -V96
 pcm file:  channels = 2  bits = 24,  rate = 44100  type = 1
 Layer III   mode 1 STEREO   44100Hz  VBR-96

-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
     388  |    2674944 /     184623 |   101%   | -2496876.96 / 145.94  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 6.973421%
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-17 11:37:49
Thanks for the bug report, I had not seen those.
Fixed the problems @Kraeved found. And since I needed test material to see if new bitdepths work on big-endian I had to add support for big-endian RIFX WAVE files.

Attached my compile + patch.
To summarize, the patch includes the following changes:
* fixed pipe encoding missing the beginning of the audio and breaking gaplessness.
* fixed the encoder to not read data beyond data chunk.
* added support for 24 bit integer and 32 bit integer and floating point input data.
* added Unicode filename support for input and output files.
* added support for WAVE_FORMAT_EXTENSIBLE file format and RIFX big endian WAVE files.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-17 16:48:29
Updated .exe and .exe with modifed source:

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case.zip (https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case.zip)

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip (https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip)

Assuming these are 'trouble-free' over the next few days, I will then post on Rarewares.  Huge thanks to @Case for a lot of hard work. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-18 17:01:13
Issues

a) Sample: bee.wav (https://helpguide.sony.net/high-res/sample1/v1/en/index.html) (96 kHz 24 bit)
* Old Helix version says “UNSUPPORTED PCM FILE TYPE”, current version says “ENCODER INIT FAIL”.
* Let's make the error message more descriptive, e.g. “ERROR. Supported sampling rates are a, b, c, d.” or a - d.
* And if the input cannot be processed, Helix should not create a zero-length output file (e.g. TEST.MP3 0 bytes).

b) Samples: 24_bit_fixed.wav,  32_bit_float.wav (https://www.sounddevices.com/sample-32-bit-float-and-24-bit-fixed-wav-files/) (48 kHz 24 bit, 48 kHz 32 bit float, they are loud)
* Error: “UNRECOGNIZED PCM FILE TYPE”, but LAME encodes them fine.

c) Samples: leftright.wav, mcgill.wav (11 kHz 16 bit, 8 kHz 32 bit integer, attached below)
* Helix unexpectedly changes the sampling rate to 22.05 kHz and 16 kHz respectively.
* It is an inherited case, but can you tell, is it a bug or a feature?

d) Filename to verify Unicode support: birds-vögel-птицы-pájaros.wav (English, German, Russian, Spanish)
* Helix successfully creates birds-vögel-птицы-pájaros.mp3 on the disk, but its name is garbled on the output screen.
“PCM input file” and “MPEG output file” fields say:
birds-vЎgel-?????-pсjaros.wav (chcp 866)
birds-vцgel-?????-pбjaros.wav (chcp 1251)
birds-v�gel-?????-p�jaros.wav (chcp 65001)
It's a minor issue, more aesthetic one than practical, but I report it nonetheless.

Questions

* Any idea why Helix input was limited to 16 bit? Is there any chance for sound degradation due to the threshold being raised to 32 bit float? E.g. a modern lossy encoder FDKAAC (https://wiki.hydrogenaud.io/index.php?title=Fraunhofer_FDK_AAC) also accepts 16 bit only.

* I divide Helix settings into 3 categories: major (e.g. bitrate choice, high frequency encoding, lowpass filter), minor (e.g. copyright and original bits), and mysterious. Let's try to understand the latter, since you've the skill to analyze the code.

Code: [Select]
A[algor_select]  0 = track input, 1=MPEG-1, 2=MPEG-2, xxxxx=sample_rate

U         u0=generic, u2=Pentium III(SSE)

Q         disable_taper, q0 = base, q1 = fast, q-1 = encoder chooses

TX        tx6, test reserved 6 or 8 seems best (startup_adjustNT1B)
            ** v5.0  TEST 1  as of 8/15/00
            ** v5.0  TEST 2  8/18/00
            ** v5.0  TEST 3  default tx6 (prev = tx8)
            ** v5.0  TEST 4  mods to short fnc_sf, ms corr. hf enable > 80
            ** v5.0  TEST 5  fix odd npart, ix clear
            ** v5.0  TEST 6  add reformatted frames
            ** v5.0  TEST 7  drop V4 amod
            ** v5.1  2005.08.09 (see CVS log for details)

SBT[short_block_threshold]
          short_block_threshold default = 700
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-18 18:26:48
I'll reply to a few points quickly.

Quote from: Kraeved on
a) Sample: bee.wav (https://helpguide.sony.net/high-res/sample1/v1/en/index.html) (96 kHz 24 bit)
* Old Helix version says “UNSUPPORTED PCM FILE TYPE”, current version says “ENCODER INIT FAIL”.
* Let's make the error message more descriptive, e.g. “ERROR. Supported sampling rates are a, b, c, d.” or a - d.
* And if the input cannot be processed, Helix should not create a zero-length output file (e.g. TEST.MP3 0 bytes).
I can improve this.

Quote from: Kraeved on
b) Samples: 24_bit_fixed.wav,  32_bit_float.wav (https://www.sounddevices.com/sample-32-bit-float-and-24-bit-fixed-wav-files/) (48 kHz 24 bit, 48 kHz 32 bit float, they are loud)
The samples have too large headers for the current code. I was actually expecting this issue to surface at some point. Thanks for the samples, I'll fix.

Quote from: Kraeved on
c) Samples: leftright.wav, mcgill.wav (11 kHz 16 bit, 8 kHz 32 bit integer, attached below)
* Helix unexpectedly changes the sampling rate to 22.05 kHz and 16 kHz respectively.
* It is an inherited case, but can you tell, is it a bug or a feature?
Feature. The engine seems to only handle rates 22050, 24000, 16000, 32000, 44100, 48000. Lower sampling rates are upsampled to nearest appropriate. There seems to also be downsampling support, not sure why its use is not allowed for higher sampling rates.

Edit: When the resampler is triggered the output length is way wrong and shifted. Not sure how motivated I am to fix this. Would be simpler to reject unsupported rates and let user resample. For example using foobar as frontend makes that super simple.

Quote from: Kraeved on
d) Filename to verify Unicode support: birds-vögel-птицы-pájaros.wav (English, German, Russian, Spanish)
* Helix successfully creates birds-vögel-птицы-pájaros.mp3 on the disk, but its name is garbled on the output screen.
This was a known issue, I didn't think mirroring the names is that important. But I can see if I can add unicode printing without too much work.

Quote from: Kraeved on
* Any idea why Helix input was limited to 16 bit? Is there any chance for sound degradation due to the threshold being raised to 32 bit float? E.g. a modern lossy encoder FDKAAC (https://wiki.hydrogenaud.io/index.php?title=Fraunhofer_FDK_AAC) also accepts 16 bit only.
Helix encoder's history seems to date back to early 2000, several years before HD audio was even close to being popular or in wide use. It is possible offering support for higher bit depths wasn't deemed necessary. Also it may have seemed unnecessary as the quality of MP3 doesn't really exceed what 16 bit PCM can offer.

I'd think that removing the 16-bit clipping and restriction is very unlikely to cause sound degradation. For 16-bit signals the output remains the same. For floating point signals you can now encode data that is way beyond the digital range without it getting clipped, with the original encoder such signals would just be hard clipped noise.

For FDK though I believe the reason for not supporting beyond 16 bits is purely commercial. They have artificially limited the open source versions so that their commercial software is more attractive.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-18 20:10:38
Quote from: Case on
Quote from: Kraeved on
c) Samples: leftright.wav, mcgill.wav (11 kHz 16 bit, 8 kHz 32 bit integer, attached below)
* Helix unexpectedly changes the sampling rate to 22.05 kHz and 16 kHz respectively.
Feature. The engine seems to only handle rates 16000 ... 48000 Hz. Lower sampling rates are upsampled to nearest appropriate. When the resampler is triggered the output length is way wrong and shifted. Not sure how motivated I am to fix this. Would be simpler to reject unsupported rates and let user resample.

Indeed, @Case. I test various resamplers (https://src.infinitewave.ca/) (including not listed there, e.g. the one (https://github.com/jniemann66/ReSampler) by Judd Niemann), but return to SoX, which is available as a standalone app and as a part of free apps such as Foobar2000. Since we are not sure how good Helix resampler is and adding a custom one such as r8brain (https://github.com/avaneev/r8brain-free-src) is unlikely an easy task, let's stay conservative so far and throw a descriptive error message that the rate is too low or too high. For audiobooks 16 kHz is enough, even too low for my taste, and AMR-NB 8 kHz diary can be upsampled via SoX before piping to Helix*. I wonder why the original team (any contacts? I'd like to write them a letter) chose the upsampling route instead of LAME omnivorous one.

* ffmpeg -i in.amr -f wav -acodec pcm_f32le -af aresample=24000:resampler=soxr:precision=28 - | hmp3 - out.mp3 -V96
Yes, pcm_f32le. Thanks to you, we can now pass the input in 32 bit float format, reducing the likelihood of clipping,
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-18 20:51:00
There was actually a typo in the original sources. The code was always supposed to throw an error when sample rate was above 48 kHz, but the check was written to fail at 480 kHz.

Here's a test version without hardcoded limits to WAVE header size so the b) problem files can be encoded. MP3 file is no longer created when initialization fails. The error message telling that source file is unsupported will print additional information about each part that fails, be it number of channels, bit depth, WAVE type or sample rate.

No other fixes or improvements for now. I should be sleeping already.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-18 21:49:15
No need to rush, @Case, give yourself time to reflect and recuperate. The state of lossy music encoding is disheartening (https://hydrogenaud.io/index.php/topic,120370.msg1040900.html#msg1040900), but the evolution of this very thread proves this is not a death sentence, but a task that can be solved. Let's go slow (https://en.wikipedia.org/wiki/Slow_movement_(culture)), but sure. Here, have a crêpe with jam, sour cream or caviar.

(https://i5.imageban.ru/out/2024/03/19/99120d9a7dce9696450cc961bb0775d7.webp)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-19 15:45:06
Another day, another test build.
* Made encoder reject input tracks with zero samples.
* Added support for printing unicode filenames to the console.
* Fixed gapless data saving, the encoder tried to store too long numbers in padding info fields and the numbers got truncated as the storage space is only 12 bits long.
* Fixed internal resampling to produce output with correct lengths.

And from the yesterday's version:
* Fixed encoding WAV files with huge headers.
* Fixed writing empty track when input had unsupported sample rate.
* Added more detailed error message for input file rejection if the audio specs weren't supported.

I'll post source patch if nothing that needs immediate fixing is found.

Edit: forgot the wav header size fix.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-19 21:28:08
* Resampler. Have you decided not to reject the input with the sampling rate below 16 kHz? For example, Exhale (https://hydrogenaud.io/index.php/topic,118888.msg1040056.html#msg1040056) (USAC converter) says “The sampling rate is 8 kHz but encoding using eSBR requires at least 22 kHz”.

* Gapless. You mentioned this word several times in this thread, but what exactly do you mean? In practical terms, I expect that there will be no click between the two files (https://hydrogenaud.io/index.php/topic,115009.msg1040572.html#msg1040572) played sequentially, but for now there is a click when encoded even in CBR mode, albeit less pronounced than with, say, Opus (https://hydrogenaud.io/index.php/topic,116605.msg1039662.html#msg1039662).

* Input larger than 4 GB. I have not tested pipe encoding in this case, but WAV RF64/BWF (https://www.lynxstudio.com/downloads/aurora-n/#Other) are not recognized.

* Progress (0%) and compress ratio (0.020440%) counters are misleading when the input comes from pipe (sample: bee.wav (https://helpguide.sony.net/high-res/sample1/v1/en/index.html)). If it is impossible to display reliable information in this case, then it makes sense to display - or N/A.

Code: [Select]
$ ffmpeg -v error -i bee.wav -f wav -acodec pcm_f32le -af aresample=44100:resampler=soxr:precision=28 - | hmp3 -V96 - bee.mp3
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    1531  |   14100480 /     877870 |     0%   | 148.26 / 175.68  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 0.020440%

* How to understand “compress ratio” in general? For example, hmp3 utopia.wav (5824 → 1092 bytes, 29.323308%), but hmp3 sine.wav (21 168 044 → 239 774 bytes, 1.132719%).

* Inherited case with CBR. Flag -B stands for constant channel bitrate, so I expect every encoded frame to use that value. Unlike LAME, Helix does not display bitrate distribution, but mp3packer (https://hydrogenaud.io/index.php/topic,32379.0.html) does, although I don't know how reliable it is. When the input frequency rate is less than or equal to 24 kHz, I see that one frame of Helix encoded output uses a lower bitrate. Is it a bug or a feature?

Code: [Select]
$ ffmpeg -v error -i bee.wav -f wav -acodec pcm_f32le -af aresample=24000:resampler=soxr:precision=28 - | hmp3 -B80 - bee.mp3
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    1670  |    7686144 /     801408 |     0%   | 159.96 / 160.00  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 0.018659%

$ mp3packer -i bee.mp3
1670 frames
Bitrate distribution:
  96: 1,0
 160: 1669,0
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-20 09:09:43
Seems that there's quite some flurry of activity over here, with some great improvement being developed by Case.

Wondering if the changes can be collected over at, e.g., https://github.com/maikmerten/hmp3 (or any other public repository)

Are these patches somewhere in a git branch?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-20 09:26:07
Reply #204, above, refers. ;)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-20 14:59:02
Quote from: Kraeved on
* Resampler. Have you decided not to reject the input with the sampling rate below 16 kHz?
I only considered disabling it as I thought fixing it is hopeless. But now that it appears to work nicely there is no reason to remove it.

Quote from: Kraeved on
* Gapless. You mentioned this word several times in this thread, but what exactly do you mean?
In this context "gapless" mean that the encoded length is 100 % correct without silence added or samples removed. Perfect audibly gapless playback requires more work, if it's at all possible when working with separate tracks. I may test extrapolation filter to see if that improves things.

Quote from: Kraeved on
* Input larger than 4 GB. I have not tested pipe encoding in this case, but WAV RF64/BWF (https://www.lynxstudio.com/downloads/aurora-n/#Other) are not recognized.
Not sure it's worth adding support for that. The Xing header where track frame counts and file size and gapless playback info is stored can only support 32 bit numbers. Things may overflow on too long inputs. Original code wasn't designed to handle anything over 2 GB, signed ints are used everywhere. I changed some parts to unsigned to properly handle up-to 4 GB, but had to revert some as you found those insane bitrate numbers with the calculations.

Quote from: Kraeved on
* Progress (0%) and compress ratio (0.020440%) counters are misleading when the input comes from pipe (sample: bee.wav (https://helpguide.sony.net/high-res/sample1/v1/en/index.html)). If it is impossible to display reliable information in this case, then it makes sense to display - or N/A.
Good point. Fixed. I also made display updating work when encoding from pipe.

Quote from: Kraeved on
* How to understand “compress ratio” in general?
The compression ratio is simply calculated by dividing the size of MP3 audio data by size of PCM data. That excludes MP3 and WAVE file headers and metadata.

Quote from: Kraeved on
* Inherited case with CBR. Flag -B stands for constant channel bitrate, so I expect every encoded frame to use that value. [...] When the input frequency rate is less than or equal to 24 kHz, I see that one frame of Helix encoded output uses a lower bitrate. Is it a bug or a feature?
That was a bug. Fixed.

Test binary + source patch attached.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-20 15:13:20
Quote from: maikmerten on
Wondering if the changes can be collected over at, e.g., https://github.com/maikmerten/hmp3 (or any other public repository)

Are these patches somewhere in a git branch?
Patches only posted here. Would be great if you don't mind including them.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-20 16:51:04
Ah, cool!

Just wondering: What codebase was your starting point? The code on GitHub (so the patches should apply cleanly) or some other snapshot?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-20 18:01:53
I used the sources bundled with the Rarewares release. I just verified and all the source files match with your git. The package from john33 just included proper Visual Studio solution files in addition.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-20 18:17:34
That's great, I'll try to merge this into a new branch, test stuff, then merge to main.

Is "Case" proper attribution?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-20 18:47:24
That is fine, unless you prefer to use real name. Then it would be Janne Hyvärinen.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-20 18:50:53
I guess in the commit message, I'd go for the real name, but in the source code, I'd go short with "Case", to avoid problems with character sets on legacy systems.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-20 20:27:38
Your patch applied wonderfully. Due to strange wording in RealNetwork's open-source license (which asks to always note the date of last change for files), I added the latest date of change to the file header.

Patched branch available at https://github.com/maikmerten/hmp3/tree/case - will merge to main after some tests on Linux (edit: To be clear, it compiles fine on Linux and also creates MP3 files, so much is known ;-) ).

Thank you very much, looking through the patch this indeed was a major rework - much more than I ever added to hmp3!

edit2: Also interested in getting a working Visual Studio build solution onto GitHub. Given I don't have Windows/Visual Studio to test things, some guidance is welcome.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-20 20:40:48
Quote from: maikmerten on
Your patch applied wonderfully. Due to strange wording in RealNetwork's open-source license (which asks to always note the date of last change for files), I added the latest date of change to the file header.

Patched branch available at https://github.com/maikmerten/hmp3/tree/case - will merge to main after some tests on Linux (edit: To be clear, it compiles fine on Linux and also creates MP3 files, so much is known ;-) ).

Thank you very much, looking through the patch this indeed was a major rework - much more than I ever added to hmp3!

edit2: Also interested in getting a working Visual Studio build solution onto GitHub. Given I don't have Windows/Visual Studio to test things, some guidance is welcome.
You will find the working VS project files in the hmp3 folder within the source included here:

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip (https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip)

I need to emphasise that these are for Visual Studio 2015. As Case will attest, the masm code does not compile with later versions of VS.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-20 22:34:35
Present

@Case, changes to modernize Helix encoder that we worked on in this iteration were tested to the best of my ability on Windows 7 x64 in direct and pipe modes using files that differ in names, sampling rates, bit depth, channels and headers, including some malformed ones. Since the psychoacoustic model was barely changed, no detailed listening was conducted, but on the surface I detect no degradation or improvement of sound quality. As a final touch, I suggest polishing the interface. God knows, I prefer incremental changes to radical ones, but the current state of Helix screen is a spaghetti for nerds that needs to be redesigned. Today, we have enough command-line apps at our disposal to borrow interface details that facilitate accessibility and bring satisfaction to a wider range of users.

Ideas for a basic screen (that pops up without -h):
* consistent alignment, indentation, line spacing, Sentence case
* vernacular language if possible, sometimes less is more
* encoder's name: is it hmp3 or Helix or Xing?
* where to get the latest version, support, more information?
* options here should be major ones only (e.g. -V, -B, -HF2)
* what are the default values and valid ranges?

Spoiler (click to show/hide)

Future

* Demystifying Helix advanced options (https://hydrogenaud.io/index.php/topic,123331.msg1041407.html#msg1041407) will give us an idea of how they affect quality and speed, how best to use them or even eliminate them altogether in order to reduce code complexity and interface confusion.

Side notes

* Comrade @danadam brought to my attention the following discussion (https://hydrogenaud.io/index.php/topic,101850.msg1041507.html#msg1041507) of the difficulties in getting SoX to work with floating point. I especially remember the phrase and hope it's not related to Helix: “Simply changing sox_sample_t from int to float would for sure break a lot of things”.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: marc2k3 on 2024-03-21 07:51:38
Quote from: maikmerten on
edit2: Also interested in getting a working Visual Studio build solution onto GitHub. Given I don't have Windows/Visual Studio to test things, some guidance is welcome.

Quote from: john33 on
I need to emphasise that these are for Visual Studio 2015. As Case will attest, the masm code does not compile with later versions of VS.

Maybe look at setting up appveyor CI on github. It looks like they still support VS 2015 unlike github itself which only supports 2019/2022. It's free for open source projects.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-21 08:09:45
Regarding the encoder name: I went for " hmp3 MPEG Layer III audio encoder [...]  Utilizing the Helix MP3 encoder" as "Helix" and "Xing" are most likely still trademarked somewhere, so cannot be used as name for this particular encoder.

As for the "TX" advanced mystery options: The value specified for the TX option ends up in the aptly named variable "test1" of the struct "BA_CONTROL" (bit allocation control). Unlike the also present "test2" and "test3", "test1" actually appears to "do" something, in the method "CBitAllo3::startup_adjustNT1B":

Code: [Select]
void
CBitAllo3::startup_adjustNT1B (  )
{
    int ch, i, a, na;
    int ab, nab;
    int d, dmax;
    int f;
    static int sthres[21] = {
        0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0,
        100, 100, 100, 200, 300, 300, 300,
    };

//#define STHRES  300
//#define STHRES  0

    if ( ba_control.test1 == 0 )
        return;
    f = ba_control.test1;

    for ( ch = 0; ch < nchan; ch++ )
    {
        na = 1;
        a = 0;
        nab = 1;
        ab = 0;
        for ( i = 0; i < nsf[ch]; i++ )
        {
            //if( snr[ch][i] > STHRES ) {
            if ( snr[ch][i] > sthres[i] )
            {
                a += NT[ch][i];
                na++;
                ab += nBand_l[i] * NT[ch][i];
                nab += nBand_l[i];
            }
        }
        a = a / na;
        ab = ab / nab;

        if ( na < 5 )
            continue;

        for ( i = 0; i < nsf[ch]; i++ )
        {
            //if( snr[ch][i] > STHRES ) {
            if ( snr[ch][i] > sthres[i] )
            {
                dmax = HX_MAX ( snr[ch][i] - 400, 0 );
                //d = (ab - NT[ch][i]) >> 1;
                d = ( f * ( ab - NT[ch][i] ) ) >> 4;
                d = HX_MIN ( d, dmax );
                NT[ch][i] = NT[ch][i] + d;
            }
        }

    }

}

The value "test1" provides a value for the variable "f", which then further down the road is used as a multiplier to compute an adjusted noise threshold (?) for 21 scalefactor bands. As is a common theme with that code base, I find it hard to actually determine what is happening there (non-descriptive variable names, magic numbers), but "test1" defaults to 6.

Why 6? Because Neal said so! ("set -1 for default, test reserved 6 or 8 seems best, Neal says 6" and "default 6 re. Neal").

Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-21 09:25:13
Quote from: john33 on
You will find the working VS project files in the hmp3 folder within the source included here:

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip (https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip)

I need to emphasise that these are for Visual Studio 2015. As Case will attest, the masm code does not compile with later versions of VS.

The relevant file is "mp3enc_asm.vcxproj", I assume?

There's also a "mp3enc_asm.vcproj" (without the x in the file extension), which appears unchanged from the version in git. Is it worthwhile to keep this file, or has it lost its usefulness?

What attribution do you like?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-21 09:29:53
Quote from: maikmerten on
Quote from: john33 on
You will find the working VS project files in the hmp3 folder within the source included here:

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip (https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip)

I need to emphasise that these are for Visual Studio 2015. As Case will attest, the masm code does not compile with later versions of VS.

The relevant file is "mp3enc_asm.vcxproj", I assume?

There's also a "mp3enc_asm.vcproj" (without the x in the file extension), which appears unchanged from the version in git. Is it worthwhile to keep this file, or has it lost its usefulness?

What attribution do you like?
You need the .sln, .vcproj and .vcxproj. Thanks for the offer, but I'm not sure I've done anything to warrant any attribution. ;)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-21 09:55:18
Quote from: john33 on
You need the .sln, .vcproj and .vcxproj. Thanks for the offer, but I'm not sure I've done anything to warrant any attribution. ;)

Thanks for elaborating! Build setups are important as well (code is otherwise useless), so if you don't mind, I'd include the project files with a "VS2015 project files provided by John" message.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-21 10:20:33
 :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-21 10:28:29
Quote from: maikmerten on
What attribution do you like?

Perhaps the proper attribution would be the umbrella term "Hydrogenaudio members" that I proposed (https://hydrogenaud.io/index.php/topic,123331.msg1041539.html#msg1041539) in the redesigned Helix screen? @john33's contribution is that he compiles builds using alternative chains (sometimes this gives more stability (https://hydrogenaud.io/index.php/topic,125651.msg1041494.html#msg1041494), compatibility with older hardware, speed), provides ad-free & hegemon-free space for uncapped downloads, and also chronicles the origins of audio tools, Helix included (https://www.rarewares.org/rrw/xing.php). Besides, the developer implements not only his own ideas, but relies on feedback, shared samples and rigorous testing, which is what local passionate users like me do.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-21 11:29:20
Okay, after some testing on Linux I now merged Case's patches to the main branch and also included John's project files. Thanks for the contributions!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-21 11:39:20
Quote from: Kraeved on
Perhaps the proper attribution would be the umbrella term "Hydrogenaudio members" that I proposed (https://hydrogenaud.io/index.php/topic,123331.msg1041539.html#msg1041539) in the redesigned Helix screen? @john33's contribution is that he compiles builds using alternative chains (sometimes this gives more stability (https://hydrogenaud.io/index.php/topic,125651.msg1041494.html#msg1041494), compatibility with older hardware, speed), provides ad-free & hegemon-free space for uncapped downloads, and also chronicles the origins of audio tools, Helix included (https://www.rarewares.org/rrw/xing.php). Besides, the developer implements not only his own ideas, but relies on feedback, shared samples and rigorous testing, which is what local passionate users like me do.

There are two dimensions to "attribution". One is to satisfy the terms of the Open Source license, to ensure it is documented who came up with the actual code (and thus has the copyright to parts of the code base). The other dimension is being nice and thankful by recognizing contributors. As you pointed out, there are contributions that are not strictly lines of code and also deserve recognition. As such, I think it'd be very fair to have a "with input from HydrogenAudio contributors" message in there.

edit: Added "With contributions from HydrogenAudio members." to hmp3 welcome message.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-21 13:41:22
@maikmerten Added where and why now? The last couple of pages convince me that the development takes place here, not on Github or anywhere else: new ideas are proposed, implemented, compiled and tested daily. In particular, @Case is now double-checking float operations and considering a proposal to a long overdue redesign of Helix screen, which also includes a new way for listing contributors. Methinks, it was worth joining the discussion or waiting until the end of this iteration before migrating the code, so as not to create two points of competing edits.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-21 14:09:06
It's standard software development practice to commit source to a proper version control. In fact, the source code Case has been basing his improvements on came from that very same GitHub repository.

This is not in any way about shifting discussion over to GitHub. Discussion is here. Keeping the repository updated is about making sure that the work done here is not lost and available in a proper fashion.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: NetRanger on 2024-03-21 16:01:55
@maikmerten - Any plans on setting up your GitHub so that you get compiles of your 'own' that gets added under Releases?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-21 16:08:20
Quote from: NetRanger on
@maikmerten - Any plans on setting up your GitHub so that you get compiles of your 'own' that gets added under Releases?

That sounds like a generally good idea, just need to tinker around how this is accomplished :-)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-21 18:56:07
Quote from: marc2k3 on
Maybe look at setting up appveyor CI on github. It looks like they still support VS 2015 unlike github itself which only supports 2019/2022. It's free for open source projects.

appveyor indeed looks promising and it indeed has a VS2015 environment.

@john33 - in https://github.com/maikmerten/hmp3/blob/main/hmp3/src/mp3enc_x86asm.vcproj there's another project file and appveyor tries to build that and fails. I guess this can be removed now that your project files are in the repo?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: NetRanger on 2024-03-21 19:58:54
Quote from: maikmerten on
Quote from: NetRanger on
@maikmerten - Any plans on setting up your GitHub so that you get compiles of your 'own' that gets added under Releases?

That sounds like a generally good idea, just need to tinker around how this is accomplished :-)

Think it's called github-actions or so, Just don't ask me how it's done. :D
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: marc2k3 on 2024-03-21 20:14:46
I use this:

https://github.com/marketplace/actions/gh-release
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-21 20:24:24
Quote from: maikmerten on
@john33 - in https://github.com/maikmerten/hmp3/blob/main/hmp3/src/mp3enc_x86asm.vcproj there's another project file and appveyor tries to build that and fails. I guess this can be removed now that your project files are in the repo?
Not sure I follow, but the old project files (.dsp and .dsw) in /hmp3/hmp3 are redundant and that folder could be removed.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-22 06:10:58
Quote from: john33 on
Not sure I follow, but the old project files (.dsp and .dsw) in /hmp3/hmp3 are redundant and that folder could be removed.

Oh, didn't even notice the .dsp/.dsw - yeah, those look obsolete.

I was wondering why there are two .vcproj, though:

./hmp3/src/mp3enc_x86asm.vcproj
./hmp3/mp3enc_asm.vcproj

and if the one in ./hmp3/src/ is obsolete.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-22 06:20:07
Quote from: Kraeved on
@Case is now double-checking float operations and considering a proposal to a long overdue redesign of Helix screen
Not sure if you meant the float part as a joke but that doesn't need any double checking. The encoder has always worked internally with floats, the input was just artifically limited to 16-bits. Before I posted the first float-enabled version I had verified that standard 16-bit behavior was unchanged and that floats exceeding -1.0..+1.0 range are nicely preserved.
The help screen can certainly use improvements.

Quote from: maikmerten on
@john33 - in https://github.com/maikmerten/hmp3/blob/main/hmp3/src/mp3enc_x86asm.vcproj there's another project file and appveyor tries to build that and fails. I guess this can be removed now that your project files are in the repo?
The .vcproj file is ancient Visual Studio project file. Visual Studio 2010 changed the format and name to include the 'x'. Since the solution uses the newer VS2010+ format the old file is absolutely useless.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-22 08:15:17
Quote from: Case on
The .vcproj file is ancient Visual Studio project file. Visual Studio 2010 changed the format and name to include the 'x'. Since the solution uses the newer VS2010+ format the old file is absolutely useless.
Thanks, good to know. I just carried them forward without giving them any thought!!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-22 10:02:35
I took the liberty of removing outdated project files (.dsp, .dsw, .vcproj) via https://github.com/maikmerten/hmp3/commit/7b677743be5fdc8c9b3fa13e5db037ed57f3661c

On appveyor, I nearly got the automated Win32 builds working, "just" failing at the linking stage:

Code: [Select]
Build started
git clone -q --branch=main https://github.com/maikmerten/hmp3.git C:\projects\hmp3
git checkout -qf 7b677743be5fdc8c9b3fa13e5db037ed57f3661c
msbuild "C:\projects\hmp3\hmp3\mp3enc_asm.sln" /p:PlatformToolset=v140 /verbosity:minimal /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll"
Microsoft (R) Build Engine version 14.0.25420.1
Copyright (C) Microsoft Corporation. All rights reserved.
  Performing Custom Build Tools
  Microsoft (R) Macro Assembler Version 14.00.24210.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
   Assembling: C:\projects\hmp3\hmp3\src\platform\win\i386\cnt.asm
  Performing Custom Build Tools
  Microsoft (R) Macro Assembler Version 14.00.24210.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
   Assembling: C:\projects\hmp3\hmp3\src\platform\win\i386\emdct.asm
  Performing Custom Build Tools
  Microsoft (R) Macro Assembler Version 14.00.24210.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
   Assembling: C:\projects\hmp3\hmp3\src\platform\win\i386\filter2.asm
  Performing Custom Build Tools
  Microsoft (R) Macro Assembler Version 14.00.24210.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
   Assembling: C:\projects\hmp3\hmp3\src\platform\win\i386\hwin.asm
  Performing Custom Build Tools
  Microsoft (R) Macro Assembler Version 14.00.24210.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
   Assembling: C:\projects\hmp3\hmp3\src\platform\win\i386\l3math.asm
  Performing Custom Build Tools
  Microsoft (R) Macro Assembler Version 14.00.24210.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
   Assembling: C:\projects\hmp3\hmp3\src\platform\win\i386\pow34.asm
  Performing Custom Build Tools
  Microsoft (R) Macro Assembler Version 14.00.24210.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
   Assembling: C:\projects\hmp3\hmp3\src\platform\win\i386\sbt.asm
  Performing Custom Build Tools
  Microsoft (R) Macro Assembler Version 14.00.24210.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
   Assembling: C:\projects\hmp3\hmp3\src\platform\win\i386\xhwin.asm
  Performing Custom Build Tools
  Microsoft (R) Macro Assembler Version 14.00.24210.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
   Assembling: C:\projects\hmp3\hmp3\src\platform\win\i386\xsbt.asm
  amodini2.c
  cnts.c
  detect.c
  emap.c
  l3init.c
  l3pack.c
  pcmhpm.c
  setup.c
  spdsmr.c
  xhead.c
  Generating Code...
  bitallo.cpp
  bitallo1.cpp
  bitallo3.cpp
  bitalloc.cpp
  bitallos.cpp
  bitallosc.cpp
  mp3enc.cpp
  srcc.cpp
  srccf.cpp
  srccfb.cpp
  tomp3.cpp
  Generating Code...
cnt.obj : error LNK2026: module unsafe for SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
emdct.obj : error LNK2026: module unsafe for SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
filter2.obj : error LNK2026: module unsafe for SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
hwin.obj : error LNK2026: module unsafe for SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
l3math.obj : error LNK2026: module unsafe for SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
pow34.obj : error LNK2026: module unsafe for SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
sbt.obj : error LNK2026: module unsafe for SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
xhwin.obj : error LNK2026: module unsafe for SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
xsbt.obj : error LNK2026: module unsafe for SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
amodini2.obj : warning LNK4075: ignoring '/EDITANDCONTINUE' due to '/SAFESEH' specification [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]
.\Debug\mp3enc.exe : fatal error LNK1281: Unable to generate SAFESEH image. [C:\projects\hmp3\hmp3\mp3enc_asm.vcxproj]

I guess that's some build settings thing that doesn't appear to be unsolvable...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-22 11:30:51
In VS, Project Properties, go to Linker/Advanced/Image Has Safe Exception Handlers/ and select 'No (/SAFESEH:NO)' from the dropdown.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-22 11:37:13
That's consistent with the information I found, e.g., on https://stackoverflow.com/questions/14710577/error-lnk2026-module-unsafe-for-safeseh-image

In practical terms, I don't have a copy of VS (the output is from an automated cloud build service), so I wonder if there's a line in the project file where this can be specified manually.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-22 12:04:06
Insert the line:
<ImageHasSafeExceptionHandlers>false</ImageHasSafeExceptionHandlers>
beneath line 72 in mp3enc_asm.vcxproj

I'll update mp3enc_asm.vcxproj in the Rarewares download.

Edit: It's already there for the 'Release' build, but not for the 'Debug'.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: marc2k3 on 2024-03-22 12:58:07
In that case, this..

Code: [Select]
msbuild "C:\projects\hmp3\hmp3\mp3enc_asm.sln" /p:PlatformToolset=v140 /verbosity:minimal /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll"

probably wants to be...

Code: [Select]
msbuild "C:\projects\hmp3\hmp3\mp3enc_asm.sln" /p:PlatformToolset=v140 /p:Configuration=Release /verbosity:minimal /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll"
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-22 13:57:46
Indeed, selecting the Release profile now yields successful automated compiles. Neat!

Now I just need to set up packaging things.

edit: Okay, after some messing around, now there are automated hmp3 win32 builds over at https://ci.appveyor.com/project/maikmerten/hmp3/build/artifacts
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: marc2k3 on 2024-03-22 18:06:48
^One month expiry though. But following a couple of links gets you here...

https://www.appveyor.com/docs/deployment/github/
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-22 19:03:37
Yeah, one month expiry, thus no permanent hosting solution. Deploying to GitHub releases is one possibility to keep things permanent.

In other news, I'm somewhat suspicious that the assembly code is a good fit for modern CPUs. It's 32-bit only and some of the assembly files look like slightly cleaned up compiler output of whatever ancient compilers they used.

With minimal tweaks to the Makefile, I can cross-compile 64-bit Windows executables with a somewhat modern GCC compiler.

Can somebody on 64 bit Windows (so a normal, modern Windows install) try the enclosed "win64" build and compare its performance to the 32-bit executable ("win32")? To enable the SSE assembly (originally for the Pentium III), one needs to pass the "-U2" hmp3 parameter.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-22 20:03:41
@maikmerten

Setup: Core2Duo SSE3, HDD, Windows 7 x64.

(https://i4.imageban.ru/out/2024/02/25/ef921b2d6df14d10ceb8beb722f16200.png)

I tested your binaries with hyperfine (https://github.com/sharkdp/hyperfine) --warmup 3 --runs 6 "command"

01.wav, 26 494 164 bytes
02.wav, 99 423 788 bytes

Code: [Select]
---------------------------------------------------------------------------
  Command                               Mean [s]        Min [s]   Max [s]  
---------------------------------------------------------------------------
  hmp3 01.wav -V150 -HF2 -D             1.806 ± 0.008   1.797     1.819    
  hmp3 01.wav -V150 -HF2 -U2 -D         1.752 ± 0.018   1.723     1.769    
  hmp3-win64 01.wav -V150 -HF2 -D       1.516 ± 0.012   1.502     1.535    
  hmp3-win64 01.wav -V150 -HF2 -U2 -D   1.527 ± 0.010   1.515     1.540    
---------------------------------------------------------------------------

---------------------------------------------------------------------------
  Command                               Mean [s]        Min [s]   Max [s]  
---------------------------------------------------------------------------
  hmp3 02.wav -V150 -HF2 -D             6.709 ± 0.076   6.618     6.843    
  hmp3 02.wav -V150 -HF2 -U2 -D         6.459 ± 0.033   6.421     6.504    
  hmp3-win64 02.wav -V150 -HF2 -D       5.618 ± 0.045   5.548     5.675    
  hmp3-win64 02.wav -V150 -HF2 -U2 -D   5.601 ± 0.027   5.552     5.629    
---------------------------------------------------------------------------
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-22 20:24:57
That is a 17 year old CPU and a 15 year old OS. What are you trying to demonstrate?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-22 20:26:44
Quote from: john33 on
That is a 17 year old CPU and a 15 year old OS. What are you trying to demonstrate?

Quote from: maikmerten on
Can somebody on 64 bit Windows (so a normal, modern Windows install) try the enclosed "win64" build and compare its performance to the 32-bit executable ("win32")? To enable the SSE assembly (originally for the Pentium III), one needs to pass the "-U2" hmp3 parameter.

@john33, are you paying attention or drinking gin?

(https://i2.imageban.ru/out/2024/03/22/b6fe6554af52d1eae14762c99c31048a.png) (https://www.rarewares.org/about.php)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-22 20:30:07
Well I'd question that meets any criteria related to 'modern'!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-23 04:34:06
I tested with otherwise similar parameters to Kraeved but I benchmarked both candidates with and without "-U2". The -U2 parameter had a tiny speed difference on 64-bit and affected the 32-bit encoder much more. My test machine is slightly newer, Core i9-12900K running on Windows 11 Pro. I combined a CD into a single 3341+ seconds long WAV file that I used as a test. Encoding was done 3 times for each test and timer64 was used to record time. I used average times to compute the speeds.

Code: [Select]
32-bit (-U2):  509x realtime
32-bit:        438x realtime
64-bit (-U2):  680x realtime, 34% faster than 32-bit
64-bit:        677x realtime, 55% faster than 32-bit
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: vina54 on 2024-03-23 07:10:20
Quote from: Case on
Quote from: Kraeved on
@Case is now double-checking float operations and considering a proposal to a long overdue redesign of Helix screen
Not sure if you meant the float part as a joke but that doesn't need any double checking. The encoder has always worked internally with floats, the input was just artifically limited to 16-bits. Before I posted the first float-enabled version I had verified that standard 16-bit behavior was unchanged and that floats exceeding -1.0..+1.0 range are nicely preserved.
The help screen can certainly use improvements.

Quote from: maikmerten on
@john33 - in https://github.com/maikmerten/hmp3/blob/main/hmp3/src/mp3enc_x86asm.vcproj there's another project file and appveyor tries to build that and fails. I guess this can be removed now that your project files are in the repo?
The .vcproj file is ancient Visual Studio project file. Visual Studio 2010 changed the format and name to include the 'x'. Since the solution uses the newer VS2010+ format the old file is absolutely useless.

Is there a reason why this encoder was artificially limited to only 16-bits?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-23 07:24:50
Quote from: vina54 on
Is there a reason why this encoder was artificially limited to only 16-bits?

I asked about it a couple of pages ago. @Case shared (https://hydrogenaud.io/index.php/topic,123331.msg1041419.html#msg1041419) his idea.

Quote from: Case on
Quote from: Kraeved on
* Any idea why Helix input was limited to 16 bit?
Helix encoder's history seems to date back to early 2000, several years before HD audio was even close to being popular or in wide use. It is possible offering support for higher bit depths wasn't deemed necessary. Also it may have seemed unnecessary as the quality of MP3 doesn't really exceed what 16 bit PCM can offer.

However, I see no consensus that it is better to send a signal to MP3 encoders in 32-bit float, 32-bit or 24-bit integer, because there are decoders, both software and hardware, that might truncate extra bits, causing distortion. So it seems that your choice depends on the capabilities of your target audience. Personally, I use 32-bit float for signal processing, but end up with 16-bit TPDF highpass dither (https://foobar.hyv.fi/?view=foo_dsp_dither) if the source has more bits or effects are added (except for resampling).

a (https://youtu.be/watch?t=965&v=2iDrbgfPjPY)) “It's probably a safer bet to encode MP3s from a 16 bit dithered file.” (video demo by Dan Worrall, sound engineer)
b (https://gearspace.com/board/mastering-forum/1298360-lossy-files-made-24-bit-vs-16-bit-wavs.html#post14529921)) “Tests of various decoders demonstrated that they don’t all meet the standard.” (specs and observations)
c (https://gearspace.com/board/mastering-forum/1379854-mp3-floating-24-16-a.html#post15999320)) “MP3 encoding breaks the statistical properties of dither noise, but it still helps a bit.” (opinion of Alexey Lukin, PhD, DSP engineer at iZotope)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-23 09:57:01
Quote from: john33 on
Well I'd question that meets any criteria related to 'modern'!

Hi John, hi Kraeved,

I think I caused the confusion here by being vague about "modern". I just meant "64-bit CPU on 64-bit OS". Thus testing on an older, but 64-bit capable x86-CPU (x86_64, with at least SSE2) such as the Intel Core 2 is still valuable, so the results Kraeved generated are still interesting to me!

These results are also in alignment with what Case found out on more recent Intel gear: It appears that 64-bit compiles of the C code are more efficient than the 32-bit compiles with the old ASM routines.

As such, I do wonder if it is worth retaining the ASM parts. Thoughts on this?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-23 10:24:39
I see no reason to remove the asm code. With Visual Studio 64-bit compile isn't as fast as the asm-enabled 32-bit version.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-23 10:33:54
Quote from: Case on
I see no reason to remove the asm code. With Visual Studio 64-bit compile isn't as fast as the asm-enabled 32-bit version.
Agreed. The GCC x64 compile is way faster than the VS x64 compile. Care to share the Makefile tweaks?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-23 10:52:42
Quote from: maikmerten on
It appears that 64-bit compiles of the C code are more efficient than the 32-bit compiles with the old ASM routines. As such, I do wonder if it is worth retaining the ASM parts. Thoughts on this?

Premature optimization (https://www.google.com/search?q=premature+optimization) makes no good. By releasing x64 version now, you gain a few seconds, but double the testing burden, which sometimes runs into tens of minutes. x86 of Helix encoder should not be deprioritized anytime soon, because MP3 isn't just for the rich and privileged, other parts of the world (https://hydrogenaud.io/index.php/topic,125597.msg1040939.html#msg1040939) have much more modest technical means. Personally, I even don't use Opus for audiobooks and music, because the complexity of decoding drains the battery much faster. I believe we first need to demystify all the Helix options, adjust them as necessary (for example, activate -U2 by default and remove this flag), polish the interface (https://hydrogenaud.io/index.php/topic,123331.msg1041539.html#msg1041539), agree that we are all happy with the result, and only then release x64 version like the cherry on the cake.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-23 12:37:01
Quote from: john33 on
Quote from: Case on
I see no reason to remove the asm code. With Visual Studio 64-bit compile isn't as fast as the asm-enabled 32-bit version.
Agreed. The GCC x64 compile is way faster than the VS x64 compile. Care to share the Makefile tweaks?
Don't worry, about tweaks, I just made a GCC13.2.0 64 bit compile that runs about twice as fast as the VS 64 bit compile. It's AVX2 optimised
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-23 12:47:55
Since you are already rolling out the updated version of Helix MP3 encoder for public use, I believe its number 5.2.1 should be increased in the sources, on the help screen and in the LAME tag. The changes are significant, so it is more appropriate to use 5.3 instead of 5.2.2.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-23 12:55:09
Quote from: john33 on
Agreed. The GCC x64 compile is way faster than the VS x64 compile. Care to share the Makefile tweaks?

Sure. The very minor tweaks to the Makefile to cross-compile win64 executables on Linux are:

Code: [Select]
CC=x86_64-w64-mingw32-gcc
(use the mingw cross-compile toolchain)

and

Code: [Select]
LFLAGS=-lm -lstdc++ -static
(added -static so the necessary libraries are included)

This alone is sufficient to compile a neat hmp3.exe
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-23 13:07:39
Quote from: Kraeved on
x86 of Helix encoder should not be deprioritized anytime soon

Oh, don't worry, I don't at all propose we should sacrifice decent x86 (as in 32-bit) support. I'm merely trying to gauge whether this support is best provided by the old ASM routines (which restrict upgrades to better compilers) or by letting modern compilers optimize the C routines.

All of this, of course, also doesn't prelude better documentation and more user-friendly program messages.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-23 13:25:29
For further investigation on the 32-bit builds: Here's hmp3 compiled with GCC for Win32. One build targets i586 (Pentium), another i686 with SSE (Pentium 3). Are there significant speed differences to the Win64 compiles and VS2015 32-bit builds?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-23 14:51:34
The 32-bit asm variant is 11% faster than the i686-SSE build and 53% faster than the i586 version here.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Replica9000 on 2024-03-23 14:56:25
I have no experience with this particular encoder.  In testing builds with FLAC and FDK-AAC, GCC 12.2.x produced the fastest binaries, especially with the -ffast-math option. 
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-23 15:23:35
Quote from: Case on
The 32-bit asm variant is 11% faster than the i686-SSE build and 53% faster than the i586 version here.

Thanks for testing! Can you give the enclosed i686-SSE build a try? I tuned the CFLAGS to not only *allow* SSE, but do floating point math only with SSE. (In my testing with Wine on Linux on AMD Ryzen, this turned out significantly faster)

Code: [Select]
CFLAGS_REL=-O3 -march=pentium3 -mtune=pentium3 -mfpmath=sse -fno-math-errno -fno-trapping-math -c -I$(SRC_PREFIX)/pub -DIEEE_FLOAT

Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-23 15:56:51
* in.wav, 99 423 788 bytes
* hmp3case.exe (https://hydrogenaud.io/index.php/topic,123331.msg1041516.html#msg1041516), SHA256: 8fa71fba6624cde0772306d09c2d922fbe36039a9f6bb126d602aeec9f0110e4

Code: [Select]
-------------------------------------------------------------------------------------
  Command                                         Mean [s]        Min [s]   Max [s]  
-------------------------------------------------------------------------------------
  hmp3case.exe in.wav -V150 -HF2 -D               7.045 ± 0.025   7.020     7.076    
  hmp3case.exe in.wav -V150 -HF2 -U2 -D           6.877 ± 0.041   6.820     6.921    
  hmp3-i586.exe in.wav -V150 -HF2 -D              6.305 ± 0.035   6.255     6.346    
  hmp3-i586.exe in.wav -V150 -HF2 -U2 -D          6.313 ± 0.039   6.266     6.378    
  hmp3-i686-sse.exe in.wav -V150 -HF2 -D          6.290 ± 0.031   6.232     6.316    
  hmp3-i686-sse.exe in.wav -V150 -HF2 -U2 -D      6.307 ± 0.055   6.201     6.354    
  hmp3-i686-sse.v2.exe in.wav -V150 -HF2 -D       5.937 ± 0.035   5.879     5.978    
  hmp3-i686-sse.v2.exe in.wav -V150 -HF2 -U2 -D   5.911 ± 0.046   5.824     5.947    
-------------------------------------------------------------------------------------
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-03-23 16:13:41
The second SSE version managed to be 11% faster than the asm build. GCC is quite impressive.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-23 17:27:05
Kraeved, Case, thanks for testing! This means we can compile faster-than-original 32-bit binaries for legacy-systems without having to rely on the ASM bits, thanks to modern compiler optimizations. I  added MinGW-based cross-compilation targets to the Linux Makefile.

As for hmp3 documentation: Would the HydrogenAudio Knowledge base (the wiki) be a proper place to start documenting hmp3?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2024-03-23 17:44:41
Quote from: maikmerten on
As for hmp3 documentation: Would the HydrogenAudio Knowledge base (the wiki) be a proper place to start documenting hmp3?

There is a separate subforum for wiki discussion, but my $2e-2 are
* why not? ...
* ... except it may "end" up like https://wiki.hydrogenaud.io/index.php?title=ReplayGain_2.0_specification .
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-03-24 01:44:52
Interesting things happen to old helix here. It compiles fine with gcc 13.2.0.
A complete album encodes in around 5 seconds and this is not even multithreaded :)
Many thanks to all involved people for your work again!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-24 11:46:49
A quick test here - 795MB .wav file:

Code: [Select]
VS2015-win32 with MASM - 15.52sec
GCC-13.2.0-win32       - 14.05sec
GCC-13.2.0-x64         - 12.09sec
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: zbutsam on 2024-03-24 20:27:20
Hello, old timer here.
Could someone please provide a Helix command line with appropriate parameters equivalent to Lame V2 vbr for use with Foobar?
Also, I'd like to ask if Helix is safe to use regarding bugs with software and hardware playback devices.
 
Thanks in advance.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-25 06:42:30
Code: [Select]
hmp3 input.wav output.mp3 -F19000 -HF2 -V115
should be somewhat -V2-comparable, at least in terms of bitrate and lowpass. At this bitrate, I expect results to be transparent in most cases, too, but your ears be judge on that.

So far, there are no reports of hardware/software player incompatibility.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-25 23:08:04
@maikmerten I got the impression that the development of Helix stalled just at the moment when they reached 16 kHz, and frequencies above this threshold remained something like “here be dragons (https://en.wikipedia.org/wiki/Here_be_dragons)”, where you are allowed at your own risk with HF flag. For example, see and listen to what happens in the following case of a noisy HF range.

Spoiler (click to show/hide)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-03-26 01:38:19
Somehow choosing AVX optimized gcc options do almost nothing here as do all other optimizations. The code compiles in a few seconds.
I'd be cautious with fastmath since the ouput of lossy codecs may suffer. At least the resulting files differ and may result in degration.
I guess the -fno-math-errno -fno-trapping-math already in the makefile is all.
I attached a simple generic x64 compile that seems not to be faster as former maikemertens x64 build but smaller.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: vina54 on 2024-03-26 02:06:34
Quote from: Kraeved on
@maikmerten I got the impression that the development of Helix stalled just at the moment when they reached 16 kHz, and frequencies above this threshold remained something like

Is there a reason why this encoder doesn't put a lowpass filter after -V80?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-26 06:18:53
Quote from: Kraeved on
@maikmerten I got the impression that the development of Helix stalled just at the moment when they reached 16 kHz, and frequencies above this threshold remained something like “here be dragons (https://en.wikipedia.org/wiki/Here_be_dragons)”, where you are allowed at your own risk with HF flag. For example, see and listen to what happens in the following case of a noisy HF range.

Spoiler (click to show/hide)

I think the funny-looking problems in the HF-range may stem from difficulties in the MP3 format to efficiently encode content beyond 16 kHz itself. Due to how the MP3 bitstream format is designed, encoding high frequencies with high precision comes at the expense of being wasteful (and thus inefficient) in the lower frequencies, due to the high-frequency band not having a separate scale factor ("sfb21 defect"). Helix appears to favor giving precision to the lower frequency bands, which in most cases should be the "sane" choice.

Your test signal doesn't just happen to have high-frequency content, its HF-energy is completely blasting over the more relevant content in the lower frequency ranges. This is certainly not something the psychoacoustic model would be expecting. I also hope that you added the high-frequency high-energy noise there artificially and isn't part of an actual recording.

For comparison, I enclosed another LAME encoding (3.100 -q0 -V0). It manages to preserve the high frequency content much better in terms of how the spectrogram looks, but sounds worse to me, adding a more pronounced metallic warbling noise. It's clear the encoder is struggling, too, using 100% 320 kbps frames, which isn't typical for a VBR encoding.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-26 06:26:28
Quote from: vina54 on
Is there a reason why this encoder doesn't put a lowpass filter after -V80?

Without the -HF switch (which is only available starting with -V80), the encoder will not encode anything beyond 16 kHz anyways, no matter what, so there's an implicit lowpass by not coding >16 kHz signals. With the -HF switch and starting with -V80, it's expected the encoder will make somewhat sane decisions on how to approximate the >16 kHz band, not expecting a lot of energy there anyways.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-26 08:47:27
Quote from: maikmerten on
For comparison, I enclosed another LAME encoding (3.100 -q0 -V0).

I replaced LAME 3.100 (there is even some 3.101 beta 3 (https://www.free-codecs.com/download/lame_encoder.htm) out there) with tweaked 3995o (https://www.rarewares.org/mp3-lame-bundle.php#lame-current-halb27), because the new psychoacoustic model (vbr-new vs vbr-old) left much to be desired (https://hydrogenaud.io/index.php/topic,120193.msg1040054.html#msg1040054). Perhaps an earlier version would have been even better, but I had to stop the exploration somewhere and settle down to stay sane. How do you like the sound of the 3995o encoded sample I attached above (https://hydrogenaud.io/index.php/topic,123331.msg1041803.html#msg1041803)?


Quote from: vina54 on
Is there a reason why this encoder doesn't put a lowpass filter after -V80?

If we run hmp3 with -EC flag, we will see more information about the encoding process:
V0-52 uses freq_limit 11713-15847, V53-150 uses freq_limit 16536, -HF2 raises freq_limit to 24000.
We can use -F flag to modify that threshold, e.g. -HF2 -F18000 makes freq_limit 18000.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-26 09:58:55
Quote from: Kraeved on
How do you like the sound of the 3995o encoded sample I attached above (https://hydrogenaud.io/index.php/topic,123331.msg1041803.html#msg1041803)?

Your LAME 3.995 encode did overall a better job, certainly better than LAME 3.100 for this sample. However, as one can see in the spectrogram, it somewhat avoided most of the sbf21 trouble by effectively lowpassing the high frequency band and thus getting rid of the excessive energy there, which might be the best tradeoff in this case.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-26 12:29:31
@maikmerten Forgive my intuitive observations and vernacular language, but it looks as if LAME 3995o -Q1 sets the upper threshold around 18.5 kHz and processes the entire material, while Helix splits the material into two parts (0-16 kHz and 16-18.5 kHz with -HF2 -F18500) and the issue arises in the very upper part, which is separated from the rest of the material by a certain (unexpected black) gap on the spectrogram and that discontinuity brings metallic vibes. What does the Helix code say: is this approach to high frequencies hardwired into the psychoacoustic model and it is difficult to change it without damaging the rest, or is it some parameter that can be adjusted? 
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-26 12:56:55
As far as I can tell, Helix chooses encoding parameters for the < 16 kHz bands (e.g., global and per-band scale factors) and then encodes > 16 kHz with whatever global scale has been selected. Helix won't trade off quality in the lower bands to accommodate the high frequency band (as might be necessary to increase >16 kHz resolution due to the sfb21 MP3 format problem). This is actually a pretty sane approach (perhaps somewhat resembling LAME's -Y parameter) and should be pretty robust apart from (hopefully rare) edge cases. It would explain why >16 kHz looks disconnected from the other frequency bands.

The code to achieve all of this is rather obtuse, with poor naming conventions regarding variable names etc., so I might misread things. I certainly am not sure of my findings. The relevant logic is in bitallo3.cpp, though.

There doesn't appear to be a parameter to influence that behavior.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-26 14:09:16
@maikmerten Hence, the serious question arises: should we use -HF2 flag at all, by itself and in conjunction with -F to limit the upper threshold? If I remove frequencies (https://hydrogenaud.io/index.php/topic,101850.msg1040331.html#msg1040331) below 16 kHz and listen to what's left above, then I hear them clearly, so it seems that I will lose not some barely audible scratching, but something significant. On the other hand, how correct is it to estimate losses in this way, since we listen to the material in its entirety (think AAC-compressed music on Youtube (https://hydrogenaud.io/index.php/topic,67376.msg1041647.html#msg1041647) that is limited to 16 kHz and yet is capable of pumping up your mood) and agree in advance that it is lossy. In particular, @halb27, who tweaked the mentioned Lame 3995o during his lifetime, was inclined to believe that we should pay less attention to -HF2 and rather enjoy a high-quality and quickly encoded range from 0 to 16 kHz, containing a musical basis. Probably, samples that sound better with -HF2 would help answer this question.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-26 14:38:55
Surely you are now venturing into areas of personal preference? The Wiki entry should offer guidance and nothing more.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-26 14:58:00
@john33 We don't have access to the original developers who could give us the certainty that we would put on the Wiki. Therefore, there is this topic where, like archaeologists, we try to reconstruct the picture (of the psychoacoustic model, in particular) from fragments as much as possible. It's a form of brainstorming that consists of researching the code, processing samples, asking uncomfortable questions and searching for answers. It may well turn out that Helix was never designed to process the entire spectrum of frequencies and -HF2 was just one of many experiments, and if so, then people will need to be warned about this. We can put it in the section “Usage guide (https://www.wavpack.com/wavpack_doc.html#usage)”, or “Recommended usage”.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: vina54 on 2024-03-26 15:22:47
Didn't this encoder come out during the days of dialup? Wasn't the main focus back then is to make the encoding as efficient as possible at lower bitrates? (similar to FDK)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-26 15:50:37
Yeah, sure, it's valid to ask how to properly operate the tools at our disposal.

Regarding -HF2, I'm not very worried. As was for the MP3 format itself, >16 kHz frequencies may have been are more of an afterthought for the Helix encoder, but the approach chosen appears sane. Also, there are listening tests with -HF2 that show that Helix is doing plenty fine, e.g., https://hydrogenaud.io/index.php/topic,113324.0.html - so if the HF implementation is an afterthought, it's a pretty solid afterthought.

As for the lullaby sample: It's a case where the actually significant audio is drowned in near-inaudible HF-noise. I attached an image with a 16 kHz lowpassed version showing the "actual" content (top tracks), compared to the sea of HF noise (bottom tracks). In the killer-sample thread, it has been demonstrated that this might have been due to noise-shapened dither dumping all that energy into the HF range. I'd strongly advise not to generalize findings from this sample, it's apparently already processed in ways that are detrimental to psychoacoustic compression.

So I wouldn't recommend always lowpassing at 16 kHz (or just leaving the -HF switch to off) for now. In my younger years, I certainly could distinguish between files with a 16 kHz lowpass and the original sample. Now it's two decades later, so I most likely don't need to worry one way or another anymore, but apparently there are still young folks with good ears out there.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-26 18:52:27
Mr. @Porcus, over the years you have been painstakingly processing your diverse library of samples with various audio tools. Perhaps you have a few samples that are rich enough above 16 kHz so they could potentially sound better with -HF2 flag, which obliges Helix MP3 encoder not to cut frequencies above the mentioned threshold?

Also, I compared this x64 build (https://hydrogenaud.io/index.php/topic,123331.msg1041807.html#msg1041807) by @Wombat (577 024 B) with the current x86 (https://hydrogenaud.io/index.php/topic,123331.msg1041703.html#msg1041703) favorite by @maikmerten (1 240 909 B).

Code: [Select]
-------------------------------------------------------------------------------------
  Command                                         Mean [s]        Min [s]   Max [s] 
-------------------------------------------------------------------------------------
  hmp3-i686-sse.v2.exe in.wav -V150 -HF2 -U2 -D   5.906 ± 0.048   5.821     5.954   
  hmp3-wombat-x64.exe in.wav -V150 -HF2 -U2 -D    5.381 ± 0.042   5.306     5.431   
-------------------------------------------------------------------------------------
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2024-03-26 22:41:28
I don't have "problem samples" for lossy encoding. (Of course I "have", but I have not catalogued them as such - I'm sticking to the lossless department.)

Anyway, 30 seconds of harpsichord which needs quite some bitrate for lossless compression: https://filebin.net/ivsp7ep0jk1gav68
See the spectrogram ... but it has been known for quite a while that harpsichord is "difficult".
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-03-27 01:09:39
@maikmerten (https://hydrogenaud.io/index.php?action=profile;u=1312)
I saw you removed "-fno-math-errno -fno-trapping-math" from the makefile because of being careful about FP-math optimizations.
I guess because i mentioned concerns about math optimizations may change lossy encoders behaviour.
The 2 flags you choosed didn't change the output here but also didn't help with speed on a 5900x.
It is more obvious when i enable fast-math or AVX2.
I really may be overcautious but i think we once had problems with a lame compile because of such things.
Sorry if i have confused anyone.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-27 05:18:57
Quote from: Wombat on
@maikmerten (https://hydrogenaud.io/index.php?action=profile;u=1312)
I saw you removed "-fno-math-errno -fno-trapping-math" from the makefile because of being careful about FP-math optimizations.
I guess because i mentioned concerns about math optimizations may change lossy encoders behaviour.
The 2 flags you choosed didn't change the output here but also didn't help with speed on a 5900x.
It is more obvious when i enable fast-math or AVX2.
I really may be overcautious but i think we once had problems with a lame compile because of such things.
Sorry if i have confused anyone.

I reconsidered including those flags because I didn't find a speed improvement on x64 (Zen2-based Ryzen here), but these optimizations might perhaps make searching for problems more difficult (no errno, no traps).

So if these options don't gain speed, but may make detecting problems (which may arise on yet-untested platforms) more difficult, I felt they were superfluous.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-03-27 15:46:28
Current git x64 generic.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: sld on 2024-03-27 16:31:49
Thanks guys, anyone and everyone who contributed to this project's revival. The encoding speed is phenomenal.

For the frequencies above 16 kHz, though they may largely be lowpassed out by default, wouldn't the lower frequencies still generate some harmonics in that range during playback?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Sunhillow on 2024-03-27 19:53:47
Quote from: sld on
wouldn't the lower frequencies still generate some harmonics in that range during playback?

They shouldn't be powerful enough to replace the cutoff! Even if the distortions in your playback chain are higher than normal  ;)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-27 20:01:28
Quote from: Wombat on
Current git x64 generic.

That's what I was afraid of. As soon as the changes began to be committed to Github, enthusiasts started publishing all sorts of binaries daily without explaining what's new. Like dig that out of the nerdy depths of the repository yourself. @Case's approach was more user-oriented: he attached own binaries here, describing the changes they contained, then processed the feedback. Nothing remained shrouded in darkness, and now the darkness is thickening again.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-27 20:08:27
Do you bother to check the github before you make these comments? Nothing has changed quality-wise.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-03-27 20:14:59
Quote from: Kraeved on
That's what I was afraid of. As soon as the changes began to be committed to Github, enthusiasts started publishing all sorts of binaries daily without explaining what's new. Dig that out of the nerdy depths of the repository yourself. @Case's approach was more user-oriented: he described changes, attached binaries and processed the feedback here.
Nobody forces you to download. Besides that above i mentioned the changes to makefile.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-27 21:58:04
Quote from: john33 on
Do you bother to check the github before you make these comments? Nothing has changed quality-wise.
(https://i3.imageban.ru/out/2024/03/28/58a3a8863cfeef2fa8068335f65d450a.png)

What is it? @john33, are you ready to put your reputation on the line and walk out the window if this change does affect the quality to speak about it so confidently? Or will we let the author of the edit have his say? Helix's legacy is already fraught with obscure code and mysterious options such as -TX and -SBT, so it is not surprising there is increased attention to changes, especially unannounced.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-28 05:48:04
These changes are only style- and safety-changes.

The memset for spd_cnt1Short shouldn't change runtime-behavior at all, as both spd_cnt1Short and spd_cnt have the same size, thus memset doing the same. This is only fixing a copy&paste oversight, and I fixed this to avoid confusion.

The memset for w_spdShort is more serious, as w_spd is a *bigger* buffer than w_spdShort and thus the memset would write zeroes beyond the boundaries of the w_spdShort-buffer. However, if the encoder doesn't crash during initialization, no effect otherwise is necessarily expected.

I checked before and after the changes, and the MP3 output remained identical.

As for providing fresh builds for whatever current git-state: Well, if several people keep their build-environment fired up, even if the changes are minimal, it is beneficial to know that no breaking changes were introduced by accident. I can only test on Linux and with whatever compiler version I have installed, so having another set of compiles available can be useful to gauge whether some recent work broke things.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-28 09:25:54
As I understand it, the latest version with reasonable and properly tested changes that can run on both x86 and x64 is the @Case's build 2024-03-20 (https://hydrogenaud.io/index.php?action=dlattach;topic=123331.0;attach=29813). After that, you went into micro-optimization, changing the makefile and now the memsets. This has already affected the size of mp3 files, so users may have a legitimate question: if the encoding algorithm is not changed, then why mp3 files are different from each other? We need to create a stable version so that users don't wade through dozens of pages, trying out every attachment, as it happens in some other forum threads.

Code: [Select]
bytes, xxhash, filename
270150 = 71f16bfc93121db3 = out-i686.mp3
270154 = 7658cb1d84627d25 = out-i586.mp3
270227 = dad23bd189aeb6c6 = out-git-fb6f2b2.mp3
270227 = dad23bd189aeb6c6 = out-i686v2.mp3
270230 = 284a80c89cdf720b = out-case.mp3
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-28 10:02:20
The Helix MP3 Encoder is using floating point arithmetic (and LAME does, too). It's perfectly expected that there are slight variations in output depending on compiler version, compiler flags and platform. Rearranging floating point operations (that should be mathematically equal) can result in small (and usually inconsequential) differences in floating point values.

I don't expect hmp3 to generate identical files for Visual Studio, GCC 10.x or GCC 13.x, let alone if different instruction sets are involved.

i586 uses the x87 instruction set for floating point.
i686 uses the SSE instructions for floating point.
The Visual Studio compiles with ASM use a mixture of x87 instructions and SSE.
The Win64 x86_64-builds can use SSE(2)
I think Case's builds were AVX-enabled.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-03-28 15:05:40
The compilles i did with GCC 13.2.0 should create the same files in size.
It is only when i use AVX or fast-math that includes -funsafe-math-optimizations the output changes.
The rarewares x64 compile for example creates the same files as when i use fast-math.
I attach a compile that has both versions with exactly the same settings but fast-math if someone wants to check.
The question is if such optimization at some point changes the sound. I am no expert here but think to remember there once was a problem with a lame compile because of some compiler option.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-28 15:16:52
Quote from: Wombat on
The compilles i did with GCC 13.2.0 should create the same files in size.
It is only when i use AVX or fast-math that includes -funsafe-math-optimizations the output changes.
The rarewares x64 compile for example creates the same files as when i use fast-math.
I attach a compile that has both versions with exactly the same settings but fast-math if someone wants to check.
The question is if such optimization at some point changes the sound. I am no expert here but think to remember there once was a problem with a lame compile because of some compiler option.
I do indeed have fast-math enabled on both x64 compiles. I too recall a lame issue, I think with a particular Intel compiler option, but I don't recall which option, and it was in any case some many years ago.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-03-28 15:31:55
Quote from: john33 on
I do indeed have fast-math enabled on both x64 compiles. I too recall a lame issue, I think with a particular Intel compiler option, but I don't recall which option, and it was in any case some many years ago.
Thanks John. Maybe it was even only a bug in some compiler version.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-28 16:04:03
Dear developers, I listen to your every word with great attention and I am taken aback. Why do you use compilation flags which impact on quality you find difficult to assess? Since when did chasing fractions of seconds become more important than quality? We have an ancient code here that needs to be dusted off first, whereas you are in a hurry to attach wings to it so that it flies even faster, although it is already faster than LAME anyway. If there is even the slightest chance that the quality could deteriorate in at least one frame by at least a bit, such compilation flag should be omitted. Aren't there other projects that can be turned into a proving ground? Helix MP3 encoder should be approached with care, in the spirit of restoration, as the title of the topic suggests.

(https://i6.imageban.ru/out/2024/03/28/03568bfbdad963de67e519cc6e5825a4.webp)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-03-28 16:28:06
Quote from: Kraeved on
Dear developers, I listen to your every word with great attention and I am taken aback. Why do you use compilation flags which impact on quality you find difficult to assess? Since when did chasing fractions of seconds become more important than quality? We have an ancient code here that needs to be dusted off first, whereas you are in a hurry to attach wings to it so that it flies even faster, although it is already faster than LAME anyway. If there is even the slightest chance that the quality could deteriorate in at least one frame by at least a bit, such compilation flag should be omitted. Aren't there other projects that can be turned into a proving ground? Helix MP3 encoder should be approached with care, in the spirit of restoration, as the title of the topic suggests.
It isn't that easy. @maikmerten already explained that compilers of different origin may change the output more as setting flags inside one and the same enviroment.
We need golden eared peoplle like you to find out if there is a way to tell them apart.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-28 16:37:28
Quote from: Kraeved on
We have an ancient code here that needs to be dusted off first, whereas you are in a hurry to attach wings to it so that it flies even faster, although it is already faster than LAME anyway.

The changes done so far are exactly that: Dusting off the codebase by fixing things that are clearly bugs (uninitialized variables, wrong buffer initialization). Such bugs can very much lead to unexpected crashes or distortions.

Nobody here is messing with the core algorithms. The compiler flag explorations are in some way a means to check that the codebase behaves reliably and predictably across platforms. The default settings always will tend towards the safe and reliable, but its good to know that even aggressive compiler optimization don't outright explode.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2024-03-28 16:52:49
Quote from: Kraeved on
compilation flags which impact on quality you find difficult to assess?
Nothing says they are any wrong. Nothing says that "no compilation flags" is any more correct than "these compilation flags".

16 bits is enough if done properly.
There is a common misconception that a floating-point number is "inaccurate". A floating-point number is a sequence of 32 given bits, but 32-bit floating-point numbers are adequate for when that is accurate enough - whether that is because the actual resolution is less and they are exact, or whether it means you have to perform an "acceptable" roundoff before fitting it into the 32 bits. (Like, try 1/3.) 
Floating-point arithmetic allow operations that might require round-offs for the result to fit a 32-bit number. Maybe this direction, maybe that direction - it is adequate when it is good enough.
And if it is good enough, than this CPU's platform's roundoff is good enough, and so is that platform's roundoff - even if the end-results are rounded off to different 32-bit floating-point numbers ...

... which, in themselves, are approximations of something that in this application is an approximation of your original signal.
Even if you dropped the latter "an approximation of", it would be good enough for audio processing (DAWs can do well with 32-bit floating-point!), and now it is even used to make a lossy file, which is a "manthematically very rough but audibly good enough" approximation to the original signal.

If you encode it on your computer and play it back on your phone, the different CPUs might differ in roundoffs in precisely the same way.


And the impact is much smaller than the lossiness of the encoder - unless a particular implementation is stupid, and if so is the case, you don't even know that "your" compiler flag set is not the stupid one.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-28 17:51:35
Quote from: maikmerten on
Nobody here is messing with the core algorithms. The default settings always will tend towards the safe and reliable.

I hope so. It is important to agree that in the case of Helix, despite the digressions to try this and that, a conservative compilation strategy is prioritized so as not to damage the explicit and implicit potential of this encoder. After all, what is Helix? It is the pinnacle of the engineering thought of its time, the product of an (almost) lost school of offline-first, non-bloated, good enough to survive decades without Patch Tuesday (https://en.wikipedia.org/wiki/Patch_Tuesday) software, just as the pigment recipes (https://heritagesciencejournal.springeropen.com/articles/10.1186/s40494-016-0070-9) with which medieval artists created their masterpieces were (almost) lost.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Markuza97 on 2024-03-28 22:55:11
Quote from: Kraeved on
potential of this encoder

What potential? Nobody is going to use this.
LAME's last quality update was over 10 years ago. Developers squeezed everything they could from MP3.
It is mature and stable encoder, tested by many experienced members here.
If you believe people will suddenly change to Helix you are wrong.

All I can see in this thread is developers wasting their time on something useless only to get yelled at in return.

Quote from: Wombat on
We need golden eared peoplle like you to find out if there is a way to tell them apart.

lol'd :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: vina54 on 2024-03-29 00:44:59
Who knew some random MP3 encoder could make someone beyond passionate...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-29 07:10:22
Quote from: Markuza97 on
LAME's last quality update was over 10 years ago. Developers squeezed everything they could from MP3.

Yeah, it's very unlikely that significant quality improvement can be squeezed out of MP3 by either LAME or Helix.

Quote from: Markuza97 on
It is mature and stable encoder, tested by many experienced members here.

Too bad the latest LAME versions apparently regressed here and there. Always nice to have another option.

Quote from: Markuza97 on
If you believe people will suddenly change to Helix you are wrong.

All I can see in this thread is developers wasting their time on something useless only to get yelled at in return.

I think your view on what motivates people to tinker with code might be a bit too focused on the number of users. I can't speak for anyone else in this thread, but for me, it's been fun to speculate about parameters, tinker with compiler flags and reviewing and applying patches.

I hope that you, too, have a fun project, no matter its usefulness or size of its user base.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-29 08:29:31
Quote from: maikmerten on
I hope that you, too, have a fun project, no matter its usefulness or size of its user base.
Amen to that, it's what's kept me sane in the last 20years! ;)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: fooball on 2024-03-29 10:14:37
Quote from: Markuza97 on
All I can see in this thread is developers wasting their time on something useless...
I'm sure most people have some kind of hobby which would be seen as a waste of time by others.  Nonetheless, sometimes hobbies turn into businesses, or knowledge/skills gained becomes applicable elsewhere.

Quote from: Markuza97 on
...only to get yelled at in return.
This thread is (or has become) a discussion place for those interested in this project, and I personally have been monitoring it with little practical interest but as a curious bystander.  Anyone with no practical involvement has no business to criticise, and should keep out of the conversation (as I have... until now!).
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-03-29 12:20:35
I took the liberty of writing something regarding the -TX parameter over at https://wiki.hydrogenaud.io/index.php?title=Helix_MP3_Encoder#Magical_Mystery_Switch_-TX
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Markuza97 on 2024-03-29 14:33:33
There was a misunderstanding.
I meant no disrespect to developers - totally opposite.

By useless I meant exactly what you guys said above - small fun side project with little to no reward that will be used only by small amount of people.
Poor word choice from my side.

My post was directed to Kraeved. I got mad with this post.
Quote from: Kraeved on
What is it? @john33, are you ready to put your reputation on the line and walk out the window if this change does affect the quality...
Quote from: Markuza97 on
only to get yelled at in return.

Instead of being happy that couple of developers are using their precious time to work on small project like this you are talking about John's reputation.
Member with rock solid reputation whose encoders have been pretty much standard for years.

Again, no disrespect to actual developers. I just don't like when "acting boss" tells actuals developers what to do and not to do.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-29 16:51:50
@Markuza97 People gather for a variety of reasons: from weddings and funerals to hobby groups like reenactors and submissives. To understand why, in the 21st century, against the backdrop of so-called progress, someone builds a castle (https://youtu.be/watch?v=Ajqort8ldXA) without the use of modern tools or voluntarily surrenders to the will of a dominant partner, it is more far-sighted to show the humility of an indigenous researcher and unobtrusively ask questions, rather than heating up the situation with frantic cries like “who needs these remnants of the past when the world is on the verge of exploring Mars”. The latter reveals elementary communication skills, close to those of a spoiled child, who considers his limited horizons to be certain and the incomprehensible to be ridiculous and outrageous. By regularly pouring arrogance out of the cup of consciousness, you are more likely to fill the vacated space with the idea that life is a symphony of diverse forces. In particular, what we have here is not and will not be a hierarchy with profanum vulgus (mob) that begs Prometheus (developers) to share the fire (updates), but a horizontal exchange of opinions between mature enthusiasts with different priorities, which sometimes includes nitpicking and assertive pushbacks (as in the case of me and John), but ultimately it makes Helix MP3 encoder better. Focus on that, and you will feel better as well.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-03-29 18:03:03
Do you genuinely have no idea how arrogant and patronising your manner is? It's actually quite breathtaking. You seem to think that it is OK to behave that way and at the same time insult in a quite creepy manner. Why say what needs to be said in a few words when you can take several paragraphs and spew BS along the way? "nitpicking and assertive pushbacks" you say. I found it quite offensive, to be honest.

I, for one, have had enough of you and will not again respond to any further comment that you post on this forum.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-29 18:39:47
@john33 Sorry to see that you don't notice your own nagging manner, which I've been tolerating for a long time. In particular, you reproached me for sharing the test results, and another developer had to explain to you that these results were acceptable (https://hydrogenaud.io/index.php/topic,123331.msg1041680.html#msg1041680). Then you stated that I had not checked Github before commenting and there were no quality-related commits in the last build, after which I had to post a screenshot of the very commit, and the same developer had to explain that it was serious (https://hydrogenaud.io/index.php/topic,123331.msg1041905.html#msg1041905), although tested. And now that BS word as an assessment. So who is patronizing here? You are not involved in development, testing, writing documentation, or thinking about the future of the project, so why are you so nervous about someone who does? Rewind a few pages and look at the respectful interaction (https://hydrogenaud.io/index.php/topic,123331.msg1040083.html#msg1040083) between me and @Case, then ask yourself why it didn't work out between us, because I did my best, starting with a list of your merits (https://hydrogenaud.io/index.php/topic,123331.msg1041562.html#msg1041562).
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-03-29 21:07:21
Quote from: maikmerten on
I took the liberty of writing something regarding the -TX parameter over at https://wiki.hydrogenaud.io/index.php?title=Helix_MP3_Encoder#Magical_Mystery_Switch_-TX

Thank you, @maikmerten.

a) Wiki says that -TX6 is used by default, but -H screen is missing the word default next to TX6 and -EC screen shows ec->test1 = 0 instead of 6 when no number is specified and -1 instead of 6 when -TX is not used at all. I believe the word should be added, but what do to with the number?

b) The last TX setting (eighth in a row) has no specific number on -H screen, it just says “v5.1  2005.08.09 (see CVS log for details)”. Have you found anything interesting there?

c) When Helix is called with -TX# -EC, we see “ec->test1 = #”. It accepts 0, positive and negative integers with no validation. Since we'll probably never know the essence of the experiments of the previous team associated with this flag and the default setting (-TX6) is recommended, maybe it makes sense to put it at the bottom of -H screen within a separate section like “--- Mysterious legacy (Read wiki for details) ---” to avoid potential questions from users?

Also, I noticed that you told the user (https://hydrogenaud.io/index.php/topic,123331.msg1041764.html#msg1041764) which Helix settings (-V115 -HF2 -F19000) might resemble LAME -V2. Let me clarify, which transition band does Helix use in this case? For LAME 3.99.5o and 3.101 beta 3, it is 18000 Hz - 18581 Hz.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-03 13:20:46
My guess is that the -TX flag triggered different experimental settings at different times. Right now it only influences one function, as documented in the Wiki. The help screen is not actually clarifying what's going on there. Users that feel adventurous can tweak as they like and perhaps find cases where it makes a difference.

I'd hope that perhaps someone more knowledgeable in MP3 psychoacoustics may actually know what that particular function is doing, so we can clarify what is happening there.

The transition band is from the LAME page of the very same Wiki.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-03 20:55:10
Quote from: maikmerten on
Regarding -HF2, I'm not very worried. There are listening tests with -HF2 that show that Helix is doing plenty fine.

I've seen the results of 3 listening tests: -V60 (2008 (https://hydrogenaud.io/index.php/topic,113324.msg1041255.html#msg1041255)), then -V146 (2013 (https://hydrogenaud.io/index.php/topic,100896.0.html)) and -V111 -HF2 (2016 (https://hydrogenaud.io/index.php/topic,113324.0.html)) by @Kamedo2. In each case, Helix receives rave reviews. But nowhere is the choice of encoding options explained. Or rather, the use of -HF2 is explained by an attempt to resemble LAME behavior. And most importantly, I haven't seen a direct comparison between the regular mode and -HF2 yet. @Destroid shared an observation (https://hydrogenaud.io/index.php/topic,100896.msg834601.html#msg834601) about the slight difference between -HF1 and -HF2.

Personally, listening to Helix's encoded music in the background (rather than focused) over the course of this spring 2024, I still can't think of a sample where the difference would be dramatic, although the increase in size is noticeable. Even a sample of harpsichord, kindly shared by @Porcus, sounds fine. Perhaps another listening test with more sound lovers involved would help to bring out the advantage of -HF2, or there would be a conclusion that its advantage is minimal and it rather distracts this robust but lossy encoder from clearly audible content (https://wiki.hydrogenaud.io/index.php?title=High-frequency_content_in_MP3s).
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: synclagz on 2024-04-04 12:25:45
Quote from: Kraeved on

Perhaps another listening test with more sound lovers involved would help to bring out the advantage of -HF2, or there would be a conclusion that its advantage is minimal and it rather distracts this robust but lossy encoder from clearly audible content (https://wiki.hydrogenaud.io/index.php?title=High-frequency_content_in_MP3s).

Perhaps you could do your own ABX test to see if you can hear the difference between HF1 and HF2 setting. This way you can eliminate any dilemma you might have. After all, personal preference is what matters the most.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-04 17:41:05
Quote from: synclagz on
Perhaps you could do your own ABX test to see if you can hear the difference… This way you can eliminate any dilemma you might have. After all, personal preference is what matters the most.
It's not about my dilemma, but about a better future where more users get the most out of the Helix MP3 encoder.

Not only do I use this tool myself, but I also generously share its results and evangelize its use among comrades who dedicated their lives to other professions and hobbies. "Kraeved, our neighbors say that you are privy to sound tools, whereas I work as a truck driver by day and knit funny socks for less fortunate by night, could you tell me how best to convert that eye-opening book by Eckhart Tolle from FLAC to use with my easily distorted car player?” Therefore, the criterion of personal satisfaction is not sufficient to justify the recommendation here. We must rely on a stronger evidence not to indulge in wishful thinking and to guide the masses in a more responsible way.

Such evidence grows as dear enthusiasts, armed with a wider range of audio samples, equipment and innate features than I can only dream of, accept the challenge of taming Helix's psychoacoustic model and identifying cases where approximate copies of the original are strikingly different from each other due to the presence of higher frequencies.

The community and its tools become more reliable when the pillars rest on the solid ground of consensus rather than on the postmodern relativism with confusing and vacillating individual pronouns truths. I am thrilled to be a part of such community, but too humble to believe that my limited experience is enough to proceed. That's why I urge us, at least those of us who like challenges, to pay more attention to comparing the encoding with and without -HF2 flag.

The road to a better future may not be smooth, but it is within our reach.

(https://i5.imageban.ru/out/2024/04/04/883ab80ddf7d508012df919bf7455d62.jpg)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-04-04 18:32:24
That has to be one of the weakest and most asinine responses I have ever seen! You suffer from verbal diarrhoea and need treatment.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: shadowking on 2024-04-07 05:02:56
HF1 vs 2 likely don't matter . The whole thing don't matter as scalefactor band 21 isn't calculated. Typically you go for 16khz @ 128 k cbr.  From 160k set lowpass/HF to highest level you want up to 20khz as long as something like -Y is enabled in vbr or using cbr/abr encoding when it happens by design.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-07 18:10:42
Oh, HF1 and HF2 makes a difference, and I think I finally figured out what it is. According to the help function, HF1 enables high frequencies for "mode-1 granules", while HF2 enables high frequencies for "all granules".

It took me a while to figure out what "mode-1 granules" are: This means "M/S stereo" ("joint stereo") granules.

With HF1, *only* parts that are coded in M/S stereo can also carry high frequency content. Of course, the encoder will dynamically switch between stereo ("mode 0") and M/S stereo ("mode 1") depending on what is more efficient. Very roughly estimated, about 80% of frames are M/S-coded, while about 20% of frames are stereo-coded.

In other words, in a typical scenario, about 20% of frames don't have high frequency content with HF1. I can imagine (but I personally cannot hear this, so this is speculation) that some persons might be able to perceive the high-frequency "dropouts" with HF1.

Thus, personally, I'd go with HF2.

Why have HF1 in the first place? I speculate that this mostly (only?) makes sense in CBR encoding. If M/S-stereo is less efficient than stereo coding, then usually the two channels don't correlate much. Thus, more bits are needed to encode the stereo signal in the first place. In this scenario, when bits are limited, it may be best to have as many bits as possible for frequencies below 16 kHz by omitting high frequency content.

In VBR encoding, though, one usually can just allocate a few more bits without starving the more relevant (lower) frequency bands.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: ha7pro on 2024-04-10 05:54:37
Hello, everyone
My English is not good and my expression may not be accurate enough, please forgive me.
Longer audio (for example, more than 3 or 4 hours) encoded by hmp3 will have a high probability of playback failure. The audio file can display the correct duration in mediainfo, but in players such as foobar2000 and mpc, the audio Only the first part of the duration can be recognized and played normally, and the extra long part cannot be recognized and played by the player. This problem exists in multiple versions up to the latest, please check it out.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 06:42:05
Thanks for the problem report. Note that you can repair existing files with foobar2000 using the 'Utilities' -> 'Fix VBR MP3 header...' feature.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: 145dBSPL on 2024-04-10 08:56:01
Quote from: maikmerten on
Oh, HF1 and HF2 makes a difference, and I think I finally figured out what it is.
[...]
With HF1, *only* parts that are coded in M/S stereo can also carry high frequency content. Of course, the encoder will dynamically switch between stereo ("mode 0") and M/S stereo ("mode 1") depending on what is more efficient. Very roughly estimated, about 80% of frames are M/S-coded, while about 20% of frames are stereo-coded.

This is an interesting insight. Thank you for your research. To visualize this, I created and encoded a pink noise file.
Settings used are -V110 -HF[1|2]. For both encoded files, an mp3 analysis shows 18.0 % Simple stereo frame
and 82.0 % Mid-side stereo frames.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: ha7pro on 2024-04-10 08:58:27
Thanks
foobar2000 'Utilities' -> 'Fix VBR MP3 header...'

It solved the problem.
But
Why does "cbr" encoded mp3 require "vbr header fix"?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 12:31:25
Quote from: ha7pro on
Why does "cbr" encoded mp3 require "vbr header fix"?
The info header at the beginning of MP3 was originally only needed and used for VBR files, otherwise players couldn't even know a track length without parsing throught the entire track. I guess foobar2000 could update its texts.

Attached is a fix for the issue ha7pro reported, both source patch and 32-bit win32 asm build. I hope I didn't break anything, but I trust @Kraeved will quickly figure out if I did.
The problem was related to input exceeding 4 GB, 32 bit variables wrapped over causing all sort of havoc. Fixed that by changing related variables to unsigned 64-bit integers.
I added an extra command line parameter -il (for ignore length) to allow ignoring source length. That way if you for example use foobar2000 to create a huge temporary WAV file it can be encoded fully without input size getting clipped to 4 GB. Ideally this should only be needed with pipe encoding, but I have made the encoder turn this mode on automatically with pipe input.

Since the encoder now supports over 4 GB input I added support for Wave64 file format. The support was only tested with basic w64 file created with foobar2000, so the support may be bugged with edge cases.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: ha7pro on 2024-04-10 13:15:36
it works
everything is ok
thanks a lot
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2024-04-10 13:18:22
Not saying this is all-important, but RF64 is a "more trivial" format once WAVE is supported?
The ITU are trying to kill it off and replace it by BW64 (with a "BW64" fourcc and discouraging WAVEFORMATEXTENSIBLE channel mask ... but is it still possible to know that a file is mono or stereo?)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-10 13:43:42
@Case, thanks a lot. I applied these patches to the GitHub version. The changes are currently living in this branch: https://github.com/maikmerten/hmp3/tree/case-64bit-fixes

On Linux, the progress indicator isn't updating on the same line anymore. I'll look into fixing this.

edit: This is caused by the line length exceeding the default terminal line width, thus the \r control working on the next line. I'll try to format the output a bit.

edit2: Fixed in https://github.com/maikmerten/hmp3/commit/3a9d3e2c9c785b97bd922d97a02441f9dec3a29c

btw, just in case anyone is wondering why I keep updating the date of last change and editor in the file header, that's caused by a term in the RPSL license...

Quote
(c) You must duplicate, to the extent it does not already exist, the notice in
Exhibit A in each file of the Source Code of all Your Modifications, and cause
the modified files to carry prominent notices stating that You changed the files
and the date of any change;
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 14:15:09
Quote from: Porcus on
Not saying this is all-important, but RF64 is a "more trivial" format once WAVE is supported?
The ITU are trying to kill it off and replace it by BW64 (with a "BW64" fourcc and discouraging WAVEFORMATEXTENSIBLE channel mask ... but is it still possible to know that a file is mono or stereo?)
Oh no, I was just coming to update my patch with RF64 support and this bit of info was waiting for me. Quick look at the specs suggests that it's not so different, so I added support for that BW64 fourcc to the RF64 support code. This addition was very last second, I don't have a clue if it works.

Quote from: maikmerten on
On Linux, the progress indicator isn't updating on the same line anymore. I'll look into fixing this. edit: This is caused by the line length exceeding the default terminal line width
Even more oh no. I forgot I had gotten annoyed at the default 80 characters not being enough for anything so all my terminals are much wider. No wonder I had so much extra space to increase the fields. 4 GB+ input required extending input byte counter from 10 characters to 20 characters. Sorry about the mess.

Attached updates with RF64 (and possibly BW64) input support.

Edit: horrible bug in the sources (should have tested with actual files).
The RF64 support code uses wrong field for data size, pcmhpm.c line 400 should read
Code: [Select]
*data_size = get_field64 ( chunk_ds64->dataSize, sizeof ( chunk_ds64->dataSize ), f_info->bigendian ); /* return actual audio data size to the frontend */
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-04-10 14:21:30
Was just about to update Rarewares!! ;) I'll wait a short time. :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 14:31:06
I updated my post with a fix to the RF64 reading. Unfortunately my Audition is too old and can't write BW64 to create a test file.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-10 14:55:17
Okay, I nuked the branch with the older patch and now pushed the current patch again to https://github.com/maikmerten/hmp3/tree/case-64bit-fixes

This also seems to include your line 400 fix in pcmhpm.c, as far as I can tell.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 14:59:38
It does. I was pleasantly surprised to see that a forum problem that prevented me from removing attachments had been fixed.
Unfortunately this still includes the printing problem that you had fixed.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-10 15:03:14
Oh, I took the liberty of taking away a few dashes in the output, so things are super-okay in the terminal.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2024-04-10 17:31:41
On BW64,
Can one use https://github.com/ebu/libbw64 ?
Like more of us, I was unaware this format existed until @bennetng posted some files made by Reaper in https://hydrogenaud.io/index.php/topic,123862.25.html reply #25. There is/was some misinformation around including Wikipedia.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 18:14:12
The RF64 and BW64 files from his archive encode fine with hmp3. But the included reaper-wave64.wav file doesn't. I don't understand what I'm missing. The fmt chunk starts at offset 40, it reports its size as 42. That means next chunk should start at offset 82. There's just zeroes there, the next real chunk is data chunk that starts at offset 88.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: bennetng on 2024-04-10 18:26:53
Quote from: Case on
reaper-wave64.wav
Quote from: MonkeysAudio on
reaper-wave64.wav is actually a W64 file.  If I use that encoder it processes fine.
So, just a misleading file extension.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 18:31:54
I know it's Wave64. It just doesn't seem to follow the specs I have. I'm using the 'Sony_Wave64.pdf' bryant linked not that long ago. The fmt chunk size is off by 6 bytes and I don't know why.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 18:46:48
Excuse my silliness. It was of course the 8 byte alignment.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 19:08:45
The bitwise AND for checking if sizes are divisible by 8 didn't work with the 64-bit integer, that's why Porcus' linked file didn't work. There was also a problem with my sanity checks to abort parsing if data doesn't make sense. The give-up-size was treated as signed and it never triggered. Those issues are fixed here.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-10 19:42:20
a) Source: bwf.wav. WAV (including RF64) is not encoded if (malformed? incorrectly padded?) bext chunk of BWF (https://en.wikipedia.org/wiki/Broadcast_Wave_Format) extension is present. Error: “UNRECOGNIZED PCM FILE TYPE”.

b) Source: gudki.wav. Differences in the number and bitrate of frames between builds.

Code: [Select]
$ hmp3johnx64.exe gudki.wav gudki.john.mp3
--------------------------------------------------------------------------------------
      Frames  |            Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
         625  |              1411200 /     104156 |   100%   |  43.13 /  51.03 Kbps
--------------------------------------------------------------------------------------
 Compress Ratio 7.380669%

$ mp3packer -i gudki.john.mp3
 Bitrate distribution:
    8: 13,0
   24: 1,0
   40: 40,0
   48: 256,0
   56: 266,0
   64: 49,0

$ hmp3case.exe gudki.wav gudki.case.mp3
--------------------------------------------------------------------------------------
      Frames  |            Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
         621  |              1411200 /     103974 |   100%   |  43.48 /  51.27 Kbps
--------------------------------------------------------------------------------------
 Compress Ratio 7.367772%

$ mp3packer -i gudki.case.mp3
 Bitrate distribution:
    8: 9,0
   24: 1,0
   32: 1,0
   40: 42,0
   48: 250,0
   56: 271,0
   64: 47,0
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-10 20:30:10
Thank you. Problem with BWF file was basically the same as with Porcus' W64 sample. The WAV reader had no support for WAV chunk alignment.
The other difference you see is related to different compilers doing the math differently so frames get allocated differently. If you look at the files in foobar2000 you will see that the decoded frame length is identical.

Fixed original WAV reader to take alignment into account. I didn't see a single mention of alignment in RF64 specs, I assume it uses the same WAV format's 2-byte alignment.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-10 23:38:48
Quote from: Case on
I hope I didn't break anything, but I trust @Kraeved will quickly figure out if I did.

To make sure Helix MP3 encoder would process a large input as expected, I took BWV 1034 Allegro (https://www.lessloss.com/bach-sonatas-in-st-martins-church-p-197.html) (44.1 kHz 16-bit) and looped it N times using FFMPEG (https://github.com/nanake/ffmpeg-tinderbox/releases/) to get 5 GB W64¹ and RF64², then played with sampling rates³ and bit depths⁴, added various metadata chunks. In each case, I checked the encoding as is and via pipe.

¹ ffmpeg -v error -stream_loop 99 -i allegro.wav -f w64 - | hmp3 - out.mp3
² -f wav -rf64 auto
³ -af aresample=22050:resampler=soxr:precision=28
⁴ -acodec pcm_f32le

So far so good. However, I was confused by two things.

a) The new -IL flag must be used if you want to encode a large input in its entirety, but there is no warning on the screen if the file is large and the flag is not used. Perhaps the encoder should activate it automatically if the file is large enough? Otherwise, you're surprised to find that MP3 is only, say, 3 h 22 min out of (source) 3 h 56 min long.

b) The encoding speed dropped at times (watch animation (https://img.imgdd.com/f210f3.36319ded-5819-4913-bc41-5c5484257f9c.webp)), the cause of which I haven't yet determined (overheating, writing to the end of a clogged disc, or something else), but if anyone else encounters this, know that you're not alone.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Markuza97 on 2024-04-10 23:52:22
Quote from: Kraeved on
... and looped it N times using FFMPEG (https://github.com/nanake/ffmpeg-tinderbox/releases/) to get 5 GB¹, gradually changing the sampling rate, bit depth, adding various metadata chunks and using W64 & RF64² formats. In each case, I checked the encoding as is and via pipe. So far so good.

So ffmpeg is finally fixed then? I remember having a problem with huge files two years ago.

Quote from: Markuza97 on
Sorry for hijacking your thread.

I have ~150 minute, 6 channel, 32-bit float, 48 kHz RF64 WAV file created by ffmpeg.

When I try to compress it using WavPack I get "xxx.wav is not a valid .WAV file!".
When I try to compress it using ffmpeg I get "100 buffers queued in out_0_0, something may be wrong."
When I try to play it in foobar2000 it says duration is over 1 day long.

Same file but this time exported by Audacity.

WavPack loves it.
ffmpeg same warning.
Duration in foobar2000 is correct.

ffmpeg bug?

Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-11 06:40:11
@Case I extracted your Wav header alignment fixes from the previous patch with interdiff and pushed them onto the case-64bit-fixes branch: https://github.com/maikmerten/hmp3/commit/9c6c125805445a7857dbe7255dc01791e4be2a76
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-11 06:41:22
Quote from: Kraeved on
a) The new -IL flag must be used if you want to encode a large input in its entirety, but there is no warning on the screen if the file is large and the flag is not used.
That isn't quite correct. When you are encoding from a pipe, that mode is automatically enabled because input length can't be trusted. The newly supported Wave64, RF64, BW64 input formats aren't limited to 4 GB sizes, the ignore length option should not be used with these. It could only be useful in a very special circumstances where you write an invalid regular WAV file that's over 4 GB long and want to encode that without using pipes.
The encoder can't just blindly encode beyond the length reported in WAV file headers. You even provided a sample file with extra chunks after the data section which simple encoders encoded as audio. With pipes it is pretty safe to assume that we are receiving just audio data. But if you are reading from a file the ending can have all sorts of metadata that isn't actually audio.

We could add a warning message when encoding from a WAV file that has a header indicating maximum supported size.

Quote from: Kraeved on
b) The encoding speed dropped at times (watch animation (https://img.imgdd.com/f210f3.36319ded-5819-4913-bc41-5c5484257f9c.webp)), the cause of which I haven't yet determined (overheating, writing to the end of a clogged disc, or something else), but if anyone else encounters this, know that you're not alone.
If your drives are regular hard drives and they are close to being full or very heavily fragmented it could be from slow seeking.
Your allegro loop test produced a 252 MB mp3 file here and even though the test drive I used shouldn't be that fragmented, it was still split in 44 different fragments.
In Windows fragmentation could be reduced by using a large write buffer and possibly eliminated entirely by telling the file system the file sizes in advance.
But this is just speculation. I haven't seen slowdowns. You'd need to use some system monitoring tools while encoding to see what hits its limits. The Task Manager in new Windows would be usable as it even shows disk utilizations. With older Windows perfmon can be used.

Quote from: maikmerten on
@Case I extracted your Wav header alignment fixes from the previous patch with interdiff and pushed them onto the case-64bit-fixes branch: https://github.com/maikmerten/hmp3/commit/9c6c125805445a7857dbe7255dc01791e4be2a76
Ah, that's great!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-11 11:20:28
Quote from: Case on
The newly supported Wave64, RF64, BW64 input formats aren't limited to 4 GB sizes, the ignore length option should not be used with these. It could only be useful in a very special circumstances where you write an invalid regular WAV file that's over 4 GB long and want to encode that without using pipes.

Hmm, I get 3 h 22 min MP3 without -IL, whereas the source WAV RF64 is 3 h 56 min.
Does it mean that FFMPEG writes invalid WAV RF64 here?

Code: [Select]
$ mediainfo allegro.wav
General
Complete name                            : allegro.wav
Format                                   : Wave
Format settings                          : PcmWaveformat
File size                                : 47.7 MiB
Duration                                 : 4 min 43 s
Overall bit rate mode                    : Constant
Overall bit rate                         : 1 411 kb/s
Producer                                 : TASCAM HD-P2
Description                              : tTAPE=090228
Encoded date                             : 2009-02-28 21:21:29

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 4 min 43 s
Bit rate mode                            : Constant
Bit rate                                 : 1 411.2 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 44.1 kHz
Bit depth                                : 16 bits
Stream size                              : 47.7 MiB (100%)

$ ffmpeg -stream_loop 49 -i allegro.wav -f wav -acodec pcm_f32le -rf64 auto -write_bext 1 out.wav

$ mediainfo out.wav
General
Complete name                            : out.wav
Format                                   : Wave
Format profile                           : RF64
Format settings                          : WaveFormatEx
File size                                : 4.66 GiB
Duration                                 : 3 h 56 min
Overall bit rate mode                    : Constant
Overall bit rate                         : 2 822 kb/s
Encoded by                               : TASCAM HD-P2
Recorded date                            : 2009-02-28
Encoded date                             :
Writing application                      : Lavf61.3.100
Comment                                  : tTAPE=090228

Audio
Format                                   : PCM
Format profile                           : Float
Codec ID                                 : 3
Codec ID/Hint                            : IEEE
Duration                                 : 3 h 56 min
Bit rate mode                            : Constant
Bit rate                                 : 2 822 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 44.1 kHz
Bit depth                                : 32 bits
Stream size                              : 4.66 GiB (100%)

$ hmp3case out.wav out.mp3
 pcm file:  channels = 2  bits = 32,  rate = 44100  type = 3
 Layer III   mode 1 STEREO   44100Hz  VBR-50
--------------------------------------------------------------------------------------
      Frames  |            Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
      466038  |           4294967296 /  196657861 |   100%   | 126.59 / 129.23 Kbps
--------------------------------------------------------------------------------------
 Compress Ratio 4.578798%

$ mediainfo out.mp3
General
Complete name                            : out.mp3
Format                                   : MPEG Audio
File size                                : 188 MiB
Duration                                 : 3 h 22 min
Overall bit rate mode                    : Variable
Overall bit rate                         : 129 kb/s
Writing library                          : LAMEH5.22

Audio
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 3
Format settings                          : Joint stereo / MS Stereo
Duration                                 : 3 h 22 min
Bit rate mode                            : Variable
Bit rate                                 : 128 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 44.1 kHz
Frame rate                               : 38.281 FPS (1152 SPF)
Compression mode                         : Lossy
Stream size                              : 188 MiB (100%)
Writing library                          : LAMEH5.22
Encoding settings                        : -m  -V 5 -q 0 -lowpass 15.8

$ hmp3case -IL out.wav out-il.mp3
 pcm file:  channels = 2  bits = 32,  rate = 44100  type = 3
 Layer III   mode 1 STEREO   44100Hz  VBR-50
--------------------------------------------------------------------------------------
      Frames  |            Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
      542882  |           5003191296 /  229035869 |   100%   |  63.72 / 129.20 Kbps
--------------------------------------------------------------------------------------
 Compress Ratio N/A

$ mediainfo out-il.mp3
General
Complete name                            : out-il.mp3
Format                                   : MPEG Audio
File size                                : 218 MiB
Duration                                 : 3 h 56 min
Overall bit rate mode                    : Variable
Overall bit rate                         : 129 kb/s
Writing library                          : LAMEH5.22

Audio
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 3
Format settings                          : Joint stereo / MS Stereo
Duration                                 : 3 h 56 min
Bit rate mode                            : Variable
Bit rate                                 : 128 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 44.1 kHz
Frame rate                               : 38.281 FPS (1152 SPF)
Compression mode                         : Lossy
Stream size                              : 218 MiB (100%)
Writing library                          : LAMEH5.22
Encoding settings                        : -m  -V 5 -q 0 -lowpass 15.8

However, W64 is encoded as expected without -IL, i.e. MP3 duration is the same as the source W64.

Code: [Select]
$ ffmpeg -stream_loop 49 -i allegro.wav -f w64 -acodec pcm_f32le out.w64

$ mediainfo out.w64
General
Complete name                            : out.w64
Format                                   : Wave64
File size                                : 4.66 GiB
Duration                                 : 3 h 56 min
Overall bit rate mode                    : Constant
Overall bit rate                         : 2 822 kb/s

Audio
Format                                   : PCM
Format profile                           : Float
Codec ID                                 : 3
Codec ID/Hint                            : IEEE
Duration                                 : 3 h 56 min
Bit rate mode                            : Constant
Bit rate                                 : 2 822 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 44.1 kHz
Bit depth                                : 32 bits
Stream size                              : 4.66 GiB (100%)

$ hmp3case out.w64 out-w64.mp3
 pcm file:  channels = 2  bits = 32,  rate = 44100  type = 3
 Layer III   mode 1 STEREO   44100Hz  VBR-50
--------------------------------------------------------------------------------------
      Frames  |            Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
      542882  |           5003160800 /  229035869 |   100%   |  63.72 / 129.20 Kbps
--------------------------------------------------------------------------------------
 Compress Ratio 4.577823%

$ mediainfo out-w64.mp3
General
Complete name                            : out-w64.mp3
Format                                   : MPEG Audio
File size                                : 218 MiB
Duration                                 : 3 h 56 min
Overall bit rate mode                    : Variable
Overall bit rate                         : 129 kb/s
Writing library                          : LAMEH5.22

Audio
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 3
Format settings                          : Joint stereo / MS Stereo
Duration                                 : 3 h 56 min
Bit rate mode                            : Variable
Bit rate                                 : 128 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 44.1 kHz
Frame rate                               : 38.281 FPS (1152 SPF)
Compression mode                         : Lossy
Stream size                              : 218 MiB (100%)
Writing library                          : LAMEH5.22
Encoding settings                        : -m  -V 5 -q 0 -lowpass 15.8
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-11 12:05:47
You are good at finding bugs. This was caused by a an oversight after fixing the WAV padding issue.
Fixes attached.

Edit: Bah. I can't remove these obsolete old attachments. Newer set can be found in a later post.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: ha7pro on 2024-04-11 12:11:52
When hmp3 accepts piped input, the -IL parameter takes effect by default and does not need to be used explicitly.

The -IL parameter is not required when a correct Wave64 file is used as input.

When a WAV file exceeding 4G is used as input, an error will occur if the -IL parameter is not used explicitly.
This should be the logic
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-11 16:27:01
I wanted to avoid adding size checks because that's more platform specific code. But I added a check for this requested feature.

I also noticed that the new status printing looks horrible on Windows with default 80 characters wide console. I experimented with various different methods that showed the sizes in KB, MB, and GB as needed but I didn't like any of these. But I think the routine I implemented for this patch is good enough: it shows input bytes as bytes when the number has 13 digits or less, if it gets longer it will switch to showing it in floating point terabytes, petabytes or exabytes as needed.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-11 16:38:57
Thanks again, pushed that patch over to the case-64bit-fixes branch in this commit: https://github.com/maikmerten/hmp3/commit/1ded92c7e412296c9d0a10e1d0f2b7ef804e9920

I took the liberty of fixing this compiler warning by allocating a slightly bigger char buffer:

Code: [Select]
hmp3/src/test/tomp3.cpp:576:55: warning: ‘%3d’ directive writing between 3 and 10 bytes into a region of size 8 [-Wformat-overflow=]
  576 |         if ( progress >= 0 ) sprintf ( progress_str, "%3d%%", progress );
      |                                                       ^~~
hmp3/src/test/tomp3.cpp:576:54: note: directive argument in the range [0, 2147483647]
  576 |         if ( progress >= 0 ) sprintf ( progress_str, "%3d%%", progress );
      |                                                      ^~~~~~~
In file included from /usr/include/stdio.h:894,
                 from hmp3/src/test/tomp3.cpp:41:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:34: note: ‘__builtin___sprintf_chk’ output between 5 and 12 bytes into a destination of size 8
   38 |   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
      |          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   39 |                                   __glibc_objsize (__s), __fmt,
      |                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   40 |                                   __va_arg_pack ());
      |                                   ~~~~~~~~~~~~~~~~~

Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-13 13:06:10
Quote from: Case on
Quote from: Kraeved on
* Gapless. You mentioned this word several times in this thread, but what exactly do you mean?
In this context "gapless" mean that the encoded length is 100 % correct without silence added or samples removed. Perfect audibly gapless playback requires more work, if it's at all possible when working with separate tracks. I may test extrapolation filter to see if that improves things.

Last night I followed the white rabbit (https://youtu.be/watch?v=6IDT3MpSCKI) and got to the Mad Hatter's ayahuasca tea party. Among other odd things, he digitized vinyl in 768 kHz 64-bit float and then exported it for delivery in 384 kHz 32-bit integer. I rummaged among the records next to the smorgasbord and pulled out a few iconic ones, mixed entirely or almost without gaps, although accompanied by a cue sheet for convenient navigation. Let's see, or rather listen to how Helix MP3 encoder handles transitions between much-loved compositions using VBR mode (https://hydrogenaud.io/index.php/topic,125684.msg1042148.html#msg1042148).

Sources (IconicGapless.zip (https://reactfiles.com/Zun0inZKFQnSYYP/file)) were only resampled to 44.1 kHz via SoX DSP x64 (https://hydrogenaud.io/index.php/topic,67376.msg1014620.html#msg1014620) before encoding.

• 1990. Enigma — MCMXC A.D. (Vinyl (https://www.discogs.com/release/158285-Enigma-MCMXC-aD), 24/192)
Code: [Select]
ffmpeg -sseof -21.400 -i "01. The Voice Of Enigma.flac" -c:a pcm_s24le -map_metadata -1 -bitexact Enigma01.wav
ffmpeg -i "02. Principles Of Lust.flac" -t 23.400 -c:a pcm_s24le -map_metadata -1 -bitexact Enigma02.wav

• 1994. Vangelis — Blade Runner (Vinyl (https://www.discogs.com/release/7097828-Vangelis-Blade-Runner), 32/384)
Code: [Select]
ffmpeg -sseof -26.000 -i "A1 Main Titles.wv" -c:a pcm_s32le -map_metadata -1 -bitexact BladeRunner01.wav
ffmpeg -i "A2 Blush Response.wv" -t 23.600 -c:a pcm_s32le -map_metadata -1 -bitexact BladeRunner02.wav

• 2000. Hans Zimmer, Lisa Gerrard — Gladiator (CD (https://www.discogs.com/release/4144077-Hans-Zimmer-And-Lisa-Gerrard-Gladiator-Music-From-The-Motion-Picture), 16/44.1)
Code: [Select]
ffmpeg -sseof -23.900 -i "02. The Wheat.flac" -map_metadata -1 -bitexact Gladiator01.wav
ffmpeg -i "03. The Battle.flac" -t 30.000 -map_metadata -1 -bitexact Gladiator02.wav
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-13 23:00:17
Finally had time to take a listen. All transitions sounded perfect to me.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-14 08:44:20
@Case As far as I can tell, your recent round of fixes seems to work plenty great. Mind if I merge this to the main branch and bump the version to 5.2.3?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-14 08:58:10
Indeed seems like no further issues have been found. I definitely don't mind at all.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-14 09:30:01
Thanks! I merged your branch to main and bumped the version to 5.2.3.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-14 10:58:48
@maikmerten, I downloaded the portable C and C++ development kit (https://github.com/skeeto/w64devkit) for Windows along with the Helix MP3 encoder source code (https://github.com/maikmerten/hmp3), then typed make. As a result, I got hmp3.exe, but there are a few alarming lines in the compilation log. Is there anything we can do about it? There is a related thread (https://stackoverflow.com/questions/21323628/warning-cast-to-from-pointer-from-to-integer-of-different-size) on Stack Oveflow.

Code: [Select]
~ # cd hmp3
~/hmp3 # make
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/amodini2.o hmp3/src/amodini2.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/cnts.o hmp3/src/cnts.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/detect.o hmp3/src/detect.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/emap.o hmp3/src/emap.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/l3init.o hmp3/src/l3init.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/l3pack.o hmp3/src/l3pack.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/mhead.o hmp3/src/mhead.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/pcmhpm.o hmp3/src/pcmhpm.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/setup.o hmp3/src/setup.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/spdsmr.o hmp3/src/spdsmr.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/xhead.o hmp3/src/xhead.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/cnt.o hmp3/src/cnt.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/emdct.o hmp3/src/emdct.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/filter2.o hmp3/src/filter2.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/hwin.o hmp3/src/hwin.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/l3math.o hmp3/src/l3math.c

hmp3/src/l3math.c: In function 'align16':
hmp3/src/l3math.c:135:27: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
  135 |     y = ( float * ) ( ( ( ( int ) x ) + 15 ) & ( ~15 ) );
      |                           ^
hmp3/src/l3math.c:135:9: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  135 |     y = ( float * ) ( ( ( ( int ) x ) + 15 ) & ( ~15 ) );
      |         ^

gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/pow34.o hmp3/src/pow34.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/sbt.o hmp3/src/sbt.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/xhwin.o hmp3/src/xhwin.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/xsbt.o hmp3/src/xsbt.c
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/bitallo.o hmp3/src/bitallo.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/bitallo1.o hmp3/src/bitallo1.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/bitallo3.o hmp3/src/bitallo3.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/bitalloc.o hmp3/src/bitalloc.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/bitallos.o hmp3/src/bitallos.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/bitallosc.o hmp3/src/bitallosc.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/mp3enc.o hmp3/src/mp3enc.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/srcc.o hmp3/src/srcc.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/srccf.o hmp3/src/srccf.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/srccfb.o hmp3/src/srccfb.cpp
gcc -O3 -c -I./hmp3/src/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 -o builds/release/test/tomp3.o hmp3/src/test/tomp3.cpp
gcc -o builds/release/hmp3 builds/release/amodini2.o builds/release/cnts.o builds/release/detect.o builds/release/emap.o builds/release/l3init.o builds/release/l3pack.o builds/release/mhead.o builds/release/pcmhpm.o builds/release/setup.o builds/release/spdsmr.o builds/release/xhead.o builds/release/cnt.o builds/release/emdct.o builds/release/filter2.o builds/release/hwin.o builds/release/l3math.o builds/release/pow34.o builds/release/sbt.o builds/release/xhwin.o builds/release/xsbt.o builds/release/bitallo.o builds/release/bitallo1.o builds/release/bitallo3.o builds/release/bitalloc.o builds/release/bitallos.o builds/release/bitallosc.o builds/release/mp3enc.o builds/release/srcc.o builds/release/srccf.o builds/release/srccfb.o builds/release/test/tomp3.o -lm -lstdc++
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-14 11:56:29
Thankfully you don't need to worry about that warning. That function is possibly some old leftover as it's not used anywhere in the code.
If it was actually used things wouldn't work at all in 64-bit build. The function truncates a 64-bit memory address when it gets stored in a 32-bit variable. Attempts to access data in the new wrong location would crash badly.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-14 12:01:32
These warning are harmless - that function is superfluous for the C code path (not even used) and the 32-bit ASM code path has its own version. It's just left in there so there's a human-readable version for what the ASM version is doing.

(The warning basically says: Hey, pointers are 64-bit long on 64-bit architectures, you're calculating new pointer values in 32-bit - that's somewhat suspicous! - the compiler is right, but given that this code is never called in the C versions, it's of no consequence.)

Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-16 22:01:49
Rescue rangers (https://youtu.be/watch?v=PELdq2SIhr4), we have a situation related to the CBR mode with a low bitrate. Unlike LAME, the -B flag of Helix sets the value for each channel, so to get 320 kbps, we enter 160, respectively, to get 24 kbps, I entered 12. But this time the encoding did not go as expected.

Code: [Select]
$ hmp3 -B12 gudki.11khz.mono.wav gudki.11khz.mono.mp3

pcm file:  channels = 1  bits = 16,  rate = 11025  type = 1
 Layer III   MONO   22050Hz   12kbps

-------------------------------------------------------------------------------
      Frames  |     Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
         621  |        352800 /      24293 |   100%   |  11.88 /  12.00 Kbps
-------------------------------------------------------------------------------
 Compress Ratio 6.885771%


$ hmp3 -B12 gudki.22khz.mono.wav gudki.22khz.mono.mp3

 pcm file:  channels = 1  bits = 16,  rate = 22050  type = 1
 Layer III   MONO   22050Hz   12kbps

-------------------------------------------------------------------------------
      Frames  |     Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
         621  |        705600 /      24293 |   100%   |  11.88 /  12.00 Kbps
-------------------------------------------------------------------------------
 Compress Ratio 3.442885%


$ hmp3 -B12 gudki.22khz.stereo.wav gudki.22khz.stereo.mp3

 pcm file:  channels = 2  bits = 16,  rate = 22050  type = 1
 Layer III   mode 1 STEREO  IS-3   22050Hz   24kbps

-------------------------------------------------------------------------------
      Frames  |     Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
         622  |       1411200 /      48744 |   100%   |  23.80 /  24.00 Kbps
-------------------------------------------------------------------------------
 Compress Ratio 3.454082%


$ hmp3 -B12 gudki.44khz.stereo.wav gudki.44khz.stereo.mp3

 pcm file:  channels = 2  bits = 16,  rate = 44100  type = 1
 ENCODER INIT FAIL

a) gudki.11khz.mono.mp3 and gudki.22khz.mono.mp3 are malformed files (24 293 bytes), apps refuse to open them
b) gudki.22khz.stereo.mp3 is the only file here that is both encoded as expected (24 kbps) and playable
c) gudki.44khz.stereo.wav is not encoded with this setting at all, but the error message is quite vague

(https://i6.imageban.ru/out/2024/04/17/05427bc42ccb9b2164ebdbbd2f669382.webp)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-17 07:02:09
A quick reply before taking a proper look...

Quote from: Kraeved on
Unlike LAME, the -B flag of Helix sets the value for each channel, [...] to get 24 kbps, I entered 12. But this time the encoding did not go as expected.
For the first files the encoding went as you specified. The mono files have only one channel so the bitrate remained at the level you asked.

Quote from: Kraeved on
a) gudki.11khz.mono.mp3 and gudki.22khz.mono.mp3 are malformed files (24 293 bytes), apps refuse to open them
The file is outside specs, also called "freeformat". There are some decoders with freeformat support, but of course it makes no sense to use such formats anywhere.
I think the best option is to disallow incorrect settings, what do you think?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-17 07:33:21
Yeah, the "malformed" files appear to be free format. I wasn't aware Helix could do those, so that's a nice finding. Lame can decode those (e.g. "lame --decode gudki.22khz.mono.mp3"). As far as I can tell, Helix did exactly what you asked it to do, sadly many decoders cannot properly deal with this form of MP3.

Helix has a few checks to discourage settings that are deemed non-recommended. For instance, for 44.1 kHz files, it won't go below -B48 and will just barf "ENCODER INIT FAIL" at the user. When taking away those checks, Helix can, e.g., create "passable" 80 kbps 44.1 kHz CBR files with intensity stereo, but I so far refrained from relaxing those checks in fear that the encoder might explode in unexpected ways.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Porcus on 2024-04-17 09:50:05
Free format is so extraordinary nowadays that one might consider to introduce a new restriction - like how LAME requires you to enter "--freeformat" to encode to such. Of course, more sorts of "overrule sanity" options may be up to some level of consideration, if one is so inclined.

(Just came to think about: Can it create "MPEG-2 only" files? Are they any problematic?)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-17 12:21:08
@Case - I hate to bother you, but could you please post the latest binary of Helix to this thread?  I try using the latest version from Rarewares (both 32 bit and 64 bit) and they both run really slow, and slow down my system...yet CPU utilization isn't near full when encoding using foobar2000.  

For some reason your version from a few days ago (5.2.2) works flawlessly and fast.  I have an AMD Ryzen 7 5700g, Windows 11, 32GB DDR4 RAM and 1TB nvme SSD if that matters. 

The last version of helix to work from Rarewares from about March 21 I think?  Every version after that either 32 or 64 bit, runs really slow, and hangs the system and just makes everything slow down to almost a crawl yet CPU usage is about half.
Thanks so much!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-04-17 12:45:55
Quote from: KevinB52379 on
@Case - I hate to bother you, but could you please post the latest binary of Helix to this thread?  I try using the latest version from Rarewares (both 32 bit and 64 bit) and they both run really slow, and slow down my system...yet CPU utilization isn't near full when encoding using foobar2000.  

For some reason your version from a few days ago (5.2.2) works flawlessly and fast.  I have an AMD Ryzen 7 5700g, Windows 11, 32GB DDR4 RAM and 1TB nvme SSD if that matters. 

The last version of helix to work from Rarewares from about March 21 I think?  Every version after that either 32 or 64 bit, runs really slow, and hangs the system and just makes everything slow down to almost a crawl yet CPU usage is about half.
Thanks so much!

That's odd. I use Ryzen 7s and 9s and I haven't noticed an issue. I'll check it out a little later. Thanks for the notification.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-17 13:46:43
I have no idea.  Honestly, this is the first time I've ever had an issue using any of your binaries at RareWares.  I've used your LAME binaries, Ogg Vorbis binaries and Opus binaries for well over a decade with no issues...so I'm not sure.

As I said the build, I believe it was from March 21, works fine, but everything newer from late March until mid April has these slow down issues, where it encodes slow, and the system hangs. 

But if I use Case's build attached to this forum from a few days ago (5.2.2) it works perfectly fine (I know it's a Win32 binary).

If you want me to test something, let me know, I'd be happy to try and troubleshoot for you! :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-17 14:00:58
Weird issue, @KevinB52379. Here's a fresh compile of the current git sources.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-17 14:17:46
@Case - This fresh compile works perfectly!  Nice and fast again and cpu utilization is near 100%
Thank you very much.
I'm still not sure why John's builds aren't working right for me though...I find that odd since his binaries are so reliable.
Thanks again!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-17 14:45:21
When helix was updated the last time on git to version 5.2.3 i did my x64 compile.
One simple and one with fast-math + AVX2 that doesn't do much for speed on my Ryzen 5900x.
You may give it a try.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-17 15:03:46
@Wombat  - I appreciate you sharing your compile with the forum, but alas, your version, like John's version, slows my system down to a crawl, and the system hangs and becomes very slow to respond to input.  CPU utilization registers about 50% or less.

When I try @Case version he kindly uploaded for me, it encodes at about 1,000x real time, where as your version encodes about 250x and slower (with the system slow down and hangs).

I tried both the regular x64 version and the fast math x64 version, same results...
I'm really not sure what the issue is. 
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-17 15:23:25
Did you try to set different Thread Counts or priority in foobars converter settings? Even non multithreaded my x64 compile is ~490x here while Case win32 is only ~280x speed.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-17 16:10:12
Where do I look at those settings?  I didn't change any priority settings, so whatever Foobar2000's default settings are, that's what it's set to. 

 My foobar2000 encoder settings are:- %d -F18000 -HF2 -V85 -X2

I have lossy selected, multi threading is enabled, 24 bit is selected.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-17 16:23:00
File -> Preferences
X
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-17 17:26:39
Quote from: KevinB52379 on
If you want me to test something, let me know, I'd be happy to try and troubleshoot for you!
@KevinB52379 A few hours ago, I sent you a private message with the Helix build to test. Do you have any feedback?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-17 18:10:29
Quote from: Kraeved on
Quote from: KevinB52379 on
If you want me to test something, let me know, I'd be happy to try and troubleshoot for you!
@KevinB52379 A few hours ago, I sent you a private message with the Helix build to test. Do you have any feedback?


I'm sorry, I missed your message.  I tried your build and I get about 259x realtime with Foobar2000 default converter options set for thread count and thread priority.  I appreciate the effor though.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-17 18:51:47
I guess all x64 gcc builds are pretty much the same in speed since hmp3 is not really to optmize with additional flags.
The high speed may overtake your io system.
Can you try a thread count of 4 only for testing?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-17 19:23:42
Quote from: Case on
Quote from: Kraeved on
a) gudki.11khz.mono.mp3 and gudki.22khz.mono.mp3 are malformed files (24 293 bytes), apps refuse to open them
The file is outside specs, also called "freeformat". There are some decoders with freeformat support, but of course it makes no sense to use such formats anywhere. I think the best option is to disallow incorrect settings, what do you think?

I believe non-spec stuff should be removed, After all, we are no longer at the beginning of the MP3 journey, when they tried all sorts of outlandish things, but proceed from what has been recognized. I tried to open a free-formatted audio file with a couple of dozen players on Windows 7 x64, and only MPV (probably with the help of mpg123 (https://www.mpg123.de/) decoder) was able to read it. Even FFplay failed.

Quote from: maikmerten on
Helix has a few checks to discourage settings that are deemed non-recommended. For instance, for 44.1 kHz files, it won't go below -B48 and will just barf "ENCODER INIT FAIL" at the user. When taking away those checks, Helix can, e.g., create "passable" 80 kbps 44.1 kHz CBR files with intensity stereo, but I so far refrained from relaxing those checks in fear that the encoder might explode in unexpected ways.

It's no coincidence that almost a month ago, when I proposed a prototype (https://hydrogenaud.io/index.php/topic,123331.msg1041539.html#msg1041539) of the simplified interface of Helix MP3 encoder, there were question marks instead of valid values for each of the encoding modes. If it is more or less clear with VBR — you can enter from 0 to 150, then what is acceptable with CBR? We should be clear about that both on the screen and on the wiki.

(https://i7.imageban.ru/out/2024/04/17/fd37d89e36b07d8a0e4abc5b7a34ec12.webp)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-17 19:57:41
Quote from: KevinB52379 on
@Case - I hate to bother you, but could you please post the latest binary of Helix to this thread?  I try using the latest version from Rarewares (both 32 bit and 64 bit) and they both run really slow, and slow down my system...yet CPU utilization isn't near full when encoding using foobar2000.

Quote from: Kraeved on
There is something wrong about Lancer mod 2021-05-11 hosted on RareWares. It is unable to correctly encode WAV when its sampling rate is less or equal to 11025 Hz: SSE2 version outputs silence, SSE3 outputs heavily distorted sound.

Quote from: john33 on
I'm off to Spain for 2 weeks tomorrow so my mind is elsewhere!! I don't think I will get to look at this until I return so please feel free to remind me if I should appear to have forgotten about it.

I was thinking about this the other day, expecting some trouble to happen. We have a few developers who release binaries here. And if the old-timers of our forum can still keep an eye on them, then what will we see if we just type "Helix MP3 encoder" in the search engine? A scattering of sites offering a version back from 2005 and some abandoned repositories. Unlike LAME, Helix MP3 encoder does not have its own page, a single point of entry. This issue cannot be solved immediately, but I see the light at the end of the tunnel. Dear @maikmerten, is it possible to compile Helix binaries automatically for x86 and x64 (two separate archives) on Github when the version is bumped, or could you update the front page by adding links to RareWares (downloads) and HA Wiki (documentation)?

(https://i4.imageban.ru/out/2024/04/17/c6c757b74653af94fecc1212a428abbb.webp)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-17 20:58:09
@Wombat - OK I tried your build again with foobar2000, and set the thread count to 4, and I tried again.

This time I get about 250x to 320x realtime, and the encoding time is about 1 hr 15 mins.  The system is more responsive....

With Case's version with these same settings I get 530x to 600x realtime with about 35 minutes to encode my library
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-18 01:14:10
Really no more idea at my site why you have only good results with the win32 binary.
Tested on Ryzen 5900x, Win11, foobar 2.1.4 x64, Thread Count auto, hmp3 5.2.3 x64 simple:
~4100x speed with temporary file (%s) and
~4400x speed stdin/stdout (-)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: autodidact on 2024-04-18 03:28:30
Here is a Clang build for comparison testing.  This does not have AVX optimizations.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-18 03:44:19
Nice idea to try clang. Last time i tried with hmp3 it was clearly slower as gcc.
I have a feeling there is some different thing at play. Maybe something hardware or even simply driver related.
@KevinB52379 has an 8 core Zen 3 5700G that does 600-1000x speed multi-threaded while my 12 core Zen 3 does ~3100x speed with the win32 binary by Case encoding mp3s from flac.

Edit: Clang version eaches ~4200x speed stdin/stdout
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-18 12:26:40
Hello!  I did try the Clang version because quite honestly I was curious and was hopeful it would fix it.  Alas it does not.  I get about 250x to 300x realtime encoding from FLAC to mp3.  The files are on an external ahrd drive and encoding to my internal NVME SSD.  I get similar results with everyone's build of Helix, and my system slows down (even more so than normal).  I even tried disabling AVG thinking that was somehow interfering, but no dice. 

When I use Case's version of Helix Win32 build, I get about 1000x to 1300x realtime encoding those same files.

I've tried everyone's 64 bit builds and 32 bit builds with the same results....

Except for Case.  Does he put some magic in his compile?? LOL

You are correct, I have a Zen3 8 core processor that can handle 16 threads (AMD Ryzen 7 5700g @ 3.8ghz base...it has integrated graphics and the system has 32GB of DDR4 RAM and 1tb nvme gen3 SSD (Samsung 980 1tb).

I'm willing to try any other builds if anyone wants me to, but so far, Case's version is the only one that seems to work properly with fast encoding speeds and doesn't cause the system to basically hang...with his version the system does slow down, but it still responds.  And encoding speed remains fast and consistent with full cup utilization.  With everyone else's builds the cpu utilization never reaches 100% even though I have foobar encoding settings set to automatic for thread count and default priority.

Oh, I have the latest chipset drivers and graphics drivers from AMD.
Sorry to be so wordy but just trying to provide info.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: fooball on 2024-04-18 12:33:28
Might be useful if you post a screen-grab of your system info.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-18 14:26:15
I still wonder what happens here and the following idea may be far fetched.
Together with my B550 chipset for AMD i had several USB problems until some AGESA BIOS update fixed it.
It may be worth to check for the latest BIOS update.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-18 14:35:19
I'll look for a BIOS update.  The one I currently have installed is from October 2023.  It's an HP system.  But, I feel that if it was a USB problem, wouldn't it be happening with Case's version as well? 

I'll search on HP's website for a newer BIOS offering for my model.

Thanks!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-18 14:41:22
Here is an x64 compile that is built with the slower compiler option -O1 that is only minimal faster as the win32 by Case.
The idea is that something overpaces your config.
Also a test with files encoded from the local SSD without USB in play can help to get an idea.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-18 15:24:35
I tried that build and I get about 300x from the external hard drive where my FLACs are and converting to mp3 to the internal SSD using foobar2000 and Helix. 

I am copying the FLACs over to the internal SSD and I'm going to try using foobar2000 to convert totally from internal SSD drive...so the source will be the internal SSD and the destination will be the same internal SSD.

My system is a PCIe Gen 3x so I didn't bother paying more for a PCIE gen 4 drive.

There were no BIOS updates.  The latest is from October 2023 and that is what I have installed.

I'll let you know how this conversion fares.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-18 15:32:18
EDIT:  OK I found a compile of Helix MP3 encoder on my external hard drive , the zip file is called helix_mp3enc_r11_x64.zip and the exe is called hmp3enc.exe.
I tried this version (I know it's one of the original Helix versions) but I was curious to see hwo this 64 bit version worked.

I did the test from my SSD and got 1300x to 1800x.

So what is different between Case's build, this old version and all newer compiles?

I believe I got that version of Helix from Rarewares or ReallyRareWares.

OK I tried copying my FLACS to the internal SSD and tried converting with foobar2000 from there.

I still average about 220x to 300x using your slow compile 64 bit version of Helix. 

Same speeds I got when converting from external hard drive...so I don't think USB is the issue.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-18 16:08:11
Quote from: Kraeved on
Dear @maikmerten, is it possible to compile Helix binaries automatically for x86 and x64 (two separate archives) on Github when the version is bumped, or could you update the front page by adding links to RareWares (downloads) and HA Wiki (documentation)?

Yeah, I probably should link to the Wiki documentation and then point to a proper download location for Windows builds. I'd of course like RandomWares to be the official download location, if john agrees.

As to building the Windows binaries, I think perhaps we should find out which builds work best and how they are generated. I'm a Linux guy and am hesitant to weigh in too much on Windows things, but am hopeful that it may become apparent what works best on Windows.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-18 18:07:00
Quote from: 145dBSPL on
This is an interesting insight. Thank you for your research. To visualize this, I created and encoded a pink noise file.
Settings used are -V110 -HF[1|2]. For both encoded files, an mp3 analysis shows 18.0 % Simple stereo frame
and 82.0 % Mid-side stereo frames.

Sorry for the late reaction, but thanks for illustrating this. Mind if I steal your images for the Wiki documentation?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: 145dBSPL on 2024-04-18 20:03:11
Oh, that's perfectly fine. Thank you for asking.
Let me know if you need higher resolution images.

Some more details if required:
Pink noise, stereo, 16 bits, 44.1 kHz, -10 dB, uniform distribution
Used spectrum analyzer is Spek (https://www.spek.cc/).
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: JoshuaChang on 2024-04-19 04:13:51
Quote from: KevinB52379 on
EDIT:  OK I found a compile of Helix MP3 encoder on my external hard drive , the zip file is called helix_mp3enc_r11_x64.zip and the exe is called hmp3enc.exe.
I tried this version (I know it's one of the original Helix versions) but I was curious to see hwo this 64 bit version worked.

I did the test from my SSD and got 1300x to 1800x.

So what is different between Case's build, this old version and all newer compiles?

I believe I got that version of Helix from Rarewares or ReallyRareWares.

OK I tried copying my FLACS to the internal SSD and tried converting with foobar2000 from there.

I still average about 220x to 300x using your slow compile 64 bit version of Helix. 

Same speeds I got when converting from external hard drive...so I don't think USB is the issue.

I think i have the same problem, my cpu is zen2 4650g(6c12t), i got around 280~300x under command line @ single thread with autodidact's clang build or rarewares' gcc build, I also compiled helix myself with clang plus linktime optimization, my version got around 480x. maybe that's the key.
here's the binary, you can try it.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-19 12:00:08
Quote from: JoshuaChang on
Quote from: KevinB52379 on
EDIT:  OK I found a compile of Helix MP3 encoder on my external hard drive , the zip file is called helix_mp3enc_r11_x64.zip and the exe is called hmp3enc.exe.
I tried this version (I know it's one of the original Helix versions) but I was curious to see hwo this 64 bit version worked.

I did the test from my SSD and got 1300x to 1800x.

So what is different between Case's build, this old version and all newer compiles?

I believe I got that version of Helix from Rarewares or ReallyRareWares.

OK I tried copying my FLACS to the internal SSD and tried converting with foobar2000 from there.

I still average about 220x to 300x using your slow compile 64 bit version of Helix. 

Same speeds I got when converting from external hard drive...so I don't think USB is the issue.

I think i have the same problem, my cpu is zen2 4650g(6c12t), i got around 280~300x under command line @ single thread with autodidact's clang build or rarewares' gcc build, I also compiled helix myself with clang plus linktime optimization, my version got around 480x. maybe that's the key.
here's the binary, you can try it.


Thanks for contributing your compile.  I tried it, but for some reason it isn't creating mp3s on my end.  I kept the settings the same as I used previously for successful encodes, but literally just replaced the hmp3.exe file.

Also when using it with foobar, the system did hang again, and I saw the conversion dialog, but the "pause" and "abort" buttons never showed up and the file progress never continued.  What I mean is the progress bar was moving, but it was still showing that it was converting the first file still.  Weird.

Thanks for trying!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: JoshuaChang on 2024-04-19 13:31:09
That's really weird, i've tried both cmd and foobar2000, everything is ok, foobar2000 2.1.4 x64 with minimal convert parameter:" - %d"
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-19 14:32:11
Also not working here "An error occurred while writing to file (The encoder has terminated prematurely with code -1073741515 (0xC0000135); please re-check parameters"
Btw. did you just set the flag -flto? I could do one with gcc.

Edit:  Did one with gcc flto. Roughly the same speed here as without.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: JoshuaChang on 2024-04-19 15:52:03
Quote from: Wombat on
Also not working here "An error occurred while writing to file (The encoder has terminated prematurely with code -1073741515 (0xC0000135); please re-check parameters"
Btw. did you just set the flag -flto? I could do one with gcc.

Edit:  Did one with gcc flto. Roughly the same speed here as without.

I've managed to reproduce the problem under my vm env, my fault for not including the -static in LDFLAGS, here's the re-complied version.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: ha7pro on 2024-04-19 16:12:00
This version compiled with clang is the fastest in the test, followed by gcc avx2. This is just the result of my test.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: punkrockdude on 2024-04-19 16:12:48
If anyone is interested

Linux amd64/x86_64 Glibc 2.39
Clang 18, O3 LTO PGO

Code: [Select]
CC=clang # was gcc

#CFLAGS_COMMON=-O3 -fprofile-generate=/tmp/ -c -I$(SRC_PREFIX)/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 # 1st run
CFLAGS_COMMON=-O3 -flto -fprofile-use=/tmp/profile.profdata -c -I$(SRC_PREFIX)/pub -DIEEE_FLOAT -D_FILE_OFFSET_BITS=64 # 2nd run

#LFLAGS=-lm -lstdc++ -O3 -fprofile-generate=/tmp/ # 1st run
LFLAGS=-lm -lstdc++ -O3 -flto -fprofile-use=/tmp/profile.profdata # 2nd run
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-19 16:42:19
Quote from: JoshuaChang on
Quote from: Wombat on
Also not working here "An error occurred while writing to file (The encoder has terminated prematurely with code -1073741515 (0xC0000135); please re-check parameters"
Btw. did you just set the flag -flto? I could do one with gcc.

Edit:  Did one with gcc flto. Roughly the same speed here as without.

I've managed to reproduce the problem under my vm env, my fault for not including the -static in LDFLAGS, here's the re-complied version.

FINALLY!  A version of Helix MP3 that converts fast besides Case's version!

This version DOES NOT hang up my system in a weird way and cause foobar to be basically nonresponsive!

ALSO - FASTEST encoding speed yet converting from internal SSD to internal SSD at 2500x encoding speed!

This version works, yay!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: punkrockdude on 2024-04-19 16:45:34
Quote from: KevinB52379 on
ALSO - FASTEST encoding speed yet converting from internal SSD to internal SSD at 2500x encoding speed!
How many cores/threads?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-19 16:55:57
Quote from: punkrockdude on
Quote from: KevinB52379 on
ALSO - FASTEST encoding speed yet converting from internal SSD to internal SSD at 2500x encoding speed!
How many cores/threads?

My system has an AMD Ryzen 7 5700g - Zen3 architecture.  8 cores 16 threads.  I have foobar running at full thread count during conversion.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-19 17:44:14
Fascinating stuff is that the "g" variants behave so different and a profiled 2pass compile fixes it.
The speed gain is nice also :)
Thanks to @punkrockdude for the insight!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: punkrockdude on 2024-04-19 18:04:15
Linux amd64/x86_64 Glibc 2.39
Clang 18, O3 LTO PGO x86-64-v3
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: punkrockdude on 2024-04-19 18:14:59
Amazing to convert a 38min album in 8 seconds. I think Deadbeef is single threaded but I am not sure.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-19 18:24:46
Yeah single threaded a normal album well encodes below 10 seconds.
@JoshuaChang binary is ~4600x speed here on the 5900x :)
The problem is what exactly causes the faulty behaviour with the "g" variants and how to fix it.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: punkrockdude on 2024-04-19 18:27:28
Quote from: Wombat on
The problem is what exactly causes the faulty behaviour with the "g" variants and how to fix it.
Like Ryzen 5600G (what I am using) etc or the builds using Og? I now tried a 1h audio (44.1kHz/16bit/2ch) encoded with no parameters and it took 06s. It was using only one logical core.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-19 18:36:55
The Windows x64 compiles using makefile don't work properly on at least the 2 "g" variants reported here.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: punkrockdude on 2024-04-19 18:43:12
I wonder if I can get up to 7200x if I can encode using all my logical cores at the same time, somehow. Let the speed competition begin! :) I might try adding OpenMP or OpenACC...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-19 19:59:51
For me (AMD Ryzen 7 5700g variant) ALL compiles of 32 bit and 64 bit cause this weird system hang and slowdown and slow encoding speed...except for the 64 bit version I quoted above in an earlier post, and Chase's 32 bit compile.

Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-20 09:53:08
Quote from: 145dBSPL on
Oh, that's perfectly fine. Thank you for asking.
Let me know if you need higher resolution images.

Some more details if required:
Pink noise, stereo, 16 bits, 44.1 kHz, -10 dB, uniform distribution
Used spectrum analyzer is Spek (https://www.spek.cc/).

Thank you, I uploaded the images with attribution and the details you provided to the Wiki and embedded them in the Helix MP3 Encoder page.

As for the various reports of the different Win32/Win64 compiles behaving vastly differently, I currently don't have a good hypothesis on what's going on there. I wouldn't, e.g., expect builds with different optimizations (e.g., PGO or no PGO) to make the difference between a machine getting unresponsive or working properly. Also, doing static builds (-static linker flag) only means that the required system libraries are included in the build - I don't quite see how this should have a meaningful performance impact (unless the system libraries are somehow misbehaving).

So I guess it might be a good idea to somehow collect data on the various build options and find a Windows build solution that works reliably.

Other than that, I'm pleased that apparently some people are getting results that indicate it's now possible to encode hours of audio in mere seconds. I remember working on machines where this used to be the other way around...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: rutra80 on 2024-04-20 12:11:53
@KevinB52379 maybe you have cooling problems and AVX(2) compiles cause throttling?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-20 14:02:14
In the Wiki, I also documented the -T parameter, which isn't part of the normal user documentation. This biasses the normal VBR quality setting (-V) and can be used to reach lower or higher VBR qualities/bitrates. The -T parameter ranges from -40 (lowest quality bias) to 50 (highest quality bias).

Examples:

This generates 44.1 kHz stereo encodings with roughly ~96 kbps (-V0 usually targets ~112 kbps) with a lowpass of 16 kHz:
Code: [Select]
hmp3 -F16000 -V0 -T-20 input.wav output.mp3

This allows encoding high-frequency content (> 16 kHz) even at low VBR bitrates such as ~128kbps:
Code: [Select]
hmp3 -V80 -T-40 -HF2 input.wav output.mp3

This increases the VBR quality target beyond what's usually possible (~310 kbps VBR instead of ~260 kbps VBR):
Code: [Select]
hmp3 -V150 -T50 -HF2 input.wav output.mp3

So that's a parameter for fun experiments.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-20 14:14:45
Quote from: rutra80 on
@KevinB52379 maybe you have cooling problems and AVX(2) compiles cause throttling?
The problem occures also without the use of AVX versions.
I think we have a problem only with some AMD APUs. At least @KevinB52379 and @JoshuaChang use a Ryzen 4650g and a 5700g.
I already offered a x64 compile with a uncommon low optimization level -O1 and this didn't help.
So we must find out if it is maybe a system problem with for example a special version of the AMD chipset/graphics driver and why it does not happen with a lto compile that also uses the high optimization level -O3.
This may be something for coding experts and not for a noob like me.
I fear only @KevinB52379 and @JoshuaChang can find out directly at their PCs.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Replica9000 on 2024-04-20 16:12:15
I have a Ryzen 7 5850U.   I can do some testing with hmp3 on Linux.  I cloned @maikmerten's github and built hmp3. 

With the default Makefile, I can encode a 1h 43m wave file in 12.42s average

Adding -march=x86-64-v3, the file encodes an average of 13.10s.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-21 00:51:42
@maikmerten Right now the build (https://hydrogenaud.io/index.php/topic,123331.msg1043104.html#msg1043104) of @Case is the most relevant for 32-bit systems, and the build (https://hydrogenaud.io/index.php/topic,123331.msg1043232.html#msg1043232) of @JoshuaChang delights HA users of 64-bit systems. In both cases, we still have no idea how their binaries are compiled from the code because the makefiles are not shared. Why? Err… Please? But new posts are coming and links to these builds will soon be lost among a dozen other testing, non-optimal, raw builds. I can't help but feel that it is not the order that is growing, but the mess. How can we trumpet the good news of the resurrection among our neighbors if we can't really point out where Helix MP3 encoder, this eighth wonder of the audio world, is? Even established search engines find it difficult to help here. Let's at least mention these builds on the HA wiki for a while, as it is done on the FhG AAC encoder (https://wiki.hydrogenaud.io/index.php?title=FhG_AAC#External_links) page.

(https://i3.imageban.ru/out/2024/04/21/a26ea13a8af6836995c53289e7ac2550.webp)
Fyodor Bronnikov — Pythagoreans celebrate the sunrise (1869)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: JoshuaChang on 2024-04-21 06:04:59
Quote from: Kraeved on
@maikmerten Right now the build (https://hydrogenaud.io/index.php/topic,123331.msg1043104.html#msg1043104) of @Case is the most relevant for 32-bit systems, and the build (https://hydrogenaud.io/index.php/topic,123331.msg1043232.html#msg1043232) of @JoshuaChang delights HA users of 64-bit systems. In both cases, we still have no idea how their binaries are compiled from the code because the makefiles are not shared. Why? Err… Please? But new posts are coming and links to these builds will soon be lost among a dozen other testing, non-optimal, raw builds. I can't help but feel that it is not the order that is growing, but the mess. How can we trumpet the good news of the resurrection among our neighbors if we can't really point out where Helix MP3 encoder, this eighth wonder of the audio world, is? Even established search engines find it difficult to help here. Let's at least mention these builds on the HA wiki for a while, as it is done on the FhG AAC encoder (https://wiki.hydrogenaud.io/index.php?title=FhG_AAC#External_links) page.

(https://i3.imageban.ru/out/2024/04/21/a26ea13a8af6836995c53289e7ac2550.webp)
Fyodor Bronnikov — Pythagoreans celebrate the sunrise (1869)

I'm using msys2's clang build env, so gcc is an alias for clang, I just put -flto in CFLAGS_COMMON and put -static in LFLAGS, see the attachment for the exact file.
ps, I've also tested compile without -flto, no large performance loss(about 5%).
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-21 07:32:57
@KevinB52379 can you try this GCC build?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: punkrockdude on 2024-04-21 11:08:32
Raspberry Pi 5, Clang 18 (-Ofast -flto -fopenacc)
276x (one core)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-21 18:11:00
Quote from: Case on
@KevinB52379 can you try this GCC build?

@Cas - I tried this version as you requested.

Result:
230x ~ 280x encoding speed.

All cores and threads utilized.

Again, system slow downs, system hang, foobar becomes nonresponsive.  Note: this slow down hang is differen than that which is expected from slow cpu load.

And as before, all cores aren't being utilized to 100% according to process lasso.

When I use your 32 bit version that you provided a few days ago and JoshuaChang 64 bit build, everything works perfectly.  No system hang/unresponsiveness, and full cpu utilization.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-21 18:15:15
Quote from: JoshuaChang on
I'm using msys2's clang build env, so gcc is an alias for clang, I just put -flto in CFLAGS_COMMON and put -static in LFLAGS, see the attachment for the exact file.
ps, I've also tested compile without -flto, no large performance loss(about 5%).
Thanks!
Unfortunately in a fresh installed msys2 using the clang64 console i only create non working binaries as if static never really works. I have to dig deeper on my side.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-22 01:35:16
One more post, sorry. Only a few packages were installed and already i blew it up in the dependencies.
I did msys2 once again and now it works and as fast as expected thanks to @JoshuaChang.
I still don't get why flto inside the clang64 enviroment gives that boost or if it is the repeated setting for flto in the linker flags. Have to play around more later i guess :)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-22 07:56:23
Thank @KevinB52379. It was a desperate attempt, I saw somewhere suggestions that Ryzens would suffer from denormals more than Intels, that version set CPU flags that would round problematic denormal numbers to zero to prevent them from causing slowdowns.

Here's another test compile but this one you'll need to run manually in command prompt. It will print timing information at the end of the encoding so we'll see where all the time is spent.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: JoshuaChang on 2024-04-22 10:27:02
Quote from: Wombat on
One more post, sorry. Only a few packages were installed and already i blew it up in the dependencies.
I did msys2 once again and now it works and as fast as expected thanks to @JoshuaChang.
I still don't get why flto inside the clang64 enviroment gives that boost or if it is the repeated setting for flto in the linker flags. Have to play around more later i guess :)

I don't think it's related to lto, you can try without -flto, the performance loss is very little(on my side, it's about 5%)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-22 14:41:30
Quote from: JoshuaChang on
I don't think it's related to lto, you can try without -flto, the performance loss is very little(on my side, it's about 5%)
5% only due compiling is significant imho.
You can measure these 5% and run the non lto binary without problems on your 4650g? Guess i misunderstood it so far.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: john33 on 2024-04-22 14:52:03
I suspect that this may have something to do with the reduced amount of level 3 cache on the Ryzen APUs compared to the standard CPUs.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-22 15:45:28
I don't really think that the architecture of the AMD Zen APUs really is the main problem here. I'm on an AMD Ryzen Pro 4750G (Zen2-based APU), and here's a 64-bit native Linux compile converting a complete CD album (hmp3 -V100 -HF2)

Code: [Select]
real	0m9,241s
user	0m9,057s
sys	0m0,184s

Here's a 64-bit Windows compile provided by Case (the binary posted on 2024-04-21 07:32:57) on Linux, in Wine, same input, same parameters:

Code: [Select]
real	0m22,239s
user	0m11,178s
sys	0m3,972s

Is the Windows version somehow spamming system calls? The process seems to spend considerable time "elsewhere".

edit:

For completeness sake: My mingw64 win64-cross compile yields very similar results:

Code: [Select]
real	0m21,472s
user	0m10,772s
sys	0m3,836s
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-22 15:59:57
For real completeness please run also the @JoshuaChang binary. https://hydrogenaud.io/index.php/topic,123331.msg1043232.html#msg1043232
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-22 16:18:06
Quote from: Wombat on
For real completeness please run also the @JoshuaChang binary. https://hydrogenaud.io/index.php/topic,123331.msg1043232.html#msg1043232

That was indeed a worthwhile test. That binary, on Linux, in Wine, on my AMD APU:

Code: [Select]
real	0m11,342s
user	0m10,098s
sys	0m0,300s

That one seems to avoid spending a lot of time "elsewhere"...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-22 16:21:21
Quote from: maikmerten on
That one seems to avoid spending a lot of time "elsewhere"...
Exactly. So far this big difference only happens on the AMD "g" APUs.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-22 17:00:53
For testing, I disabled the usual Spectre/Meltdown CPU security mitigations ("mitigations=off" as Linux boot parameter). This significantly improved performance.

Binary provided by Case *with* security mitigations (see my previous post):
Code: [Select]
real	0m22,239s
user	0m11,178s
sys	0m3,972s

Binary provided by Case *without* CPU security mitigations:
Code: [Select]
real	0m15,819s
user	0m10,493s
sys	0m1,701s

The CPU security mitigations impact the speed of system calls. For me, this indicates that the affected hmp3.exe binaries are somehow unusually syscall-heavy. This would also explain why the system would get "unresponsive", as reported.

(The native 64-bit Linux binary doesn't get faster when disabling the security mitigations.)
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-22 17:24:54
Since you have an affected system, care to test the compile I posted today? It ought to at least give a hint where the slowdowns happen.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-22 17:27:36
Quote from: Case on
Since you have an affected system, care to test the compile I posted today? It ought to at least give a hint where the slowdowns happen.

Gladly!

Code: [Select]
Time used:
main:     195.430800 ms
wav:      5.713400 ms
encode:   9885.130700 ms
xing:     189.984200 ms
file io:  191.137400 ms
print io: 10774.042400 ms
kbhit:    15.513600 ms
real	0m22,051s
user	0m11,398s
sys	0m3,645s

Spending more time on printing stuff than actual encoding?!

edit: And here's with -D flag (don't print progress):

Code: [Select]
Time used:
main:     197.418600 ms
wav:      6.954600 ms
encode:   9804.481700 ms
xing:     189.424400 ms
file io:  200.825100 ms
print io: 0.000000 ms
kbhit:    15.052200 ms
real	0m11,049s
user	0m10,044s
sys	0m0,233s
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-22 17:29:05
Status print. Adding -D should make it go much faster in that case. Could you compile a test version without the fflush(stderr)?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-22 17:40:39
Quote from: Case on
Status print. Adding -D should make it go much faster in that case. Could you compile a test version without the fflush(stderr)?

Removing the fflush(stderr) doesn't really improve stuff:

Code: [Select]
real	0m21,684s
user	0m10,887s
sys	0m3,854s

Commenting out the sprintf and fprintf does, however, ...

Code: [Select]
real	0m10,610s
user	0m9,634s
sys	0m0,181s

So, I guess the fix would be to only print stuff, e.g., every 100 milliseconds or so...

Mind if I engineer a fix?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-22 17:55:43
Thanks. That sounds like a good workaround for improving performance. But I'd like to know why the printing is so slow on those AMD systems. The GCC binary seems to use old functions from msvcrt.dll. Visual Studio compile is static and only calls Windows API functions externally. Clang compile calls functions from much newer Universal CRT.

Either way the msvcrt functions aren't slow on other machines...
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-22 19:02:32
Turns out it's surprisingly unnerving to determine the current time in milliseconds across platforms and across compiler-versions in C/C++, so here's my ugly little patch that cuts down the number of updates by counting:

https://github.com/maikmerten/hmp3/commit/fad361020b066bddaeaa2191022a70d8ba1e7eb9

(currently on the "dev" branch)

In my win64-in-Wine tests, this avoids the performance penality.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-22 20:59:35
I just tried this version.  The encoding speed with foobar is 800x...but the system still hangs and becomes unresponsive.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-23 05:31:06
Quote from: KevinB52379 on
I just tried this version.  The encoding speed with foobar is 800x...but the system still hangs and becomes unresponsive.

Thanks for testing! Do the system hangs also happen with the -D (don't print progress) option?
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-23 10:16:04
I did some benchmarking and GCC's fprintf takes 100 times longer even here compared to MSVC's or clang's function.
Attached is a patch that changes progress printing to use WriteConsole on Windows when stderr isn't redirected to file. That gives consistently fast progress display on all compilers. I also noticed that stopping encoding by keyboard press didn't work when not compiled on Visual Studio, fixed that too.

I also changed the progress printing to happen 4 times less often than originally. I think maikmerten's progress update happened too seldom, it looked buggy when it was so sluggish.

I hope @KevinB52379 can test this and report if things now work correctly.

Edit: replaced with static compile.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-23 10:45:44
Hmmm.

* in.wav, 99 423 788 bytes
* hmp3.case.x64, 513 024 bytes, xxhash 820182a208834328
* hmp3.chang.x64, 839 680 bytes, xxhash 4080008cf88453e9

Code: [Select]
$ hyperfine --warmup 3 --runs 3 -L encoder hmp3.case.x64,hmp3.chang.x64 "{encoder} -V150 -HF2 -F20000 in.wav out.mp3"

  Command          Mean [s]        Relative     
 ---------------- --------------- ------------- 
  hmp3.case.x64    6.860 ± 0.051   1.41 ± 0.01  
  hmp3.chang.x64   4.850 ± 0.019   1.00     
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: JoshuaChang on 2024-04-23 10:48:26
Quote from: Case on
I did some benchmarking and GCC's fprintf takes 100 times longer even here compared to MSVC's or clang's function.
Attached is a patch that changes progress printing to use WriteConsole on Windows when stderr isn't redirected to file. That gives consistently fast progress display on all compilers. I also noticed that stopping encoding by keyboard press didn't work when not compiled on Visual Studio, fixed that too.

I also changed the progress printing to happen 4 times less often than originally. I think maikmerten's progress update happened too seldom, it looked buggy when it was so sluggish.

I hope @KevinB52379 can test this and report if things now work correctly.


I think you might forgot to add -static in the LFLAGS since it ask for libstdc++-6.dll in my env.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-23 10:57:04
Thanks for noticing. fixed.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-23 11:17:37
Quote from: Kraeved on
Hmmm.
[...]
Code: [Select]
  hmp3.case.x64    6.860 ± 0.051   1.41 ± 0.01  
  hmp3.chang.x64   4.850 ± 0.019   1.00     
Note that the intention is to find and solve the abnormal system slowdown KevinB reported. His problem is in no way related to relative performance of different compiles.

Though I do wonder how JoshuaChang's clang compile is so fast. Usually GCC has been able to produce faster binaries. And if I compile a clang build with joshua's makefile, I get a slower binary than my GCC compile.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: JoshuaChang on 2024-04-23 11:30:01
I think that might be related to the compile's link stage, my clang binary under exeinfo shows a microsoft signature code, and all other gcc binaries and clang binaries shows a mingw signature code
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-23 11:45:12
Quote from: Case on
I did some benchmarking and GCC's fprintf takes 100 times longer even here compared to MSVC's or clang's function.
Attached is a patch that changes progress printing to use WriteConsole on Windows when stderr isn't redirected to file. That gives consistently fast progress display on all compilers. I also noticed that stopping encoding by keyboard press didn't work when not compiled on Visual Studio, fixed that too.

I also changed the progress printing to happen 4 times less often than originally. I think maikmerten's progress update happened too seldom, it looked buggy when it was so sluggish.

I hope @KevinB52379 can test this and report if things now work correctly.

Edit: replaced with static compile.

EDIT:  I tried the -D switch and that didn't improve anything.

Also, please note, I am able to use FAAC, LAME, Vorbis, Opus from rarewares with all thrads going with foobar2000 and I have fast encoding speeds and no system hang.  Yes, I know there are slowdowns because all the cores are being used, but this system hang I'm experiencing with Helix is totally different.
I tried this version.

At first it started off promising, encoding started at around 1300x, but then about 20 seconds or so in, things started to slow down.  They system started to hang, things became very unresponsive.

When encoding first started with foobar it started about 1400x to 1500x and then slowed down drastically.

In fact, in foobar the file list of what was currently being encoded stopped updating, the time elapsed stopped updating, the estimated time complete incrased and then stopped updating.  The only thing that was changing, although very slowly was the progress bar of the encoding task.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-23 12:04:45
@KevinB52379, what if you discard Foobar2000 as an intermediary for a while and encode right in the terminal using Hyperfine (https://hydrogenaud.io/index.php/topic,123331.msg1043434.html#msg1043434), a cross-platform tool for warming up the system and measuring the execution time of the binaries? For example, @misio discovered and I verified that visualization via VST (https://hydrogenaud.io/index.php/topic,123342.msg1042721.html#msg1042721) plug-ins works much slower in Foobar2000 than the same plug-ins in audio editors (DAWs), and surprisingly, interaction with WavPack files turned out to be the slowest.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-23 12:32:54
If you don't mind, you could install Windows Performance Toolkit (https://learn.microsoft.com/en-us/windows-hardware/test/wpt/) and use Windows Performance Recorder to record what goes on with the machine when encoding. You can enable all options under Resource Analysis and you can stop recording when the system goes slow. If you share the recording it should reveal why your machine acts so badly.

@Kraeved - I already tried to reply in the related topic (https://hydrogenaud.io/index.php/topic,123342.msg1043012.html#msg1043012) why VSTs aren't smooth. But let me clarify. The VSTs are given big blobs of audio data to process at once, the exact size depends on design of decoder components and what other DSPs may be before them. The janky VSTs apparently have trouble updating their UIs when dealing with such data. All buffers in foobar2000 are big to support smooth and glitchless audio playback. DAWs have tiny buffers to work as realtime as possible.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-23 13:47:50
@Case - I would try that toolkit, but the problem is, once the system starts to hang, I can't do anything.  For example, that start menu button doesn't respond, and when I hit "abort" on foobar, it can take several minutes for it to finally abort (if it even stops at all).  There have been times I have to let it finish encoding the process because I can't cancel it.

I do find it odd that the cpu utilization doesn't even go to 90 - 100% usage like it does with your 32 bit binary from a few days ago and @JoshuaChang version does.  So far those are the only 2 builds that work properly.

Oh, I also am legally blind and use Screen Magnification software, but this program ISN'T the cause of the problem, because even when I shut it down before encoding, the system still hangs.
Let me look into it.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-23 14:23:44
Quote from: Case on
Though I do wonder how JoshuaChang's clang compile is so fast. Usually GCC has been able to produce faster binaries. And if I compile a clang build with joshua's makefile, I get a slower binary than my GCC compile.
Clang 18 in the msys2 clang64 enviroment creates these fast binaries for hmp3 here also. Seems the code reacts perfectly to this enviroment.
It is not a magic bullet since flac is still faster with a standard gcc compile in my old mingw64 enviroment.

@Case already mentioned clang64 uses ucrt in this case https://www.msys2.org/docs/environments/
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-23 15:05:33
Quote from: Kraeved on
@KevinB52379, what if you discard Foobar2000 as an intermediary for a while and encode right in the terminal using Hyperfine (https://hydrogenaud.io/index.php/topic,123331.msg1043434.html#msg1043434), a cross-platform tool for warming up the system and measuring the execution time of the binaries? For example, @misio discovered and I verified that visualization via VST (https://hydrogenaud.io/index.php/topic,123342.msg1042721.html#msg1042721) plug-ins works much slower in Foobar2000 than the same plug-ins in audio editors (DAWs), and surprisingly, interaction with WavPack files turned out to be the slowest.

How come when I click your Hyperfire link it brings me back to this hydorgenaudio Helix thread?

Can you provide a download link to try this? 
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-23 15:10:55
Quote from: KevinB52379 on
How come when I click your Hyperfire link it brings me back to this hydorgenaudio Helix thread?
Can you provide a download link to try this? 

The link in that message takes you to an example of how to use Hyperfine (https://github.com/sharkdp/hyperfine), so that you don't have to go through the user's guide in search of the proper commands. To date, the latest version available for download on Github is 1.18 (https://github.com/sharkdp/hyperfine/releases/tag/v1.18.0).

When troubleshooting, the first thing I do is look for a minimal reproducible example (https://en.wikipedia.org/wiki/Minimal_reproducible_example), i.e. I simplify the chain of events that lead to the problem. By measuring the audio encoding using several builds of Helix MP3 encoder in the terminal, we will know for sure whether Foobar2000 affects the slowdown in any way or not. If there is a comparable slowdown to the point that the system becomes unresponsive in the terminal as well, you can continue to encode via Foobar2000. Also, when the encoder outputs its report in the terminal, you may witness phenomena like counters going crazy (https://hydrogenaud.io/index.php/topic,123331.msg1041279.html#msg1041279), which will give the developers a clue on how to fix things.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-23 15:21:53
Don't confuse @KevinB52379 to much.
Old hmp3 versions work, @JoshuaChang version works within foobar.
A solution at the source is worthwile in this case.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-23 16:29:52
Quote from: Wombat on
Don't confuse @KevinB52379 to much.
Old hmp3 versions work, @JoshuaChang version works within foobar.
A solution at the source is worthwile in this case.

I was willing to give Hyperfire a shot...but to be honest I thought the same thing as you. 

If the original version of Helix 64 bit works from ReallyRareWares, and @JoshaChan version works perfectly with foobar...then something else is the issue.

Also as I mentioned, I've used LAME (both 32 bit and 64 bit), FAAC 1.30, OPUS, Ogg Vorbis AoTuV, FDKAAC - all from RareWares, and they all behave normally, so I don't think it's a temperature/overheating issue.  This is with foobar and all cores/threads being used.  Perfect fine speed and system responsiveness.

I do think it is interesting that CPU utilization never reaches 100% like it does with Joshua's version, old Helix and Case's 32 bit binary.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-24 07:32:24
Here's a test build with GCC linked against UCRT. All the compiles you have tried that have shown issues have used MSVCRT (the GCC builds and the clang build from @autodidact). The compiles that have worked have either been fully static or have been linked against the Universal CRT.

Not sure how useful this test is, but at least it's another data point. Publishing encoder binaries requiring UCRT isn't really an option in my opinion, that runtime is present by default only on Windows 10 and newer.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-24 12:31:11
@Case  - Thanks for this compile, but unfortunately, same results.  System hang.  The file progress bar increases, but the files being encoded with foobar2000 never changes, and I never see the elapsed time, estimated time, encoding speed, and the 'pause" and "abort" buttons never appear.  The only thing that changes is the progress bar.  It didn't take over an hour to encode like all the other slow compiles...but it was no where near as fast as @JoshuaChang version.  His version encodes over 5,000 flac files in under 9 minutes with no system hangs or slowdowns.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-24 13:04:23
Good to know. There are no longer many options that can cause the issue you are facing. I think antivirus or other security product that scans or sandboxes executables is the only thing that can do this.
The converter in foobar2000 uses very low priority by default and you said you haven't increased it. These encoders don't touch priorities so they do their tasks with leftover cycles. Memory consumption is no different between compiles.

You must have some high priority process(es) running on the same core as the UI processes and steal all the processing power from the UI operations. Or a sandbox goes haywire and consumes all the system memory. Swapping will make machine slow even with SSD.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: JoshuaChang on 2024-04-24 13:18:17
I've managed to compile the binary using visual studio 2022, it runs quite slow at my env(about 1/3 speed compared to clang version), but you can test if it will hang your system @KevinB52379 .
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-24 13:23:54
Quote from: Case on
Good to know. There are no longer many options that can cause the issue you are facing. I think antivirus or other security product that scans or sandboxes executables is the only thing that can do this.
The converter in foobar2000 uses very low priority by default and you said you haven't increased it. These encoders don't touch priorities so they do their tasks with leftover cycles. Memory consumption is no different between compiles.

You must have some high priority process(es) running on the same core as the UI processes and steal all the processing power from the UI operations. Or a sandbox goes haywire and consumes all the system memory. Swapping will make machine slow even with SSD.

Hi @Case - I can provide some more information if you'd like.

1.  I use AVG Internet Security as my security suite.  In AVG I have excluded my foobar2000 directory (I use foobar in portable mode).  At one point, I even disabled realtime protection before encoding to see if this helped, nothing changed.  Would you like me to try uinstalling AVG?

2.  As I stated, I have an HP Pavilion with AMD Ryzen 7 5700g, and I'm using the integrated graphics.  I cannot install a dedicated graphics card as the system is limited to a 180 watt PSU and the motherboard and power supply use proprietary connections.  My system also has 32GB DDR4 3200 RAM (this is the max capacity and the RAM is matching).  I'm using a 1TB Samsung 980 SSD (PCIe Gen3).

3.  As stated, I do use a screen magnification program called Zoomtext, and I thought for sure this was the culprit, but even when this program is shut down, I still have these issues.

4.  I do use a program called Process Lasso - this program is supposed to prevent the system from slowing down.  Now that I think about this, maybe I should try uninstalling this to see if this is the culprit too.

P.S.  I do think it's interesting though that no other encoders with foobar cause this issue, don't you?  Or that Joshua's build and your 32 bit build work as expected.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Kraeved on 2024-04-24 13:27:02
Quote from: KevinB52379 on
System hang.  The file progress bar increases, but the files being encoded with foobar2000 never changes, and I never see the elapsed time, estimated time, encoding speed, and the 'pause" and "abort" buttons never appear.
You keep describing what the problem looks like in Foobar2000 and ignore the suggestion to test Helix MP3 encoder in the terminal. Start encoding one relatively large WAV file (e.g. merge songs from the same album into one WAV file) and describe what you see. The encoder outputs its own status report with various counters.

(https://i2.imageban.ru/out/2024/04/24/bd20a2afaae1e0195fcbfef831380cde.gif)

Quote from: JoshuaChang on
I've managed to compile the binary using visual studio 2022, it runs quite slow at my env
(about 1/3 speed compared to clang version), but you can test if it will hang your system
Code: [Select]
           Command         Mean [s]      Relative  
 ------------------ ---------------- ------------- 
     hmp3chang.exe    4.890 ± 0.021          1.00  
  hmp3chang_vc.exe   10.558 ± 0.041   2.16 ± 0.01
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-24 13:49:18
Quote from: JoshuaChang on
I've managed to compile the binary using visual studio 2022, it runs quite slow at my env(about 1/3 speed compared to clang version), but you can test if it will hang your system @KevinB52379 .


This version works perfectly too.  It's a little slower than the clang version you uploaded a few days ago, but it doesn't cause that system hang!

@JoshuaChang - so far your two binaries have worked as expected on my system, thank you!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-24 13:53:22
Quote from: Kraeved on
Quote from: KevinB52379 on
System hang.  The file progress bar increases, but the files being encoded with foobar2000 never changes, and I never see the elapsed time, estimated time, encoding speed, and the 'pause" and "abort" buttons never appear.
You keep describing what the problem looks like in Foobar2000 and ignore the suggestion to test Helix MP3 encoder in the terminal. Start encoding one relatively large WAV file (e.g. merge songs from the same album into one WAV file) and describe what you see. The encoder outputs its own status report with various counters.

I planned on trying this actually, it's just that new compiles keep coming in, so I tried them with my current encoding setup with foobar2000, that's all.

I am hoping I can try this process soon!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Wombat on 2024-04-24 14:17:55
Quote from: JoshuaChang on
I think i have the same problem, my cpu is zen2 4650g(6c12t), i got around 280~300x under command line @ single thread with autodidact's clang build or rarewares' gcc build, I also compiled helix myself with clang plus linktime optimization, my version got around 480x. maybe that's the key.
here's the binary, you can try it.
To be honest since i did read this i thought you have the same problem as @KevinB52379.
How does foobar multithreaded react on your system with the other builds?

As a sidenote: fast-math + AVX2 gives another nice boost to the fast clang compile.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: JoshuaChang on 2024-04-25 03:20:24
Quote from: Wombat on
Quote from: JoshuaChang on
I think i have the same problem, my cpu is zen2 4650g(6c12t), i got around 280~300x under command line @ single thread with autodidact's clang build or rarewares' gcc build, I also compiled helix myself with clang plus linktime optimization, my version got around 480x. maybe that's the key.
here's the binary, you can try it.
To be honest since i did read this i thought you have the same problem as @KevinB52379.
How does foobar multithreaded react on your system with the other builds?

As a sidenote: fast-math + AVX2 gives another nice boost to the fast clang compile.

Well, after test, I think I didn't have multithread problem like KevinB52379, my foobar2000 hmp3 batch convert have normal behavior like all other encoders(didn't slow down the system at all, my thread count is set to 0), different compilers seems just affect performance.
As I mentioned above, seems all microsoft link signature works for KevinB52379, and all mingw link signature doesn't.
You can find the signature using exeinfo pe, as can be found at github.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-25 14:05:19
Hi everyone!

@Wombat , @Case , @JoshuaChang

I figured out the issue!

The issue is totally on my end, and I apologize to everyone, and I appreciate everyone looking into this issue.

It turns out that the issue occurs with my screen magnification software, which is called ZoomText.

Closing down the software wasn't enough, I had to uninstall it completely, and then use the Windows built in magnifier so I could at least try encoding again.

This time @Case last binary he uploaded worked perfectly.

I had tried uninstalling Process Lasso, AVG Internet Security, and then because the issue seemed to be with the way things were updating on the screen so slowly (If not at all), I decided to try and uninstall Zoomtext.

Still, I wonder why @JoshuaChang version doesn't cause this issue, and his clang binary has been the fastest by far with @Case version being second (the latest 64 bit build he uploaded to this thread).

Anyway, I just wanted to say I'm so sorry for driving people crazy and have everyone go through hoops to try and help me troubleshoot the issue.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-25 14:49:58
Thanks for the update and happy to hear you got it solved. ZoomText sounds very invasive if it can cause the system to slow down. Would be fun to figure out what the problem is, but the software seems to be expensive and I don't even see demo option.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-25 15:05:34
Quote from: Case on
Thanks for the update and happy to hear you got it solved. ZoomText sounds very invasive if it can cause the system to slow down. Would be fun to figure out what the problem is, but the software seems to be expensive and I don't even see demo option.

@Case  - I mentioned before that I am legally blind, so Zoomtext is a vital piece of software for me to have in order to use the computer. 

You are correct, Zoomtext is expensive.  I purchase a Home license, which is a subscription based plan.  So I buy 5 years at a time for $500.  For me, it's wroth it.

You can download a trial version of Zoomtext, but the caveat is that it doesn't expire in a certain amount of days, but instead, it can only run 40 minutes at a time, and then the system must restart to reset the activation timer.

I am not quite sure why this is happening with Zoomtext.  To be honest, I've had other issues with other things and Zoomtext, and I'm usually told I'm the only one who reports such issues, or that the program I'm having issues with (in this case Helix) is NOT officially supported by the Zoomtext software.

I do know that Zoomtext mirrors the screen and obviously enlarges it.  So I believe there may be some type of mirroring driver installed so it interfaces with windows and the screen.  I know for a fact there are services running in the background.

My guess to as to why exiting Zoomtext didn't work, is because it isn't completely unloading everything in the background, so services, etc are still running.

Anyway, if you're interested in trying Zoomtext, it can be downloaded here:
https://software.vfo.digital/ZoomText/2024/2024.2402.66.400/5082DF51-83F9-4DC5-BE8B-545B0CCC1345/ZT2024.2402.66.400-Offline-x64.exe
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: Case on 2024-04-25 16:07:31
Thanks for the link. I installed ZoomText inside a virtualized Windows 11 and tried triggering the machine freeze, but I don't see any difference in behavior with different hmp3 compiles. Putting a conversion job running for my entire FLAC library keeps the machine fully usable and I can abort the task without issues. I tested both GCC and clang compiles for a few minutes.
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: KevinB52379 on 2024-04-25 16:25:08
Quote from: Case on
Thanks for the link. I installed ZoomText inside a virtualized Windows 11 and tried triggering the machine freeze, but I don't see any difference in behavior with different hmp3 compiles. Putting a conversion job running for my entire FLAC library keeps the machine fully usable and I can abort the task without issues. I tested both GCC and clang compiles for a few minutes.

@Chase - I appreciate you taking a look.  Honestly, I have no idea. 

The only builds that works are the clang versions and VS2022 helix builds from @JoshuaChang.

I also find it odd that I can encode with LAME, Vorbis, FAAC, FDKAAC, Opus, Musepack and speeds stay consistent with no system hang...it's just so weird to me.

If I want to use Helix, I guess I'm going to have to use @JoshuaChang versions if he would be willing to provide future builds.

I just love the speed and quality of Helix, but I could also use LAME in the future as well.

@Chase - Quick question, are there quality differences between the original Helix and the current 5.2.3 version?  I know LAME tags have been implemented, and gapless playback, right?  But was just curious if there are any quality/speed improvements?

Thank you again for trying to troubleshoot with Zoomtext, I am very appreciative of the time you took to try and help!
Title: Re: Resurrecting/Preserving the Helix MP3 encoder
Post by: maikmerten on 2024-04-25 17:39:06
I reverted my hacky "less printf" workaround on the dev branch, because a) it was implemented poorly and b) clearly wouldn't be effective if -D (displaying no progress update) doesn't alleviate the issue.

Case, do you feel that your patch is the way to proceed?

Text only | Text with Images