Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Resurrecting/Preserving the Helix MP3 encoder (Read 109041 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Resurrecting/Preserving the Helix MP3 encoder

Hello there,

given that the original Helix MP3 Encoder thread (https://hydrogenaud.io/index.php/topic,35531.0.html) has been inactive for years and I'm not sure if resurrecting threads *that* old is welcome, I'm starting a fresh one. If this is in itself welcome, I hope mods can merge threads to keep things tidy. Sorry for any inconvenience.

Given my interest in Text-to-Speech, I've been looking into efficiently coding speech with MP3 as "works everywhere" fallback. Lame certainly is the obvious choice for a MP3 encoder to investigate, but I was eager to also look into the Helix MP3 encoder for its reputation of being fast and robust at lowish bitrates.

The Helix community project was closed many years ago, but an archive with apparently the newest (?)  source code was recently posted over at https://hydrogenaud.io/index.php/topic,122473.0.html thankfully. I'm on Linux and am interested in doing server-side MP3 encodes. With really minor source code modifications (about three lines of code) and a custom Makefile (attached to this post) I could get this old code to compile with only some minor warnings (which in the future I might be able to rectify).

Some observations and opinions:
  • The code not only compiles, it also works. Given I'm doing 64-bit compiles of ~20 years old code this was a pleasant surprise.
  • On my AMD Zen2 machine, Helix MP3 is about six times faster than Ubuntu's LAME compile, with apparently respectable quality. The clear performance advantage somewhat surprised me, given that I'm compiling the C-version codepath, without the ASM SSE bits.
  • Porting the old x86 ASM snippets to, e.g., SSE intrinsics might unlock additional speed. Even more interesting might be adding some ARM NEON code to have lighting-fast encodes on low-power single board computers.

I'm somewhat tempted to put the code on GitHub for preservation, future compile fixes, and code maintenance. However, while the code is open source, it comes with a custom RealNetworks license that reeks of "open source yes, but let's keep all control". I believe it is possible to publish the source code, but the license reads a bit awkward. Is there somebody here with insights into the actual intent there?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #1 –
I now published a version of the Helix MP3 Encoder, with some minor code maintenance having been done: https://github.com/maikmerten/hmp3

My main platform is Linux, but the original project files for Windows are still in there. I fixed some compiler warnings, fixed some typos in help texts and also enabled writing Xing TOC headers by default, so VBR files now seek properly. The Makefile also includes targets for debugging and profiling.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #2 –
I have also successfully compiled this, 32 and 64 bit, but using the C codepath. "sbt.asm" and "xsbt.asm" generate errors when compiling within VS2019 and trying to correct those is above my pay grade, I'm afraid! The other .asm modules appear to compile OK but, obviously, I can't know for sure without these other two compiling. Even without the asm code, the speed is very respectable. I can upload these if anyone is particularly interested. They are, of course, Windows compiles.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #3 –
Thank you, John, for looking into the Windows side of things!

I looked over the ASM files - those are good, old, plain 32-bit x86 and won't integrate into 64-bit compiles. However, compilers got a lot better, and I'm seeing GCC do some pretty neat stuff with SSE by its own. I'm profiling things here and there and my current approach to speed is to see if one can do faster approaches in pure C.

For instance, rounding floats to int in the original C code path was done as such:

Code: [Select]
    int ix;

    if ( x >= 0.0 )
        ix = ( int ) ( x + 0.5f );
    else
        ix = ( int ) ( x - 0.5f );
    return ix;

However, this introduces hard to predict branches. A faster way to get to the same result is

Code: [Select]
return (int)(x + copysignf(0.5f, x));

(copying over the sign from the float to be rounded onto the 0.5 constant - yielding -0.5 for negative values, then adding that offset before truncating to int.)

GCC generates some pretty neat, branch-free code from that, operating with SSE bit operations to get the job done. Fast!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #4 –
Well, the asm code compiles perfectly well in VS2015 for 32 bit. Not tried to compile for 64 bit, but would expect a fail.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #5 –
@john33 Can you provide a compiled Windows release for testing?
It's nice that there is some development on this codec. 😀

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #6 –
For any distribution of binaries, please include information on the license and where to get the source code. I guess having LICENSE.txt, RPSL.txt and RCSL.txt and a link to https://github.com/maikmerten/hmp3 in the archive with the binary should help being compliant with the licensing terms. Or just ZIP up the complete source code, license texts included.

Quote
(e) if You Deploy Covered Code in object code, executable form only, You must
include a prominent notice, in the code itself as well as in related
documentation, stating that Source Code of the Covered Code is available under
the terms of this License with information on how and where to obtain such
Source Code. You must also include the Object Code Notice set forth in Exhibit A
in the "about" box or other appropriate place where other copyright notices are
placed, including any packaging materials.

(That's not me trying to annoy anybody, but those are the terms RealNetworks released the code under.)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #7 –
Edit: err not so fast, Swine.
This RPSL.txt is a newer version than the one described at https://en.wikipedia.org/wiki/RealNetworks_Public_Source_License and that is classified as "free" by FSF/OSI.
And the differences are quite significant - there is a ton about "Research" purposes.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #8 –
Quote from: KrajzegaX on
@john33 Can you provide a compiled Windows release for testing?
It's nice that there is some development on this codec. 😀
Here you go:

https://www.rarewares.org/files/mp3/hmp3-main.zip

Slightly larger than required as source code, etc, is included with updated VS2015 project files. Executable is in the hmp3\Release folder.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #9 –
Quote from: Porcus on
Edit: err not so fast, Swine.
This RPSL.txt is a newer version than the one described at https://en.wikipedia.org/wiki/RealNetworks_Public_Source_License and that is classified as "free" by FSF/OSI.
And the differences are quite significant - there is a ton about "Research" purposes.

Hi Porcus, I think you may have confused RPSL.txt and RCSL.txt here.

RPSL.txt is the "proper" open-source license and, as far as I can see, matches https://opensource.org/licenses/RPSL-1.0 . It happens to contain the word "research" only once.

However, the RCSL.txt is a completely different beast and is not a "proper" open-source (in the sense of "libre") license:
Quote
(i) Source. Covered Code may be distributed in Source Code form only to
another Licensee (except for students as provided below).

Its main purpose appears to be "research use", with 24 instances of the word "research". Another use case of that license appears to be for commercial usage, for which additional terms may be agreed upon on top of the RCSL, as explained in the preamble.

Thankfully, LICENSE.txt makes clear that the awkward RCSL license only applies to those who actively sought to license the code in that fashion:

Quote
The contents of this directory, and (except where otherwise
 indicated) the directories included within this directory, are
 subject to the current version of the RealNetworks Public Source
 License (the "RPSL") available at RPSL.txt in this directory, unless
 you have licensed the directory under the current version of the
 RealNetworks Community Source License (the "RCSL") available at
 RCSL.txt in this directory, in which case the RCSL will apply.
(emphasis mine)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #10 –
Ah good. I just hurried to edit when I saw different version numbers and discrepancies, and I didn't to the checking properly. Sorry to anyone I have confused (that, hopefully, amounts to "myself").

Anyway, this means Helix is free software, just with a slightly more awkward "freedom comes with this note attached".

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #11 –
Quote from: Porcus on
Anyway, this means Helix is free software, just with a slightly more awkward "freedom comes with this note attached".

Yeah, it's free software as far as I can tell, but also a clear example of needless "license proliferation" (inflating the number of open source licenses in existence). I speculate that releasing the code under a more traditional license, such as the LGPL, may have made it easier to actually get a community started, instead of Helix "fizzling out".

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #12 –
I guess it has to do with what Real thought of themselves? If they are important enough, they set the terms - and Real was everywhere on the 'net twenty-five years ago.
Not excusing, explaining ... but from what I can read at https://en.wikipedia.org/wiki/Comparison_of_free_and_open-source_software_licenses , most FOSS licenses are actually newer.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #13 –
Yeah, I guess that's a realistic assessment. To be fair, albeit I wished they would have gone with a more popular/known license, getting a company of that size to open the sources for key products is an achievement in itself. Their software, their terms - and that's perfectly fine.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #14 –
Everytime i stumbled across the helix encoder i wondered if it is possible to hack in the gapless effort lame uses?
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #15 –
Quote from: Wombat on
Everytime i stumbled across the helix encoder i wondered if it is possible to hack in the gapless effort lame uses?

Well, should be pretty possible, I guess. Is gapless MP3 done using some tags expressing how many samples at the beginning and how many samples at the end shall be cut? Is there documentation somewhere?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #16 –
Quote from: maikmerten on
Well, should be pretty possible, I guess. Is gapless MP3 done using some tags expressing how many samples at the beginning and how many samples at the end shall be cut? Is there documentation somewhere?
I think that really is how it works. No idea where a complete documentation is.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #17 –
The values for encoder delay and padding are included in the 'lametag' added to the lame encoded mp3. The Helix encoder does not currently write such a tag.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #18 –
Quote from: john33 on
The values for encoder delay and padding are included in the 'lametag' added to the lame encoded mp3. The Helix encoder does not currently write such a tag.

Yeah, for Lame there's https://lame.sourceforge.io/tech-FAQ.txt

edit: even more relevant may be http://gabriel.mp3-tech.org/mp3infotag.html

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #19 –
There seems to be a bigger problem here, at least when using lame and foobar2000 as decoders, as both decode to fewer wave samples than in the original wave file on the file tested, although both to the same size. This suggests that the decoders are making delay/padding assumptions that don't hold true which is no great surprise but is a real problem for playback. Without a resolution, this makes the encoder unusable so far as I am concerned.

Edit: Actually, more worrying is that the encoder seems to 'throw samples away', or foobar properties is not reporting the correct number of samples. The encoded mp3 files show fewer samples that the original wave files and decoding to wav shows a further reduction in samples!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #20 –
Quote from: john33 on
both decode to fewer wave samples than in the original wave file on the file tested
Dumb question, but have you checked the actual wave files, not just fb2k's user output?
There have been instances where fb2k versions have reported wrong sample counts.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #21 –
Quote from: Porcus on
Quote from: john33 on
both decode to fewer wave samples than in the original wave file on the file tested
Dumb question, but have you checked the actual wave files, not just fb2k's user output?
There have been instances where fb2k versions have reported wrong sample counts.

No, not a dumb question at all. I hadn't appreciated that fb2k had that issue. Thanks for the heads-up. I will check but the issue appears consistent over all the files tested so far.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #22 –
Yeah, I'm seeing samples getting lost at the end as well. This might be an "off-by-one" error, where after reaching the end of the input file the last frame is not properly flushed to the output file or something of that sort. I also think that I'm seeing a difference depending on whether the Xing headers are written (-X2, Xing headers with TOC, the new default) or not (-X0 - no Xing headers).

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #23 –
I've just uploaded a fresh compile. It's here:

https://www.rarewares.org/files/mp3/hmp3-main.zip

The VS2015 project files have changed and there are a couple of preprocessor directives added which were in the old builds but somehow didn't get carried over.

The net result is that files encoded with this version seem to play in fb2k in gapless mode without any issue. I am currently listening to an album that runs continuously without gaps and all appears fine.

Let me know if there are any issues, but, otherwise, have fun. :)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #24 –
Hi John,

I guess the preprocess definitions happen to be the following?

Quote
<PreprocessorDefinitions>NDEBUG;WIN32;_M_IX86;IEEE_FLOAT;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>

(I don't see how those would lead to different behavior regarding sample counts, but perhaps things are pretty well hidden...)

Under Linux, with Wine, I think I'm still seeing samples being cut off at the end with your provided build. To test, I prepared a 1 second, 44.1 kHz (so exactly 44100 samples) mono WAV file, with a 441 Hz tone. To mark the beginning and the end, I reduced the volume for short sections (exactly three sine periods) at the beginning and the end ("sine441-notches.wav").

As can be seen in the enclosed screenshot, the Lame encode perfectly lines up with the original input file.

The Helix CBR encode is a bit longer then the original, but there's a considerable gap at the beginning. For VBR, the gap at the start appears to be identical, but there are fewer samples than in the input file. It doesn't seem to matter if I include a full Xing header (-X2) or none whatsoever (-X0).

Is there some kind of MP3 frame analyzer available somewhere? I very faintly remember Lame having had some kind of graphical analysis tool, but that's like 20 years ago, most likely written with GTK1 and most likely not operable nowadays if it hasn't seen updates. I'd love to see if the cut off samples are in the MP3 and decoders being a bit too eager to remove padding.