Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Resurrecting/Preserving the Helix MP3 encoder (Read 46755 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #200
Updated .exe and .exe with modifed source:

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case.zip

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip

Assuming these are 'trouble-free' over the next few days, I will then post on Rarewares.  Huge thanks to @Case for a lot of hard work. :)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #201
Issues

a) Sample: bee.wav (96 kHz 24 bit)
* Old Helix version says “UNSUPPORTED PCM FILE TYPE”, current version says “ENCODER INIT FAIL”.
* Let's make the error message more descriptive, e.g. “ERROR. Supported sampling rates are a, b, c, d.” or a - d.
* And if the input cannot be processed, Helix should not create a zero-length output file (e.g. TEST.MP3 0 bytes).

b) Samples: 24_bit_fixed.wav,  32_bit_float.wav (48 kHz 24 bit, 48 kHz 32 bit float, they are loud)
* Error: “UNRECOGNIZED PCM FILE TYPE”, but LAME encodes them fine.

c) Samples: leftright.wav, mcgill.wav (11 kHz 16 bit, 8 kHz 32 bit integer, attached below)
* Helix unexpectedly changes the sampling rate to 22.05 kHz and 16 kHz respectively.
* It is an inherited case, but can you tell, is it a bug or a feature?

d) Filename to verify Unicode support: birds-vögel-птицы-pájaros.wav (English, German, Russian, Spanish)
* Helix successfully creates birds-vögel-птицы-pájaros.mp3 on the disk, but its name is garbled on the output screen.
“PCM input file” and “MPEG output file” fields say:
birds-vЎgel-?????-pсjaros.wav (chcp 866)
birds-vцgel-?????-pбjaros.wav (chcp 1251)
birds-v�gel-?????-p�jaros.wav (chcp 65001)
It's a minor issue, more aesthetic one than practical, but I report it nonetheless.

Questions

* Any idea why Helix input was limited to 16 bit? Is there any chance for sound degradation due to the threshold being raised to 32 bit float? E.g. a modern lossy encoder FDKAAC also accepts 16 bit only.

* I divide Helix settings into 3 categories: major (e.g. bitrate choice, high frequency encoding, lowpass filter), minor (e.g. copyright and original bits), and mysterious. Let's try to understand the latter, since you've the skill to analyze the code.

Code: [Select]
A[algor_select]  0 = track input, 1=MPEG-1, 2=MPEG-2, xxxxx=sample_rate

U         u0=generic, u2=Pentium III(SSE)

Q         disable_taper, q0 = base, q1 = fast, q-1 = encoder chooses

TX        tx6, test reserved 6 or 8 seems best (startup_adjustNT1B)
            ** v5.0  TEST 1  as of 8/15/00
            ** v5.0  TEST 2  8/18/00
            ** v5.0  TEST 3  default tx6 (prev = tx8)
            ** v5.0  TEST 4  mods to short fnc_sf, ms corr. hf enable > 80
            ** v5.0  TEST 5  fix odd npart, ix clear
            ** v5.0  TEST 6  add reformatted frames
            ** v5.0  TEST 7  drop V4 amod
            ** v5.1  2005.08.09 (see CVS log for details)

SBT[short_block_threshold]
          short_block_threshold default = 700
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #202
I'll reply to a few points quickly.

a) Sample: bee.wav (96 kHz 24 bit)
* Old Helix version says “UNSUPPORTED PCM FILE TYPE”, current version says “ENCODER INIT FAIL”.
* Let's make the error message more descriptive, e.g. “ERROR. Supported sampling rates are a, b, c, d.” or a - d.
* And if the input cannot be processed, Helix should not create a zero-length output file (e.g. TEST.MP3 0 bytes).
I can improve this.

b) Samples: 24_bit_fixed.wav,  32_bit_float.wav (48 kHz 24 bit, 48 kHz 32 bit float, they are loud)
The samples have too large headers for the current code. I was actually expecting this issue to surface at some point. Thanks for the samples, I'll fix.

c) Samples: leftright.wav, mcgill.wav (11 kHz 16 bit, 8 kHz 32 bit integer, attached below)
* Helix unexpectedly changes the sampling rate to 22.05 kHz and 16 kHz respectively.
* It is an inherited case, but can you tell, is it a bug or a feature?
Feature. The engine seems to only handle rates 22050, 24000, 16000, 32000, 44100, 48000. Lower sampling rates are upsampled to nearest appropriate. There seems to also be downsampling support, not sure why its use is not allowed for higher sampling rates.

Edit: When the resampler is triggered the output length is way wrong and shifted. Not sure how motivated I am to fix this. Would be simpler to reject unsupported rates and let user resample. For example using foobar as frontend makes that super simple.

d) Filename to verify Unicode support: birds-vögel-птицы-pájaros.wav (English, German, Russian, Spanish)
* Helix successfully creates birds-vögel-птицы-pájaros.mp3 on the disk, but its name is garbled on the output screen.
This was a known issue, I didn't think mirroring the names is that important. But I can see if I can add unicode printing without too much work.

* Any idea why Helix input was limited to 16 bit? Is there any chance for sound degradation due to the threshold being raised to 32 bit float? E.g. a modern lossy encoder FDKAAC also accepts 16 bit only.
Helix encoder's history seems to date back to early 2000, several years before HD audio was even close to being popular or in wide use. It is possible offering support for higher bit depths wasn't deemed necessary. Also it may have seemed unnecessary as the quality of MP3 doesn't really exceed what 16 bit PCM can offer.

I'd think that removing the 16-bit clipping and restriction is very unlikely to cause sound degradation. For 16-bit signals the output remains the same. For floating point signals you can now encode data that is way beyond the digital range without it getting clipped, with the original encoder such signals would just be hard clipped noise.

For FDK though I believe the reason for not supporting beyond 16 bits is purely commercial. They have artificially limited the open source versions so that their commercial software is more attractive.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #203
c) Samples: leftright.wav, mcgill.wav (11 kHz 16 bit, 8 kHz 32 bit integer, attached below)
* Helix unexpectedly changes the sampling rate to 22.05 kHz and 16 kHz respectively.
Feature. The engine seems to only handle rates 16000 ... 48000 Hz. Lower sampling rates are upsampled to nearest appropriate. When the resampler is triggered the output length is way wrong and shifted. Not sure how motivated I am to fix this. Would be simpler to reject unsupported rates and let user resample.

Indeed, @Case. I test various resamplers (including not listed there, e.g. the one by Judd Niemann), but return to SoX, which is available as a standalone app and as a part of free apps such as Foobar2000. Since we are not sure how good Helix resampler is and adding a custom one such as r8brain is unlikely an easy task, let's stay conservative so far and throw a descriptive error message that the rate is too low or too high. For audiobooks 16 kHz is enough, even too low for my taste, and AMR-NB 8 kHz diary can be upsampled via SoX before piping to Helix*. I wonder why the original team (any contacts? I'd like to write them a letter) chose the upsampling route instead of LAME omnivorous one.

* ffmpeg -i in.amr -f wav -acodec pcm_f32le -af aresample=24000:resampler=soxr:precision=28 - | hmp3 - out.mp3 -V96
Yes, pcm_f32le. Thanks to you, we can now pass the input in 32 bit float format, reducing the likelihood of clipping,
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #204
There was actually a typo in the original sources. The code was always supposed to throw an error when sample rate was above 48 kHz, but the check was written to fail at 480 kHz.

Here's a test version without hardcoded limits to WAVE header size so the b) problem files can be encoded. MP3 file is no longer created when initialization fails. The error message telling that source file is unsupported will print additional information about each part that fails, be it number of channels, bit depth, WAVE type or sample rate.

No other fixes or improvements for now. I should be sleeping already.

 

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #205
No need to rush, @Case, give yourself time to reflect and recuperate. The state of lossy music encoding is disheartening, but the evolution of this very thread proves this is not a death sentence, but a task that can be solved. Let's go slow, but sure. Here, have a crêpe with jam, sour cream or caviar.

• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #206
Another day, another test build.
* Made encoder reject input tracks with zero samples.
* Added support for printing unicode filenames to the console.
* Fixed gapless data saving, the encoder tried to store too long numbers in padding info fields and the numbers got truncated as the storage space is only 12 bits long.
* Fixed internal resampling to produce output with correct lengths.

And from the yesterday's version:
* Fixed encoding WAV files with huge headers.
* Fixed writing empty track when input had unsupported sample rate.
* Added more detailed error message for input file rejection if the audio specs weren't supported.

I'll post source patch if nothing that needs immediate fixing is found.

Edit: forgot the wav header size fix.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #207
* Resampler. Have you decided not to reject the input with the sampling rate below 16 kHz? For example, Exhale (USAC converter) says “The sampling rate is 8 kHz but encoding using eSBR requires at least 22 kHz”.

* Gapless. You mentioned this word several times in this thread, but what exactly do you mean? In practical terms, I expect that there will be no click between the two files played sequentially, but for now there is a click when encoded even in CBR mode, albeit less pronounced than with, say, Opus.

* Input larger than 4 GB. I have not tested pipe encoding in this case, but WAV RF64/BWF are not recognized.

* Progress (0%) and compress ratio (0.020440%) counters are misleading when the input comes from pipe (sample: bee.wav). If it is impossible to display reliable information in this case, then it makes sense to display - or N/A.

Code: [Select]
$ ffmpeg -v error -i bee.wav -f wav -acodec pcm_f32le -af aresample=44100:resampler=soxr:precision=28 - | hmp3 -V96 - bee.mp3
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    1531  |   14100480 /     877870 |     0%   | 148.26 / 175.68  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 0.020440%

* How to understand “compress ratio” in general? For example, hmp3 utopia.wav (5824 → 1092 bytes, 29.323308%), but hmp3 sine.wav (21 168 044 → 239 774 bytes, 1.132719%).

* Inherited case with CBR. Flag -B stands for constant channel bitrate, so I expect every encoded frame to use that value. Unlike LAME, Helix does not display bitrate distribution, but mp3packer does, although I don't know how reliable it is. When the input frequency rate is less than or equal to 24 kHz, I see that one frame of Helix encoded output uses a lower bitrate. Is it a bug or a feature?

Code: [Select]
$ ffmpeg -v error -i bee.wav -f wav -acodec pcm_f32le -af aresample=24000:resampler=soxr:precision=28 - | hmp3 -B80 - bee.mp3
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    1670  |    7686144 /     801408 |     0%   | 159.96 / 160.00  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 0.018659%

$ mp3packer -i bee.mp3
1670 frames
Bitrate distribution:
  96: 1,0
 160: 1669,0
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #208
Seems that there's quite some flurry of activity over here, with some great improvement being developed by Case.

Wondering if the changes can be collected over at, e.g., https://github.com/maikmerten/hmp3 (or any other public repository)

Are these patches somewhere in a git branch?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #209
Reply #204, above, refers. ;)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #210
* Resampler. Have you decided not to reject the input with the sampling rate below 16 kHz?
I only considered disabling it as I thought fixing it is hopeless. But now that it appears to work nicely there is no reason to remove it.

* Gapless. You mentioned this word several times in this thread, but what exactly do you mean?
In this context "gapless" mean that the encoded length is 100 % correct without silence added or samples removed. Perfect audibly gapless playback requires more work, if it's at all possible when working with separate tracks. I may test extrapolation filter to see if that improves things.

* Input larger than 4 GB. I have not tested pipe encoding in this case, but WAV RF64/BWF are not recognized.
Not sure it's worth adding support for that. The Xing header where track frame counts and file size and gapless playback info is stored can only support 32 bit numbers. Things may overflow on too long inputs. Original code wasn't designed to handle anything over 2 GB, signed ints are used everywhere. I changed some parts to unsigned to properly handle up-to 4 GB, but had to revert some as you found those insane bitrate numbers with the calculations.

* Progress (0%) and compress ratio (0.020440%) counters are misleading when the input comes from pipe (sample: bee.wav). If it is impossible to display reliable information in this case, then it makes sense to display - or N/A.
Good point. Fixed. I also made display updating work when encoding from pipe.

* How to understand “compress ratio” in general?
The compression ratio is simply calculated by dividing the size of MP3 audio data by size of PCM data. That excludes MP3 and WAVE file headers and metadata.

* Inherited case with CBR. Flag -B stands for constant channel bitrate, so I expect every encoded frame to use that value. [...] When the input frequency rate is less than or equal to 24 kHz, I see that one frame of Helix encoded output uses a lower bitrate. Is it a bug or a feature?
That was a bug. Fixed.

Test binary + source patch attached.


Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #212
Ah, cool!

Just wondering: What codebase was your starting point? The code on GitHub (so the patches should apply cleanly) or some other snapshot?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #213
I used the sources bundled with the Rarewares release. I just verified and all the source files match with your git. The package from john33 just included proper Visual Studio solution files in addition.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #214
That's great, I'll try to merge this into a new branch, test stuff, then merge to main.

Is "Case" proper attribution?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #215
That is fine, unless you prefer to use real name. Then it would be Janne Hyvärinen.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #216
I guess in the commit message, I'd go for the real name, but in the source code, I'd go short with "Case", to avoid problems with character sets on legacy systems.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #217
Your patch applied wonderfully. Due to strange wording in RealNetwork's open-source license (which asks to always note the date of last change for files), I added the latest date of change to the file header.

Patched branch available at https://github.com/maikmerten/hmp3/tree/case - will merge to main after some tests on Linux (edit: To be clear, it compiles fine on Linux and also creates MP3 files, so much is known ;-) ).

Thank you very much, looking through the patch this indeed was a major rework - much more than I ever added to hmp3!

edit2: Also interested in getting a working Visual Studio build solution onto GitHub. Given I don't have Windows/Visual Studio to test things, some guidance is welcome.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #218
Your patch applied wonderfully. Due to strange wording in RealNetwork's open-source license (which asks to always note the date of last change for files), I added the latest date of change to the file header.

Patched branch available at https://github.com/maikmerten/hmp3/tree/case - will merge to main after some tests on Linux (edit: To be clear, it compiles fine on Linux and also creates MP3 files, so much is known ;-) ).

Thank you very much, looking through the patch this indeed was a major rework - much more than I ever added to hmp3!

edit2: Also interested in getting a working Visual Studio build solution onto GitHub. Given I don't have Windows/Visual Studio to test things, some guidance is welcome.
You will find the working VS project files in the hmp3 folder within the source included here:

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip

I need to emphasise that these are for Visual Studio 2015. As Case will attest, the masm code does not compile with later versions of VS.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #219
Present

@Case, changes to modernize Helix encoder that we worked on in this iteration were tested to the best of my ability on Windows 7 x64 in direct and pipe modes using files that differ in names, sampling rates, bit depth, channels and headers, including some malformed ones. Since the psychoacoustic model was barely changed, no detailed listening was conducted, but on the surface I detect no degradation or improvement of sound quality. As a final touch, I suggest polishing the interface. God knows, I prefer incremental changes to radical ones, but the current state of Helix screen is a spaghetti for nerds that needs to be redesigned. Today, we have enough command-line apps at our disposal to borrow interface details that facilitate accessibility and bring satisfaction to a wider range of users.

Ideas for a basic screen (that pops up without -h):
* consistent alignment, indentation, line spacing, Sentence case
* vernacular language if possible, sometimes less is more
* encoder's name: is it hmp3 or Helix or Xing?
* where to get the latest version, support, more information?
* options here should be major ones only (e.g. -V, -B, -HF2)
* what are the default values and valid ranges?

Spoiler (click to show/hide)

Future

* Demystifying Helix advanced options will give us an idea of how they affect quality and speed, how best to use them or even eliminate them altogether in order to reduce code complexity and interface confusion.

Side notes

* Comrade @danadam brought to my attention the following discussion of the difficulties in getting SoX to work with floating point. I especially remember the phrase and hope it's not related to Helix: “Simply changing sox_sample_t from int to float would for sure break a lot of things”.
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #220
edit2: Also interested in getting a working Visual Studio build solution onto GitHub. Given I don't have Windows/Visual Studio to test things, some guidance is welcome.

I need to emphasise that these are for Visual Studio 2015. As Case will attest, the masm code does not compile with later versions of VS.

Maybe look at setting up appveyor CI on github. It looks like they still support VS 2015 unlike github itself which only supports 2019/2022. It's free for open source projects.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #221
Regarding the encoder name: I went for " hmp3 MPEG Layer III audio encoder [...]  Utilizing the Helix MP3 encoder" as "Helix" and "Xing" are most likely still trademarked somewhere, so cannot be used as name for this particular encoder.

As for the "TX" advanced mystery options: The value specified for the TX option ends up in the aptly named variable "test1" of the struct "BA_CONTROL" (bit allocation control). Unlike the also present "test2" and "test3", "test1" actually appears to "do" something, in the method "CBitAllo3::startup_adjustNT1B":

Code: [Select]
void
CBitAllo3::startup_adjustNT1B (  )
{
    int ch, i, a, na;
    int ab, nab;
    int d, dmax;
    int f;
    static int sthres[21] = {
        0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0,
        100, 100, 100, 200, 300, 300, 300,
    };

//#define STHRES  300
//#define STHRES  0

    if ( ba_control.test1 == 0 )
        return;
    f = ba_control.test1;

    for ( ch = 0; ch < nchan; ch++ )
    {
        na = 1;
        a = 0;
        nab = 1;
        ab = 0;
        for ( i = 0; i < nsf[ch]; i++ )
        {
            //if( snr[ch][i] > STHRES ) {
            if ( snr[ch][i] > sthres[i] )
            {
                a += NT[ch][i];
                na++;
                ab += nBand_l[i] * NT[ch][i];
                nab += nBand_l[i];
            }
        }
        a = a / na;
        ab = ab / nab;

        if ( na < 5 )
            continue;

        for ( i = 0; i < nsf[ch]; i++ )
        {
            //if( snr[ch][i] > STHRES ) {
            if ( snr[ch][i] > sthres[i] )
            {
                dmax = HX_MAX ( snr[ch][i] - 400, 0 );
                //d = (ab - NT[ch][i]) >> 1;
                d = ( f * ( ab - NT[ch][i] ) ) >> 4;
                d = HX_MIN ( d, dmax );
                NT[ch][i] = NT[ch][i] + d;
            }
        }

    }

}

The value "test1" provides a value for the variable "f", which then further down the road is used as a multiplier to compute an adjusted noise threshold (?) for 21 scalefactor bands. As is a common theme with that code base, I find it hard to actually determine what is happening there (non-descriptive variable names, magic numbers), but "test1" defaults to 6.

Why 6? Because Neal said so! ("set -1 for default, test reserved 6 or 8 seems best, Neal says 6" and "default 6 re. Neal").


Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #222
You will find the working VS project files in the hmp3 folder within the source included here:

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip

I need to emphasise that these are for Visual Studio 2015. As Case will attest, the masm code does not compile with later versions of VS.

The relevant file is "mp3enc_asm.vcxproj", I assume?

There's also a "mp3enc_asm.vcproj" (without the x in the file extension), which appears unchanged from the version in git. Is it worthwhile to keep this file, or has it lost its usefulness?

What attribution do you like?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #223
You will find the working VS project files in the hmp3 folder within the source included here:

https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240317-Case%28with-source%29.zip

I need to emphasise that these are for Visual Studio 2015. As Case will attest, the masm code does not compile with later versions of VS.

The relevant file is "mp3enc_asm.vcxproj", I assume?

There's also a "mp3enc_asm.vcproj" (without the x in the file extension), which appears unchanged from the version in git. Is it worthwhile to keep this file, or has it lost its usefulness?

What attribution do you like?
You need the .sln, .vcproj and .vcxproj. Thanks for the offer, but I'm not sure I've done anything to warrant any attribution. ;)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #224
You need the .sln, .vcproj and .vcxproj. Thanks for the offer, but I'm not sure I've done anything to warrant any attribution. ;)

Thanks for elaborating! Build setups are important as well (code is otherwise useless), so if you don't mind, I'd include the project files with a "VS2015 project files provided by John" message.