Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Resurrecting/Preserving the Helix MP3 encoder (Read 74804 times) previous topic - next topic
0 Members and 15 Guests are viewing this topic.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #175 –
Quote from: Case on
I should learn to shut up sometimes. I just tested the Helix encoder and the latest version from Rarewares does write proper gapless info to the headers. But it seems to have some limitations or bugs as it doesn't do it when the data is fed through a pipe.

That's why I asked @Apesbrain exactly how he encodes. Yesterday I was looking for the optimal way to compress an audiobook and came across that Helix-encoded files sound different when switching from one to another if we abandon the pipe (-) in favor of temporary files (%s). I was inspired to try that by @KrajzegaX comment, which dates back to version 5.1.1. But I am afraid your Fake Gapless DSP is still required.

Quote
Detecting this type of error is super simple. Just look at file length as samples and you'll see the original and the encode have different lengths. Such files can never be gapless.

Hmm.
FLAC 01: 0:10.117 (485 603 samples), Musepack 01: 0:10.117 (446 148 samples)
FLAC 02: 0:05.447 (261 445 samples), Musepack 02: 0:05.447 (240 203 samples)
Yet gapless playback without DSPs. Perhaps you meant “never” applicable to MP3 format.
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #176 –
Quote from: john33 on
Quote from: Kraeved on
Quote from: Apesbrain on
Quote from: LedHed8 on
Lack of gapless encoding is the only drawback I know of in the Helix encoder.
I also noticed that Helix introduces sector boundary errors; LAME does not.

RareWares hosts a newer version of Helix than you used back in 2017, namely 5.2.1 2022-12-19.
@Kamedo2 @LedHed8 @Apesbrain Could you please check if there has been any improvement since you commented?


To the best of my recollection @maikmerten's work was about making both the encoded and decoded files more standards compliant and had nothing to do with tuning of the encoder. I stand to be corrected, but I believe that is the case.

That is indeed correct. I only added compile support for modern Linux and headers for (hopefully) gapless playback. No work on audio quality.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #177 –
Quote from: Kraeved on
Quote from: Case on
Detecting this type of error is super simple. Just look at file length as samples and you'll see the original and the encode have different lengths. Such files can never be gapless.

Hmm.
FLAC 01: 0:10.117 (485 603 samples), Musepack 01: 0:10.117 (446 148 samples)
FLAC 02: 0:05.447 (261 445 samples), Musepack 02: 0:05.447 (240 203 samples)
Yet gapless playback without DSPs. Perhaps you meant “never” applicable to MP3 format.
Ok I should have been more specific with my wording. Since this was about "Sector Boundary Error", thus about audio CDs, I kind of only thought about 44.1 kHz material. The sample count matching rule only applies when no resampling is involved.

Your example files show 48 kHz FLACs and 44.1 kHz Musepacks. Resampling has a possibility to introduce gapless issues of its own, though there are now nice resamplers for foobar2000 that extrapolate the signal to prevent that problem.

This is getting way off topic, but those gapless files from your example would need some adjustments to burn on audio CD. The FLACs of course are in wrong sample rate and would need to be resampled to 44100 Hz. And since the lengths don't align to sector boundaries, the track endings and beginnings would need to be shifted so that they align on sector boundaries. I believe foobar2000 Audio CD Writer component does this automatically for you.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #178 –
Here's a patch to hmp3 encoder (sources from Rarewares) to fix the pipe encoding bug. I also fixed one undefined behavior bug and initialized some variables so they aren't used with random data - these issues caused the encoder to freeze at least with newer compilers.
Unfortunately the asm code didn't want to compile, perhaps it needs an older MASM version, so I can't post an asm-enabled binary.
But I added a basic MSVC compile so any interested party can give it a try.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #179 –
@Case, thank you for trying to improve Helix MP3 encoder.
I believe the ASM part is crucial for speed, we need a hero to integrate it.

Code: [Select]
$ lptimer105.exe hmp3.exe storm.wav -V96

 PCM input file: storm.wav
MPEG ouput file: TEST.MP3
 pcm file:  channels = 2  bits = 16,  rate = 48000  type = 0
 Layer III   mode 1 STEREO   48000Hz  VBR-96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553984 |   100%   | 120.52 / 129.81  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455782%

Kernel Time  =     0.109 = 00:00:00.109 =    5%
User Time    =     1.341 = 00:00:01.341 =   71%
Process Time =     1.450 = 00:00:01.450 =   77%
Global Time  =     1.873 = 00:00:01.873 =  100%

$ lptimer105.exe hmp3case20240314a.exe storm.wav -V96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553984 |   100%   | 120.57 / 129.81  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455782%

Kernel Time  =     0.093 = 00:00:00.093 =    3%
User Time    =     1.965 = 00:00:01.965 =   75%
Process Time =     2.059 = 00:00:02.059 =   78%
Global Time  =     2.608 = 00:00:02.608 =  100%
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #180 –
And here you are: https://www.rarewares.org/files/mp3/hmp3-5.2.1-20240314-Case.zip

Visual Studio 2015 with MASM. :)

Edit: If we can confirm all is well (it works fine on the quick test that I did), I'll post it to Rarewares officially. And, I guess, @maikmerten would then probably like to update his git repository as well. :)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #181 –
Thank you. The encoder seems to be just as fast as previous build from you and in my testing I get bit-identical output from piped encoding in foobar and from the same source wav using old binary on command line. But you'll probably want a confirmation from an unbiased third party.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #182 –
I see the ASM part is back, as is the speed.
Kudos to @john33, there's plenty of pep in you yet.

Code: [Select]
 $ lptimer105.exe hmp3case20240314b.exe storm.wav -V96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553984 |   100%   | 120.52 / 129.81  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455782%

Kernel Time  =     0.093 = 00:00:00.093 =    4%
User Time    =     1.419 = 00:00:01.419 =   75%
Process Time =     1.513 = 00:00:01.513 =   80%
Global Time  =     1.888 = 00:00:01.888 =  100%

Now let me invoke hmp3case20240314b.exe in three ways:
a) via direct call in the command-line shell
b) via Foobar2000 with a temporary file (%s)
c) via Foobar2000 with a pipe (-)

$ for %# in (*.mp3) do @echo %~z# = %#
2553984 = storm-direct.mp3
2553984 = storm-pipe.mp3
2553984 = storm-tempfile.mp3

$ b3sum *.mp3
424f76a649f35de484c8980a50fd13675dd17a702f0eaa60bed8ba5e5b48bc93  storm-direct.mp3
fe0dd60dcf3e5341631850f4b6a3a2444b5d34a9b19fda1022c287598366f5ec  storm-tempfile.mp3
fe0dd60dcf3e5341631850f4b6a3a2444b5d34a9b19fda1022c287598366f5ec  storm-pipe.mp3

What is the difference between direct and the other two?
Since MediaInfo and ExifTool reported nothing special, let's dig a bit deeper.

$ for %# in (*.mp3) do mp3packer -i %# | rg -oA999 INFO: > %~n#.txt
$ fc /N storm-direct.txt storm-pipe.txt

***** storm-direct.txt
    9:   18516192 bits of payload data (117.625858 kbps)
   10:   2317405 bytes of payload data (117.772272 kbps)
   11:   23048 bits wasted from partially-full bytes (0.146415 kbps)
   12:   2553529 bytes of MP3 data (129.772272 kbps) = minimum bitrate possible
   13:   455 bytes of padding (0.023123 kbps)
***** storm-pipe.txt
    9:   18516087 bits of payload data (117.625191 kbps)
   10:   2317392 bytes of payload data (117.771612 kbps)
   11:   23049 bits wasted from partially-full bytes (0.146421 kbps)
   12:   2553516 bytes of MP3 data (129.771612 kbps) = minimum bitrate possible
   13:   468 bytes of padding (0.023784 kbps)
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #183 –
I assume there is something special about the WAV file that causes the difference. When you encode with foobar2000, the encoder doesn't see the original wav but a new fresh temporary WAV created only from the audio bits. If you shared the source track I could take a look at some point.
 

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #184 –
@Case, I hope you'll understand why the source storm.wv¹ is compressed with WavPack. My trust in FLAC has been shaken recently due to its inability to reliably preserve non-audio data.

By the way, since you took a peek at the Helix code, is it difficult to raise the bit depth threshold of incoming data from the current 16-bit to 32-bit floating-point, as LAME team did?

¹ SHA256: 6516c87b4d81851b59b5d0ca54944f135bf04a2b3014a58862a8fb815588a6ad
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #185 –
The hmp3 frontend handles the wav incorrectly. The encoded file has gained 14 extra samples that should not be there. I haven't yet looked at why, busy with actual work.

Just to clarify, that issue has nothing to do with my pipe encoding fix. The original encoder produces bit-identical output when not using pipes.

My intention was to see if the frontend is worth improving by making it at least support basic input bit depths (24 and 32-bit float) and fix file accesses to use 64-bit interfaces. Now it can't handle source files over 2 GB correctly. But that is something potential stuff for the future, if this thing has any users.

PS. I can decode WavPack, thanks. Though as I'm not a musician I have zero need to keep non-audio information in audio files, other than tags.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #186 –
I think I must be missing something here.

I have decoded 'storm.wv' to wav and encoded it, on the command line, using hmp3 at default VBR;
I have encoded the 'wv' file, in foobar, using the same default, above;
I have done the same with the wav file;
I have decoded the file encoded from the .wv file and ALL have exactly the same duration and number of samples!

What am I missing?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #187 –
Quote from: john33 on
I have decoded 'storm.wv' to wav and encoded it, on the command line, using hmp3 at default VBR;
Did you decode it with foobar2000? If you did, it should have created a lovely fresh WAV without the issue. Use wvunpack.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #188 –
Quote from: Case on
Quote from: john33 on
I have decoded 'storm.wv' to wav and encoded it, on the command line, using hmp3 at default VBR;
Did you decode it with foobar2000? If you did, it should have created a lovely fresh WAV without the issue. Use wvunpack.
Yes, I did. Obviously what I was missing! I'll try again. ;)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #189 –
Examined the situation. The file has 56 bytes of extra at the end which the encoder happily took in. I changed the encoder to only encode as much data as data chunk's size indicates.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #190 –
I just looked at the files in a hex editor and came to the same conclusion, which is great! Have you amended the patch file linked to above?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #191 –
Here's an updated patch to the original Rarewares sources that fixes the pipe encoding + unitialized variables + prevents encoding beyond indicated data section size.

And it looks like support for higher bitdepths could be added. It appears this encoder converts the data to 32-bit floating point for processing anyway.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #193 –
Code: [Select]
$ hmp3case20240315a.exe storm.wav storm-direct.mp3 -V96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553984 |   100%   | 120.52 / 129.81  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455810%

$ b3sum *.mp3
d92f158207015b10480f35f6c08a8e2dff550a5c6adc320588f1880687a534a1  storm-direct.mp3
dbbd7bec1b2275315e0e7222f09724e09523bd5f9f40141c2dac4f0050d85380  storm-pipe.mp3
dbbd7bec1b2275315e0e7222f09724e09523bd5f9f40141c2dac4f0050d85380  storm-tempfile.mp3

Sorry, but storm-direct.mp3 checksum is still different on my end.
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #194 –
So it seems. Some patches are not in. Here's my compile, I dug up old version of Microsoft's assembler.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #195 –
Code: [Select]
$ hmp3case20240316a.exe storm.wav storm-direct.mp3 -V96
-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
    6559  |   30219264 /    2553936 |   100%   | 120.44 / 129.80  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 8.455651%

$ b3sum *.mp3
3090e261c6eb89ab813029e9f0756b01d7fabfb50aba513dc56fcb56d1199aba  storm-direct.mp3
3090e261c6eb89ab813029e9f0756b01d7fabfb50aba513dc56fcb56d1199aba  storm-pipe.mp3
3090e261c6eb89ab813029e9f0756b01d7fabfb50aba513dc56fcb56d1199aba  storm-tempfile.mp3

Well done, @Case.
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #197 –
I added support for handling WAV files in WAVE_FORMAT_EXTENSIBLE format, 24-bit and 32-bit input support. 32-bit handles both integer and float. And I added unicode file name support. I performed basic tests, like confirmed that 16-bit input produces bit-perfect output with standard version. The tool can now encode @Kraeved 's dolle.wav from command line.

I'll upload source patch soon if no bugs are found.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #198 –
Progress and bitrate counters have gone crazy: -6116.64, Kbpsss, 101%.

Sample: dolle.wav

Code: [Select]
$ hmp3 dolle.wav -V96
 pcm file:  channels = 1  bits = 24,  rate = 44100  type = 1
 Layer III   MONO   44100Hz  VBR-96

-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
     883  |    3048192 /     233881 |   100%   | -6116.64 /  81.15  Kbpsss
-------------------------------------------------------------------------------
 Compress Ratio 7.702456%

Sample: bestia.wav

Code: [Select]
$ hmp3 bestia.wav -V96
 pcm file:  channels = 2  bits = 24,  rate = 44100  type = 1
 Layer III   mode 1 STEREO   44100Hz  VBR-96

-------------------------------------------------------------------------------
  Frames  |  Bytes In  /  Bytes Out | Progress | Current/Average Bitrate
     388  |    2674944 /     184623 |   101%   | -2496876.96 / 145.94  Kbps
-------------------------------------------------------------------------------
 Compress Ratio 6.973421%
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #199 –
Thanks for the bug report, I had not seen those.
Fixed the problems @Kraeved found. And since I needed test material to see if new bitdepths work on big-endian I had to add support for big-endian RIFX WAVE files.

Attached my compile + patch.
To summarize, the patch includes the following changes:
* fixed pipe encoding missing the beginning of the audio and breaking gaplessness.
* fixed the encoder to not read data beyond data chunk.
* added support for 24 bit integer and 32 bit integer and floating point input data.
* added Unicode filename support for input and output files.
* added support for WAVE_FORMAT_EXTENSIBLE file format and RIFX big endian WAVE files.