Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Resurrecting/Preserving the Helix MP3 encoder (Read 45166 times) previous topic - next topic
0 Members and 3 Guests are viewing this topic.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #450
I did some benchmarking and GCC's fprintf takes 100 times longer even here compared to MSVC's or clang's function.
Attached is a patch that changes progress printing to use WriteConsole on Windows when stderr isn't redirected to file. That gives consistently fast progress display on all compilers. I also noticed that stopping encoding by keyboard press didn't work when not compiled on Visual Studio, fixed that too.

I also changed the progress printing to happen 4 times less often than originally. I think maikmerten's progress update happened too seldom, it looked buggy when it was so sluggish.

I hope @KevinB52379 can test this and report if things now work correctly.


I think you might forgot to add -static in the LFLAGS since it ask for libstdc++-6.dll in my env.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #451
Thanks for noticing. fixed.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #452
Hmmm.
[...]
Code: [Select]
  hmp3.case.x64    6.860 ± 0.051   1.41 ± 0.01  
  hmp3.chang.x64   4.850 ± 0.019   1.00     
Note that the intention is to find and solve the abnormal system slowdown KevinB reported. His problem is in no way related to relative performance of different compiles.

Though I do wonder how JoshuaChang's clang compile is so fast. Usually GCC has been able to produce faster binaries. And if I compile a clang build with joshua's makefile, I get a slower binary than my GCC compile.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #453
I think that might be related to the compile's link stage, my clang binary under exeinfo shows a microsoft signature code, and all other gcc binaries and clang binaries shows a mingw signature code

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #454
I did some benchmarking and GCC's fprintf takes 100 times longer even here compared to MSVC's or clang's function.
Attached is a patch that changes progress printing to use WriteConsole on Windows when stderr isn't redirected to file. That gives consistently fast progress display on all compilers. I also noticed that stopping encoding by keyboard press didn't work when not compiled on Visual Studio, fixed that too.

I also changed the progress printing to happen 4 times less often than originally. I think maikmerten's progress update happened too seldom, it looked buggy when it was so sluggish.

I hope @KevinB52379 can test this and report if things now work correctly.

Edit: replaced with static compile.

EDIT:  I tried the -D switch and that didn't improve anything.

Also, please note, I am able to use FAAC, LAME, Vorbis, Opus from rarewares with all thrads going with foobar2000 and I have fast encoding speeds and no system hang.  Yes, I know there are slowdowns because all the cores are being used, but this system hang I'm experiencing with Helix is totally different.
I tried this version.

At first it started off promising, encoding started at around 1300x, but then about 20 seconds or so in, things started to slow down.  They system started to hang, things became very unresponsive.

When encoding first started with foobar it started about 1400x to 1500x and then slowed down drastically.

In fact, in foobar the file list of what was currently being encoded stopped updating, the time elapsed stopped updating, the estimated time complete incrased and then stopped updating.  The only thing that was changing, although very slowly was the progress bar of the encoding task.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #455
@KevinB52379, what if you discard Foobar2000 as an intermediary for a while and encode right in the terminal using Hyperfine, a cross-platform tool for warming up the system and measuring the execution time of the binaries? For example, @misio discovered and I verified that visualization via VST plug-ins works much slower in Foobar2000 than the same plug-ins in audio editors (DAWs), and surprisingly, interaction with WavPack files turned out to be the slowest.
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #456
If you don't mind, you could install Windows Performance Toolkit and use Windows Performance Recorder to record what goes on with the machine when encoding. You can enable all options under Resource Analysis and you can stop recording when the system goes slow. If you share the recording it should reveal why your machine acts so badly.

@Kraeved - I already tried to reply in the related topic why VSTs aren't smooth. But let me clarify. The VSTs are given big blobs of audio data to process at once, the exact size depends on design of decoder components and what other DSPs may be before them. The janky VSTs apparently have trouble updating their UIs when dealing with such data. All buffers in foobar2000 are big to support smooth and glitchless audio playback. DAWs have tiny buffers to work as realtime as possible.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #457
@Case - I would try that toolkit, but the problem is, once the system starts to hang, I can't do anything.  For example, that start menu button doesn't respond, and when I hit "abort" on foobar, it can take several minutes for it to finally abort (if it even stops at all).  There have been times I have to let it finish encoding the process because I can't cancel it.

I do find it odd that the cpu utilization doesn't even go to 90 - 100% usage like it does with your 32 bit binary from a few days ago and @JoshuaChang version does.  So far those are the only 2 builds that work properly.

Oh, I also am legally blind and use Screen Magnification software, but this program ISN'T the cause of the problem, because even when I shut it down before encoding, the system still hangs.
Let me look into it.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #458
Though I do wonder how JoshuaChang's clang compile is so fast. Usually GCC has been able to produce faster binaries. And if I compile a clang build with joshua's makefile, I get a slower binary than my GCC compile.
Clang 18 in the msys2 clang64 enviroment creates these fast binaries for hmp3 here also. Seems the code reacts perfectly to this enviroment.
It is not a magic bullet since flac is still faster with a standard gcc compile in my old mingw64 enviroment.

@Case already mentioned clang64 uses ucrt in this case https://www.msys2.org/docs/environments/
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #459
@KevinB52379, what if you discard Foobar2000 as an intermediary for a while and encode right in the terminal using Hyperfine, a cross-platform tool for warming up the system and measuring the execution time of the binaries? For example, @misio discovered and I verified that visualization via VST plug-ins works much slower in Foobar2000 than the same plug-ins in audio editors (DAWs), and surprisingly, interaction with WavPack files turned out to be the slowest.

How come when I click your Hyperfire link it brings me back to this hydorgenaudio Helix thread?

Can you provide a download link to try this? 

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #460
How come when I click your Hyperfire link it brings me back to this hydorgenaudio Helix thread?
Can you provide a download link to try this? 

The link in that message takes you to an example of how to use Hyperfine, so that you don't have to go through the user's guide in search of the proper commands. To date, the latest version available for download on Github is 1.18.

When troubleshooting, the first thing I do is look for a minimal reproducible example, i.e. I simplify the chain of events that lead to the problem. By measuring the audio encoding using several builds of Helix MP3 encoder in the terminal, we will know for sure whether Foobar2000 affects the slowdown in any way or not. If there is a comparable slowdown to the point that the system becomes unresponsive in the terminal as well, you can continue to encode via Foobar2000. Also, when the encoder outputs its report in the terminal, you may witness phenomena like counters going crazy, which will give the developers a clue on how to fix things.
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #461
Don't confuse @KevinB52379 to much.
Old hmp3 versions work, @JoshuaChang version works within foobar.
A solution at the source is worthwile in this case.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #462
Don't confuse @KevinB52379 to much.
Old hmp3 versions work, @JoshuaChang version works within foobar.
A solution at the source is worthwile in this case.

I was willing to give Hyperfire a shot...but to be honest I thought the same thing as you. 

If the original version of Helix 64 bit works from ReallyRareWares, and @JoshaChan version works perfectly with foobar...then something else is the issue.

Also as I mentioned, I've used LAME (both 32 bit and 64 bit), FAAC 1.30, OPUS, Ogg Vorbis AoTuV, FDKAAC - all from RareWares, and they all behave normally, so I don't think it's a temperature/overheating issue.  This is with foobar and all cores/threads being used.  Perfect fine speed and system responsiveness.

I do think it is interesting that CPU utilization never reaches 100% like it does with Joshua's version, old Helix and Case's 32 bit binary.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #463
Here's a test build with GCC linked against UCRT. All the compiles you have tried that have shown issues have used MSVCRT (the GCC builds and the clang build from @autodidact). The compiles that have worked have either been fully static or have been linked against the Universal CRT.

Not sure how useful this test is, but at least it's another data point. Publishing encoder binaries requiring UCRT isn't really an option in my opinion, that runtime is present by default only on Windows 10 and newer.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #464
@Case  - Thanks for this compile, but unfortunately, same results.  System hang.  The file progress bar increases, but the files being encoded with foobar2000 never changes, and I never see the elapsed time, estimated time, encoding speed, and the 'pause" and "abort" buttons never appear.  The only thing that changes is the progress bar.  It didn't take over an hour to encode like all the other slow compiles...but it was no where near as fast as @JoshuaChang version.  His version encodes over 5,000 flac files in under 9 minutes with no system hangs or slowdowns.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #465
Good to know. There are no longer many options that can cause the issue you are facing. I think antivirus or other security product that scans or sandboxes executables is the only thing that can do this.
The converter in foobar2000 uses very low priority by default and you said you haven't increased it. These encoders don't touch priorities so they do their tasks with leftover cycles. Memory consumption is no different between compiles.

You must have some high priority process(es) running on the same core as the UI processes and steal all the processing power from the UI operations. Or a sandbox goes haywire and consumes all the system memory. Swapping will make machine slow even with SSD.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #466
I've managed to compile the binary using visual studio 2022, it runs quite slow at my env(about 1/3 speed compared to clang version), but you can test if it will hang your system @KevinB52379 .

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #467
Good to know. There are no longer many options that can cause the issue you are facing. I think antivirus or other security product that scans or sandboxes executables is the only thing that can do this.
The converter in foobar2000 uses very low priority by default and you said you haven't increased it. These encoders don't touch priorities so they do their tasks with leftover cycles. Memory consumption is no different between compiles.

You must have some high priority process(es) running on the same core as the UI processes and steal all the processing power from the UI operations. Or a sandbox goes haywire and consumes all the system memory. Swapping will make machine slow even with SSD.

Hi @Case - I can provide some more information if you'd like.

1.  I use AVG Internet Security as my security suite.  In AVG I have excluded my foobar2000 directory (I use foobar in portable mode).  At one point, I even disabled realtime protection before encoding to see if this helped, nothing changed.  Would you like me to try uinstalling AVG?

2.  As I stated, I have an HP Pavilion with AMD Ryzen 7 5700g, and I'm using the integrated graphics.  I cannot install a dedicated graphics card as the system is limited to a 180 watt PSU and the motherboard and power supply use proprietary connections.  My system also has 32GB DDR4 3200 RAM (this is the max capacity and the RAM is matching).  I'm using a 1TB Samsung 980 SSD (PCIe Gen3).

3.  As stated, I do use a screen magnification program called Zoomtext, and I thought for sure this was the culprit, but even when this program is shut down, I still have these issues.

4.  I do use a program called Process Lasso - this program is supposed to prevent the system from slowing down.  Now that I think about this, maybe I should try uninstalling this to see if this is the culprit too.

P.S.  I do think it's interesting though that no other encoders with foobar cause this issue, don't you?  Or that Joshua's build and your 32 bit build work as expected.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #468
System hang.  The file progress bar increases, but the files being encoded with foobar2000 never changes, and I never see the elapsed time, estimated time, encoding speed, and the 'pause" and "abort" buttons never appear.
You keep describing what the problem looks like in Foobar2000 and ignore the suggestion to test Helix MP3 encoder in the terminal. Start encoding one relatively large WAV file (e.g. merge songs from the same album into one WAV file) and describe what you see. The encoder outputs its own status report with various counters.



I've managed to compile the binary using visual studio 2022, it runs quite slow at my env
(about 1/3 speed compared to clang version), but you can test if it will hang your system
Code: [Select]
           Command         Mean [s]      Relative  
 ------------------ ---------------- -------------
     hmp3chang.exe    4.890 ± 0.021          1.00 
  hmp3chang_vc.exe   10.558 ± 0.041   2.16 ± 0.01
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #469
I've managed to compile the binary using visual studio 2022, it runs quite slow at my env(about 1/3 speed compared to clang version), but you can test if it will hang your system @KevinB52379 .


This version works perfectly too.  It's a little slower than the clang version you uploaded a few days ago, but it doesn't cause that system hang!

@JoshuaChang - so far your two binaries have worked as expected on my system, thank you!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #470
System hang.  The file progress bar increases, but the files being encoded with foobar2000 never changes, and I never see the elapsed time, estimated time, encoding speed, and the 'pause" and "abort" buttons never appear.
You keep describing what the problem looks like in Foobar2000 and ignore the suggestion to test Helix MP3 encoder in the terminal. Start encoding one relatively large WAV file (e.g. merge songs from the same album into one WAV file) and describe what you see. The encoder outputs its own status report with various counters.

I planned on trying this actually, it's just that new compiles keep coming in, so I tried them with my current encoding setup with foobar2000, that's all.

I am hoping I can try this process soon!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #471
I think i have the same problem, my cpu is zen2 4650g(6c12t), i got around 280~300x under command line @ single thread with autodidact's clang build or rarewares' gcc build, I also compiled helix myself with clang plus linktime optimization, my version got around 480x. maybe that's the key.
here's the binary, you can try it.
To be honest since i did read this i thought you have the same problem as @KevinB52379.
How does foobar multithreaded react on your system with the other builds?

As a sidenote: fast-math + AVX2 gives another nice boost to the fast clang compile.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

 

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #472
I think i have the same problem, my cpu is zen2 4650g(6c12t), i got around 280~300x under command line @ single thread with autodidact's clang build or rarewares' gcc build, I also compiled helix myself with clang plus linktime optimization, my version got around 480x. maybe that's the key.
here's the binary, you can try it.
To be honest since i did read this i thought you have the same problem as @KevinB52379.
How does foobar multithreaded react on your system with the other builds?

As a sidenote: fast-math + AVX2 gives another nice boost to the fast clang compile.

Well, after test, I think I didn't have multithread problem like KevinB52379, my foobar2000 hmp3 batch convert have normal behavior like all other encoders(didn't slow down the system at all, my thread count is set to 0), different compilers seems just affect performance.
As I mentioned above, seems all microsoft link signature works for KevinB52379, and all mingw link signature doesn't.
You can find the signature using exeinfo pe, as can be found at github.