Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Resurrecting/Preserving the Helix MP3 encoder (Read 59897 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #425 –
Quote from: Kraeved on
@maikmerten Right now the build of @Case is the most relevant for 32-bit systems, and the build of @JoshuaChang delights HA users of 64-bit systems. In both cases, we still have no idea how their binaries are compiled from the code because the makefiles are not shared. Why? Err… Please? But new posts are coming and links to these builds will soon be lost among a dozen other testing, non-optimal, raw builds. I can't help but feel that it is not the order that is growing, but the mess. How can we trumpet the good news of the resurrection among our neighbors if we can't really point out where Helix MP3 encoder, this eighth wonder of the audio world, is? Even established search engines find it difficult to help here. Let's at least mention these builds on the HA wiki for a while, as it is done on the FhG AAC encoder page.


Fyodor Bronnikov — Pythagoreans celebrate the sunrise (1869)

I'm using msys2's clang build env, so gcc is an alias for clang, I just put -flto in CFLAGS_COMMON and put -static in LFLAGS, see the attachment for the exact file.
ps, I've also tested compile without -flto, no large performance loss(about 5%).

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #426 –
@KevinB52379 can you try this GCC build?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #427 –
Raspberry Pi 5, Clang 18 (-Ofast -flto -fopenacc)
276x (one core)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #428 –
Quote from: Case on
@KevinB52379 can you try this GCC build?

@Cas - I tried this version as you requested.

Result:
230x ~ 280x encoding speed.

All cores and threads utilized.

Again, system slow downs, system hang, foobar becomes nonresponsive.  Note: this slow down hang is differen than that which is expected from slow cpu load.

And as before, all cores aren't being utilized to 100% according to process lasso.

When I use your 32 bit version that you provided a few days ago and JoshuaChang 64 bit build, everything works perfectly.  No system hang/unresponsiveness, and full cpu utilization.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #429 –
Quote from: JoshuaChang on
I'm using msys2's clang build env, so gcc is an alias for clang, I just put -flto in CFLAGS_COMMON and put -static in LFLAGS, see the attachment for the exact file.
ps, I've also tested compile without -flto, no large performance loss(about 5%).
Thanks!
Unfortunately in a fresh installed msys2 using the clang64 console i only create non working binaries as if static never really works. I have to dig deeper on my side.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #430 –
One more post, sorry. Only a few packages were installed and already i blew it up in the dependencies.
I did msys2 once again and now it works and as fast as expected thanks to @JoshuaChang.
I still don't get why flto inside the clang64 enviroment gives that boost or if it is the repeated setting for flto in the linker flags. Have to play around more later i guess :)
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #431 –
Thank @KevinB52379. It was a desperate attempt, I saw somewhere suggestions that Ryzens would suffer from denormals more than Intels, that version set CPU flags that would round problematic denormal numbers to zero to prevent them from causing slowdowns.

Here's another test compile but this one you'll need to run manually in command prompt. It will print timing information at the end of the encoding so we'll see where all the time is spent.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #432 –
Quote from: Wombat on
One more post, sorry. Only a few packages were installed and already i blew it up in the dependencies.
I did msys2 once again and now it works and as fast as expected thanks to @JoshuaChang.
I still don't get why flto inside the clang64 enviroment gives that boost or if it is the repeated setting for flto in the linker flags. Have to play around more later i guess :)

I don't think it's related to lto, you can try without -flto, the performance loss is very little(on my side, it's about 5%)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #433 –
Quote from: JoshuaChang on
I don't think it's related to lto, you can try without -flto, the performance loss is very little(on my side, it's about 5%)
5% only due compiling is significant imho.
You can measure these 5% and run the non lto binary without problems on your 4650g? Guess i misunderstood it so far.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #434 –
I suspect that this may have something to do with the reduced amount of level 3 cache on the Ryzen APUs compared to the standard CPUs.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #435 –
I don't really think that the architecture of the AMD Zen APUs really is the main problem here. I'm on an AMD Ryzen Pro 4750G (Zen2-based APU), and here's a 64-bit native Linux compile converting a complete CD album (hmp3 -V100 -HF2)

Code: [Select]
real	0m9,241s
user	0m9,057s
sys	0m0,184s

Here's a 64-bit Windows compile provided by Case (the binary posted on 2024-04-21 07:32:57) on Linux, in Wine, same input, same parameters:

Code: [Select]
real	0m22,239s
user	0m11,178s
sys	0m3,972s

Is the Windows version somehow spamming system calls? The process seems to spend considerable time "elsewhere".

edit:

For completeness sake: My mingw64 win64-cross compile yields very similar results:

Code: [Select]
real	0m21,472s
user	0m10,772s
sys	0m3,836s

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #437 –
Quote from: Wombat on
For real completeness please run also the @JoshuaChang binary. https://hydrogenaud.io/index.php/topic,123331.msg1043232.html#msg1043232

That was indeed a worthwhile test. That binary, on Linux, in Wine, on my AMD APU:

Code: [Select]
real	0m11,342s
user	0m10,098s
sys	0m0,300s

That one seems to avoid spending a lot of time "elsewhere"...

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #438 –
Quote from: maikmerten on
That one seems to avoid spending a lot of time "elsewhere"...
Exactly. So far this big difference only happens on the AMD "g" APUs.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #439 –
For testing, I disabled the usual Spectre/Meltdown CPU security mitigations ("mitigations=off" as Linux boot parameter). This significantly improved performance.

Binary provided by Case *with* security mitigations (see my previous post):
Code: [Select]
real	0m22,239s
user	0m11,178s
sys	0m3,972s

Binary provided by Case *without* CPU security mitigations:
Code: [Select]
real	0m15,819s
user	0m10,493s
sys	0m1,701s

The CPU security mitigations impact the speed of system calls. For me, this indicates that the affected hmp3.exe binaries are somehow unusually syscall-heavy. This would also explain why the system would get "unresponsive", as reported.

(The native 64-bit Linux binary doesn't get faster when disabling the security mitigations.)

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #440 –
Since you have an affected system, care to test the compile I posted today? It ought to at least give a hint where the slowdowns happen.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #441 –
Quote from: Case on
Since you have an affected system, care to test the compile I posted today? It ought to at least give a hint where the slowdowns happen.

Gladly!

Code: [Select]
Time used:
main:     195.430800 ms
wav:      5.713400 ms
encode:   9885.130700 ms
xing:     189.984200 ms
file io:  191.137400 ms
print io: 10774.042400 ms
kbhit:    15.513600 ms
real	0m22,051s
user	0m11,398s
sys	0m3,645s

Spending more time on printing stuff than actual encoding?!

edit: And here's with -D flag (don't print progress):

Code: [Select]
Time used:
main:     197.418600 ms
wav:      6.954600 ms
encode:   9804.481700 ms
xing:     189.424400 ms
file io:  200.825100 ms
print io: 0.000000 ms
kbhit:    15.052200 ms
real	0m11,049s
user	0m10,044s
sys	0m0,233s

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #442 –
Status print. Adding -D should make it go much faster in that case. Could you compile a test version without the fflush(stderr)?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #443 –
Quote from: Case on
Status print. Adding -D should make it go much faster in that case. Could you compile a test version without the fflush(stderr)?

Removing the fflush(stderr) doesn't really improve stuff:

Code: [Select]
real	0m21,684s
user	0m10,887s
sys	0m3,854s

Commenting out the sprintf and fprintf does, however, ...

Code: [Select]
real	0m10,610s
user	0m9,634s
sys	0m0,181s

So, I guess the fix would be to only print stuff, e.g., every 100 milliseconds or so...

Mind if I engineer a fix?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #444 –
Thanks. That sounds like a good workaround for improving performance. But I'd like to know why the printing is so slow on those AMD systems. The GCC binary seems to use old functions from msvcrt.dll. Visual Studio compile is static and only calls Windows API functions externally. Clang compile calls functions from much newer Universal CRT.

Either way the msvcrt functions aren't slow on other machines...

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #445 –
Turns out it's surprisingly unnerving to determine the current time in milliseconds across platforms and across compiler-versions in C/C++, so here's my ugly little patch that cuts down the number of updates by counting:

https://github.com/maikmerten/hmp3/commit/fad361020b066bddaeaa2191022a70d8ba1e7eb9

(currently on the "dev" branch)

In my win64-in-Wine tests, this avoids the performance penality.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #446 –
I just tried this version.  The encoding speed with foobar is 800x...but the system still hangs and becomes unresponsive.

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #447 –
Quote from: KevinB52379 on
I just tried this version.  The encoding speed with foobar is 800x...but the system still hangs and becomes unresponsive.

Thanks for testing! Do the system hangs also happen with the -D (don't print progress) option?

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #448 –
I did some benchmarking and GCC's fprintf takes 100 times longer even here compared to MSVC's or clang's function.
Attached is a patch that changes progress printing to use WriteConsole on Windows when stderr isn't redirected to file. That gives consistently fast progress display on all compilers. I also noticed that stopping encoding by keyboard press didn't work when not compiled on Visual Studio, fixed that too.

I also changed the progress printing to happen 4 times less often than originally. I think maikmerten's progress update happened too seldom, it looked buggy when it was so sluggish.

I hope @KevinB52379 can test this and report if things now work correctly.

Edit: replaced with static compile.
 

Re: Resurrecting/Preserving the Helix MP3 encoder

Reply #449 –
Hmmm.

* in.wav, 99 423 788 bytes
* hmp3.case.x64, 513 024 bytes, xxhash 820182a208834328
* hmp3.chang.x64, 839 680 bytes, xxhash 4080008cf88453e9

Code: [Select]
$ hyperfine --warmup 3 --runs 3 -L encoder hmp3.case.x64,hmp3.chang.x64 "{encoder} -V150 -HF2 -F20000 in.wav out.mp3"

  Command          Mean [s]        Relative     
 ---------------- --------------- ------------- 
  hmp3.case.x64    6.860 ± 0.051   1.41 ± 0.01  
  hmp3.chang.x64   4.850 ± 0.019   1.00     
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data