Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Speeding up codecs with faster CRC calculations (Read 39736 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Speeding up codecs with faster CRC calculations

Reply #25
@Enzo:
Just ran my test set of FLACs again with John33's ICL18 x64 compile from 18.11.2018.
It took 68.852 sec compared to 70.646 sec with an ICL17 x64 compile from October 2017 (before the CRC patch).

That said, the compile you provided is still the fastest one (66.794 sec) on my i5-2400...

 

Re: Speeding up codecs with faster CRC calculations

Reply #26
That said, the compile you provided is still the fastest one (66.794 sec) on my i5-2400...
Might be due to my build using --enable-64-bit-words which is a bit faster on x64. Not sure if John33's builds have that option enabled.

Re: Speeding up codecs with faster CRC calculations

Reply #27
Quote
Built for Win64 with GCC 4.9.2 and -O3 -march=nehalem -funroll-loops. FLAC is configured with --enable-64-bit-words

Since your build is still the fastest out there:
Did you try to compile the current FLAC version 1.33 with your settings?

Maybe I could try to set up the current version of GCC's C++ compiler and use your settings. You got any tips there?
What's the version you are using atm?

Re: Speeding up codecs with faster CRC calculations

Reply #28
Did you try to compile the current FLAC version 1.33 with your settings?
I have compiled the new release and attached it here. Used the same settings as above and it's even slightly (~1.5%) faster than the fastcrc preview build (likely due to other optimizations added since 1.3.2).

If you'd like to try building it yourself:

I'm using my custom fre:ac Component Development Kit for compiling. It's still based on GCC 4.9.2 as I plan to update to a newer version only after the fre:ac 1.1 release. The toolkit can be found here: https://github.com/enzo1982/BoCA/releases

The script for building libogg and flac can be found here: https://github.com/enzo1982/freac/blob/ebbe9cb95b507617b5d3cd205329337be80a8012/tools/build-codecs
Changes for this build are replacing -march=nocona with -march=nehalem and removing the -mno-sse3 option in line 27 as well as adding --disable-shared in line 302.

Re: Speeding up codecs with faster CRC calculations

Reply #29
Hi enzo,

thank you so much for your build of FLAV v1.33. Your settings are AWESOME!
I tested it with my set of 40 WAV files (~ 1.9 GB) and compared it to your fastcrc version of v1.32 and Wombat's VS 2017 compile.
I used an i7-6700 @ 3.4 GHz, the WAVs stored on a Samsung EVO 850 SSD. Timings were done with Igor Pavlov's timer64.exe.

In this configuration, the difference between your fastcrc and the 1.33 build is marginal, with a tiny advantage for the 1.33 version, but that's within the range of measurement inaccuracies. But compared to the VS build, the difference is REMARKABLE:

Code: [Select]
flac132-fastcrc.exe (enzo):
Kernel  Time =     2.609 =    6%
User    Time =    35.421 =   89%
Process Time =    38.031 =   96%    Virtual  Memory =     14 MB
Global  Time =    39.359 =  100%    Physical Memory =     14 MB

flac133_enzo.exe:
Kernel  Time =     2.234 =    5%
User    Time =    35.750 =   90%
Process Time =    37.984 =   96%    Virtual  Memory =     14 MB
Global  Time =    39.342 =  100%    Physical Memory =     17 MB

flac133_vs2017.exe:
Kernel  Time =     2.796 =    6%
User    Time =    39.218 =   89%
Process Time =    42.015 =   95%    Virtual  Memory =     14 MB
Global  Time =    43.897 =  100%    Physical Memory =     16 MB

Thanks also for the instructions to set up the toolchain for GCC. Will try that on the next rainy weekend... ;-)

Re: Speeding up codecs with faster CRC calculations

Reply #30
Quote
I have compiled the new release and attached it here.

thanks a lot, @enzo - very appreciated! i was using your initial test-build up until now, because it is remarkably faster for me than the build from rarewares!  got to test this now in comparison 8)

Re: Speeding up codecs with faster CRC calculations

Reply #31
Quote
i was using your initial test-build up until now, because it is remarkably faster for me than the build from rarewares!
Same here! And thanks to enzo's excellent tuning of the compiler switches, the free GCC compiler outperforms those from Intel and Microsoft. Btw, just tested the flac.exe in foobar's "Free_Encoder_Pack_2019-08-04" and sadly, this one is (by far) the slowest:
Code: [Select]
Kernel  Time =     2.671 =    5%
User    Time =    47.625 =   91%
Process Time =    50.296 =   96%    Virtual  Memory =     14 MB
Global  Time =    51.852 =  100%    Physical Memory =     14 MB

Re: Speeding up codecs with faster CRC calculations

Reply #32
Btw, just tested the flac.exe in foobar's "Free_Encoder_Pack_2019-08-04" and sadly, this one is (by far) the slowest:
flac.exe from Free_Encoder_Pack was made to work even on ancient CPUs like Pentium 2.

Re: Speeding up codecs with faster CRC calculations

Reply #33
Btw, just tested the flac.exe in foobar's "Free_Encoder_Pack_2019-08-04" and sadly, this one is (by far) the slowest:
Probably the main reason for it is that it's 32-bit. Because yes, compatibility.

Re: Speeding up codecs with faster CRC calculations

Reply #34
Quote
i was using your initial test-build up until now, because it is remarkably faster for me than the build from rarewares!
Same here! And thanks to enzo's excellent tuning of the compiler switches, the free GCC compiler outperforms those from Intel and Microsoft.

i just made a little 1.33 comparison, not as profound as yours and under wine 64bit with fb2k. anyway here is the result. surprisingly, netrangers build is the fastest right now :)

netranger: 345x
enzo: 300x
wombat: 244x

enzo flac 1.32: 370x

Re: Speeding up codecs with faster CRC calculations

Reply #35
Quote
Probably the main reason for it is that it's 32-bit. Because yes, compatibility.
Quote
flac.exe from Free_Encoder_Pack was made to work even on ancient CPUs like Pentium 2
That makes perfect sense! And is very important for the targeted audiance = Everyone!
Thanks for the explanation!

Re: Speeding up codecs with faster CRC calculations

Reply #36
@sundance, care to try this build?

Re: Speeding up codecs with faster CRC calculations

Reply #37
Atm, I just can run the test on my office computer:
Code: [Select]
i5-6500 @ 3.2 GHz, 16 GB RAM

flac133_NetRanger:
Kernel  Time =     3.109 =    5%
User    Time =    52.562 =   90%
Process Time =    55.671 =   95%    Virtual  Memory =     14 MB
Global  Time =    58.036 =  100%    Physical Memory =     14 MB

flac133_VS2017:
Kernel  Time =     3.109 =    6%
User    Time =    46.343 =   90%
Process Time =    49.453 =   96%    Virtual  Memory =     14 MB
Global  Time =    51.172 =  100%    Physical Memory =     13 MB

flac133_enzo.exe:
Kernel  Time =     2.890 =    6%
User    Time =    40.734 =   90%
Process Time =    43.625 =   96%    Virtual  Memory =     14 MB
Global  Time =    45.233 =  100%    Physical Memory =     14 MB

flac133_Case.exe:
Kernel  Time =     2.890 =    6%
User    Time =    39.796 =   90%
Process Time =    42.687 =   97%    Virtual  Memory =     14 MB
Global  Time =    43.916 =  100%    Physical Memory =     14 MB

Look and behold! Another speed increase of ~3%...
As soon I'm back on my own computer, I'll keep you posted on the results there.
What compiler/settings did you use?

Re: Speeding up codecs with faster CRC calculations

Reply #38
I used GCC 4.9.3 (oldest one I had archived) with CFLAGS -O3 -m64 -march=haswell -funroll-loops -fno-stack-protector.

It looks like the newer the GCC version is the slower the compiled FLAC binary becomes. With the same settings as enzo used my compile was a tiny fraction slower. GCC 7.3 produced somewhat slower binary and GCC 9.1 somewhat slower than that.

Re: Speeding up codecs with faster CRC calculations

Reply #39
Did another run on an i7 at hand:
Code: [Select]
i7-4790 @ 3.6 GHz, 16 GB RAM

flac133_enzo.exe:
Kernel  Time =     2.046 =    4%
User    Time =    39.484 =   92%
Process Time =    41.531 =   97%    Virtual  Memory =     14 MB
Global  Time =    42.661 =  100%    Physical Memory =     14 MB

flac133_Case.exe:
Kernel  Time =     2.437 =    5%
User    Time =    42.265 =   92%
Process Time =    44.703 =   97%    Virtual  Memory =     14 MB
Global  Time =    45.705 =  100%    Physical Memory =     14 MB
Quite surprisingly, on that CPU your GCC 4.9.3 build was slower than enzo's AND even slower compared to my i5 office computer despite having less core speed...
Now I'm really curious how it performs on my i7-6700 @ 3.4 GHz, which is the same CPU family as the office PC (Skylake)

Re: Speeding up codecs with faster CRC calculations

Reply #40
Here we are with the results on my i7-6700 @ 3.4 GHz CPU. For both binaries, I did 6 runs (only 3 for the previous tests) and calculated the average value of timer64's "Global time":
Code: [Select]
flac133_enzo.exe:
Global Time: 39.174 sec

flac133_Case.exe:
Global Time: 38.058 sec
So we see a ~3% benefit also, like with the i5-Skylake CPU in my office.

Re: Speeding up codecs with faster CRC calculations

Reply #41
Thanks for testing. Weird that the i7-4790 performed the way it did as my i7-4771 shows my build to be faster.

Would be curious to know how these builds behave on AMD or older Intels. I'm a bit worried about processor specific optimization flags there.

Re: Speeding up codecs with faster CRC calculations

Reply #42
BTW, running configure with --enable-64-bit-words option should give some slight speedup for 64-bit executables.

Re: Speeding up codecs with faster CRC calculations

Reply #43
I used that in all GCC compile tests but neglected to mention it. Otherwise there would be no hope of getting speeds anywhere close to enzo's.

Re: Speeding up codecs with faster CRC calculations

Reply #44
@sundance, care to try this build?
Is this STILL the latest recommended FLAC binary for Windows with fastcrc mod?
• We can do better about lossy music: Opus complexity & qAAC dependence on Apple is a departure from Vorbis & Musepack breakthroughs
• Clipping-free MP3 encoding for vintage gear: SoX to 44.1 kHz → LoudMax -1 dB ISP → ADClip8 → Smart dither 16 bit → Lame3995o -Q1
• Plz, update WavPack hybrid & FSLAC

Re: Speeding up codecs with faster CRC calculations

Reply #45
Is this STILL the latest recommended FLAC binary for Windows with fastcrc mod?
fastcrc patch was included in official repository. Latest binaries are  available on RareWares and here.