Hydrogenaudio Forums

Lossless Audio Compression => FLAC => Topic started by: NetRanger on 2019-08-04 16:55:01

Title: FLAC v1.3.3
Post by: NetRanger on 2019-08-04 16:55:01
FLAC v1.3.3 (August 4th, 2019)

https://xiph.org/flac/changelog.html
https://xiph.org/flac/download.html

No Official Windows binaries available at this time.
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-08-04 17:13:21
FLAC v1.3.3
Built on August 04, 2019, GCC 9.1.0

Latest commit included : f764434

https://xiph.org/flac/

https://github.com/xiph/flac/commits/master
https://git.xiph.org/?p=flac.git;a=summary
Title: Re: FLAC v1.3.3
Post by: soundping on 2019-08-04 17:49:16
Thanks NR  :D
Title: Re: FLAC v1.3.3
Post by: Heliologue on 2019-08-04 19:22:43
In case anyone's wondering, this release does not contain any improvements in compression (or at least not at --best), so there's no utility in re-encoding your existing 1.3.x files.

I am, however, pleased that it should no longer spam my console output when I specify -f --no-error-on-compression-fail.

Title: Re: FLAC v1.3.3
Post by: Wombat on 2019-08-07 04:20:22
I did a VS 2017 compile done from the source of the official 1.3.3 Xiph release.
I don't do these things often atm. It would be nice someone can check it.
It should be a bit faster against a GCC version.
Title: Re: FLAC v1.3.3
Post by: Rollin on 2019-08-07 21:45:58
I did a VS 2017 compile done from the source of the official 1.3.3 Xiph release.
I don't do these things often atm. It would be nice someone can check it.
It should be a bit faster against a GCC version.
Works OK on Windows 7 32 bit and core i3 3245. Indeed a bit faster than build from NetRanger - ~156x vs ~148x
Title: Re: FLAC v1.3.3
Post by: TuNk77 on 2019-08-07 22:23:47
Wombat's build is indeed a bit faster compared to NetRanger's build.
Same file encoded at level 8
Wombat's build: 174,91x
NetRanger's build: 156.85x

BTW, using x64
Title: Re: FLAC v1.3.3
Post by: Wombat on 2019-08-08 01:13:57
Thanks for testing how it works in w32 and x64!
This compile i only did because i see no sign of official 1.3.3 Windows binaries and some may also like these.
In no way i want to compete with Netranger and his nice effort to always deliver recent GIT builds. Cheers!
Title: Re: FLAC v1.3.3
Post by: Anakunda on 2019-08-10 16:29:43
Can I know what's the difference between NR binaries and those at Rarewarez? The latter seem be bigger.
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-08-10 19:34:23
Made with different compliers.  John33 use ICL for his compiles.
Title: Re: FLAC v1.3.3
Post by: EmersonWal on 2019-08-13 17:01:35
Is there going to be an official FLAC 1.3.3 version that gets released on the official site?
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-08-14 12:57:32
v1.3.4 is around the corner so to say.

Quote
Verion 1.3.3 was borked because the version number in CMakeLists.txt did
not get updated.

https://github.com/xiph/flac/pull/119
Title: Re: FLAC v1.3.3
Post by: Wombat on 2019-08-15 01:59:31
https://github.com/xiph/flac/pull/119
Thanks for the info.
The compiled versions report as libFLAC 1.3.3 20190804 and one day later as libFLAC 1.3.4 20190805
I wonder why 1.3.4 didn't make it to any official xiph page yet?
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-08-15 08:03:30
'erikd' haven't pushed it yet on Github.
https://github.com/xiph/flac/pull/119
Thanks for the info.
The compiled versions report as libFLAC 1.3.3 20190804 and one day later as libFLAC 1.3.4 20190805
I wonder why 1.3.4 didn't make it to any official xiph page yet?

https://github.com/xiph/flac/issues/111#issuecomment-521431000
Title: Re: FLAC v1.3.3
Post by: Rollin on 2019-08-17 13:25:49
Quote
Verion 1.3.3 was borked because the version number in CMakeLists.txt did
not get updated.
So, Erik cares about typo in CMakeLists.txt, but he doesn't care about fact that offical site still hosts 32-bit binary of 1.3.2 for windows with bug (that was fixed in git more than 2 years ago) that prevents encoding  on 32-bit system if size of source PCM stream is larger than 4GB?
Title: Re: FLAC v1.3.3
Post by: kode54 on 2019-08-19 02:28:21
Erik has already made it quite clear that he couldn't care any less about proprietary ("dead") operating systems.
Title: Re: FLAC v1.3.3
Post by: Rollin on 2019-09-01 22:05:31
BTW, FLAС still needs --channel-map=none to encode some non-standard channels configurations (e.g. 4.1) and still --channel-map=none is not documented.
Title: Re: FLAC v1.3.3
Post by: VandalayBoss on 2019-09-16 16:52:45
I discovered some 24-bit WAV files that are NOT decoded properly after encoding-decoding using the flac 1.3.3 Windows 64-bit binaries at rarewares. Specifically the fmt chunk is messed up and does not contain the correct information. Unfortunately they are wav files I cannot release to the public. Flac 1.3.2 from xiph works correctly.
Title: Re: FLAC v1.3.3
Post by: VandalayBoss on 2019-09-16 18:05:52
More specifically, the original wav header looks like this:

Code: [Select]
52 49 46 46 86 8C 80 02 57 41 56 45    RIFF†Œ€.WAVE

But after encoding and decoding with flac 1.3.3 it looks like this:

Code: [Select]
52 49 46 46 9E 8C 80 02 57 41 56 45    RIFFžŒ€.WAVE

The fmt chunk in the original wav is:

Code: [Select]
66 6D 74 20 10 00 00 00 01 00 02 00    fmt ........

but in the encoded/decoded wav it's:

Code: [Select]
66 6D 74 20 28 00 00 00 FE FF 02 00    fmt (...þÿ..

It looks like flac is "fixing" the wav file to be a correct 24-bit PCM file with the 0xFFFE flag, but I don't know if "fixing" files is appropriate or not.
Title: Re: FLAC v1.3.3
Post by: [JAZ] on 2019-09-16 19:12:47
I believe this was intentional.
IIRC, FLAC is not storing the "fmt" header itself, but reconstructs it instead, and in this commit from 2017, it was modified to use WAVEFORMATEXTENSIBLE for bits different than 8 or 16

https://git.xiph.org/?p=flac.git;a=commit;h=bb750734287a4079ca3de9ff85c71cc62160ac46

So it all depends on what kind of compatibility do you need when decoding a flac file.
Title: Re: FLAC v1.3.3
Post by: VandalayBoss on 2019-09-16 21:19:57
I wouldn't have even noticed except that when batch converting some files I was double checking the conversion by running a SHA256 sum on the original wav and comparing it against a SHA256 sum on the decoded flac file. I don't mind if flac is going to fix files, but it will make it more difficult to make sure decoded PCM is bit-for-bit identical to the original. I'm not the type who can just "trust" that it is. I suppose I can just compare the data chunk, but that's a lot more difficult from a batch file than comparing the entire file.
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-09-18 13:06:11
FLAC v1.3.3-Git-2019-09-15
Built on September 18, 2019, GCC 9.2.0

Latest commit included : 5598543


https://xiph.org/flac/

https://github.com/xiph/flac/commits/master
https://git.xiph.org/?p=flac.git;a=summary
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-10-09 02:53:16
FLAC v1.3.3-Git-2019-10-07
Built on October 08, 2019, GCC 9.2.0

Latest commit included : 2e7931c

https://xiph.org/flac/

https://github.com/xiph/flac/commits/master
https://git.xiph.org/?p=flac.git;a=summary
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-10-09 20:56:44
You can compile libFLAC_dynamic.dll? Thanks
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-10-10 12:49:09
You can compile libFLAC_dynamic.dll? Thanks

Can't do that, the 'media auto-build suite' doesn't offer any such option.
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-10-10 19:24:02
FLAC v1.3.3-Git-2019-10-10
Built on October 10, 2019, GCC 9.2.0

Latest commit included : 952d511


https://xiph.org/flac/

https://github.com/xiph/flac/commits/master
https://git.xiph.org/?p=flac.git;a=summary
Title: Re: FLAC v1.3.3
Post by: Em on 2019-10-10 22:48:08
You can compile libFLAC_dynamic.dll? Thanks
This is a x64 build using MSVC 2017 (v141) SDK version 10.0.17763.0 from the 2019-10-06 commit (2e7931c2). Tried using the latest commit but it breaks on windows_unicode_filenames.c

Please backup the current dll you're using in case this doesn't work for you
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-11-03 06:09:19
LibFLAC_dynamic x64
MSVC 2019 (v142) SDK version 10.0.18362.0 from the 2019-10-06 commit b84ff55 (2019-10-23)
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-11-03 10:41:48
FLAC v1.3.3-Git-2019-10-23
Built on November 03, 2019, GCC 9.2.0

Latest commit included : b84ff55


https://xiph.org/flac/

https://github.com/xiph/flac/commits/master
https://git.xiph.org/?p=flac.git;a=summary
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-11-05 15:40:03
FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
MSVC 2019 (v142) SDK version 10.0.18362.0
commit b84ff55 (2019-10-23)
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-11-11 15:26:28
FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
MSVC 2019 (v142) SDK version 10.0.18362.0
commit 2907d49 (2019-11-11)
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-11-16 11:20:24
FLAC v1.3.3-Git-2019-11-15
Built on November 16, 2019, GCC 9.2.0

Latest commit included : cdcf0d5


https://xiph.org/flac/

https://git.xiph.org/?p=flac.git;a=summary
https://github.com/xiph/flac/commits/master
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-11-18 15:43:22
FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
MSVC 2019 (v142) SDK version 10.0.18362.0
commit ed1b67b (2019-11-18)
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-11-19 17:17:09
FLAC v1.3.3-Git-2019-11-19
Built on November 19, 2019, GCC 9.2.0

Latest commit included : 3bb5d8c

https://xiph.org/flac/

https://git.xiph.org/?p=flac.git;a=summary
https://github.com/xiph/flac/commits/master
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-11-20 15:27:17
Date: 2019-11-20
Commit: f706f28
Compiler: MSVC 2019 (v142) SDK version 10.0.18362.0

FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-11-21 20:42:18
Commit d518e13a1f (2019-11-21)
We're waiting NetRanger.  :)
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-11-22 18:03:53
Date: 2019-11-21
Commit: d518e13
Compiler: MSVC 2019 (v142) SDK version 10.0.18362.0

FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-11-24 09:01:34
Date: 2019-11-24
Commit: b02e159
Compiler: MSVC 2019 (v142) SDK version 10.0.18362.0

FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-11-30 09:40:17
Date: 2019-11-30
Commit: 6455e47
Compiler: MSVC 2019 (v142) SDK version 10.0.18362.0

FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-12-08 15:59:23
Date: 2019-12-09
Commit: a9d9f4d
Compiler: MSVC 2019 (v142) SDK version 10.0.18362.0

FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2019-12-10 05:17:29
FLAC v1.3.3-Git-2019-12-08
Built on December 10, 2019, GCC 9.2.0

Latest commit included : a3d8927

https://xiph.org/flac/

https://git.xiph.org/?p=flac.git;a=summary
https://github.com/xiph/flac/commits/master
Title: FLAC v1.3.3-Git-2019-12-22
Post by: NetRanger on 2019-12-23 12:49:39
FLAC v1.3.3-Git-2019-12-22
Built on December 23, 2019, GCC 9.2.0

Latest commit included : 0dfe235


https://xiph.org/flac/

https://git.xiph.org/?p=flac.git;a=summary
https://github.com/xiph/flac/commits/master
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-12-29 19:31:56
Date: 2019-12-27
Commit: cffe389
Compiler: MSVC 2019 (v142) SDK version 10.0.18362.0

FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
Title: Re: FLAC v1.3.3
Post by: Ozz on 2019-12-29 19:33:02
Date: 2019-12-27
Commit: cffe389
Compiler: MSVC 2019 (v142) SDK version 10.0.18362.0

FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
Title: Re: FLAC v1.3.3
Post by: sanskrit44 on 2020-01-03 06:37:04
does there exist amd optimized flac (or ogg vorbis) builds somewhere? i would like to see them performing in contrast to the builds offered at rarewares or in this thread.
Title: Re: FLAC v1.3.3
Post by: Ozz on 2020-03-08 09:22:19
Date: 2020-03-06
Commit: c8fca6b
Compiler: MSVC 2019 (v142) SDK version 10.0.18362.0

FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)
Title: Re: FLAC v1.3.3
Post by: 2tec on 2020-03-08 12:16:18
Date: 2020-03-06
Commit: c8fca6b
Compiler: MSVC 2019 (v142) SDK version 10.0.18362.0

FLAC 1.3.3 + LibFLAC_dynamic.dll  (x86 & x64)

virustotal reports ten positives
see https://www.virustotal.com/gui/file/54e596b27c2b631617521cb3e425b117647e71ac5f083237990531842d21ebe9/detection
Title: Re: FLAC v1.3.3
Post by: kode54 on 2020-03-08 13:14:11
Of course it does. That's what it's good at doing. Scaring people. Especially people who use MSVC >2015. Better report that false positive to Microsoft, and all the others will just go away with a rescan.
Title: Re: FLAC v1.3.3
Post by: TuNk77 on 2020-03-08 16:30:26
metaflac.exe in flac-win-1.3.3-c8fca6b-2020-03-06\Win32 triggers a false positive
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2020-03-10 05:23:35
FLAC v1.3.3-Git-2020-03-06
Built on March 08, 2020, GCC 9.2.0

Latest commit included : 1d5299d

https://xiph.org/flac/

https://git.xiph.org/?p=flac.git;a=summary
https://github.com/xiph/flac/commits/master
Title: Re: FLAC v1.3.3
Post by: SigHunter on 2020-03-10 08:11:04
1.3.3 was released over 6 months ago and never even made the news on the homepage, that's a little sad IMHO
https://xiph.org/flac/news.html
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-03-24 00:50:06
FLAC 1d5299d (https://github.com/xiph/flac/tree/1d5299d67b4f4d8172b2fab702696bce1082b91a) MSYS2/mingw-w64-x86_64 2020-03-23 (64bit, GCC 9.3)

Tuned for modern Intel CPUs and linked against similarly built libogg 1.3.4.

Simple recompression benchmark on an AMD 2700X:

This build:
Code: [Select]
./timer64.exe ./flac.exe --verify --best --exhaustive-model-search --qlp-coeff-precision-search ./test.flac -o test_grieverv.flac

test.flac: Verify OK, wrote 377669674 bytes, ratio=0.992


Kernel  Time =     1.171 =    0%
User    Time =   335.281 =   99%
Process Time =   336.453 =   99%    Virtual  Memory =     15 MB
Global  Time =   337.578 =  100%    Physical Memory =     14 MB

NetRanger's build:
Code: [Select]
./timer64.exe ./flac.exe --verify --best --exhaustive-model-search --qlp-coeff-precision-search ./test.flac -o test_netranger.flac

test.flac: Verify OK, wrote 377669674 bytes, ratio=0.992


Kernel  Time =     1.531 =    0%
User    Time =   409.140 =   99%
Process Time =   410.671 =   99%    Virtual  Memory =     15 MB
Global  Time =   411.926 =  100%    Physical Memory =     14 MB

Build information:
Spoiler (click to show/hide)
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-03-26 00:12:29
FLAC 1d5299d (https://github.com/xiph/flac/tree/1d5299d67b4f4d8172b2fab702696bce1082b91a) MSYS2/mingw-w64-x86_64 2020-03-25 (64bit, GCC 9.3)

Rebuilt with more aggressive GCC optimizations taken from GentooLTO (https://github.com/InBetweenNames/gentooLTO).
Compared to the previous build, it's around 4.2% faster on the previously tested file which saved another ~14 seconds.

Build information:
Spoiler (click to show/hide)
Title: Re: FLAC v1.3.3
Post by: itisljar on 2020-03-30 18:42:46
Speedup is great, from average 125x with Netranger's on my i7-2600 to 145x with GrieverV's build here.
Title: Re: FLAC v1.3.3
Post by: sundance on 2020-04-01 09:11:06
On my i7-6700, Case's build is still the fastest...
https://hydrogenaud.io/index.php?topic=115900.msg974249#msg974249
Title: Re: FLAC v1.3.3
Post by: itisljar on 2020-04-01 19:44:43
His build just doesn't work on my computer. It starts, shows help, but when it needs to encode, it just... exits.
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-02 01:46:53
His build just doesn't work on my computer. It starts, shows help, but when it needs to encode, it just... exits.

Code: [Select]
-march=haswell
Means that the binary has been fully optimized for the Haswell CPU family and is unlikely to run on any other family unless they have full support for the instruction sets GCC enables for the Haswell family. Fortunately, I believe all current Intel CPUs since Haswell support the Haswell optimized code generated by GCC.
Title: Re: FLAC v1.3.3
Post by: sundance on 2020-04-02 08:46:20
@GrieverV
Just tested your latest compile against Case's here on my i7-6700 with my set of 40 WAV files (1.77 GB).
All reading/writing is from/to SSD, but since I do at least 5 runs, the WAVs are likely to be cached.

Code: [Select]
flac_Griever_1d5299d.exe
Kernel  Time =     2.203 =    5%
User    Time =    38.453 =   91%
Process Time =    40.656 =   96%    Virtual  Memory =     14 MB
Global  Time =    42.164 =  100%    Physical Memory =     14 MB
Code: [Select]
Case´s build:
Kernel  Time =     2.562 =    6%
User    Time =    35.390 =   90%
Process Time =    37.953 =   96%    Virtual  Memory =     14 MB
Global  Time =    39.238 =  100%    Physical Memory =     14 MB
I have no idea what causes this difference (platform specific compile, compiler version used, compiler optimisations ...)
What I noticed is that your binary uses way less "kernel time" and more "user time" than Case's. But interpreting these figures is way beyond my knowledge...
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-02 12:05:30
@sundance
Their toolchain is likely very different so there could be a lot of possible explanations, but the most likely would be -march=haswell and GCC 4.9.

If I remember correctly, there was a big performance regression on GCC 5 with FLAC encoding and it's possible GCC never recovered it or further regressed. I've attached a build that was built with -march=haswell and -funroll-loops if you want to try it. As far as I could tell, there wasn't much difference with -funroll-loops alone when I tested.
Title: Re: FLAC v1.3.3
Post by: sundance on 2020-04-02 15:23:49
Getting pretty close now:
Code: [Select]
flac_GrieverV_1d5299d_haswell.exe
Kernel  Time =     2.390 =    5%
User    Time =    36.468 =   90%
Process Time =    38.859 =   96%    Virtual  Memory =     14 MB
Global  Time =    40.137 =  100%    Physical Memory =     14 MB
Title: Re: FLAC v1.3.3
Post by: SigHunter on 2020-04-06 09:20:13
did some testing of my own with GrieverV's aggressively optimized Skylake 1d5299d 1.3.3 flac.exe. I don't use -ep so I left it out and compared it to my up until recently used flac 1.3.2 (official xiph generic windows binary, flac-1.3.2-win.zip). SSD to SSD, i7-7700K, it's 25 % faster! (I know, apples and oranges, but I switched from this 1.3.2 to GrieverV's 1.3.3 just now and I'm gonna stick with it, so this is my real life improvement)


Code: [Select]
PS C:\Program Files (x86)\foobar2000\Encoders\skylakeflac> Measure-Command { ..\flac.exe --verify --best E:\flac_reenc_AIO\Image.wav -o E:\flac_reenc_AIO\Image_new.flac }

flac 1.3.2
Copyright (C) 2000-2009  Josh Coalson, 2011-2016  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

Image.wav: Verify OK, wrote 2816137583 bytes, ratio=0,789


Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 50
Milliseconds      : 85
Ticks             : 1100852308
TotalDays         : 0,00127413461574074
TotalHours        : 0,0305792307777778
TotalMinutes      : 1,83475384666667
TotalSeconds      : 110,0852308
TotalMilliseconds : 110085,2308


Code: [Select]
PS C:\Program Files (x86)\foobar2000\Encoders\skylakeflac> Measure-Command { .\flac_1d5299d_skylake_2020-03-25.exe --verify --best E:\flac_reenc_AIO\Image.wav -o E:\flac_reenc_AIO\Image_new.flac }

flac 1.3.3
Copyright (C) 2000-2009  Josh Coalson, 2011-2016  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

Image.wav: Verify OK, wrote 2816138302 bytes, ratio=0,789


Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 23
Milliseconds      : 653
Ticks             : 836538482
TotalDays         : 0,000968215835648148
TotalHours        : 0,0232371800555556
TotalMinutes      : 1,39423080333333
TotalSeconds      : 83,6538482
TotalMilliseconds : 83653,8482

Title: Re: FLAC v1.3.3
Post by: sanskrit44 on 2020-04-06 10:43:13
wow, this is really impressive. seems like there is always room for further optimizations.

flac2flac conversion of 25 tracks in foobar via wine on i5-ivy.

netranger compile:
Track converted successfully.
Total encoding time: 0:22.661, 239.69x realtime


grieverv compile:
Track converted successfully.
Total encoding time: 0:17.350, 313.07x realtime

thanks a lot @grieverv.
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2020-04-07 03:08:32
FLAC v1.3.3-Git-2020-04-06
Built on April 07, 2020, GCC 9.3.0

Latest commit included : dd975a9


https://xiph.org/flac/

https://github.com/xiph/flac/commits/master
https://git.xiph.org/?p=flac.git;a=summary
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-07 16:40:39
FLAC 7a35c52 (https://github.com/xiph/flac/tree/7a35c528495b6d8a51e87e007f6810b5553101bc) MSYS2/mingw-w64-x86_64 2020-04-07 (64bit, GCC 9.3)

Aggressively optimized builds tuned for modern processors. Included is a generic x86_64 build and an Intel Haswell optimized build that will only run on Haswell or newer Intel CPUs and Zen or newer AMD CPUs. All FLAC tests passed for both builds.

Compared to my previous builds, the generic build is now built with -funroll-loops and FLAC is now configured with --enable-64-bit-words.

Build information:
Spoiler (click to show/hide)

This is likely to be my last build so I've included my makepkg configuration and PKGBUILDs for libogg and flac.
Title: Re: FLAC v1.3.3
Post by: sanskrit44 on 2020-04-07 18:50:33
no changes in speed with latest netranger build - but grievervs build got tamed from 313x to 264x.

Track converted successfully.
Total encoding time: 0:20.609, 263.56x realtime

same setup as before.
Title: Re: FLAC v1.3.3
Post by: sundance on 2020-04-07 19:00:03
@GrieverV:
Well done, mate! This time your Haswell compile is the fastest I ever tested!
These are my test results (under today's full moon conditions  ;) )
Code: [Select]
flac133_ GrieverV_7a35c528.exe:
Kernel  Time =     2.921 =    7%
User    Time =    35.781 =   89%
Process Time =    38.703 =   96%    Virtual  Memory =     14 MB
Global  Time =    40.034 =  100%    Physical Memory =     14 MB

flac133_ GrieverV_7a35c528_haswell.exe:
Kernel  Time =     2.359 =    6%
User    Time =    35.359 =   90%
Process Time =    37.718 =   96%    Virtual  Memory =     14 MB
Global  Time =    39.087 =  100%    Physical Memory =     14 MB

flac133_Case.exe:
Kernel  Time =     2.812 =    7%
User    Time =    35.109 =   89%
Process Time =    37.921 =   96%    Virtual  Memory =     14 MB
Global  Time =    39.223 =  100%    Physical Memory =     14 MB

@sanskrit44: What CPU are you running, and which of GrieverV's compiles did you test?
Title: Re: FLAC v1.3.3
Post by: sanskrit44 on 2020-04-08 07:00:52
@sanskrit44: What CPU are you running, and which of GrieverV's compiles did you test?
it is a i5-3230m running the non-haswell build.
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-08 10:58:23
I see consistent gains on a 2700X with this test file: BIS1447-002-flac_16.flac (https://web.archive.org/web/20200408091726/https://www.eclassical.com/custom/eclassical/files/BIS1447-002-flac_16.flac)
Code: [Select]
--best:                                                                 4.5%
--verify --best:                                                        5.8%
--verify --best --qlp-coeff-precision-search:                           2.5%
--verify --best --exhaustive-model-search --qlp-coeff-precision-search: 1.2%
The builds I tested were generic x86_64 from 2020-03-25 (https://hydrogenaud.io/index.php?topic=118008.msg981392#msg981392) and 2020-04-07 (https://hydrogenaud.io/index.php?topic=118008.msg981823#msg981823). The differences between the two are just -funroll-loop and enabling 64 bit words for FLAC (upstream commits do not have any effect).

@sanskrit44
Were your tests perhaps done at different times? Background processes can change the results and each build should be tested at least three times to ensure the results are consistent.

Are you able to reproduce similar results with different FLAC files or the one linked above?
Title: Re: FLAC v1.3.3
Post by: sanskrit44 on 2020-04-08 12:24:43
Are you able to reproduce similar results with different FLAC files or the one linked above?
i retested flac2flac with foobar and booted into a fresh system for each run.


mingw-w64-x86_64-flac-git-1.3.3.r3906.1d5299d6:

my set of 25 flacs:
Total encoding time: 0:17.488, 310.60x realtime

25x flac testfile:
Total encoding time: 0:32.065, 326.21x realtime



mingw-w64-x86_64-flac-git-1.3.3.r3909.7a35c528:

my set of 25 flacs:
Total encoding time: 0:16.978, 319.93x realtime

25x flac testfile:
Total encoding time: 0:30.801, 339.59x realtime


looking much better now :)
Title: Re: FLAC v1.3.3
Post by: SigHunter on 2020-04-08 12:48:33
why are those called haswell builds when they are compiled with -mtune=skylake?


FLAC 1d5299d MSYS2/mingw-w64-x86_64 2020-03-25 (64bit, GCC 9.3)
Code: [Select]
PS C:\Program Files (x86)\foobar2000\Encoders> Measure-Command { .\flac.exe --verify --best E:\Musik_reenc_AIO\Image.w64 -o E:\Musik_reenc_AIO\imageout.flac }

flac 1.3.3
Copyright (C) 2000-2009  Josh Coalson, 2011-2016  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

Image.w64: Verify OK, wrote 4116685875 bytes, ratio=0,779


Days              : 0
Hours             : 0
Minutes           : 2
Seconds           : 16
Milliseconds      : 485
Ticks             : 1364857502
TotalDays         : 0,00157969618287037
TotalHours        : 0,0379127083888889
TotalMinutes      : 2,27476250333333
TotalSeconds      : 136,4857502
TotalMilliseconds : 136485,7502


FLAC 7a35c52 MSYS2/mingw-w64-x86_64 2020-04-07 (64bit, GCC 9.3) haswell/skylake on i7-7700K
Code: [Select]
PS C:\Program Files (x86)\foobar2000\Encoders> Measure-Command { .\flacneu\flac.exe --verify --best E:\Musik_reenc_AIO\Image.w64 -o E:\Musik_reenc_AIO\imageout2.flac }

flac 1.3.3
Copyright (C) 2000-2009  Josh Coalson, 2011-2016  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

Image.w64: Verify OK, wrote 4116686510 bytes, ratio=0,779


Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 58
Milliseconds      : 800
Ticks             : 1188007802
TotalDays         : 0,00137500903009259
TotalHours        : 0,0330002167222222
TotalMinutes      : 1,98001300333333
TotalSeconds      : 118,8007802
TotalMilliseconds : 118800,7802
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-08 13:02:16
why are those called haswell builds when they are compiled with -mtune=skylake?

I only included the generic builds flags in the build info since all that's done for Haswell is changing march and removing mtune. The binary is still built for Haswell and the relevant makepkg configuration is included in the download.

You can think of -mtune=skylake as a more modern default for x86_64 CPUs since every Intel CPU from the last decade will likely show benefits over generic/x86_64; AMD Zen also shows improvements and I wouldn't be surprised if bulldozer benefits, too.
Title: Re: FLAC v1.3.3
Post by: rutra80 on 2020-04-08 17:34:50
Size difference?
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-11 12:36:02
Size difference?
Executables are larger due to enabling loop unrolling, FLAC compression is different due to -march=haswell.

I was able to narrow the FLAC compression difference down to the -fma flag. AVX+AVX2 is fine and FLAC built with -fma and configured with --disable-avx --disable-asm-optimizations has the same difference so it's probably GCC/FMA's fault.
Title: Re: FLAC v1.3.3
Post by: rutra80 on 2020-04-11 23:10:33
Yeah I meant compression difference. Was surprised as FLAC does integer math only doesn't it? It made me wonder how recompile could make the resulting files differ...
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-12 10:04:25
AFAIU, decoding is integer only but libFLAC uses floating point unless FLAC__INTEGER_ONLY_LIBRARY (https://github.com/xiph/flac/blob/c6318e9dd3f7a91f40340911bcf57bf36768910e/src/libFLAC/include/private/float.h#L42) macro is defined.

For those watching this thread, I'll probably be releasing new builds soon with FMA disabled so compression is reproducible with other builds (confirmed MSVC reproduces same compression). I also noticed I had -D__USE_MINGW_ANSI_STDIO=1 disabled for the previous builds, although I'll leave that disabled since it incurs a very minor performance penalty that shouldn't be necessary for ogg/FLAC.

I'll have to build, test, package and benchmark before I release, however I did see ~4-5% gain on a 2700X --verify --best after disabling FMA.
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-12 13:51:54
FLAC 7a35c52 (https://github.com/xiph/flac/tree/7a35c528495b6d8a51e87e007f6810b5553101bc) MSYS2/mingw-w64-x86_64 2020-04-12 (64bit, GCC 9.3)

Aggressively optimized builds tuned for modern processors. Included is a generic x86_64 build and an Intel Haswell optimized build that will only run on Haswell or newer Intel CPUs and Zen or newer AMD CPUs. All FLAC tests passed for both builds.

Changes:

The new Haswell build is 8.1% faster than the previous Haswell build and 5% faster than the generic build on a 2700X (--verify --best) with this test file: BIS1447-002-flac_16.flac (https://web.archive.org/web/20200408091726/https://www.eclassical.com/custom/eclassical/files/BIS1447-002-flac_16.flac)

Build information:
Spoiler (click to show/hide)

I'd appreciate it if anyone else can confirm the compression difference is now gone and how much of a performance hit or increase Intel CPUs receive.
Title: Re: FLAC v1.3.3
Post by: sundance on 2020-04-13 08:42:00
@GrieverV:
- The compression difference is gone; same FLACs as with NetRanger's binaries here
- On my i6700 I can not confirm a speed increase (haswell build); a quick test resulted in ~ the same performance as Case's build, but your FMA enabled build is still a bit faster (~1%).

Pardon my ignorance: What does this "FMA support" do? I was under the impression that compiler options can (and should) have an impact on the binary size/speed, but change the output files of a lossless encoder (although both files are valid FLACs with identical MD5)? Isn't this algorithm fixed by the source code?
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-13 09:49:10
Thanks for confirming and testing on Intel!

I'm not a math person and floating-point (approximate) stuff is like black magic to me, but as far as I understand, fused multiply-add (https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation#Fused_multiply%E2%80%93add) can result in slight differences due to different rounding compared to the usual multiply-add. Most modern CPUs should have hardware support for fused multiply-add operations which can be used through the FMA instruction set and GCC can make use of those instructions when performing optimizations by specifying the -mfma flag which is enabled when -march=haswell is set. Compilers usually have options (https://gcc.gnu.org/wiki/FloatingPointMath) that affect floating-point math.

libFLAC uses floating-point math for encoding that can be optimized by GCC and it seems enabling FMA will change the result in some cases.
Title: Re: FLAC v1.3.3
Post by: itisljar on 2020-04-13 09:56:18
What must I do to make compile compatible with SandyBridge CPU, but not with generations before?
Is there some step-by-step which I can follow?
I've tried yesterday, by installing MinGW and setting things up, but it just wouldn't work.
Title: Re: FLAC v1.3.3
Post by: GrieverV on 2020-04-13 10:03:54
@itisljar
Assuming you're using MSYS2, all you have to do is change -march=haswell to -march=sandybridge in the makepkg_mingw64.conf included with the Haswell build, copy to msys64/etc/, start MSYS2 MinGW 64-bit shell and run 'MINGW_INSTALLS=mingw64 makepkg-mingw -sic' in each of the folders containing the PKGBUILDs. It's easiest to copy the libogg/ and flac-git/ folders to msys64\home\<user> so all you have to do is 'cd <folder>/'

If you're using plain Mingw-w64, you'll need to set the environment variables manually and refer to the PKGBUILDs to see how ogg and FLAC are built.

Somewhat step-by-step for MSYS2:
Spoiler (click to show/hide)

edit: Fair warning, anti-viruses hate MSYS2 and may completely break your MSYS2 install when you try to build any package (curl.exe, bash.exe, etc.). You might want to add an exception for the MSYS2 folder while building packages and then remove it from exceptions when done to avoid any security issues in the future while not using MSYS2.
Title: Re: FLAC v1.3.3
Post by: sundance on 2020-04-13 11:11:26
@GrieverV:
Thank you very much for shedding some light on FMA optimisation. Very interesting stuff.

Btw, here are the results of my ususal test set:
Code: [Select]
Intel Core i7-6700 CPU @ 3.40GHz, 24 GB RAM, Samsung SSD 850 EVO 500GB
Windows 10 Pro 64-bit, Version 1909, Build 18363.720

flac133_case.exe:
39.303 sec (avg. runtime/5 runs, fastest run: 39.228 sec)
total size of FLAC files: 1.167.279.038 bytes -> different build!

flac133_GrieverV_7a35c528_haswell_FMA.exe (07.04.2020):
38.964 sec (avg. runtime/5 runs, fastest run: 38.814 sec)
total size of FLAC files: 1.167.276.028 bytes

flac133_GrieverV_7a35c528_haswell_noFMA.exe (12.04.2020):
39.169 sec (avg. runtime/5 runs, fastest run: 39.055 sec)
total size of FLAC files: 1.167.278.241 bytes

flac133_GrieverV_7a35c528.exe (12.04.2020):
40.192 sec (avg. runtime/5 runs, fastest run: 40.068 sec)
total size of FLAC files: 1.167.278.241 bytes

Title: Re: FLAC v1.3.3
Post by: itisljar on 2020-04-13 11:40:57
Somewhat step-by-step for MSYS2:

Oh, thank you very much! I will try these in next few days!
Title: Re: FLAC v1.3.3
Post by: itisljar on 2020-04-13 13:06:50
Can't edit the post, but... do I edit these:

Code: [Select]
#-- Compiler and Linker Flags
# -march (or -mcpu) builds exclusively for an architecture
# -mtune optimizes for an architecture, but builds for whole processor family
CPPFLAGS="-D__USE_MINGW_ANSI_STDIO=1"
CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe"
CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe"
LDFLAGS="-pipe"

Like this?

Code: [Select]
#-- Compiler and Linker Flags
# -march (or -mcpu) builds exclusively for an architecture
# -mtune optimizes for an architecture, but builds for whole processor family
CPPFLAGS="-D__USE_MINGW_ANSI_STDIO=1"
CFLAGS="-march=x86-64 -mtune=sandybridge -O2 -pipe"
CXXFLAGS="-march=x86-64 -mtune=sandybridge -O2 -pipe"
LDFLAGS="-pipe"

Or like this?

Code: [Select]
#-- Compiler and Linker Flags
# -march (or -mcpu) builds exclusively for an architecture
# -mtune optimizes for an architecture, but builds for whole processor family
CPPFLAGS="-D__USE_MINGW_ANSI_STDIO=1"
CFLAGS="-march=sandybridge -mtune=generic -O2 -pipe"
CXXFLAGS="-march=sandybridge -mtune=generic -O2 -pipe"
LDFLAGS="-pipe"
Title: Re: FLAC v1.3.3
Post by: rutra80 on 2020-04-13 13:09:03
sundance: From what I understand FLAC uses floating point math in Linear Predictive Coding. Different floating point instructions have different precision, plus there are rounding errors. Most lossless encoders try to predict the next sample(s) and store error in the resulting file - due to slightly different math the predictions differ, so stored correction information differs, and so file size is different, but still lossless. Hopefully...
In case of lossy encoders, different compile may produce different file too, but in this case decoded file will be different too, inaudibly though. Hopefully...
Title: Re: FLAC v1.3.3
Post by: sundance on 2020-04-13 19:31:44
Thanks a lot for your explanation. That makes perfect sense.
Somehow I tended to think that lossless compression and (imprecise) floating point calculations can't go together...
Title: Re: FLAC v1.3.3
Post by: Listener-felinegrace on 2020-04-16 07:56:13
FLAC uses integer math not floating point.  From the WIKIpedia article:


Thanks
Title: Re: FLAC v1.3.3
Post by: Octocontrabass on 2020-04-16 09:40:58
The FLAC format uses only integer math, but FLAC encoders can use floating-point math to select the (integer) parameters used for the linear prediction.
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2020-05-03 16:31:07
FLAC v1.3.3-Git-2020-05-03-37e675b
Built on May 03, 2020, GCC 9.3.0

Latest commit included : 37e675b

https://xiph.org/flac/

https://github.com/xiph/flac/commits/master
https://gitlab.xiph.org/xiph/flac
Title: Re: FLAC v1.3.3
Post by: NetRanger on 2020-05-14 17:36:23
FLAC v1.3.3-Git-2020-05-14-ce6dd6b
Built on May 14, 2020, GCC 10.1.0

Latest commit included : ce6dd6b

https://xiph.org/flac/

https://github.com/xiph/flac/commits/master
https://gitlab.xiph.org/xiph/flac
SimplePortal 1.0.0 RC1 © 2008-2020