• TAK Applications 2.3.3.

• TAK Winamp plugin 2.3.3

• TAK Decoder library 2.3.3 (x86/x64)

• TAK SDK 2.3.3

• An 64-bit decoder library for the SDK.
• I have also created 64-bit versions of the applications. As expected they are bigger and slower without any advantage. As long as Windows supports 32-bit applications i see no reason to release them. But i will continously maintain them, so that they are ready when needed.
• Unicode support of the GUI version is no longer limited to the open file dialogs. The required switch to a newer version of my development environment is responsible for a 3.5 times bigger program file.

• Tiny encoding speed improvements of not more than 3 percent for Intel cpus based upon the skylake microarchitecture (6th to 10th Generation Core). I could have squeezed out more but only at the expense of significantly slower processing on older platforms. As rule of thumb i am taking into account cpu microarchitectures of the last 10 years.

Preset  Enco-Speed                Deco-Speed
---------------------------------------------------------
        2.3.2    2.3.3    Win %   2.3.2    2.3.3    Win %
---------------------------------------------------------
-p0     868.72   893.94    2.90   806.84   810.57    0.46
-p0e    698.08   711.16    1.87   812.44   817.09    0.57
-p0m    418.70   425.54    1.63   815.07   819.19    0.51
-p1     726.44   748.04    2.97   787.03   788.37    0.17
-p1e    482.99   491.72    1.81   788.73   791.12    0.30
-p1m    314.46   322.68    2.61   790.61   794.60    0.50
-p2     580.58   591.20    1.83   715.05   718.12    0.43
-p2e    347.07   348.35    0.37   714.52   719.10    0.64
-p2m    203.02   206.26    1.60   715.99   718.71    0.38
-p3     301.25   306.83    1.85   697.20   703.97    0.97
-p3e    241.91   244.79    1.19   697.88   702.06    0.60
-p3m    131.89   133.65    1.33   699.12   701.40    0.33
-p4     183.23   186.54    1.81   650.44   657.64    1.11
-p4e    158.75   160.61    1.17   651.01   658.40    1.14
-p4m     82.49    83.73    1.50   651.21   656.84    0.86
---------------------------------------------------------

Preset  Enco-Speed                Deco-Speed
---------------------------------------------------------
        32 bit   64 bit   Win %   32 bit   64 bit   Win %
---------------------------------------------------------
-p0     893.94   827.49   -7.43   810.57   700.80  -13.54
-p0e    711.16   661.05   -7.05   817.09   709.15  -13.21
-p0m    425.54   398.79   -6.29   819.19   710.48  -13.27
-p1     748.04   698.29   -6.65   788.37   689.93  -12.49
-p1e    491.72   461.79   -6.09   791.12   693.09  -12.39
-p1m    322.68   303.56   -5.93   794.60   695.48  -12.47
-p2     591.20   557.27   -5.74   718.12   634.55  -11.64
-p2e    348.35   336.29   -3.46   719.10   633.90  -11.85
-p2m    206.26   193.64   -6.12   718.71   636.49  -11.44
-p3     306.83   287.58   -6.27   703.97   619.50  -12.00
-p3e    244.79   233.77   -4.50   702.06   620.18  -11.66
-p3m    133.65   123.27   -7.77   701.40   621.06  -11.45
-p4     186.54   172.30   -7.63   657.64   577.94  -12.12
-p4e    160.61   145.73   -9.26   658.40   579.34  -12.01
-p4m     83.73    76.29   -8.89   656.84   578.45  -11.93
---------------------------------------------------------

• Port to Lazarus / Freepascal. Nice for Linux support.
• Fast integrity check without decoding based upon the checksums only.
• Transcode mode.
• Tuning of the encoder for the problem files which have been reported in the past months.

Currently it's not clear what i will do next.

    Unterstützung für Unicode-Zeichensätze.
    Eine deutschsprachige Version.
    Noch ein bißchen mehr Geschwindigkeit und Kompressionseffizienz...
    Anwendungen für andere Plattformen als Windows.
    Unterstützung für mehr als 6 Audiokanäle.

• For FLAC I think there is still a bit to gain by improving quantization of predictor coefficients, but it seems this is not applicable to TAK, as it doesn't store raw predictor coefficients like FLAC does. I'm sharing the idea just in case it does make sense
• You could consider switching from MD5 to something faster. I've been told most SHA checksums are faster simply because they don't have a long dependency chain and can more efficiently use the superscalar properties and out-of-order execution capabilities of modern CPUs. (That would obviously break backward compatibility, but only for checksumming) For FLAC checksumming is quite a significant part of decoding CPU load (and encoding for the fastest presets), so I imagine this is also the case for TAK.
• You could consider looking into the arithmetic coder used in Daala/AV1. That is an arithmetic coder that was specifically designed to evade any existing patents. I don't know whether that is fast enough for your liking

An 64-bit decoder library for the SDK.

You could consider switching from MD5 to something faster. I've been told most SHA checksums are faster simply because they don't have a long dependency chain and can more efficiently use the superscalar properties and out-of-order execution capabilities of modern CPUs. (That would obviously break backward compatibility, but only for checksumming) For FLAC checksumming is quite a significant part of decoding CPU load (and encoding for the fastest presets), so I imagine this is also the case for TAK.

You could consider looking into the arithmetic coder used in Daala/AV1. That is an arithmetic coder that was specifically designed to evade any existing patents. I don't know whether that is fast enough for your liking

• Port to Lazarus / Freepascal. Nice for Linux support.

Future

The next release should add support for the AVX2 instruction set. I achieved encoding speed improvements of about 14 percent for preset -p4m on my primary system (Intel Skylake based CPU), less for other presets. But results of my secondary (Haswell based) system were discouraging: Maximum improvement of 8 percent for presets p4 and p4e and up to 23 percent slower encoding for p2m, p3m and p4m!

Those presets make the most use of AVX2-instructions and should also benefit the most. But they seem to trigger the automatic down clocking mechanism of the cpu. AVX2 base and turbo frequencies are lower than the regular ones. This wouldn't hurt too much if the encoder would mostly use AVX2 instructions, but that's not the case. I havent profiled the code yet but i would estimate that about 30 percent of the encoding time goes to AVX2 instructions. And this is no continuous block, instead blocks of x86/SSE2 and AVX2 instructions are alternating.

That's bad, beacuse it will cause many transitions between the different clock rates. During such transitions the speed can be much slower than the lower clock rate would suggest. After the last AVX2 instruction the lower clock rate will be maintained for a considerable amount of time, therfore succeeding non-AVX2 instructions will also be excecuted slower.

Well, my haswell cpu is an 35w low power quad core, quite a challenge for an older desktop microarchitecture. The difference between regular and AVX2 clock most likely is considerably bigger than for the common 65W+ cpus.

Nevertheless i am really hesitant to release an AVX2-version which will make encoding on an unknown number of older systems slower. And  imho the possible advantage isn't big enough to justify an elaborate study and implementation of a cpu dependend code path.
...

...
You could consider switching from MD5 to something faster. I've been told most SHA checksums are faster simply because they don't have a long dependency chain and can more efficiently use the superscalar properties and out-of-order execution capabilities of modern CPUs. (That would obviously break backward compatibility, but only for checksumming) For FLAC checksumming is quite a significant part of decoding CPU load (and encoding for the fastest presets), so I imagine this is also the case for TAK.[/li][/list]
...

...
In TAK and WavPack, MD5 is optional and disabled by default - in WavPack it is viewed more as a fingerprint, and even more so after having implemented the non-decoding integrity check.
From that point of view, where MD5 is an optional fingerprint, I think it is a great advantage to have the same [note2] algorithm across FLAC/TAK/WavPack(/OptimFROG if anyone cares):
...

Quote

You could consider looking into the arithmetic coder used in Daala/AV1. That is an arithmetic coder that was specifically designed to evade any existing patents. I don't know whether that is fast enough for your liking

That hasn't stopped Microsoft from patenting the rANS entropy method, regardless.

Which to me leaves us in the exact same situation as if we use ordinary arithmetic coding. Might as well just use that or range coding. :/

Modern CPU's have acceleration for sha1 and sha256, IMO that's a big reason to use them. For a similar reason crc32c is interesting: https://www.corsix.org/content/fast-crc32c-4k

https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#ig_expand=1566,6495&cats=Cryptography

Is there any lossless audio format that uses anything "cryptographic", except TTA's encryption feature that nothing supports [note]?

Also, back around 2005, TTA went for a model where there are no presets. Imagine a Monkey's Audio format which has just one mode (circa "fast", for CDDA purposes). That makes it "kinda useful" for torrenting out, because no user will retrieve it and then mess up the shareability by a recompression in-place. There is a collection of fan art that uses TTA for that purpose, and also claims that one reason to use it is the lack of metadata ... which is not true, but saying so discourages users from trying to tag and thus change the "CD image" files.

For file sharing purposes, it would make sense to have a cryptographic hash per block, and share each block as a segment. Would require its own P2P thing that will never happen.

don't think it's necessary, for sending files you rely on the transport layer to verify that it's transporting properly

In a perfect world frames wouldn't even need a crc checksum to validate, everyone would store their audio on a modern filesystem that does block-level checksumming and filesystem API's would be mature to the point that a decoder could determine when a block fails and where the next good block is to be able to skip the corruption cleanly.

Quote from: TBeck on 2022-06-30 15:51:32

Future

The next release should add support for the AVX2 instruction set. I achieved encoding speed improvements of about 14 percent for preset -p4m on my primary system (Intel Skylake based CPU), less for other presets. But results of my secondary (Haswell based) system were discouraging: Maximum improvement of 8 percent for presets p4 and p4e and up to 23 percent slower encoding for p2m, p3m and p4m!

Those presets make the most use of AVX2-instructions and should also benefit the most. But they seem to trigger the automatic down clocking mechanism of the cpu. AVX2 base and turbo frequencies are lower than the regular ones. This wouldn't hurt too much if the encoder would mostly use AVX2 instructions, but that's not the case. I havent profiled the code yet but i would estimate that about 30 percent of the encoding time goes to AVX2 instructions. And this is no continuous block, instead blocks of x86/SSE2 and AVX2 instructions are alternating.

That's bad, beacuse it will cause many transitions between the different clock rates. During such transitions the speed can be much slower than the lower clock rate would suggest. After the last AVX2 instruction the lower clock rate will be maintained for a considerable amount of time, therfore succeeding non-AVX2 instructions will also be excecuted slower.

Well, my haswell cpu is an 35w low power quad core, quite a challenge for an older desktop microarchitecture. The difference between regular and AVX2 clock most likely is considerably bigger than for the common 65W+ cpus.

Nevertheless i am really hesitant to release an AVX2-version which will make encoding on an unknown number of older systems slower. And  imho the possible advantage isn't big enough to justify an elaborate study and implementation of a cpu dependend code path.
...

FWIW modern CPU's downclock much less or not at all in the presence of AVX2 or even AVX512 (I'm thinking desktop, laptop or other heavily power/thermally-constrained hardware may vary). AMD had AVX2 support on paper earlier, but parity support for AVX2 (one instruction per cycle) didn't come until Zen2 (before that AMD's throughput was half intel's clock for clock, but there still a minor benefit to the AVX2 path from less instructions so better performance of cpu-frontend/instruction-cache). AVX512 in particular had a rocky start with intel but AMD's implementation in Zen4 was solid out of the gate. https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html

Haswell was the first arch that had AVX2 so low power haswell has to be the worst-case, I'm guessing a large majority of AVX2 systems in use will see a benefit always. IMO it would be more than sufficient if you had a --no-avx2 flag that a user could use if they have hardware that supports avx2 but that path performs worse.

Quote from: ktf on 2022-06-30 18:42:39

...
You could consider switching from MD5 to something faster. I've been told most SHA checksums are faster simply because they don't have a long dependency chain and can more efficiently use the superscalar properties and out-of-order execution capabilities of modern CPUs. (That would obviously break backward compatibility, but only for checksumming) For FLAC checksumming is quite a significant part of decoding CPU load (and encoding for the fastest presets), so I imagine this is also the case for TAK.[/li][/list]
...

Modern CPU's have acceleration for sha1 and sha256, IMO that's a big reason to use them. For a similar reason crc32c is interesting: https://www.corsix.org/content/fast-crc32c-4k

https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#ig_expand=1566,6495&cats=Cryptography

Quote from: Porcus on 2022-07-01 09:14:04

...
In TAK and WavPack, MD5 is optional and disabled by default - in WavPack it is viewed more as a fingerprint, and even more so after having implemented the non-decoding integrity check.
From that point of view, where MD5 is an optional fingerprint, I think it is a great advantage to have the same [note2] algorithm across FLAC/TAK/WavPack(/OptimFROG if anyone cares):
...

That is MD5's one saving grace IMO, but it also means I'll die before MD5 does.

Quote from: mudlord on 2022-08-13 00:42:16

Quote

You could consider looking into the arithmetic coder used in Daala/AV1. That is an arithmetic coder that was specifically designed to evade any existing patents. I don't know whether that is fast enough for your liking

That hasn't stopped Microsoft from patenting the rANS entropy method, regardless.

Which to me leaves us in the exact same situation as if we use ordinary arithmetic coding. Might as well just use that or range coding. :/

I believe ANS is faster and is easier to multithread. I don't think patent trolling is a reason to discount something, if anything it's proof that a novel development is worth a damn. Just give the middle finger to Microsoft and carry on.