Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: TAK 2.3.2 Beta thread (Read 14211 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

TAK 2.3.2 Beta thread

Beta release 1 of TAK 2.3.2 ((T)om's lossless (A)udio (K)ompressor)

It consists of:

- TAK Applications 2.3.2 Beta 1 in "\Applications".
- TAK Winamp plugin 2.3.2 Beta 1 in "\WinAmp".
- TAK Decoding library 2.3.2 Beta 1 in "\Deco_Lib".

The final release will additionally contain the SDK.

TAK 2.3.2 has been released.

Re: TAK 2.3.2 Beta thread

Reply #1 –
What's new

This release brings unicode support for the command line version and somewhat limited for the GUI version.

Improvements:

- Tiny compression improvements of usually less than 0.01 percent. I only mention it to explain why compressed files may differ from those created by the previous version.
- Considerably smaller and more portable code for the MD5 calculation.

New features:

- Unicode support. The command line version accepts unicode parameters (file names, tags) and writes unicode to the screen. The GUI version supports unicode file names in the open file dialogs, but any screen output is still limited to the Ansi character set. Therefore characters outside of the windows codepage will be displayed as question mark.
- Log files can now alternatively be stored with UTF-16 encoding. This may compensate for the current limitations of the GUI version output. The encoding can be selected in the General options dialog respectively via the new command line switch -lf.

Cleanup:

- Removed detection of the command line parameter -sts (SeekTableSize). It was silently ignored since version 1.1.1 and now will cause an error.

Beta testing

The unicode implementation affects many program parts i haven't touched for years.
For instance it revealed a bug in a module i have written 20 years ago. The core of the codec itself isn't affected but any higher level functions like the command line and gui interface. Bugs are more likely than in previous releases.

Thanks for testing and have fun

Thomas

Re: TAK 2.3.2 Beta thread

Reply #2 –
Results

Here the results for my primary file set.

Code: [Select]
Preset  Compression %
-----------------------------
        2.3.1   2.3.2    Win
-----------------------------
-p0     58.74   58.74    0.00
-p0e    58.33   58.33    0.00
-p0m    58.22   58.22    0.00
-p1     57.84   57.84    0.00
-p1e    57.52   57.52    0.00
-p1m    57.42   57.42    0.00
-p2     56.90   56.89    0.01
-p2e    56.70   56.69    0.01
-p2m    56.59   56.58    0.00
-p3     56.36   56.35    0.01
-p3e    56.27   56.26    0.01
-p3m    56.20   56.19    0.01
-p4     56.03   56.00    0.02
-p4e    55.94   55.92    0.01
-p4m    55.89   55.87    0.01
-----------------------------
Compression in percent relative to the original file size.

There is no good reason to evaluate the speed of this beta version. There are no speed improvements and the beta may be slower than the final version.

Re: TAK 2.3.2 Beta thread

Reply #3 –
There are some improvements in compression and encoding speed (p4m):
Code: [Select]
Version 231, 10 passes ..........
Version 232, 10 passes ..........
Carl Orff Carmina Burana.wav, 00:56:15, 2 ch, 24 bit, 88200 Hz

Version Compress Size_bytes Enc_dur_sec Enc_speed Dec_speed
------- -------- ---------- ----------- --------- ---------
WAV          100 1786113548
231        62,05 1108329449      21,46    157,32    241,49
232        62,05 1108245762      20,88    161,68    240,91

Version 231, 10 passes ..........
Version 232, 10 passes ..........
Fleshgod Apocalypse King.wav, 00:57:30, 2 ch, 24 bit, 48000 Hz

Version Compress Size_bytes Enc_dur_sec Enc_speed Dec_speed
------- -------- ---------- ----------- --------- ---------
WAV          100  993600044
231        76,03  755477876      14,83    232,57    497,99
232        76,03  755452762      14,58    236,63    496,19

Version 231, 10 passes ..........
Version 232, 10 passes ..........
The Prodigy No Tourists.wav, 00:37:45, 2 ch, 24 bit, 44100 Hz

Version Compress Size_bytes Enc_dur_sec Enc_speed Dec_speed
------- -------- ---------- ----------- --------- ---------
WAV          100  599319626
231        78,45  470190270        9,69    233,86    523,7
232        78,45  470174434        9,58    236,5    520,53
“We are not stuff that abides, but patterns that perpetuate themselves.” – Norbert Wiener

Re: TAK 2.3.2 Beta thread

Reply #4 –
Quote from: TBeck on
Beta release 1 of TAK 2.3.2 ((T)om's lossless (A)udio (K)ompressor)
Great to hear @TBeck! And luckily you did not repeat the "fault" of the very first version which you announced on an April 1st IIRC ;)

Re: TAK 2.3.2 Beta thread

Reply #5 –
Quote from: Sunhillow on
And luckily you did not repeat the "fault" of the very first version which was announced on April 1st ;)
35 minutes close: https://hydrogenaud.io/index.php?topic=122317.msg1009743#msg1009743 , last line.

Now testing: 4 channels, 5.1, ...

Re: TAK 2.3.2 Beta thread

Reply #8 –
Testing a few multi-channel files found at https://thedigitaltheater.com/dolby-trailers/
No, not the 7.1 as they are, that isn't supported - I split them into 4ch file + 4ch file, into 5.1 file + stereo (SL+SR) and into stereo+stereo+stereo+stereo - just to see how TAK would compete against those codecs that support 8 channels.
Guess what, TAK does damn well even if having only stereo to play with. MPEG-4 ALS can beat TAK if given 40x to 100x the encoding time, so who knows what would have happened if TAK implemented a -pATIENCEPLEASE.

The corpus has lots of wasted bits, so TAK sizes are about 1/3 of the 1.46 GB PCM. And about 1/2 of Dolby TrueHD.


Anyway, space saved over TAK 2.3.1: around 0.012 percent for -p4 or -p4m. Not percentage points.

Re: TAK 2.3.2 Beta thread

Reply #9 –
Quote from: eddie.zato on
There are some improvements in compression and encoding speed (p4m):
Nice. I don't see any speed improvement on my system. Maybe it's cpu dependend.
Quote from: Sunhillow on
Great to hear @TBeck! And luckily you did not repeat the "fault" of the very first version which you announced on an April 1st IIRC ;)
I really tried! But there seemed to be a jinx on this release. When my automated encoder/decoder test set was finished i realized that unicode output of the command line version was gone. Took me quite some time to fix it. And more went wrong. It wasn't meant to be...
Quote from: Porcus on
Testing a few multi-channel files found at https://thedigitaltheater.com/dolby-trailers/
No, not the 7.1 as they are, that isn't supported - I split them into 4ch file + 4ch file, into 5.1 file + stereo (SL+SR) and into stereo+stereo+stereo+stereo - just to see how TAK would compete against those codecs that support 8 channels.
Could it be that you would like to see 7.1 support in Tak?

Well, it's now quite high on my to do list, but in the past days i have come to the conclusion that full unicode support for the GUI version and an 64-bit implementation are most important to keep Tak usable.

I had to switch to a newer programming environment to implement this. Yesterday i could run the first GUI version with complete unicode support. Unfortunately the exe file is now 3.5 times bigger.

Just now i decoded the first file with an 64-bit decoder... (Still without assembler optimizations.)

I don't want to raise wrong expectations, the transition to 64 bit is still quite a lot of work and will probably require 2 further releases: One to make it basically work and a second with optimizations for the 64-bit mode.

But If nothing totally unexpected happens, the release frequency will clearly rise.

Re: TAK 2.3.2 Beta thread

Reply #10 –
Quote from: TBeck on
Quote from: Porcus on
Testing a few multi-channel files found at https://thedigitaltheater.com/dolby-trailers/
No, not the 7.1 as they are, that isn't supported - I split them into 4ch file + 4ch file, into 5.1 file + stereo (SL+SR) and into stereo+stereo+stereo+stereo - just to see how TAK would compete against those codecs that support 8 channels.
Could it be that you would like to see 7.1 support in Tak?

Well, it's now quite high on my to do list

I have no 7.1 files except for testing, so ...
 
What I am thinking is that this is one of those things that could give TAK a push:
* it compresses 5.1 better than anything that isn't awfully CPU-expensive, so nothing else would do 7.1 as well,
* if you got a Matroska profile and ffmpeg support for 7.1 TAK before they bother to support 5.1 Monkey's, there is pretty much no reason left to choose Monkey's over TAK. (Sure, Monkey's can handle AU/SND and now also > 4 GB Wave64/RF64, and yes there are *n*x ports of 3.99 for encoding, but ...)

But whether to do it? Unqualified guess is it would depend on whether interchannel decorrelation has to be part of the format spec, or can be left to the encoder to choose? (How is it about 5.1? Is decorrelation strategy written into the format, or could you write a Takc.exe version that would encode them as 6xmono and say ffmpeg would still decode?)
If it doesn't require format tuning, then why not? It will already compress better than most, and compression could be left to improve later. (Same probably goes for 32 bit integer I guess?)

Looking at the 7/8 channel status of the others:
* WavPack: Isn't that the only thing that - as long as there is playback support - is actually sure to work, allocating the channels where they are supposed to be?
* FLAC: format requires a certain channel allocation. Seems to work with Blu-Ray 7.1 allocation, but otherwise there is no requirement that a third-party decoder be able to understand a waveformat_extensible channel mask tag. The first 4 channels in the Microsoft order means 3.1 (and WavPack defaults to that) - while a FLAC decoder that does not read a channel mask tag, shall default to 4.0 as per SMPTE/ITU.
* ALAC: not unlike FLAC, the format uses a certain channel allocation - or you will probably be in trouble. And, does not compress too well.
* Monkey's: ffmpeg doesn't support multi-channel .ape, backwards compatibility has been broken in the builds, 6x as CPU hungry decoding as TAK (that could matter when there are eight channels ...) - and once there are wasted bits around, it performs bad.
* TTA: no support for channel allocations. Sucks big time. Avoid.

Re: TAK 2.3.2 Beta thread

Reply #12 –
Maybe i should put the winamp plugin into a separate zombie release...

The 64-bit version is nearly finished.

I had to switch to a newer Delphi compiler (XE7) with 64-bit support and had to adapt the assembler code. When i performed the first tests, the 64-bit version was encoding more than 12 percent slower (preset -p4m)!

I knew that the 64-bit code would be slower but this dimension was unexpected. To check, if the new compiler was responsible, i also used it to compile the 32-bit- version: 6 percent slower than before.

This seemed to make sense: Part of the speed disadvantage attributed to the 64-bit penalty, part to a probably less efficient compiler.

Since i don't like a new version to be slower than the previous one i tried some optimizations. Then things got really strange: Speed variations of up to 13 percent depending on the code alignment occured. And this although i was controlling the alignment of any real number cruncher function / loop! And even if i wouldn't care for code aligment at all, the effect on recent cpus  should not be bigger than maybe 2 to 3 percent.

It took me days of thinking, trying and googleing. This time the answer is not "42" but "Intel":  Mitigations for Jump Conditional Code Erratum (PDF)

Skylake microarchitecture based cpus up do Coffee Lake suffer a significant speed penalty if a jump instruction crosses or ends at an 32 byte memory address boundary. In case of macro-op fusion this also applies to the preceding instruction.

I had never heard of it before.

Maybe this info is helpful for someone.

Revision of my assembler code wasn't fun but at least the 32 bit version is now (depending on the preset) up to 6 percent faster than V2.3.2. Revision of the 64 bit code comes later, therfore no speed values for now. Cpus not affected by the bug may loose a really tiny bit of speed.

Re: TAK 2.3.2 Beta thread

Reply #13 –
Quote from: TBeck on
It took me days of thinking, trying and googleing. This time the answer is not "42" but "Intel":  Mitigations for Jump Conditional Code Erratum (PDF)
That's one part of what I really like about programming with SIMD intrinsics: it is almost assembler, but the compiler can still do some smart stuff reordering instructions. Compilers usually optimize for intricacies such as this you link. From a quick google search it seems something similar does not seem to exist in Delphi, which is a pity.
Music: sounds arranged such that they construct feelings.

Re: TAK 2.3.2 Beta thread

Reply #14 –
Yes, Delphi is not really intended to be used for this kind of low level stuff.

I just finished the adaption of the 64-bit assembler code. Unfortunately decoding still is 10 to to 13 percent slower than with the 32-bit version, encoding 2 to 5 percent.

Looks as if i will only release an 64-bit version of the decoder library to support third party 64-bit applications. As long as 32-bit applications are supported by the OS, i see no reason to publish a version of Takc.exe that has only disadvantages.

Re: TAK 2.3.2 Beta thread

Reply #15 –
64-bit library still serves a mission.
refalac can decode both FLAC and WavPack files (including splitting image with embedded cue to individual .wav files) but require 64-bit.

BTW, will a beta2 change decoded files or are you just doing speed optimization? As I had started a little bit of testing on channel decorrelation when you were about to post the first beta, I included it, and I have a couple of more running - if there is a beta2 with any compression changes coming up soon, I can wait for that.

Re: TAK 2.3.2 Beta thread

Reply #16 –
Quote from: TBeck on
Yes, Delphi is not really intended to be used for this kind of low level stuff.
You could consider moving to C for just the number crunching. It suppose it makes stuff more complicated but I haven't got a clue how much.

For inspiration you could consider looking at the libFLAC code with intrinsics, as both TAK and FLAC are predictive codecs (AFAIK) there could be some overlap. libFLAC is licensed 3-clause BSD, which has no 'viral' aspect like GPL, if you by any chance copy some code directly.
Music: sounds arranged such that they construct feelings.

Re: TAK 2.3.2 Beta thread

Reply #17 –
Quote from: Porcus on
BTW, will a beta2 change decoded files or are you just doing speed optimization? As I had started a little bit of testing on channel decorrelation when you were about to post the first beta, I included it, and I have a couple of more running - if there is a beta2 with any compression changes coming up soon, I can wait for that.
Sorry, posting about the next version in this thread was misleading. Small speed optimizations for skylake based cpus and the 64-bit library will come soon as V2.3.3. No compression improvements are planned for this version. The tiny improvements in 2.3.2 are a by-product of a revision of some very old code that revealed a suboptimal storage of the predictor coefficients.

V2.3.2 Final will contain 2 bug fixes for the encoder which theoretically could cause an "invalid floating point operation' respectively an "illegal memory access" error. The bugs were slumbering in the now revised code for many years,  probably not a big deal since i never heard of users encountering those errors.

Quote from: ktf on
Quote from: TBeck on
Yes, Delphi is not really intended to be used for this kind of low level stuff.
You could consider moving to C for just the number crunching. It suppose it makes stuff more complicated but I haven't got a clue how much.
Now that i have finished the boring job of aligning the code to avoid intels bug i am happy again writing pure assembler code.

Things i want to try:

1.  Port to Lazarus to check the quality of the Free Pascal compiler. Also the fastest way to a Linux version.
2.  Port the decoder library to C.
3. Create an encoder library in C.

The last two points obviously are similar to your proposal.

But no promises. 2 and 3 will definitely not happen within the next couple of months.

I also want to try some ideas for compression improvements, but i am not to optimistic about this. Over the past years i already tried various ideas but either the code was painfully slow or the conpression improvements were too small to justify an incompatible format change.  Because ot the small prospects of success i am only occasionally motivated to invest another couple of weeks into the evaluation of a new idea.

Optimizations for some of the special respectively problematic files Porcus and others have reported lately are more likely.

Re: TAK 2.3.2 Beta thread

Reply #18 –
Quote from: TBeck on
I also want to try some ideas for compression improvements, but i am not to optimistic about this. Over the past years i already tried various ideas but either the code was painfully slow or the conpression improvements were too small to justify an incompatible format change.
Meaning, a "-p5" would have to break compatibility and a "-p4mmm" would not be worth waiting for?

Just thinking aloud: An easily available "recompress TAK files" feature (like FLAC/WavPack) could make a "-p4mmm" more tempting - especially if it would leave untouched files that are already maxed out. I would imagine that many users will be happy about running -p0, tag & RG, and then periodically let the GUI scan the TAK collection, find the less-than-max and recompress those only.

Still I'd be cautious: I guess that people might judge TAK by the maximum compression setting, whatever that is. Many users apply -8 or -8e on autopilot - and some even complain about WavPack's "x6" (Then.Do.Not.Use.It!!)

Re: TAK 2.3.2 Beta thread

Reply #19 –
Quote from: Porcus on
As I had started a little bit of testing on channel decorrelation when you were about to post the first beta, I included it, and I have a couple of more running
... updated with un-dramatic figures on the impact of the beta for "near-mono" signals.

Re: TAK 2.3.2 Beta thread

Reply #20 –
Quote from: Porcus on
Meaning, a "-p5" would have to break compatibility and a "-p4mmm" would not be worth waiting for?
You never really now until you try... But i think chances for compatible compression improvements are small. Ok, i could increase the maximum predictor order from 160 to 256. Any official decoder since V2.0. supports it. But this only helps some files and reduces the speed of -p4m by the factor 2. My primary sample set contains relatively many such files and gains about 0.1 percent of compression. For most other sets its not more than 0.05 percent.

Then there is a currently unused option regarding channel correlation: A frame can be split into two parts of arbritrary size which use different parameters for channel decorrelation. Compression of my primary sample set is improved by 0.075 percent. Again it's less for most other sets. But i found very significant improvements for some problem samples. Of course files with rapidly changing characteristics benefit the most from a finer temporal adaption. But again this comes at the price of considerably slower encoding and additionally more encoder complexity.

Generally it does not make sense to sacrifice a significant amount of speed to improve the compression of some problem samples. Would i do this for all of them, Tak would get a lot slower.

About an incompatible compression improvement i tried that was far too slow.

Quote from: Porcus on
Just thinking aloud: An easily available "recompress TAK files" feature (like FLAC/WavPack) could make a "-p4mmm" more tempting - especially if it would leave untouched files that are already maxed out. I would imagine that many users will be happy about running -p0, tag & RG, and then periodically let the GUI scan the TAK collection, find the less-than-max and recompress those only.
Indeed, this sounds interesting.

Quote from: Porcus on
Still I'd be cautious: I guess that people might judge TAK by the maximum compression setting, whatever that is. Many users apply -8 or -8e on autopilot - and some even complain about WavPack's "x6" (Then.Do.Not.Use.It!!)
100% agree. Tak's speed will be primarily judged by the speed of it's strongest setting. It's nice to have those super fast presets, but would i use them? Maybe for some niche applications.

Tak's formula is "High Speeed + High compression".

It would require a very significant compression improvement to make me dilute this.

But there is still hope for improvements that fit.

Re: TAK 2.3.2 Beta thread

Reply #21 –
BTW: I intend to release the final version within the next days. Automated testing has been done. Now i am working on some additions to the documentation of the SDK.

Re: TAK 2.3.2 Beta thread

Reply #22 –
Quote from: TBeck on
100% agree. Tak's speed will be primarily judged by the speed of it's strongest setting.
Its strongest preset I'd think. 

Reference FLAC encoding can be slowed down indefinitely by throwing options at the end (here I let it run for five to six days on 8 GB) - and even if it allows users direct access to the filterbank, I guess you need to be more than a HA regular to bother about the speed penalty from custom "-A".

As long as "-p0" to "-p4m" are highlighted as the TAK presets, you might consider "auxiliary options" as for example a -exhaustive-channel-decorrelation-search (that, without need to tout it loudly: it could also improve stereo, right?).
And you already have settings outside presets - block size limits, which I forgot about in this test, reply 38, and only afterwards saw that I had mentioned myself (hurry do another and edit, that's what happened).

Out of presets = out of sight = out of mind  - even if "--super-secret-totally-impractical-compression-level" was a formulation that got some attention ;-)


Quote from: TBeck on
It's nice to have those super fast presets, but would i use them?
Even myself didn't really pay attention to how good -p0 is. Of course when you have boldly claimed high-ape size FLAC speed (haha, that'd be the day!) and even practically met it if decoding from spinning drive ... who cares about encoding faster than FLAC -0?
(One should - especially if you offer an automated compress-fast-first, recompress-later ...)

Actually, since TAK loses to FLAC at RAMdisk-measured decoding speed - but still has so much margin down to ALAC/WavPack -f - then that is probably the only thing that can be sacrificed without people whining about "slow!"

Re: TAK 2.3.2 Beta thread

Reply #23 –
Quote from: Porcus on
Its strongest preset I'd think. 
And it was the swiney snout who used the word "setting" initially.  Nobody's Fault But Mine.

(943 kbit/s with ALAC,     922 with FLAC -8,      913 for WavPack -hhx6,     905 for TAK -p4m,     901 Monkey's Insane     and     892 for OptimFROG --preset max, all on the un-remastered version.)

Re: TAK 2.3.2 Beta thread

Reply #24 –
@Tbeck:  Is there a beta version of the 64-bit tak_deco_lib.dll available for public testing?