Beta release 1 of TAK 2.3.1 ((T)om's lossless (A)udio (K)ompressor)
It consists of:
- TAK Applications 2.3.1 Beta 1 in "\Applications".
- TAK Winamp plugin 2.3.1 Beta 1 in "\WinAmp".
- TAK Decoding library 2.3.1 Beta 1 in "\Deco_Lib".
The final release will additionally contain the SDK.
Download:
What's new
This release - i like to call it the back-to-work release - brings significant speed optimizations for the encoder and a lot of source code cleanups in preparation of a migration from Delphi to Lazarus and/or C. Practical goals are Linux binaries and open source releases. This will be done step by step depending on my spare time. The cleanup revealed several bugs which affected the compression efficiency, but never the data integrity.
Improvements:
- Encoding speed improvements between 6 and 28 percent for my primary file set. The slower presets benefit most. System: i5-4460 (Haswell)
- Decoding speed improvements between 2 and 7 percent for my primary file set. The slower presets benefit most. System: i5-4460 (Haswell)
- Really tiny compression improvements because of some bug fixes. See below.
- Better source code and smaller binaries.
New features:
- The multi-threaded encoder now supports up to 8 instead of 4 threads.
- The cpu optimization option None now really disables any assembler optimizations. Previous versions still used some i386-assembly that could only be disabled by compiler switches. The new option ASM enables this code and is equivalent to None of earlier versions.
Fixes:
- Small bugs in the encoder decreased compression by usually not more than 0.01 percent. One of my file sets lost 0.06 percent. Some special files will show stronger effects.
- A bug in the plain pascal code path significantly decreased compression of some presets: up to 1.19 percent for my primary file set. To encounter this bug, you had to use V2.2.0 or 2.3.0 and explicitly disable assembler optimizations (-cpuNone) or run Tak on a cpu without even the MMX-instruction set (e.g. Pentium 1).
- Added new tests to my already extensive validation procedure to detect regressions of the plain pascal code path. Til now i only checked the data integrity.
- None of these bugs affected the data integrity.
Cleanup:
- Removed the assembler optimizations from the TAK 1.x decoder and made it a lot more compact.
- Replaced MMX with SSE2 assembly.
- Replaced FPU with SSE2 assembly.
- Removed assembler optimizations which had little effect on the speed.
- Removed a lot of partial redundancies which had been introduced to gain some speed.
Known issues:
- If you use pipe decoding and the application reading the pipe is beeing terminated before the whole file has been read, TAKC may get into an endless loop and has to be manually killed with the task manager. I don't think this is a big issue but i will try to fix it in one of the next versions. BTW: Big thanks to shnutils for testing the pipe decoding!
- There seem to be some compatibility issues with pipe decoding to some other applications ("crc1632.exe" has been reported). I will try to fix it in the next release.
Beta testing
This beta release has already gone through extensive testing performed by my automatic scripts. Nevertheless there may be bugs left. Therefore i would like you to verify the proper function of the codec: Compress -> Decompress -> Compare resulting wave with the original file, either by a binary compare or by the use MD5-check sums.
BTW: I just noticed a missing update of the command line help output: The option -tn# is now limited to 8 and not 4 threads. It's right in the applications readme.
Since i only own a quad core, it would be nice if someone with an octa core could try preset -p4m with -tn8 (8 threads).
Certainly i am very interested into efficiency comparisons. The speed of the final release may vary in the range of about +/- 1 percent, simply because i remove the "beta" from the program name and this alters the code alignment.
Thanks for testing and have fun
Thomas
Here the results for my primary file set.
Test system: Intel i5-4460 (3.4 GHz Turbo / 1 Thread), Windows 7.
Preset Compression % Enco-Speed Deco-Speed
---------------------------------------------------------------------------------
2.3.0 2.3.1 Win 2.3.0 2.3.1 Win % 2.3.0 2.3.1 Win %
---------------------------------------------------------------------------------
-p0 58.75 58.74 0.01 755.81 800.87 5.96 764.62 779.10 1.89
-p0e 58.33 58.33 0.00 567.60 612.74 7.95 763.90 784.35 2.68
-p0m 58.22 58.22 0.00 312.09 348.79 11.76 766.54 786.71 2.63
-p1 57.84 57.84 0.00 607.58 657.42 8.20 720.04 756.14 5.01
-p1e 57.52 57.52 0.00 372.65 412.89 10.80 718.55 757.34 5.40
-p1m 57.42 57.42 0.00 231.08 262.61 13.64 720.15 759.25 5.43
-p2 56.90 56.90 0.00 432.65 511.77 18.29 634.70 677.83 6.80
-p2e 56.70 56.70 0.00 236.19 296.02 25.33 629.59 676.11 7.39
-p2m 56.59 56.59 0.00 130.25 165.97 27.42 630.84 678.14 7.50
-p3 56.36 56.36 0.00 211.95 257.29 21.39 603.48 645.93 7.03
-p3e 56.27 56.27 0.00 165.70 203.79 22.99 601.90 646.04 7.33
-p3m 56.20 56.20 0.00 84.92 109.08 28.45 601.13 645.24 7.34
-p4 56.03 56.03 0.00 131.12 155.15 18.33 553.99 591.48 6.77
-p4e 55.94 55.94 0.00 109.73 130.42 18.86 553.04 591.65 6.98
-p4m 55.89 55.89 0.00 53.26 68.33 28.30 552.41 591.60 7.09
---------------------------------------------------------------------------------
Compression in percent relative to the original file size. Speed as multiple of realtime playback.
Thank you Thomas for the progress, still like TAK and stay with it as I really like the efforts and results you achieved.
I would like to fire the TAK-beta on 3 machines, one is an octa-core btw.
Therefore I copied some wavs I accidentally had nearby (Classic+ModernTalking+PhilCollins) as a file-set. 48 tracks.
Can still put some mixed techno-stuff (Thunderdome, Prodigy, ChemicalBrothers, ..) in, if this helps. But would mean some effort.
I remember you once had some sort of testing-mode..or is there any parameter or simple advice to easily support you with a bunch of values of different modes?
There's no need to waste time on this, otherwise it still results in single values, which also might be okay.
AMD PhenomII X4 960T BlackEdition, 3.0 GHz, 6 threads (anno 2011), Win7 64bit (GUI, p4m; @compression: all 6 cores @100%, @decompression: seems only 1 core involved)
Compression -p4m 52.56 %
Enco-Speed 186.00
Deco-Speed 381.92/387.67
I ran a simple test using max (-p4m) compression on a CD image (play duration 57:56) on Core 2 1.83GHz 2GB RAM 120GB HDD.
It was known this laptop throttles down to about to about 1GHz when nearing the upper temperature range so the individual tests were run several minutes apart from another. Despite this, the image file took is large enough that it appears throttling occurs during encoding. The decode times are puzzling and the laptop could not give consistent results after re-running the test several times, so I posted the best result. It's better to do future testing on a desktop system.
The improvements in the new beta are quite impressive!
binary settings ratio enc decode
------ --------- ------ ----- ------
2.3.0 -p4m 46.92% 14.51 168.86
2.3.1b -p4m 46.91% 15.54 165.54
2.3.0 -p4m -tn2 46.92% 14.91 ...
2.3.1b -p4m -tn2 46.91% 16.56 ...
I also did some tests using -p4m on my Xeon X5675 (Westmere EP from 2011, 6 cores/12 threads @ 3,06 GHz). The encoding speed went from around 49 to around 60 which is ~22 percent faster. Very nice!
Thanks to this thread I now also know that TAK has a multi-threaded encoding mode. Converting CD images using CueTools is now much faster. If only I had known this before :).
TAK is my favourite format for archiving music. I would also like to use it for playback (I use FLAC for this), but the provided Winamp plugin (still) doesn't work correctly in MediaMonkey; it can play tracks, but not all tags are displayed or are "deleted" when you try to edit them. But this is probably a MediaMonkey problem, although other Winamp input plugins work correctly. Maybe I will report it to MediaMonkey or switch to MusicBee sometime in the future if this can't be fixed otherwise.
new results
Intel Xeon W-2145 (skylake)(Q3 2017), 3.70 GHz, 8 cores / 16 threads possible, Win10, SSD
binary settings ratio enc decode
------ --------- ------ ----- ------
2.3.0 -p4m -tn4 52.56 % 236.04 614.41
2.3.1b -p4m -tn4 52.56 % 340.85 698.34
2.3.1b -p4m -tn8 52.56 % 575.82 697.55
AMD PhenomII X4 960T BlackEdition (anno 2011), 3.0 GHz, 6 threads, Win7 64bit, SSD
binary settings ratio enc decode
------ --------- ------ ----- ------
2.3.0 -p4m -tn4 52.56 % 99.50 391.07
2.3.1b -p4m -tn4 52.56 % 142.69 456.39
2.3.1b -p4m -tn6 52.56 % 181.52 456.25
Thomas, great job ! :D
I wrote myself an autoit-script (.au3) to easily collect the results, the content normally shown at the cmd-prompt.
The idea is to have all stuff in 1 simple TEST-folder and save the results. There are some must-haves regarding the directory-structure and stuff needed. The directory-name itself should not contain any white spaces, so keep it simple then it's a no-brainer.
You HAVE TO make COPIES of your wav-files and DON'T use EXISTING directories. CREATE a NEW directory for TESTING (as the scripts deletes ALL .tak-files witihn your used music-directory).
See attachment or
#cs ----------------------------------------------------------------------------
AutoIt: https://www.autoitscript.com
Author: sPeziFisH
Script
Function:
run 2 different TAK-Binaries (the CLI-version 'takc') and save the results of the command-windows as .txt (meant as log-file)
with each run a seprate directory with log-files is created
Name:
TAK_HA.au3
#ce ----------------------------------------------------------------------------
#include <AutoItConstants.au3>
;directory-name (next to this script) containing copies of wav-files - your music-file-set to test TAK with
Local $dir_music = @ScriptDir & "\" & "wav-music" ; <<< CHANGE if neccessary
;TAK-Version 1
Local $dir_tak = "TAK_2.3.0"
;TAK-Version 2 - the test/beta-version
Local $dir_tak_test = "TAK_2.3.1_Beta_1"
;Location of takc within TAK-dir
Local $dir_takc = "\Applications\takc"
Local $dir_tak_full = @ScriptDir & "\" & $dir_tak & $dir_takc
Local $dir_tak_test_full = @ScriptDir & "\" & $dir_tak_test & $dir_takc
;MsgBox(0, "info",@ScriptDir)
;preset (encoding)
Local $preset[13]
$preset[12] = "-p4m" ; <<< CHANGE if neccessary
;further parameter-set
Local $params = " -tn4 -cpuSSSE3 " ; <<< CHANGE if neccessary
;further parameter-set test/beta-version
Local $params_test = " -tn6 -cpuSSSE3 " ; <<< CHANGE if neccessary
;create directory nearby script containing the txt-files
;name-sheme year-month-day hour.minute.sec (eg. 2018-04-06 19.50.22)
Local $dir_name = @ScriptDir & "\" & @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & "." & @MIN & "." & @SEC
DirCreate($dir_name)
;complete cmd incl. pipe
$cmd_enc_tak = @ComSpec & " /c " & $dir_tak_full & " -e " & $preset[12] & $params & '"' & $dir_music & "\*" & '"'
$cmd_enc_tak = $cmd_enc_tak & " > " & '"' & $dir_name & "\" & $dir_tak & "_enc_" & $preset[12] & ".txt" & '"' ;pipe
$cmd_enc_tak_test = @ComSpec & " /c " & $dir_tak_test_full & " -e " & $preset[12] & $params_test & '"' & $dir_music & "\*" & '"'
$cmd_enc_tak_test = $cmd_enc_tak_test & " > " & '"' & $dir_name & "\" & $dir_tak_test & "_enc_" & $preset[12] & ".txt" & '"' ;pipe
;MsgBox(0, "info", $cmd_enc_tak_test)
$cmd_dec_tak = @ComSpec & " /c " & $dir_tak_full & " -t " & '"' & $dir_music & "\*" & '"'
$cmd_dec_tak = $cmd_dec_tak & " > " & '"' & $dir_name & "\" & $dir_tak & "_dec.txt" & '"' ;pipe
$cmd_dec_tak_test = @ComSpec & " /c " & $dir_tak_test_full & " -t " & '"' & $dir_music & "\*" & '"'
$cmd_dec_tak_test = $cmd_dec_tak_test & " > " & '"' & $dir_name & "\" & $dir_tak_test & "_dec.txt" & '"' ;pipe
;MsgBox(0, "Info", "existing .tak-files get deleted")
FileDelete($dir_music & "\*.tak")
sleep(2000)
;MsgBox(0, "Info", "next: " & $dir_tak_test)
RunWait($cmd_enc_tak_test) ;encoding/compression
sleep(1000)
RunWait($cmd_dec_tak_test) ;decoding/decompression
sleep(1000)
;MsgBox(0, "Info", "existing .tak-files get deleted")
FileDelete($dir_music & "\*.tak")
sleep(2000)
;MsgBox(0, "Info", "next: " & $dir_tak)
RunWait($cmd_enc_tak) ;encoding/compression
sleep(1000)
RunWait($cmd_dec_tak) ;decoding/decompression
sleep(1000)
MsgBox(0, "Info", "finished !")
Great!
Anyone took the time to compare TAK, WavPack and FLAC already with different settings to see which is best/fastest/slowest etc.?
Would love to know the results, but I just don't have the time to do it at the moment.
http://wiki.hydrogenaud.io/index.php?title=Lossless_comparison
http://www.audiograaf.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%204.pdf
http://wiki.hydrogenaud.io/index.php?title=Lossless_comparison
http://www.audiograaf.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%204.pdf
That comparison is quite old. Isn't there a more recent version?
even though nothing >4 threads:
Intel Core i3-350M (2010, mobile/notebook), 2.26 GHz, 2 cores / 4 threads possible, Win7, SSD
binary settings ratio enc decode
------ --------- ------ ----- ------
2.3.0 -p4m -tn2 52.56 % 38 295
2.3.0 -p4m -tn4 52.56 % 51 295
2.3.1b -p4m -tn2 52.56 % 48 317
2.3.1b -p4m -tn4 52.56 % 65 317
AMD A6-5400K (Q3 2012), 3.6 GHz, 2 cores / 2 threads possible, Win10, SSD
binary settings ratio enc decode
------ --------- ------ ----- ------
2.3.0 -p4m -tn2 52.56 % 49 355
2.3.1b -p4m -tn2 52.56 % 61 430
I also ran several runs with all systems (again) to be sure to have consistent results..nothing to change.
That comparison is quite old. Isn't there a more recent version?
guess what.
Keep in mind everyone still wants this or let's say sort of, things don't change that much. It means effort and a lot of people don't have time at moments.
Automation makes sense, maybe an automated contribution system (like LuxMark). All this would have to be done.. it's effort too and it's more effort.
That comparison is quite old. Isn't there a more recent version?
As far as I know, no, there isn't newer comparison of lossless formats.
Also that comparison table doesn't reflect even near the reality of multichannel audio compression. Some formats that may tuned better for compression ratios in Stereo, in multichannel they don't so well and vice versa.
I archive my multichannel DVD-Audio and Blu-Ray Audio discs in TAK since I found that it's essentially the state of art in compression ratios in multichannel audio.
BTW, I recompressed by curiosity my 5.1 Blu-Ray rip of Coldplay's "A Head Full Of Dreams" album and I got a total save of 12KB (Files are around ~2GB in total)! Also I didn't keep exact times but it was no doubt noticably faster.
Excited to see work resumed on the format! Keep up the good work!
As far as I know, no, there isn't newer comparison of lossless formats.
After that, OptimFrog and WavPack have new major versions. I am not convinced that speed/ratio tests matter too much anymore, except that geeks like us enjoy to be impressed over improvements.
Maybe after the next TAK, it could be worthwhile to rerun the comparison, but I guess that whatever codec update that misses next train, will have to wait for loooong.
- Replaced MMX with SSE2 assembly.
CHKCPU32.exe /V | find "extensions"
Instr set extensions: MMX, 3DNow!, SSE
Takc_2.3.0.exe -te -pMax test_sample.wav
test_sample.wav .......... 42.41% 15*
Compression: 42.41 %
Duration: 160.63 sec
Speed: 15.24 * real time
Takc_2.3.1b1.exe -te -pMax test_sample.wav
test_sample.wav .......... 42.41% 5*
Compression: 42.41 %
Duration: 453.33 sec
Speed: 5.40 * real time
:(
CoRoNe:
The AMD chips got support for SSE2 quite later than Intel (Mostly since Athlon 64), but sure that PC of yours has more than 10 years (and possibly more than 12).
While it always feels bad that some software that you use stop working, in this case I would consider this the natural course of events and not try to stop this change. More so when this change does not stop it working, but only make it work slower.
( MMX and SSE code does not work well together. Also, SSE (non-2) cannot do integer arithmetic like MMX, so it's either MMX or SSE2 )
Hi Tom,
I like TAK and would like to use it, but I need a codec without CUE for images, because I buy music mostly online and CUE cant handle them right. Could you maybe add something different to it like ALAC does?
ALAC does not have anything for multi-track encoding, but its container, MP4, already has every supported format covered by chapter support.
I dont care what it is, but nearly all lossless codecs have problems with this kind of stuff and I hope TAK will be the next FLAC but better.
I like TAK and would like to use it, but I need a codec without CUE for images, because I buy music mostly online and CUE cant handle them right.
Care to clarify what kind of features music bought online has that CUE has problems with? I'm genuinely curious.
If you wish a container for TAK you should try to get it supported in Matroska.
When sample rate is different from 44100Hz, or duration is not multiple of 1/75 sec CD frame (or 588 samples), it's not something
CUE is meant for.
There are many formats with chapter support now but they may not be sample accurate (millisecond accuracy is not enough).
I'm going to seize the fact this old thread has been risen from the dead to say:
Thank you Thomas
@TBeck , for creating a CODEC with enables me to reach (unlike its other lossless competitors) bitrates
lower (~290Kbps) than the original 320Kbps-MP3 RCA Victor recordings I'd downloaded from The Web Archive and later done some sound processing in.
There are many formats with chapter support now but they may not be sample accurate (millisecond accuracy is not enough).
FLAC in oga chained into multitrack file can be solution.
FLAC in oga chained into multitrack file can be solution.
Yes. but it's not as well supported as ordinary FLAC files.
As for chapters, Matroska chapter timestamp has nanoseconds precision internally, so it's more than enough.
MP4 QuickTime style chapters are described in media timescale (usually equal to samplerate) so it's sample accurate.
MP4 Nero style chapters are in 100 nanoseconds precision, so it's also enough.
So, both of them have enough accuracy as for container format specification.
However, what we see everywhere is a "simple chapter format" descrbed at https://mkvtoolnix.download/doc/mkvmerge.html#mkvmerge.chapters and https://wiki.xiph.org/Chapter_Extension, which uses only 3 decimal places (implies millisecond precision).
So it's more about tooling and input format.
Hi Thomas, it's been 4 years since you said you will support Unicode. I am glad to see an update but without Unicode support I cannot use TAK as my archiver format :(
Unicode support is not needed if your converting software uses ASCII compatible temporary filenames, and your player software handles file paths outside of the TAK decoder library.
Unicode was already reported to be a major issue to fix, since the whole library and encoder are made with Delphi, which apparently never got past the ANSI stage, so never supported Unicode API?
FreePascal Lazarus is said to be (somewhat) compatible to Delphi and is supporting Unicode. Maybe this could be an option for the future?
Unfortunately my experience with Pascal ended with Turbo Pascal 6/7 and its clone Virtual Pascal for OS/2 many years ago ...
Unicode support is not needed if your converting software uses ASCII compatible temporary filenames, and your player software handles file paths outside of the TAK decoder library.
While fb2k solves issue with lack of Unicode support for
file name, it cant help if
folder name contains Unicode characters. (well, actually it can, but only with using option "Convert to temporary location" in Advanced preferences which i not enabled by default)
Unicode was already reported to be a major issue to fix, since the whole library and encoder are made with Delphi, which apparently never got past the ANSI stage, so never supported Unicode API?
https://hydrogenaud.io/index.php/topic,101386.msg838399.html#msg838399
Because my old Delphi 6 from 2001 provides very little (none for the GUI) unicode support and TAK uses some libraries i have written long ago which too don't support unicode.
https://hydrogenaud.io/index.php/topic,101386.msg886066.html#msg886066
While i intended to port TAK to C, this looks unrealistic at the moment. Therefore the next version still has to be written in Delphi. Since my Delphi 6 ist lacking a lot required for the planned features, i finally bought an upgrade to Delphi XE 7 (There was a time limited offer for users of very old versions).
https://hydrogenaud.io/index.php/topic,101386.msg952703.html#msg952703
Delphi XE 7 has built in unicode support. But unfortunately now (3 years later) it does not seem to be a good option for future TAK development. Embarcadero just wrote me, that they will no longer provide upgrade options for older versions. You have to subscribe to a permanent and costly update. For my purposes that's a killer... Last time i looked for a free version, it did not support 64 bit.
Therefore the planned step-wise switch to Lazarus. I wanted to start this as soon as possible to avoid the feeling of wasting time on an inadequate (because too costly) development tool. This obviously changed priorities.
As already noted above, there is FreePascal + Lazarus, supporting both Unicode and 64bit. Switching shoudn't be hard for a project of TAKs size (for example TotalCommander, which is a full blown file manager orders of magnitude more complex did such a switch for its 64bit version while still keeping the source compatible with Delphi for 32bit).
TAK uses a royal shedload of inline assembly.
SSE/MMX? Would be better to use something like Xbyak from a C++ file for ASM optimizations.
Huge speed boost