Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: TAK 2.3.1 Beta 1 (Read 107509 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

TAK 2.3.1 Beta 1

Beta release 1 of TAK 2.3.1 ((T)om's lossless (A)udio (K)ompressor)

It consists of:

- TAK Applications 2.3.1 Beta 1 in "\Applications".
- TAK Winamp plugin 2.3.1 Beta 1 in "\WinAmp".
- TAK Decoding library 2.3.1 Beta 1 in "\Deco_Lib".

The final release will additionally contain the SDK.

Download:

Re: TAK 2.3.1 Beta 1

Reply #1
What's new

This release - i like to call it the back-to-work release - brings significant speed optimizations for the encoder and a lot of source code cleanups in preparation of a migration from Delphi to Lazarus and/or C. Practical goals are Linux binaries and open source releases. This will be done step by step depending on my spare time. The cleanup revealed several bugs which affected the compression efficiency, but never the data integrity.

Improvements:

- Encoding speed improvements between 6 and 28 percent for my primary file set. The slower presets benefit most. System: i5-4460 (Haswell)
- Decoding speed improvements between 2 and 7 percent for my primary file set. The slower presets benefit most. System: i5-4460 (Haswell)
- Really tiny compression improvements because of some bug fixes. See below.
- Better source code and smaller binaries.

New features:

- The multi-threaded encoder now supports up to 8 instead of 4 threads.
- The cpu optimization option None now really disables any assembler optimizations. Previous versions still used some i386-assembly that could only be disabled by compiler switches. The new option ASM enables this code and is equivalent to None of earlier versions.

Fixes:

- Small bugs in the encoder decreased compression by usually not more than 0.01 percent. One of my file sets lost 0.06 percent. Some special files will show stronger effects.
- A bug in the plain pascal code path significantly decreased compression of some presets: up to 1.19 percent for my primary file set. To encounter this bug, you had to use V2.2.0 or 2.3.0 and explicitly disable assembler optimizations (-cpuNone) or run Tak on a cpu without even the MMX-instruction set (e.g. Pentium 1).
- Added new tests to my already extensive validation procedure to detect regressions of the plain pascal code path. Til now i only checked the data integrity.
- None of these bugs affected the data integrity.

Cleanup:

- Removed the assembler optimizations from the TAK 1.x decoder and made it a lot more compact.
- Replaced MMX with SSE2 assembly.
- Replaced FPU with SSE2 assembly.
- Removed assembler optimizations which had little effect on the speed.
- Removed a lot of partial redundancies which had been introduced to gain some speed.

Known issues:

- If you use pipe decoding and the application reading the pipe is beeing terminated before the whole file has been read, TAKC may get into an endless loop and has to be manually killed with the task manager. I don't think this is a big issue but i will try to fix it in one of the next versions. BTW: Big thanks to shnutils for testing the pipe decoding!
- There seem to be some compatibility issues with pipe decoding to some other applications ("crc1632.exe" has been reported). I will try to fix it in the next release.

Beta testing

This beta release has already gone through extensive testing performed by my automatic scripts. Nevertheless there may be bugs left. Therefore i would like you to verify the proper function of the codec: Compress -> Decompress -> Compare resulting wave with the original file, either by a binary compare or by the use MD5-check sums.

BTW: I just noticed a missing update of the command line help output: The option -tn# is now limited to 8 and not 4 threads. It's right in the applications readme.

Since i only own a quad core, it would be nice if someone with an octa core could try preset -p4m with -tn8 (8 threads).

Certainly i am very interested into efficiency comparisons. The speed of the final release may vary in the range of about +/- 1 percent, simply because i remove the "beta" from the program name and this alters the code alignment.

Thanks for testing and have fun

Thomas

Re: TAK 2.3.1 Beta 1

Reply #2
Here the results for my primary file set.

Test system: Intel i5-4460 (3.4 GHz Turbo / 1 Thread), Windows 7.

Code: [Select]
Preset  Compression %           Enco-Speed                Deco-Speed             
---------------------------------------------------------------------------------                            
        2.3.0   2.3.1    Win    2.3.0    2.3.1    Win %   2.3.0    2.3.1    Win %
---------------------------------------------------------------------------------                            
-p0     58.75   58.74    0.01   755.81   800.87    5.96   764.62   779.10    1.89
-p0e    58.33   58.33    0.00   567.60   612.74    7.95   763.90   784.35    2.68
-p0m    58.22   58.22    0.00   312.09   348.79   11.76   766.54   786.71    2.63
-p1     57.84   57.84    0.00   607.58   657.42    8.20   720.04   756.14    5.01
-p1e    57.52   57.52    0.00   372.65   412.89   10.80   718.55   757.34    5.40
-p1m    57.42   57.42    0.00   231.08   262.61   13.64   720.15   759.25    5.43
-p2     56.90   56.90    0.00   432.65   511.77   18.29   634.70   677.83    6.80
-p2e    56.70   56.70    0.00   236.19   296.02   25.33   629.59   676.11    7.39
-p2m    56.59   56.59    0.00   130.25   165.97   27.42   630.84   678.14    7.50
-p3     56.36   56.36    0.00   211.95   257.29   21.39   603.48   645.93    7.03
-p3e    56.27   56.27    0.00   165.70   203.79   22.99   601.90   646.04    7.33
-p3m    56.20   56.20    0.00    84.92   109.08   28.45   601.13   645.24    7.34
-p4     56.03   56.03    0.00   131.12   155.15   18.33   553.99   591.48    6.77
-p4e    55.94   55.94    0.00   109.73   130.42   18.86   553.04   591.65    6.98
-p4m    55.89   55.89    0.00    53.26    68.33   28.30   552.41   591.60    7.09
---------------------------------------------------------------------------------
Compression in percent relative to the original file size. Speed as multiple of realtime playback.

Re: TAK 2.3.1 Beta 1

Reply #3
Thank you Thomas for the progress, still like TAK and stay with it as I really like the efforts and results you achieved.

I would like to fire the TAK-beta on 3 machines, one is an octa-core btw.
Therefore I copied some wavs I accidentally had nearby (Classic+ModernTalking+PhilCollins) as a file-set. 48 tracks.
Can still put some mixed techno-stuff (Thunderdome, Prodigy, ChemicalBrothers, ..) in, if this helps. But would mean some effort.

I remember you once had some sort of testing-mode..or is there any parameter or simple advice to easily support you with a bunch of values of different modes?
There's no need to waste time on this, otherwise it still results in single values, which also might be okay.

AMD PhenomII X4 960T BlackEdition, 3.0 GHz, 6 threads (anno 2011), Win7 64bit    (GUI, p4m; @compression: all 6 cores @100%, @decompression: seems only 1 core involved)
Code: [Select]
Compression -p4m 52.56 %
Enco-Speed 186.00
Deco-Speed 381.92/387.67

Re: TAK 2.3.1 Beta 1

Reply #4
I ran a simple test using max (-p4m) compression on a CD image (play duration 57:56) on Core 2  1.83GHz 2GB RAM 120GB HDD.

It was known this laptop throttles down to about to about 1GHz when nearing the upper temperature range so the individual tests were run several minutes apart from another. Despite this, the image file took is large enough that it appears throttling occurs during encoding. The decode times are puzzling and the laptop could not give consistent results after re-running the test several times, so I posted the best result. It's better to do future testing on a desktop system.

The improvements in the new beta are quite impressive!
Code: [Select]
binary  settings   ratio   enc    decode
------  ---------  ------  -----  ------
2.3.0   -p4m       46.92%  14.51  168.86
2.3.1b  -p4m       46.91%  15.54  165.54
2.3.0   -p4m -tn2  46.92%  14.91  ...
2.3.1b  -p4m -tn2  46.91%  16.56  ...
"Something bothering you, Mister Spock?"

Re: TAK 2.3.1 Beta 1

Reply #5
I also did some tests using -p4m on my Xeon X5675 (Westmere EP from 2011, 6 cores/12 threads @ 3,06 GHz). The encoding speed went from around 49 to around 60 which is ~22 percent faster. Very nice!

Thanks to this thread I now also know that TAK has a multi-threaded encoding mode. Converting CD images using CueTools is now much faster. If only I had known this before :).

TAK is my favourite format for archiving music. I would also like to use it for playback (I use FLAC for this), but the provided Winamp plugin (still) doesn't work correctly in MediaMonkey; it can play tracks, but not all tags are displayed or are "deleted" when you try to edit them. But this is probably a MediaMonkey problem, although other Winamp input plugins work correctly. Maybe I will report it to MediaMonkey or switch to MusicBee sometime in the future if this can't be fixed otherwise.

Re: TAK 2.3.1 Beta 1

Reply #6
new results

Intel Xeon W-2145 (skylake)(Q3 2017), 3.70 GHz, 8 cores / 16 threads possible, Win10, SSD
Code: [Select]
binary  settings   ratio   enc    decode
------  ---------  ------  -----  ------
2.3.0 -p4m -tn4 52.56 % 236.04 614.41
2.3.1b -p4m -tn4 52.56 % 340.85 698.34
2.3.1b -p4m -tn8 52.56 % 575.82 697.55

AMD PhenomII X4 960T BlackEdition (anno 2011), 3.0 GHz, 6 threads, Win7 64bit, SSD
Code: [Select]
binary  settings   ratio   enc    decode
------  ---------  ------  -----  ------
2.3.0 -p4m -tn4 52.56 % 99.50 391.07
2.3.1b -p4m -tn4 52.56 % 142.69 456.39
2.3.1b -p4m -tn6 52.56 % 181.52 456.25

Thomas, great job !  :D

I wrote myself an autoit-script (.au3) to easily collect the results, the content normally shown at the cmd-prompt.
The idea is to have all stuff in 1 simple TEST-folder and save the results. There are some must-haves regarding the directory-structure and stuff needed. The directory-name itself should not contain any white spaces, so keep it simple then it's a no-brainer.

You HAVE TO make COPIES of your wav-files and DON'T use EXISTING directories. CREATE a NEW directory for TESTING (as the scripts deletes ALL .tak-files witihn your used music-directory).
See attachment or
Code: [Select]
#cs ----------------------------------------------------------------------------

 AutoIt: https://www.autoitscript.com
 Author: sPeziFisH

 Script
Function:
run 2 different TAK-Binaries (the CLI-version 'takc') and save the results of the command-windows as .txt (meant as log-file)
with each run a seprate directory with log-files is created
Name:
TAK_HA.au3

#ce ----------------------------------------------------------------------------

#include <AutoItConstants.au3>

;directory-name (next to this script) containing copies of wav-files  - your music-file-set to test TAK with
Local $dir_music = @ScriptDir & "\" & "wav-music"  ; <<< CHANGE if neccessary

;TAK-Version 1
Local $dir_tak = "TAK_2.3.0"
;TAK-Version 2 - the test/beta-version
Local $dir_tak_test = "TAK_2.3.1_Beta_1"

;Location of takc within TAK-dir
Local $dir_takc = "\Applications\takc"

Local $dir_tak_full = @ScriptDir & "\" & $dir_tak & $dir_takc
Local $dir_tak_test_full = @ScriptDir & "\" & $dir_tak_test & $dir_takc
;MsgBox(0, "info",@ScriptDir)

;preset (encoding)
Local $preset[13]
$preset[12] = "-p4m" ; <<< CHANGE if neccessary

;further parameter-set
Local $params = " -tn4 -cpuSSSE3 " ; <<< CHANGE if neccessary
;further parameter-set test/beta-version
Local $params_test = " -tn6 -cpuSSSE3 " ; <<< CHANGE if neccessary

;create directory nearby script containing the txt-files
;name-sheme year-month-day hour.minute.sec (eg. 2018-04-06 19.50.22)
Local $dir_name = @ScriptDir & "\" & @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & "." & @MIN & "." & @SEC
DirCreate($dir_name)

;complete cmd incl. pipe
$cmd_enc_tak = @ComSpec & " /c " & $dir_tak_full & " -e " & $preset[12] & $params & '"' & $dir_music & "\*" & '"'
$cmd_enc_tak = $cmd_enc_tak & " > " & '"' & $dir_name & "\" & $dir_tak & "_enc_" & $preset[12] & ".txt" & '"' ;pipe
$cmd_enc_tak_test = @ComSpec & " /c " & $dir_tak_test_full & " -e " & $preset[12] & $params_test & '"' & $dir_music & "\*" & '"'
$cmd_enc_tak_test = $cmd_enc_tak_test & " > " & '"' & $dir_name & "\" & $dir_tak_test & "_enc_" & $preset[12] & ".txt" & '"' ;pipe
;MsgBox(0, "info", $cmd_enc_tak_test)
$cmd_dec_tak = @ComSpec & " /c " & $dir_tak_full & " -t " & '"' & $dir_music & "\*" & '"'
$cmd_dec_tak = $cmd_dec_tak  & " > " & '"' & $dir_name & "\" & $dir_tak & "_dec.txt" & '"' ;pipe
$cmd_dec_tak_test = @ComSpec & " /c " & $dir_tak_test_full & " -t " & '"' & $dir_music & "\*" & '"'
$cmd_dec_tak_test = $cmd_dec_tak_test & " > " & '"' & $dir_name & "\" & $dir_tak_test & "_dec.txt" & '"' ;pipe



;MsgBox(0, "Info", "existing .tak-files get deleted")
FileDelete($dir_music & "\*.tak")
sleep(2000)

;MsgBox(0, "Info", "next: " & $dir_tak_test)
RunWait($cmd_enc_tak_test) ;encoding/compression
sleep(1000)
RunWait($cmd_dec_tak_test) ;decoding/decompression

sleep(1000)
;MsgBox(0, "Info", "existing .tak-files get deleted")
FileDelete($dir_music & "\*.tak")
sleep(2000)

;MsgBox(0, "Info", "next: " & $dir_tak)
RunWait($cmd_enc_tak) ;encoding/compression
sleep(1000)
RunWait($cmd_dec_tak) ;decoding/decompression

sleep(1000)
MsgBox(0, "Info", "finished !")

Re: TAK 2.3.1 Beta 1

Reply #7
Great!

Anyone took the time to compare TAK, WavPack and FLAC already with different settings to see which is best/fastest/slowest etc.?

Would love to know the results, but I just don't have the time to do it at the moment.



Re: TAK 2.3.1 Beta 1

Reply #10
even though nothing >4 threads:

Intel Core i3-350M (2010, mobile/notebook), 2.26 GHz, 2 cores / 4 threads possible, Win7, SSD
Code: [Select]
binary  settings   ratio   enc    decode
------  ---------  ------  -----  ------
2.3.0 -p4m -tn2 52.56 % 38 295
2.3.0 -p4m -tn4 52.56 % 51 295
2.3.1b -p4m -tn2 52.56 % 48 317
2.3.1b -p4m -tn4 52.56 % 65 317

AMD A6-5400K (Q3 2012), 3.6 GHz, 2 cores / 2 threads possible, Win10, SSD
Code: [Select]
binary  settings   ratio   enc    decode
------  ---------  ------  -----  ------
2.3.0 -p4m -tn2 52.56 % 49 355
2.3.1b -p4m -tn2 52.56 % 61 430

I also ran several runs with all systems (again) to be sure to have consistent results..nothing to change.

That comparison is quite old. Isn't there a more recent version?
guess what.
Keep in mind everyone still wants this or let's say sort of, things don't change that much. It means effort and a lot of people don't have time at moments.
Automation makes sense, maybe an automated contribution system (like LuxMark). All this would have to be done.. it's effort too and it's more effort.

Re: TAK 2.3.1 Beta 1

Reply #11
That comparison is quite old. Isn't there a more recent version?
As far as I know, no, there isn't newer comparison of lossless formats.

Re: TAK 2.3.1 Beta 1

Reply #12
Also that comparison table doesn't reflect even near the reality of multichannel audio compression. Some formats that may tuned better for compression ratios in Stereo, in multichannel they don't so well and vice versa.
I archive my multichannel DVD-Audio and Blu-Ray Audio discs in TAK since I found that it's essentially the state of art in compression ratios in multichannel audio.

BTW, I recompressed by curiosity my 5.1 Blu-Ray rip of Coldplay's "A Head Full Of Dreams" album and I got a total save of 12KB (Files are around ~2GB in total)! Also I didn't keep exact times but it was no doubt noticably faster.
Excited to see work resumed on the format! Keep up the good work!

Re: TAK 2.3.1 Beta 1

Reply #13
As far as I know, no, there isn't newer comparison of lossless formats.

After that, OptimFrog and WavPack have new major versions. I am not convinced that speed/ratio tests matter too much anymore, except that geeks like us enjoy to be impressed over improvements. Maybe after the next TAK, it could be worthwhile to rerun the comparison, but I guess that whatever codec update that misses next train, will have to wait for loooong.

Re: TAK 2.3.1 Beta 1

Reply #14
- Replaced MMX with SSE2 assembly.
Code: [Select]
CHKCPU32.exe /V | find "extensions"
 Instr set extensions: MMX, 3DNow!, SSE
Code: [Select]
Takc_2.3.0.exe -te -pMax test_sample.wav
test_sample.wav ..........  42.41%   15*

Compression:     42.41 %
Duration:       160.63 sec
Speed:           15.24 * real time

Takc_2.3.1b1.exe -te -pMax test_sample.wav
test_sample.wav ..........  42.41%    5*

Compression:     42.41 %
Duration:       453.33 sec
Speed:            5.40 * real time
:(

Re: TAK 2.3.1 Beta 1

Reply #15
CoRoNe:

The AMD chips got support for SSE2 quite later than Intel (Mostly since Athlon 64), but sure that PC of yours has more than 10 years (and possibly more than 12).

While it always feels bad that some software that you use stop working, in this case I would consider this the natural course of events and not try to stop this change. More so when this change does not stop it working, but only make it work slower.


( MMX and SSE code does not work well together. Also, SSE (non-2) cannot do integer arithmetic like MMX, so it's either MMX or SSE2 )

Re: TAK 2.3.1 Beta 1

Reply #16
Hi Tom,

I like TAK and would like to use it, but I need a codec without CUE for images, because I buy music mostly online and CUE cant handle them right. Could you maybe add something different to it like ALAC does?

Re: TAK 2.3.1 Beta 1

Reply #17
ALAC does not have anything for multi-track encoding, but its container, MP4, already has every supported format covered by chapter support.

Re: TAK 2.3.1 Beta 1

Reply #18
I dont care what it is, but nearly all lossless codecs have problems with this kind of stuff and I hope TAK will be the next FLAC but better.

Re: TAK 2.3.1 Beta 1

Reply #19
I like TAK and would like to use it, but I need a codec without CUE for images, because I buy music mostly online and CUE cant handle them right.
Care to clarify what kind of features music bought online has that CUE has problems with? I'm genuinely curious.

If you wish a container for TAK you should try to get it supported in Matroska.

Re: TAK 2.3.1 Beta 1

Reply #20
When sample rate is different from 44100Hz, or duration is not multiple of 1/75 sec CD frame (or 588 samples),  it's not something 
 CUE is meant for.
There are many formats with chapter support now but they may not be sample accurate (millisecond accuracy is not enough).

Re: TAK 2.3.1 Beta 1

Reply #21
I'm going to seize the fact this old thread has been risen from the dead to say:

Thank you Thomas @TBeck‍ ‍, for creating a CODEC with enables me to reach (unlike its other lossless competitors) bitrates lower   (~290Kbps) than the original 320Kbps-MP3 RCA Victor recordings I'd downloaded from The Web Archive and later done some sound processing in.

Listen to the music, not the media it's on.
União e reconstrução

Re: TAK 2.3.1 Beta 1

Reply #22
There are many formats with chapter support now but they may not be sample accurate (millisecond accuracy is not enough).
FLAC in oga chained into multitrack file can be solution.

Re: TAK 2.3.1 Beta 1

Reply #23
FLAC in oga chained into multitrack file can be solution.

Yes. but it's not as well supported as ordinary FLAC files.

As for chapters, Matroska chapter timestamp has nanoseconds precision internally, so it's more than enough.
MP4 QuickTime style chapters are described in media timescale (usually equal to samplerate) so it's sample accurate.
MP4 Nero style chapters are in 100 nanoseconds precision, so it's also enough.
So, both of them have enough accuracy as for container format specification.

However, what we see everywhere is a "simple chapter format" descrbed at  https://mkvtoolnix.download/doc/mkvmerge.html#mkvmerge.chapters and https://wiki.xiph.org/Chapter_Extension, which uses only 3 decimal places (implies millisecond precision).
So it's more about tooling and input format.

Re: TAK 2.3.1 Beta 1

Reply #24
Hi Thomas, it's been 4 years since you said you will support Unicode. I am glad to see an update but without Unicode support I cannot use TAK as my archiver format :(