HydrogenAudio

Hydrogenaudio Forum => Validated News => Topic started by: TBeck on 2021-03-28 18:36:10

Title: TAK 2.3.1
Post by: TBeck on 2021-03-28 18:36:10
Final release of TAK 2.3.1 ((T)om's lossless (A)udio (K)ompressor)

It consists of:

- TAK Applications 2.3.1
- TAK Winamp plugin 2.3.1
- TAK Decoding library 2.3.1
- TAK SDK 1.1.1

Download

Title: Re: TAK 2.3.1
Post by: TBeck on 2021-03-28 18:47:01
What's new

This release brings significant speed optimizations for the encoder and a lot of source code cleanups in preparation of a migration from Delphi to Lazarus and/or C. Practical goals are Linux binaries and open source releases. This will be done step by step depending on my spare time. The cleanup revealed several bugs which affected the compression efficiency, but never the data integrity.

Improvements:


New features:


Fixes:


Cleanup:


Known issues:


More information

You can find some useful information and comparisons in the beta thread (https://hydrogenaud.io/index.php?topic=115769.0).

Have fun...

Thomas
Title: Re: TAK 2.3.1
Post by: TBeck on 2021-03-28 18:51:08
Results

Here the results for my primary file set.

Test system: Intel i3-8100 (3.6 GHz / 1 Thread), Windows 10.
Code: [Select]
Preset  Compression %           Enco-Speed                Deco-Speed
---------------------------------------------------------------------------------
        2.3.0   2.3.1    Win    2.3.0    2.3.1    Win %   2.3.0    2.3.1    Win %
---------------------------------------------------------------------------------
-p0     58.75   58.74    0.01   809.08   878.29    8.55   807.68   800.12   -0.94
-p0e    58.33   58.33    0.00   622.98   697.03   11.89   806.35   806.26   -0.01
-p0m    58.22   58.22    0.00   346.10   413.76   19.55   809.03   808.92   -0.01
-p1     57.84   57.84    0.00   651.21   731.41   12.32   780.98   782.47    0.19
-p1e    57.52   57.52    0.00   407.25   467.87   14.89   780.58   784.63    0.52
-p1m    57.42   57.42    0.00   253.37   307.77   21.47   781.59   787.19    0.72
-p2     56.90   56.90    0.00   465.33   581.92   25.06   671.37   710.83    5.88
-p2e    56.70   56.70    0.00   259.44   341.04   31.45   666.22   709.60    6.51
-p2m    56.59   56.59    0.00   143.59   198.09   37.96   666.78   710.65    6.58
-p3     56.36   56.36    0.00   231.98   300.15   29.39   645.18   697.12    8.05
-p3e    56.27   56.27    0.00   184.86   239.29   29.44   643.99   696.21    8.11
-p3m    56.20   56.20    0.00    92.80   131.10   41.27   643.44   697.58    8.41
-p4     56.03   56.03    0.00   144.21   182.79   26.75   595.68   652.26    9.50
-p4e    55.94   55.94    0.00   123.28   157.95   28.12   594.05   653.27    9.97
-p4m    55.89   55.89    0.00    57.47    82.42   43.41   593.92   652.78    9.91
---------------------------------------------------------------------------------
Compression in percent relative to the original file size. Speed as multiple of realtime playback.

Future

Title: Re: TAK 2.3.1
Post by: lvqcl on 2021-03-28 19:25:21
Long time, no see!
Title: Re: TAK 2.3.1
Post by: TBeck on 2021-03-28 20:15:13
"Sometimes they come back."

From 'My life in Stephen King quotes'.
Title: Re: TAK 2.3.1
Post by: sveakul on 2021-03-29 00:59:43
Terrific, thank you sir.
Title: Re: TAK 2.3.1
Post by: Destroid on 2021-03-30 09:17:47
A release ahead of schedule ;)

I am excited about the future developments too.
Title: Re: TAK 2.3.1
Post by: eddie.zato on 2021-03-30 09:42:26
Tried to test albums of different genres (i5-3570K 3.4 GHz, Win 10, -p4m -tn4):  :)

Code: [Select]
PS > Get-ChildItem *.wav -File -Name | Foreach-Object { .\taktest 10 $_ }
Version 230, 10 passes ..........
Version 231, 10 passes ..........

carl_orff_carmina_burana.wav, 00:56:15, 2 ch, 24 bit, 88200 Hz

Version Compress Filesize_bytes Enc_dur_sec Encode_speed Decode_speed
------- -------- -------------- ----------- ------------ ------------
WAV          100     1786113572
230        62,06     1108390330       28,12       120,05       218,88
231        62,05     1108333112       21,24       158,88       238,42


Version 230, 10 passes ..........
Version 231, 10 passes ..........

fleshgod_apocalypse_kng.wav, 00:57:30, 2 ch, 24 bit, 48000 Hz

Version Compress Filesize_bytes Enc_dur_sec Encode_speed Decode_speed
------- -------- -------------- ----------- ------------ ------------
WAV          100      993600068
230        76,06      755692758       17,74       194,52       470,69
231        76,03      755481355       14,77       233,65       491,04


Version 230, 10 passes ..........
Version 231, 10 passes ..........

the_prodigy_the_fat_of_the_land.wav, 01:06:07, 2 ch, 16 bit, 44100 Hz

Version Compress Filesize_bytes Enc_dur_sec Encode_speed Decode_speed
------- -------- -------------- ----------- ------------ ------------
WAV          100      699800012
230        69,39      485577995       21,54       184,18       568,81
231        69,39      485569546       17,56       225,88       616,79
Title: Re: TAK 2.3.1
Post by: NetRanger on 2021-03-30 14:36:14
Nice. Thnx :)
Title: Re: TAK 2.3.1
Post by: kode54 on 2021-04-01 21:21:47
Sorry, didn't realize this topic was in News Submissions, I'll bump it into the News feed.
Title: Re: TAK 2.3.1
Post by: Destroid on 2021-04-01 22:03:02
I ran a CD image WAV test on Sandy Bridge 2500k WinXP 32-bits and the results show incredible speed improvements (although using HDD is definitely hampering the decoding speeds):
Code: [Select]
mode       2.3.0                2.3.1
-tn4  %ratio  enc   dec    %ratio  enc   dec
----  ------  ----  ----   ------  ----  ----
-p0   70.75%  319*  281*   70.75%  352*  323*
-p0e  70.45%  669*  255*   70.45%  792*  301*
-p0m  70.34%  627*  253*   70.34%  837*  296*
-p1   68.89%  645*  265*   68.89%  820*  298*
-p1e  68.70%  677*  221*   68.70%  895*  287*
-p1m  68.59%  645*  225*   68.45%  812*  300*
-p2   68.45%  709*  141*   68.59%  685*  296*
-p2e  68.27%  711*  238*   68.27%  835*  304*
-p2m  68.16%  401*  270*   68.15%  498*  288*
-p3   68.04%  644*  270*   68.04%  724*  309*
-p3e  68.01%  551*  286*   68.00%  611*  311*
-p3m  67.95%  298*  302*   67.95%  350*  292*
-p4   67.93%  441*  308*   67.93%  504*  295*
-p4e  67.90%  371*  301*   67.90%  429*  292*
-p4m  67.87%  191*  198*   67.87%  228*  321*
Great work, Thomas! (I can PM you the CPU measurements if you want.)

P.S. My earlier comment was regarding the March 28th release was a few days early :)
Title: Re: TAK 2.3.1
Post by: TBeck on 2021-04-01 23:00:25
A release ahead of schedule ;)
Hm... Beta released on April 2, Final on March 28. Huh? Time leap backwards?
No, just the longest beta testing phase ever...

Tried to test albums of different genres (i5-3570K 3.4 GHz, Win 10, -p4m -tn4):  :)
...
Thank you. Nice to see some albeit tiny compression imrovements caused by the bugfix.

I ran a CD image WAV test on Sandy Bridge 2500k WinXP 32-bits and the results show incredible speed improvements (although using HDD is definitely hampering the decoding speeds):
Thank you too. I am very glad to see another familar er... face. My english never was good but the long abstinence from hydrogen definitely has worsened it. I am struggling for words...

I am a bit ashamed about the long delay but i am happy to say that im am just working on the next version.
Title: Re: TAK 2.3.1
Post by: VegasI15 on 2021-04-09 19:39:51
Very welcoming news TBeck...TY!  Was also nice to find the new streamlined foobar2000 plugin for TAK.  Something good came out of having to reinstall windows from scratch.  Cheers!
Title: Re: TAK 2.3.1
Post by: skamp on 2021-04-10 17:42:29
Long time, no see!
"Sometimes they come back."

'sup?

I just got an M1 Mac Mini. I wonder how fast TAK would encode files under ideal conditions (8 threads, native ARM build)…

Title: Re: TAK 2.3.1
Post by: kode54 on 2021-04-10 23:24:46
It would need to have its assembly portions rewritten for aarch64 and NEON intrinsics or opcodes. For now, Rosetta 2 can do a decent enough job converting up to SSE4.2 or SSSE3 to that.

I'm benchmarking FLAC decoding a -8 encoded album, 8 threads, 5 passes:

Total length: 1d 12:24:43.200
Opening time: 0:36.564
Decoding time: 9:51.934
1668.526x realtime

And the encode:

Encode to -p2 took 13 seconds.

And a decoder benchmark of the resulting TAKs, using latest foo_input_tak:

Total length: 1d 12:24:43.200
Opening time: 0:20.804
Decoding time: 9:45.604
1729.307x realtime
Title: Re: TAK 2.3.1
Post by: mycroft on 2021-04-11 10:06:17
Too late.
Title: Re: TAK 2.3.1
Post by: kode54 on 2021-04-11 20:05:48
Too late for what?
Title: Re: TAK 2.3.1
Post by: Porcus on 2021-04-11 22:42:41
Well ... for world domination I guess.

More than fifteen years ago, @bryant wrote some thoughts about why WavPack never really caught on (https://hydrogenaud.io/index.php?topic=36408.msg321408#msg321408). Arguably, 2005 was already too late to get big - unless being pushed by some major player who for all the stupid reasons insisted on having their own thing that is worse than what was already around. WMAL (2003) arguably was more ephemera than ALAC (2004).

But if the purpose of 2.3.1 was "just" to maintain something that impresses enthusiasts ... well who cares that it didn't arrive in 2020. Right now I see more reason to relegate WMAL to the "Other" section at https://wiki.hydrogenaud.io/index.php?title=Lossless_comparison , since it is abandonware without the raison d'être that TAK/WavPack can claim.
Title: Re: TAK 2.3.1
Post by: Destroid on 2021-04-12 06:59:52
A release ahead of schedule ;)

Dudes, it was an April 1st reference (in 2006).
Title: Re: TAK 2.3.1
Post by: Porcus on 2021-04-12 10:33:52
Tested. Felt a bit bored during the week-end, played around with -p2 and -p0, and then put some FOR loops at nighttime work for -p4m. I know that gives "unfair" speeds when compared to FLAC that I usually use.

I deliberately chose something high-bitrate:
* All my eight 96/24 albums. Seven hours. NIN: The Slip; The Tea Party: Tx 20; Cult of Luna: The Raging River and A Dawn to Fear; Kayo Dot: Hubardo; Clouds Taste Satanic: The Satanic Singles Series; Cascades s/t; and finally the https://opengoldbergvariations.org
* One 96/16 single (https://sunn.bandcamp.com/album/--2) (6 minutes) snuck in and was treated as if it were a 96/24. My only 88.2/24 "album" (https://analtrump.bandcamp.com/album/that-makes-me-smart) to see how it fares with three minutes of grindcore.
* And for the hell of it: a Merzbow CD which has been the worst I could expose TAK to. (https://hydrogenaud.io/index.php?topic=101386.msg837834#msg837834) Yes CDDA this one, 50 minutes.

In addition to putting a hi-rez I am using a fanless computer with a fairly slow CPU, that explains the speeds.

-p0 test encodes, the 96kHz part
Compression:     60.75 %     both 2.3.0 and 2.3.1
Duration:       172.20 sec     for 2.3.0, improves to      163.75      for 2.3.1
Speed:          148.91 * real time     for 2.3.0, improves to      156.60

-p4m encode to spinning SATA drive with verify, the 96kHz part
Compression:     58.38 %     for 2.3.0, improves to      58.37      for 2.3.1, that is 1/100 of a percentage point
Duration:      1025.68 sec    for 2.3.0, improves to      846.12 sec      for 2.3.1
Speed:           25.00 * real time, improves to 30.31.
For comparison, a 2.3.0 test encode took 927.76 sec, 27.64 * real time.

Decoding -p4m using -t (all files combined). Both the HDD and the SSD are SATA-ed to the motherboard. Speeds sorted from slowest to fastest:
211.55: encoded with .1 decoded with .0 on HDD
212.40: encoded with .0 decoded with .0 on HDD
219.58: encoded with .1 decoded with .0 on SSD
220.14: encoded with .0 decoded with .0 on SSD
222.42: encoded with .0 decoded with .1 on HDD
224.67: encoded with .1 decoded with .1 on HDD
225.761 encoded with .1 decoded with foo_input_tak using fb2k's "Decoding speed test" utility (not Takc.exe, but put here in the order)
230.19: encoded with .0 decoded with .1 on SSD
231.17: encoded with .1 decoded with .1 on SSD

It is unfair to compare the high-compression level TAK to FLAC, but for reference the corresponding FLAC -8 files decode at 395 using the fb2k speed test.
Title: Re: TAK 2.3.1
Post by: Porcus on 2021-04-12 10:38:44
Oh, and the 88.2/24 Anal Trump actually compressed worse with 2.3.1.
Available for free here: https://analtrump.bandcamp.com/album/that-makes-me-smart
Title: Re: TAK 2.3.1
Post by: VEG on 2021-04-12 16:45:21
Well ... for world domination I guess.
Support in web browsers could make a huge difference.

Did you hear about FLIF image codec? Its author is working on the JPEG XL now, and all the best ideas from FLIF were used in the new codec, with some cool additions which make the new image codec truly universal. The format itself is basically a mixture of FUIF (from the author of FLIF), PIK and Brunsli image codecs. Google is already testing it (https://bugs.chromium.org/p/chromium/issues/detail?id=1178058) in Chromium. When it is supported by major browsers, the format will become much more popular.

A similar thing could happen with a new cool lossless audio codec.
Title: Re: TAK 2.3.1
Post by: Porcus on 2021-04-12 18:13:00
Additional test, -p0-encoding the 96k .wavs with verification to real .tak files on a real HDD (but SATA to mobo) - compared to FLAC. Nothing surprising here I think, just a confirmation that bloopers like this didn't happen to TAK (https://hydrogenaud.io/index.php?topic=120750.0).

Takc (version 2.3.1) -e -p0 -v 
329 seconds encoding time (read from the output). Compare to 604 seconds for flac (version 1.3.3) --best --verify (read from echo:|time)
2786 bitrate, compare to 2677 for -p4m. 2786 is slightly better than the 2797 for FLAC [note at the end].
fb2k decoding speed test: 244.8. Not much worse than the 225.7 for -p4m. Quite a bit slower than FLAC's 372.7.


I would not have expected anything strange here except if there were some bad assembler optimization. It is qualitatively as what we know from http://www.audiograaf.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%204.pdf  :
Best FLAC compression is on par with TAK weakest compression, and FLAC wins big at decoding time.
Fastest FLAC encoding time (--verify -0, but who wants to use that?) is actually on par with fastes TAK encoding time, and then TAK wins big at compression (compared to 2955) and FLAC wins big at decoding speed.


As for the compression ratio: For the eight albums,
* there are two for which TAK -p0 compresses better than FLAC --best on every track in the album
* there are two for which FLAC --best compresses better than TAK -p0 on every track in the album
* the remaining four has at least one track of each.
Title: Re: TAK 2.3.1
Post by: TBeck on 2021-04-13 19:33:47
First: Thanks to everyone for your interest and testing! It's very encouraging, and i am glad to tell you, that i am making good progress in the implementation of unicode support for the next release of the command line version Takc. Basically it's working, but i have to perform a lot of testing because i had to modify a lot of code. The core of the codec is unaffected.

A release ahead of schedule ;)

Dudes, it was an April 1st reference (in 2006).
Ahhh... You must be a veteran!

I deliberately chose something high-bitrate:
That's definitely interesting. Most of my test files are CD-Audio and i haven't really checked the processing speed of hires audio lately. I will catch up.

In addition to putting a hi-rez I am using a fanless computer with a fairly slow CPU, that explains the speeds.
The new assembly optimizations are using unaligned SSE2 read instructions which is faster on relatively new cpu architectures (Skylake and later) but often considerably slower (than the old code)  on older cpus. This penalty will eat up a lot of the other speed optimazions and the advantage of the new version will be quite small.

Oh, and the 88.2/24 Anal Trump actually compressed worse with 2.3.1.
Available for free here: https://analtrump.bandcamp.com/album/that-makes-me-smart
Strange! How big is the difference? I would have checked it myself but i seem to be too dumb to downlad it for free...

--------------------
Sorry for bad English...
I can do badder!
Title: Re: TAK 2.3.1
Post by: sPeziFisH on 2021-04-13 19:48:32
Hi Thomas, nice to see progress
Code: [Select]
Enc with -p4m
2.3.0 Enc  131.72s / 98.09x  (52.56%)
2.3.1 Enc  74.01s / 174.39x  (52.56%)

2.3.0 Dec  35.11s / 368.02x
2.3.1 Dec  29.96s / 431.28x
Osna-rocket launched 8)
Title: Re: TAK 2.3.1
Post by: Porcus on 2021-04-13 20:48:17
Oh, and the 88.2/24 Anal Trump actually compressed worse with 2.3.1.
Available for free here: https://analtrump.bandcamp.com/album/that-makes-me-smart
Strange! How big is the difference? I would have checked it myself but i seem to be too dumb to downlad it for free...

Click "Buy Digital Album  name your price" and enter "0" as price ;-)

The worst is -p4m:
51 064 065 bytes with 2.3.0
51 083 360 bytes with 2.3.1

It still isn't that bad, 1 kbit/s on average?! Largest bitrate difference: 2181 vs 2189 for track 10.  But there is also a track with an improvement of 3.


And now I see that there are some strange (but small!) results:
-p4e, this improves:
50 983 669 bytes with 2.3.0
50 979 810 bytes with 2.3.1.

-p3e
51 150 393 bytes with 2.3.0
51 150 669 bytes with 2.3.1

-p3:
50 965 890 bytes with 2.3.0
50 967 229 bytes with 2.3.1

and at the low end:
-p0m
64 403 789 bytes with 2.3.0
64 326 035 bytes with 2.3.1

-p0
64 723 721 bytes with 2.3.0
64 723 693 bytes with 2.3.1
Title: Re: TAK 2.3.1
Post by: TBeck on 2021-04-13 21:52:25
Code: [Select]
2.3.0 Enc  131.72s / 98.09x  (52.56%)
2.3.1 Enc  74.01s / 174.39x  (52.56%)
The encoding speed improvement nearly is to good to be true! Are you now using more cores (maybe 6 vs. 4)?

Click "Buy Digital Album  name your price" and enter "0" as price ;-)
Well, i thougt so, but somehow had scruples... At first. ;)

These are some really interesting files. I agree, that the difference is not very significant, but i really want to understand, why this happens. It might be caused by the loss of arithmetic accuracy because ot the switch from FPU to SSE2-floating point caclculations (80 vs. 64 bits), but i am not sure. Maybe it's a random effect, maybe it's something systematic, which bears an opportunity for a tiny optimization.  I will dig into it, but this may take some time. I really want to release 2.3.2 soon. Hopefully in a couple of weeks.
Title: Re: TAK 2.3.1
Post by: sPeziFisH on 2021-04-13 22:20:46
The encoding speed improvement nearly is to good to be true! Are you now using more cores (maybe 6 vs. 4)?
oh well, jupp, I squeezed all out of it - as possible. sorry for missing noting this detail.
2.3.0 -p4m -tn4 -cpuSSSE3
2.3.1 -p4m -tn6 -cpuSSSE3

while 2.3.1-tn6 used 6cores with ~98% total CPU-Load, 2.3.0-tn4 used 6cores with ~82%-85% total CPU-Load.
Might be some sort of internal balancing of this CPU, even though you told 2.3.0 could only address 4 cores.

Code: [Select]
Enc always with -p4m -t4n
2.3.0 Enc  132.16s / 97.77x  (52.56%)
2.3.1 Enc  92.25s / 140.06x  (52.56%)

2.3.0 Dec  33.87s / 381.50x
2.3.1 Dec  28.86s / 447.70x
Title: Re: TAK 2.3.1
Post by: d4k0 on 2021-04-18 14:49:17
Great to hear that TAK 2.3.1 is finally released  :). The new 8 threads setting really speeds up the encoding process.

But I have a question: What is the easiest/best way to update existing TAK files to newer versions of the codec? FLAC allows FLAC files as input, but TAK requires WAV as input. There is support for pipe encoding, but it seems that this is only useful for applications that support it (like Foobar200, but some don't recommend it for reencoding). Is there a way to reencode the files using a batch file or is something like Foobar2000 or Xrecode III the better way?
Title: Re: TAK 2.3.1
Post by: Rollin on 2021-04-18 14:56:02
Foobar200, but some don't recommend it for reencoding
Who are those "some" and why they don't recommend foobar2000 for reencoding?
Many people here can confirm that using foobar2000 for re-encoding is safe. Just be sure not to enable DSP, Additional decoding and ReplayGain in Converter settings. And you can use Binary Comparator ( https://www.foobar2000.org/components/view/foo_bitcompare ) after conversion to compare resulted files with sources.
Title: Re: TAK 2.3.1
Post by: d4k0 on 2021-04-18 19:47:19
Who are those "some" and why they don't recommend foobar2000 for reencoding?
Many people here can confirm that using foobar2000 for re-encoding is safe.

I searched and found one post I remembered again on Hydrogenaudio:

https://hydrogenaud.io/index.php?topic=106974.msg879004#msg879004

To be more precise: There is no technical problem with Foobar2000 and reencoding, but Foobar2000 can't replace/update existing files, it always creates new one which creates extra work afterwards.

The "easiest" solution seems to add some characters to the new file name. When the encoding is done, you have to delete all files without the the extra characters and remove these characters from the new files:

I have a conversion preset set up in Foobar, it converts to Leve 8 and adds "~~" to the beginning of every file name. Once conversion is done, delete every file that doesn't have the prefix and use something like MP3tag to remove it from the new files.
I also like to bit-compare the new files before deleting the old ones, i might just be paranoid though.
https://hydrogenaud.io/index.php?topic=106974.msg878938#msg878938

You can still use the source folder option, but rename the output file to include something that differentiates it from the original. Of course, you'll need to have space available for the converted tracks plus the original until you delete the originals.

What I would do, use source folder, and and then use %filename%-16 for the output filenames, this will allow you to convert, then you can easily filter out all the tracks that don't have "-16" in the filename and remove them.

Depending on how much space you have available, you can convert an album or group of tracks at a time.
https://hydrogenaud.io/index.php?topic=98750.msg854514#msg854514

There also seems to be foo_run, but there doesn't seem to be much documentation:

You can't replace the source files with converter. The term "converter" may be a bit misleading as it doesn't convert the originals. It allows encoding a new copy.
If you absolutely must replace files in place it can be performed with foo_run (http://www.foobar2000.org/components/view/foo_run).
https://hydrogenaud.io/index.php?topic=117340.msg968867#msg968867

I think I will probably use the extra character solution.



Just be sure not to enable DSP, Additional decoding and ReplayGain in Converter settings. And you can use Binary Comparator ( https://www.foobar2000.org/components/view/foo_bitcompare ) after conversion to compare resulted files with sources.

Thanks, I'm already using Binary Comparator, it's a really useful plugin  ;).
Title: Re: TAK 2.3.1
Post by: Rollin on 2021-04-19 22:05:08
https://hydrogenaud.io/index.php?topic=106974.msg879004#msg879004
foobar never really worked for that purpose without doing some afterwork
All "afterwork" can be done right from fb2k window in few clicks. Also, nowadays, ReplayGain tags can be automatically transferred during conversion.
Title: Re: TAK 2.3.1
Post by: phw on 2021-06-30 14:13:43
Really glad to see TAK is still alive and kicking! Thanks a lot Thomas!

But I have a question: What is the easiest/best way to update existing TAK files to newer versions of the codec?

As I understand it there is no change to the codec itself, so no need to reencode at all. You can encode the files faster now than with previous releases, but the result should be the same.
SimplePortal 1.0.0 RC1 © 2008-2021