HydrogenAudio

Lossless Audio Compression => Lossless / Other Codecs => Topic started by: TBeck on 2022-05-30 11:45:30

Title: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 11:45:30
Beta release 3 of TAK 2.3.3 ((T)om's lossless (A)udio (K)ompressor)

TAK 2.3.3 brings an 64-bit decoding library and better unicode support for the GUI version.

New in Beta 3

The bug that has been fixed in beta 2 hid another bug, which unfortunately only occured on some systems (not on my test system). It would crash the application. It is fixed in the new 64-bit library.

New in Beta 2

The interface of the 64-bit decoder library contained a bug. This release brings a new library and a modified interface for the pascal version which is technically backwards compatible. The C interface remains unchanged. The SDK documentation has been updated.

It consists of:


Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 11:46:48
What's new

New features:


Improvements:


Beta testing

For most users this release is not directly relevant. The most important feature is the 64-bit decoder library. Quite useless for the average user unless a developer incorporates it into his software. Relevant if a software dropped 32-bit support. If you know of such a case, please inform the author about the new library and / or report it here. I already contacted the author of the great Mp3Tag.

The new library passed my Delphi (Pascal) test set. I also ported some of my tests to C (Code::Blocks environment) to test the header file. Since i am not really fluid in writing C code, i am always a bit uncertain if i did everything right.

Thanks for testing and have fun

Thomas
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 11:47:36
Results

Here the results for my primary file set.

Test system: Intel i3-8100 (3.6 GHz / 1 Thread), Windows 10.

Code: [Select]
Preset  Enco-Speed                Deco-Speed
---------------------------------------------------------
        2.3.2    2.3.3    Win %   2.3.2    2.3.3    Win %
---------------------------------------------------------
-p0     868.72   893.94    2.90   806.84   810.57    0.46
-p0e    698.08   711.16    1.87   812.44   817.09    0.57
-p0m    418.70   425.54    1.63   815.07   819.19    0.51
-p1     726.44   748.04    2.97   787.03   788.37    0.17
-p1e    482.99   491.72    1.81   788.73   791.12    0.30
-p1m    314.46   322.68    2.61   790.61   794.60    0.50
-p2     580.58   591.20    1.83   715.05   718.12    0.43
-p2e    347.07   348.35    0.37   714.52   719.10    0.64
-p2m    203.02   206.26    1.60   715.99   718.71    0.38
-p3     301.25   306.83    1.85   697.20   703.97    0.97
-p3e    241.91   244.79    1.19   697.88   702.06    0.60
-p3m    131.89   133.65    1.33   699.12   701.40    0.33
-p4     183.23   186.54    1.81   650.44   657.64    1.11
-p4e    158.75   160.61    1.17   651.01   658.40    1.14
-p4m     82.49    83.73    1.50   651.21   656.84    0.86
---------------------------------------------------------
Speed as multiple of realtime playback.

And to illustrate the speed disadvantage of the 64-bit version:

Code: [Select]
Preset  Enco-Speed                Deco-Speed
---------------------------------------------------------
        32 bit   64 bit   Win %   32 bit   64 bit   Win %
---------------------------------------------------------
-p0     893.94   827.49   -7.43   810.57   700.80  -13.54
-p0e    711.16   661.05   -7.05   817.09   709.15  -13.21
-p0m    425.54   398.79   -6.29   819.19   710.48  -13.27
-p1     748.04   698.29   -6.65   788.37   689.93  -12.49
-p1e    491.72   461.79   -6.09   791.12   693.09  -12.39
-p1m    322.68   303.56   -5.93   794.60   695.48  -12.47
-p2     591.20   557.27   -5.74   718.12   634.55  -11.64
-p2e    348.35   336.29   -3.46   719.10   633.90  -11.85
-p2m    206.26   193.64   -6.12   718.71   636.49  -11.44
-p3     306.83   287.58   -6.27   703.97   619.50  -12.00
-p3e    244.79   233.77   -4.50   702.06   620.18  -11.66
-p3m    133.65   123.27   -7.77   701.40   621.06  -11.45
-p4     186.54   172.30   -7.63   657.64   577.94  -12.12
-p4e    160.61   145.73   -9.26   658.40   579.34  -12.01
-p4m     83.73    76.29   -8.89   656.84   578.45  -11.93
---------------------------------------------------------
Speed as multiple of realtime playback.
Title: Re: TAK 2.3.3 Beta thread
Post by: Case on 2022-05-30 15:24:51
Thanks for the new 64-bit DLL. I used it without trouble in a foobar2000 component.

For some reason I see much slower decoding with it than I expected. Same component with the 32-bit DLL reaches 1111x realtime speed, the 64-bit component is limited to 400x realtime speed. 64-bit foobar2000 with ffmpeg decodes at 650x realtime speed.
Tested on Intel Core i9-12900K.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 15:35:03
Great news that it is working! And a mistery...

I have no idea how the 64-bit version could be so slow. I never test the speed of the deco_lib, because it is using the same code as the application with a very thin layer on top. I will write a test program to check if i can replicate your results.

Can you tell me, which encoder preset is affected?
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 15:38:59
Can you tell me, which encoder preset is affected?
The results of the extremes -p0 and -p4m would be interesting.
Title: Re: TAK 2.3.3 Beta thread
Post by: Case on 2022-05-30 16:19:22
The test file used p4m compression. I created new test files with the latest beta-encoder with p0 and p4m presets and got these speed results.

32-bit dll:
p0:  1379x
p4m: 1134x

64-bit dll:
p0:  961x
p4m: 408x

64-bit ffmpeg:
p0:  871x
p4m: 653x

Each decode test was ran three times and I recorded the fastest result. Interestingly the new DLL beats ffmpeg with the p0 file.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 16:50:59
Thank you. The 32-bit version on your system is 1.43 respectively 2.78 times faster.

My newly created speed test gives 1.14 and 1.10, as would be to expected.

Please tell me which options you use when creating the decoder:
    TtakCpuOptions Cpu;
    TtakInt32      Flags;

I suspect your 64 bit decoder is using no assembler optimizations. If i compare 32-bit with cpuAny with 64-bit with cpuNone i get ratios of 1.40 and 3.62, quite similar to your results.
Title: Re: TAK 2.3.3 Beta thread
Post by: Case on 2022-05-30 18:09:22
Cpu option is set to tak_Cpu_Any (1023) and Flags to 0. I set Flags to "tak_ssd_opt_SequentialRead | tak_ssd_opt_CheckMd5" for integrity verification.

I ran speed tests on the 64-bit DLL with Cpu set to 0:
p0:  768x
p4m: 366x
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 18:19:29
Will check this soon.

I already wrote that i am not very familiar with (modern) C / C++. I am unsure, if the aligment of fields in a struct could cause problems (especially in 64-bit mode).  From the SDK:
Quote
Structures (record, struct) have to be packed; insertion of filling bytes for data alignment is not allowed. All current structures exclusively contain the data types Int32 and Int64. It should be possible to cope with activated alignment by replacing Int64 with two Int32 for low and high. Then all elements have a size of 4.
But by some googling i stumbled upon a post at stackoverflow:
Quote
First of all, structure alignment is not an exact science and may depend on architecture and compiler. In many case, all structure members are padded according the biggest variable (in byte).

This definitely would mean trouble. Is there a good way to modify the C header file to avoid such problems? Possibly something like the "packed"-attribute in Delphi Pascal?
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 18:34:25
Cpu option is set to tak_Cpu_Any (1023) and Flags to 0. I set Flags to "tak_ssd_opt_SequentialRead | tak_ssd_opt_CheckMd5" for integrity verification.
Can't replicate it. Here anything is w├Ârking as expected. Could you imagine a problem with the header / interface declaration?
Title: Re: TAK 2.3.3 Beta thread
Post by: Case on 2022-05-30 18:40:52
I actually had added #pragma pack(1) (https://docs.microsoft.com/en-us/cpp/preprocessor/pack?view=msvc-170) to the beginning of my tak_deco_lib.h long ago as the documentation stated things need to be byte aligned, IIRC. Removing the pragma command or changing it to for example 4 byte alignment results in the decoder to stop working. Library returns error.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 18:51:25
I suppose  "#pragma pack(1) " is compiler specific? Maybe i should put a note at the top of the header file about alignment for those developers not reading the doucumentation as thoroughly as you.

Regarding the speed problem: Currently i have no idea what is going on. I will go through the code to find a new approach.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 18:54:30
BTW: Which function do you use to create the decoder?
Title: Re: TAK 2.3.3 Beta thread
Post by: Case on 2022-05-30 19:03:02
I use tak_SSD_Create_FromStream. I sent you PM with a link to full sources.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 19:33:50
Got it, thank you. I am just struggling with the (for me) new message system. Sent you two replies, but i am unsure, if you received them.
Title: Re: TAK 2.3.3 Beta thread
Post by: Case on 2022-05-30 19:50:30
I saw the first one as I happened to still be online at the time. I hadn't received mail notification about the second one yet but I would have eventually. Replied.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-30 21:17:02
Here are the values of the parameter AOptions which tak_SSD_Create_FromStream () receives from foo_input_tak :

Play
  Cpu:   0x03A01760  60823392 (dec)
  Flags:  0x00000000  0

Verify integrity
  Cpu:   0x04BA1600  79304192
  Flags:  0x00000000  0

Strange. Cpu is garbage.

Somehow i have the suspicion that we have a nasty case of a program part overwriting data in another place.

In your class declaration input_tak StreamInfo precedes the Options variable.

    Ttak_str_StreamInfo_V22 StreamInfo;
    TtakSSDOptions Options;

It could be worthwhile to insert some dummy fields (maybe 16 bytes or so) between the two to check, if some write acceess to StreamInfo is overwriting Options.  And i have to check tak_SSD_GetStreamInfo_V22().
Title: Re: TAK 2.3.3 Beta thread
Post by: rutra80 on 2022-05-31 13:49:08
64-bit foobar2000
Wait... where??  :o
Title: Re: TAK 2.3.3 Beta thread
Post by: Porcus on 2022-05-31 14:05:15
Wait... where??  :o
In the test lab.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-05-31 21:31:17
I saw the first one as I happened to still be online at the time. I hadn't received mail notification about the second one yet but I would have eventually. Replied.
I'v got it!

Three functions are affected:

  tak_SSD_Create_FromFile ()
  tak_SSD_Create_FromFileW ()
  tak_SSD_Create_FromStream ()

The troublesome parameter (a struct) is declared in pascal as:

   const AOptions : TtakSSDOptions;

And in C:

    const TtakSSDOptions * AOptions,

Obviously a pointer to the struct should be passed.

For decades i  made a false assumption about the effect of const when passing a struct in pascal. I assumed it would always be passed by reference (as pointer). Wrong! If it fits into a register, it will be passed by value. Since TtakSSDOptions is 8 bytes large, it fits perfectly in 64 bit mode.

My only excuse: "It ever worked as i expected!"I never had to question my assumption.

Sorry for the trouble! I will release a new version tomorrow.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-06-01 16:54:19
Beta 2 has been added to the first post.

It hopefully fixes the reported bugs in the 64-bit decoder library.

Maybe a Mod could delete the Beta 1 attachement? I seem to have no sufficient rights to do it.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-06-01 20:07:15
Seems i was too hasty... Just received the report it's still crashing. Surprising. It ran so well in my virtual machine. Then i tried it directly on my primary system and it crashed! Good it did, makes it a lot easier to find the bug by trying things out.

What could be the difference between the systems? First hypothesis: The library gets loaded at a different adress. What if my assembler code is not perfectly relocatable? Switched it off and everything is running fine. Now i have to dig a bit deeper into 64-bit assembly.

Funny: The bug in beta 1 was likely to disable the assembler code and therfore hid the new bug.

It's been a long time since i experienced such a bumpy release.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-06-01 22:33:57
Good news: I fixed a bug in my assembler code that indeed depended on the memory location of the library. Now the library is working well on both of my systems and 'Case' too has reported success. Big thanks for his support!

I see a pattern: A new day, a new Beta...
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-06-02 09:57:32
Beta 3 has been added to the first post.
Title: Re: TAK 2.3.3 Beta thread
Post by: kode54 on 2022-06-02 11:09:22
Would you like me to delete the first two?
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-06-02 11:34:51
Yes. Thank you.
Title: Re: TAK 2.3.3 Beta thread
Post by: nu774 on 2022-06-02 15:33:52
beta3 tak_deco_lib.dll works fine with unmodified qaac64/refalac64.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-06-02 20:48:46
Cool! Thank you.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-06-30 16:00:46
TAK 2.3.3 (https://hydrogenaud.io/index.php/topic,122639.html) has been released.
Title: Re: TAK 2.3.3 Beta thread
Post by: Studio 308 on 2022-08-03 00:59:23
So great that TAK is alive! Encoding whole collection for years and everything new incoming.
Title: Re: TAK 2.3.3 Beta thread
Post by: MonkeysAudio on 2022-08-12 20:37:02
Hi TBeck.  I was just wondering if you would let TAK be included in Monkey's Audio like FLAC and WavPack are already?

I have the APX file authored and it supports compression, decompression, and verify.

I just don't want to include it unless you give your blessing.

Also, really impressive everything you've done!

Thanks.
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-09-11 15:27:48
So great that TAK is alive! Encoding whole collection for years and everything new incoming.
Thank you. That's very motivating!
Title: Re: TAK 2.3.3 Beta thread
Post by: TBeck on 2022-09-11 16:02:35
Hi TBeck.  I was just wondering if you would let TAK be included in Monkey's Audio like FLAC and WavPack are already?

I have the APX file authored and it supports compression, decompression, and verify.

I just don't want to include it unless you give your blessing.

Also, really impressive everything you've done!

Thanks.
Hi Matt,

it is an honour for me. Really. Please do it.

Without MonkeysAudio there would be no TAK. TAK's early development was primarily driven by my desire to create a fast asymmetrical compressor that could compete with MA's High preset. It wasn't easy...

Sorry for the late reply. I just noticed you sent me me a PM too. For some reason i received no notification.
Title: Re: TAK 2.3.3 Beta thread
Post by: MonkeysAudio on 2022-09-11 16:28:45
Thanks Thomas!

Just released:
https://monkeysaudio.com/download.html

I tested on my Windows 7 bottle and my Windows 11 main machine and it works well.  Just let me know if you want anything tuned.

Really amazing work  :)
Title: Re: TAK 2.3.3 Beta thread
Post by: Destroid on 2022-09-17 10:43:24
I'm really overwhelmed by these last few messages, in a good way. :)

With MA, I had the tools needed to start home recording  in Cool Edit Pro and be able to save on storage. I became obsessed with compression ratio vs. performance.

TAK (aka YALAC) was really fresh when it appeared and I was instantly intrigued, and was not disappointed. :)

Before I ramble further, I want to thank Matt and Thomas and David and all the developers and contributors and HA for making audio an awesome experience.

edit: typo