Skip to main content

Topic: TAK 2.2.0 - Alpha release (Read 28817 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Alpha release 1 of TAK 2.2.0 ((T)om's lossless (A)udio (K)ompressor)

It consists of:

- TAK Applications 2.2.0 Alpha 1 b in "\Applications".
- TAK Winamp plugin 2.2.0 Alpha 1 in "\WinAmp".
- TAK Decoding library 2.2.0 Alpha 1 in "\Deco_Lib".

The final release will additionally contain the SDK.

Download:

Download link removed. TAK 2.2.0 Final has been released.

What's new

This release brings support for multi-channel audio and speed optimizations for encoder and decoder.

New features:

- Support for multi-channel audio. While the stream format supports up to 16 channels, the codec currently is restricted to a maximum of 6 channels.
- Support for the "Wave Format Extensible" file format.

Improvements:

- Encoding speed improvements of up to 10 percent for my primary file set. Most of it is achieved by a modification of the algorithm which selects the optimal predictor order for each subframe. It will now often use less predictors than before, what may on average result in about 0.01 percent worse compression. You will only notice an speed advantage, if your files benefit from high predictor orders.
- Decoding speed improvements of up to 18 percent for my primary file set. Some of it is attributed to the above-noted modification of the encoder's predictor order selection algorithm. Therefore it will only take effect when decoding files encoded with this version and only, if they benefit from high predictor orders. Additionally SSSE3-instructions can be used for predictor counts of 32 and more. This affects the presets p3, p4 and sometimes p2, but only, if a particular file benefits from high predictor orders.

Known issues:

- If you use pipe decoding and the application reading the pipe is beeing terminated before the whole file has been read, TAKC may get into an endless loop and has to be manually killed with the task manager. I don't think this is a big issue but i will try to fix it in one of the next versions. BTW: Big thanks to shnutils for testing the pipe decoding!
- There seem to be some compatibility issues with pipe decoding to some other applications ("crc1632.exe" has been reported). I will try to fix it in the next release.

Alpha testing

This alpha release has already gone through extensive testing performed by my automatic scripts. Nevertheless there may be bugs left. Especially because i had to write a lot of new code for the support of multi-channel audio. This also involved a lot of minor modifications of the existing code. Therefore i would like you to verify the proper function of the codec: Compress -> Decompress -> Compare resulting wave with the original file, either by a binary compare or by the use MD5-check sums.

Certainly i am very interested into efficiency comparisons of the new multi-channel audio codec and other compressors.

Some remarks:

The most time consuming part of the new codec is it's channel decorrelation mechanism. The strongest presets sometimes check any possible channel combination. Principially you have n * (n - 1) (n = channel count) possible combinations, if you count "A predicts B" and "B predicts A" as two combinations.

Some figures:

2 channels ->  2 combinations
4 channels -> 12 combinations
5 channels -> 20 combinations
6 channels -> 30 combinations
8 channels -> 56 combinations

This rapid increase is the reason, why the codec currently is restricted to a maximum of 6 channels. I have to find and evaluate more heuristics for a fast estimation of optimal pairings which doesn't require a full evaluation of all possibilities.

Some are alreaday working. Most of them rely on the presence of a speaker assignment mask in the source wave file. If present, some faster presets will only test those combinations, which were most useful in my evaluations. A low frequency channel will never be evaluated.

But this only works, if the speaker assignment is known. Therefore the encoding time of the same audio data may differ considerably dependent on the presence of a speaker mask in the original wave file.

Im my evaluations the new codec often did beat Mpeg4Als -7 compressionwise, if a particular file provided good opportunities for channel decorrelation. Unfortunately for some files there are zero opportunities. My test corpus is still too small to make any generalized statement regarding the new codecs efficiency.

Therefore i am very interested into compression results and comparisons with other codecs.

Thanks for testing and have fun

Thomas
  • Last Edit: 10 July, 2011, 06:53:19 PM by TBeck

  • _m²_
  • [*][*][*]
TAK 2.2.0 - Alpha release
Reply #1
I get "Wave file not supported" on all files.
Sample:
http://www.hydrogenaudio.org/forums/index....showtopic=88969
BTW flood protection sucks.
Come on, what a problem with users writing 2 simple posts within a minute?

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #2
I get "Wave file not supported" on all files.
Sample:
http://www.hydrogenaudio.org/forums/index....showtopic=88969

Hm, works for me. I tried encoding with both Tak and Takc and decoding with Tak. Binary compare of the resulting wave file is also ok.

edit: Here is TAK's file info result:

Code: [Select]
=== d2.tak ====================================================

  File size:                     2.37 MB
  Header size:                   0.13 KB
    Unused:                      0.00 KB
  Compression:                  36.01 %
  Samples per channel:         384000
  File duration:                 8.00 sec
  Frame duration:                 125 ms
  Seek table:              Not available
  Audio format:            PCM, 48000 Hz, 24 Bits, 6 Channels
  Encoder:                 V 2.2.0, -p2
  Codec:                   4 Integer 24 bit MC (TAK 2.2)
  Wave file meta data:     Header 68, Footer 0 Bytes
  MD5:                     Not available
  APEv2-Tag:               No
  Status:                  Ok
  • Last Edit: 06 June, 2011, 12:37:10 PM by TBeck

  • lvqcl
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #3
Quote
Hm, works for me.

Works here too.

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #4
Although there is no hint, i just looked at my wave file reader and found some unfortunate limitation. It seems to insist on the wValidBitsPerSample value to be zero or equal to wBitsPerSample. That's not how it is meant to work. This could be a source of your problem. Maybe the editor you are using to create the sample file is modifying this field in the way TAK expects.

edit: Addendum

I can post a quick fix for you to try.
  • Last Edit: 06 June, 2011, 01:08:14 PM by TBeck

  • _m²_
  • [*][*][*]
TAK 2.2.0 - Alpha release
Reply #5
The file was created with sox. I just tried Tak instead of Takc with the same result. I use Windows XP x64 if that matters.
I can only repeat my suggestion from long ago:
Make Takc more verbose. "Wave file not supported" is not really helpful.

  • lvqcl
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #6
Quote
The file was created with sox.

What version? 14.3.2 or earlier?

Added: sox 14.3.1 and earlier writes dwChannelMask = 0.
  • Last Edit: 06 June, 2011, 01:15:11 PM by lvqcl

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #7
The file was created with sox. I just tried Tak instead of Takc with the same result. I use Windows XP x64 if that matters.

Can you please try this TAK.exe: Attachment removed
  • Last Edit: 06 June, 2011, 03:03:04 PM by TBeck

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #8
I can only repeat my suggestion from long ago:
Make Takc more verbose. "Wave file not supported" is not really helpful.

Well, since this is possibly a bug, i am not sure if a more informative message would have been created.

Nevertheless i will think about it. I don't want to break the current design of the internal error handling, but maybe i can add a function to request extended error info.

Quote
The file was created with sox.

Added: sox 14.3.1 and earlier writes dwChannelMask = 0.

Thanks for the hint. But this should(!) be no problem.

  • lvqcl
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #9
I decoded d2.flac to 2 wav files with SoX 14.3.1 and 14.3.2.

They have different RIFF chunk sizes: 0x0069783c for 14.3.1 and 0x00697848 for 14.3.2.

takc.exe can encode the latter file, but not the former.

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #10
I decoded d2.flac to 2 wav files with SoX 14.3.1 and 14.3.2.

They have different RIFF chunk sizes: 0x0069783c for 14.3.1 and 0x00697848 for 14.3.2.

takc.exe can encode the latter file, but not the former.

A difference of 12 bytes? I am having some difficulties to interpret this. Well, i am really not the master of sound file formats...

Could you please send me the maybe first 512 bytes of both files?

  • lvqcl
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #11
--
  • Last Edit: 06 June, 2011, 02:26:19 PM by lvqcl

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #12
I decoded d2.flac to 2 wav files with SoX 14.3.1 and 14.3.2.

They have different RIFF chunk sizes: 0x0069783c for 14.3.1 and 0x00697848 for 14.3.2.

takc.exe can encode the latter file, but not the former.

A difference of 12 bytes? I am having some difficulties to interpret this. Well, i am really not the master of sound file formats...

Could you please send me the maybe first 512 bytes of both files?

Sorry, no longer needed! I had some trouble downloading 14.3.1 from sourceforge, but finally it worked. Now i can reproduce the failure.

edit: I was too slow! Thank you.
  • Last Edit: 06 June, 2011, 02:11:14 PM by TBeck

  • _m²_
  • [*][*][*]
TAK 2.2.0 - Alpha release
Reply #13
Sorry, my bad, seems to be a flake bug, the flac doesn't decompress correctly!
I'm going to replace it in a minute with a wavpack file which I just verified to be OK. The latest build shows the same problem.
Sox 14.3.2 BTW.
ADDED: I can confirm that both versions of tak compress the decompressed flac correctly.
ADDED: The wavpack compressed file is uploaded. Same place as before, http://www.hydrogenaudio.org/forums/index....showtopic=88969
  • Last Edit: 06 June, 2011, 02:20:39 PM by _m²_

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #14
Sorry, my bad, seems to be a flake bug, the flac doesn't decompress correctly!
I'm going to replace it in a minute with a wavpack file which I just verified to be OK. The latest build shows the same problem.
Sox 14.3.2 BTW.
ADDED: I can confirm that both versions of tak compress the decompressed flac correctly.
ADDED: The wavpack compressed file is uploaded. Same place as before, http://www.hydrogenaudio.org/forums/index....showtopic=88969

Thank you. I can reproduce the failure.

They have different RIFF chunk sizes: 0x0069783c for 14.3.1 and 0x00697848 for 14.3.2.

takc.exe can encode the latter file, but not the former.

Often i may be a bit too restrictive when reading foreign file formats...

My wave reader starts like this:

Code: [Select]
  if not FFile.Seek (0) then begin
    Result := frr_IoError;
    EXIT;
  end;
  if FFile.Length < SizeOf (Chunk) then begin
    Result := frr_InvalidType;
    EXIT;
  end;
  if not FFile.ReadBuffer (Chunk, SizeOf (Chunk)) then begin
    Result := frr_IoError;
    EXIT;
  end;
  if Chunk.Id <> WaveRiffId then begin
    Result := frr_InvalidType;
    EXIT;
  end;
  if Chunk.nBytes <> FFile.Length - SizeOf (Chunk) then begin
    Result := frr_StructError;
    EXIT;
  end;

It's the last check which is causing the trouble. The reader insits, that nothing comes after the RIFF-chunk and it's data. Could it be that Sox is adding some meta data? I am a bit irritated, because i never before received a report about such a failure.

  • lvqcl
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #15
From SoX changelog:

"Fix bug where FACT chunk was not accounted for in RIFF length calculation.  Patch by David Bryant."

Probably that's why SoX 14.3.2 works fine but 14.3.1 doesn't.

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #16
From SoX changelog:

"Fix bug where FACT chunk was not accounted for in RIFF length calculation.  Patch by David Bryant."

Probably that's why SoX 14.3.2 works fine but 14.3.1 doesn't.

Thank you so much! This explains everything. After inspecting the file with a hex editor, i just wanted to post the suspicion, that the file is invalid.

Then everything is ok (for now...).

Thank you all for helping me to resolve this issue.

  • [JAZ]
  • [*][*][*][*][*]
TAK 2.2.0 - Alpha release
Reply #17
Indeed, that last check shouldn't be made.

Microsoft's Wave format is a RIFF based format, and the RIFF format is just a union of labeled chunks of data.
I don't know if it is usual nowadays, but wave editors used to add an INFO FACT chunk adding their name to the files. This is just one of many things that can be also in the file. And then, there is the people that insist in adding tags to wave files...

You have to assert that the chunk's size of the "data" chunk is not bigger than the current position minus file length, but only that. Read up to chunk size, and if you are not at the file end, then there's something else that you don't care.


So:

Open
Read 8 bytes-> 4bytes Id and 4 bytes size (in little endian, in the case of RIFF files). Verify that it's RIFF and the chunk size is smaller than the file size.
Read next 4 bytes if it's WAVE, continue.
Read 8 byes -> 4 bytes ID and 4 bytes size. Verify that the ID is "fmt " (there's a space in there, yes). The chunk size can also be an indication of which waveformat you have. Read up to fmt's chunk size.
Read 8 bytes -> 4 bytes ID and 4 byte size. loop while this chunk ID isn't "data", skipping the chunk size.
Read wave data up to this chunk size.
Close.

This is a basic Wave file reader. Something simpler is not a wave reader, just a reader that happens to read most wave files.
  • Last Edit: 06 June, 2011, 03:08:15 PM by [JAZ]

  • lvqcl
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #18
Quote
Read up to chunk size, and if you are not at the file end, then there's something else that you don't care.


...but please don't break -ihs switch

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #19

Indeed, that last check shouldn't be made.

Microsoft's Wave format is a RIFF based format, and the RIFF format is just a union of labeled chunks of data.
I don't know if it is usual nowadays, but wave editors used to add an INFO chunk adding their name to the files. This is just one of many things that can be also in the file. And then, there is the people that insist in adding tags to wave files...

You are right.

I think my motivation for including this check was that i wanted to make sure, that i don't unintentionally ignore additional wave data chunks possibly added by applications trying to circumvent the file formats size limitations. Maybe this is a non-issue, but as i wrote above, i am (obviously) not an expert for audio file formats.

I will modify the wave reader for the next release.

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #20
Quote
Read up to chunk size, and if you are not at the file end, then there's something else that you don't care.


...but please don't break -ihs switch

How can you say something so horrible?! I never would (intentionally...)!

  • Destroid
  • [*][*][*][*][*]
TAK 2.2.0 - Alpha release
Reply #21
I managed to finish a preview of a TAK benchmark due to a conversion issue.* This short test only covers TAK and not other multichannel encoders. The end result is that TAK 2.2.0 completed the test successfully without any errors, also noted encoding speed-ups with stereo material. Machine used: Athlon XP 3000+, 2GB, WD1.5GB SATA, WinXP SP3, NTFS.*
Code: [Select]
Source files:
DTS source  6ch 16bit 48KHz    3.29GB  102:19.125  RG -5.61dB
CD audio 1  2ch 16bit 44.1KHz  297MB  29:26.392  RG -9.75dB
CD audio 2  2ch 16bit 44.1KHz  304MB  30:07.258  RG -10.71dB 


encoder / file  enc                            timer301.exe (enc / dec)
 (-p2 setting)  ratio  enc / dec speed    (Kernel)  (User)      (Process)
--------------  ------  -----------------  ----------------------------------
TAK 2.2.0 DTS  40.30%  15.49x /  59.00x  (2% / 13%) (95% / 75%) (98% / 88%)
TAK 2.2.0 CD 1  67.12%  115.10x / 197.68x  (6% / 14%) (85% / 83%) (91% / 98%)
TAK 2.2.0 CD 2  68.96%  114.53x / 191.90x  (5% / 12%) (86% / 84%) (91% / 96%)
TAK 2.1.0 CD 1  67.12%  110.49x / 192.62x  (5% / 14%) (86% / 80%) (91% / 94%)
TAK 2.1.0 CD 2  68.96%  106.71x / 191.48x  (5% / 12%) (84% / 83%) (90% / 95%)


bitcompared decoded TAK files with original WAV files = No differences.
[!--sizeo:1--][span style=\"font-size:8pt;line-height:100%\"][!--/sizeo--]*conversion issue: TAK reports Wave file not supported on WAV files with durations >02:04:16 (converted DTS->WAV using FB2K v1.1.6, foo_input_dts.dll v0.3.0 & specs above). Sizes of resulting WAV files are shown to be 4.04GB and 4.10GB even though every media player reports a durations of 02:04:16 for both those files. Anyone give me hint at fixing this (possibly) WAV header issue?[/size]
"Something bothering you, Mister Spock?"

  • lvqcl
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #22
Quote
Anyone give me hint at fixing this (possibly) WAV header issue?


1) Convert DTS to W64.
2) Convert W64 to WAV, encode to TAK with (the above-mentioned) -ihs switch.
3) Compare W64 and TAK.

Added: according to takc help, -ihs works only in pipe mode... 
takc.exe -e -ihs -p2 - out.tak < in.wav
  • Last Edit: 07 June, 2011, 08:23:00 AM by lvqcl

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #23
I managed to finish a preview of a TAK benchmark due to a conversion issue.* This short test only covers TAK and not other multichannel encoders. The end result is that TAK 2.2.0 completed the test successfully without any errors, also noted encoding speed-ups with stereo material. Machine used: Athlon XP 3000+, 2GB, WD1.5GB SATA, WinXP SP3, NTFS.*
...
bitcompared decoded TAK files with original WAV files = No differences.

Thank you! That's good news.

*conversion issue: TAK reports Wave file not supported on WAV files with durations >02:04:16 (converted DTS->WAV using FB2K v1.1.6, foo_input_dts.dll v0.3.0 & specs above). Sizes of resulting WAV files are shown to be 4.04GB and 4.10GB even though every media player reports a durations of 02:04:16 for both those files. Anyone give me hint at fixing this (possibly) WAV header issue?

Well, add about half a second to 02:04:16 and your files exceed the 4 GB limit of the wave file format. There is little i can do besides adding support for something like the Wave64-Format.

  • TBeck
  • [*][*][*][*][*]
  • Developer
TAK 2.2.0 - Alpha release
Reply #24
takc.exe -e -ihs -p2 - out.tak < in.wav

Good idea! Currently i have no time to look at the source to check if this works for files larger than 4 GB. TAK's stream functions definitely support more than 4 GB, but i am not sure, if i have somewhere coded some restriction. Anyway, you will not be able to decode such a file while TAK only supports the standard wave format as output format...