Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: TAK 1.0 - Beta testing (Read 104053 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

TAK 1.0 - Beta testing

Reply #50
I found an very interesting file, that answers in an almost linear way to the increase of compression time, in case of TAK and MAC, and in others it is an jump in compression in the very high setings:

Very interesting file! I have downloaded it and played with it.

The compression delta for TAK is 4.65 %, wich seems very high for TAK. And the compression delta between Flac -0 and this extreme mode in OptimFrog is 12.04 %. Flac is the "left zero", with an delta of only 2.55, but Flake got the trick somewere beteween -8 and -12.

Usually there are 2 possible reasons for such a big delta:

1) The file likes the PreFilter.
2) The file likes many predictors.

This time the answer is 2).

Another thing intersting in this file is the "max" swich in TAK. Here the gains in compression with max for each preset:

Code: [Select]
Turbo Max over Turbo        0.64 %
Fast Max over Fast         0.55 %
Normal Max over Normal      0.20 %
High Max over High          0.35 %
Extra Max over Extra        0.46 %

It increases after "normal"! The High Max compress beter than extra simple!

You are right: Usually the "max" switch isn't doing much for presets higher than FAST, because starting with NORMAL most of the options "max" could add are already part of the presets default setting.

The option "Frame partition calculator - Validate" is slow and quite useless (on average less than 0.05 percent advantage) most of the time. But it can help some problem files (especially with high predictor orders) and this is one of them:

Extra + Max: 67.77
Extra + Max without Validate: 68.16
Difference: 0.39

There are some more hidden options in TAK, which can help rare problem files. Currently they are disabled, because they are very slow and on average have as little effect on compression as Validate.

Thanks for providing such an interesting sample for my test corpus! It can help me with further optimizations.

TAK 1.0 - Beta testing

Reply #51
And no, i don't intend to drop the Turbo mode, exactly for the same reasons as you!
Maybe i will try to make it a bit faster sometime...


Thank you very much to keep this! The high speed and low CPU usage is very suitable for video capture.
Looking forward to TAK access via ACM or DirectShow!
Hong Kong - International Joke Center (after 1997-06-30)

TAK 1.0 - Beta testing

Reply #52
There are some more hidden options in TAK, which can help rare problem files. Currently they are disabled, because they are very slow and on average have as little effect on compression as Validate.

Just tried some of the hidden options: About 0.20 percent better compression, but 2.5 times slower encoding.

Well, there is room for some more encoder optimizations (without breaking the current design and compatibility) in the future, but don't expect too much. Maybe 0.15 to 0.30 percent on average and only with a heavy speed penality for encoding.

TAK 1.0 - Beta testing

Reply #53
Well, there is room for some more encoder optimizations (without breaking the current design and compatibility) in the future, but don't expect too much. Maybe 0.15 to 0.30 percent on average and only with a heavy speed penality for encoding.

Compression improvement from 0.2 to 0.3 percent is enough to beat MAC Extra high (or at least to be on par), so it sounds very promising anyway. Encoding speed penalty isn't a big problem in this case until decoding still much faster than MAC.

TAK 1.0 - Beta testing

Reply #54
I found a file that benefits from prefilter, but only from "fast" and up. It is from an 448kbps AC3 track of a DVD. I'm omiting the normal and high modes as they behave in the same way as fast and extra in all the situations I tested.

Code: [Select]
prefilter-test.wav 

TAK and diferent presets and prefilters strenghts

Mode             Compression     Speed

Turbo + off        40.15 %        45x
Turbo + low        40.20 %        35x
Turbo + medium     40.31 %        35x
Turbo + high       40.38 %        35x

fast + off         39.95 %        35x
fast + low         37.83 %        26x
fast + medium      37.83 %        25x
fast + high        37.83 %        25x

extra + off        39.77 %        12x
extra + low        37.33 %        10x
extra + medium     37.33 %        10x
extra + high       37.33 %        10x

If I re-encode to vorbis q4, the benefits of pre-filter grows a little for all presets, except turbo, where the compression hurt is even higher.

If I re-encode to vorbis q-2 (~32kbps) turbo starts to get benefits, but not much, and in an strange patern:
Code: [Select]
Mode             Compression     

Turbo + off        37.45 %      
Turbo + low        37.28 %      
Turbo + medium     37.31 %      
Turbo + high       37.30 %      

fast + off         37.12 %      
fast + low         34.83 %        
fast + medium      34.83 %  
fast + high        34.83 %      

extra + off        36.12 %      
extra + low        34.15 %      
extra + medium     34.15 %      
extra + high       34.15 %

If I re-encode (it is geting a bit repetitive, isn't?) to Winamp's AACv2 32kbps, the figure is totaly diferent. Now Turbo gains a little with prefilter, and all other presets loose compression:
Code: [Select]
Mode             Compression     

Turbo + off        37.42 %      
Turbo + low        37.29 %      
Turbo + medium     37.30 %      
Turbo + high       37.33 %      

fast + off         36.83 %      
fast + low         37.16 %        
fast + medium      37.29 %  
fast + high        37.32 %      

extra + off        36.64 %      
extra + low        36.88 %      
extra + medium     37.00 %      
extra + high       37.00 %

Now we have an extra mode that compress by defaut worse than the fast mode.     

I can send it to you if you want. It is only 23 seconds long, and I'm keeping the vorbis and AAC compressed ones. May help you tune the prefilter, or even understand it better.

TAK 1.0 - Beta testing

Reply #55
Quote
You found the secret String "tBaK"... I have to hide it better the next time...

I also saw this sometime ago, when cutting the files... it isn't very secret... what is the great deal?
Quote
2) The file likes many predictors.

This time the answer is 2).

Indeed. The ground breaking change from Flake -8 to Flake -12 shoud be the increased prediction order from 12 to 32. Maybe the same thing between the wavepack's hardware capable -h and the non hardware capable -hh. But that brings a question: is TAK turbo, fast and normal modes hardware capable, even with such high predictors number?
Quote
Well, there is room for some more encoder optimizations (without breaking the current design and compatibility) in the future, but don't expect too much. Maybe 0.15 to 0.30 percent on average and only with a heavy speed penality for encoding.

That is still good. When compressing those 30s samples for upload, or if you have an quad quad-core K8L system (AMD showed one of those one month ago...) with an future multithreaded version of TAK, encoding speed isn't a problem.

By the way, my feature request list, in the order of importance to me:

- Unicode Suport
- DirectShow filter
- GStreamer plugin
- Matroska embedding (I'm not sure if this is the correct word)
- Multitreading (for dualcores and quadcores at least)
- More compression!
- Native suport for TAK to TAK transcoding
- ACM filter

I don't know if all them are possible, and I know that some of them are very tricky, especialy the unicode... windows vs rest-of-the-world problems, developer system problems (you are in windows 98, isn't?), etc. So you can put it in your TODO list in any order you want. And I don't listed the other things that are even more important, but already are in your TODO list.

TAK 1.0 - Beta testing

Reply #56
I'll test with either a Mac binary (linux i guess is ok too) or source... but Windows = nope, can't, sorry
err... i'm not using windows any more ;)

TAK 1.0 - Beta testing

Reply #57
Quote
You found the secret String "tBaK"... I have to hide it better the next time...

I also saw this sometime ago, when cutting the files... it isn't very secret... what is the great deal?

I think he was just a bit ironic!
Not a big deal, indeed; I just tought it would have be nice to add TAK to the other various  filetypes that can be identified.

Bye!

TAK 1.0 - Beta testing

Reply #58

Quote
You found the secret String "tBaK"... I have to hide it better the next time...

I also saw this sometime ago, when cutting the files... it isn't very secret... what is the great deal?

I think he was just a bit ironic!
Not a big deal, indeed; I just tought it would have be nice to add TAK to the other various  filetypes that can be identified.

Bye!

Yeah... I thought so... but curiosity was bigger. 

TAK 1.0 - Beta testing

Reply #59
Here a file that benefits a lot from the prefilter. Up to 4.30% in turbo, down to 4.15% in extra. It is from an 192kbps MP2 layer II file.

No significant anomalities in this one. Only an loose of 0.01% in fast on "medium" and "high" settings.

Code: [Select]
prefilter friendly.wav

Mode             Compression    

Turbo + off        66.17 %      
Turbo + low        61.88 %      
Turbo + medium     61.88 %      
Turbo + high       61.87 %      
Turbo Max          65.25 %

fast + off         65.41 %      
fast + low         61.20 %        
fast + medium      61.21 %  
fast + high        61.21 %      
fast max           64.69 %

normal + off       65.11 %      
normal + low       60.90 %        
normal + medium    60.90 %  
normal + high      60.90 %      
normal max         64.53 %

high + off         64.96 %      
high + low         60.81 %        
high + medium      60.81 %  
high + high        60.81 %      
high max           60.46 %

extra + off        64.92 %      
extra + low        60.77 %      
extra + medium     60.77 %      
extra + high       60.77 %
extra max          60.44 %

MAC fast           70.60 %
MAC normal         63.87 %
MAC extra high     59.12 %

The Max also helps a lot. 0.93% in turbo, 0.88 in normal, 0.33 in extra.

When re-encoding to vorbis q2 (96kbps) no strange things also:
Code: [Select]
prefilter friendly vorbis q2.wav
Mode             Compression    

Turbo + off        64.08 %      
Turbo + low        62.27 %      
Turbo + medium     62.21 %      
Turbo + high       62.19 %      
Turbo Max          63.26 %

fast + off         63.79 %      
fast + low         60.31 %        
fast + medium      60.31 %  
fast + high        60.31 %      
fast max           62.97 %

normal + off       63.16 %      
normal + low       59.92 %        
normal + medium    59.92 %  
normal + high      59.92 %      
normal max         62.64 %

high + off         62.89 %      
high + low         59.77 %        
high + medium      59.77 %  
high + high        59.77 %      
high max           59.43 %

extra + off        62.79 %      
extra + low        59.69 %      
extra + medium     59.69 %      
extra + high       59.69 %
extra max          59.40 %

MAC fast           69.26 %
MAC normal         62.14 %
MAC extra high     56.88 %

The tread is about TAK, but the compression delta of MAC in this file is impressive. The compression level at extra high also is so. If weren't for the prefilter...

I didn't compressed at lower bitrates because anything under this sounds horrible. Probably because it is an relatively low bitrate transcoding...

The files are also avaliable on request.

TAK 1.0 - Beta testing

Reply #60
Quote
You found the secret String "tBaK"... I have to hide it better the next time...

I also saw this sometime ago, when cutting the files... it isn't very secret... what is the great deal?
Quote
2) The file likes many predictors.

This time the answer is 2).

Indeed. The ground breaking change from Flake -8 to Flake -12 shoud be the increased prediction order from 12 to 32. Maybe the same thing between the wavepack's hardware capable -h and the non hardware capable -hh. But that brings a question: is TAK turbo, fast and normal modes hardware capable, even with such high predictors number?

Turbo is using up to 16, Fast up to 32 predictors (like flake -12). I don't know, how many hardware players are fast enough for this.

Prefilter

I found a file that benefits from prefilter, but only from "fast" and up. It is from an 448kbps AC3 track of a DVD. I'm omiting the normal and high modes as they behave in the same way as fast and extra in all the situations I tested.
...
If I re-encode to vorbis q4, the benefits of pre-filter grows a little for all presets, except turbo, where the compression hurt is even higher.
...
If I re-encode to vorbis q-2 (~32kbps) turbo starts to get benefits, but not much, and in an strange patern:
...
If I re-encode (it is geting a bit repetitive, isn't?) to Winamp's AACv2 32kbps, the figure is totaly diferent. Now Turbo gains a little with prefilter, and all other presets loose compression:
...
Now we have an extra mode that compress by defaut worse than the fast mode.     
...
I can send it to you if you want. It is only 23 seconds long, and I'm keeping the vorbis and AAC compressed ones. May help you tune the prefilter, or even understand it better.

Here a file that benefits a lot from the prefilter. Up to 4.30% in turbo, down to 4.15% in extra. It is from an 192kbps MP2 layer II file.
...


I really appreciate your work! But to be honest, for me it does not make much sense to check the (lossless) compressability of lossy encoded files.

A bit of the PreFilter history can be found in the Yalac evaluation thread. Summary:

1) The PreFilter sigificantly affects the decompression speed.
2) The PreFilter is usually more efficient on files which like higher predictor orders.
3) On average it's advantage lies in the range of 0.1 to 0.3 percent, sometimes up to 1 percent.
4) Lossy encoded files can benefit tremendously from the PreFilter. I have seen 2 to 6 percent better compression.
5) I could improve the PreFilter efficiency for lossy files further, but what is it worth?
6) Not all lossy compressed files are benefiting from the PreFilter, it depends on the encoder and it's parameters.

Because of the decoding speed penality i earlier thought about dropping the PreFilter. One reason why i haven't:

Mpeg4Als - 7 is quite efficient in compressing this lossy encoded kind of files. Here it can be up to 2 percent better than Tak, while it usually isn't more than about 0.3 percent better.

It's possible, that someone publishes comparisons of lossless compressors based on evaluations of lossy encoded file! This shouldn't happen but it does. I simply don't want Tak to look bad in such (quite useless) evaluations. One important reason for keeping the PreFilter.

TAK 1.0 - Beta testing

Reply #61
Quote
I really appreciate your work! But to be honest, for me it does not make much sense to check the (lossless) compressability of lossy encoded files.

I've read in the read-me that the prefilter is good for lossy files, so I selected some to test. It realy isn't very meaningfull. I've only losslessly compressed an lossy file one time before, and wasn't for archieve purposes...

Some reasons for losslessly recompress lossy files:
1)  You want to convert an rare or proprietary format without loss of quality in something you can use. May be because you can't hear it in another OS (ie: Linux, FreeBSD...) or in an hardware player.
2) You want all your colection in only one format.
3) You don't know it was lossy compressed before.
4) You decoded it to wave for whatever reason and deleted the source by accident (prefilter testing anyone? =P). Don't want to loose quality again because of this.
5) You is crazy, or don't understand audio compression.   

This is all that I can think. It don't need much atention... there are many more important things to do. May be in future when you are bored...   

TAK 1.0 - Beta testing

Reply #62
Though more compression results aren't super important, I decided to do a few myself.  I wanted to compare flac -6 (what I currently use) against tak -p3 for a few different albums.  I did not bother to list decoding speeds since both flac -6 and tak -p3 were blazing fast to decode - about 90x.  For reference, this test was done on an Athlon64 3400+ running Win2000. 
Code: [Select]
Album:  Sarah McLachlan - Solace
Style of music:  Quieter stuff, and it's a bit older so it's mastered much lower than albums today.
       Mode        Size       Encoding Rate
Original Image:   486 MB          n/a
Flac -6:          262 MB          50x
Tak -p3:          246 MB          34x


Album:  Green Day - American Idiot
Style of music:  Harder rock, mastered very loudly and compressed.
       Mode        Size       Encoding Rate
Original Image:   578 MB          n/a
Flac -6:          415 MB          41x
Tak -p3:          407 MB          34.5x


Album:  Techmaster P.E.B. - Bass Computer
Sytle of music:  Bass music.  Virtually no vocals, all electronic with LOTS of low frequency content.  Well mastered.
       Mode        Size       Encoding Rate
Original Image:   624 MB          n/a
Flac -6:          351 MB          46x
Tak -p3:          331 MB          35.5x


Fun results.  Flac -6 has tak -p3 beat in encoding speed, though encoding at 35x like Tak -p3 does on my computer is still very fast.  But in terms of compression, tak -p3 absolutely destroys flac -6, and tak seems to do especially well on material that has not been mastered super loudly.  Cutting 16 MB off the Sarah McLachlan album is outstanding!

Lastly, I really want to say thank you to TBeck and all the alpha testers for working on this codec.  I remember long ago when TBeck first posted about his lossless codec and I was excited that I was going to be able to follow along as it developed.  This first public release is certainly a major milestone.  Congratulations, and thanks.

TAK 1.0 - Beta testing

Reply #63
According to my comparison TAK Normal only compresses 0.2% less than High, but encodes faster than FLAC -6 - about the same speed as FLAC -5.

Code: [Select]
Encoder Setting    Compression    Encode Rate    Decode Rate
============================================================
TAK Normal             63.875%            40x            83x
FLAC -5                65.926%            38x            79x
TAK Turbo Max          64.520%            36x            83x
FLAC -6                65.881%            34x            79x
TAK High               63.684%            27x            76x
I'm on a horse.

TAK 1.0 - Beta testing

Reply #64
Is there an easy way I can locate the frame headers within a TAK file using a hex editor or such? More importantly is there an easy way I can locate the checksums within the headers(and at the start of the file) I'm going to try and get a bit more crafty with my damage tests as so far every error gets reported correctly.

TAK 1.0 - Beta testing

Reply #65
Some reasons for losslessly recompress lossy files:
...

Very creative! I agree, there can be reasons for compressing lossy files.

Fun results.  Flac -6 has tak -p3 beat in encoding speed, though encoding at 35x like Tak -p3 does on my computer is still very fast.  But in terms of compression, tak -p3 absolutely destroys flac -6, and tak seems to do especially well on material that has not been mastered super loudly.  Cutting 16 MB off the Sarah McLachlan album is outstanding!
...
Lastly, I really want to say thank you to TBeck and all the alpha testers for working on this codec.  I remember long ago when TBeck first posted about his lossless codec and I was excited that I was going to be able to follow along as it developed.  This first public release is certainly a major milestone.  Congratulations, and thanks.

Thank you! Indeed Tak isn't as good in compressing loud and (dynamic wise) compressed material.

Is there an easy way I can locate the frame headers within a TAK file using a hex editor or such? More importantly is there an easy way I can locate the checksums within the headers(and at the start of the file) I'm going to try and get a bit more crafty with my damage tests as so far every error gets reported correctly.

Each frame header starts with a 16 bit sync code (byte aligned):

0A0FFh =  41215 (decimal)  = 10100000.11111111 (bits)

I wrote a littel tool to find the pattern with the lowest probability (1 to about 80,000) and this came out.

Nevertheless this pattern is also likely to be found in the audio data.

For the checksums: There are two per frame: One 24 bit CRC following the frame header to protect the header and to make the identification of headers (avoid false alarm caused by sync code only) more safe.

Next comes the compressed audio data and then (byte aligned) another 24 bit CRC to protect the audio data.

Illustration:
Code: [Select]
[Frame header]  [Frame data]  [Frame header]  [Frame data]

Frame header is:

[Sync code - Header data - CRC]

Frame data is:

[Audio data - CRC]


Hope this helps.

TAK 1.0 - Beta testing

Reply #66
According to my comparison TAK Normal only compresses 0.2% less than High, but encodes faster than FLAC -6 - about the same speed as FLAC -5.

Good point, Synthetic Soul.  Personally I like as much compression as possible while mantaining good speeds.  That's why I use flac -6 rather than -7 or -8.  At this point, with tak -p3 going at about 35x on my computer, I'd be more than happy with that speed, I think. 

Anyway, I have looked at your website and the extensive lossless tests you've done.  While tak was in alpha, it was a key source of information for me.  But now that tak is in public beta, I couldn't resist doing a few quick tests of my own. 

TAK 1.0 - Beta testing

Reply #67
Some reasons for losslessly recompress lossy files:
...
Very creative! I agree, there can be reasons for compressing lossy files.

But of course none of them are a valid reason to optimize lossless compression for it.
In theory, there is no difference between theory and practice. In practice there is.

TAK 1.0 - Beta testing

Reply #68
Last chance to report bugs in Beta 1

I want to release the second (and hopefully final) beta soon. Please tell me, if you have found any bugs in beta 1 not contained in the list below:

Fixed:

- If the last 1 to 3 bytes of a (damaged) file had been cut off, the automatic repair function of the decoder crashed.

Improved:

- If one file was undecodable, Tak shows the hint "Try disabling "Restore wave file meta data" (-wm0)."
- The state of the Verify-option is beeing preserved when changing presets in the GUI version.
- Now i know that i have to write "already exists" instead of "allready exists"...

  Thomas

TAK 1.0 - Beta testing

Reply #69
Other than the issue you have fixed I've been unable to make any unreported or undecodable errors.

TAK 1.0 - Beta testing

Reply #70
Last chance to report bugs in Beta 1

I have found one by myself...

The Tak stream contains a seek table for fast random access to play positions. Because the seek table has not to be accessed for continuous decoding, it's integrity could not be checked. Therefore i implemented a new operation mode "Info" (-i), which outputs many details about the stream (Audio format, codec version, preset, meta data structures...) and additionally performs an exhaustive check of the seek table (it jumps to any file position contained in the seek table, decodes the frame header at this position and checks if it is the right one).

Unfortunately this new check revealed one small error which is now fixed. Files encoded with beta 1 are still decodable without any problems, but later player plugins will have trouble to use their seek table. Sorry, but that's beta testing.

The new info option will not be part of the first final release, because i don't want to apply bigger modifications to the allready well tested beta code. It will be included into the next public version.

TAK 1.0 - Beta testing

Reply #71
Maybe I'm missing something but where is the decoder so I can actually play the files in Foobar2000?

TAK 1.0 - Beta testing

Reply #72
There is none, you cannot play TAK files at the moment.

This is beta1, simply for testing encoding and decoding to WAVE.
I'm on a horse.


TAK 1.0 - Beta testing

Reply #74
[deleted]