Hydrogenaudio Forums

Lossless Audio Compression => Lossless / Other Codecs => Topic started by: TBeck on 2009-09-20 21:51:50

Title: TAK 2.0 Development
Post by: TBeck on 2009-09-20 21:51:50
TAK 2.0 Development

Well, it looks as if i will be able to release TAK 2.0 this year... Therefore it's very unlikely that there will be an 1.1.3 release unless a bug fix for 1.1.2 is needed.

There are several good reasons for me to open a TAK 2.0 development thread now:

- Most of the new or improved algorithms have been implemented.
- Therefore i can now provide some performance data which will not differ very much from the final release.
- At some point i may need help from interested users to tune the new algorithms.
- It's easier for me to ask for help or opinions if there is already an appropriate thread.
- With this thread started and my goal stated, i have to stop digging even deeper for further improvements...

What to expect from V2.0:

It will introduce a new codec, but no new features from my todo list. The new codec will be no revolution, but an evolution. When i arrived at hydrogen and published first evaluation versions of YALAC, i was focussing more on maximum compression than on maximum speed. But it didn't take long to get aware of what users liked most of YALAC/TAK: It should be as fast as the fastest existing codecs but provide significantly higher compression. Despite of this clear mission statement, it took quite long for me to fully follow this order. Probably the reduction of the maximum predictor count from 256 to 160 in TAK 1.1.0 marks the last step of this process.

The goal of the TAK 2.0 development can be defined as follows:

Get the most bang for the buck. More specifically:

- Try to make it faster.
- Try to make it stronger (compression wise) but without sacrificing (decoding-) speed.

With these constraints present, i haven't managed to perform something revolutionary. But please don't forget, that many users already regarded the efficiency of the TAK 1.x line as revolutionary...

Ok, now for some data. First the results of my primary sample set. It consists of 46 files taken from rarewares.org (http://www.rarewares.org/test_samples/), combined in one file. This set has proven (in my evaluations) to stress lossless codecs in many different ways and it's my primary choice if i want a quite representative result covering different genres of music. Surely, i could think of an even more representative set, but this one is nice, because it's publically available.

Here we go with a comparison of V1.1.2 and 2.0:

Test system: AMD Sempron 2,2 GHz

Code: [Select]
Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                          
-p0     58.83   58.78   0.05   264.10   278.26    5.36%   283.52   293.90    3.66%
-p0e    58.67   58.34   0.33   200.29   146.38  -26.92%   284.84   293.52    3.05%
-p0m    58.53   58.27   0.26    88.43    85.90   -2.86%   285.66   293.83    2.86%
-p1     57.98   57.90   0.08   193.18   199.29    3.16%   275.80   285.30    3.44%
-p1e    57.86   57.59   0.27   123.55   114.51   -7.32%   276.19   285.75    3.46%
-p1m    57.73   57.46   0.27    64.98    63.13   -2.85%   276.48   286.04    3.46%
-p2     57.07   56.95   0.12   131.39   133.28    1.44%   250.36   259.74    3.75%
-p2e    56.89   56.80   0.09    81.92    85.53    4.41%   250.04   259.36    3.73%
-p2m    56.78   56.68   0.10    44.80    43.25   -3.46%   249.90   261.00    4.44%
-p3     56.52   56.43   0.09    55.97    61.21    9.36%   190.88   220.69   15.62%
-p3e    56.44   56.33   0.11    43.16    41.62   -3.57%   188.53   219.72   16.54%
-p3m    56.37   56.25   0.12    25.44    23.66   -7.00%   187.45   219.03   16.85%
-p4     56.16   56.10   0.06    32.07    32.94    2.71%   166.89   176.79    5.93%
-p4e    56.10   56.01   0.09    18.57    26.34   41.84%   166.27   176.15    5.94%
-p4m    56.07   55.95   0.12    17.81    13.81  -22.46%   166.47   177.37    6.55%

Compression is relative to the original file size, Enco- and Deco-Speed expressed as multiple of real time.

Some remarks:

- Faster decoding for any preset, especially for -p3x.
- Faster encoding for any basic preset (without additional evaluation level).
- Some compression improvements for any preset.
- -p4m is now compressing a bit stronger than any TAK version before, and this despite the reduction of the maximum predictor count from 256 to 160 in V1.1.0.
- Nice improvement of 0.12 percent for the default preset.
- Highest improvements for p0e/p0m,p1e/p1m, which -while only using 4 respectively 12 predictors- will easily (on average) outperform FLAC -8 with 12 predictors.

Now only the basic presets:

AMD Sempron 2.2 GHz
                                                                       
Code: [Select]
Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                        
-p0     58.83   58.78   0.05   264.10   278.26    5.36%   283.52   293.90    3.66%
-p1     57.98   57.90   0.08   193.18   199.29    3.16%   275.80   285.30    3.44%
-p2     57.07   56.95   0.12   131.39   133.28    1.44%   250.36   259.74    3.75%
-p3     56.52   56.43   0.09    55.97    61.21    9.36%   190.88   220.69   15.62%
-p4     56.16   56.10   0.06    32.07    32.94    2.71%   166.89   176.79    5.93%
-p4m    56.07   55.95   0.12    17.81    13.81  -22.46%   166.47   177.37    6.55%

Some speed differences on a Intel Pentium Dual Core 2 GHz:

Code: [Select]
                                                                        
Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                        
-p0     58.83   58.78   0.05   261.83   271.77    3.80%   298.67   314.33    5.24%
-p1     57.98   57.90   0.08   201.75   205.87    2.04%   291.72   306.68    5.13%
-p2     57.07   56.95   0.12   146.14   144.41   -1.18%   262.58   272.50    3.78%
-p3     56.52   56.43   0.09    64.64    68.20    5.51%   214.36   240.91   12.39%
-p4     56.16   56.10   0.06    36.90    36.37   -1.44%   193.97   206.40    6.41%
-p4m    56.07   55.95   0.12    21.07    15.76  -25.20%

Compression advantage for other bit depths than 16:

Code: [Select]
       24 Bit                          8 Bit
       44-48 KHz  96 KHz   192 KHz     8-44 KHz              
-p0     0.02        0.05    0.31        1.24
-p1     0.10        0.16    0.41        1.32
-p2     0.17        0.30    2.07        1.41
-p3     0.03        0.01    1.03        1.48
-p4    -0.02       -0.08    1.03        1.00
-p4m    0.01       -0.03    1.12        0.99

Some remarks:

- Currently there is some regression for 96 KHz. The new algorithms need some fine-tuning.
- In my evaluations only Optimfrog can beat TAK's compression of 192 KHz sample data.

Last but not least some results for LossyWav:

Code: [Select]
Quality Preset  Compression             
                1.1.2   2.0     Win
                                
-q00    -p2m    19.37   19.05   0.32
        -p4m    19.43   19.04   0.39
-q25    -p2m    26.38   26.12   0.26
        -p4m    26.40   26.11   0.29
-q50    -p2m    32.34   32.10   0.24
        -p4m    32.36   32.08   0.28

And no, that's not the results of the dedicated LossyWav codec i intend to develop later.

Well, i hope you are not disappointed!

  Thomas
Title: TAK 2.0 Development
Post by: 2E7AH on 2009-09-21 02:28:28
Well, it looks as if i will be able to release TAK 2.0 this year...

I love this codec because of it speed to encode and decode (at least in foobar) and prefer it then FLAC when some compatibility isn't in question.
I didn't read the tables but I guess it will still be fast enc/dec
Only the date?

[edit] just forgot MD5 -

[edit2] I just saw them quickly, and encoding speed graph is little strange. Default -p3 just jumps from the neighbors (why?) Also -p2 looks like a candidate for default setting.

[edit3] well, -p2 is default, I somehow thought it was -p3, nevermind
Title: TAK 2.0 Development
Post by: gib on 2009-09-21 03:54:50
Great news, Thomas!  Nice to see my lossless codec of choice is continuing to be improved upon.

Thank you.
Title: TAK 2.0 Development
Post by: Alexxander on 2009-09-21 08:46:19
It's closed source until the day it isn't anymore, and maybe that day will never come. Pressuring with each release by remembering extensively the words said looong ago seems to me a desperate scream to get a personal desire fulfilled.

I do agree with the fact that the evolution of TAK doesn't help getting TAK more widely adopted (lack of specifications for developers, lack of multiplatform, not being open source, lack of features FLAC does have, etc.). In fact, I have the impression that with each TAK release less people participate (posting & testing).

But then, Thomas can do what he want, it's his work.

Back to TAK 2.0. Great to see the improvements, how did you got them? You rewrote the encoder and decoder? Do you think there's a room to make it even faster?

When it's out I will compare it to FLAC and TAK 1.1.2, if I have time by that time
Title: TAK 2.0 Development
Post by: Destroid on 2009-09-21 14:54:05
I am glad to see a new milestone being set for TAK. However, users fixated on the topic of code availability is becoming very irritating. I think certain members should quit threatening the developer by declaring TAK unusable because of their narrow philosophy. For them I say, "Go write your own open-source lossless program," if only just to get an idea of what it takes to make a codec that is different and better.

Kicking aside the soapbox (and getting back to point of main discussion), I also wanted to throw out other questions: Is TAK looking at adding multichannel support sometime in version 2.x? What other additional tools might 2.x be included in the SDK that programmers can utilize?

Title: TAK 2.0 Development
Post by: IgorC on 2009-09-22 07:35:11
Thomas, congratulation on new step of evolution of your codec.
I see that all standard presets -q0 ...-q4 have received enco- and deco- speed improvements while sub-presets "extra" and "max" have gone to more asymmetric balance  (slower encoding, faster decoding and better compression). It's great.  The CD ripping (EAC) is more slower than encoding to -p0m or -p1m and many time there is a lot of time to encode without rush while decoding speed is important for playback and converting to the lossy formats. I wouldn't mind see even more asymmetric speeds. Maybe new profile ( -p1u ultra?) could bring more compression and/or faster decoding at cost of something like 2-2.5x slower speed comparing to -p0m and -p1m (which is still comparable with speed of FLAC -8)?
Most of FLAC users encode to -8 level. http://www.hydrogenaudio.org/forums/index....c=58731&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=58731&hl=)

Likewise those numbers are ideal as they don't consider I/O bottleneck. The difference between speeds of -p1 and -p1m is considerably smaller in real scenario.
Title: TAK 2.0 Development
Post by: TBeck on 2009-09-23 23:35:22
[edit2] I just saw them quickly, and encoding speed graph is little strange. Default -p3 just jumps from the neighbors (why?)

I have replaced something i called "PreFilter", which was present for -p3x and -p4x. The new approach is a lot faster for -p3x.

In fact, I have the impression that with each TAK release less people participate (posting & testing).

I think it would be quite easy for me to evoke more responses, if i would release evaluation versions and would frequently ask for opinions on some tuning deceisions like i did in the YALAC-days. But there is far less need for me to do this, because i now know a lot more about the nature of many of TAK's compression techniques and their effect on even extraordinary audio files. This because of the great support of some past testers of YALAC and TAK.

And as i alreaday wrote: The speed-compression-efficiency of the first YALAC/TAK-versions has been regarded as revolutionary by quite some people, but now there is only some -less exciting- evolution happening. Something users possibly are expecting.

Back to TAK 2.0. Great to see the improvements, how did you got them? You rewrote the encoder and decoder?

What i did:

- Further improvements of the bitcoder for the final residuals. FLAC for instance is using fast but not very efficient rice codes. Monkey's Audio is using range coding: more efficient but noticeable slower. TAK is using codes that are nearly as fast as rice codes and nearly as efficient as range coding. TAK 2 further improves the bitcoder efficiency for very low amplitude signals (here conventional rice codes are very inefficient). You can see a significant effect for 8-bit data, low amplitude parts of for instance classical music and for LossyWav-files.

- I have implemented a simplified variant of the advanced channel decorrelation algorithm used by presets p2 and above. It's about as fast as simple Mid/Side-encoding used by -p0/-p1 in earlier versions. Because of it's speed, it was now possible to bring a good part of the advanced channel decorrelation to the presets p0/p1.

- Presets -p3 and -p4 of the TAK 1.x line are sending each sample through something i called "PreFilter". It's primary effect was the reduction of the space requirements for the storage of the predictor coefficients. Another approach i tried in the YALAC-days was the storage of the predictors as parcor coefficients. Unfortunately this approach was quite slow for high predictor orders, because encoder and decoder have to transform parcor into filter coefficients and this operation required very slow 32*32=64 bit multiplications and the count of those multiplications is growing exponentially with the predictor count.

But now i have managed to modify the algorithm in a way, that it needs only 16*32=32 bit multiplications, what is faster than the PreFilter, if you don't go for extremely high predictor counts > 256.

- I have performed a lot of quite small optimizations to simplify the algorithms and to gain a bit of speed.

Do you think there's a room to make it even faster?

I am quite sure i can speed up encoding for the encoder evaluation levels e and m. Regarding decoding, i have my doubts.

When it's out I will compare it to FLAC and TAK 1.1.2, if I have time by that time

Fine!

Kicking aside the soapbox (and getting back to point of main discussion), I also wanted to throw out other questions: Is TAK looking at adding multichannel support sometime in version 2.x? What other additional tools might 2.x be included in the SDK that programmers can utilize?

Yes, i really would like to implement multichannel support. But it's not really easy to do, because i first want to evaluate the possibilities to take advantage of channel correlations of such files.

My todo list for  the SDK:

- Encoding capabilitities.
- Unicode support.
- Access to other meta data than APEv2-Tags.

Maybe new profile ( -p1u ultra?) could bring more compression and/or faster decoding at cost of something like 2-2.5x slower speed comparing to -p0m and -p1m (which is still comparable with speed of FLAC -8)?

Well, i could implement a brute-force-approach, which possibly would increase compression by 0.05 to 0.1 percent, but might reduce the encoding speed by a factor of 10 to 50...

Likewise those numbers are ideal as they don't consider I/O bottleneck. The difference between speeds of -p1 and -p1m is considerably smaller in real scenario.

How true. And p1 is probably only a tiny bit slower than p0. For now i have modified p1 to be nearly as strong p1e while loosing only very little encoding speed (<= 10 %). I will post some results later.

  Thomas
Title: TAK 2.0 Development
Post by: TBeck on 2009-10-26 23:13:25
Updated results for Version 2.0 Eval 03

Yeah, there is some progress...

Results for my primary sample set:

Code: [Select]
AMD Sempron 2.2 GHz                                                                     
                                                                        
Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                        
-p0     58.83   58,74   0,09   264.10   290,95   10,17%   283.52   294,06    3,72%
-p1     57.98   57,66   0,32   193.18   192,05   -0,58%   275.80   285,83    3,64%
-p2     57.07   56,93   0,14   131.39   135,30    2,98%   250.36   258,87    3,40%
-p3     56.52   56,40   0,12    55.97    61,18    9,31%   190.88   218,93   14,70%
-p4     56.16   56,08   0,08    32.07    33,83    5,49%   166.89   176,69    5,87%
-p4m    56.07   55,93   0,14    17.81    14,43  -18,98%

Intel Pentium Dual Core 2 GHz                                                                  
                                                                        
Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                        
-p0     58.83   58,74   0,09   261.83   284,54    8,67%   298.67   311,81    4,40%
-p1     57.98   57,66   0,32   201.75   201,09   -0,33%   291.72   304,66    4,44%
-p2     57.07   56,93   0,14   146.14   148,89    1,88%   262.58   271,37    3,35%
-p3     56.52   56,40   0,12    64.64    69,37    7,32%   214.36   240,48   12,19%
-p4     56.16   56,08   0,08    36.90    37,86    2,60%   193.97   205,73    6,06%
-p4m    56.07   55,93   0,14    21.07    16,68  -20,84%

Compression is relative to the original file size, Enco- and Deco-Speed expressed as multiple of real time.

Some results for LossyWav:

Code: [Select]
Quality Preset  Compression                  
                1.1.2   2.0     Win  
                                    
-q00    -p2m    19.37   18,92   0,45
-q25    -p2m    26.38   26,00   0,38
-q50    -p2m    32.34   31,97   0,37

Some remarks:

- Again some tiny compression improvements.
- Again -p0 is encoding significantly faster. Encoding is now nearly as fast as decoding on the Sempron.
- I have deceided, to make -p1 stronger, therefore it's now encoding a bit slower.

  Thomas

Title: TAK 2.0 Development
Post by: _m²_ on 2009-10-27 14:49:05
Any news about release date?
Title: TAK 2.0 Development
Post by: SokilOff on 2009-10-27 20:28:51
Updated results for Version 2.0 Eval 03


Congrats, Thomas. Great to hear about TAK 2 development progress. Looks really promising.
According to your todo list, there are:
Quote
- Encoding capabilitities.
- Unicode support.
- Access to other meta data than APEv2-Tags


Do you plan to add also brute-force mode before public release ? This mode could still be very useful despite slow encoding speed.
And yes, is there any estimated release date for beta version ?
Title: TAK 2.0 Development
Post by: lvqcl on 2009-10-27 21:19:40
Quote
This mode could still be very useful despite slow encoding speed.

Monkey's Audio, OptimFROG, LA - they all have better compression ratio than TAK -pMax. But they aren't particularly popular.

There are no DAPs that support TAK. Winamp plugin doesn't support tagging. And yes, no Unicode support. (what else? embedded cover art?)  I think it's much more important for TAK.
Title: TAK 2.0 Development
Post by: SokilOff on 2009-10-27 21:43:14
Quote
This mode could still be very useful despite slow encoding speed.

Monkey's Audio, OptimFROG, LA - they all have better compression ratio than TAK -pMax. But they aren't particularly popular.

TAK has one huge advantage over those codecs - very fast decompression.
I can easily live with slower encoding in 'ultra-heavy-brute-force mode', since I compress each album just once. Fast decoding speed stays anyway. So if such mode is possible - why not to have it ?

Quote
There are no DAPs that support TAK. Winamp plugin doesn't support tagging. And yes, no Unicode support. (what else? embedded cover art?)  I think it's much more important for TAK.

Yep. But it seems all these features are already on Thomas's todo-list.
Title: TAK 2.0 Development
Post by: nZero on 2009-10-27 22:10:36
Quote
This mode could still be very useful despite slow encoding speed.

Monkey's Audio, OptimFROG, LA - they all have better compression ratio than TAK -pMax. But they aren't particularly popular.

TAK has one huge advantage over those codecs - very fast decompression.


2 advantages.  Monkey's and FROG have honestly really dumb names that inhibit adoption.  I remember when I first found out about lossless encoding, the only 2 codecs I was aware of were FLAC and Monkey's.  Before being aware of the other advantages and disadvantages of each (hey, it's lossless after all, I can switch anytime with no quality loss) I chose FLAC because it had a descriptive name with a cool high-tech acronym and file extension, as opposed to littering my hard drive with APE files.  TAK is even better because it sounds like the villain from Stephen King's Desperation.

LA reminds me of vintage Roland synthesizers, which is good, but... I don't even know where to get tools for that which would work on my system anymore.
Title: TAK 2.0 Development
Post by: sauvage78 on 2009-10-27 23:47:26
_m²_ & SokilOff :

According to TBeck, Tak 2.0 is supposed to be released before the end of this year.
Needless to say, I'm looking forward for its license.
Title: TAK 2.0 Development
Post by: greynol on 2009-10-28 03:49:07
Needless to say, I'm looking forward for its license.

Now let's keep this on-topic, sauvage78.  Don't push your luck.
Title: TAK 2.0 Development
Post by: ChronoSphere on 2009-11-04 21:13:35
- Unicode support.
What do you mean by that? I have a couple of TAK files with Japanese tags and they show up fine on non-JP systems...
Do you mean unicode support for the command line encoder? (Well, file names fed to the encoder)
Title: TAK 2.0 Development
Post by: Zarggg on 2009-11-04 23:59:27
I believe he does... which is something I've been greatly missing.
Title: TAK 2.0 Development
Post by: TBeck on 2009-11-05 21:36:50
Any news about release date?

"Before christmas" seems to be a reasonable date for the final release. But i don't know, how long the beta testing phase will last. Because i have modified so much code, i may need quite a lot of positive feedback (verification of proper encoding and decoding) before i will feel safe enough to perform a final release... Currently i am not too far away from a beta release (notice: i am calling it "beta" instead of "alpha" because i am quite confident, that it is working well).

Do you plan to add also brute-force mode before public release ? This mode could still be very useful despite slow encoding speed.

A complete brute-force-approach would mean to fully encode any combination of all parameter variations of TAK's filters and to choose the best output. This would be very slow!

If i am right, there is something like a "-secret-totally-impracticable" setting for FLAC, that does something like this. I don't know, how significant the effect on the FLAC compression ratio is, but for TAK it would be very small. And extremely slow!

TAK is using more filters and more variations of their parameters than FLAC does. If you want to calculate the complexity of a complete encode, you have (simplified!) to multiply the possible parameter variations (N) of the filters (fx): N(f1) * N(f2) * N(f3) ... It's a bit like putting 2 pieces of grain on the first field of a checkerboard, 4 on the next, then 8 and so on.

But during the development of the codec i have often calculated the maximum possible compression of selected filter combinations by brute-force and than worked very hard to find a much faster heuristic to estimate the optimal parameters. Surprisingly -after many hundreds of hours- i have always found a heuristic that was extremely close to the brute-force-approach.

Summary: A real brute-force-approach may encode about 50 to 100 times slower and gain less than 0.1 percent of compression.

TAK is even better because it sounds like the villain from Stephen King's Desperation.

Good point! I have read the alternative story based upon the TAK character, called "Regulators". I liked it very much and this was one important reason to call the codec TAK.

- Unicode support.
What do you mean by that? I have a couple of TAK files with Japanese tags and they show up fine on non-JP systems...
Do you mean unicode support for the command line encoder? (Well, file names fed to the encoder)

I believe he does... which is something I've been greatly missing.

Yes!

  Thomas

Title: TAK 2.0 Development
Post by: Polar on 2009-11-06 10:26:27
If i am right, there is something like a "-secret-totally-impracticable" setting for FLAC, that does something like this. I don't know, how significant the effect on the FLAC compression ratio is, but for TAK it would be very small. And extremely slow!
(...)
Summary: A real brute-force-approach may encode about 50 to 100 times slower and gain less than 0.1 percent of compression.
Ditto with FLAC.  Cf. Omion's test (http://people.ucsc.edu/~rswilson/flactest/).  Even though it only covers decoding speed, you can clearly see that FLAC's --super-secret-totally-impractical-compression-level yields only nominally, neglectably better compression than -8.  Even if I can't provide hard figures, I can tell by personal experience that it's indeed totally impractically slow at encoding.  Even on a fast machine, encoding occurs at real-time or hardly better.

Apart from that, let me still repeat that - imho - it's a shame that you ditched the -5 compression setting, making -4 the highest preset.  TAK's overall speed and efficiency are its well-known advantages, and to me, too, -5 was far from unreasonably slow at encoding, that is, the extra compression was well worth the wait.  Just a reminder: almost 2/3 of FLAC's userbase (at least the ones who voted in this poll (http://www.hydrogenaudio.org/forums/index.php?showtopic=58731)) use the maximum compression setting.  As TAK is a similar, asymmetric codec, I regret that by omitting -5, focus has shifted from the higher to the lower presets.
Title: TAK 2.0 Development
Post by: alvaro84 on 2009-11-06 11:04:56
Just a reminder: almost 2/3 of FLAC's userbase (at least the ones who voted in this poll (http://www.hydrogenaudio.org/forums/index.php?showtopic=58731)) use the maximum compression setting.


Just some interesting data, without too much significance:

I use FLAC at -8 but TAK at -p2m and enjoy its incredible encoding and near-flac decoding speed (well, officially it should be even faster but in fb2k it seems a bit different) while its compression is significantly better than flac -8.

Title: TAK 2.0 Development
Post by: _m²_ on 2009-11-06 13:49:44
Any news about release date?

"Before  christmas" seems to be a reasonable date for the final release. But i  don't know, how long the beta testing phase will last. Because i have  modified so much code, i may need quite a lot of positive feedback  (verification of proper encoding and decoding) before i will feel safe  enough to perform a final release... Currently i am not too far away  from a beta release (notice: i am calling it "beta" instead of "alpha"  because i am quite confident, that it is working well).

Thank you for the answer.

Apart from that, let me still repeat that - imho - it's a shame that you ditched the -5 compression setting, making -4 the highest preset.  TAK's overall speed and efficiency are its well-known advantages, and to me, too, -5 was far from unreasonably slow at encoding, that is, the extra compression was well worth the wait.  Just a reminder: almost 2/3 of FLAC's userbase (at least the ones who voted in this poll (http://www.hydrogenaudio.org/forums/index.php?showtopic=58731)) use the maximum compression setting.  As TAK is a similar, asymmetric codec, I regret that by omitting -5, focus has shifted from the higher to the lower presets.

I agree. For me disk space is much more valuable resource than CPU time - I do conversions in background anyway.
The slowest thing that I used (and at the limit of tolerability for me) was wavpack -hhx6... compared to it, p4m is a lightning and I'd be happy to trade some speed to shave a few kilos more.
Title: TAK 2.0 Development
Post by: Destroid on 2009-11-07 03:45:52
Quote
I'd be happy to trade some speed to shave a few kilos more.
  McGruff the crime dog says, "Kids, don't try this at the breakfast table!"  (couldn't resist, lol)

But I did have a serious thought: maybe -p3 and -p4 could get something more aggressive towards compression (i.e. -p3em and -p4em) with the disclaimer that, "This is going to compress reeeally slow for almost an extra 0.04% over -p#m. 
Title: TAK 2.0 Development
Post by: tEiS on 2009-11-08 17:47:16
is there any beta or so i could test already?

i'm interested in how good this codec will be
Title: TAK 2.0 Development
Post by: TBeck on 2009-11-10 23:51:27
Apart from that, let me still repeat that - imho - it's a shame that you ditched the -5 compression setting, making -4 the highest preset.  TAK's overall speed and efficiency are its well-known advantages, and to me, too, -5 was far from unreasonably slow at encoding, that is, the extra compression was well worth the wait.  Just a reminder: almost 2/3 of FLAC's userbase (at least the ones who voted in this poll (http://www.hydrogenaudio.org/forums/index.php?showtopic=58731)) use the maximum compression setting.  As TAK is a similar, asymmetric codec, I regret that by omitting -5, focus has shifted from the higher to the lower presets.


Apart from that, let me still repeat that - imho - it's a shame that you ditched the -5 I agree. For me disk space is much more valuable resource than CPU time - I do conversions in background anyway.
The slowest thing that I used (and at the limit of tolerability for me) was wavpack -hhx6... compared to it, p4m is a lightning and I'd be happy to trade some speed to shave a few kilos more.


Quote
I'd be happy to trade some speed to shave a few kilos more.
  McGruff the crime dog says, "Kids, don't try this at the breakfast table!"  (couldn't resist, lol)

But I did have a serious thought: maybe -p3 and -p4 could get something more aggressive towards compression (i.e. -p3em and -p4em) with the disclaimer that, "This is going to compress reeeally slow for almost an extra 0.04% over -p#m. 


Yes, the "focus has shifted from the higher to the lower presets". This was happening from the first presentation of YALAC (as TAK was named before it's first stable release). I have always been affected by user comments at hydrogen. Most users (who posted) wanted it to be as fast as possible. A significant amount of posters (not necessarily users...) kept emphasizing TAK's slower decoding speed than FLAC (especially when using foobar). Maybe i have taken this too serious... Possibly a lot of users indeed would welcome a bit stronger compression in exchange for a bit slower decoding. Possibly i should create a poll?

I have modified the codec to achieve more speed especially for the lower presets. Those modifications may lead to a more significant loss of decoding speed than in previous versions, if i again raise the maximum predictor order for -p4. I will have to try it.

is there any beta or so i could test already?

i'm interested in how good this codec will be

Only a bit better than before.

Currently i am modifying the decoder to be capable to decode files created by later, more sophisticated TAK encoders. Then i can prepare a beta release.

  Thomas
Title: TAK 2.0 Development
Post by: Destroid on 2009-11-11 07:11:06
Quote
Yes, the "focus has shifted from the higher to the lower presets". This was happening from the first presentation of YALAC (as TAK was named before it's first stable release). I have always been affected by user comments at hydrogen. Most users (who posted) wanted it to be as fast as possible.
I believe this is still true. Most current TAK users are interested in this still, especially the ones who use the default or lower settings.
Quote
A significant amount of posters (not necessarily users...) kept emphasizing TAK's slower decoding speed than FLAC (especially when using foobar).
Decoding speed is definitely a priority, although in this age of computing I think current TAK decoding speeds will be fine for current portables (FLAC decoding speed may be a bit overkill nowadays as compared to 5 years ago.
Quote
Possibly a lot of users indeed would welcome a bit stronger compression in exchange for a bit slower decoding. Possibly i should create a poll?
Why not? I predict the people who use -p3 and up would be in favor of better compression, whereas -p2 users want the fastest encode/decode performance. I am not sure how much die-hard users who want the best compression would care about slower decompression speed. As far as -p3 settings and higher I think the poll might as well ask if decode speed is a priority (i.e. DAW and HTPC don't need to be battery-saving).
Title: TAK 2.0 Development
Post by: Destroid on 2009-11-11 12:38:35
And no, that's not the results of the dedicated LossyWav codec i intend to develop later.
Pardon the double-post, but I somehow overlooked this part of the OP. Thomas, can you elaborate about this?
Title: TAK 2.0 Development
Post by: SokilOff on 2009-11-11 14:51:04
Yes, the "focus has shifted from the higher to the lower presets". This was happening from the first presentation of YALAC (as TAK was named before it's first stable release). I have always been affected by user comments at hydrogen. Most users (who posted) wanted it to be as fast as possible. A significant amount of posters (not necessarily users...) kept emphasizing TAK's slower decoding speed than FLAC (especially when using foobar). Maybe i have taken this too serious... Possibly a lot of users indeed would welcome a bit stronger compression in exchange for a bit slower decoding. Possibly i should create a poll?


I would say, the strongest side of TAK is efficiency - size/speed rate. It's great to have a codec that has presets with encoding/decoding speed comparable to FLAC and presets with compression levels comparable to APE High.

So I think you shouldn't have hard time choosing "speed or compression". Lower presets are for better speed, higher ones - for better compression
Title: TAK 2.0 Development
Post by: TBeck on 2009-11-12 13:46:01
Quote
Possibly a lot of users indeed would welcome a bit stronger compression in exchange for a bit slower decoding. Possibly i should create a poll?
Why not? I predict the people who use -p3 and up would be in favor of better compression, whereas -p2 users want the fastest encode/decode performance. I am not sure how much die-hard users who want the best compression would care about slower decompression speed. As far as -p3 settings and higher I think the poll might as well ask if decode speed is a priority (i.e. DAW and HTPC don't need to be battery-saving).

I would say, the strongest side of TAK is efficiency - size/speed rate. It's great to have a codec that has presets with encoding/decoding speed comparable to FLAC and presets with compression levels comparable to APE High.

So I think you shouldn't have hard time choosing "speed or compression". Lower presets are for better speed, higher ones - for better compression

I think you both are right: Let's regard -p0 to -p2 as the speed settings focussing on maximum decoding speed and -p3/-p4 as the power settings which sacrifice some more decoding speed for a bit better compression, even if the proportion may get somewhat insane efficiencywise.

I have alreday raised the maximum predictor count to 256 and will post some results soon.

And no, that's not the results of the dedicated LossyWav codec i intend to develop later.
Pardon the double-post, but I somehow overlooked this part of the OP. Thomas, can you elaborate about this?

I am quite confident that i can modify the codec to achieve at least 1 percent better compression for LossyWav-files. Furthermore it may perform significantly better when compressing files with sections where LossyWav hasn't removed any bits and the efficiency falls behind that of a pure lossless encode.

Earlier i wanted to integrate those modifications into the TAK 2.0 codec, but now i plan to create a dedicated codec (the file format supports up to 64 different codecs) . That's a bit easier to do and doesn't stall the 2.0 release.

  Thomas


Title: TAK 2.0 Development
Post by: Bylie on 2009-11-13 07:56:39
Thomas,

I'm curious on your views of GPGPU and how it could be beneficial for TAK. Do you have any plans to look at this? Mind you I only have a general understanding of how this stuff works but it seems to be getting more interesting now things are going to get more standardized.

The following thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=64628&hl=flacuda) also handles this and even has a proof of concept FLAC implementation based on CUDA.
Title: TAK 2.0 Development
Post by: TBeck on 2009-11-13 21:43:52
I have alreday raised the maximum predictor count to 256 and will post some results soon.

Here they are:

Code: [Select]
AMD Sempron 2.2 GHz                                                                     
                                                                        
Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                        
-p3     56.52   56.32   0.20    55.97    55.53   -0.79%   190.88   210.42   10.24%
-p4     56.16   55.96   0.20    32.07    25.10  -21.73%   166.89   148.77  -10.86%
-p4m    56.07   55.80   0.27    17.81     9.34  -47.56%


Intel Pentium Dual Core 2 GHz                                                                  
                                                                        
Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                        
-p3     56.52   56.32   0.20    64.64    63.45   -1.84%   214.36   234.40    9.35%
-p4     56.16   55.96   0.20    36.90    28.64  -22.38%   193.97   178.50   -7.98%
-p4m    56.07   55.80   0.27    21.07    10.38  -50.74%

Compression is relative to the original file size, Enco- and Deco-Speed expressed as multiple of real time.

-p3 is now using 96 instead of 80, -p4 256 instead of 160 predictors.

Some results for other sample rates and bit depths:       

Code: [Select]
                   Preset  Compression         
                           1.1.2     2.0 Eval  Win
                    
24 bit /  44 khz    -p4m   56.78     56.67     0.11
24 bit /  96 khz    -p4m   50.69     50.61     0.08
24 bit / 192 khz    -p4m   44.86     43.76     1.10
  8 bit             -p4m   38.23     37.08     1.15


Decoding of -p4 is still at least 150 times faster than realtime, what should be acceptable.
Title: TAK 2.0 Development
Post by: zombiewerewolf on 2009-11-14 00:06:04
Code: [Select]
Intel Pentium Dual Core 2 GHz                                                                   
                                                                        
Preset  Compression            Enco-Speed
        1.1.2   2.0     Win    1.1.2    2.0       Win
                                                                        
-p4     56.16   55.96   0.20    36.90    28.64  -22.38%
-p4m    56.07   55.80   0.27    21.07    10.38  -50.74%

That's quite an impressive improvement actually, when compare old p4m to new p4.
Compression ratio from both presets are about the same (0.11 difference) but encoding speed improves 35.93%.
Title: TAK 2.0 Development
Post by: yerma on 2009-11-14 12:13:28
Compression ratio from both presets are about the same (0.11 difference) but encoding speed improves 35.93%.
One of us is getting it wrong here. If I understand correctly, compression time drops from 20 times real time to 10 times real time, which means the -p4m compression will take twice as long with TAK 2.0...

Edit: Now I get it, you compared old p4m with the new p4. Sorry, my bad...
Title: TAK 2.0 Development
Post by: TBeck on 2009-11-15 23:45:02
I'm curious on your views of GPGPU and how it could be beneficial for TAK. Do you have any plans to look at this? Mind you I only have a general understanding of how this stuff works but it seems to be getting more interesting now things are going to get more standardized.

The following thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=64628&hl=flacuda) also handles this and even has a proof of concept FLAC implementation based on CUDA.

I am always interested into new opportunities to optimize TAK and GPGPU is no exception. It's definitely possible to utilize GPGPU for encoding, but i have no practical experience yet. And it will take some time until i will try it. The most important reason:

I am very concerned about the reliability of the encoder. I am worried about failures of the GPU-memory resulting in unrecognized encoder errors. Unless i find a trustable study which shows that GPU-memory isn't failing more often than system memory or unless more GPU's come with ECC for their memory, i really dont wan't to take a risk.

Currently i would prefer to add multicore support to the encoder. It's easier to do, will probably result in a larger speed gain (especially on systems with slow GPU's) and i am feeling more safe regarding the reliability.
Title: TAK 2.0 Development
Post by: Bylie on 2009-11-16 07:15:04
...
I am very concerned about the reliability of the encoder. I am worried about failures of the GPU-memory resulting in unrecognized encoder errors. Unless i find a trustable study which shows that GPU-memory isn't failing more often than system memory or unless more GPU's come with ECC for their memory, i really dont wan't to take a risk.
...

Sounds perfectly reasonable to me and it might be the safest approach for new tech, although I think nVidia is actually going to include ECC into their new Fermi architecture. I actually don't know if it will also be made available in normal consumerproducts as marketsegmentation sometimes dictates otherwise. Thanks for the answer and nice to hear that you're looking into multicore CPU support.
Title: TAK 2.0 Development
Post by: skamp on 2009-11-16 07:29:21
Currently i would prefer to add multicore support to the encoder. It's easier to do, will probably result in a larger speed gain (especially on systems with slow GPU's) and i am feeling more safe regarding the reliability.

As I've had to deal with bit-flip errors with RAM myself, I understand your concern. I also recall reading an article recently about how GPU RAM errors weren't considered critical since one wrong pixel in a video game doesn't matter much.
Here's the thing with lossless codecs though: you can always compare the encode to the source. So, I'm not too worried.

As for the multi-core CPU vs. GPU: think of laptops, HTPCs and generally lower-end computers. They're more likely to have only one or two CPU cores, and a GPU. My new laptop is something of a special case, but a GPU-enabled encoder would run much faster on it: it has an Intel Atom N270 CPU (one core, two threads) and an NVIDIA Ion GPU (GeForce 9400M). The latter's benefit becomes clear when playing high-definition videos, where CPU usage stays below 20% (most of the time, below 10%).

Now, the question is: what machines would benefit most from encoding speed improvements on TAK, which is already quite fast? In my case, my Atom/Ion netbook would certainly benefit more from a GPU-enabled encoder than my quad-core AMD Phenom PC would benefit from a multi-threaded implementation, especially since I can already encode multiple files in parallel.
Title: TAK 2.0 Development
Post by: TBeck on 2009-11-16 14:17:03
I am very concerned about the reliability of the encoder. I am worried about failures of the GPU-memory resulting in unrecognized encoder errors. Unless i find a trustable study which shows that GPU-memory isn't failing more often than system memory or unless more GPU's come with ECC for their memory, i really dont wan't to take a risk.

A quick search with google revealed this interesting page: MemtestG80: A Memory and Logic Tester for NVIDIA CUDA-enabled GPUs (https://simtk.org/home/memtest). It contains a link to an pdf of the study "Hard Data on Soft Errors: A Large-Scale Assessment of Real-World Error Rates in GPGPU". I had no time to really read or even critically rate the article, but here is an excerpt from the conclusions:

Quote
"We have presented the first large-scale study of error rates in GPGPU hardware, conducted over more than 20,000 GPUs on the Folding@home distributed computing network. Our control experiments on consumer-grade and dedicated-GPGPU hardware in a controlled environment found no errors. However, our large-scale experimental results show that approximately two-thirds of tested cards exhibited a pattern-sensitive susceptibility to soft errors in GPU memory or logic, confirming concerns about the reliability of the installed base of GPUs for GPGPU computation. We have further demonstrated that this nonzero error rate cannot be adequately explained by overclocking or time of day of execution (a proxy for ambient temperature). However, it appears to correlate strongly with GPU architecture, with boards based on the newer GT200 GPU having much lower error rates than those based on the older G80/G92 design. While we cannot rule out user error, misconfiguration on the part of Folding@home donors, or environmental effects as the cause behind nonzero error rates, our results strongly suggest that GPGPU is susceptible to soft errors under normal conditions on non-negligible timescales."


As I've had to deal with bit-flip errors with RAM myself, I understand your concern. I also recall reading an article recently about how GPU RAM errors weren't considered critical since one wrong pixel in a video game doesn't matter much.
Here's the thing with lossless codecs though: you can always compare the encode to the source. So, I'm not too worried.

Yes, this seems to be the way to go. I forgot about TAK's verify option, which will decode each frame after encoding and compare the output with the original. Since decoding is so fast, this could be performed by the CPU without sacrificing too much encoding speed.

Now, the question is: what machines would benefit most from encoding speed improvements on TAK, which is already quite fast? In my case, my Atom/Ion netbook would certainly benefit more from a GPU-enabled encoder than my quad-core AMD Phenom PC would benefit from a multi-threaded implementation, especially since I can already encode multiple files in parallel.

I agree, this is the question... The answer may require a lot of testing of real implementations. Currently i don't want to put too much effort into it, but i will keep an eye on evaluations of GPU-implementations of similar algorithms as TAK is using.
Title: TAK 2.0 Development
Post by: _m²_ on 2009-11-16 14:35:19
I'm curious on your views of GPGPU and how it could be beneficial for TAK. Do you have any plans to look at this? Mind you I only have a general understanding of how this stuff works but it seems to be getting more interesting now things are going to get more standardized.

The following thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=64628&hl=flacuda) also handles this and even has a proof of concept FLAC implementation based on CUDA.

I am always interested into new opportunities to optimize TAK and GPGPU is no exception. It's definitely possible to utilize GPGPU for encoding, but i have no practical experience yet. And it will take some time until i will try it. The most important reason:

I am very concerned about the reliability of the encoder. I am worried about failures of the GPU-memory resulting in unrecognized encoder errors. Unless i find a trustable study which shows that GPU-memory isn't failing more often than system memory or unless more GPU's come with ECC for their memory, i really dont wan't to take a risk.

Currently i would prefer to add multicore support to the encoder. It's easier to do, will probably result in a larger speed gain (especially on systems with slow GPU's) and i am feeling more safe regarding the reliability.

I'm glac that there are more people like me who are concerned about GPU encoding correctness and it's great that you're one of them.
I've seen somewhere a study done with NVIDIA collaboration that showed that GPU memory errors were indeed an issue...as well as some silicon issues, IIRC GPUs with few processing units damaged passing validation.

But there's a simple solution: Encode on GPU, move to the main memory and then use CPU to verify the results. Retry (on CPU?) in case of a failure.
It should warranty correctness with overhead low enough to still provide a great speed boost.
Verification that parameters are best chosen would be infeasible, which would hurt compression ratio somehow, but I think that the speed improvement is well worth it...especially that there are reliable GPUs from NVIDIA on the way.
Title: TAK 2.0 Development
Post by: TBeck on 2009-11-20 07:58:01
Now i am preparing a first release... Probably i will call it a beta.

I am looking at code i have written weeks or months ago to find errors. Today i caught one. Fortunately: Even my quite exhaustive script-based test set possibly wouldn't have caught it because it was based upon very rare conditions. Mathematically possible but extremely rare in practice.

Addditionally i am feeding the encoder with random data to test the decoder regarding features of the encoder, which will later be implemented. I want to be sure, that the TAK 2.0 decoder will decode files created by later, more sophisticated encoders without any problems.

Some bad news for some power hungry users: For now i have again reduced the maximum predictor count from 256 to 160. But the decoder will be laid out to support even up to 320 predictors; this way i am able to add a really insane preset -p5 later ( if i want) and the files will be decodable by the V 2.0 decoder.

I hope, this wasn't boaring. I will release a first version as soon as i am feeling confident enough about the reliability of the new codec.

  Thomas
Title: TAK 2.0 Development
Post by: GeSomeone on 2009-11-20 15:37:24
Some bad news for some power hungry users: For now i have again reduced the maximum predictor count from 256 to 160.
  Thomas

Not boring at all, about the max predictors, would a value like 192 be a reasonable compromise?
Title: TAK 2.0 Development
Post by: TBeck on 2009-11-25 00:15:06
Not boring at all, about the max predictors, would a value like 192 be a reasonable compromise?

Well, the problem is, how you define reasonable. For users looking for optimum efficiency (speed/compression), the compression advantage would be too small to justify slower encoding and decoding. And users looking for maximum compression would still ask for more...

For V2.0 i will limit the predictor count to 160.

This also because higher predictor counts will require a tuning of the algorithm, which estimates the optimum predictor order. Currently it is failing sometimes when using more the 160 predictors. This tuning can be a lot of work and i prefer to first tune some other encoder options, which hardly affect the decoding speed and nevertheless may provide some nice improvements even for the fast presets. This will be my task for 2.0.1.

Later (2.0.2 ?) i may release an evaluation version with up to 320 predictors and let the users try themselves, if it is really advantageous for their files.

I need good reasons to increase the predictor count, because doing so will -besides the initial tuning- also mean more work regarding future modifications of the encoder, because there tends to be a lot of interaction between high predictor orders and other parts of the codec.

BTW: In the past days i have performed a lot of tests to verify the proper function of the new codec and so far everything was ok. I hope to release a first version within one week. 
Title: TAK 2.0 Development
Post by: gottkaiser on 2009-11-25 01:19:55
Big thanks for developing and improving TAK!
Was there any progress for album cover support in the Winamp plug-in?

Title: TAK 2.0 Development
Post by: TBeck on 2009-11-29 09:14:23
Was there any progress for album cover support in the Winamp plug-in?

I suppose this would require a modified plugin which uses a new interface to communicate with WinAmp. I haven't done this yet, but it should be possible to write such a plugin based upon the "TAK_deco_lib.dll". Maybe someone more experienced with WinAmp than me could quite easily do this... Otherwises i will have do it sooner or later, but probably not before releasing TAK 2.0.1 or TAK 2.0.2.

Actually i wanted to release a beta of V2.0 within the next days, but then i remembered some encoder option i was using years ago (with little success) but later have removed. Now i tried it again and achieved some tiny improvements of up to 0.05 percent for some presets. Not really much but it comes without nearly no speed penality and it's about as much (or sometimes more) as could be gained by increasing the predictor count from 160 to 192. Therefore i think it's worth to implement it. This may delay the release by a couple of days.
Title: TAK 2.0 Development
Post by: Soap on 2009-11-29 13:25:32
Quote
A significant amount of posters (not necessarily users...) kept emphasizing TAK's slower decoding speed than FLAC (especially when using foobar).
Decoding speed is definitely a priority, although in this age of computing I think current TAK decoding speeds will be fine for current portables (FLAC decoding speed may be a bit overkill nowadays as compared to 5 years ago.

A DAP with a voltage scaling CPU will gain power efficiency and thus battery life with an easier-to-decode format, even if it has enough raw horsepower to decode APE realtime.
DAPs, however, are shifting quickly away from HDDs, and therefore the battery cost of larger files has diminished significantly.
Title: TAK 2.0 Development
Post by: TBeck on 2009-12-10 13:28:17
Prepearing the release

Currently my secondary PC is performing a lot of automated tests to validate the proper function of the new codec. This may take one or two days.

In the meantime i have to deceide, if i want to add another optimization which improves the compression of 192 KHz files by about 0.25 percent but unfortunately has no significant effect on files with lower sample rates. There are good reasons against this optimization:

- It will either make the encoder slower (for any sampling rate!) or require a higher code compexity to avoid this speed penality.
- TAK's compression efficiency for 192 KHz files is already on top, only beaten by OptimFrog in my tests, therefore it's not really neccessary to add a bit more, especially because 192 KHz files are a bit exotic.

So it's very likely that we will see a first release within the next days.

For now some (final) results for my primary sample set:

Code: [Select]
AMD Sempron 2.2 GHz                                                                     
                                                                        
Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                        
-p0     58.83   58.74   0.09   264.10   289.24    9.52%   283.52   293.55    3.54%
-p1     57.98   57.84   0.14   193.18   205.66    6.46%   275.80   288.77    4.70%
-p2     57.07   56.90   0.17   131.39   135.32    2.99%   250.36   257.09    2.69%
-p3     56.52   56.36   0.16    55.97    60.78    8.59%   190.88   218.79   14.62%
-p4     56.16   56.02   0.14    32.07    34.35    7.11%   166.89   174.72    4.69%
-p4m    56.07   55.89   0.18    17.81    14.88  -16.45%

Intel Pentium Dual Core 2 GHz                                                                  
                                                                        
Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                        
-p0     58.83   58.74   0.09   261.83   283.94    8.44%   298.67   311.12    4.17%
-p1     57.98   57.84   0.14   201.75   215.76    6.94%   291.72   305.56    4.74%
-p2     57.07   56.90   0.17   146.14   149.26    2.13%   262.58   271.21    3.29%
-p3     56.52   56.36   0.16    64.64    68.66    6.22%   214.36   239.69   11.82%
-p4     56.16   56.02   0.14    36.90    38.09    3.22%   193.97   203.54    4.93%
-p4m    56.07   55.89   0.18    21.07    16.98  -19.41%
                   
Compression is relative to the original file size, Enco- and Deco-Speed expressed as multiple of real time.

I like those results: Better compression and higher encoding speed and higher decoding speed for any basic preset (without additionally evaluation level like p4m). For me this combination justifies the introduction of a new codec.

Some results for other file types and for LossyWav:

Code: [Select]
                   Preset  Compression         
                           1.1.2     2.0 Eval  Win
                    
24 bit /  44 khz    -p4m   56.78     56.73     0.05
24 bit /  96 khz    -p4m   50.69     50.70    -0.01
24 bit / 192 khz    -p4m   44.86     43.71     1.15
8 bit               -p4m   38.23     37.19     1.04
LossyWav -q00       -p2m   19.37     18.92     0.45
LossyWav -q25       -p2m   26.38     26.00     0.38
LossyWav -q50       -p2m   32.34     31.97     0.37


  Thomas
Title: TAK 2.0 Development
Post by: _m²_ on 2009-12-10 14:51:02
In the meantime i have to deceide, if i want to add another optimization which improves the compression of 192 KHz files by about 0.25 percent but unfortunately has no significant effect on files with lower sample rates. There are good reasons against this optimization:

- It will either make the encoder slower (for any sampling rate!) or require a higher code compexity to avoid this speed penality.
- TAK's compression efficiency for 192 KHz files is already on top, only beaten by OptimFrog in my tests, therefore it's not really neccessary to add a bit more, especially because 192 KHz files are a bit exotic.

If it doesn't reduce decompression speed, I'm for it.
Title: TAK 2.0 Development
Post by: SokilOff on 2009-12-10 18:32:47
In the meantime i have to deceide, if i want to add another optimization which improves the compression of 192 KHz files by about 0.25 percent but unfortunately has no significant effect on files with lower sample rates. There are good reasons against this optimization:

- It will either make the encoder slower (for any sampling rate!) or require a higher code compexity to avoid this speed penality.
- TAK's compression efficiency for 192 KHz files is already on top, only beaten by OptimFrog in my tests, therefore it's not really neccessary to add a bit more, especially because 192 KHz files are a bit exotic.


Then you may delay it until 2.0.1 (or 2.0.2), it won't hurt. 192 KHz files is really a bit exotic thing for now.

Looking forward to first TAK 2.0 beta


Title: TAK 2.0 Development
Post by: TBeck on 2009-12-11 00:16:16
If it doesn't reduce decompression speed, I'm for it.

Well, after some fine tuning the possible advantage is down to 0.1 percent...

Then you may delay it until 2.0.1 (or 2.0.2), it won't hurt. 192 KHz files is really a bit exotic thing for now.

Unfortunately the modification would make the format incompatible, therefore it has to be done before the 2.0 release.
Title: TAK 2.0 Development
Post by: SokilOff on 2009-12-11 23:49:19
Unfortunately the modification would make the format incompatible, therefore it has to be done before the 2.0 release.


Well... the best thing from 'end-user point of view' would be:

- no additional speed penalty for any (especially most used lower) sample rates
- additional 0.25% for 192 kHz files (though for many users who don't use such files it's rather virtual advantage)

The price in this case is a higher code compexity. As a developer you decide whether you'd like to implement it or not. It's up to you.
Title: TAK 2.0 Development
Post by: Destroid on 2009-12-12 02:06:06
The 192KHz is a strange one, it is poised to predict the future of digital audio.

Today, this sample rate (likely at 24-bit) is mostly used in commercial pro-studio and some DVD-A (and not very many that I can recall, I'm not a fan of DVD-A).

The more popular sampling rate out there is 96KHz used in most consumer pro-audio and (most?) DVD-A.

Of course, someday, everybody will listen to everything in 192KHz  , but will leaving out the slight 192KHz advantage today have more dramatic improvements later that hamper TAK later?

IMO, 192KHz is exotic and will likely be used very little in comparison to 96KHz. For how long will this remain? 5 years? Maybe 10? I'm not exactly sure. :shrug:
Title: TAK 2.0 Development
Post by: skamp on 2009-12-12 02:49:23
Of course, someday, everybody will listen to everything in 192KHz

I'm not so sure about that. The industry seems to be pushing for MP3s instead of CD/DVD-A/lossless.
Title: TAK 2.0 Development
Post by: Destroid on 2009-12-12 03:51:08
Excuse me, but that is off-topic.

Recording industry is already using higher sampling rates and eventually they all will, not much different from the HDTV transition. I suppose my great grandchildren will take it for granted they have home stereos with Dolby Lossless 15.1 at 192KHz
Title: TAK 2.0 Development
Post by: Seeking_Lossless on 2009-12-12 07:27:15
can human really differentiate between 44.1kHz and 192khz? This is pointless to the industry when they people cant actually feel the difference. Moreover, this will cost a lot of fortunes.
Title: TAK 2.0 Development
Post by: Zarggg on 2009-12-12 07:30:46
I can't even hear a difference between 44.1kHz and 48kHz. I think it's a bit optimistic to think that listeners will get much benefit out of 192kHz or even 96kHz.

Producers, on the other hand, could definitely take advantage of that... but for the purpose of released tracks for public consumption, 48kHz should be just fine.

For that matter, do we even need 15 spatial channels? Where would you put the speakers?
Title: TAK 2.0 Development
Post by: Destroid on 2009-12-12 08:44:37
Good questions, I'll have to consult consumer-gurus.

As for lossless, I think I'm right in saying there are no boundaries of what is "good enough," especially from the stance of audio production (something that would be suicidal if using lossy schemes). I think of lossy is the by-product of consumer-ism.

Anyways, 192KHz is currently the new bar for excellence in digital audio production, something for the studio management to brag about. Having lossless is the only choice in that environment, so obviously the best contender will see fairly high savings for such an extreme sampling rate.
Title: TAK 2.0 Development
Post by: _m²_ on 2009-12-12 09:49:58
Well, the highest standard is DXD @ 384 kHz, 24bit. Supposed to be used in audio production.
BTW can TAK encode it? FLAC can't.
Title: TAK 2.0 Development
Post by: lvqcl on 2009-12-12 10:20:27
IIRC DXD has samplerate 44.1*8 = 352.8 kHz. Also, FLAC can encode 352.8 kHz/24bit WAV file.
Title: TAK 2.0 Development
Post by: _m²_ on 2009-12-12 21:34:45
IIRC DXD has samplerate 44.1*8 = 352.8 kHz.

Correct, my bad.
Also, FLAC can encode 352.8 kHz/24bit WAV file.

I've had once a wav file that I remembered to be DXD, though it might me 384 kHz. FLAC couldn't encode it.
Title: TAK 2.0 Development
Post by: jcoalson on 2009-12-14 06:10:33
The 192KHz is a strange one, it is poised to predict the future of digital audio.

let's see some evidence of that.  192kHz might have some advantages as an intermediate format but for final distribution anything over 48kHz is overkill because no one will hear it.

Well, the highest standard is DXD @ 384 kHz, 24bit. Supposed to be used in audio production.
BTW can TAK encode it? FLAC can't.

yes it can, the max flac sample rate is 655350Hz and is not even really practically limited to that.
Title: TAK 2.0 Development
Post by: TBeck on 2009-12-14 12:59:36
In the meantime i have to deceide, if i want to add another optimization which improves the compression of 192 KHz files by about 0.25 percent but unfortunately has no significant effect on files with lower sample rates. There are good reasons against this optimization:

- It will either make the encoder slower (for any sampling rate!) or require a higher code compexity to avoid this speed penality.
- TAK's compression efficiency for 192 KHz files is already on top, only beaten by OptimFrog in my tests, therefore it's not really neccessary to add a bit more, especially because 192 KHz files are a bit exotic.

So it's very likely that we will see a first release within the next days.

And i was about to release V2.0 Beta on last saturday...

But then i deceided to dig a bit deeper into the said optimization. Fortunately!

- The compression of my 192 KHz test files improved by 0.45 percent. If i exclude files converted from DSD-Sources from my test set, the advantage is more than 1 percent. That's because the extreme high frequency noise of the DSD files kills nearly any attempt to improve the signal prediction.

- The new optimization can also help some CD-Audio files a bit, but this will require further tuning of the encoder in later releases.

- As a side effect the encoding speed of 16 bit audio is again a bit higher for some presets.

Now i again have to perform a lot of testing to prepare the release.
Title: TAK 2.0 Development
Post by: Destroid on 2009-12-15 21:05:57
Excellent gains there, Thomas. It also reminds me of a dilemma I frequently run into with projects: "Is it done yet, or should I keep fine-tuning it?" I have only to thank exterior deadlines for simplifying the decision.

192kHz might have some advantages as an intermediate format but for final distribution anything over 48kHz is overkill because no one will hear it.
Yes, I omitted a lot in that claim. I keep forgetting a majority of end-users deal with 44.1 and 48 KHz material. To clarify, the studio (where lossless is a must) would use such high sampling rates for production, and that is where I meant that there are no boundaries (practically speaking and audibility aside). Given the idea that higher sampling rates are yet to become more popular later on (not sure how much or how soon), I was addressing Thomas' question if leaving out better 192KHz compression in version 2.0 release might encumber the codec later (a format/container change).
Title: TAK 2.0 Development
Post by: TBeck on 2009-12-16 02:21:51
Excellent gains there, Thomas. It also reminds me of a dilemma I frequently run into with projects: "Is it done yet, or should I keep fine-tuning it?" I have only to thank exterior deadlines for simplifying the decision.

How true...

The said optimizations may provide opportunities for further improvements, which would not be compatible with the current file format. If i could yet specify the necessary modifications of the format, i could prepare the decoder for it and then later optimize the encoder to take advantage of it. Unfortunately i can't exactly specify the modifications without writing the encoder part and performing a lot of evaluations first. Since this means much work and i still want to release at least a beta before christmas, i will not implement it now.
Title: TAK 2.0 Development
Post by: _m²_ on 2009-12-16 08:29:45
Excellent gains there, Thomas. It also reminds me of a dilemma I frequently run into with projects: "Is it done yet, or should I keep fine-tuning it?" I have only to thank exterior deadlines for simplifying the decision.

How true...

The said optimizations may provide opportunities for further improvements, which would not be compatible with the current file format. If i could yet specify the necessary modifications of the format, i could prepare the decoder for it and then later optimize the encoder to take advantage of it. Unfortunately i can't exactly specify the modifications without writing the encoder part and performing a lot of evaluations first. Since this means much work and i still want to release at least a beta before christmas, i will not implement it now.


Well, one can make a really future-proof decoder the way zpaq did - by specifying an interepreted language for writing (de)compressors and attaching algorithms to the files. However implementing it efficiently (JIT compiling?) would take a lot of time....I guess it's not an idea for TAK, but for some future project.
Title: TAK 2.0 Development
Post by: TBeck on 2009-12-18 19:13:59
Well, one can make a really future-proof decoder the way zpaq did - by specifying an interepreted language for writing (de)compressors and attaching algorithms to the files. However implementing it efficiently (JIT compiling?) would take a lot of time....I guess it's not an idea for TAK, but for some future project.

That could be interesting for a codec focussing only on optimum compression. But it would be too slow for TAK. Maybe i will play a bit with symmetric compression, when i am done with TAK's high efficiency codec... There such an approach could fit. Well, you have to try it to be sure.

BTW: Now i am definitely preparing a beta release. Execution of my validation scripts started some hours ago, but they will take some more hours to complete.
SimplePortal 1.0.0 RC1 © 2008-2019