Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: TAK - Alpha testing (Read 72336 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

TAK - Alpha testing

Reply #75 –
Quote from: TrNSZ on
I'll post a full report later, but this one is very interesting.  I decided to throw away Sade, Dire Straits, Pink Floyd, Blue Nile, Cowboy Junkies, Police, Donald Fagen, Mina, etc. and all those other "audiophile" discs and go with a much more eclectic and hopefully more representive test set that will give better results.  We'll call this the "A" test set.  The selection was chosen to be diverse and have a wide variety of different mastering techniques and musical styles represented to give a fair mix.  Hopefully a preview will be available tonight.

Very exciting! I hope i will have time to respond.
Quote from: TrNSZ on
... and hopefully codecs don't update while I'm testing.

No changes for Tak before the first public release.

But then i will look for better compression...

  Thomas

TAK - Alpha testing

Reply #76 –
[deleted]

TAK - Alpha testing

Reply #77 –
I've uploaded my test results for an album image.

Download Links:
Zip File Containing XLS, PDF

Album Details:
Album Tested: Renegades by Rage Against the Machine (Genre: Metal/Rap/"electronic"/weird)
Album Length: ~60 minutes (3636 seconds)
Uncompressed Size: 611.69 MB (641402204 bytes)

Test Bed:
Dell e1505 Laptop: Intel Core 2 Duo T7200 @ 2 Ghz, 1 GB Ram, 120 GB HDD 5400 RPM

Purpose:
1. To test Tak vs other codecs in a more realistic setting than previous test: to compress an album image.
2. To test Wavpack's new switches in a more realistic setting.

Methodology:
A single wav file of the album was created using foobar. The timing of compression/decompression was done by Synthetic Soul's scripts.


Encoders Versions and Switches Tested
1. TAK 0.15 Alpha: All Switches + Max for each
2. FLAC 1.1.3: -0 to -8
3. Wavpack 4.40.0: All Switches + (x1 through x6 for each)
4. Monkey's Audio 4.01: Fast through Insane
5. Optimfrog 4.600 ex: Default (no switches) and (fast, seek fast)
6. ALS RM18: Default (no switches), -7 ("maximum compression"), -a -o32
7. LA 0.4b: Default
8. TTA 3.3: Default

Remarks
1. Encoding/Decoding Performance
- Tak has the fastest encoding speed (Tak0 encodes faster than even Flac0).
- Tak also has the fastest decoding speed (Tak4M decodes faster than Flac1, and wavpack Fast)

2. Compression
- Tak is able to beat Flac hands down (tak 0 compresses better than Flac 8), and also outperforms Wavpack (Wavpack HH x 6 is matched by TAK 1). The performance vs ALS is also quite commendable: only ALS at the maximum compression beats TAK Correction: In this data set, TAK does perform better than ALS
- However, Tak doesn't quite measure up to Monkey's audio. Monkey's Audio at Normal compressed better than Tak 4m. Similarly, it can't match up with Optimfrog even at the most lax setting i could think of (fast with fast seeking). Higher compression is something TAK could improve on

3. Wavpack specific observations
- x1, x2 and x3 are definitely usable and beneficial but x4, x5, x6 are very hard to justify... -hh x6 in particular encodes at 0.98X, which is slower than even LA or ALS at maximum (although the decoding speed is very good: 74x).

4. Weird Observations
- Flac1 seems to compress faster than Flac0. I noticed this in the drug me sample too.


Future Plans
- Test on a spoken/comedy album (Trsnz seems to have taken care of all other genres  )
- Full swing damage test (i'm sure that will be loads of fun)
- Maybe a small scale test with Tak vs Flac optimized for P4 vs Wavpack optimized for MMX.
- Maybe test the same album with optimfrog maximum compression experimental to torture my computer (and to see if it can match LA).

PS: Sorry for the delay (i had my finals)

TAK - Alpha testing

Reply #78 –
Quote from: kanak on
I've uploaded my test results for an album image.

Download Links:
Zip File Containing XLS, PDF

Great presentation. Thank you!

Quote from: kanak on
2. Compression
- Tak is able to beat Flac hands down (tak 0 compresses better than Flac 8), and also outperforms Wavpack (Wavpack HH x 6 is matched by TAK 1). The performance vs ALS is also quite commendable: only ALS at the maximum compression beats TAK (although the compression time for ALS at maximum is abysmally slow)

Hm... From your data it looks as if Tak 1 already beats ALS -7 (The strongest mode of your results). Are you talking about ALS's z-modes?

Quote from: kanak on
Future Plans
- Test on a spoken/comedy album (Trsnz seems to have taken care of all other genres  )
- Full swing damage test (i'm sure that will be loads of fun)
- Maybe a small scale test with Tak vs Flac optimized for P4 vs Wavpack optimized for MMX.
- Maybe test the same album with optimfrog maximum compression experimental to torture my computer (and to see if it can match LA).

Very impressive!

Nice that all this testing isn't only useful  for TAK development. This thread for instance provides some good examples for the difficulty to judge a codecs performance by only limited file sets.

TAK - Alpha testing

Reply #79 –
Quote from: TBeck on
Quote from: kanak on

2. Compression
- Tak is able to beat Flac hands down (tak 0 compresses better than Flac 8), and also outperforms Wavpack (Wavpack HH x 6 is matched by TAK 1). The performance vs ALS is also quite commendable: only ALS at the maximum compression beats TAK (although the compression time for ALS at maximum is abysmally slow)

Hm... From your data it looks as if Tak 1 already beats ALS -7 (The strongest mode of your results). Are you talking about ALS's z-modes?


Oops. Corrected.

TAK - Alpha testing

Reply #80 –
I've uploaded my test results for an album image (genre: spoken).

Download Links:
UPDATED: Zip File Containing XLS, PDF
ORIGINAL: Zip File Containing XLS, PDF

Album Details:
Album Tested: People's History of the United States by Howard Zinn (Genre: Spoken Word)
Album Length: ~50 minutes (3030 seconds)
Uncompressed Size: ~510 MB (534521900 bytes)
Album is Stereo, but both sides contain nearly identical data

Test Bed:
Dell e1505 Laptop: Intel Core 2 Duo T7200 @ 2 Ghz, 1 GB Ram, 120 GB HDD 5400 RPM

Purpose:
1. To test the Tak on spoken albums. I thought it would be interesting to test codecs on audio samples which may be stereo but have identical data on both channels (like this one), and whose audio content is primarily human voice.
2. To test the (speed) optimized versions of FLAC and Wavpack as well, to see how much improvement can be expected.


Methodology:
A single wav file of the album was created using foobar. The timing of compression/decompression was done by Synthetic Soul's scripts.

Encoders Versions and Switches Tested
1. TAK 0.15 Alpha: All Switches + Max for each

2. FLAC 1.1.3: -0 , -1, -5, -8
3. P4 Optimized FLAC 1.1.3: -0 , -1, -5, -8
    (available Here)

4. Wavpack 4.40.0: Fast, Normal, HH
5. MMX Optimized Wavpack build: Fast, Normal, High
    (Available Here)

5. Monkey's Audio 4.01: Fast, Insane
6. Optimfrog 4.600 ex: Default (no switches) and (fast, seek fast)
7. ALS RM18: Default (no switches)
8. LA 0.4b: Default
9. TTA 3.3: Default

Note: Wavpack optimized build is based on Wavpack 4.31 while the "non-optimized" version i tested is version 4.40. Thus, there is a difference in resultant file size. Also, i've used the -HH switch for the official build because it is the "equivalent" of the old high.


Remarks

1. Compression
-  This has got to be the first sample i've seen in which Monkey's audio performs better than both LA and OptimFrog AND more importantly, TAK at 4M is the best compression (i haven't tested it on OptimFrog --maximumcompression --experimental yet).
- the performance of Flac with switch -0 is pretty abysmal: double of Flac -1.
- 4.40 compresses better than 4.31, this is evident in the Fast and Default switches. No changes in the -HH setting (which is expected)

2. Encoding Speed
- Again, performance of TAK is pretty good, but TAK 4M compresses only about as fast as Monkey at Insane. (TAK 4M has always compressed faster than Monkey (even at Extra High; also see Synthetic Soul's tests)).
- The optimized builds are pretty impressive. In case of Flac, the improvement is very significant at -8 (~70 % faster). The performance is more subtle for Wavpack (~7 % at most)
- Once again, Flac -1 compresses Faster than Flac -0. Something fishy is going on with Flac -0.

TAK - Alpha testing

Reply #81 –
just wanted to point out that TAK 4m should be 20.29, which would be TAK's best result instead of 21.72 which would be its worst.
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame

TAK - Alpha testing

Reply #82 –
[deleted]

TAK - Alpha testing

Reply #83 –
Quote from: kanak on
I've uploaded my test results for an album image (genre: spoken).

Download Links:
Zip File Containing XLS, PDF

Album Details:
Album Tested: People's History of the United States by Howard Zinn (Genre: Spoken Word)
Album Length: ~50 minutes (3030 seconds)
Uncompressed Size: ~510 MB (534521900 bytes)
Album is Stereo, but both sides contain nearly identical data

Looking at these results makes me think that this is truly mono material (but in 2 channels, obviously). This would explain the very poor performance of WavPack (compared to the others). It also may explain the weird performance of flac -0. Josh (or someone else familar with the internals) would have to confirm, but perhaps the -0 option doesn't even include joint stereo so both channels are completely independent, and that would explain getting only about half the compression of -1.

I suspect that using the --optimize-mono option in WavPack would put in right back into the pack with the others.

Thanks again for your thorough testing. 

David

TAK - Alpha testing

Reply #84 –
Quote from: music_man_mpc on
just wanted to point out that TAK 4m should be 20.29, which would be TAK's best result instead of 21.72 which would be its worst.


Thank you. There was a problem with the formulas in excel. I've corrected it.


Quote from: bryant on
I suspect that using the --optimize-mono option in WavPack would put in right back into the pack with the others.


Updated the table. You were right, it does make a  helluva difference. Would it be possible for you to make the encoder automatically use the optimize mono if the material is "pseudomono"?

TAK - Alpha testing

Reply #85 –
Quote from: bryant on
Looking at these results makes me think that this is truly mono material (but in 2 channels, obviously). This would explain the very poor performance of WavPack (compared to the others). It also may explain the weird performance of flac -0. Josh (or someone else familar with the internals) would have to confirm, but perhaps the -0 option doesn't even include joint stereo so both channels are completely independent, and that would explain getting only about half the compression of -1.

yep, that's exactly right.

Josh

TAK - Alpha testing

Reply #86 –
Josh, is there any reason why Flac -0 encodes slower than Flac -1 on my computer? All three samples i've tested so far exhibit this.

TAK - Alpha testing

Reply #87 –
Quote from: kanak on
Quote from: bryant on

I suspect that using the --optimize-mono option in WavPack would put in right back into the pack with the others.


Updated the table. You were right, it does make a  helluva difference. Would it be possible for you to make the encoder automatically use the optimize mono if the material is "pseudomono"?

I could, but the resulting files are not compatible with all decoders (specifically the current tiny_decoder and DirectShow filter). At some point in the future I will surely do this.

Anyway, thanks for trying that. I'm glad that it helped...

TAK - Alpha testing

Reply #88 –
What's tiny_decoder?

TAK - Alpha testing

Reply #90 –
Quote from: kanak on
Josh, is there any reason why Flac -0 encodes slower than Flac -1 on my computer? All three samples i've tested so far exhibit this.
I suspect for the same reason why the compression ratio is so bad compared to -1, which bryant has already explained. Since both channels are processed independently with -0, I suspect that the joint-stereo processing that takes place with -1 and onwards fastens things up tremedously for "true mono in 2 channel" material, whereas this j-s step would normally slow the encoding down for true stereo material compared to indepedent processing of both channels only.

TAK - Alpha testing

Reply #91 –
Quote from: kanak on
Josh, is there any reason why Flac -0 encodes slower than Flac -1 on my computer? All three samples i've tested so far exhibit this.

not sure, depends on how much slower it is, but on this sample I would guess at least some is due to 2x the disk writes.

TAK - Alpha testing

Reply #92 –
Preparing the public beta

That's what i would like to do know.

Please don't take me wrong: I don't want to stop you the alpha testing! Both can be done simultaneously.

Some questions:

Do you know any reason why i shouldn't release the beta soon?

If you have found any bugs, now it's really time to shout...

Is the documentation ok?

I know, it's really bad english, but is it still understandable? Would it be even sufficient for someone less familiar with lossless audio compression than you?

I definitely want to improve it in the future, but there will be only minor changes for the first public release:

1) Add disclaimer of warranty.
2) Very short description of the purpose of the software.
3) A list of important features which are missing put planned to be implemented into the next release (tagging, media player support).

Try tagging

While Tak does not support tagging on it's own, the current version should ignore any data appended to the end of the encoded file. Therefore it should be possible to use an external software to apply tags to the file end. The files should still be decodable without any problems. Could someone please try this?

It's different for tags inserted at the beginning of the file. The current decoder will give you an error message. No need to try this yet.

  Thomas

TAK - Alpha testing

Reply #93 –
Thomas. I'm of the opinion that a public beta shouldn't be released without a foobar2000 decoding plugin. Why? Well if you want your testers to truly test the codec they need to be able to use it in the real world environment they're familiar with. If you do the public beta as you did the alpha you'll probably get a lot of compression and speed comparisons (which is of course great) and maybe a few UI or general format issues like the 2gb thing that came up. But the test won't give you nearly the kind of mileage you'd get if the lossless codec could actually be used as a lossless codec. You'd want to put all kinds of warning about not to use it for anything but testing of course

I'm also wondering what you're going to do with the presets. Are you planning on making "Max" part of the public "final"? I ask because in my opinion the highest preset and the Max option are backwards. Max of course means maximum or the best or highest possible. And extra means a boost or something in addition that was perhaps not specifically planned for. It would make more sense if the Extra preset was called Max and vise versa.

TAK - Alpha testing

Reply #94 –
Quote from: Gnerma on
Thomas. I'm of the opinion that a public beta shouldn't be released without a foobar2000 decoding plugin. Why? Well if you want your testers to truly test the codec they need to be able to use it in the real world environment they're familiar with. If you do the public beta as you did the alpha you'll probably get a lot of compression and speed comparisons (which is of course great) and maybe a few UI or general format issues like the 2gb thing that came up. But the test won't give you nearly the kind of mileage you'd get if the lossless codec could actually be used as a lossless codec. You'd want to put all kinds of warning about not to use it for anything but testing of course

I'm also wondering what you're going to do with the presets. Are you planning on making "Max" part of the public "final"? I ask because in my opinion the highest preset and the Max option are backwards. Max of course means maximum or the best or highest possible. And extra means a boost or something in addition that was perhaps not specifically planned for. It would make more sense if the Extra preset was called Max and vise versa.


I agree with Gnerma on both points.

I don't think any significant additional testing can be done unless there is a decoder for some audio program. If there were a decoder, we could actually compress albums with tags and see how TAK decodes in "real life" applications as opposed to how it decodes to wav. So please consider releasing a foobar plugin along with the public beta.

I too was confused initially by the Max option. I thought it was something like OFR's --maximumcompression option, and didn't pay much attention towards it (i only realized it was like wavpack's x option after looking at synthetic soul's comparison charts). To avoid this "confusion", i guess you could have an -x system like Wavpack's (x for extra); it's just a matter of calling it something different to avoid possible confusion.

TAK - Alpha testing

Reply #95 –
Quote from: Gnerma on
I'm also wondering what you're going to do with the presets. Are you planning on making "Max" part of the public "final"? I ask because in my opinion the highest preset and the Max option are backwards. Max of course means maximum or the best or highest possible. And extra means a boost or something in addition that was perhaps not specifically planned for. It would make more sense if the Extra preset was called Max and vise versa.


Quote from: kanak on
I too was confused initially by the Max option. I thought it was something like OFR's --maximumcompression option, and didn't pay much attention towards it (i only realized it was like wavpack's x option after looking at synthetic soul's comparison charts). To avoid this "confusion", i guess you could have an -x system like Wavpack's (x for extra); it's just a matter of calling it something different to avoid possible confusion.


Thanks for telling me about this irritation. Possibly some of the confusion comes from the layout of the options dialog in the gui version, where "Additionally Evaluation - Max" is placed following the strongest preset. This might look as if the additional evaluation is another (the really strongest) preset.

But on the command line it should be quite clear.

I will think about a better solution. Indeed someting like -x (extra) would make more sense (especially for people familar with WavPack). There are historical reasons why it isn't called extra: Earlier there have been two levels of additional (extra) evaluation, "extra" and "max"...

Quote from: Gnerma on
Thomas. I'm of the opinion that a public beta shouldn't be released without a foobar2000 decoding plugin. Why? Well if you want your testers to truly test the codec they need to be able to use it in the real world environment they're familiar with. If you do the public beta as you did the alpha you'll probably get a lot of compression and speed comparisons (which is of course great) and maybe a few UI or general format issues like the 2gb thing that came up. But the test won't give you nearly the kind of mileage you'd get if the lossless codec could actually be used as a lossless codec. You'd want to put all kinds of warning about not to use it for anything but testing of course


Quote from: kanak on
I don't think any significant additional testing can be done unless there is a decoder for some audio program. If there were a decoder, we could actually compress albums with tags and see how TAK decodes in "real life" applications as opposed to how it decodes to wav. So please consider releasing a foobar plugin along with the public beta.

I agree that a foobar plugin would be neccessary for a comprehensive real life beta test. But unfortunately it will take quite much time for me to build one. Probably we will see a WinAmp plugin first.

To be honest, i definitely don't want to wait any longer with a public release, this simply because it would lighten me. Now it are nine month since my first post about Tak. People have often been asking for a release, have told, how excited they are. I really want to make Tak available.

Do you think it would severely hurt, if Tak's first release is missing some important features? People who are interested into Tak at least would be able to try it and judge, if it is worth to wait for a full blown release.

Well, Tak's development history may be a bit different: The first release will be enormously optimized performance wise, but lack some general features. Later releases will improve more the usability and less the performance.

  Thomas

TAK - Alpha testing

Reply #96 –
Quote from: TBeck on
Do you know any reason why i shouldn't release the beta soon?
No.  A foobar or Winamp component would be nice, but it isn't necessary to let more people start testing the codec.  If users aren't happy with just encoding and decoding to WAVE then let them wait, they don't have to try it!

 
Quote from: TBeck on
Is the documentation ok?
I need to scrutinise it further, but AFAIR it is absolutely fine.  It's a very simple thing to improve in the future in any case.

 
Quote from: TBeck on
Try tagging
I tagged a file with APEv2 tags using the following Tag command line:

Code: [Select]
TAG.EXE --ape2 --nocheck --artist "My Artist" --album "My Album" test.tak

... and it decoded fine, is the same size as the source, and is bit-identical according to foobar.

I think inbuilt tagging is desirable for the future, but some codecs can't tag themselves still, Monkey's Audio being one.
I'm on a horse.

TAK - Alpha testing

Reply #97 –
[deleted]

TAK - Alpha testing

Reply #98 –
Quote from: TBeck on
Do you think it would severely hurt, if Tak's first release is missing some important features? People who are interested into Tak at least would be able to try it and judge, if it is worth to wait for a full blown release.


Sorry for bumping in ...

I don't think Tak's reputation will hurt if the most simple compress or decompress function works fine.

People asking for features can wait and tolerate because features usually don't hurt the core functionality. I think if the coder fail the simplest function (CRC error, extremely slow, crashes) and the author [don't care] , that will surely hurt the reputation.

Quote
Well, Tak's development history may be a bit different: The first release will be enormously optimized performance wise, but lack some general features. Later releases will improve more the usability and less the performance.


I can understand you want to release a "bug free" and good performance binary. But without a large amount of testing in different combination, there may be hidden bugs or usage problems, so I think if the basic functions work fine, a BETA version can be release anytime.

Like LAME, without huge number of people testing, it won't be so good now, IMO the most difficult decision is "official release" because it MUST be completely bug free.
Hong Kong - International Joke Center (after 1997-06-30)

TAK - Alpha testing

Reply #99 –
Quote from: TrNSZ on
APEv2 tagging as an accepted standard would be ideal, IMHO.
Indeed. I will be disappointed if not.  As TAK ignores anything stuck on the end I guess ID3v1 and APEv2 both would be valid, and easily implemented.  I see from another thread this morning that WavPack is OK with both.
I'm on a horse.