Skip to main content
Topic: TAK - Alpha testing (Read 52420 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

TAK - Alpha testing

Reply #50
Thanks for the update Thomas. No change on the my greater than 2gb WAVs. The program still throws "Wave file not supported" for both.

I had the opportunity to examine the file headers.

First comes a valid RIFF structure indicating a size of 2 GB and containing about 2 GB of wave data.

I suppose, that the remaining audio data is embedded into another RIFF or data chunk following the first one. I could not verify this, because i only have the first 100 KB of the file.

Probably the application which generated this file has created this layout to make the file compatible to older applications, which are only able to read up to 2 GB. They may read the first chunk and ignore the remaining data.

But to my knowledge this file structure is definitely non-standard.

Gnerma told me, that this file could be compressed by WavPack. I have looked into it's source code and from my understanding WavPack will do this:

1) Read the first data chunk of 2GB and compress it.
2) Store the remaining (audio) data as uncompressed meta data.

If that's true, then TAK isn't behaving differently. But Tak currently limits the the size of wave file meta data to 1 MByte, not enough to store the remaining audio data.

For now i would not like to put too much effort into the support of such a non-standard format. Sorry.

But possibly someone more knowledgeable than myself will convince me, that this format is common practice...

TAK - Alpha testing

Reply #51

Thanks for the update Thomas. No change on the my greater than 2gb WAVs. The program still throws "Wave file not supported" for both.

...
For now i would not like to put too much effort into the support of such a non-standard format. Sorry.
...


Well, Destroid's (the second tester with trouble compressing files bigger than 2 GB) files have the same structure...

Currently Tak is capable to compress files up to 4 GB if the whole audio data is contained in one data chunk. What would be neccessary:

1) Support for reading and writing multiple data chunks.
2) A new meta data structure to store not only the file header and the non-audio data following the last audio data chunk, but also the segmentation (partition into multiple audio data chunks) of the audio data.

Some problems:

1) My 40 GB hard drive is nearly filled.
2) I myself have no application which creates files with such a structure.

But possibly Destroid or Gnerma could create a special big file for me:

1) Create a file of more than 2 GB which contains only silence!
2) After compression this should be small enough to be sent to me (I hope so...).

  Thomas

TAK - Alpha testing

Reply #52
If I remember correctly the thread "The 2GB Wav-limit!?" is a good read.  It may be of some help, but it also may not be relevant here.
I'm on a horse.

TAK - Alpha testing

Reply #53
Thomas, you probably do have the tools necessary to create such a file. In my case the only things that could have done this we're foobar2000 when I used convert -> convert to single file. Or Wavpack when the file was compressed.

TAK - Alpha testing

Reply #54
If I remember correctly the thread "The 2GB Wav-limit!?" is a good read.  It may be of some help, but it also may not be relevant here.


Thomas, you probably do have the tools necessary to create such a file. In my case the only things that could have done this we're foobar2000 when I used convert -> convert to single file. Or Wavpack when the file was compressed.

Thanks for the info!

  Thomas

TAK - Alpha testing

Reply #55
Thomas, you probably do have the tools necessary to create such a file. In my case the only things that could have done this we're foobar2000 when I used convert -> convert to single file. Or Wavpack when the file was compressed.

I will try this later.

Until now i have checked CoolEdit 2000 and Audacity. Both generated wave files bigger than 2 GB with only one audio data chunk, which could be read by TAK.

Off topic: I let Audacity generate 250 min of silence, compressed it with Tak and surprise: It could only be compressed to about 15 percent! Too much for silence. I examined the wave file and found, that Audacity had generated alternating sample values of -1 and +1 instead of all 0. Well, i am not familar with Audacity, possibly there is an option to create real silence...

Back to the topic:

I hope you don't mind, but i deceided not to deal with those specific big wave file issues for the first public release, because

- there are more important topics on my to do list (for instance tagging)
- the solution would involve much time consuming testing (processing of large wave files is rather slow on my  good old system...)

I hope it's ok for now.

Nevertheless your findings are really helpful for future implemtations of Tak! Sooner or later i will deal with this problem.

  Thomas

TAK - Alpha testing

Reply #56
No problem TBeck. I agree that working on things related to getting the codec "usable" should get higher priority. Thanks for all the hard work.

TAK - Alpha testing

Reply #57
Off topic: I let Audacity generate 250 min of silence, compressed it with Tak and surprise: It could only be compressed to about 15 percent! Too much for silence. I examined the wave file and found, that Audacity had generated alternating sample values of -1 and +1 instead of all 0. Well, i am not familar with Audacity, possibly there is an option to create real silence...


That's weird, here an "empty" file is created, zeros only. But I haven't tested it with such a big file. Are you sure you used the menu entry "Generate->Silence..."?

PS: It wouldn't surprise me though when Audicity produces such weird files after a certain file limit was exceeded. This program has some unfixed bugs, and maybe this is just one of them. Unfortunately I can't test it here, because I also got no HD space left at the moment.

TAK - Alpha testing

Reply #58
Off topic: I let Audacity generate 250 min of silence, compressed it with Tak and surprise: It could only be compressed to about 15 percent! Too much for silence. I examined the wave file and found, that Audacity had generated alternating sample values of -1 and +1 instead of all 0. Well, i am not familar with Audacity, possibly there is an option to create real silence...


That's weird, here an "empty" file is created, zeros only. But I haven't tested it with such a big file. Are you sure you used the menu entry "Generate->Silence (process)..."?

That's what i did:

1) Start Audacity (V 1.2.4).
2) Project -> New stereo track (i am using a german version, translation may be different)
3) Generate -> Silence
4) File -> Export (Track) as wave (44 Khz, 16 bit, stereo)

Internal project sample format is 32 bit float, hence a sample conversion has to be performed.

Same happens with one second of silence.

My description of the sample values wasn't exact. Here a small part (of a 1 second file):

Code: [Select]
Left   0  0  1  0  0 -1  1  0
Right  0 -1  1  1 -1  1 -2  2


Possibly there is some dithering beeing performed when converting from 32 bit float to 16 bit int.

TAK - Alpha testing

Reply #59
Yeah, most likely it's dithered silence.

TAK - Alpha testing

Reply #60
It is dithering. If you disable it (under settings > quality (Einstellungen > Qualit├Ąt, using German version as well)) it produces a file containing only zeroes (apart from the header of course).

TAK - Alpha testing

Reply #61
[deleted]

TAK - Alpha testing

Reply #62
uncompressed.wav   756,297,404

0.tak         459,680,772
0m.tak         457,282,414
1.tak         456,444,623
1m.tak         454,754,042
2.tak         454,146,970
2m.tak         453,596,130
3.tak         451,564,637
3m.tak         450,963,823
4.tak         449,585,555
4m.tak         449,183,998

fast.ape      460,594,920
normal.ape      454,237,188
high.ape      450,957,024
extra high.ape      445,940,648
insane.ape      441,223,400

0.flac         502,612,998
2.flac         491,362,729
3.flac         478,381,725
6.flac         467,162,472
7.flac         466,275,676
8.flac         465,710,069

TAK - Alpha testing

Reply #63
uncompressed.wav   756,297,404

0.tak         459,680,772
...
8.flac         465,710,069

Thank you!

Could you please tell, what kind of music this is?

TAK - Alpha testing

Reply #64
uncompressed.wav   756,297,404

I took the freedom to bring audiofreak's data into another form:
Code: [Select]
                   Size (MB)  Ratio   Diff Best   Diff Tak 4m
uncompressed.wav    721,26    100,00    -41,66    -40,61
0.flac              479,33     66,46     -8,12     -7,06
2.flac              468,60     64,97     -6,63     -5,58
3.flac              456,22     63,25     -4,91     -3,86
6.flac              445,52     61,77     -3,43     -2,38
7.flac              444,68     61,65     -3,31     -2,26
8.flac              444,14     61,58     -3,24     -2,19
fast.ape            439,26     60,90     -2,56     -1,51
0.tak               438,39     60,78     -2,44     -1,39
0m.tak              436,10     60,46     -2,12     -1,07
1.tak               435,30     60,35     -2,01     -0,96
1m.tak              433,69     60,13     -1,79     -0,74
normal.ape          433,19     60,06     -1,72     -0,67
2.tak               433,11     60,05     -1,71     -0,66
2m.tak              432,58     59,98     -1,64     -0,58
3.tak               430,65     59,71     -1,37     -0,31
3m.tak              430,07     59,63     -1,29     -0,24
high.ape            430,07     59,63     -1,29     -0,23
4.tak               428,76     59,45     -1,11     -0,05
4m.tak              428,38     59,39     -1,05      0,00
extra high.ape      425,28     58,96     -0,62      0,43
insane.ape          420,78     58,34      0,00      1,05

Sorted by compression.
"Diff Best" is the difference between the row and the best result.
"Diff Tak 4m" is the difference between the row and Tak's strongest mode.

TAK - Alpha testing

Reply #65
Currently Tak is capable to compress files up to 4 GB if the whole audio data is contained in one data chunk. What would be neccessary:

1) Support for reading and writing multiple data chunks.
2) A new meta data structure to store not only the file header and the non-audio data following the last audio data chunk, but also the segmentation (partition into multiple audio data chunks) of the audio data.

Some problems:

1) My 40 GB hard drive is nearly filled.
2) I myself have no application which creates files with such a structure.

But possibly Destroid or Gnerma could create a special big file for me:

1) Create a file of more than 2 GB which contains only silence!
2) After compression this should be small enough to be sent to me (I hope so...).

  Thomas

I have another idea how you could create the file yourself: use a compressed folder. I hope you have a NTFS partition, then this should be possible without any problem. The tricky part is to make your audio editor "play along". In case it uses temp files (which I'm pretty sure of) you should compress the corresponding temp directories as well. Oh and be sure they are empty. In other words: no other files opened in the audio editor, because non silent audio data may slow down the computer tremendously when it is compressing those non-silent temp files.

I tested it with a WAV file filled with 6 hours of silence (16bit/44.1kHz of course), it compressed to aprox. 8.192 bytes!  Uncompressed size is 3.810.242.762 bytes.  I think testing TAK with very big WAV files is no problem anymore... at least silent ones.

TAK - Alpha testing

Reply #66

Some problems:

1) My 40 GB hard drive is nearly filled.

I have another idea how you could create the file yourself: use a compressed folder. I hope you have a NTFS partition, then this should be possible without any problem. The tricky part is to make your audio editor "play along". In case it uses temp files (which I'm pretty sure of) you should compress the corresponding temp directories as well. Oh and be sure they are empty. In other words: no other files opened in the audio editor, because non silent audio data may slow down the computer tremendously when it is compressing those non-silent temp files.

Nice idea! Well, unfortunately i am using Win 98... Probably i will buy Win XP and possibly a bigger hard disk next month.

TAK - Alpha testing

Reply #67
Gnerma told me, that this file could be compressed by WavPack. I have looked into it's source code and from my understanding WavPack will do this:

1) Read the first data chunk of 2GB and compress it.
2) Store the remaining (audio) data as uncompressed meta data.

If that's true, then TAK isn't behaving differently. But Tak currently limits the the size of wave file meta data to 1 MByte, not enough to store the remaining audio data.


I'd like to add that Wavpack's WAV file limit is 4GB, which makes sense since the maximum size of one RIFF chunk seems to be 4GB*.  So the size limit for all standard compliant WAV files is 4GB. I've just tested it right now (in a compressed folder ) and the limit is definitely 4GB, the rest is appended to the WV file, bloating it up. So the effective WAV size limit for Wavpack is also 4GB, compressing bigger files makes no sense. Btw, wavpack, will report the same compression ratio (99.96%) as before, but the compression takes longer since appending the additional chunk also takes quite some time. 

I'd say stick to a 4GB limit for now. Any WAV file bigger than that is definitely not standard. (For example Adobe Audition warned before saving a WAV bigger than 4GB, telling me the file will only open in Audition.)

*=Quote from Wikipedia's IFF article:
Quote
An IFF file is built up from chunks. Each chunk begins with what the spec calls a "Type ID" (what the Macintosh called an OSType and Windows developers might call a FourCC). This is followed by a 32-bit unsigned integer (all integers in IFF files' structure are big-endian) specifying the size of the following data (the chunk content) in bytes.

TAK - Alpha testing

Reply #68

Gnerma told me, that this file could be compressed by WavPack. I have looked into it's source code and from my understanding WavPack will do this:

1) Read the first data chunk of 2GB and compress it.
2) Store the remaining (audio) data as uncompressed meta data.

If that's true, then TAK isn't behaving differently. But Tak currently limits the the size of wave file meta data to 1 MByte, not enough to store the remaining audio data.


I'd like to add that Wavpack's WAV file limit is 4GB, which makes sense since the maximum size of one RIFF chunk seems to be 4GB*.  So the size limit for all standard compliant WAV files is 4GB. I've just tested it right now (in a compressed folder ) and the limit is definitely 4GB, the rest is appended to the WV file, bloating it up. So the effective WAV size limit for Wavpack is also 4GB, compressing bigger files makes no sense. Btw, wavpack, will report the same compression ratio (99.96%) as before, but the compression takes longer since appending the additional chunk also takes quite some time. 

Fine. This confirms my understanding of WavPacks source.

I'd say stick to a 4GB limit for now. Any WAV file bigger than that is definitely not standard. (For example Adobe Audition warned before saving a WAV bigger than 4GB, telling me the file will only open in Audition.)

The file i was talking about contained a valid standard conforming first data chunk of 2 GB. Then followed another data chunk of up to 2 GB. This is non standard. Probably the application which generated this file partitions bigger files into multiple data chunks of 2 GB.

This (non-standard) variation can not be read by TAK, even if the whole file size is less than 4 GB.


TAK - Alpha testing

Reply #70
[deleted]

TAK - Alpha testing

Reply #71
[deleted]

TAK - Alpha testing

Reply #72
Edit 2: For the next "full" test (probably going to use a very wide eclectic mix of music, about 2.5GB of total samples), I would be in a position to include DTS-HD and Meridian MLP (Dolby TrueHD) results, if anyone is interested.
I would certainly be interested.  I'd like to see more testing with files that aren't 16bit 44100Hz stereo, but personally I have no resources.  Keep up the good work.
I'm on a horse.

TAK - Alpha testing

Reply #73
Here TrNSZ's latest results in the format i am used to:

Classical Test:
Dvorak: Silent Woods. Cello Concerto. Op. 104. in B Minor
Code: [Select]
                      Size (MB)  Ratio  Diff Best  Diff Tak 4m
Image (original).wav    697.95   100.00   -64.02   -63.55
Flac (-8).flac          271.57    38.91    -2.93    -2.46
True Audio.tta          269.78    38.65    -2.67    -2.20
Flake r112 (-12 -v 1)   269.52    38.62    -2.64    -2.17
WavPack (hhx6).wv       266.65    38.20    -2.22    -1.75
Extra Maximum.tak       254.41    36.45    -0.47     0.00
Monkey Insane.ape       251.13    35.98     0.00     0.47

Some rock/New Wave comparing only the strongest WavPack mode to TAK.
Code: [Select]
                             Size (MB)  Ratio  Diff Best  Diff Tak 4m
Senses-Working-Overtime.wav   720.86    100.00   -35.08   -35.08
0.tak                         477.05     66.18    -1.26    -1.26
0m.tak                        474.09     65.77    -0.85    -0.85
1.tak                         471.85     65.46    -0.54    -0.54
wavpack-hhx6.wv               471.33     65.38    -0.47    -0.47
2.tak                         469.94     65.19    -0.27    -0.27
1m.tak                        469.75     65.17    -0.25    -0.25
2m.tak                        469.37     65.11    -0.19    -0.19
3.tak                         468.89     65.05    -0.13    -0.13
4.tak                         468.37     64.97    -0.06    -0.06
3m.tak                        468.36     64.97    -0.05    -0.05
4m.tak                        467.96     64.92     0.00     0.00

TAK vs. MPEG-4 ALS and Flake -12 on more classical a very dynamic Telarc recording.
Code: [Select]
              Size (MB)  Ratio  Diff Best  Diff Tak 4m
Telarc.wav     595.86    100.00   -58.07   -58.07
Telarc.tta     274.26     46.03    -4.09    -4.09
flake-12.flac  272.75     45.77    -3.84    -3.84
ALS.mp4        271.58     45.58    -3.65    -3.65
ALS-p.mp4      267.92     44.96    -3.03    -3.03
0.tak          260.90     43.79    -1.85    -1.85
0m.tak         259.67     43.58    -1.65    -1.65
1.tak          258.56     43.39    -1.46    -1.46
1m.tak         257.27     43.18    -1.24    -1.24
2.tak          255.77     42.93    -0.99    -0.99
2m.tak         255.05     42.80    -0.87    -0.87
3.tak          252.10     42.31    -0.38    -0.38
3m.tak         251.37     42.19    -0.25    -0.25
4.tak          250.22     41.99    -0.06    -0.06
4m.tak         249.86     41.93     0.00     0.00

"Diff Best"  is the difference of the best result and the row.
"Diff Tak 4m" is the difference of Tak's strongest mode and the row.

Thank you so much for all your interesting and helpful work! We may have to stop you sometime... But not yet.

Edit: Note: MPEG-4 ALS using -7 -p and even just -7 is really too slow to do routine testing, but I am interested to seeing if it performs better, but the encoding is really, REALLY slow.  Slow enough to make WavPack -hhx6 or even OptimFROG look fast.  I didn't test the ALS decoding speeds.  I also experienced a few crashes using the latest MPEG-4 reference coder, mainly when enabling switches for extra compression.  It took several hours before actually crashing too, which makes it really frustrating.

Too bad. Just to be sure: You know, that you are not allowed to use the z-options together with -7 ? MPEG-4 ALS should warn you, but it doesn't.

Edit 2: For the next "full" test (probably going to use a very wide eclectic mix of music, about 2.5GB of total samples), I would be in a position to include DTS-HD and Meridian MLP (Dolby TrueHD) results, if anyone is interested.

Wow, that would be something really new and absolutely interesting for me!

Again, thanks for all your hard work!

  Thomas

TAK - Alpha testing

Reply #74
[deleted]

 
SimplePortal 1.0.0 RC1 © 2008-2019