Hi,
(sorry, my english isn't very good...)
i have been working (for fun) on lossless audio compression since about 1997. Finally i would like to bring this thing (especially the never-ending-search for just a tenth of a percent more compression...) to an end. In the light of the big bunch of existing Compressors, i am not quite sure, if it would be of any use to add one more Compressor to the public. The preparations for a useful release would be much further work, and i wouldn't like to waste my time for something not needed.
My Compressor uses similar techniques like FLAC, but far more elaborated. Compression ratios lie between Monkey's Audios High- and Extra-High-Mode (Can be better than Extra High at the expense of a considerable increase of encoding time). Encoding Speed is a bit slower than Monkey's, Decoding Speed is much higher on most Files. Seek-Times should also be better cause of the maximum (independent) frame length of 250 ms.
I would like to read some opinions. Would it make any sense to release it?
Thanks
Thomas
In the beginning, I really didn't understand lossless (this was back in 2003). I didn't really see the 10% compression ratio as being significantly smaller than the original wav and, to me, a much smaller mp3 had the same sound.
Now I understand the importance of lossless encoders in that they can retain tags (something wav files can't do), some offer better than 10% compression ratios, and many programs now support them.
I would welcome another lossless encoder. After all, competition is good. Your encoder may not use newer technology when compared with Apple lossless for Monkey's Audio but still, it would be nice to see.
I am currently in the process of ripping my entire library to Apple lossless (I live in a iPod world).
Then again, my interest is purely for curiousity. Sadly, I am afraid that many people are set with their lossless encoders with most people going with FLAC. Unless your lossless encoder would obtain a high compatibility with current software and hardware, then I really don't see it going anywhere beyond curiosity and testing. Please, don't take this in the wrong way what so ever. I would love to see a lossless format developed by a HA poster. I just don't know if it would have any practical use or not.
In my opinion, competition is always welcome. Take a look at the Linux world. More than 100 distros, but all distros are nearly the same at the heart (linux kernel + a lot of packages).
If the compressor has some "selling points" (such as a better-than-average compression ratio, or high performance, or both), then I believe it certainly deserves some mention, especially on a site that is dedicated to audio compression.
So please, go ahead.
TBeck,
Sounds interesting. I'm really happy to see another good lossless audio compressor available. Will you release your compressor under GPL license or something like?
Unless your lossless encoder would obtain a high compatibility with current software and hardware, then I really don't see it going anywhere beyond curiosity and testing. Please, don't take this in the wrong way what so ever. I would love to see a lossless format developed by a HA poster. I just don't know if it would have any practical use or not.
[a href="index.php?act=findpost&pid=377693"][{POST_SNAPBACK}][/a]
Thanks.
My Thinking goes into the same direction. Sigh...
Building of the compression engine has been very much work. The creation and promotion of a new (free) format, which seems to be necessary to make the technology useful, would be even more work. My biggest respect for Josh Coalson, who has made FLAC some standard.
I'm in doubt that i myself would be able to establish some new standard. And i'm not sure, if it would make sense. It's a pity, that i am too late. Some years ago my work possibly would have had a chance to enrich the development of FLAC.
Thomas
If the compressor has some "selling points" (such as a better-than-average compression ratio, or high performance, or both), then I believe it certainly deserves some mention, especially on a site that is dedicated to audio compression.
So please, go ahead.
[a href="index.php?act=findpost&pid=377694"][{POST_SNAPBACK}][/a]
Many thanks for the encouragement!
The selling points could be speed, low hardware requirements if someone wanted to implement the algorithms on a DSP (Digital Signal Processor, 16 Bit would be enough) and patent free algorithms (I hope so).
Compression efficiency should be on par or better than the most common audio compressors i know, that use forward prediction (FLAC, LPAC, MPEG4 Lossless if you use Mode -7).
It will never achieve the same Compression ratios as the compressors with forward prediction e.g. Optimfrog (possibly Optimfrog is some hybrid one, that uses both methods).
On the other hand it provides asymmetric Speeds: Decoding is much faster then encoding.
Thomas
TBeck,
Sounds interesting. I'm really happy to see another good lossless audio compressor available. Will you release your compressor under GPL license or something like?
[a href="index.php?act=findpost&pid=377710"][{POST_SNAPBACK}][/a]
I would like to do so, but possibly somewhat later. If I should go on with this project, the next step could be a (closed source) release of some simple compression tool, which would allow some public testing of the efficiency. If it should prove to be useful, it would make sense to invest more time into the development. Although my source code is quite clean, it would be much work to bring it into a form, that others could use. It's mainly written in Pascal (Borland Delphi) with some assembler. I have to translate it to C, to make it handy for other developers.
Thomas
Unless your lossless encoder would obtain a high compatibility with current software and hardware, then I really don't see it going anywhere beyond curiosity and testing. Please, don't take this in the wrong way what so ever. I would love to see a lossless format developed by a HA poster. I just don't know if it would have any practical use or not.
[a href="index.php?act=findpost&pid=377693"][{POST_SNAPBACK}][/a]
Thanks.
My Thinking goes into the same direction. Sigh...
Building of the compression engine has been very much work. The creation and promotion of a new (free) format, which seems to be necessary to make the technology useful, would be even more work. My biggest respect for Josh Coalson, who has made FLAC some standard.
I'm in doubt that i myself would be able to establish some new standard. And i'm not sure, if it would make sense. It's a pity, that i am too late. Some years ago my work possibly would have had a chance to enrich the development of FLAC.
Thomas
[a href="index.php?act=findpost&pid=377718"][{POST_SNAPBACK}][/a]
TBeck, you won't have to promote this new thing when it's good. the community will do this for you
en/decoding speed aren't the only points which make a format useable, what about tagging (RG-tags and the like) multi-channel support and stuff?
don't be afraid to announce your codec - if you din't your work would be lost!
regards, j~
Hi,
(sorry, my english isn't very good...)
i have been working (for fun) on lossless audio compression since about 1997. Finally i would like to bring this thing (especially the never-ending-search for just a tenth of a percent more compression...) to an end. In the light of the big bunch of existing Compressors, i am not quite sure, if it would be of any use to add one more Compressor to the public. The preparations for a useful release would be much further work, and i wouldn't like to waste my time for something not needed.
My Compressor uses similar techniques like FLAC, but far more elaborated. Compression ratios lie between Monkey's Audios High- and Extra-High-Mode (Can be better than Extra High at the expense of a considerable increase of encoding time). Encoding Speed is a bit slower than Monkey's, Decoding Speed is much higher on most Files. Seek-Times should also be better cause of the maximum (independent) frame length of 250 ms.
I would like to read some opinions. Would it make any sense to release it?
Thanks
Thomas
[a href="index.php?act=findpost&pid=377674"][{POST_SNAPBACK}][/a]
Hey, the 1st of April is today! It's not quite fair!
I would like to read some opinions. Would it make any sense to release it?
No, until it will provide some extraordinary features.
See here (http://wiki.hydrogenaudio.org/index.php?title=Lossless_comparison)
In my opinion, competition is always welcome. Take a look at the Linux world. More than 100 distros, but all distros are nearly the same at the heart (linux kernel + a lot of packages).
So please, go ahead.
[a href="index.php?act=findpost&pid=377694"][{POST_SNAPBACK}][/a]
yes go ahead.
(off topic: i personally think that one of linux problems is just that, 100 distributions, a lot of problems, a lot of wasted resources)
Hey, the 1st of April is today! It's not quite fair!
[a href="index.php?act=findpost&pid=377779"][{POST_SNAPBACK}][/a]
Huch... Did i miss something?
I would like to read some opinions. Would it make any sense to release it?
No, until it will provide some extraordinary features.
See here (http://wiki.hydrogenaudio.org/index.php?title=Lossless_comparison)
[a href="index.php?act=findpost&pid=377800"][{POST_SNAPBACK}][/a]
Would much higher speed be relevant?
I've worked on something similiar the last years.
my encoder uses an adaptive wavelet filterbank on the prediction
residuals.
it has id3v1 and apev2 support but i stopped developing because
i think nobody needs yet another lossless compressor.
best regards
Nobody needs another one that is only 1-2% better. Personally I will stick with FLAC until someone matches its features and at least 15% better.
Triza
An encoder with compression ratio as high as Monkey's Audio but faster at decoding would be certainly a nice thing. Count me as a first user if additionally it has these features:
1. Piping support (without requiring input's accurate lenght)
2. APEv2 tags support
3. Always up-to-date & not-hackish plugin for fb2k
4. Simple (no more than 2 switches for altering compression ratio & speed)
An encoder with compression ratio as high as Monkey's Audio but faster at decoding would be certainly a nice thing. Count me as a first user if additionally it has these features:
1. Piping support (without requiring input's accurate lenght)
2. APEv2 tags support
3. Always up-to-date & not-hackish plugin for fb2k
4. Simple (no more than 2 switches for altering compression ratio & speed)
[a href="index.php?act=findpost&pid=377878"][{POST_SNAPBACK}][/a]
Things like that are why I personally use WavPack. Encodes fast even on high mode, smaller than flac, and decodes only a bit slower. IMHO Monkey's Audio is a poor codec simply because it is slow as hell if you want decent compression. If you are going to try to compete it would be more reasonable to try and get compression ratios like OptimFrog, MAC or LA, but with WavPack or FLAC's encoding/decoding speeds along with APEv2 tags, proper seeking and other nice features. Then you'd have something worthy of competition.
you really don't need an official release right now - why don't you just pack up the source and have some people test it.
the bigger question is if it is something you want to continue maintaining it...
later
IMHO Monkey's Audio is a poor codec simply because it is slow as hell if you want decent compression. If you are going to try to compete it would be more reasonable to try and get compression ratios like OptimFrog, MAC or LA, but with WavPack or FLAC's encoding/decoding speeds along with APEv2 tags, proper seeking and other nice features. Then you'd have something worthy of competition.
..define slow as hell, if you would.. this is just a near-troll..
MAC extra high decodes @18~22x on my pc (athlon 2800+), and is probably the best choice for decoding speed vs. compression ratios there out there..
sure, OFR gets better compression ratios, but that's decoding at 3x playback or slower.. flac -8 is faster at decoding, but the compression ratios are significantly lower.. wavpack is better at compressing than flac, but still worse than MAC extra high (in that you can save GBs of space on modern sized HD), which i consider the benchmark here (since imo 20x playback is more than fast enough)..
LA i haven't tested, but the benchmarks seem to suggest it's impractically slow at comparable compression levels (<5x playback)
regardless.. you want: awesome compression, combined with high decoding speeds, and preferably encoding too.. obviously well-taggable, without bugs in it..
so basically you ask the world of someone who hasn't even released a .01 version of his en/decoder yet..
how encouraging this must be
anyway, TSer.. feel free to do with your free time what you want, i'm sure there'll be people here that are interested without saying you should invent cold fusion before you're allowed to post about it here
An encoder with compression ratio as high as Monkey's Audio but faster at decoding would be certainly a nice thing.
[a href="index.php?act=findpost&pid=377878"][{POST_SNAPBACK}][/a]
Agree here, but personaly I think that it will be quite hard to achieve high compression ratios while making format non-symmetric.
Agree here, but personaly I think that it will be quite hard to achieve high compression ratios while making format non-symmetric.
[a href="index.php?act=findpost&pid=377941"][{POST_SNAPBACK}][/a]
Yeah, i like asymmetrical Compression. Possibly time for some Data:
Participiants:
TAK (Toms Audio K©ompression) Alpha, unreleased
FLAC V 1.1.2 Win
Monkey V 3.99
OptimFrog V 4.520b [2006.03.02] (beta)
System:
Pentium III / 866 MHz
TAK FLAC Monkey 3.99 OptimFrog
Mode: High Extra Insane | -8 | High Extra | Best |
-----------------------------------+---------+-----------------+---------+
Song_02 48.41 47.87 47.73 | 51.03 | 48.00 47.28 | 47.88 |
Song_04 33.15 32.59 32.56 | 37.27 | 33.58 32.35 | 32.76 |
Song_06 33.74 33.34 33.20 | 37.04 | 33.77 33.09 | 33.01 |
Song_08 44.97 44.56 44.45 | 49.74 | 44.81 43.59 | 43.82 |
Song_10 56.41 56.00 55.94 | 59.10 | 55.95 54.97 | 55.03 |
Song_12 53.86 53.33 53.27 | 57.62 | 53.04 51.99 | 51.57 |
Song_14 48.97 48.51 48.44 | 51.87 | 48.65 47.76 | 47.69 |
Song_16 74.16 73.82 73.79 | 75.95 | 73.70 73.44 | 73.27 |
-----------------------------------+---------+-----------------+---------+
Sum: 47.86 47.41 47.33 | 51.35 | 47.62 46.70 | 46.81 |
EncoTime: 53.01 270.94 595.41 | --.-- | 57.15 109.02 | --.-- |
DecoTime: 13.50 14.90 15.19 | --.-- | 63.92 114.71 | --.-- |
-----------------------------------+---------+-----------------+---------+
Bach_01 58.29 54.40 53.20 | 64.67 | 60.05 55.42 | 59.27 |
Bartok_01 48.69 48.19 48.13 | 53.30 | 49.17 47.74 | 48.40 |
Debussy_01 26.17 26.04 25.96 | 28.55 | 26.48 26.12 | 25.59 |
Mahler_01 48.32 47.69 47.42 | 49.87 | 47.73 47.09 | 47.51 |
Speech_01 30.96 30.46 30.17 | 34.15 | 31.81 31.78 | 29.10 |
-----------------------------------+---------+-----------------+---------+
Sum: 42.50 41.40 41.02 | 46.00 | 43.02 41.62 | 42.04 |
EncoTime: 8.87 46.97 104.87 | --.-- | 9.23 18.12 | --.-- |
DecoTime: 2.20 2.56 2.65 | --.-- | 11.74 20.52 | --.-- |
-----------------------------------+---------+-----------------+---------+
S_24_44_01 59.56 59.41 59.38 | 61.69 | 59.23 58.97 | --.-- |
S_24_44_02 62.27 62.08 62.05 | 65.87 | 61.85 61.70 | --.-- |
S_24_44_03 52.42 52.18 52.15 | 54.65 | 52.32 51.88 | --.-- |
S_24_48_01 62.86 62.58 62.51 | 65.16 | 62.83 62.45 | --.-- |
S_24_48_02 54.27 54.13 54.09 | 55.89 | 53.99 53.92 | --.-- |
S_24_48_03 78.72 78.57 78.52 | 97.93 | 79.26 79.30 | --.-- |
-----------------------------------+---------+-----------------+---------+
Sum: 58.13 57.94 57.90 | 60.88 | 57.89 57.65 | --.-- |
EncoTime: 23.37 113.12 247.91 | --.-- | 28.01 49.20 | --.-- |
DecoTime: 5.79 6.21 6.28 | --.-- | 30.83 53.37 | --.-- |
-----------------------------------+---------+-----------------+---------+
I don't know, how long a post can be. So to be continued in the next post.
Thomas
The Table in the previous post contains results from my main test corpus,
which is divided into three parts:
Song_02..Song_16
CD-Ripps (Rock, Pop, Songwriter), 44 KHz, 16 Bit, about 90 Seconds per file.
Bach_01..Speech_01
Classic music from a compression test side, 44 KHz, 16 Bit, about 30 Seconds
per file.
S_24_44_01..S_24_48_03
Some Samples from a ADC/DAC (Analog-Digital-Converter)-Test-Side. 44/48 KHz, 24 Bit.
I have tested far more files in the past, but these ones have proven to be
the most representative ones.
You can see the relative compression ratio for each single file. Sum is the
mean Compression for each part, Enco and DecoTime is Duration of Encoding and
Decoding in seconds.
You can easily see the assymmetry in encoding vs. decoding time.
Thomas
Truly speaking I'm little bit impressed. The ratios are very close to MA while providing fast decoding! Thats realy cool ! Can we get alpha version of your TAK to play with?
By the way I advise you to change the name to something else cause the whole name is little bit complicated and further more there were some lossy project called TAC from K+K Research so some people can be confused.
-----------------------------------+---------+-----------------+---------+
Sum: 58.13 57.94 57.90 | 60.88 | 57.89 57.65 | --.-- |
EncoTime: 23.37 113.12 247.91 | --.-- | 28.01 49.20 | --.-- |
DecoTime: 5.79 6.21 6.28 | --.-- | 30.83 53.37 | --.-- |
-----------------------------------+---------+-----------------+---------+
[a href="index.php?act=findpost&pid=377983"][{POST_SNAPBACK}][/a]
That's good. Too good. It's indeed truly remarkable!
What day is today again?
Truly speaking I'm little bit impressed. The ratios are very close to MA while providing fast decoding! Thats realy cool ! Can we get alpha version of your TAK to play with?
By the way I advise you to change the name to something else cause the whole name is little bit complicated and further more there were some lossy project called TAC from K+K Research so some people can be confused.
[a href="index.php?act=findpost&pid=378000"][{POST_SNAPBACK}][/a]
Nice. That's very motivating!
Unfortunately this Alpha version definitely needs some further work, before i want to give it away. It still generates some single bit errors on some files. It's not a conceptual problem, so compression efficiency will not be reduced by the necessary corrections. Furthermore my code for reading the source (wave) files is very rudimentary and could possibly fail on some files. I don't think, that the actual release would make a really good impression.
But i will try to provide a working evaluation version as soon as possible. If nothing unexpected will happen, this should be done in about one month.
TAK was the first thing that came to my mind. And it did fit into the table, so i used it. But confusion with other projects would be bad, so i will look for another name.
Thanks
Thomas
-----------------------------------+---------+-----------------+---------+
Sum: 58.13 57.94 57.90 | 60.88 | 57.89 57.65 | --.-- |
EncoTime: 23.37 113.12 247.91 | --.-- | 28.01 49.20 | --.-- |
DecoTime: 5.79 6.21 6.28 | --.-- | 30.83 53.37 | --.-- |
-----------------------------------+---------+-----------------+---------+
[a href="index.php?act=findpost&pid=377983"][{POST_SNAPBACK}][/a]
That's good. Too good. It's indeed truly remarkable!
What day is today again?
[a href="index.php?act=findpost&pid=378007"][{POST_SNAPBACK}][/a]
No, no, no... I should have carefully looked into my calendar before posting... Bad timing.
But it's true. The whole design of the compressor was made for speed. When i started with the work (1997) my pc was simply too slow to perform the necessary test runs for evaluation in tolerable time. Then i got my first Pentium MMX with 200 MHz. And that it was: MMX made the things 2 to 3 times faster. So i built the whole compressor for the 16 Bit-Arithmetic, that MMX allowed. To be exact, most calculations are beeing done in 13 Bit.
This choice makes things fast, but the reduced resolution sacrifies some compression efficiency. I too did a comparison with MPEG4 Lossless (mpeg4als). If you use the switch -7, you activate a asymmetric compression mode which is very similar to my Compressor or FLAC. But it's about 0.40 percent better on my test corpus than TAC. One reason seems to be the use of 64-Bit arithmetic, which on the other hand is quite slow.
And to lower my credibility just a little bit more: With the SSE3 instruction set found in newer X86-CPU's, Tac could become even faster, because it provides 8 instead of 4 of the necessary calculations simultaniously...
Should i officially confirm my posts when the 1.4. is over?
Thomas
But it's true.
Ok, I'm sorry. But cut me some slack. You register here out of nowhere right on April 1st, comes up with a completely groudbreaking innovation in lossles encoding, and still asks "Would it make any sense to release it?"
Should i officially confirm my posts when the 1.4. is over?[a href="index.php?act=findpost&pid=378017"][{POST_SNAPBACK}][/a]
It's over already in Germany, right?
But it's true.
Ok, I'm sorry. But cut me some slack. You register here out of nowhere right on April 1st, comes up with a completely groudbreaking innovation in lossles encoding, and still asks "Would it make any sense to release it?"
Should i officially confirm my posts when the 1.4. is over?[a href="index.php?act=findpost&pid=378017"][{POST_SNAPBACK}][/a]
It's over already in Germany, right?
[a href="index.php?act=findpost&pid=378021"][{POST_SNAPBACK}][/a]
Yes. And i confirm, that it isn't a joke!
I didn't rate my work as groundbreaking. I know for example, that the highest compression modes of OptimFrog give up to 2 percent better compression than TAC. Nevertheless i am a bit proud of the speed of my compressor. That's true.
But i understand, that my post could be a bit suspicious. Possibly a successful troll would even present something a bit surprising instead of something hard to believe. Sigh...
But i initially wanted some motivation to force the release of my compressor. And in this context the need to prove, that my compressor isn't Vaporware, is useful.
Thomas
Yes. And i confirm, that it isn't a joke!
I didn't rate my work as groundbreaking.
Go ahead, and see how well your baby fares.
Building of the compression engine has been very much work. The creation and promotion of a new (free) format, which seems to be necessary to make the technology useful, would be even more work.
I would agree with that... from my experience, development of the actual compression algorithm takes the least part of time for a successful codec. algorithm-wise FLAC is not that much different than shorten. the vast majority of time for me was spent in trying to make it useful (format spec, portable libraries, docs, test suites, all the features people want in a codec like a metadata system, support tools, etc.) and external stuff like project administration, releases, ...
I'm in doubt that i myself would be able to establish some new standard. And i'm not sure, if it would make sense. It's a pity, that i am too late. Some years ago my work possibly would have had a chance to enrich the development of FLAC.
it still could if it's compatible with the FLAC goals (http://flac.sourceforge.net/goals.html); it's not too late. your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.
Josh
This discussion reminds me of that episode of the Simpsons where Bart had that Dad-at-Large and Homer had that kid-at-large and they were competing against each other for some reason. Anyway, after that big fight Homer had with Bart's "bigger-brother", when Bart and Homer got back together again, and right at the end, when the kid (I think his name was Pepe) and the bigger-brother were sitting at the curb lamenting about how crappy things were for them (Pepe had no dad now, and the bigger-brother had all this food that would go to waste) and they didn't realize that they should partner up since the Pepe kid was perfect with the super-good bigger-brother guy. Then Bart introduced them to each other and there was a happy ending. Let me be like Bart here, and encourage TBeck to join up with the FLAC team, and...... Well you get the idea.
Furthermore my code for reading the source (wave) files is very rudimentary and could possibly fail on some files.
if you like i can send you a c++-class which can read
Mu/Alaw 8,16,24,32-bit wave,wave-extended and aiff files...
IHMO, if you could transform your ideas into some methods / a library for other lossless codec, and thus enhancing them, it would be more appreciate by the user than a brand new codec.
Hey, that's really something special! I don't really think your codec will survive in an ongoing lossless competition without some extra features and/or promotion, but you could join up with either of the open-source codec developers to try to merge your ideas into one but very effective codec (if that is possible, of course).
Anyway, good luck with your work.
Let me be like Bart here, and encourage TBeck to join up with the FLAC team, and..
[a href="index.php?act=findpost&pid=378087"][{POST_SNAPBACK}][/a]
Oh god, I'm afraided it but why FLAC? Don't think that I'm flaming but I just don't like direct biasing.
I don't really think your codec will survive in an ongoing lossless competition without some extra features and/or promotion
[a href="index.php?act=findpost&pid=378187"][{POST_SNAPBACK}][/a]
Agree, but partly. Lossless compressors are always have been pieces of software mainly for experinced and interested users so even if your compressor will have only one extra feuture - it will have success, although among quite limited group of users.
For TBeck I can only advise the following:
+ Make your compressor open-source to spread it everywhere without limits
+ First of all develope the format. In ideal it must:
- support any kind of input files. 8/16/24/32 bit/IEEE float, mono/2 channel/multichannel, any sample rate.
- support non-audio data embeded into files (RIFF chunks)
- be error resistant. At least error detection and decode-throught-errors, at most the error correction.
- provide tagging possibility. APEv2 tags are highly recomended.
- be asymmetric but providing high compression ratios, same or better than Monkey's Audio.
- be very fast at decoding and to be hardware friendly
- provide ReplayGain support
- provide possibility to improve compression ratios without breaking the backward compatibility
- provide full Unicode support
+ After you developed the format just provide native reference compilations for
most well known platforms and also plugins for at least 2-3 famous players on
such platforms. Also support for burning/editing applications will be just great.
Good luck !
I would agree with that... from my experience, development of the actual compression algorithm takes the least part of time for a successful codec. algorithm-wise FLAC is not that much different than shorten. the vast majority of time for me was spent in trying to make it useful (format spec, portable libraries, docs, test suites, all the features people want in a codec like a metadata system, support tools, etc.) and external stuff like project administration, releases, ...
I'm in doubt that i myself would be able to establish some new standard. And i'm not sure, if it would make sense. It's a pity, that i am too late. Some years ago my work possibly would have had a chance to enrich the development of FLAC.
it still could if it's compatible with the FLAC goals (http://flac.sourceforge.net/goals.html); it's not too late. your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.
Josh
[a href="index.php?act=findpost&pid=378083"][{POST_SNAPBACK}][/a]
Hello Josh,
very glad to read this "It's not too late..."! To be honest, there is an EMail in my draft box, that should have been sent to the FLAC Dev List, but never found its way...
I'm aware of the really big work needed to make a good format. And to be honest, i wouldn't like to do this. Especially as it would be some reinvention of the wheel and so quite useless.
It has been some time since i last carefully read the flac goals, but if there wasn't a big change, there should be no problem.
I didn't provide speed data for flac in my comparison, cause i was a bit in a hurry. But i will add that soon. It should be no problem to reach the speed advantage needed, cause the fastest of my compression modes allready uses 128 Predictors, which could easily be changed to 64 or 32 too get more speed.
If it would be possible to to add my compression methods (possibly as an alternative to keep backwards compatibility) to FLAC, i would be very happy.
But it would take some time and i would need help.
My code is written in Borland delphi (pascal) and nasm. I', not a wizzard in c, so it will take some time to translate it. And i don't know nearly nothing about programming for platform compatibility. Last but not least, my skills in writing english aren't very good, so a proper documentation would't be easy. Would there be help from the flac-dev-community? Could it be practicable?
Very exited
Thomas
For TBeck I can only advise the following:
+ Make your compressor open-source to spread it everywhere without limits
+ First of all develope the format. In ideal it must:
- support any kind of input files. 8/16/24/32 bit/IEEE float, mono/2 channel/
multichannel, any sample rate.
- support non-audio data embeded into files (RIFF chunks)
- be error resistant. At least error detection and decode-throught-errors, at
most the error correction.
- provide tagging possibility. APEv2 tags are highly recomended.
- be asymmetric but providing high compression ratios, same or better than
Monkey's Audio.
- be very fast at decoding and to be hardware friendly
- provide ReplayGain support
- provide possibility to improve compression ratios without breaking the
backward compatibility
+ After you developed the format just provide native reference compilations for
most well known platforms and also plugins for at least 2-3 famous players on
such platforms. Also support for burning/editing applications will be just great.
Good luck !
[a href="index.php?act=findpost&pid=378215"][{POST_SNAPBACK}][/a]
One more thing: Please make it Unicode-based, or at least support filenames in multibyte chars. That's important for i18n, ie for the whole world includking China Japan Korea to enjoy your app!!
wavpack and ttaenc didn't like multibyte-char filenames before (the problem was already fixed now tho) and was not able to compress the file when the filename contains a double-byte char whose 2nd half is 0x5C, misunderstanding it as \
I'm looking forward to the Alpha release of this!
it still could if it's compatible with the FLAC goals (http://flac.sourceforge.net/goals.html); it's not too late. your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.
Josh
[a href="index.php?act=findpost&pid=378083"][{POST_SNAPBACK}][/a]
Here some more comparison now including Timings for FLAC.
Cause i had to use a stop watch for flac, i only tested the
biggest files to reduce the effect of the measuring equipment
(my thumb isn't too accurate...).
The first two columns now include some variations of the HIGH-Mode
with reduced predictor numbers. They could compress better without
increasing decoding time, if they used the same methods for paramter
calculation as Extra or Insane modes. I just wasn't to lazy.
The bottom rows show the speed, that TAK achieves without the use
of MMX-Instructions or general assembler optimizations. This could be
interesting, if the code would be ported to some platform without such
instructions.
TAK TAK TAK FLAC
Mode: High High High Extra Insane | -8 |
Predictors: 32 64 128 256 384 | |
---------------------------------------------------+---------+
Song_02 48,96 48,69 48.41 47.87 47.73 | 51.03 |
Song_04 34,19 33,84 33.15 32.59 32.56 | 37.27 |
Song_06 34,30 34,04 33.74 33.34 33.20 | 37.04 |
Song_08 45,74 45,31 44.97 44.56 44.45 | 49.74 |
Song_10 56,84 56,71 56.41 56.00 55.94 | 59.10 |
Song_12 54,13 53,93 53.86 53.33 53.27 | 57.62 |
Song_14 49,14 49,07 48.97 48.51 48.44 | 51.87 |
Song_16 74,17 74,16 74.16 73.82 73.79 | 75.95 |
---------------------------------------------------+---------+
Sum: 48,40 48,15 47.86 47.41 47.33 | 51.35 |
---------------------------------------------------+---------+
Times with the use of MMX: | |
EncoTime: 41,59 45,14 53.01 270.94 595.41 | 191.00 |
DecoTime: 11,35 12,24 13.50 14.90 15.19 | 20.00 |
---------------------------------------------------+---------+
Times without the use of MMX: | |
EncoTime: 63,08 73,87 91,65 638,81 1350,89 | ---.-- |
DecoTime: 16,40 19,46 24,77 31,46 33,38 | ---.-- |
---------------------------------------------------+---------+
To be honest, the comparison of the three leftmost columns of TAK with FLAC isn't
quite fair, because TAK actually doesn't measure the time needed for Disk-IO (my
40 GB Disk is quite slow). This shouldn't significantly affect the validity of
the other two modes, were the calculation overhead is much higher.
Possibly interesting to see, that the MMX-Implementation of the Modes with
low predictor order isn't considerably faster than the implementation without.
That's caused by some overhead introduced by the scaling and other preparations
of the data needed for the use of MMX.
Thomas
Is there any way we can get a sourcecode release of this to examine / integrate in other projects?
If I understand your table correctly, you are saying that your compressor (using the 32 predictors) compresses about 3% better than FLAC (-8) and does this with significantly faster encode and decode speeds. And it does this using pure Pascal with no optimized assembly.
Assuming this is correct, I think that your best (and easiest) course of action would simply be to publish a paper on the superb method(s) that you have discovered, become famous, and let others bother with all the unpleasant implemention details. It would obviously be a shame to let this go to waste, especially because the magnitude of the method's improvement (0.5 bits per sample!) could have ramifications across many disciplines.
However, I would stongly suggest that you fix that random bit error issue before getting too excited. It might be that those error bits actually need to be transmitted in the bitstream, wiping out your advantage in the process!
Congratulations!
Well, I am currently performing a Lossless Compression Test (described in this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=42734)).
My test system relies on scripts and the computer's timers so there's no thumb-accuracy issue
If you can possibly send me a link (via PM if you prefer) to download your binary then I will objectively contest your algorithm against some others I am testing... my test covers both encoding and decoding tests.
If I understand your table correctly, you are saying that your compressor (using the 32 predictors) compresses about 3% better than FLAC (-8) and does this with significantly faster encode and decode speeds. And it does this using pure Pascal with no optimized assembly.
Assuming this is correct, I think that your best (and easiest) course of action would simply be to publish a paper on the superb method(s) that you have discovered, become famous, and let others bother with all the unpleasant implemention details. It would obviously be a shame to let this go to waste, especially because the magnitude of the method's improvement (0.5 bits per sample!) could have ramifications across many disciplines.
However, I would stongly suggest that you fix that random bit error issue before getting too excited. It might be that those error bits actually need to be transmitted in the bitstream, wiping out your advantage in the process!
Congratulations!
[a href="index.php?act=findpost&pid=378259"][{POST_SNAPBACK}][/a]
Many Thanks!
You are totally right. In the past many revolutionary improvements of my compression efficiency were really bugs. But i did fix the error last night. The maximum reduction in compression efficiency after the fix is 0.01 percent.
I'm not quite sure if this would be the right place to release details of my private life...But...
I'm working self employed (mainly developing software for psychological research) and this is actually not going very well.
I'm suffering from the fact, that i don't have a formal education in informatics and not many references.
So i would like to get some profit out of the publication of my compressor in form of some reputation. I don't know, what could be the best way to achieve this goal.
Thomas
If I understand your table correctly, you are saying that your compressor (using the 32 predictors) compresses about 3% better than FLAC (-8) and does this with significantly faster encode and decode speeds. And it does this using pure Pascal with no optimized assembly.
Assuming this is correct, I think that your best (and easiest) course of action would simply be to publish a paper on the superb method(s) that you have discovered, become famous, and let others bother with all the unpleasant implemention details. It would obviously be a shame to let this go to waste, especially because the magnitude of the method's improvement (0.5 bits per sample!) could have ramifications across many disciplines.
However, I would stongly suggest that you fix that random bit error issue before getting too excited. It might be that those error bits actually need to be transmitted in the bitstream, wiping out your advantage in the process!
Congratulations!
[a href="index.php?act=findpost&pid=378259"][{POST_SNAPBACK}][/a]
One more:
You read the table right. The reduction of the calculation accuracy to 13 bit, that was necessary for the MMX-implementation, helps the pascal implemenation too, because the accumulator for the results of the many predictor * sample multiplications only has to be 32 bit wide instead of the 64 bit needed for many other compressors. So the compiler can use the faster x * y = eax instead of x * y = edx:eax instructions. And the use of this instruction allows pipelining (CPU) of the multiplications which isn't possible wit a 64 bit result. So on the pentium 3 you can get a troughput of 1 multiplication per clock cycle with 32 bit instead of 4 clock cycles per every single multiplication for 64 bit.
The higher compression efficiency of my compressor is based on many new or optimized methods. It's the sum of single improvements from about 0.1 to 0.5 percent each.
Thomas
I'm sure many people will wish to donate for using your compressor. Also, you could always make commercial licenses to the product / codec need to be paid.
So i would like to get some profit out of the publication of my compressor in form of some reputation. I don't know, what could be the best way to achieve this goal.
[a href="index.php?act=findpost&pid=378266"][{POST_SNAPBACK}][/a]
Well, again, a paper would probably be the best path for academic recognition. If you don't feel comfortable writing it yourself, you might team up with someone else in the field like Tilman Liebchen who is right there in Germany. I suspect that he would be interested in talking to you about your methods.
Of course, it would always be a good idea to quickly write them all out in letter form and mail it to yourself, just to be protected (there's probably a more modern way to do that, but you get the idea.)
Possibly time for some summary. Especially because it seems impracticable to anwer to every single post.
First thanks for the many encouraging words! They have provided me with some new motivation needed to continue and actually force the progress of my work.
I'm not quite sure what the future will bring. Actually my favourite option would be the inclusion of my technology into an existing free project like FLAC (if possible). But this will take a considerable amount of time (cleaning up and translation of my code to C, writing some documentation).
But the first thing on my priority list is the release of some evaluation version without any gimmicks and without any guarantee for further support of this first incarnation of the format.
I'm feeeling more and more bad because i provide promising results without the public prove of function. So this thing has to be done fast.
And i will enable my email-adress for this forum, if someone wants to send me some personal mail.
It includes the adress of my home page. But be warned. It is only written in german and doesn't contain any audio compression software yet.
Don't take me wrong. This post schould not stop this discussion! I would be really glad to receive further encouragement or constructive critics! This post is only a summary of my present thinking.
Thomas
it still could if it's compatible with the FLAC goals (http://flac.sourceforge.net/goals.html); it's not too late. your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.
Here some more comparison now including Timings for FLAC.
Cause i had to use a stop watch for flac, i only tested the
biggest files to reduce the effect of the measuring equipment
(my thumb isn't too accurate...).
Thomas, this definitely looks promising. the design of FLAC allows new methods to be added while preserving backwards compatibility. the main reasons I haven't added any so far is because they all either 1) had too little compresion gain; 2) significantly increased decode complexity; 3) were patent encumbered.
it's hard to say in this crazy world but assuming you're method doesn't run afoul of some patent (will take some research), you seem to have overcome all 3. there are other methods I have not added to FLAC just because the compression increase wasn't worth it, since the code is running in many devices and it would be some work to get it out to all the manufactures. but a consistent +10% would be worth it; at the same time I could add all the little stuff I haven't put in yet.
as you know FLAC and libFLAC are meant to be as open and free as possible so if you are OK with that, there is no reason your method cannot be incorporated. (if you're looking to make some economic benefit from it, that's much harder. the only I options see for that are to join a giant like dolby or free it and use any success of it as a reference.)
the next step would be to go over the details of your method. that could be done here or on the flac -dev list (http://lists.xiph.org/mailman/listinfo/flac-dev) where more FLAC devs would see it.
Josh
Thomas, this definitely looks promising. the design of FLAC allows new methods to be added while preserving backwards compatibility. the main reasons I haven't added any so far is because they all either 1) had too little compresion gain; 2) significantly increased decode complexity; 3) were patent encumbered.
it's hard to say in this crazy world but assuming you're method doesn't run afoul of some patent (will take some research), you seem to have overcome all 3. there are other methods I have not added to FLAC just because the compression increase wasn't worth it, since the code is running in many devices and it would be some work to get it out to all the manufactures. but a consistent +10% would be worth it; at the same time I could add all the little stuff I haven't put in yet.
as you know FLAC and libFLAC are meant to be as open and free as possible so if you are OK with that, there is no reason your method cannot be incorporated. (if you're looking to make some economic benefit from it, that's much harder. the only I options see for that are to join a giant like dolby or free it and use any success of it as a reference.)
the next step would be to go over the details of your method. that could be done here or on the flac -dev list (http://lists.xiph.org/mailman/listinfo/flac-dev) where more FLAC devs would see it.
Josh
[a href="index.php?act=findpost&pid=378368"][{POST_SNAPBACK}][/a]
Good News!
The patent thing is a really crazy one! To my knowledge there shouldn't be any problems with my Methods. But who knows.
As i wrote earlier, the improvements are mainly a sum of many optimizations and tricks. So even if there would be a need to remove one or two of them because of patent issues, there should still be a considerable advantage left.
I did try to avoid any patent issues. So i played a bit around with arithmetic encoding, which gave me about 0.9 percent better compression than standard rice. Then i dropped this thing and generated a variation of rice encoding, which gave me at least 0.6 percent more than standard rice. But may be my variation is patented, i don't know for sure.
Actually i'm working hard on a evaluation release. It could be done very soon, cause i dropped my earlier plan to first complete a really usable format.
After this is done, i would like to join the flac-dev-list.
Thomas
If nothing unexpected happens, i will release an evaluation version of my compressor within the next days. It will be a very quick and dirty thing. No error handling, only support for 44 KHz, 16 Bit Stereo and an ugly user interface.
To be honest, i never before did write compressed files to nor did i decompress files from the disk. All this had be done in RAM. So there has been a very little chance, that i faked myself about the compression efficiency. But after some strange errors and much excitement i could compare the first files and everthing is ok!
Would it be possible to store the program archive (less than 1 MB) at hydrogen audio? My free homepage traffic is very limited, so i would be glad about an alternative.
Very exhausted
But happy
Thomas
If none of the moderators object, you can use the Uploads (http://www.hydrogenaudio.org/forums/index.php?showforum=35) section of HA and give a link to the archive in a post here.
This development is really interesting. Personally, I think a collaboration will benefit both the community and yourself more than doing a one man show. If (for example) FLAC can benefit from your findings, you already have a well-known and quite established format to put in your CV.
TBeck, could you describe your basic algorithm ideas in short? Will you publish a paper?
TBeck, could you describe your basic algorithm ideas in short? Will you publish a paper?
[a href="index.php?act=findpost&pid=378512"][{POST_SNAPBACK}][/a]
There isn't any paper yet. I have been working on this since about 1997, but never even talked about it. Ok, if my friends wanted to invite me for some beer, i told seem, that i couldn't go with them, because i was near to a breakthrough in audio compression technology (This never really happens...).
My decision to go into public was a spontaneous one, in no way prepared. So actually there is no paper.
My improvements are mostly the sum of many single optimizations. One of them may be remarkable, because it allowed me to use reduced precision arithmetic without a significant penality in compression efficiency. It's so simple, that i could explain it in one short sentence. But i don't want to publish details yet.
Sorry... It will take some time until i will (and can) publish any details.
Thomas
Would it be possible to store the program archive (less than 1 MB) at hydrogen audio? My free homepage traffic is very limited, so i would be glad about an alternative.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=378503")
I think Rjamorim will be glad to put it on [a href="http://www.rarewares.org]Rarewares[/url]
if you're looking to make some economic benefit from it, that's much harder. the only I options see for that are to join a giant like dolby or free it and use any success of it as a reference.
At first I wanted to recommend you to approach some company that is into media hard- and software that will want to add proprietary lossless to its bouquet of formats in the future, while at the same time not „infect” its assets with open-source → SONY.
But seeing it might be viable your work is added as a new method to libflac is a very appealing thought. Putting TAK standalone into the open-source world might only lead to more format fragmentation, with a miniscule adoption share for TAK unless you want to get into direct concurrence with FLAC, Wavpack, APE and so on...and win.
BTW, I'm doing Cognitive Science in OS, they are quite blossoming and heavily software-dependent, so possible devel opportunities?
if you're looking to make some economic benefit from it, that's much harder. the only I options see for that are to join a giant like dolby or free it and use any success of it as a reference.
BTW, I'm doing Cognitive Science in OS, they are quite blossoming and heavily software-dependent, so possible devel opportunities?
[a href="index.php?act=findpost&pid=378904"][{POST_SNAPBACK}][/a]
I'm not quite sure, if i understand it right. If your question is: Would you like to do some work for us, the answer would be: Definitely yes!
Otherwise please clarify your question for me.
Nice to meet someone from osnabrück in this global village!
Thomas
BTW, I'm doing Cognitive Science in OS, they are quite blossoming and heavily software-dependent, so possible devel opportunities?
[a href="index.php?act=findpost&pid=378904"][{POST_SNAPBACK}][/a]
I'm not quite sure, if i understand it right. If your question is: Would you like to do some work for us, the answer would be: Definitely yes!
Otherwise please clarify your question for me.
Nice to meet someone from osnabrück in this global village!
Thomas
[a href="index.php?act=findpost&pid=378908"][{POST_SNAPBACK}][/a]
No, no. I did my B.Sc. at Osnabrück but am shifting to Tübingen currently. The CogSci institute is quite well funded currently by the DAAD and so on. They need a lot of tools for experiments like the Psi (although they train their students to do a lot themselves). So, if Kuhl has everything he needs by now, you could approach them instead (i.e. no offer but an idea)
BTW, I'm doing Cognitive Science in OS, they are quite blossoming and heavily software-dependent, so possible devel opportunities?
[a href="index.php?act=findpost&pid=378904"][{POST_SNAPBACK}][/a]
I'm not quite sure, if i understand it right. If your question is: Would you like to do some work for us, the answer would be: Definitely yes!
Otherwise please clarify your question for me.
Nice to meet someone from osnabrück in this global village!
Thomas
[a href="index.php?act=findpost&pid=378908"][{POST_SNAPBACK}][/a]
No, no. I did my B.Sc. at Osnabrück but am shifting to Tübingen currently. The CogSci institute is quite well funded currently by the DAAD and so on. They need a lot of tools for experiments like the Psi (although they train their students to do a lot themselves). So, if Kuhl has everything he needs by now, you could approach them instead (i.e. no offer but an idea)
[a href="index.php?act=findpost&pid=378912"][{POST_SNAPBACK}][/a]
Now it's clear.
Thanks
Thomas
Would it be possible to store the program archive (less than 1 MB) at hydrogen audio? My free homepage traffic is very limited, so i would be glad about an alternative.
As was pointed out, the upload section should work nicely.
If you need other hosting related things, feel free to PM about what you need, and I'll see what I can do.
Would it be possible to store the program archive (less than 1 MB) at hydrogen audio? My free homepage traffic is very limited, so i would be glad about an alternative.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=378503")
I think Rjamorim will be glad to put it on [a href="http://www.rarewares.org]Rarewares[/url]
Thomas, I'm sure if you PM, or mail, Roberto (rjamorim) he'll happily sort out something on Rarewares for you. He's away over the Easter period but will be back shortly thereafter.
Links to 24 bit files i have usedI think, they are hard to find.
44 KHz, 24 bit
mytek_8X96_24bit_web.wav
Mytek-stereo96adc_evans.wav
Mytek-stereo96adc_ravel.wav
Source: http://www.mytekdigital.com/compare/comparison1.htm
48 KHz, 24 bit
McDougalsMen24bit_48kHz.wav
sister24bit_48kHz.wav
Source: http://ff123.net/samples.html
Would it be possible to store the program archive (less than 1 MB) at hydrogen audio? My free homepage traffic is very limited, so i would be glad about an alternative.
As was pointed out, the upload section should work nicely.
If you need other hosting related things, feel free to PM about what you need, and I'll see what I can do.
Thanks.
Thomas, I'm sure if you PM, or mail, Roberto (rjamorim) he'll happily sort out something on Rarewares for you. He's away over the Easter period but will be back shortly thereafter. smile.gif
Thanks to John to.