After an exceedingly long apparent pause, the OptimFROG 5.000 stable release is finally ready!
Summary of changes:
- several new supported platforms, including SSE2 enabled builds;
- a large number of internal code updates and improvements;
- various speed improvements for all compression modes;
- updated SDK layout, documentation, and website content;
- a new unified, simplified, and more permissive license.
You can check all the new release packages on the Downloads (http://LosslessAudio.org/Downloads.php) page and the comprehensive list of changes on the Changes (http://LosslessAudio.org/Changes.php) page.
More information about the OptimFROG Lossless / DualStream / IEEE Float Audio Codec can be found on the OptimFROG website (http://LosslessAudio.org/).
Code name: Lazarus? Welcome back. I downloaded a copy to kick the tires.
Good to see this. Welcome back.
Thank you, and nice to find you here again! Indeed, time-wise the release code name would be a good fit, however I personally have never actually left the field.
During my long public absence I have actively worked on my PhD thesis about lossless audio compression, "Modeling and Optimization for Lossless Audio Coding with Flexible Complexity Profiles", and I am very happy to say that the end result is a completely new codec member, OptimFROG Asymmetric: from compression ratios similar to OptimFROG maximum compression experimental, while being 5 up to 20 times faster at decoding, and going down to decoding speeds in the range of FLAC.
There is work slowly going on about the bitsream format finalization and analysis, and also integration and harmonization of different codec modules, but I definitely plan to release the full bitstream format specification and the official open-source reference implementations within a time frame of 6 to 10 months from now.
On the short term, I am currently working on finalizing the port of the entire OptimFROG release package for Android with armv7, mips, and x86 platforms.
All feedback and especially help with creating new player plug-ins and improving the range of software support for @OptimFROG (https://twitter.com/OptimFROG) would be greatly appreciated!
I noticed that, like in previous versions, the MD5 hash for OFR files still don't display in the File Properties within foobar2000, forcing me to use OFR.exe to see those hashes via command line. Would this be the fault of the OptimFROG plugin that foobar2000 has to use? TAK used to have a similar problem until its plugin was updated.
Also noticed that foobar2000 still refuses to let OptimFROG Dualstream use pipe encoding, which forces you to use the old school temp wav encoding option just to make it work. Oddly, you can use that option via command line with ofr.exe
I noticed that, like in previous versions, the MD5 hash for OFR files still don't display in the File Properties within foobar2000, forcing me to use OFR.exe to see those hashes via command line. Would this be the fault of the OptimFROG plugin that foobar2000 has to use? TAK used to have a similar problem until its plugin was updated.
The MD5 hash for an OFR file is currently not available through the OptimFROG SDK interface, therefore plug-ins cannot obtain this stored MD5 hash value. I will update the SDK interface and also the foobar2000 plug-in in order to support this new and useful feature. Thank you!
Also noticed that foobar2000 still refuses to let OptimFROG Dualstream use pipe encoding, which forces you to use the old school temp wav encoding option just to make it work. Oddly, you can use that option via command line with ofr.exe
The ofr and ofs programs fully support pipes and should work identically in the foobar2000 Converter, without a temporary file. This requires adding the command-line option --incorrectheader, which allows to correctly encode data with a generic WAV header, having a fixed data chunk size of around 4 GB.
The compressed file will play normally, because the actual data size is stored separately. However, if you decode it back to WAV using the command-line programs, it will have the original WAV header. I will add a command-line option to correct the data chunk size in the original WAV header during decoding.
Uses 4 GB or just says 4 GB?
This is the command line for OptimFROG DualStream I use in fb2k
--encode --mode fast --seek fast --optimize fast --quality 6 --ans --silent --md5 --incorrectheader %s --output %d
The highlighted section? It shows the temp wav command line switch I need to make everything work, because fb2k just won't let me use the "-" (pipe endoding) switch to do the same thing. This has been the case since the previous version of DualStream was released in 2011.
This is the command line for OptimFROG DualStream I use in fb2k
--encode --mode fast --seek fast --optimize fast --quality 6 --ans --silent --md5 --incorrectheader %s --output %d
The highlighted section? It shows the temp wav command line switch I need to make everything work, because fb2k just won't let me use the "-" (pipe endoding) switch to do the same thing. This has been the case since the previous version of DualStream was released in 2011.
Indeed, the WAVE parser, even with --incorrectheader option, was too strict and rejected some size combinations. I have fixed it to accept all WAV files with incorrect RIFF and data chunk sizes in the WAV header, and also correct these sizes in the stored WAV header right after encoding.
The new version 5.001, using either "-" with --incorrectheader option (pipe encoding) or "%s" (temporary file), produces identical compressed files, therefore identical WAV files after decoding with the command-line programs. When using "%s", it is not necessary to add --incorrectheader option.
You can check the new release packages for the version 5.001 on the Downloads (http://LosslessAudio.org/Downloads.php) page.
Ahh my favourite lossless encoder. I remember how my files were taking over 70% CPU time when playing back on Athlon XP 3200+, now on Celeron G1820 when EIST clocks it down to 800MHz the load is unmeasurable. Hardware has made some major strides, hopefully ofr will exploit them (how about OptimFroGPU? ).
Congrats on the new release, Florin.
During my long public absence I have actively worked on my PhD thesis about lossless audio compression, "Modeling and Optimization for Lossless Audio Coding with Flexible Complexity Profiles", and I am very happy to say that the end result is a completely new codec member, OptimFROG Asymmetric: from compression ratios similar to OptimFROG maximum compression experimental, while being 5 up to 20 times faster at decoding, and going down to decoding speeds in the range of FLAC.
That sounds great, what about encoding speeds? The wording above reads as if one of the "decoding" up there should be an "encoding" to me...
Hardware has made some major strides, hopefully ofr will exploit them (how about OptimFroGPU? ).
That would certainly help with an otherwise rather slow encoding speed
edit: It could also be nice if someone made a decoder plugin for deadbeef, for those of us on the linux/mac side of things.
It could also be nice if someone made a decoder plugin for deadbeef, for those of us on the linux/mac side of things.
+1
Hi all,
As usually with new releases of lossless audio codecs, I've made a comparison between the versions. The tests have been done with the same setup as in my previous overall lossless codec comparison (http://www.audiograaf.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%204.pdf), that is, AMD A4-3400, Windows 7, CPU clock locked to max, RAMdisk. The presets are, from right to left and from up to down: fast, normal, high, best, highnew and bestnew.
Long story short: I don't see any improvements.
(http://www.audiograaf.nl/misc_stuff/optimfrog5-enc.png)
(http://www.audiograaf.nl/misc_stuff/optimfrog5-dec.png)
P.S.: ghido, the presets I tested are the same as I use(d) in my lossless codec comparison (http://www.audiograaf.nl/downloads.html). If you have a different (better) set of presets for me to test that takes about as long as it does now, please let me know. OptimFROG has too many buttons to try them all, currently it already takes more than half the testing time, so if you have something sane just tell me.
BTW, i compared OptimFROG_Win_x86_5001 with OptimFROG_Win_x86_SSE2_5001, and the former (SSE auto-detect) is faster than the latter (SSE2).
Florin, thanks for the new release!
1.) Technical internals (for those who missed them)The
--maximumcompression option is the preset and stands for
--mode ultranew --optimize best --seek slow options.
The ultranew mode is one of the undocumented modes but it works as any other mode so you can utilize it in order to perform a precise tuning. Another undoc mode is extrafast.
Please note that --maximumcompression does not guarantee the maximum compression for all of the data. In some rare instances the lower modes (like extranew or bestnew) can give smaller output file.
The --seek option controls the size of the block. Its not guaranteed that --seek slow will do the best for every kind of data so keep it in mind.
For really hardcore compression you can go further and use another undocumented switch:
It shaves off a really small amount of bytes while slowing down the encoding considerably so don't waste your time with it.
2.) Faster and slower Squarepusher - Stadium Ice (46 210 472 bytes, 44100Hz, 16bit, 2ch)
4.910 5.001 5.001 SSE2 5.001 x64
---------------- ---------------- ---------------- ----------------
enc dec enc dec enc dec enc dec
------- ------- ------- ------- ------- ------- ------- -------
--mode ultranew --optimize best --seek slow 183.875 27.875 176.734 26.296 178.937 27.109 181.234 28.828
--mode ultranew --optimize best --seek slow --experimental 186.266 29.281 178.281 27.640 180.640 28.515 179.234 28.984
--mode normal --optimize fast --seek normal 4.000 2.843 3.875 2.812 4.218 3.093 3.906 3.000
--mode normal --optimize fast --seek normal --experimental 9.265 3.484 8.234 3.281 8.718 3.578 6.985 3.328
--mode fast --optimize none --seek fast 2.156 2.031 2.062 1.953 2.203 2.109 2.000 2.015
--mode fast --optimize none --seek fast --experimental 7.859 2.656 6.828 2.468 7.078 2.640 5.390 2.312
------- ------- ------- ------- ------- ------- ------- -------
TOTAL: 393.421 68.170 376.014 64.450 381.794 67.044 378.749 68.467
Good news is that v5.001 is
4.4% faster on encoding and
5.5% faster on decoding than 4.910 version.
Bad news is that both SSE2 and x64 builds of v5.001 are
slower than plain x86 build of v5.001
3.) The bugThere is the severe bug in OptimFrog. If you take 44100, 2ch, 16bit WAV file with length of 1323001 samples and compress it with any mode\optimize option combined with --experimental and --seek fast\normal then you'll get the following:
srcFile: <Test.wav>
dstFile: <Test.ofr>
Compressing 85.7%
Exception OUTOFMEMORY in file unknown, line 0
variable mpData, size 4294934529
Errors occurred compressing <Test.wav>
srcFile: <Test.ofr>
dstFile: <Test_OUT.wav>
Decompressing 100.0%
Exception UNEXPECTEDEOF in file unknown, line 0
function readMin
Errors occurred decompressing <Test.ofr>
Comparing files test.wav and TEST_OUT.WAV
FC: test.wav longer than TEST_OUT.WAV
I have uploaded an example with ready-to-use batch file and binary. Feel free to test manually in case of security concerns.
OptimFROG Bug (https://drive.google.com/open?id=0B9lNQY6a_Ptkalo1WGNYUjdPTjA)
Many thanks to everyone for the feedback!
Hardware has made some major strides, hopefully ofr will exploit them (how about OptimFroGPU? ;) ).
I am already working on updating the OptimFROG encoder to use OpenMP, which will be able to utilize multiple cores to encode one file. After the public release of OptimFROG Asymmetric, I will also update its encoder to use OpenMP; in principle, its encoder algorithms would be also well suited to achieve significant speed-up using OpenCL (GPGPU).
During my long public absence I have actively worked on my PhD thesis about lossless audio compression, "Modeling and Optimization for Lossless Audio Coding with Flexible Complexity Profiles", and I am very happy to say that the end result is a completely new codec member, OptimFROG Asymmetric: from compression ratios similar to OptimFROG maximum compression experimental, while being 5 up to 20 times faster at decoding, and going down to decoding speeds in the range of FLAC.
That sounds great, what about encoding speeds? The wording above reads as if one of the "decoding" up there should be an "encoding" to me...
All the mentions are about decoding speed. My focus was mostly on decoding speed, which is related to the theoretical optimality of the compressed representation and the architectural design of the compressed format. The main goal was to create a single decoder format flexible enough to allow both low complexity and simple encoders with good compression, and also high or very high complexity encoders, which would however achieve for every individual file the best possible compression for a user-selected decoding complexity.
edit: It could also be nice if someone made a decoder plugin for deadbeef, for those of us on the linux/mac side of things.
I was also thinking to ask here about some suggestions of players under Linux and OS X, for which to create decoder plug-ins. There are already some plans for an Audacious decoder plug-in under Linux. I believe a decoder plug-in for deadbeef would be a very good suggestion, which could cover both platforms!
Long story short: I don't see any improvements.
There are indeed no compression differences between 4.910b and 5.00x. As mentioned in the changelog.txt file, all compressed sizes should be identical to those from the 4.910b version. However, the upcoming 5.002 version will have significant compatible compression improvements, around 4 %, for the fast, normal, and high modes using optimize none, for stereo files with near mono content.
P.S.: ghido, the presets I tested are the same as I use(d) in my lossless codec comparison (http://www.audiograaf.nl/downloads.html). If you have a different (better) set of presets for me to test that takes about as long as it does now, please let me know. OptimFROG has too many buttons to try them all, currently it already takes more than half the testing time, so if you have something sane just tell me.
Thank you for taking the time to test the new version! I am currently working on defining encoding presets for OptimFROG, which will allow choosing the optimum encoder operating points in a simple way, similar to FLAC. Until then, I would suggest to use a list very similar to the one you described, however with mode best replaced with mode extra and the additional switch --advanced-compression-analysis (ACA), which gives on average 0.6 % better compression on my 82 CDs test corpus: fast, fast + ACA, normal + ACA, high + ACA, extra + ACA, highnew + ACA, and bestnew + ACA.
BTW, i compared OptimFROG_Win_x86_5001 with OptimFROG_Win_x86_SSE2_5001, and the former (SSE auto-detect) is faster than the latter (SSE2).
Good news is that v5.001 is 4.4% faster on encoding and 5.5% faster on decoding than 4.910 version.
Bad news is that both SSE2 and x64 builds of v5.001 are slower than plain x86 build of v5.001
This is true, and the reason why the x86 build is recommended for all Windows versions on the Downloads page. Currently, the x86 build uses x87 FPU plus SSE auto-detect, and contains some hand-optimized x87 FPU and SSE code. Both the x86_SSE2 and the x64 builds use SSE2 only, they include the SSE optimizations, but do not have hand-optimized SSE2 code for the other parts yet.
There is the severe bug in OptimFrog. If you take 44100, 2ch, 16bit WAV file with length of 1323001 samples and compress it with any mode\optimize option combined with --experimental and --seek fast\normal then you'll get the following:
srcFile: <Test.wav>
dstFile: <Test.ofr>
Compressing 85.7%
Exception OUTOFMEMORY in file unknown, line 0
variable mpData, size 4294934529
Errors occurred compressing <Test.wav>
Thank you for the summary of technical internals and for reporting the bug! I can confirm that the OptimFROG encoder, when used with the --experimental option, triggers an out-of-memory exception during encoding of the above file. I have made a quick check and the file, which has 1323001 sample points, is encoded into 4 independent blocks. The last independent block, having only 1 sample point, is the one triggering the exception in the code corresponding to the --experimental option. I will fix the issue in the upcoming 5.002 version, to be released later during this week.
The --experimental option name is now deprecated, and renamed to --advanced-compression, because the corresponding code is not experimental anymore. I would like to add to your summary of technical internals two new and currently undocumented options: --advanced-compression-analysis (ACA) and --advanced-compression-modeling (ACM). Using only ACA will give almost all of the compression gains of --advanced-compression, but with smaller overhead for both the encoding and the decoding time. Using both ACA and ACM is fully equivalent to --advanced-compression.
OptimFROG version 5.002 is ready! You can check the new release packages for the version 5.002 on the Downloads (http://losslessaudio.org/Downloads.php) page.
It includes the compatible compression improvement for --optimize none I mentioned above, and a few fixes and updates.
I was recently told that a Qmmp decoder plug-in is almost ready, and that someone will write a deadbeef decoder plug-in.
OptimFROG version 5.002 is ready! You can check the new release packages for the version 5.002 on the Downloads (http://losslessaudio.org/Downloads.php) page.
It includes the compatible compression improvement for --optimize none I mentioned above, and a few fixes and updates.
I was recently told that a Qmmp decoder plug-in is almost ready, and that someone will write a deadbeef decoder plug-in.
The Qmmp plugin is now ready: https://github.com/cspiegel/qmmp-optimfrog (https://github.com/cspiegel/qmmp-optimfrog)
In addition, I've written an Audacious plugin: https://github.com/cspiegel/audacious-optimfrog (https://github.com/cspiegel/audacious-optimfrog)
Thanks to Florin for OptimFROG!
OptimFROG DualStream isn't working correctly in fb2k at all. I get this error message whenever I use temp wav or pipe encoding:
ofs.exe has stopped working
A problem has caused the program to stop working correctly.
Windows will close the program and notify you if a solution is available.
For some reason, the last file always has the smallest file size (Like something with a bitrate of 4.) And Replaygain always takes 7-10 minutes to complete for an average album. At this point, the standard version of DualStream is looking very good, and stable. The experimental version? Not so much.
Is there a list of command line options somewhere? I can't find one in the download or on the website.
Not trying to be funny or smart ass, I just want to understand, why is anyone using OptimFROG today?
Is there a list of command line options somewhere? I can't find one in the download or on the website.
Nevermind, I got them in the OptimFROG_All_Windows_x86_4520b1.zip download.
edit: It could also be nice if someone made a decoder plugin for deadbeef, for those of us on the linux/mac side of things.
I was also thinking to ask here about some suggestions of players under Linux and OS X, for which to create decoder plug-ins. There are already some plans for an Audacious decoder plug-in under Linux. I believe a decoder plug-in for deadbeef would be a very good suggestion, which could cover both platforms!
I think people on OS X would appreciate it if XLD (http://tmkk.undo.jp/xld/index_e.html) could support OptimFROG.
The Qmmp plugin is now ready: https://github.com/cspiegel/qmmp-optimfrog (https://github.com/cspiegel/qmmp-optimfrog)
In addition, I've written an Audacious plugin: https://github.com/cspiegel/audacious-optimfrog (https://github.com/cspiegel/audacious-optimfrog)
Thanks to Florin for OptimFROG!
Many thanks for the Qmmp and Audacious plug-ins!
OptimFROG DualStream isn't working correctly in fb2k at all. I get this error message whenever I use temp wav or pipe encoding:
ofs.exe has stopped working
A problem has caused the program to stop working correctly.
Windows will close the program and notify you if a solution is available.
For some reason, the last file always has the smallest file size (Like something with a bitrate of 4.) And Replaygain always takes 7-10 minutes to complete for an average album. At this point, the standard version of DualStream is looking very good, and stable. The experimental version? Not so much.
Thank you for reporting the problem! I was not able to reproduce it and I wrote you a PM some time ago, asking for more details. I guess the problem may be related to using the --experimental, --advanced-compression, or --advanced-compression-analysis command-line options with OptimFROG DualStream. These options were meant, in principle, for the OptimFROG Lossless variant only. The command-line parser code is common to all OptimFROG variants, with particular options being enabled only in some variants. The above options are currently implicitly enabled in all variants, but I will investigate and disable them accordingly in the upcoming OptimFROG version.
Is there a list of command line options somewhere? I can't find one in the download or on the website.
Nevermind, I got them in the OptimFROG_All_Windows_x86_4520b1.zip download.
Indeed, there is no up-to-date document with a detailed description of the command-line commands and options. Using --help on the command-line may help, as it will show the complete list of commands and options, most of them including a short explanation. I am planning to write and maintain such a document in the future.
I think people on OS X would appreciate it if XLD (http://tmkk.undo.jp/xld/index_e.html) could support OptimFROG.
Thank you for the very good suggestion! Hopefully, decoder plug-ins for both deadbeef and XLD will become available, some time soon.
What about plug-in for foobar 2000? How can I use it?
What about plug-in for foobar 2000? How can I use it?
To use the foobar2000 plug-in, the files foo_input_ofr.dll (from foo_input_ofr_131_test.zip or foo_input_ofr_130.zip) and OptimFROG.dll (for example, from the OptimFROG_Win_x86_5003.zip release package, in the SDK/Library subfolder) must be copied into the foobar2000/components folder.
OptimFROG 5.003 is ready! It adds a standard interactive installer install.sh, pkg-config support, shared library versioning support, --writefreshheader option to write a fresh standard WAV header during decoding, has some small decoder speed improvements, and is now available on FreeBSD.
You can check the new release packages on the Downloads (http://losslessaudio.org/Downloads.php) page and the comprehensive list of changes on the Changes (http://losslessaudio.org/Changes.php) page.
Among others, speed improvements using hand-optimized SSE2 code are in progress, and a decoder plug-in for XLD will be available some time soon.
Many thanks to Angenial for the DeaDBeeF plug-in, which is now ready: https://github.com/cspiegel/deadbeef-optimfrog (https://github.com/cspiegel/deadbeef-optimfrog)
What about plug-in for foobar 2000? How can I use it?
To use the foobar2000 plug-in, the files foo_input_ofr.dll (from foo_input_ofr_131_test.zip or foo_input_ofr_130.zip) and OptimFROG.dll (for example, from the OptimFROG_Win_x86_5003.zip release package, in the SDK/Library subfolder) must be copied into the foobar2000/components folder.
OptimFROG 5.003 is ready! It adds a standard interactive installer install.sh, pkg-config support, shared library versioning support, --writefreshheader option to write a fresh standard WAV header during decoding, has some small decoder speed improvements, and is now available on FreeBSD.
You can check the new release packages on the Downloads (http://losslessaudio.org/Downloads.php) page and the comprehensive list of changes on the Changes (http://losslessaudio.org/Changes.php) page.
Among others, speed improvements using hand-optimized SSE2 code are in progress, and a decoder plug-in for XLD will be available some time soon.
Many thanks to Angenial for the DeaDBeeF plug-in, which is now ready: https://github.com/cspiegel/deadbeef-optimfrog (https://github.com/cspiegel/deadbeef-optimfrog)
I had a big expectation from this.
But seems that it's useless. Apparantly every single file must be converted one by one because a fucking windows can't support several unicode languages (more than two) in filepath in a bat file. Who could have thought?! Foobar converter doesn't work at all (doesn't rename temp-somethingsomethinfile). I give up. I will better delete half of files than use a command promt software again. Now I understand why most people have stuck to flac.
Foobar converter doesn't work at all (doesn't rename temp-somethingsomethinfile).
I just tried and can't see any problem.
Foobar converter doesn't work at all (doesn't rename temp-somethingsomethinfile).
I just tried and can't see any problem.
You are right. My bad.
Just to mention it somewhere where someone might be interested: Kermit seems to be unavailable at the author's location, but archive.org has it: http://web.archive.org/web/20140826164213/...peek/kermit.htm (http://web.archive.org/web/20140826164213/http://members.home.nl/w.speek/kermit.htm)
OptimFROG 5.003 is ready!
Firstly, I just want to say it's great to see you back in action with v5. Thank you for all your work!
5.002 x64 (Windows) still seems to be the fastest encoding for me, although compared with 5.003 only by a fraction of a percent. Negligible, but consistent during repetition.
Not trying to be funny or smart ass, I just want to understand, why is anyone using OptimFROG today?
I can't speak for everyone, but I use it because no other lossless encoder reaches its level of compression.
Not trying to be funny or smart ass, I just want to understand, why is anyone using OptimFROG today?
The numbers speak for themselves
2014 codec poll (https://www.hydrogenaud.io/forums/index.php?showtopic=105188&hl=) and 2015 codec poll (https://www.hydrogenaud.io/forums/index.php?showtopic=108125&hl=)
So, optim frog is unable to embed ALL of the flac tags and can't show embedded pictures in popular foobar2000 and other players.
It can't be embedded into MKV container also. So, what can I use it for at all?
tried 5.003. (if 5.002 works, how could I know it?)
It was seeming as a good idea to get some free space (up to 5%, up to 10% with some other tricks, but). But now it just an arhiver software, not a codec.
Please tell me I am wrong. Because it feels weird, that author doesn't care about these things.
During my long public absence I have actively worked on my PhD thesis about lossless audio compression, "Modeling and Optimization for Lossless Audio Coding with Flexible Complexity Profiles", and I am very happy to say that the end result is a completely new codec member, OptimFROG Asymmetric: from compression ratios similar to OptimFROG maximum compression experimental, while being 5 up to 20 times faster at decoding, and going down to decoding speeds in the range of FLAC.
There is work slowly going on about the bitsream format finalization and analysis, and also integration and harmonization of different codec modules, but I definitely plan to release the full bitstream format specification and the official open-source reference implementations within a time frame of 6 to 10 months from now.
Should we expect a release of the reference implementation of this new codec soon?
Should we expect a release of the reference implementation of this new codec soon?
Thank you for asking! The work has been progressing well, however significantly slower than anticipated, mainly due to lack of time. Also, I have been redirecting some of the time into updating and improving the existing OptimFROG codec.
Along the way, some new great ideas appeared and need to be investigated. I will surely let everyone know when I am getting closer to releasing the full bitstream format specification and the official open-source reference implementations!