Skip to main content
Topic: OptimFROG 5.000 has been released (Read 26427 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

OptimFROG 5.000 has been released

After an exceedingly long apparent pause, the OptimFROG 5.000 stable release is finally ready!

Summary of changes:
  • several new supported platforms, including SSE2 enabled builds;
  • a large number of internal code updates and improvements;
  • various speed improvements for all compression modes;
  • updated SDK layout, documentation, and website content;
  • a new unified, simplified, and more permissive license.

You can check all the new release packages on the Downloads page and the comprehensive list of changes on the Changes page.

More information about the OptimFROG Lossless / DualStream / IEEE Float Audio Codec can be found on the OptimFROG website.

OptimFROG 5.000 has been released

Reply #1
Code name: Lazarus? Welcome back. I downloaded a copy to kick the tires.
--
Eric

OptimFROG 5.000 has been released

Reply #2
Good to see this. Welcome back.
wavpack 4.8 -b4x4c

OptimFROG 5.000 has been released

Reply #3
Thank you, and nice to find you here again! Indeed, time-wise the release code name would be a good fit, however I personally have never actually left the field.

During my long public absence I have actively worked on my PhD thesis about lossless audio compression, "Modeling and Optimization for Lossless Audio Coding with Flexible Complexity Profiles",  and I am very happy to say that the end result is a completely new codec member, OptimFROG Asymmetric: from compression ratios similar to OptimFROG maximum compression experimental, while being 5 up to 20 times faster at decoding, and going down to decoding speeds in the range of FLAC.

There is work slowly going on about the bitsream format finalization and analysis, and also integration and harmonization of different codec modules, but I definitely plan to release the full bitstream format specification and the official open-source reference implementations within a time frame of 6 to 10 months from now.

On the short term, I am currently working on finalizing the port of the entire OptimFROG release package for Android with armv7, mips, and x86 platforms.

All feedback and especially help with creating new player plug-ins and improving the range of software support for @OptimFROG would be greatly appreciated!

OptimFROG 5.000 has been released

Reply #4
I noticed that, like in previous versions, the MD5 hash for OFR files still don't display in the File Properties within foobar2000, forcing me to use OFR.exe to see those hashes via command line. Would this be the fault of the OptimFROG plugin that foobar2000 has to use? TAK used to have a similar problem until its plugin was updated.
ghostman

OptimFROG 5.000 has been released

Reply #5
Also noticed that foobar2000 still refuses to let OptimFROG Dualstream use pipe encoding, which forces you to use the old school temp wav encoding option just to make it work. Oddly, you can use that option via command line with ofr.exe
ghostman

OptimFROG 5.000 has been released

Reply #6
I noticed that, like in previous versions, the MD5 hash for OFR files still don't display in the File Properties within foobar2000, forcing me to use OFR.exe to see those hashes via command line. Would this be the fault of the OptimFROG plugin that foobar2000 has to use? TAK used to have a similar problem until its plugin was updated.

The MD5 hash for an OFR file is currently not available through the OptimFROG SDK interface, therefore plug-ins cannot obtain this stored MD5 hash value. I will update the SDK interface and also the foobar2000 plug-in in order to support this new and useful feature. Thank you!

Also noticed that foobar2000 still refuses to let OptimFROG Dualstream use pipe encoding, which forces you to use the old school temp wav encoding option just to make it work. Oddly, you can use that option via command line with ofr.exe

The ofr and ofs programs fully support pipes and should work identically in the foobar2000 Converter, without a temporary file. This requires adding the command-line option --incorrectheader, which allows to correctly encode data with a generic WAV header, having a fixed data chunk size of around 4 GB.
The compressed file will play normally, because the actual data size is stored separately. However, if you decode it back to WAV using the command-line programs, it will have the original WAV header. I will add a command-line option to correct the data chunk size in the original WAV header during decoding.

OptimFROG 5.000 has been released

Reply #7
Uses 4 GB or just says 4 GB?

OptimFROG 5.000 has been released

Reply #8
This is the command line for OptimFROG DualStream I use in fb2k

--encode --mode fast --seek fast --optimize fast --quality 6 --ans --silent --md5 --incorrectheader %s --output %d

The highlighted section? It shows the temp wav command line switch I need to make everything work, because fb2k just won't let me use the "-" (pipe endoding) switch to do the same thing. This has been the case since the previous version of DualStream was released in 2011.
ghostman

OptimFROG 5.000 has been released

Reply #9
This is the command line for OptimFROG DualStream I use in fb2k

--encode --mode fast --seek fast --optimize fast --quality 6 --ans --silent --md5 --incorrectheader %s --output %d

The highlighted section? It shows the temp wav command line switch I need to make everything work, because fb2k just won't let me use the "-" (pipe endoding) switch to do the same thing. This has been the case since the previous version of DualStream was released in 2011.

Indeed, the WAVE parser, even with --incorrectheader option, was too strict and rejected some size combinations. I have fixed it to accept all WAV files with incorrect RIFF and data chunk sizes in the WAV header, and also correct these sizes in the stored WAV header right after encoding.

The new version 5.001, using either "-" with --incorrectheader option (pipe encoding) or "%s" (temporary file), produces identical compressed files, therefore identical WAV files after decoding with the command-line programs. When using "%s", it is not necessary to add --incorrectheader option.

You can check the new release packages for the version 5.001 on the Downloads page.

OptimFROG 5.000 has been released

Reply #10
Ahh my favourite lossless encoder. I remember how my files were taking over 70% CPU time when playing back on Athlon XP 3200+, now on Celeron G1820 when EIST clocks it down to 800MHz the load is unmeasurable. Hardware has made some major strides, hopefully ofr will exploit them (how about OptimFroGPU?  ).

OptimFROG 5.000 has been released

Reply #11
Congrats on the new release, Florin.
Vital papers will demonstrate their vitality by spontaneously moving from where you left them to where you can't find them.

OptimFROG 5.000 has been released

Reply #12
During my long public absence I have actively worked on my PhD thesis about lossless audio compression, "Modeling and Optimization for Lossless Audio Coding with Flexible Complexity Profiles",  and I am very happy to say that the end result is a completely new codec member, OptimFROG Asymmetric: from compression ratios similar to OptimFROG maximum compression experimental, while being 5 up to 20 times faster at decoding, and going down to decoding speeds in the range of FLAC.
That sounds great, what about encoding speeds? The wording above reads as if one of the "decoding" up there should be an "encoding" to me...

Hardware has made some major strides, hopefully ofr will exploit them (how about OptimFroGPU?  ).
That would certainly help with an otherwise rather slow encoding speed

edit: It could also be nice if someone made a decoder plugin for deadbeef, for those of us on the linux/mac side of things.


OptimFROG 5.000 has been released

Reply #14
Hi all,

As usually with new releases of lossless audio codecs, I've made a comparison between the versions. The tests have been done with the same setup as in my previous overall lossless codec comparison, that is, AMD A4-3400, Windows 7, CPU clock locked to max, RAMdisk. The presets are, from right to left and from up to down: fast, normal, high, best, highnew and bestnew.

Long story short: I don't see any improvements.




P.S.: ghido, the presets I tested are the same as I use(d) in my lossless codec comparison. If you have a different (better) set of presets for me to test that takes about as long as it does now, please let me know. OptimFROG has too many buttons to try them all, currently it already takes more than half the testing time, so if you have something sane just tell me.
Music: sounds arranged such that they construct feelings.

OptimFROG 5.000 has been released

Reply #15
BTW, i compared OptimFROG_Win_x86_5001 with OptimFROG_Win_x86_SSE2_5001, and the former (SSE auto-detect) is faster than the latter (SSE2).

OptimFROG 5.000 has been released

Reply #16
Florin, thanks for the new release!

1.) Technical internals (for those who missed them)
The --maximumcompression option is the preset and stands for --mode ultranew --optimize best --seek slow options.
The ultranew mode is one of the undocumented modes but it works as any other mode so you can utilize it in order to perform a precise tuning. Another undoc mode is extrafast.
Please note that --maximumcompression does not guarantee the maximum compression for all of the data. In some rare instances the lower modes (like extranew or bestnew) can give smaller output file.
The --seek option controls the size of the block. Its not guaranteed that --seek slow will do the best for every kind of data so keep it in mind.
For really hardcore compression you can go further and use another undocumented switch: Spoiler (click to show/hide)

It shaves off a really small amount of bytes while slowing down the encoding considerably so don't waste your time with it.

2.) Faster and slower
Code: [Select]
                                                                      Squarepusher - Stadium Ice (46 210 472 bytes, 44100Hz, 16bit, 2ch)

                                                                      4.910               5.001             5.001 SSE2          5.001 x64
                                                                 ----------------    ----------------    ----------------    ----------------
                                                                   enc      dec        enc      dec        enc      dec        enc      dec
                                                                 -------  -------    -------  -------    -------  -------    -------  -------
--mode ultranew --optimize best --seek slow                      183.875   27.875    176.734   26.296    178.937   27.109    181.234   28.828
--mode ultranew --optimize best --seek slow   --experimental     186.266   29.281    178.281   27.640    180.640   28.515    179.234   28.984
--mode normal   --optimize fast --seek normal                      4.000    2.843      3.875    2.812      4.218    3.093      3.906    3.000
--mode normal   --optimize fast --seek normal --experimental       9.265    3.484      8.234    3.281      8.718    3.578      6.985    3.328
--mode fast     --optimize none --seek fast                        2.156    2.031      2.062    1.953      2.203    2.109      2.000    2.015
--mode fast     --optimize none --seek fast   --experimental       7.859    2.656      6.828    2.468      7.078    2.640      5.390    2.312
                                                                 -------  -------    -------  -------    -------  -------    -------  -------
                                                      TOTAL:     393.421   68.170    376.014   64.450    381.794   67.044    378.749   68.467

Good news is that v5.001 is 4.4% faster on encoding and 5.5% faster on decoding than 4.910 version.
Bad news is that both SSE2 and x64 builds of v5.001 are slower than plain x86 build of v5.001

3.) The bug
There is the severe bug in OptimFrog. If you take 44100, 2ch, 16bit WAV file with length of 1323001 samples and compress it with any mode\optimize option combined with --experimental and --seek fast\normal then you'll get the following:
Code: [Select]
srcFile: <Test.wav>
dstFile: <Test.ofr>
Compressing    85.7%
Exception OUTOFMEMORY in file unknown, line 0
variable mpData, size 4294934529

Errors occurred compressing <Test.wav>

srcFile: <Test.ofr>
dstFile: <Test_OUT.wav>
Decompressing 100.0%
Exception UNEXPECTEDEOF in file unknown, line 0
function readMin

Errors occurred decompressing <Test.ofr>
Comparing files test.wav and TEST_OUT.WAV
FC: test.wav longer than TEST_OUT.WAV


I have uploaded an example with ready-to-use batch file and binary. Feel free to test manually in case of security concerns.
OptimFROG Bug
Gabber, Jazz and IDM

OptimFROG 5.000 has been released

Reply #17
Many thanks to everyone for the feedback!

Hardware has made some major strides, hopefully ofr will exploit them (how about OptimFroGPU? ;) ).

I am already working on updating the OptimFROG encoder to use OpenMP, which will be able to utilize multiple cores to encode one file. After the public release of OptimFROG Asymmetric, I will also update its encoder to use OpenMP; in principle, its encoder algorithms would be also well suited to achieve significant speed-up using OpenCL (GPGPU).

During my long public absence I have actively worked on my PhD thesis about lossless audio compression, "Modeling and Optimization for Lossless Audio Coding with Flexible Complexity Profiles",  and I am very happy to say that the end result is a completely new codec member, OptimFROG Asymmetric: from compression ratios similar to OptimFROG maximum compression experimental, while being 5 up to 20 times faster at decoding, and going down to decoding speeds in the range of FLAC.
That sounds great, what about encoding speeds? The wording above reads as if one of the "decoding" up there should be an "encoding" to me...

All the mentions are about decoding speed. My focus was mostly on decoding speed, which is related to the theoretical optimality of the compressed representation and the architectural design of the compressed format. The main goal was to create a single decoder format flexible enough to allow both low complexity and simple encoders with good compression, and also high or very high complexity encoders, which would however achieve for every individual file the best possible compression for a user-selected decoding complexity.

edit: It could also be nice if someone made a decoder plugin for deadbeef, for those of us on the linux/mac side of things.

I was also thinking to ask here about some suggestions of players under Linux and OS X, for which to create decoder plug-ins. There are already some plans for an Audacious decoder plug-in under Linux. I believe a decoder plug-in for deadbeef would be a very good suggestion, which could cover both platforms!

Long story short: I don't see any improvements.

There are indeed no compression differences between 4.910b and 5.00x. As mentioned in the changelog.txt file, all compressed sizes should be identical to those from the 4.910b version. However, the upcoming 5.002 version will have significant compatible compression improvements, around 4 %, for the fast, normal, and high modes using optimize none, for stereo files with near mono content.

P.S.: ghido, the presets I tested are the same as I use(d) in my lossless codec comparison. If you have a different (better) set of presets for me to test that takes about as long as it does now, please let me know. OptimFROG has too many buttons to try them all, currently it already takes more than half the testing time, so if you have something sane just tell me.

Thank you for taking the time to test the new version! I am currently working on defining encoding presets for OptimFROG, which will allow choosing the optimum encoder operating points in a simple way, similar to FLAC. Until then, I would suggest to use a list very similar to the one you described, however with mode best replaced with mode extra and the additional switch --advanced-compression-analysis (ACA), which gives on average 0.6 % better compression on my 82 CDs test corpus: fast, fast + ACA, normal + ACA, high + ACA, extra + ACA, highnew + ACA, and bestnew + ACA.

BTW, i compared OptimFROG_Win_x86_5001 with OptimFROG_Win_x86_SSE2_5001, and the former (SSE auto-detect) is faster than the latter (SSE2).

Good news is that v5.001 is 4.4% faster on encoding and 5.5% faster on decoding than 4.910 version.
Bad news is that both SSE2 and x64 builds of v5.001 are slower than plain x86 build of v5.001

This is true, and the reason why the x86 build is recommended for all Windows versions on the Downloads page. Currently, the x86 build uses x87 FPU plus SSE auto-detect, and contains some hand-optimized x87 FPU and SSE code. Both the x86_SSE2 and the x64 builds use SSE2 only, they include the SSE optimizations, but do not have hand-optimized SSE2 code for the other parts yet.

There is the severe bug in OptimFrog. If you take 44100, 2ch, 16bit WAV file with length of 1323001 samples and compress it with any mode\optimize option combined with --experimental and --seek fast\normal then you'll get the following:
Code: [Select]
srcFile: <Test.wav>
dstFile: <Test.ofr>
Compressing    85.7%
Exception OUTOFMEMORY in file unknown, line 0
variable mpData, size 4294934529

Errors occurred compressing <Test.wav>

Thank you for the summary of technical internals and for reporting the bug! I can confirm that the OptimFROG encoder, when used with the --experimental option, triggers an out-of-memory exception during encoding of the above file. I have made a quick check and the file, which has 1323001 sample points, is encoded into 4 independent blocks. The last independent block, having only 1 sample point, is the one triggering the exception in the code corresponding to the --experimental option. I will fix the issue in the upcoming 5.002 version, to be released later during this week.

The --experimental option name is now deprecated, and renamed to --advanced-compression, because the corresponding code is not experimental anymore. I would like to add to your summary of technical internals two new and currently undocumented options: --advanced-compression-analysis (ACA) and --advanced-compression-modeling (ACM). Using only ACA will give almost all of the compression gains of --advanced-compression, but with smaller overhead for both the encoding and the decoding time. Using both ACA and ACM is fully equivalent to --advanced-compression.

OptimFROG 5.000 has been released

Reply #18
OptimFROG version 5.002 is ready! You can check the new release packages for the version 5.002 on the Downloads page.
It includes the compatible compression improvement for --optimize none I mentioned above, and a few fixes and updates.
I was recently told that a Qmmp decoder plug-in is almost ready, and that someone will write a deadbeef decoder plug-in.

OptimFROG 5.000 has been released

Reply #19
OptimFROG version 5.002 is ready! You can check the new release packages for the version 5.002 on the Downloads page.
It includes the compatible compression improvement for --optimize none I mentioned above, and a few fixes and updates.
I was recently told that a Qmmp decoder plug-in is almost ready, and that someone will write a deadbeef decoder plug-in.

The Qmmp plugin is now ready: https://github.com/cspiegel/qmmp-optimfrog

In addition, I've written an Audacious plugin: https://github.com/cspiegel/audacious-optimfrog

Thanks to Florin for OptimFROG!

OptimFROG 5.000 has been released

Reply #20
OptimFROG DualStream isn't working correctly in fb2k at all. I get this error message whenever I use temp wav or pipe encoding:


ofs.exe has stopped working

A problem has caused the program to stop working correctly.
Windows will close the program and notify you if a solution is available.



For some reason, the last file always has the smallest file size (Like something with a bitrate of 4.) And Replaygain always takes 7-10 minutes to complete for an average album. At this point, the standard version of DualStream is looking very good, and stable. The experimental version? Not so much.
ghostman

OptimFROG 5.000 has been released

Reply #21
Is there a list of command line options somewhere? I can't find one in the download or on the website.

OptimFROG 5.000 has been released

Reply #22
Not trying to be funny or smart ass, I just want to understand, why is anyone using OptimFROG today?

OptimFROG 5.000 has been released

Reply #23
Is there a list of command line options somewhere? I can't find one in the download or on the website.

Nevermind, I got them in the OptimFROG_All_Windows_x86_4520b1.zip download.

OptimFROG 5.000 has been released

Reply #24
edit: It could also be nice if someone made a decoder plugin for deadbeef, for those of us on the linux/mac side of things.

I was also thinking to ask here about some suggestions of players under Linux and OS X, for which to create decoder plug-ins. There are already some plans for an Audacious decoder plug-in under Linux. I believe a decoder plug-in for deadbeef would be a very good suggestion, which could cover both platforms!

I think people on OS X would appreciate it if XLD could support OptimFROG.
Every night with my star friends / We eat caviar and drink champagne
Sniffing in the VIP area / We talk about Frank Sinatra
Do you know Frank Sinatra? / He's dead

 
SimplePortal 1.0.0 RC1 © 2008-2018