Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Lossless compression for IEEE Float audio data! (Read 5731 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Lossless compression for IEEE Float audio data!

Hi!

First, thank you all for your interest, help and feedback for further developing and improving OptimFROG.

As I was promising, the surprize is ready! Advanced lossless compression for IEEE Float audio data... 

To the best of my knowledge, this is the first publicly available lossless compressor for IEEE Float audio data. I know other multimedia-aware lossless compressors like RAR3.x, ACE2.x and SZIP, but no specialized one.


I need testing because I had a limited amount of IEEE Float samples to test it on. Compression comparisons (with numerical results), bug reports and even feature request are welcome... I will also open an upload ftp site, where you can send me interesting samples.

Download it (both Win32 and Linux versions availabe) from http://LosslessAudioCompression.com/Downloads.php or by visiting http://LosslessAudioCompression.com

Florin Ghido


Features
========

This is an experimental version, please use it with care. The file
compatibility with any future version is not implied or intended.

Feedback is greatly appreciated, especially problem reports with the
program crashing or worse, producing incorrectly decompressed audio
files. If you are not already familiar with the mainstream OptimFROG
Lossless, it will be helpful to read the ofr.txt file.

It is a branch from the main OptimFROG Lossless version 4.507 project,
with modifications and new algorithms to implement support for the IEEE
Float formats. The branch with support for IEEE Float started in
July 2002, and I recently decided to publicly release it.

This version has no input plug-ins available for real-time playback
and the file format is not compatible and recognized by the mainstream
OptimFROG Lossless 4.507 version. Also, it does not support any integer
PCM wave formats.


OptimFROG 4.508e [2003.10.04] has the following notable features:
  - first publicly available lossless audio compressor for IEEE Float
  - full support for 16.8 type 1 (Cool Edit), 24.0 type 1 (Cool Edit)
    and 0.24 type 3 (standard), normal and extended wave formats
  - full support for infinities, NaNs, negative zeros and denormals
  - optimal support for floating point data with integer-like content
  - on average 10-15% better lossless compression than any other
    multimedia-aware lossless compressor (like RAR3.x, ACE2.x, SZIP)
  - 2-10 times faster than other multimedia-aware lossless compressors
  - fast operation, default mode encodes 44100 Hz, 32 bits, stereo
    audio data at 9.0x real-time and decodes at 11.7x real-time on
    AMD Athlon XP 1800+
  - Win32 and Linux command line versions
  - extensible, streamable compressed format, tagging compatible
  - optimize option, further improving compression at no decoding cost


Usage
=====

The command line usage is identical to the mainstream OptimFROG
Lossless 4.507 version. See the ofr.txt file for the detailed command
line explanations. The only differences are that the executable has
been renamed to OFF (OptimFROG Float), and the raw files are not
supported.

It is very useful to use the --md5 option for all encodings, because
you can later verify for correct decompression with the --check command.

This is even more important because OFF is an experimental version, and
may have hidden bugs, and thus you can find the possible problems.
For example, to encode all the (IEEE Float only) wave files in the
current directory, using default compression:

  off --md5 *.wav

and to check all the (IEEE Float only) wave files in the current
directory for correct decompression, right after encoding:

  off --check *.wav

In case of any error, the program stops with a non-zero exit code at
the first file producing the error, so it is very easy to find it.

Lossless compression for IEEE Float audio data!

Reply #1 –
Looks promising!

Just tested it, but it failed:
Untitled.wav is a "32-bit Normalized float (type 3)" according to Cool Edit.

Code: [Select]
>off --md5 *.wav

OptimFROG Lossless IEEE Float Audio Compressor (Win32), v4.508e [2003.10.04]
Copyright (C) 1996-2003 Florin Ghido, all rights reserved.
Visit http://LosslessAudioCompression.com for updates
Experimental version, please use it with care! E-mail: FlorinGhido@yahoo.com


WAV File: <Untitled.wav>
OFR File: <Untitled.ofr>
Compressing done.
Error processing WAVE file:
Exception READERROR in file Unknown, line 0
function ReadFile, code 998, Invalid access to memory location.

Errors occurred compressing <Untitled.wav>


Despite the error it producced a Untitled.ofr file.
Trying to run --check on this file:

Code: [Select]
>off --check *.ofr

OptimFROG Lossless IEEE Float Audio Compressor (Win32), v4.508e [2003.10.04]
Copyright (C) 1996-2003 Florin Ghido, all rights reserved.
Visit http://LosslessAudioCompression.com for updates
Experimental version, please use it with care! E-mail: FlorinGhido@yahoo.com


OFR File: <Untitled.ofr>
Checking  52.8%
Exception STREAMERROR in file Unknown, line 0

Errors occurred checking <Untitled.ofr>


File is available here (5 MB, WinRAR 3 compression)

Lossless compression for IEEE Float audio data!

Reply #2 –
Please rerun with 'off --md5 --verbose *.wav'  This will show you how the WAVE file is parsed.

It is very likely that your WAV file (incorrectly) has an APEv2 tag at the end. The WAV parser tries to correctly parse the WAV file, and finds an invalid chunk of size ~10^9, which is in fact the start of the APE v2 tag.

I have been reported exactly the same problem with OFS (by Atlantis two days before).

Currently OFR, OFS and OFF do not support any tags appended to WAV files.

I will take a more detailed look at the file, but it will take a while as it comes very slowly on my analogue 33.6 kbps line...

Lossless compression for IEEE Float audio data!

Reply #3 –
The file has no tags, but it has some extra RIFF chunks.

Code: [Select]
>off --md5 --verbose Untitled.wav

OptimFROG Lossless IEEE Float Audio Compressor (Win32), v4.508e [2003.10.04]
Copyright (C) 1996-2003 Florin Ghido, all rights reserved.
Visit http://LosslessAudioCompression.com for updates
Experimental version, please use it with care! E-mail: FlorinGhido@yahoo.com


WAV File: <Untitled.wav>
OFR File: <Untitled.ofr>
Found fmt  chunk, size         16
  wFormatTag               3
  nChannels                2
  nSamplesPerSec       44100
  wBitsPerSample          32
  containerSize           32
  foundContainerSize      32
Found data chunk, size    6680576
data (start   44, size    6680576)
Compressing done.
Found LIST chunk, size        303
Found ISP? chunk, size   16777216
Error processing WAVE file:
Exception READERROR in file Unknown, line 0
function ReadFile, code 998, Invalid access to memory location.

Errors occurred compressing <Untitled.wav>


Edit: seems that OptimFrog parses the chunks wrong? it overshoots the start of the last chunk, "DISP", by 1 byte... making the size of it 16 MB instead of 8 bytes...

Lossless compression for IEEE Float audio data!

Reply #4 –
Quote
Found LIST chunk, size        303
Found ISP? chunk, size   16777216
Error processing WAVE file:
Exception READERROR in file Unknown, line 0
function ReadFile, code 998, Invalid access to memory location.

Edit: seems that OptimFrog parses the chunks wrong? it overshoots the start of the last chunk, "DISP", by 1 byte... making the size of it 16 MB instead of 8 bytes...


The problem is that the RIFF WAVE standard mandates that all chunks must have a multiple of 2 size, so even if a chunk has an uneven number of bytes, a padding byte will be added (uncounted).
This causes OptimFROG to eat the first byte of the DISP chunk and after that the disaster happens

I will see what can be done to workaround the inconsistencies (both even and uneven number of bytes in chunks).

Meantime, you can try opening the file and saving with another name (without the 'Save extra non-audio information' option in Cool Edit) to see the compression at work 

BTW: Cool Edit introduced in the past another problem. You cannot distinguish between 16.8 float type 1 (which they introduced) and 32 bit integer type 1, which is a bad design decision. Later, in Cool Edit Pro 2.1 they added a dummy cbSize=2 in the fmt_ chunk to inicate the difference and made it 'Obsolete/Compatibility'. 

Lossless compression for IEEE Float audio data!

Reply #5 –
I've done a little test, with samples from CEP 2.0 (all three formats) and fb2k diskwriter, and everything seemes to work fine.

There is an error in "off.txt" though:
off --check *.wav"  should be "off --check *.ofr" i think. 

Good work

Lossless compression for IEEE Float audio data!

Reply #6 –
Quote
Meantime, you can try opening the file and saving with another name (without the 'Save extra non-audio information' option in Cool Edit) to see the compression at work 


Yeah it works fine after resaving and without any extra chunks

Lossless compression for IEEE Float audio data!

Reply #7 –
"Advanced lossless compression for IEEE Float audio data"

I've never heard of this before, which really surprises me. Can some tell me, in layman's terms, what IEEE Float audio data is and where it is used?

Lossless compression for IEEE Float audio data!

Reply #8 –
Quote
"Advanced lossless compression for IEEE Float audio data"

I've never heard of this before, which really surprises me. Can some tell me, in layman's terms, what IEEE Float audio data is and where it is used?

Standard PCM as used on audio CD allows you to store an amplitude from -32768...0...+32767, giving 16 bit precision.

IEEE Float gives you a much wider and precise range by using floating point numbers of 32 to 64 bits.

Lossless compression for IEEE Float audio data!

Reply #9 –
Quote
IEEE Float gives you a much wider and precise range by using floating point numbers of 32 to 64 bits.

Just wondering, is there any loss when converting to and from 32-bit float? The reason I'm asking is that CoreAudio, the Mac OS X sound API, uses it as it's native format. All sound is converted to it and sent to the sound card driver which then converts it to whatever format it wants.

Lossless compression for IEEE Float audio data!

Reply #10 –
Quote
Just wondering, is there any loss when converting to and from 32-bit float? The reason I'm asking is that CoreAudio, the Mac OS X sound API, uses it as it's native format. All sound is converted to it and sent to the sound card driver which then converts it to whatever format it wants.

There is always a loss, but it's going to be at the 32'nd bit. If you know that real life equipment generally doesn't perform better than 20 bits, you've got yourself quite the headroom there.

Lossless compression for IEEE Float audio data!

Reply #11 –
Quote
Just wondering, is there any loss when converting to and from 32-bit float? The reason I'm asking is that CoreAudio, the Mac OS X sound API, uses it as it's native format. All sound is converted to it and sent to the sound card driver which then converts it to whatever format it wants.

IEEE Float (32 bits) can losslessly represent any signed integer up to and including 25 bits (-2^24...0...2^24-1).
This means that even 24 bit PCM (-2^23...0...2^23-1) audio data can be losslessly converted to and from IEEE Float.

If the audio data is sent to the hardware unprocessed, there will be absolutely no loss by using the IEEE Float as an intermediary format. However, if there takes place some post-processing like equalization or other effects, as Garf said  , there will be a slight loss when converting back to integer values (the least significant bit, due to integer rounding), needed for hardware playback.

Lossless compression for IEEE Float audio data!

Reply #12 –
Good job, Florin! I tested out your encoder on some IEEE material I have and it worked, well, losslessly! 

However, your server was a little slow. I got less than 1k bytes/second on the download and I was getting ping roundtrips of 2-3 seconds from here! 
 

Lossless compression for IEEE Float audio data!

Reply #13 –
Quote
If the audio data is sent to the hardware unprocessed, there will be absolutely no loss by using the IEEE Float as an intermediary format. However, if there takes place some post-processing like equalization or other effects, as Garf said  , there will be a slight loss when converting back to integer values (the least significant bit, due to integer rounding), needed for hardware playback.

I had a feeling it was something like this - anything else would be quite silly if they want the platform to be taken seriously. I guess a rounding error of 1/2^16 or 1/2^24 can be safely ignored