Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Ogg+Flac as lossy+recovery file! (Read 13793 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Ogg+Flac as lossy+recovery file!

Here's a quick test I did:

I encoded a wav-file (Eurythmics' Sweet Dreams) to ogg using Oggenc RC3 -q 5.0, then decoded it to wav. The decoded Ogg was then inverse-mixed (using cool edit) with the original wav file, then compressed with Flac.

Original WAV size: 49.4 MB
Original FLAC compressed: 28.3 MB

Ogg: 5.56 MB
Flac 'Recovery' file: 24.8 MB
Total (ogg+recovery): 30.4 MB

As you can see, the Ogg+Recovery solution is only 7% bigger than the original flac.

To reverse the process (ie create the original wav): just mix the decoded Ogg and the decoded Flac-recovery-file togheter!

Since both Ogg and Flac are open-source, this could easily be automated.

I know that WavPack and Monkey's Audio have this lossy+recovery, but they are closed-source windows-only.

So what do everybody think about this?

 

Ogg+Flac as lossy+recovery file!

Reply #1
I don't get the point of this. FLAC is already lossless. You'll get the original WAV when you decode from FLAC.

And what makes you so sure that a lossless inverse-mix and a lossy Ogg will amount to the original WAV file? I don't think this works.

Ogg+Flac as lossy+recovery file!

Reply #2
Umm difference signal and decoded ogg mixed together should amount the original.

The idea is that you can first download the ogg, if you later want to make it lossless you'll save few MBs worth of downloading by downloading the recovery file instead of original lossless.

Also I don't know if RIAA can attack you for hosting only difference signal..

Not sure how useful this would be though.
Juha Laaksonheimo

Ogg+Flac as lossy+recovery file!

Reply #3
Quote
Originally posted by JohnV
Umm difference signal and decoded ogg mixed together should amount the original.


Do you really think that the mixing, the decoding and the additional mixing are accurate enough to give you a WAV with the same CRC as the original? I have to see proof before i believe that.

Quote
The idea is that you can first download the ogg, if you later want to make it lossless you'll save few MBs worth of downloading by downloading the recovery file instead of original lossless.


Okay, but you could as well download the file off WinMX in 128 kbit MP3 'quality' first, listen to it, and if you like it, download the full FLAC (from whereever) and delete the MP3. Cause if you don't like the downloaded OGG, why would you wanna keep it?

Ogg+Flac as lossy+recovery file!

Reply #4
I think this is an awesome development if in fact it would result in identically CRCd files.... when WAVpack came out with this feature a few weeks ago, I tried it out and thought it was an awesome idea, but wished it could be implemented in a codec better tuned for lossy compression  - wavpack's lossy 320 kbps, although I didn't do extensive listening tests, I would imagine would be nowhere near the quality of ogg, mpc, or even mp3, as it has not undergone the extensive tuning those formats have.

For those who don't see the point of this, the reason I think its useful is because I want to encode my entire CD collection, but the current state of audiocoding is in such flux right now, I am having a hard time deciding on a format... do I go with MPC for quality, but then have to transcode to mp3 for hardware players? Do I go with ogg and hope that hardware player support that has been talked about actually comes through? If so, do I go ahead and encode now, or wait for 1.0? A feature such as this solves this problem, as I could encode in ogg for now, burn all the correction files to backups, so I have something to listen to for now, without requiring the massive disk space that would be required to store them all lossless on my HD... but then when the state of audiocoding settles down and I decide on a final format, I can restore the original waves and reencode without worrying about quality loss from transcoding.

Ogg+Flac as lossy+recovery file!

Reply #5
Lossless coders a la FLAC, LPAC and Monkeys Audio aren't really suited for compressing a recovery file.

If you inverse-mix the OGG decoded and original, it'll give you mostly fuzz, which these lossless coders really dont handle good. If there would be some lossless encoder dedicated to compress these fuzz files, you'd achieve a higher ratio.

Ogg+Flac as lossy+recovery file!

Reply #6
I still have a hard time understanding the philosophy behind this all, Randum.

Are you telling me that you prefer a backup with two seperate files, of which a) the Ogg file will possibly become worthless when you settle on a different format/version in the future, b) the FLAC file _alone_ is already worthless, and c) where you can't re-encode without taking both files and joining them first...  over a backup with one lossless FLAC copy and one small file for your hard disk, encoded with what you like best at this very moment, and where you can easily re-encode from the FLAC files off your CDs, when - for instance - Ogg Vorbis 1.0 is available?

Ogg+Flac as lossy+recovery file!

Reply #7
Well, the more I think about it, the less convinced I am that madah's proposed method would actually result in bit-identical original wav's... however, if it could be made to work, I still think it would be very useful. CiTay, your suggestion that for my purposes I should just encode 2 versions, a lossy and a lossless, and burn the lossless to backup... that would, of course, work for my purposes, it would just bug me that I'm using backup media unnecesssarily for redundant data. With wavpack, the correction files are significantly smaller than the entire lossless file, so it would increase the number of backups I could fit on a CDR (or whatever backup medium I use)... if the same could be said of the hybrid ogg method (or... if it does work, it could be equally applicable to mpc) you would have the same benfit, just with better sounding lossy files. Essentially the only reason to do your way over mine/madah's would be all the additional steps required for the process of generating and compressing the correction files... which is of course a totally valid reason - I would never consider using this method unless someone (me?) took the time to write an automated tool to do this kind of thing in a simple one step frontend.

Ogg+Flac as lossy+recovery file!

Reply #8
Quote
Originally posted by CiTay
Do you really think that the mixing, the decoding and the additional mixing are accurate enough to give you a WAV with the same CRC as the original? I have to see proof before i believe that.
Sure, you can verify this easily with CoolEdit.
Juha Laaksonheimo

Ogg+Flac as lossy+recovery file!

Reply #9
Now I get it!

Create lossy file - Create difference file lossy/original - Compress difference file using a lossless scheme.
With this it is possible to get a bit accurate reproduction of the original. Interesting idea yet I do not see the benefits.

The size difference of the two lossless files in this case is only little more than 12%.  If you want to backup for later reencode then why not just "waste" those 12% and have files you can reuse? Imagine: you decide to settle on ogg 1.0 but maybe someday you will want to reencode that also. So you need to create another set of recovery files etc. Even if automated this is still time and media consuming. Or another one: you loose your lossy files (hdd crash, accidental deletion...) and think: if only I had not saved those 12% I could still listen to the music today.

Perhaps you will be better off using general purpose compression since, as Tom pointed out, you are not necessarily encoding tonal data.

Ogg+Flac as lossy+recovery file!

Reply #10
Quote
Originally posted by Gecko
Or another one: you loose your lossy files (hdd crash, accidental deletion...) and think: if only I had not saved those 12% I could still listen to the music today.

Perhaps you will be better off using general purpose compression since, as Tom pointed out, you are not necessarily encoding tonal data.


Yes, I thought of this too, and I think its the most convincing argument against this method - and why I'm probably not going to actually do it. Still, I just think the idea has a coolness factor to it

Ogg+Flac as lossy+recovery file!

Reply #11
Quote
And what makes you so sure that a lossless inverse-mix and a lossy Ogg will amount to the original WAV file? I don't think this works.


I've tested the files and they where bit-identical. But, if the decoder is changed or different decoding method is used (replaygain/dither/etc...) this will of course fail...

Quote
The idea is that you can first download the ogg, if you later want to make it lossless you'll save few MBs worth of downloading by downloading the recovery file instead of original lossless.


One thing that this could be very useful for is that you can download a preview mp3/ogg for free (from an artist' page) and then pay to get the recovery file if you want full quality...

Ogg+Flac as lossy+recovery file!

Reply #12
I think that FLAC (and other lossless compressors) would probably do pretty well on the difference files. They are still 16-bit stereo audio and all those programs are adaptive to some extent, although a little alternate tuning might help. I'm sure they would still compress much better than a general purpose compressor.

However, if someone wants to use this method, they should store the decoder executable along with the ogg files. Different versions of the decoder would very likely produce slightly different decoded versions (just like different MP3 decoders). While these differences would probably not be audible, they would cause the "recovery" file to no longer work exactly the same (i.e. bye-bye lossless).

madah: Wrote this before your last post showed up!

Ogg+Flac as lossy+recovery file!

Reply #13
Quote
Originally posted by madah

One thing that this could be very useful for is that you can download a preview mp3/ogg for free (from an artist' page) and then pay to get the recovery file if you want full quality...


If bandwidth is really the issue here, why not pay for the CD, instead of downloading an additional recovery file...

I can think of no situation that would make the space saving worth the extra effort and doubled risk of data loss.

Ogg+Flac as lossy+recovery file!

Reply #14
:confused:

I fail to see the value in this.

There are two situations: a) I want to encode an album where I have the original CD. b) I want to encode an album where I will not have the original CD in the future.

For case a), there is no need to create a recovery file and burn it to disk. Just re-rip the original CD and re-encode.

For case b), I pick a lossy format that offers significant headroom and just bite the bullet and realize it will be a permanently lossy copy. For these situations, I currently use "mppenc --xtreme --nmt 16 --tmn 32". Avg bitrate ranges from 275-325kbps...a filesize that I can manage with current disk space realities. It will be extremely rare if I can tell these MPCs apart from the WAVs.

For case b), if in the future I realize that I shouldn't have used a lossy format and wished I had the original WAVs again to re-encode, I could always buy the CD. If this CD was known to be OOP/rare/etc, I would have used a lossless format like LPAC or MAC.

Ogg+Flac as lossy+recovery file!

Reply #15
Quote
Originally posted by mithrandir
I fail to see the value in this.

You're neglecting a couple of scenarios. The whole point of this is to not HAVE to 'bite the bullet'.... as stated, the reason I want to do this is so that I can have small file sizes on my hard drive, yet still be able to reencode them from the original wav's at a later time should either a) a new audio codec comes out that I like better, or b) I hear artifacts in my lossy versions, and decide that I should have encoded at a higher bitrate.

Of course, there are other reasons which have been pointed out not to do it this way, most notably the fact that just storing complete, lossless versions rather than correction files would take only a bit more space, and serve the dual purpose of avoiding 'bullet-biting'  and serve as an actual backup should you have a drive crash (or brain fart) and lose your lossy versions. This, however, is the rationale for wanting this kind of functionality in the first place.

Edit: I kant spel gud

Ogg+Flac as lossy+recovery file!

Reply #16
What is the point? Isn't that a lot of trouble to save 3 megabytes?

Ogg+Flac as lossy+recovery file!

Reply #17
Quote
The whole point of this is to not HAVE to 'bite the bullet'.

Sometimes we need to take a step back and ask "what exactly are we doing?"

There are two things we must consider: diminishing returns and opportunity costs.

Tonight I encoded Live's A Distance To Here album in MAC's Normal mode and the average bitrate came to 988kbps. This is simply unacceptable compression from a practicality standpoint; not that it's MAC's fault, but lossless compression does not pay for its benefits, IMO. The same album encoded with my MPC settings required 284kbps, less than 1/3 of the lossless size. I simply cannot tell the MPC files from the originals. It's more than "good enough"...my ears think they are flawless.

If my original album were destroyed and I could never, ever get another original copy, I probably wouldn't "care" from an audio standpoint. What am I losing? Some wispy fine detail -60dB down in level that my ears can't hear anyway? Oh well, life goes on.

I personally believe you can get "perfect sound" from existing lossy codecs. It's all a matter of bitrate. Can any human tell the difference between a 450kbps MPC/Ogg file and a 900kbps lossless? I don't think so. That's diminishing returns.

We also have time limitations. Time IS money, to use a oft-used phrase. To go through this effort of making two files with two separate encoders, then burning one of them to media that can only handle 700MB (just a few albums worth)...forget it. In 10-15 years, I will probably rebuy my albums all over again in 24-bit, 96KHz format, so really, I think you can really go overboard with perfectionist tendencies. I found an encoder and a command switch combo that I'm very happy with and I am sticking to it.

From an academic standpoint, this lossy+FLAC exercise might be interesting, but it's not practical nor particularly necessary for "normal listening" needs.

Ogg+Flac as lossy+recovery file!

Reply #18
One more thing, correct me if i have got this idea all wrong, but you'd store the 'lossless' file on CD for example and the ogg's on your harddrive and combine them again to create the original wav so you save a few megs on the media. What happens if you lose all your ogg's (recovery files) in the event of a hard disk faliure, won't the archived files then be of no use because they can't be recovered??

Sorry if i got this idea all wrong, seems i may have done since this hasn't been mentioned yet, but if it is correct then it's an important factor, you would infact need to bck up the ogg onto the CD or other medium as well!

Kris

Ogg+Flac as lossy+recovery file!

Reply #19
Just a quick question to mithrandir. Is your modified mpc command line "mppenc --xtreme --nmt 16 --tmn 32" better than the standard --insane ? And if so why didn't you use a modified version of insane. I ask this because your average bitrate seemed to be a lot higher than I get with insane and 0.90s.

Thanks:)

Ogg+Flac as lossy+recovery file!

Reply #20
Quote
What happens if you lose all your ogg's (recovery files) in the event of a hard disk faliure, won't the archived files then be of no use because they can't be recovered??


Yes you're right. I personally would never use this for backup purposes, unless both the ogg and flac are stored. But storing the original flac alone will result in less space so this is useless...

I do find the idea of a lossy+recovery combo very interesting though...

It would better be used for other purposes, for example: You have a link to an insane 1400 kbps Ogg (let's say it's lossless). You download only so you get 128 kbps bitrate. Later you find you'd like to have better quality, so you download a difference-stream of 64 kbps, so total bitrate of the file will be 192 kbps. And if you want full quality, you only have to download 1208 kbps (1400-192) compared to the full file.

Maybe Ogg bitrate-coupling will work this way, or is it only capable of scaling the bitrate down?

Ogg+Flac as lossy+recovery file!

Reply #21
Quote
Originally posted by madah
Maybe Ogg bitrate-coupling will work this way, or is it only capable of scaling the bitrate down?


i think it is called bitrate peeling and it would be really nice if it could work the way you described it

Ogg+Flac as lossy+recovery file!

Reply #22
Quote
Originally posted by Timmy The Turtle
Is your modified mpc command line "mppenc --xtreme --nmt 16 --tmn 32" better than the standard --insane ? And if so why didn't you use a modified version of insane. I ask this because your average bitrate seemed to be a lot higher than I get with insane and 0.90s.

As I learned from Dibrom in another thread, --xtreme sounds better than --insane at equal bitrates. This is because --xtreme uses the ATH whereas --insane encodes the full 22.05KHz bandwidth, whether your ears can hear all of it or not. Therefore, --insane "wastes" bits that could otherwise be applied to other parts of the audible frequency spectrum. That's why I modified --xtreme rather than --insane: optimization.

--xtreme produces smaller files than --insane, so generally --insane sounds "better" by default. However, the --nmt and --tmn switches allow you to increase headroom beyond a preset's default setting (thereby eating up a lot more bits). Indeed, --xtreme --nmt 16 --tmn 32 produces files that are larger than --insane. That's because --insane's default nmt and tmn settings are lower (i.e. less sensitive, requiring less bits).

Yes, my settings produce files that are measurably better than --insane. It uses more bits and allocates them more optimally. But that's not to say that --insane is inferior from an audible perspective. I'm not claiming that I can hear a difference between --insane and my settings...I use these settings because I don't mind the larger files and know that I have files that are well-suited for transcoding, should the future need arise.

Ogg+Flac as lossy+recovery file!

Reply #23
lossy+recovery is not useless. There are definitely uses for this. Wavpack, for example has an implementation of this. The uses of this idea has been discussed before too. I admit it is not that useful compared to many other things, definitely not to the point of being crucial, but it is useful nevertheless.

@Benjamin
What you described is a perfect optimal implementation of bitrate peeling, but unfortunately it won't be that simple. You can't peel a -q5 vorbis stream down to -q3 bitrate and expect it to sound as good as a -q3 encode, but it won't be that far behind. The way I understand how bitrate peeling works is that the bits in the frames are packed so that the most significant information are packed at the start of the frame and the least significant information at the end of the frame. When you need to peel, you can chop off or truncate the end of the frame and lose only the least significant information, and the stream plays back without any problems.

Ogg+Flac as lossy+recovery file!

Reply #24
i did not explain anything