Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Lossless Extensions for Opus (Backwards Compatible) (Read 69355 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.


Lossless Extensions for Opus (Backwards Compatible)

Reply #27
Your second paragraph, on the other hand, is mainly a matter of philosophy.  You also appear to be a bit confused.  "We" aren't going to be having any extension.  "I" will be having an extension that others will be free to use.  I do not require your blessing to continue, nor quite frankly, do I care if I have it.  Especially not with that attitude.
He's not being philosophical and he's not confused, he's just trying to dissuade you from trying to do something which he has good reason to believe does not make sense. Nitpicking his choice of pronoun and getting all huffy and confrontative isn't going to make your idea make more sense. Nor will being childish and sarcastic to 2Bdecided.

Assume, for the sake of argument, that you somehow came up with a way to specify an Opus decoder which was bit-exact regardless of seeking etc, so you could actually transmit a delta. It is true that the deltas are more easily losslessly compressible than the original. But not by all that much; certainly not by anywhere near enough that each bit spent on the opus version is a bit you don't have to spend on the delta. (Opus optimizes for fidelity perceived by human listeners, not for bitwise similarity. >20kHz content, for instance, will all show up in the delta no matter how high the opus rate is.) So sending opus+losslessly compressed delta isn't much better than sending opus + losslessly compressed original.

A quick test with three quite different files (speech, classical, heavy metal) showed that jmv's estimate of 7% savings at Opus's default bitrate was on the money. For several different bitrates the savings is about half of what it would be if each bit spent on the Opus file were a bit you didn't have to spend on the delta.

Maybe it's possible to just transmit the lossless version in the padding. Then you don't have to worry about bit-exact Opus decoding and seeking, and the bitrate is only a small proportion higher. But then you're just duplicating with a hack the muxing work the container is already able to do reliably and well. Why not just include two streams at the container level?

You've said that the main reason you're considering this is video. Many videos already carry multiple audio streams.

I don't know all the details of why they did DTS-HD as an extension scheme rather than as another stream. Maybe there are compatibility reasons I don't know about. I do know DTS is frequently used at bitrates 4x higher per channel than Opus, so there's considerably more reason to be trying to look for savings in reducing redundancy. Because DTS is relatively primitive/ psychoacoustically naive, for a given audible quality level (for which DTS will need a much higher bitrate than Opus) the delta between DTS and the original is likely much more compressible than the delta between Opus and the original.

Lossless Extensions for Opus (Backwards Compatible)

Reply #28
Assume, for the sake of argument, that you somehow came up with a way to specify an Opus decoder which was bit-exact regardless of seeking etc, so you could actually transmit a delta. It is true that the deltas are more easily losslessly compressible than the original. But not by all that much; certainly not by anywhere near enough that each bit spent on the opus version is a bit you don't have to spend on the delta. (Opus optimizes for fidelity perceived by human listeners, not for bitwise similarity. >20kHz content, for instance, will all show up in the delta no matter how high the opus rate is.) So sending opus+losslessly compressed delta isn't much better than sending opus + losslessly compressed original.

I've been thinking about this.  I'm not inherently limited to FLAC.  I can use whatever I want to compress the deltas, within legal and technical reasons.  It seems to me that WavPack's lossless encoder should already be optimized for this.  Perhaps I could assess that project's legal terms and apply that to encode the Opus deltas.  That way you have something like WavPack, but will likely be playable on far more devices.  Granted, WavPack's lossless encoder is not optimized for Opus deltas.

A quick test with three quite different files (speech, classical, heavy metal) showed that jmv's estimate of 7% savings at Opus's default bitrate was on the money. For several different bitrates the savings is about half of what it would be if each bit spent on the Opus file were a bit you didn't have to spend on the delta.

Maybe it's possible to just transmit the lossless version in the padding. Then you don't have to worry about bit-exact Opus decoding and seeking, and the bitrate is only a small proportion higher. But then you're just duplicating with a hack the muxing work the container is already able to do reliably and well. Why not just include two streams at the container level?

You've said that the main reason you're considering this is video. Many videos already carry multiple audio streams.

So long as there are savings compared to two discrete tracks, the project is worth it to me.

I don't know all the details of why they did DTS-HD as an extension scheme rather than as another stream. Maybe there are compatibility reasons I don't know about. I do know DTS is frequently used at bitrates 4x higher per channel than Opus, so there's considerably more reason to be trying to look for savings in reducing redundancy. Because DTS is relatively primitive/ psychoacoustically naive, for a given audible quality level (for which DTS will need a much higher bitrate than Opus) the delta between DTS and the original is likely much more compressible than the delta between Opus and the original.

When DTS-HD started gaining popularity in 2006, the vast majority of home theater owners already had DTS decoders.  The DTS-HD stream could be sent (without the lossess deltas) over SPDIF and the existing decoders could understand it.

Having a lossless codec that can also be played on a large number of existing devices is the main premise of my project, and I believe that Opus will be widely adopted.

Lossless Extensions for Opus (Backwards Compatible)

Reply #29
There are no existing hybrid lossless decoders for Opus, so if your goal is to be compatible with an existing lossless implementation...


Now I think I see the problem.  There is a fundamental misunderstanding of what I am saying.  Here is the principal of operation for the encoder:

1. Take the current chunk of input PCM and use an Opus encoder to generate a compressed, lossy, Opus packet.
2. Decode this packet (probably with the fixed-point decoder) and hang on to the PCM values it returns.
3. Subtract these PCM values from the input PCM values and compress those with a lossless codec.
4. Store this this new, losslessly-compressed chunk of data in the Opus padding area.


This is just what I said above:

Quote
If your goal is to be compatible only with an existing lossy format, then you can use any of the above mentioned formats, either extending it to your needs or simply using existing software and standards. In this case, I'd probably recommend just using MPEG-4 SLS, as it is backwards compatible with anything that supports AAC and is already standardized. You could also use MP3 and simply define your own correction format, but since you want 5.1 support, it would probably be easier to use AAC since there is already widespread support for 5.1 AAC.


This is still your best bet to come up with something actually useful.

Lossless Extensions for Opus (Backwards Compatible)

Reply #30
I can't really say that I feel motivated to extend something that's proprietary.  To that end, I am the one being philosophical.

To me, Opus is the sweet spot between what is royalty-free and what I believe will be well-supported.

Lossless Extensions for Opus (Backwards Compatible)

Reply #31
The problem here is on 2 levels:

a) Technical ones: no seeking (unless you can solve that issue somehow, maybe by changing the encoder and seeking back far enough), can't support 44.1kHz (unless you also specify a fixed-point resampler, do an encode and decode passs, and have the correction layer work on the original sampling rate).

b) Even if you surmount those, your advantage over shipping Opus+FLAC (for example) is going to be minimal (1-2%, it seems). But you will do so at a large compatibility disadvantage, because existing FLAC players won't support your new format. Only the new ones to support your Opus+lossless one will do. For technical reasons the CPU requirements will also be substantially higher.

So, you're trading a minimal gain in compression efficiency over a vast increase of computation requirements and a big loss in playback compatibility. There's good reasons to believe such a tradeoff isn't worthwhile and won't get broad acceptance, which is something to consider before you invest engineerigng resources into it.

MPEG-4 SLS has extra advantages such as the ability to slice the correction stream to an arbitrary bitrate and to have it pretty much guaranteed that every bit improves the quality, but it also hasn't taken off very much (you didn't even know it existed).

Lossless Extensions for Opus (Backwards Compatible)

Reply #32
wswartzendruber: The very reason that saratoga mentions MPEG-4 SLS is that you don't need to extend it. It already does what you want.

Lossless Extensions for Opus (Backwards Compatible)

Reply #33
I knew of an MPEG-4 scheme that involved an AAC track and something like an MLP track with deltas, but I didn't know MPEG-4 had a hybrid scheme like DTS-HD.  It still doesn't really change things for me, though.  I want something like this to exist in the royalty-free domain, and be compatible with a large number of devices.

But I think I'm getting ahead of myself by going even this far.  I think it would be best if I created some Opus streams with random padding of ridiculous lengths.  I need to see if existing decoders can handle that.  The Ogg+Opus specs seem to suggest that players should discard packets that are even slightly too large, although I had a very hard time understanding that paragraph.

If I can't have ridiculous padding work on existing decoders, then assessing this any farther is pointless regardless.

Lossless Extensions for Opus (Backwards Compatible)

Reply #34
I want something like this to exist in the royalty-free domain, and be compatible with a large number of devices.


Those two requirements are mutually exclusive at the moment.  Probably though your best bet is to use MP3 and wait another year or two for the patents to expire.

Lossless Extensions for Opus (Backwards Compatible)

Reply #35
I'm looking more towards the future, in terms of who is backing Opus and what level of support I expect it to achieve.  Needless to say, I need to see if my proposed method of embedding deltas even works.  If it does, then I'll probably spend several months playing with different lossless codecs and Opus encoder settings.

Lossless Extensions for Opus (Backwards Compatible)

Reply #36
If your response to saratoga's comments about lack of OPUS-using devices isn't "well, I want to be backwards compatible with what's already in the market" (ie MPEG-4 SLS) but rather what you said "I'm looking towards the future", then why waste your time testing existing decoders?

You've already decided that the existing decoder base isn't worth designing for in your abandonment of existing technologies.  Design a practical OPUS-based solution now and if it's of any value the decoders you've decided already to wait for will support it.

Creature of habit.

Lossless Extensions for Opus (Backwards Compatible)

Reply #37
There is a “tie your hands to commitment” story to tell here. Maximum flexibility is always a good as long as you are a wise dictator and nobody knows about your flexibility. But if not?

This is a codec where some deliberate restrictions have been made. E.g., a short list of allowed sample rates, of which only one single value covers the audible range. Not only is it a restriction, it is a very clear-cut way to verify compliance, and it is so much to the core that messing with it is a very serious violation of the format. Which means, it discourages fragmentation.  If you want to play the “let's make this twist and maybe sooner or later it becomes so widespread that it will be retrofitted into the de facto format if not the formal standard” game, then go somewhere else. That is a benefit of a narrow specification with few but simple rules.

And it works.  Not that I would expect the OP to rule the world hacking the format to allow lossless, but in addition to the “it's not compliant” there is this “even if you do it for 48, you still have only a fraction of what you were after”.

(Another benefit is of course ease of implementation.)

Lossless Extensions for Opus (Backwards Compatible)

Reply #38
If you want to play the “let's make this twist and maybe sooner or later it becomes so widespread that it will be retrofitted into the de facto format if not the formal standard” game, then go somewhere else.

I am not trying to change how Opus works.  My (maybe) future encoder covers a completely different use case:  Lossless compression.  It utilizes standardized Opus decoding for graceful fallback.

By the way, why bring up sample rates?  If I do decide to support 96 kHz, the Opus frames themselves will still be sampled at 48 kHz, and the extra frequency response will be in the deltas.  I will not violate RFC 6716 decoding constraints.


Lossless Extensions for Opus (Backwards Compatible)

Reply #40
the extra frequency response will be in the deltas.


I'm curious how you intend to implement this.

That's a big if.  I assume some sort of resampling from 48 kHz to 96 kHz.  Then I apply the deltas.

But bringing up 96 kHz is entirely premature.

Lossless Extensions for Opus (Backwards Compatible)

Reply #41
It sounding more and more like you're going to end up with a file thats larger then just sending PCM

Lossless Extensions for Opus (Backwards Compatible)

Reply #42
It sounding more and more like you're going to end up with a file thats larger then just sending PCM

Here's my current list of priorities:

1. Determine if I can freely use the padding area without breaking existing decoders.  Success occurs when I can create streams with utterly deformed padding areas that still play flawlessly.

2. Determine if I can resolve this seeking issue.  Success metric is yet to be determined.

3. Find the optimal combination of Opus encoder settings with lossless settings.  Absolute success is measured by beating the size of FLAC by itself, as FLAC is the commonly used lossless codec.  Partial success is measured by gaining some savings compared to bundling Opus + FLAC together.

Failing to resolve #1 or #3 means the project is dead in the water.  Failing on #2, well, we'll see.

Lossless Extensions for Opus (Backwards Compatible)

Reply #43
I am not trying to change how Opus works.  My (maybe) future encoder covers a completely different use case:  Lossless compression.  It utilizes standardized Opus decoding for graceful fallback.


OK, so you want a hybrid format which in principle is a lossless + an Opus, but zipped? A request for Opus will get you the Opus, and a request for lossless will make the hybrid residual file call up the Opus and transfer both?


By the way, why bring up sample rates?  If I do decide to support 96 kHz

But what if you decide to support 44.1 kHz?

Lossless Extensions for Opus (Backwards Compatible)

Reply #44
OK, so you want a hybrid format which in principle is a lossless + an Opus, but zipped? A request for Opus will get you the Opus, and a request for lossless will make the hybrid residual file call up the Opus and transfer both?

Correct.  The stream should be playable through any Opus decoder, but a specialized decoder will know to also decode the lossless information.

EDIT:  Err, maybe not.  I'm confused by your question.  Both the Opus core and lossless differences will be stored in the same stream, but in such a way that is valid to Opus decoding specification.  Those who designed the spec seem to have left two different ways for future extension.

But what if you decide to support 44.1 kHz?

Honestly, I can't say that I care.  I mean I should, but every time I see the characters "44.1 kHz" the only thing that comes to mind is "Yuck."  Still, archiving media libraries that can later be dumped onto Android devices without transcoding would probably be a rather large use case.

Anyway, I went over the Opus spec again.  It states that invalid packets (violating one of the six listed constraints) are reserved for future use and should be ignored by the decoder.  I'm wondering if this isn't the more appropriate way to embed the lossless information, as opposed to embedding it in the padding of each packet.  My main concern with using Opus padding has been that a decoder would simply decide the packet is too large and discard it, thereby discarding the main audio frames that were in it.  But if I keep Opus frames and lossless data in separate packet types, I should be able to avoid this.  I don't care if standardized decoders unconditionally drop the lossless information based solely on the packet's size.

EDIT:  Hrm, packet constraint R5 states that a Code 3 packet must have at least one frame, the full count of which are stored in bits 2-7 of that appropriate byte.  I might store the lossless deltas by having a Code 3 packet with ZERO frames (violates R5) and the appropriate padding.  Essentially, I would have illegal packets composed entirely of Opus padding that would contain the lossless differences.  Because they are illegal (reserved for future use, according to the spec) standardized decoders should simply ignore them.

Lossless Extensions for Opus (Backwards Compatible)

Reply #45
OK, so you want a hybrid format which in principle is a lossless + an Opus, but zipped? A request for Opus will get you the Opus, and a request for lossless will make the hybrid residual file call up the Opus and transfer both?

Correct.  The stream should be playable through any Opus decoder, but a specialized decoder will know to also decode the lossless information.

EDIT:  Err, maybe not.  I'm confused by your question.  Both the Opus core and lossless differences will be stored in the same stream, but in such a way that is valid to Opus decoding specification.  Those who designed the spec seem to have left two different ways for future extension.


Why? Once you are in a world where bandwidth of audio is no issue (and you have to be for this to make sense), what do you need two streams for, when you anyway have to deliver the entire load?



But what if you decide to support 44.1 kHz?

Honestly, I can't say that I care.  I mean I should, but every time I see the characters "44.1 kHz" the only thing that comes to mind is "Yuck."


So, you want a format which
- looks like Opus in order to be played back lossy
- still requires the bandwidth of a lossless
- and cannot fit the CD format
?

Don't bet your neck on world domination ...

Lossless Extensions for Opus (Backwards Compatible)

Reply #46
World domination is irrelevant.  Anyway, I looked at HA's lossless audio comparison page, and it looks like the most efficient codec there ("Lossless Audio") is no longer maintained.  I need to get in touch with this guy.  It seems that he used to work at Skype, but spends his present time doing iOS apps.

Lossless Extensions for Opus (Backwards Compatible)

Reply #47
OptimFROG is showing promise.

Lossless Extensions for Opus (Backwards Compatible)

Reply #48
The 1.1 alpha build has further increased the efficiency of compressing the deltas with OptimFROG, but I haven't yet begun to tweak the Opus encoding parameters.  I'm still getting familiar with them.

Also, after further thought, it seems sensible to include support for arbitrary sample rates.  I can decode the Opus stream, resample to the original sample rate, and then apply the deltas.  I don't like that more CPU overhead is going to be added, but the entire decoding process can be pipelined via threads.  Normal decoders should, of course, ignore the delta packets all together.  I currently plan to violate R6 for code three packets to signal the decoder to ignore them.  I have not actually tested this yet.  There may be complications when it comes to packing this complete scheme inside Ogg.

I would also prefer to keep the Opus bitrate as high as possible, so as to be as transparent as possible on normal decoders.

My continued work on this insanity shall resume on Saturday.

Lossless Extensions for Opus (Backwards Compatible)

Reply #49
Crap, I meant to say that I'm going use R5 violations, not R6 ones.