HydrogenAudio

Lossy Audio Compression => Opus => Topic started by: wswartzendruber on 2013-02-24 18:58:02

Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-24 18:58:02
After reading a good chunk of the Opus RFC, I believe it may be possible to use Opus padding to embed lossless deltas for each packet.  It states that this padding may be of any size, and that while an Opus encoder itself must set any padding to zero, the decoder must accept any value.  It also states that when the decoder has finished reading bytes from a frame for decoding, it may not spill over into the padding for further input.  Hrm...  Sounds like an arbitrary extension field to me.

My only concern is about just how much padding decoders would be willing to accept before deciding that the packet is a DOS attack.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: NullC on 2013-02-24 19:58:03
to embed lossless deltas for each packet.
Deltas are not going to do you any good. The opus specification is quite intentionally not bit exact, and the decoder gives slightly different output on different systems and different versions of the decoder. The design of the format also does not reliably result in compact simple lossless deltas (e.g. the phase shift in hybrid mode, or how HF gets filled with 'folded' noise).

For completeness, the RFC also specifies that the padding MUST be set to all zeros— though certainly some future revision to the RFC could make use of the freedom that comes from also mandating that the decoder ignore the padding.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Garf on 2013-02-24 20:09:52
The opus specification is quite intentionally not bit exact, and the decoder gives slightly different output on different systems and different versions of the decoder.


This is easily solvable: mandate that decoders that want to use this lossless extension use a specified decoder that *is* bit exact with some reference (for example a fixed point implementation) before applying the correction data.

Old decoders will decode not-bit exact and sound fine, and the new ones will decode bit exact and know how to apply the correction info.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: lvqcl on 2013-02-24 20:23:40
And how to losslessly encode 44.1 kHz audio?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Garf on 2013-02-24 20:25:41
And how to losslessly encode 44.1 kHz audio?


Ah, good point. NullC also pointed out to me on IRC this setup will fail while seeking.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-24 20:36:29
When I came up with this idea, my primary use case was movies.  With that said, I can't say that I really care about encoding 44.1 kHz audio.  Is this idea worthless without this feature?

Also, I'm wondering if DTS doesn't already hold a patent on extendable audio streams (DTS, DTS ES, DTS 96/24).
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-24 21:08:29
When I came up with this idea, my primary use case was movies.  With that said, I can't say that I really care about encoding 44.1 kHz audio.  Is my idea worthless without this feature?


Why Opus?  Something like Wavpack would probably work more efficiently unless you want ultra-low bitrate lossy files.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-24 21:10:38
Compatibility with existing Opus decoders.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: db1989 on 2013-02-24 21:14:36
Which, as has already been explained, is not possible.

Opus was not designed as a lossless format. If they had wanted it to have that capability in the future, they would not have implemented various features that rule out such usage. In other words, either it was never considered as an option, or it was considered and dismissed on technical or other grounds.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-24 21:21:22
Nowhere in this thread has it been explained that this isn't possible.  NullC has said that RFC6716 encoders are to set any padding to zero.  But this isn't going to be an RFC6716 encoder.  I aim to write a custom encoder that generates lossless streams that can still be played back by RFC6716 decoders.

Do DTS-ES encoders follow the specs for DTS encoders?  Heck no.  But they're still compatible.

EDIT:  Now with that said, some good points have been raised and there are issues that need to be solved.  Bit-exact inaccuracy is the main one.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-24 21:34:13
Compatibility with existing Opus decoders.


Yeah, but why?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-24 21:43:37
Compatibility with existing Opus decoders.


Yeah, but why?

To have a single encode that you can use on the greatest number of devices.  Looking at the parties backing Opus, I see it becoming a widely-supported codec.  Consider Dolby Digital EX and DTS-ES.  They would have never taken off had they not been compatible with existing implementations of their respective decoders.

In order to convince me this isn't a good idea, I would need to be pointed to a widely-supported lossless audio codec, or one that will arguably become this way in the near future.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-24 21:54:33
Mp3, aac.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-24 22:00:23
Mp3, aac.

Why yes, MP3 does have lossless extensions.  It's also heavily patented.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-24 23:10:32
Do you really care?  The MP3 patent license is almost always paid anyway on hardware devices (e.g. android, dvd players, etc) and no one is going to go after some niche hybrid lossless format for royalties since theres no money in it.

It sounded like you wanted compatibility.  If so, just use MP3 or AAC.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-25 00:32:39
mp3HD can't go beyond stereo and 16-bit.  I'm also unaware of any lossless extensions for AAC.  And I do care about patents because this isn't just about me.  I would also like for others to be able to use this, as well.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-25 00:36:54
mp3HD can't go beyond stereo and 16-bit.  I'm also unaware of any lossless extensions for AAC.


There are actually lossless extensions for both, but aren't you planning to implement your own system?  Who cares what already exists?

And I do care about patents because this isn't just about me.  I would also like for others to be able to use this, as well.


Realistically the number of people who would ever use something like this is so small you will never attract enough attention for patent licensing to matter.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-25 00:43:27
There are actually lossless extensions for both, but aren't you planning to implement your own system?  Who cares what already exists?

Realistically the number of people who would ever use something like this is so small you will never attract enough attention for patent licensing to matter.

Interestingly enough, you answered your own question.  The new system must remain compatible with existing decoders precisely because nobody will have a new one.  At least at first.

Think of it like this, "Hey, here's this lossless codec you can use, and it still plays on anything that can decode Opus."
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-25 01:54:41
There are no existing hybrid lossless decoders for Opus, so if your goal is to be compatible with an existing lossless implementation (which is not what you said above but what you seem to be implying here), you cannot use Opus.  You would have to use the existing Mp3 or AAC variants.

If your goal is to be compatible only with an existing lossy format, then you can use any of the above mentioned formats, either extending it to your needs or simply using existing software and standards.  In this case, I'd probably recommend just using MPEG-4 SLS, as it is backwards compatible with anything that supports AAC and is already standardized.  You could also use MP3 and simply define your own correction format, but since you want 5.1 support, it would probably be easier to use AAC since there is already widespread support for 5.1 AAC.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-25 02:10:39
There are no existing hybrid lossless decoders for Opus, so if your goal is to be compatible with an existing lossless implementation...


Now I think I see the problem.  There is a fundamental misunderstanding of what I am saying.  Here is the principal of operation for the encoder:

1. Take the current chunk of input PCM and use an Opus encoder to generate a compressed, lossy, Opus packet.
2. Decode this packet (probably with the fixed-point decoder) and hang on to the PCM values it returns.
3. Subtract these PCM values from the input PCM values and compress those with a lossless codec.
4. Store this this new, losslessly-compressed chunk of data in the Opus padding area.

Here's the principal of operation for a standard Opus decoder:

1. Take the current input Opus packet and decode it to PCM.
2. Ignore the padding.

Here's the principal of operation for the decoder I would make.

1. Take the current input Opus packet and decode it to PCM with the fixed-point decoder.
2. Take the padding and feed it into the lossless decoder.
3. Add the decoded Opus value and the decoded lossless value together and store it as output PCM.

Does this make more sense now?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-25 04:18:09
I also missed the part above where this proposition will mess up seeking.  I'm hoping someone can explain why.  That seems more like a container issue to me.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: jmvalin on 2013-02-25 04:43:17
I also missed the part above where this proposition will mess up seeking.  I'm hoping someone can explain why.  That seems more like a container issue to me.


It's not a container issue. if you seek within an Opus file, what you decode from that point will not be bit-exact with a decoder that started from the beginning of the file. The difference is so small that it's not audible, but it still means losing the lossless property as soon as you seek.

But really, I think most comments have so far missed the real issue here: A lossless extension to Opus would be mostly pointless. A 48 kHz 16-bit PCM stereo file is 1536 kb/s, and on average a codec like FLAC will compress about 50%, so that means 768 kb/s. If you consider that Opus gives you pretty good quality at 96 kb/s, then it means that Opus+FLAC would cost you 864 kb/s. OTOH, a lossless extension to Opus, would compress slightly worse then than a "native" lossless codec like FLAC (partly due to things like intensity stereo and spectral folding), probably costing us about at *least* 5%, meaning that we'd end up at 806 kb/s. So in the end, we have a messy new extension that saves us only 7% compared to just carrying two sets of files. There's just no point, really. Get over it.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-25 05:02:54
Your first paragraph raises a very good and valid point.  This is a technical issue that I will have to assess.

Your second paragraph, on the other hand, is mainly a matter of philosophy.  You also appear to be a bit confused.  "We" aren't going to be having any extension.  "I" will be having an extension that others will be free to use.  I do not require your blessing to continue, nor quite frankly, do I care if I have it.  Especially not with that attitude.

I am looking for technical issues.  So far, two very good ones have been raised.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: 2Bdecided on 2013-02-25 11:23:34
we have a messy new extension that saves us only 7% compared to just carrying two sets of files.
..and sometimes it's worse, and it's always worse than just a pure lossless file that doesn't depend on a lossy core.

If you must go down that route and want to hack around with something "free", WavPack Hybrid and lossyWAV/lossyFLAC + correction files are built with this in mind.

Cheers,
David.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-25 16:00:09
Why does DTS-HD have a lossy core stream?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: 2Bdecided on 2013-02-25 16:17:49
Why does DTS-HD have a lossy core stream?
To be backwards compatible with already deployed standard DTS decoders.

Cheers,
David.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-25 18:22:27
Why does DTS-HD have a lossy core stream?

To be backwards compatible with already deployed standard DTS decoders.

Ding, ding, ding!  We have a winner!
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: jensend on 2013-02-25 19:17:49
Your second paragraph, on the other hand, is mainly a matter of philosophy.  You also appear to be a bit confused.  "We" aren't going to be having any extension.  "I" will be having an extension that others will be free to use.  I do not require your blessing to continue, nor quite frankly, do I care if I have it.  Especially not with that attitude.
He's not being philosophical and he's not confused, he's just trying to dissuade you from trying to do something which he has good reason to believe does not make sense. Nitpicking his choice of pronoun and getting all huffy and confrontative isn't going to make your idea make more sense. Nor will being childish and sarcastic to 2Bdecided.

Assume, for the sake of argument, that you somehow came up with a way to specify an Opus decoder which was bit-exact regardless of seeking etc, so you could actually transmit a delta. It is true that the deltas are more easily losslessly compressible than the original. But not by all that much; certainly not by anywhere near enough that each bit spent on the opus version is a bit you don't have to spend on the delta. (Opus optimizes for fidelity perceived by human listeners, not for bitwise similarity. >20kHz content, for instance, will all show up in the delta no matter how high the opus rate is.) So sending opus+losslessly compressed delta isn't much better than sending opus + losslessly compressed original.

A quick test with three quite different files (speech, classical, heavy metal) showed that jmv's estimate of 7% savings at Opus's default bitrate was on the money. For several different bitrates the savings is about half of what it would be if each bit spent on the Opus file were a bit you didn't have to spend on the delta.

Maybe it's possible to just transmit the lossless version in the padding. Then you don't have to worry about bit-exact Opus decoding and seeking, and the bitrate is only a small proportion higher. But then you're just duplicating with a hack the muxing work the container is already able to do reliably and well. Why not just include two streams at the container level?

You've said that the main reason you're considering this is video. Many videos already carry multiple audio streams.

I don't know all the details of why they did DTS-HD as an extension scheme rather than as another stream. Maybe there are compatibility reasons I don't know about. I do know DTS is frequently used at bitrates 4x higher per channel than Opus, so there's considerably more reason to be trying to look for savings in reducing redundancy. Because DTS is relatively primitive/ psychoacoustically naive, for a given audible quality level (for which DTS will need a much higher bitrate than Opus) the delta between DTS and the original is likely much more compressible than the delta between Opus and the original.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-25 19:48:14
Assume, for the sake of argument, that you somehow came up with a way to specify an Opus decoder which was bit-exact regardless of seeking etc, so you could actually transmit a delta. It is true that the deltas are more easily losslessly compressible than the original. But not by all that much; certainly not by anywhere near enough that each bit spent on the opus version is a bit you don't have to spend on the delta. (Opus optimizes for fidelity perceived by human listeners, not for bitwise similarity. >20kHz content, for instance, will all show up in the delta no matter how high the opus rate is.) So sending opus+losslessly compressed delta isn't much better than sending opus + losslessly compressed original.

I've been thinking about this.  I'm not inherently limited to FLAC.  I can use whatever I want to compress the deltas, within legal and technical reasons.  It seems to me that WavPack's lossless encoder should already be optimized for this.  Perhaps I could assess that project's legal terms and apply that to encode the Opus deltas.  That way you have something like WavPack, but will likely be playable on far more devices.  Granted, WavPack's lossless encoder is not optimized for Opus deltas.

A quick test with three quite different files (speech, classical, heavy metal) showed that jmv's estimate of 7% savings at Opus's default bitrate was on the money. For several different bitrates the savings is about half of what it would be if each bit spent on the Opus file were a bit you didn't have to spend on the delta.

Maybe it's possible to just transmit the lossless version in the padding. Then you don't have to worry about bit-exact Opus decoding and seeking, and the bitrate is only a small proportion higher. But then you're just duplicating with a hack the muxing work the container is already able to do reliably and well. Why not just include two streams at the container level?

You've said that the main reason you're considering this is video. Many videos already carry multiple audio streams.

So long as there are savings compared to two discrete tracks, the project is worth it to me.

I don't know all the details of why they did DTS-HD as an extension scheme rather than as another stream. Maybe there are compatibility reasons I don't know about. I do know DTS is frequently used at bitrates 4x higher per channel than Opus, so there's considerably more reason to be trying to look for savings in reducing redundancy. Because DTS is relatively primitive/ psychoacoustically naive, for a given audible quality level (for which DTS will need a much higher bitrate than Opus) the delta between DTS and the original is likely much more compressible than the delta between Opus and the original.

When DTS-HD started gaining popularity in 2006, the vast majority of home theater owners already had DTS decoders.  The DTS-HD stream could be sent (without the lossess deltas) over SPDIF and the existing decoders could understand it.

Having a lossless codec that can also be played on a large number of existing devices is the main premise of my project, and I believe that Opus will be widely adopted.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-25 19:54:29
There are no existing hybrid lossless decoders for Opus, so if your goal is to be compatible with an existing lossless implementation...


Now I think I see the problem.  There is a fundamental misunderstanding of what I am saying.  Here is the principal of operation for the encoder:

1. Take the current chunk of input PCM and use an Opus encoder to generate a compressed, lossy, Opus packet.
2. Decode this packet (probably with the fixed-point decoder) and hang on to the PCM values it returns.
3. Subtract these PCM values from the input PCM values and compress those with a lossless codec.
4. Store this this new, losslessly-compressed chunk of data in the Opus padding area.


This is just what I said above:

Quote
If your goal is to be compatible only with an existing lossy format, then you can use any of the above mentioned formats, either extending it to your needs or simply using existing software and standards. In this case, I'd probably recommend just using MPEG-4 SLS, as it is backwards compatible with anything that supports AAC and is already standardized. You could also use MP3 and simply define your own correction format, but since you want 5.1 support, it would probably be easier to use AAC since there is already widespread support for 5.1 AAC.


This is still your best bet to come up with something actually useful.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-25 20:47:56
I can't really say that I feel motivated to extend something that's proprietary.  To that end, I am the one being philosophical.

To me, Opus is the sweet spot between what is royalty-free and what I believe will be well-supported.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Garf on 2013-02-26 09:50:44
The problem here is on 2 levels:

a) Technical ones: no seeking (unless you can solve that issue somehow, maybe by changing the encoder and seeking back far enough), can't support 44.1kHz (unless you also specify a fixed-point resampler, do an encode and decode passs, and have the correction layer work on the original sampling rate).

b) Even if you surmount those, your advantage over shipping Opus+FLAC (for example) is going to be minimal (1-2%, it seems). But you will do so at a large compatibility disadvantage, because existing FLAC players won't support your new format. Only the new ones to support your Opus+lossless one will do. For technical reasons the CPU requirements will also be substantially higher.

So, you're trading a minimal gain in compression efficiency over a vast increase of computation requirements and a big loss in playback compatibility. There's good reasons to believe such a tradeoff isn't worthwhile and won't get broad acceptance, which is something to consider before you invest engineerigng resources into it.

MPEG-4 SLS has extra advantages such as the ability to slice the correction stream to an arbitrary bitrate and to have it pretty much guaranteed that every bit improves the quality, but it also hasn't taken off very much (you didn't even know it existed).
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Jplus on 2013-02-26 12:57:46
wswartzendruber: The very reason that saratoga mentions MPEG-4 SLS is that you don't need to extend it. It already does what you want.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-26 18:59:58
I knew of an MPEG-4 scheme that involved an AAC track and something like an MLP track with deltas, but I didn't know MPEG-4 had a hybrid scheme like DTS-HD.  It still doesn't really change things for me, though.  I want something like this to exist in the royalty-free domain, and be compatible with a large number of devices.

But I think I'm getting ahead of myself by going even this far.  I think it would be best if I created some Opus streams with random padding of ridiculous lengths.  I need to see if existing decoders can handle that.  The Ogg+Opus specs seem to suggest that players should discard packets that are even slightly too large, although I had a very hard time understanding that paragraph.

If I can't have ridiculous padding work on existing decoders, then assessing this any farther is pointless regardless.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-26 19:54:20
I want something like this to exist in the royalty-free domain, and be compatible with a large number of devices.


Those two requirements are mutually exclusive at the moment.  Probably though your best bet is to use MP3 and wait another year or two for the patents to expire.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-26 20:57:34
I'm looking more towards the future, in terms of who is backing Opus and what level of support I expect it to achieve.  Needless to say, I need to see if my proposed method of embedding deltas even works.  If it does, then I'll probably spend several months playing with different lossless codecs and Opus encoder settings.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Soap on 2013-02-26 21:16:43
If your response to saratoga's comments about lack of OPUS-using devices isn't "well, I want to be backwards compatible with what's already in the market" (ie MPEG-4 SLS) but rather what you said "I'm looking towards the future", then why waste your time testing existing decoders?

You've already decided that the existing decoder base isn't worth designing for in your abandonment of existing technologies.  Design a practical OPUS-based solution now and if it's of any value the decoders you've decided already to wait for will support it.

Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Porcus on 2013-02-26 23:20:22
There is a “tie your hands to commitment” story to tell here. Maximum flexibility is always a good as long as you are a wise dictator and nobody knows about your flexibility. But if not?

This is a codec where some deliberate restrictions have been made. E.g., a short list of allowed sample rates, of which only one single value covers the audible range. Not only is it a restriction, it is a very clear-cut way to verify compliance, and it is so much to the core that messing with it is a very serious violation of the format. Which means, it discourages fragmentation.  If you want to play the “let's make this twist and maybe sooner or later it becomes so widespread that it will be retrofitted into the de facto format if not the formal standard” game, then go somewhere else. That is a benefit of a narrow specification with few but simple rules.

And it works.  Not that I would expect the OP to rule the world hacking the format to allow lossless, but in addition to the “it's not compliant” there is this “even if you do it for 48, you still have only a fraction of what you were after”.

(Another benefit is of course ease of implementation.)
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-27 01:49:59
If you want to play the “let's make this twist and maybe sooner or later it becomes so widespread that it will be retrofitted into the de facto format if not the formal standard” game, then go somewhere else.

I am not trying to change how Opus works.  My (maybe) future encoder covers a completely different use case:  Lossless compression.  It utilizes standardized Opus decoding for graceful fallback.

By the way, why bring up sample rates?  If I do decide to support 96 kHz, the Opus frames themselves will still be sampled at 48 kHz, and the extra frequency response will be in the deltas.  I will not violate RFC 6716 decoding constraints.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-27 02:22:14
the extra frequency response will be in the deltas.


I'm curious how you intend to implement this.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-27 02:25:16
the extra frequency response will be in the deltas.


I'm curious how you intend to implement this.

That's a big if.  I assume some sort of resampling from 48 kHz to 96 kHz.  Then I apply the deltas.

But bringing up 96 kHz is entirely premature.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-02-27 02:32:43
It sounding more and more like you're going to end up with a file thats larger then just sending PCM
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-27 03:06:05
It sounding more and more like you're going to end up with a file thats larger then just sending PCM

Here's my current list of priorities:

1. Determine if I can freely use the padding area without breaking existing decoders.  Success occurs when I can create streams with utterly deformed padding areas that still play flawlessly.

2. Determine if I can resolve this seeking issue.  Success metric is yet to be determined.

3. Find the optimal combination of Opus encoder settings with lossless settings.  Absolute success is measured by beating the size of FLAC by itself, as FLAC is the commonly used lossless codec.  Partial success is measured by gaining some savings compared to bundling Opus + FLAC together.

Failing to resolve #1 or #3 means the project is dead in the water.  Failing on #2, well, we'll see.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Porcus on 2013-02-27 18:55:00
I am not trying to change how Opus works.  My (maybe) future encoder covers a completely different use case:  Lossless compression.  It utilizes standardized Opus decoding for graceful fallback.


OK, so you want a hybrid format which in principle is a lossless + an Opus, but zipped? A request for Opus will get you the Opus, and a request for lossless will make the hybrid residual file call up the Opus and transfer both?


By the way, why bring up sample rates?  If I do decide to support 96 kHz

But what if you decide to support 44.1 kHz?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-27 21:27:55
OK, so you want a hybrid format which in principle is a lossless + an Opus, but zipped? A request for Opus will get you the Opus, and a request for lossless will make the hybrid residual file call up the Opus and transfer both?

Correct.  The stream should be playable through any Opus decoder, but a specialized decoder will know to also decode the lossless information.

EDIT:  Err, maybe not.  I'm confused by your question.  Both the Opus core and lossless differences will be stored in the same stream, but in such a way that is valid to Opus decoding specification.  Those who designed the spec seem to have left two different ways for future extension.

But what if you decide to support 44.1 kHz?

Honestly, I can't say that I care.  I mean I should, but every time I see the characters "44.1 kHz" the only thing that comes to mind is "Yuck."  Still, archiving media libraries that can later be dumped onto Android devices without transcoding would probably be a rather large use case.

Anyway, I went over the Opus spec again.  It states that invalid packets (violating one of the six listed constraints) are reserved for future use and should be ignored by the decoder.  I'm wondering if this isn't the more appropriate way to embed the lossless information, as opposed to embedding it in the padding of each packet.  My main concern with using Opus padding has been that a decoder would simply decide the packet is too large and discard it, thereby discarding the main audio frames that were in it.  But if I keep Opus frames and lossless data in separate packet types, I should be able to avoid this.  I don't care if standardized decoders unconditionally drop the lossless information based solely on the packet's size.

EDIT:  Hrm, packet constraint R5 states that a Code 3 packet must have at least one frame, the full count of which are stored in bits 2-7 of that appropriate byte.  I might store the lossless deltas by having a Code 3 packet with ZERO frames (violates R5) and the appropriate padding.  Essentially, I would have illegal packets composed entirely of Opus padding that would contain the lossless differences.  Because they are illegal (reserved for future use, according to the spec) standardized decoders should simply ignore them.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Porcus on 2013-02-28 00:07:16
OK, so you want a hybrid format which in principle is a lossless + an Opus, but zipped? A request for Opus will get you the Opus, and a request for lossless will make the hybrid residual file call up the Opus and transfer both?

Correct.  The stream should be playable through any Opus decoder, but a specialized decoder will know to also decode the lossless information.

EDIT:  Err, maybe not.  I'm confused by your question.  Both the Opus core and lossless differences will be stored in the same stream, but in such a way that is valid to Opus decoding specification.  Those who designed the spec seem to have left two different ways for future extension.


Why? Once you are in a world where bandwidth of audio is no issue (and you have to be for this to make sense), what do you need two streams for, when you anyway have to deliver the entire load?



But what if you decide to support 44.1 kHz?

Honestly, I can't say that I care.  I mean I should, but every time I see the characters "44.1 kHz" the only thing that comes to mind is "Yuck."


So, you want a format which
- looks like Opus in order to be played back lossy
- still requires the bandwidth of a lossless
- and cannot fit the CD format
?

Don't bet your neck on world domination ...
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-02-28 23:52:24
World domination is irrelevant.  Anyway, I looked at HA's lossless audio comparison page, and it looks like the most efficient codec there ("Lossless Audio") is no longer maintained.  I need to get in touch with this guy.  It seems that he used to work at Skype, but spends his present time doing iOS apps.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-01 02:05:48
OptimFROG is showing promise.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-15 03:47:20
The 1.1 alpha build has further increased the efficiency of compressing the deltas with OptimFROG, but I haven't yet begun to tweak the Opus encoding parameters.  I'm still getting familiar with them.

Also, after further thought, it seems sensible to include support for arbitrary sample rates.  I can decode the Opus stream, resample to the original sample rate, and then apply the deltas.  I don't like that more CPU overhead is going to be added, but the entire decoding process can be pipelined via threads.  Normal decoders should, of course, ignore the delta packets all together.  I currently plan to violate R6 for code three packets to signal the decoder to ignore them.  I have not actually tested this yet.  There may be complications when it comes to packing this complete scheme inside Ogg.

I would also prefer to keep the Opus bitrate as high as possible, so as to be as transparent as possible on normal decoders.

My continued work on this insanity shall resume on Saturday.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-15 06:13:43
Crap, I meant to say that I'm going use R5 violations, not R6 ones.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: LithosZA on 2013-03-15 07:37:03
Lossless extensions to Opus would be completely pointless IMO except maybe if the resulting (lossless extensions + lossy) file would be smaller than other widely supported lossless formats.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-16 22:53:59
Yeah, I hear you there.  I've contacted the OptimFROG developer to see if he's interested, but haven't heard anything back.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-19 03:42:05
I have heard back from the OptimFROG developer.  He has expressed great interest in the idea and we are corresponding.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: C.R.Helmrich on 2013-03-19 20:45:53
When I came up with this idea, my primary use case was movies.

I still don't get this. Sorry, I haven't followed this thread much and might have missed something, but: How do movies benefit from lossless and backward compatibility to Opus at the same time? Only Blu-Ray does lossless audio beside video, I think, and if you're going for Hi-res video, why not use FLAC from the start? It's much more widely supported than Opus on mobile and media-Center devices right now (and probably also in the next few months or even years, industry can be quite slow).

Second question for clarification: so now you're trying to get Florin to develop a coding scheme for the lossless Opus-extension layer based on OptimFrog?

Chris
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-20 00:12:28
I solicited Florin for interest in the project and pointed him to this thread.  He has responded with a very positive attitude and some ideas of his own.  We have different approaches on the details (API, where certain information is stored, etc...), so I am probably going to be making a lot of concessions.  I look forward to his valuable input on my ideas, and most importantly, his expertise with lossless compression.  If this project becomes successful, it will be because of him.

EDIT:  The initial use case was for movies.  He and I are talking about other sample rates and need to work that out before I make anymore statements.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: NullC on 2013-03-25 21:28:38
I have heard back from the OptimFROG developer.  He has expressed great interest in the idea and we are corresponding.
Is he aware that the developers of Opus consider this to be a bad idea and do not intend to support it and will not be maintaining a bit exact decoder at this time? Or that the resulting streams are not losslessly seekable?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-25 22:23:37
I have heard back from the OptimFROG developer.  He has expressed great interest in the idea and we are corresponding.
Is he aware that the developers of Opus consider this to be a bad idea and do not intend to support it and will not be maintaining a bit exact decoder at this time? Or that the resulting streams are not losslessly seekable?

He struck me as very unimpressed with the criticisms provided in this thread.  With that said, I suspect he doesn't have a lot of interest in whether or not the Opus developers agree.  What fundamentally matters is that Opus is properly specified and those specifics allow for extension packets.  I don't see how our use of those extension packets is really any of their concern, given that we make our extension formatting readily identifiable (so as not to throw off future decoders that use other extensions).

As far as bit-exact decoding goes, he and I currently agree that using an existing fixed point implementation is the way to go.  This would likely be used to generate the deltas and later to decode the Opus core stream.

He has not mentioned lossless seeking, but we may be able to use the extension packets to help the decoder out.

I'm still waiting to hear back from him.  It's possible he has lost interest in the project.  If this is the case, I will continue alone.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: jmvalin on 2013-03-25 22:53:51
He struck me as very unimpressed with the criticisms provided in this thread.  With that said, I suspect he doesn't have a lot of interest in whether or not the Opus developers agree.  What fundamentally matters is that Opus is properly specified and those specifics allow for extension packets.  I don't see how our use of those extension packets is really any of their concern, given that we make our extension formatting readily identifiable (so as not to throw off future decoders that use other extensions).


Readily identifiable assuming people actually know about your stuff -- which they won't. The current spec specifically says that the padding *MUST* be set to zero. This is done intentionally, so that any future extension has to be agreed on in the IETF WG to avoid any issue. What you're suggesting is essentially a rogue, incompatible extension that disregards the standard process.

As far as bit-exact decoding goes, he and I currently agree that using an existing fixed point implementation is the way to go.  This would likely be used to generate the deltas and later to decode the Opus core stream.


You're aware that the output of the fided-point decoder *will* change in the future, right?

I'm still waiting to hear back from him.  It's possible he has lost interest in the project.  If this is the case, I will continue alone.


Fortunately, that is indeed likely.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Soap on 2013-03-25 22:54:42
As far as bit-exact decoding goes, he and I currently agree that using an existing fixed point implementation is the way to go.  This would likely be used to generate the deltas and later to decode the Opus core stream.


Ok, help me out here.

The whole idea of doing this hack (and don't get me wrong, "hack" isn't a bad word), IIRC, was so that it was a playable OPUS stream with padding which would allow you to make it lossless if you wanted.  OPUS was chosen over existing solutions because it was "not proprietary".

Yet you've sought out the help of a coder who has not released the code to his proprietary codec.  A codec you're going to put in the padding in violation of the OPUS spec.  A codec which likely is patent encumbered just not tested because its ripple in the pond is so small.

And not just any OPUS decoder will work with this stream.  We'll need to use a specific OPUS decoder which isn't even the official code.

I'm not sure we agree on what "proprietary" means.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-25 23:44:55
I am no longer seeking to use Opus padding.  I now intend to have these extension packets violate constraint R5.  Per the RFC, doing so triggers the standardized decoder to ignore them because they are reserved for future extensions.

Florin's recent communication with me indicated that he's going to be far more open with his new stuff.  The issue of his work possibly being patented concerns me, though.  I had not thought of that.

And who says I have to update the fixed point decoder I choose to use?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: jmvalin on 2013-03-25 23:55:31
I am no longer seeking to use Opus padding.  I now intend to have these extension packets violate constraint R5.  Per the RFC, doing so triggers the standardized decoder to ignore them because they are reserved for future extensions.


Well, I wouldn't bet on that rule actually being widely followed.

And who says I have to update the fixed point decoder I choose to use?


Congratulations, you are now the proud maintainer of a fork of the entire Opus codebase. Best of luck in your new responsabilities.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Soap on 2013-03-26 00:15:55
Florin's recent communication with me indicated that he's going to be far more open with his new stuff.


Those are oft-said words, less frequently acted upon.
The issue of his work possibly being patented concerns me, though.  I had not thought of that.



It doesn't really matter, you are unlikely to ever be a target of action.

Neither of those points of mine matter in the big picture.  Except that I'm trying to get you to think about your apparently contradictory views on "proprietary".
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-26 02:53:43
I am no longer seeking to use Opus padding.  I now intend to have these extension packets violate constraint R5.  Per the RFC, doing so triggers the standardized decoder to ignore them because they are reserved for future extensions.


Well, I wouldn't bet on that rule actually being widely followed.

I've been thinking about that for a couple weeks.  Right now my plan is to declare a packet with no frames (violate R5) and then have the deltas compressed in "padding" for that packet.  So when I say I'm no longer using padding, I mean I'm not using padding in a packet with frames.  The deltas will be in what is essentially a code 3 packet with zero frames and lots of padding.  This increases my chances of a non-compliant decoder just going, "Durrrr, zero frames, now padding, uh...next..."  I've also thought of violating another R5 rule by declaring a padding size inconsistent with the total frame length and packet size.  I think that this second way is maybe not a good idea.

And who says I have to update the fixed point decoder I choose to use?


Congratulations, you are now the proud maintainer of a fork of the entire Opus codebase. Best of luck in your new responsabilities.

I need to retain the entire code base to keep the fixed-point decoder? 
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: jmvalin on 2013-03-26 03:47:42
And who says I have to update the fixed point decoder I choose to use?


Congratulations, you are now the proud maintainer of a fork of the entire Opus codebase. Best of luck in your new responsabilities.

I need to retain the entire code base to keep the fixed-point decoder? 


Well, I was assuming you also wanted to have tweaks in the encoder to make it not use the many features that improve the perceptual quality while making the residual larger... Not to mention the mess of taking the encoder and decoder from different codebases (many functions are shared). Lots of fun!
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: 2Bdecided on 2013-03-26 09:35:29
I wonder why you've chosen opus? Rather than a more widely implemented codec that supports 44.1kHz natively?

By the time you've finished, mp3's patents will probably have expired.

Not that mp3+FLAC in the way you're proposing it would be attractive.

Cheers,
David.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-26 19:59:10
MP3 stops at stereo.  Besides that, there is already mp3HD.  It works by storing the deltas as ID3v2 tags. 

Quote
Well, I was assuming you also wanted to have tweaks in the encoder to make it not use the many features that improve the perceptual quality while making the residual larger... Not to mention the mess of taking the encoder and decoder from different codebases (many functions are shared). Lots of fun!

That kind of thing might happen if I get a proof-of-concept working.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: db1989 on 2013-03-26 20:04:36
MP3 stops at stereo.
http://en.wikipedia.org/wiki/MP3_Surround (http://en.wikipedia.org/wiki/MP3_Surround)

Quote
Besides that, there is already mp3HD.  It works by storing the deltas as ID3v2 tags. 
I don’t see how mp3HD being implemented in a way that you find distasteful (as many people will find your attempt to kludge lossless into Opus) rules out your ability to develop a scheme you like more, thus making this not a very informative reply as to the reasoning behind your dismissal of MP3 as a base.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-27 01:26:18
First off, mp3 Surround (which I would hypothetically build on) is not open and I have no desire to extend it.  Second, if their best answer for mp3HD was to embed the deltas in freaking header metadata, that tells me the bitstream is fundamentally non-extendable.  Opus was designed to be extended from the beginning, similar to DTS.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-03-27 01:52:57
First off, mp3 Surround (which I would hypothetically build on) is not open


OptimFROG, the proprietary format on which you have hypothetically decided to build on is not open either.

Second, if their best answer for mp3HD was to embed the deltas in freaking header metadata, that tells me the bitstream is fundamentally non-extendable.


MP3 is a compression format, not a container.  If you want an extensible container, use one of the many that supports MP3, such as MP4 or MKV.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Soap on 2013-03-27 01:57:23
MP3 is a compression format, not a container.  If you want an extensible container, use one of the many that supports MP3, such as MP4 or MKV.


Now that's just silly, saratoga.  If a container was an acceptable answer one could just put an OPUS stream and a FLAC stream in that!

/s
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-03-27 02:02:24
Actually I suppose if the goal is to have this work on any old software device, the obvious format to use is AAC, since every AAC player in existence supports MP4 with >2 channels, and you can easily insert the lossless correction data into the MP4 file as a separate stream without breaking anything.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-27 03:08:47
I see that reiteration is necessary.

1. I have no desire to extend something that is not open and royalty-free.
2. Florin has stated his intention of having his new work be open.
3. AAC is not royalty free.
4. Packing Opus and FLAC together in the same container takes more space than I would like, and seems redundant.
5. I have lost contact with Florin, and an OptimFROG-derived codec for deltas may be out of the question.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2013-03-27 03:23:46
I see that reiteration is necessary.
1. I have no desire to extend something that is not open and royalty-free.


How do you reconcile this statement with all of the posts you made from Feb 28 onward detailing your plans to extend a format that was not open?

2. Florin has stated his intention of having his new work be open.


He told you he planned to open OptimFROG?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-27 03:44:41
1. I am not out to extend OptimFROG, I'm out to extend Opus.  If this succeeds, these streams will not be decodable by OptimFROG, but by Opus.  OptimFROG may be a catalyst.

2. He seems to have greatly improved OptimFROG and plans to make that standardized and open.  I don't know about royalties, though.  I'm also much more concerned about possible patent "infringement."

EDIT:  It would be neat if he would get back to me.  Even if just to tell me I'm a total idiot.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-29 03:37:54
I sent Florin one last email asking for input.  If I don't get a response within a week, I'm going on without him.

I'm not going to stop until I see for myself that this will not work well.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: C.R.Helmrich on 2013-03-29 14:10:27
I'm not going to stop until I see for myself that this will not work well.

This is an exemplary attitude! Sometimes I wish we had a bit more of this in the scientific and engineering world, since it's often the only way to achieve progress.

Honestly, good luck with your project. And please, do let us know about your findings.

Chris
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-30 21:58:00
I'll just use this thread as sort of a log.

Anyway, I've tested Monkey's Audio against Opus residuals at a variety of bitrates and it is proving to be competitive against OptimFROG.  Given that I have no idea what lossless codec may eventually be used to compress the residuals, I need to redo the principles of operation to accommodate one that uses variable frame sizes (that is, other than 60 ms, which I have been assuming up until now).  It is highly likely that there will be many Opus packets for each delta packet, and this ratio is likely to vary throughout a stream.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Dynamic on 2013-03-31 10:54:01
With mention of an open lossy codec being a requirement, the other option is Ogg Vorbis, which supports specific sampling rates and isn't too far behind Opus in bitrate efficiency and is an open format that has fairly widespread support in the wild (main exception being Apple, despite Wikipedia's use of Vorbis). Then you have no worry about resampling consistently so that the hybrid remains lossless. You then just need to presumably place non-Vorbis packets in the Ogg stream that all Vorbis decoders (except your hybrid decoder) will ignore. Presumably it would also be easy to strip out only Vorbis packets from the Ogg stream to obtain a smaller lossy file.

The other option where sampling rate can be specified and still be expected to play is Xiph.org's CELT (forerunner to the CELT half of Opus you're thinking of), but plain CELT decoder support in the wild is very limited and will very likely remain so now Opus has superceded it, even though I believe Google's plugin for Chat/Hangouts uses CELT in its new Studio mode (click Learn More in this link (http://support.google.com/plus/answer/1216370?hl=en)) and will possibly implement Opus in due course, possibly piggybacking on WebRTC integration.

On the positive side for Opus, it's possible that being standardized by IETF with clearly specified open patent licensing, support will eventually surpass that of Vorbis.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Destroid on 2013-03-31 11:55:27
Dynamic's suggestion is very interesting. In addition to the points mentioned there are a few other things I wanted rehearse here:

- former lossless Vorbis [q 10] discontinued;
- new iteration of Vorbis, possibly ditched due to Ghost/OPUS and the possibility of broken compatibility;
- the AoTuV project low-bitrate settings (q-1, q-2) sounded better to me than Speex at comparable bitrate, IMO.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-03-31 15:40:04
One of the main reasons for choosing Opus is because of the IETF standardization Dynamic mentioned.  Also, Opus lets me extend it at the sub-container level.  There should be no reason a lossless/hybrid stream can't be placed in any container that can handle standardized Opus.  My hybrid encoder API so far is completely container agnostic.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-04-01 22:52:22
Florin got back to me.  He's just in a bit of a personal dilemma and estimates reviewing my comments in one week.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-04-19 06:38:33
One week, right.  I think he's in a bit more of a jam than he admitted.

Whatever.  As soon as libopus-1.1 comes out, I'm gonna start gutting the tree until I have a fixed-point decoder and nothing else.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-08-09 22:24:48
One week, right.  I think he's in a bit more of a jam than he admitted.

Whatever.  As soon as libopus-1.1 comes out, I'm gonna start gutting the tree until I have a fixed-point decoder and nothing else.

All right, now that I've graduated from college (Bachelor in Computer Science) and have myself a job, I can get back to the task at hand.  After having thought about this for a while, I think the best answer may just be to implement my own Opus (CELT) decoder in fixed-point arithmetic.  This is going to really suck, but I need stability with a base I understand.  I also don't see myself needing the SILK parts of the reference implementation.  I should know more this weekend after I've gone over the appropriate parts of the RFC.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2013-11-30 20:08:46
Florin finally got back to me!  He's been all tied up with real-life stuff.  Moving and whatnot.  He still seems determined to help.  I won't share what he told me because I don't have his permission to do so, but I'll share what I said to him:

Quote
Hi Florin,

I'm on my tablet so I won't type a lot.  I have outlined some main points below.

1. Thank you for getting back to me.

2. I am in the United States (California).

3. I have determined that we should have our own implementations of Opus.

4. We only need the MDCT-based CELT layer.

5. I don't know how to do any of this, so I am learning from the ground up.  I have just finished, as practice, my first FFT implementation, and then used that to create (from scratch) an FFT-based convolver that uses overlap-and-add.  In the coming days I will move onto learn and implement the MDCT transform.

6. I hate Jean-Marc for trying to shoot this down so fast, but he has a point.  Extension packets need to be standardized first.

7. I've lost the work I did, but it wasn't applicable anyway.  I've since started using Git.

8. I still have every intention of making this happen.

9. You can track my learning progress here:  https://github.com/wswartzendruber/logic/ (https://github.com/wswartzendruber/logic/)

-William
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: darkbyte on 2014-08-07 10:52:15
This stuff died quickly.

Nevertheless i wonder why CELT was not designed in mind to be extensible to lossless. Most new video codecs support lossless coding as well (H.264, H.265, Daala) which is a much more complex area and still achievable. I wonder why audio codecs are not developed with this criteria in mind. Would it reduce CELT's efficiency considerably?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-08-07 14:09:49
This stuff died quickly.

Nevertheless i wonder why CELT was not designed in mind to be extensible to lossless. Most new video codecs support lossless coding as well (H.264, H.265, Daala) which is a much more complex area and still achievable. I wonder why audio codecs are not developed with this criteria in mind. Would it reduce CELT's efficiency considerably?

This isn't dead.  I just haven't come back to run my mouth because I don't have anything besides wrapper stuff to show for it, and upon further consideration, that code has to change.  The wrapper code was written to put CELT and lossless data in different packets.  That was before I understood how multistream worked.  Now that idea just seems ridiculous.  Also, there's no way I'm trusting stuff like mkvmerge to say, "Hey, it's a packet full of jibberish with no real Opus data in it.  Yes, I will mux this in."

No, the lossless deltas will be embedded directly in the padding for each packet.  The wrapper around libopus will be written as a proof of concept that I can embed junk in Opus packets and get away with it in a variety of scenarios.  Then after that, I will start on my own CELT implementation and the fun stuff will begin...

There seems to be some sort of decoder state I will have to synthesize when seeking.  That might be a nail.  If I give up on this project, everyone here will know.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: darkbyte on 2014-08-07 14:35:24
Okay sorry, it's just there was no update for months, but now i understand.

I don't know by the way which is the better path. Putting correction data into the same container, the same way MPEG4-SLS do by mp4, or into a separate file how WavPack Hybrid does it. The latter makes very easy to copy only the lossy part to somewhere else but the number of files explodes if you have lot of audio on your drive. The first requires a tool to copy only the lossy part.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-08-07 19:13:17
I've already decided on using the same container.  Makes for simpler management.  I had trouble deciding if I wanted the deltas in the same packets as the lossy CELT bits or if I wanted them in their own extension packets.

EDIT:  JMV has expressed concern that I'm creating a rogue extension.  Each padding area will start off with its own identifier like "wswartzendruber_whacked_out_experiment" or some such thing.  If future, standards-compliant decoders that support some new official extension mistake my lossless delta bits for something else, that's on him and the IETF.

EDIT:  Stuffing the lossless deltas in the padding of the CELT packets also absolves some time stamping issues I've been concerned about when it comes to Ogg encapsulation.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2014-08-07 19:28:13
This stuff died quickly.

Nevertheless i wonder why CELT was not designed in mind to be extensible to lossless. Most new video codecs support lossless coding as well (H.264, H.265, Daala) which is a much more complex area and still achievable.


I wouldn't say that video codecs are more complex.  Since the input data size is is ~ 3 orders of magnitude larger, and the encoding speed isn't exponentially slower, the complexity is probably much lower than a typical audio codec.  Theres just a lot more data to process.

I wonder why audio codecs are not developed with this criteria in mind. Would it reduce CELT's efficiency considerably?


Lossless mode in H.264 basically skips most of the codec and just does lossless packing.  Not quite the same as being extensible to lossless, more like just including two different codecs in one format.  I think from the point of view of CELT, there is a lossless mode, and it is called "FLAC".
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: jmvalin on 2014-08-07 19:34:48
EDIT:  JMV has expressed concern that I'm creating a rogue extension.  Each padding area will start off with its own identifier like "wswartzendruber_whacked_out_experiment" or some such thing.  If future, standards-compliant decoders that support some new official extension mistake my lossless delta bits for something else, that's on him and the IETF.


So what you're saying is that the IETF is no longer free to have an extension system based on an 8-bit ID because it would make collisions likely (in your example, if an 8-bit ID were "w" there would be a collision). That pretty much fits the definition of a rogue extension.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-08-07 20:07:28
So what you're saying is that the IETF is no longer free to have an extension system based on an 8-bit ID because it would make collisions likely (in your example, if an 8-bit ID were "w" there would be a collision). That pretty much fits the definition of a rogue extension.

What I'm saying is that you've been getting in my way from day one and that's about all you've been doing.  I have approached you, very tactfully, in the Opus mailing list about how to best have an extension.  I recall you telling me to get lost.

Despite what you may think, I really do care about where the IETF is going to take Opus and I do wish to remain compatible.  That is why I approached you.  But if your only response is "get over it" and you guys somehow decide that an 8-bit extension ID without a null terminator is a good idea, then yes, it will be a rogue extension.

Tell me something...  After openly standardizing a royalty-free codec, how did you not realize this would eventually happen?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: saratoga on 2014-08-07 20:12:29
I think people generally expect poorly thought out extensions that break things. There is nothing you can do to completely prevent them and historically a great many have cropped up for popular formats.

The best you can do is discourage them.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: jmvalin on 2014-08-11 19:10:11
Tell me something...  After openly standardizing a royalty-free codec, how did you not realize this would eventually happen?


When winter arrives, I realize I'm likely to catch a cold. Doesn't mean I look forward to it.

On a different topic, how about you start by showing us how great you new extension works in practice and then I may take you a little bit more seriously.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: skamp on 2014-08-11 20:13:55
how did you not realize this would eventually happen?


When winter arrives, I realize I'm likely to catch a cold. Doesn't mean I look forward to it.


(http://i.imgur.com/eDbCQC2.png)
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: 2Bdecided on 2014-08-12 13:41:42
Most new video codecs support lossless coding as well (H.264, H.265, Daala) which is a much more complex area and still achievable.
The adoption and use of scalable coding is almost non-existent. If you count MVC (using just the left image to get 2D, and both to get 3D) then that's one (tenuous) example.

As for scalable to lossless, does anyone have a single example of where a significant number of people actually use this?

I know there are standards for scalable, and scalable to lossless - and I know people play with this stuff - and I know of some situations where it even kind of makes sense (e.g. wavepack lossy) - but even then it's a niche of a niche. Doing it from a psychoacoustic/psycho-visual lossy core makes very little sense and becomes a niche of a niche of a niche.

Don't get me wrong - it's great fun to play with this, and you'll always learn something by experimenting - but please understand there's at least one very good reason why almost no one is using this approach. If you want to solve the problem of lossless/lossy co-existance, there are other things you could do that more people would use - you might even be the first to do them exceptionally well based on Opus. e.g.
http://sysadminsjourney.com/content/2008/1...-mp3-fly-mp3fs/ (http://sysadminsjourney.com/content/2008/12/11/convert-flac-mp3-fly-mp3fs/)
https://github.com/robinbowes/flac2mp3 (https://github.com/robinbowes/flac2mp3)
or some new idea.

Cheers,
David.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: skamp on 2014-08-12 14:03:10
If you want to solve the problem of lossless/lossy co-existance, there are other things you could do that more people would use - you might even be the first to do them exceptionally well based on Opus. e.g.
http://sysadminsjourney.com/content/2008/1...-mp3-fly-mp3fs/ (http://sysadminsjourney.com/content/2008/12/11/convert-flac-mp3-fly-mp3fs/)
https://github.com/robinbowes/flac2mp3 (https://github.com/robinbowes/flac2mp3)
or some new idea.

I don't understand what transcoding software has to do with "scaling"?

Never mind, I misread your post.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-09-06 17:58:18
Well, none of the players I've tested are falling for a Code 3 packet with padding and zero frames.  MPlayer outputs massive warnings and Chrome just goes into cardiac arrest.

EDIT:  That means I have to take the alternative route and embed the deltas in packets after the CELT info.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-09-06 18:03:58
As for scalable to lossless, does anyone have a single example of where a significant number of people actually use this?

Go to your local retail store and find the Blu-rays.  Most of those will have the primary audio track in DTS-HD.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-09-06 19:56:03
Good grief.  Firefox can't even handle packets (made with opus_packet_pad) that have had 64 bytes added to them.  Chrome and GStreamer do just fine handling a 4x padding increase.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-09-07 00:01:53
Apparently editing posts has some kind of expiration.

Anyway, here's a sample file with random garbage for padding.  Each packet has a different amount.  I yanked the source from 2L's free-to-download demo page.

http://www.wswartzendruber.name/uploads/op...g-test-001.opus (http://www.wswartzendruber.name/uploads/opus-padding-test-001.opus)

I'm interested in finding out about different people's playing experiences with it.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: CoRoNe on 2014-09-07 11:41:59
Plays fine with LAV Filters, MPC-BE's internal filters and DC-Bass Source Mod.
But, bitrate 2628kbps?!
"Maverick Tools", something of your creation?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-09-07 19:27:58
Plays fine with LAV Filters, MPC-BE's internal filters and DC-Bass Source Mod.
But, bitrate 2628kbps?!

Good.  And it's a stress test to see how well players honor section 3.2.5 of the RFC.

"Maverick Tools", something of your creation?

Yes.  maverick-tools is a the front-end that also acts as a sandbox for testing higher-level principles.

https://github.com/wswartzendruber/maverick-tools (https://github.com/wswartzendruber/maverick-tools)

libmaverick is going to be the backend.  It will initially wrap libopus.  There is already work here, but it has to be redone.

https://github.com/wswartzendruber/libmaverick (https://github.com/wswartzendruber/libmaverick)
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: 2Bdecided on 2014-09-08 10:45:52
As for scalable to lossless, does anyone have a single example of where a significant number of people actually use this?

Go to your local retail store and find the Blu-rays.  Most of those will have the primary audio track in DTS-HD.
You are right, a very good example - thank you.

I think the mandatory status and extremely high bitrate of the lossy DTS core make this quite a different proposition from Opus.

Cheers,
David.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-09-09 00:55:25
You are right, a very good example - thank you.

You are most welcome.

I think the mandatory status and extremely high bitrate of the lossy DTS core make this quite a different proposition from Opus.

I'm not sure what you mean by "mandatory status."  As far as the Opus bitrate goes, I can take that quite high.

It's not so much what Opus was intended to do.  I don't care about that.  It's about what Opus can do.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: Seren on 2014-09-13 03:24:52
Apparently editing posts has some kind of expiration.

Anyway, here's a sample file with random garbage for padding.  Each packet has a different amount.  I yanked the source from 2L's free-to-download demo page.

http://www.wswartzendruber.name/uploads/op...g-test-001.opus (http://www.wswartzendruber.name/uploads/opus-padding-test-001.opus)

I'm interested in finding out about different people's playing experiences with it.


Firefox Nightly is saying "Video can not be played because the file is corrupt."...
Video?
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: wswartzendruber on 2014-09-13 03:38:40
Firefox Nightly is saying "Video can not be played because the file is corrupt."...
Video?

I believe this is happening because I don't fundamentally understand how granule postitioning works.  This is something I'm working on.  I'm guessing the other players just don't care.  opusinfo complains about it as well.
Title: Lossless Extensions for Opus (Backwards Compatible)
Post by: itisljar on 2014-09-13 08:47:28
Chrome with VLC plugin just plays it; Foobar plays it, but when checking integrity, Warning: Nonsense Ogg granulepos value, file is not properly seekable, Warning: Reported length is inaccurate : 3:48.360000 vs 3:48.380000 decoded. But the file plays. Seeking works, though. I don't know how precise, it seems pretty precise.