Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: How important is being syncable? (Read 3920 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

How important is being syncable?

Syncable meaning you can start at some arbitrary point in the bytestream (possibly not seekable) with no other context and after some fumbling find the start of a frame to start decoding.

  • The obvious use case is picking up in the middle of an actual stream where for whatever reason there is no onboarding process. Does streaming even use this property of an audio bitstream or do they use the syncable trait of a container, or not rely on syncable at all?
  • A file player could use it to seek, but do file players do this? To me it seems better to use a combination of a seek table and filling in gaps in the seek table as useful intermediate seek points are encountered for better resolution
  • Corrupt frames can be skipped by fumbling until synced, minimising the damage of a corruption. This doesn't necessarily need "syncable" to do, but it is required to do it efficiently

Using flac just as an example that's well known and because poking around flac prompted the question, flac adds about 3KB per minute of normal audio (CDDA, fixed blocking strategy with normal-sized blocks) to allow the bitstream to be syncable. Flac's implementation of syncable goes to lengths to make it performant so a different codec might add even less. This isn't much for a lossless codec so in practical terms the question isn't hugely relevant, but theoretically it's still interesting IMO.

The way flac is designed to be syncable basically adds or ensures 0's exist in the bitstream wherever possible (so that encountering what looks like a sync code is likely to actually be a sync code, for performance). It also repeats information necessary to decode in the header of every frame. The entire design of the bitstream is to allow syncable to exist so it must have been important at the time, but is it just a legacy feature that is rarely actually used?

 

Re: How important is being syncable?

Reply #1
  • The obvious use case is picking up in the middle of an actual stream where for whatever reason there is no onboarding process. Does streaming even use this property of an audio bitstream or do they use the syncable trait of a container, or not rely on syncable at all?
I don't know a lot about streaming, but I have looked at Icecast. It serves a separate stream to each client, first serving the necessary headers. This means most software wouldn't even see a difference between a file and a stream other than not being able to seek. I'm unsure whether this is how internet radio works in general?

  • A file player could use it to seek, but do file players do this? To me it seems better to use a combination of a seek table and filling in gaps in the seek table as useful intermediate seek points are encountered for better resolution
The way FLAC is designed forces you to do so, there is no other way to seek except for having a seektable pointing to every single frame. Other codecs, like for example WavPack, store a frame length making it possible to quickly skip a frame, but I wouldn't be surprised when players only do this for the last few seeking steps.

Quote
  • Corrupt frames can be skipped by fumbling until synced, minimising the damage of a corruption. This doesn't necessarily need "syncable" to do, but it is required to do it efficiently
Just having a sync code is enough for this and seeking, there is no need for a complete context (sample rate, bit depth, number of channels) like FLAC does.

Quote
Using flac just as an example that's well known and because poking around flac prompted the question, flac adds about 3KB per minute of normal audio (CDDA, fixed blocking strategy with normal-sized blocks) to allow the bitstream to be syncable. Flac's implementation of syncable goes to lengths to make it performant so a different codec might add even less. This isn't much for a lossless codec so in practical terms the question isn't hugely relevant, but theoretically it's still interesting IMO.
All of this is really unnecessary in streaming as we see it these days. It would have been useful if multicast streaming would have picked up, but it didn't.

Quote
The way flac is designed to be syncable basically adds or ensures 0's exist in the bitstream wherever possible (so that encountering what looks like a sync code is likely to actually be a sync code, for performance). It also repeats information necessary to decode in the header of every frame. The entire design of the bitstream is to allow syncable to exist so it must have been important at the time, but is it just a legacy feature that is rarely actually used?
So, the TL;DR here is: the sync code is (by design) useful in seeking and for handling of corruption. The rest of the frame header isn't, because multicasting never took off.

There is one marginally relevant format of which I'm sure decoding cannot be picked up in the middle of the stream for certain streams: MPEG-4 ALS. On encoding, it is possible to select an option that makes 'seeking' impossible, by having each frame depend on the frame before it.
Music: sounds arranged such that they construct feelings.

Re: How important is being syncable?

Reply #2
  • A file player could use it to seek, but do file players do this? To me it seems better to use a combination of a seek table and filling in gaps in the seek table as useful intermediate seek points are encountered for better resolution
The way FLAC is designed forces you to do so, there is no other way to seek except for having a seektable pointing to every single frame. Other codecs, like for example WavPack, store a frame length making it possible to quickly skip a frame, but I wouldn't be surprised when players only do this for the last few seeking steps.
Another possibility is using the seektable if available to seek to the closest previous point then decode ASAP without playing the audio until at the desired location. You're not skipping the last section as such but from the users point of view you are. For a codec as fast as flac it should be transparent unless a seektable doesn't exist, even then it's probably near-transparent with track-per-file on a PC.

Quote
  • Corrupt frames can be skipped by fumbling until synced, minimising the damage of a corruption. This doesn't necessarily need "syncable" to do, but it is required to do it efficiently
Just having a sync code is enough for this and seeking, there is no need for a complete context (sample rate, bit depth, number of channels) like FLAC does.
True, the bundled context is a separate thing that combined with sync could allow a stream to be picked up without prior context (if that's how some streaming works TBD). Probably shouldn't have included it in the definition.

Possibly not even sync is required if there's enough reserved values in the header to make it unlikely that a full decode is triggered (which there would be if only sync was removed from flac say). But if bundled context is removed because useless then crc8 probably is too, and we're back to needing sync so that a full decode isn't triggered every dozens of attempts.

Quote
The way flac is designed to be syncable basically adds or ensures 0's exist in the bitstream wherever possible (so that encountering what looks like a sync code is likely to actually be a sync code, for performance). It also repeats information necessary to decode in the header of every frame. The entire design of the bitstream is to allow syncable to exist so it must have been important at the time, but is it just a legacy feature that is rarely actually used?
So, the TL;DR here is: the sync code is (by design) useful in seeking and for handling of corruption. The rest of the frame header isn't, because multicasting never took off.

There is one marginally relevant format of which I'm sure decoding cannot be picked up in the middle of the stream for certain streams: MPEG-4 ALS. On encoding, it is possible to select an option that makes 'seeking' impossible, by having each frame depend on the frame before it.
Sounds like the concept of I/P frames from video encoding, taken to the extreme with a single I frame followed only by P frames. Say a codec similar to flac did have the concept of I/P frames with an I frame every dozen frames or so, a P frame could use the previous frames samples as the warmup samples instead of having to encode them. There probably would be a benefit but it would be marginal even if model orders increased by a lot. It would be ridiculous and incredibly encoder-expensive to have an LPC of order 100 right?

I know this is diving into the weeds just because they exist, thank you for responding.

Re: How important is being syncable?

Reply #3
I chose FLAC for good reasons and for bad reasons. This was a misunderstanding of mine: who doesn't want streamability, isn't that essential? (Spoiler alert, it isn't always.)
But lossless streaming services like Tidal do need it?

As for consumer playback ... how were Squeezeserver/Squeezebox working? Were they serving streaming audio, or did they actually work as any other file player? ( @Wombat, do I remember correctly that you had one such, and discovered its buggy FLAC implementation?)


There is one marginally relevant format of which I'm sure decoding cannot be picked up in the middle of the stream for certain streams: MPEG-4 ALS. On encoding, it is possible to select an option that makes 'seeking' impossible, by having each frame depend on the frame before it.
If that is the -z option you are referring to, then that is a quite different beast altogether.
Also ... not that it will ever get a large userbase, but it should not be trusted with your data: the reference implementation is buggy and might crash rather than decoding what it just encoded. And, you cannot resort to ffmpeg, it cannot decode the -z's.

Re: How important is being syncable?

Reply #4
As for consumer playback ... how were Squeezeserver/Squeezebox working? Were they serving streaming audio, or did they actually work as any other file player? ( @Wombat, do I remember correctly that you had one such, and discovered its buggy FLAC implementation?)
The Squeezebox/Logitech media server can send files directly to the boxes and its firmware decodes or the server decodes or encodes to several formats you can choose. The server is very flexible and even can do dsd conversation to the limits of the player.
You can stream flac as wav, aiff, pcm, mp3 and more or whatever the players firmware can handle.
The firmare of my player is limited to 24/96. Its native flac implementation can't decode variable bitrate files and can't handle a blocksize above 4608 at 24/96. It even struggles on some 24/96 files with a blocksize of 4608.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: How important is being syncable?

Reply #5
Sounds like the concept of I/P frames from video encoding, taken to the extreme with a single I frame followed only by P frames. Say a codec similar to flac did have the concept of I/P frames with an I frame every dozen frames or so, a P frame could use the previous frames samples as the warmup samples instead of having to encode them. There probably would be a benefit but it would be marginal even if model orders increased by a lot. It would be ridiculous and incredibly encoder-expensive to have an LPC of order 100 right?

I know this is diving into the weeds just because they exist, thank you for responding.

Well, depends on how you use it. MPEG-4 ALS got pretty crazy with high orders. It has no problem going to LPC order 1000, and additionally, it has something it calls long-term predictors. TAK does something rather long term too I seem to recall, but I don't know the details.

I chose FLAC for good reasons and for bad reasons. This was a misunderstanding of mine: who doesn't want streamability, isn't that essential? (Spoiler alert, it isn't always.)
But lossless streaming services like Tidal do need it?
I would be surprised if they would. 'Streaming' is really serving files on demand over a network. The capability FLAC has, able to be multicast, has more strict requirements.

There is one marginally relevant format of which I'm sure decoding cannot be picked up in the middle of the stream for certain streams: MPEG-4 ALS. On encoding, it is possible to select an option that makes 'seeking' impossible, by having each frame depend on the frame before it.
If that is the -z option you are referring to, then that is a quite different beast altogether.
No, I was referring to the r option. See the help text of the reference encoder
Quote
-r# : Random access (multiples of 0.1 sec), -1 = each frame, 0 = off (default)
So MPEG-4 ALS is rather complicated, having frames that depend on each other unless otherwise noted (a random-access frame), and within each frame there are subframes as well (see the option -g, block switching level).
Music: sounds arranged such that they construct feelings.

Re: How important is being syncable?

Reply #6
Sounds like the concept of I/P frames from video encoding, taken to the extreme with a single I frame followed only by P frames. Say a codec similar to flac did have the concept of I/P frames with an I frame every dozen frames or so, a P frame could use the previous frames samples as the warmup samples instead of having to encode them. There probably would be a benefit but it would be marginal even if model orders increased by a lot. It would be ridiculous and incredibly encoder-expensive to have an LPC of order 100 right?

I know this is diving into the weeds just because they exist, thank you for responding.

Well, depends on how you use it. MPEG-4 ALS got pretty crazy with high orders. It has no problem going to LPC order 1000, and additionally, it has something it calls long-term predictors. TAK does something rather long term too I seem to recall, but I don't know the details.

...

Quote
-r# : Random access (multiples of 0.1 sec), -1 = each frame, 0 = off (default)
So MPEG-4 ALS is rather complicated, having frames that depend on each other unless otherwise noted (a random-access frame), and within each frame there are subframes as well (see the option -g, block switching level).

That makes sense, so a random access frame can be thought of as an I frame with everything else being P frames.

Looking at ALS's whitepaper and the sourcecode (wayback machine required) ALS seems to use backwards-adaptive FIR (with LMS algorithm) meaning coefficients aren't transmitted, this takes care of the other big blocker to using large orders. One downside to backwards-adaptive is it makes the codec symmetric killing decode performance. This is my best interpretation based on a lot of this being a black box so excuse any inaccuracy. I assume forwards-adaptive with large orders is not viable because at some point the overhead of warmup samples and coefficients outweighs increasing order and that the reason for this is that LPC efficiency is logarithmic as order increases.

They also try to encode the warm up samples in I-frames smartly, I wonder how common that is in other codecs:
Quote
However, prediction at the beginning of random access frames still constitutes a problem. A conventional K-th order predictor would normally need K samples from the previous frame in order the predict the current frame’s first sample. Since samples from previous frames may not be used, the encoder has either to assume zeros, or to transmit the first K original samples directly, starting the prediction at position K + 1.

As a result, compression at the beginning of random access frames would be poor. In order to minimize this problem, the MPEG-4 ALS codec uses progressive prediction [17], which makes use of as many available samples as possible. While it is of course not feasible to predict the first sample of a random access frame, we can use first-order prediction for the second sample, second-order prediction for the third sample, and so forth, until the samples from position K + 1 on are predicted using the full K-th order predictor (Figure 7).

The paper has a little to say about the long term prediction, mainly that signals have periodic and harmonic components due to fundamental frequencies of instruments, and that the LTP uses lag and gain parameters (presumably a gain of 0 turns off the use of LTP). The LTP prediction is subtracted from the residual and it's this form that is stored. It's interesting but I'm skeptical as to how well it performs when there's multiple instruments with different frequencies at play. It also seems counter-intuitive that a strong corelation will still exist in a useful way after LPC has done its thing (the lag is beyond the orders that flac uses but well within the ~1k orders used with ALS, so it seems they're working on the same data), but maybe I'm misunderstanding or LTP is only used for smaller orders.

Re: How important is being syncable?

Reply #7
Looking at ALS's whitepaper and the sourcecode (wayback machine required) ALS seems to use backwards-adaptive FIR (with LMS algorithm) meaning coefficients aren't transmitted, this takes care of the other big blocker to using large orders.
Not really. The paper says the following

Quote
MPEG-4 ALS specifies an optional backward-adaptive predictor as well [2], but in the following, only the forward-adaptive predictor and related tools are discussed.

The main difference with FLAC is that FLAC stores the integer prediction coefficients directly (this is straightforward and difficult to mess up), MPEG-4 ALS stores so-called parcor coefficients, which one can think of as an intermediate step between autocorrelation and predictor coefficients. Then it uses some more trickery to have these coefficients take up even less space. The first two parcor values are deemed the most important and stored companded, the other values are quantized and fed through an entropy coder. So, this is quite a bit more complicated than FLACs approach. The benefit here is that it is possible to store very high predictor orders with not too much overhead.

If you want to know how well MPEG-4 ALS performs, check my latest lossless codec comparison. I've tested predictor orders up to 5, 10, 20, 40 and 1023.
Music: sounds arranged such that they construct feelings.

Re: How important is being syncable?

Reply #8
Thanks for the parcor explanation, the pieces are there but it would have taken a long time for me to parse it even with the motivation to understand ALS fully (which I don't have TBH, broadstrokes is enough). I did look at the codec comparison and made bad assumptions about why ALS took so long. The main point for me is that they have done "something" to reduce the impact of coefficients on bitstream size, that seemed necessary for large orders to be viable and it looks like it probably is.

The conclusion I have from this thread is that GOP is interesting to a maximal-compression codec or one that deals with huge orders, only mildly interesting to a more moderate codec because of the complexity. The bigger takeaway is that were a codec to be created now, that compute power has gotten to a point where relatively expensive encoding is viable for fractional percentage point gains; and that being the case, not wasting bits on inefficient overhead would be very important as it's the low hanging fruit. Sync (or at least the many trappings flac has to improve sync performance) not being very important is good news in that respect.

Re: How important is being syncable?

Reply #9
bad assumptions about why ALS took so long.
For one thing, the code isn't well optimized. I don't know how much one can speed up by improving, but I remember seeing test runs when FLAC was in its infancy - whether it was before or after 1.0 I don't know - where encoding would be much slower than Monkey's. Like, a factor of two or three. Fast forward a decade and you could flip that ratio. I guess the big part of it is that -e wasn't anymore necessary; I don't remember the exact details, but some are still around early HA threads and I think others I found by wayback machine versions of flac.sourceforge.net

Re: How important is being syncable?

Reply #10
For one thing, the code isn't well optimized.
Where do you base this on?

FLAC's first release was in 2000, but only just: The 23th of December. See the FLAC changelog for that information. LPACs first release was already in 1999, and the program I could still get my hands on from 2002 states (c) 1998-2002 Tilman Liebchen, Technical University Berlin. So, LPACs development already predates FLACs development by two full years. Back then, Liebchen was already teaching courses, not just a student interested in the subject.

LTAC was selected as a basis for MPEG-4 ALS in 2003. There is proof of this here. It says the following (emphasis mine):

Quote
Important notice: An improved version of the LPAC algorithm was recently chosen as reference model for "MPEG-4 Audio Lossless Coding (ALS)", so I decided to freeze the development of LPAC at its current state. Once standardization is completed, I will supply tools to convert LPAC files into MPEG-4 ALS files (which will not require any reencoding).

This makes it very clear MP4ALS' design was not a blank sheet of paper, it is clearly a continuation of LPAC, but in a different format with room for extra features. The core algorithm has stayed the same.

There are also comparisons with old (not very optimized) versions of FLAC: https://hydrogenaud.io/index.php/topic,40124.0.html See specifically reply #20

Anyway, MP4ALS development continued until about 2009. FLAC development stalled in 2007. In 2009 I did my first lossless audio codec comparison. This is comparing 11 years of work by a university professor as part of research with 7 years of work by a volunteer who occasionally had some help. I don't think there is much to say about FLAC having an advantage, really. Still, FLAC was much faster.

There have also been people working on optimizations in the past. Decoding got 20% faster, encoding got 250% faster. That was compared to MP4ALS RM16. There has also been academic work on a faster codec, which can be found here.  There, FLAC 1.1.2 is compared to MP4ALS RM18 and MP4ALS Fast. Their numbers say FLAC 1.1.2 was as fast as MP4ALS RM18 in both encoding and decoding, and their MP4ALS Fast is twice as fast. However, this comparison seems to conflict with all others I've come across in stating that MP4ALS was at some point as fast as FLAC 1.1.2 in decoding, so I'm not sure which is more reliable.

So, all in all, I'm not sure you can expect enormous improvements anymore after 11 years of development, including scrutiny by academics.
Music: sounds arranged such that they construct feelings.

Re: How important is being syncable?

Reply #11
This is comparing 11 years of work by a university professor as part of research with 7 years
I naively assumed this, but this seems untrue. Employment was apparently as teaching and research assistant. Still, I think that comparing someone getting paid to develop a codec with someone doing the same work voluntary in less time doesn't tip the scales in favour of the latter.
Music: sounds arranged such that they construct feelings.

Re: How important is being syncable?

Reply #12
For one thing, the code isn't well optimized.
Where do you base this on?

The readme.txt in the RM23 distribution:
- The ALS reference software is not optimized, particularly not in terms
  of encoder speed.


And nothing more. I took their word for it without making any comparisons.

Re: How important is being syncable?

Reply #13
Ah, yes, that makes sense. I think it is strange that so many people (there are quite a lot of authors named in the changelogs of all source files) worked on this for so many years, and the encoder is still considered unoptimized. With that in mind, such a statement feels a bit like hyping stuff up: this compresses great, and it might get fast in the future! Or it might not.

Maybe it is also interesting to note that quite a few people have been working on getting an ALS encoder in ffmpeg (see here and here) but it never got merged. Perhaps ALS really is too complicated to write a proper encoder for.
Music: sounds arranged such that they construct feelings.

Re: How important is being syncable?

Reply #14
Of course, if that line is in just because nobody removed it ... But it could be for the hype. Members of the development team have published several papers about improvements.

ffmpeg encoder? That was news to me, the ways of ffmpeg are a mystery to say the least - but I think they are doing right focusing on decoding support over encoding. If they are to do any more work on ALS, I'd say it makes more sense to support decoding of the ALS "-z" adaptive RLS-LMS codec. Especially since I got the reference encoder to create files it cannot decode.

(Meanwhile, ffmpeg just got APAC support. Yes, in 2022. I have a hunch that was more for fun, it doesn't appear to be a very sophisticated codec.)

Re: How important is being syncable?

Reply #15
Yes, recently more decoders have been added for oddball lossless audio formats: Bonk and WavArc.
Music: sounds arranged such that they construct feelings.

Re: How important is being syncable?

Reply #16
I'd also assume academics to generally be more interested in the algorithms than the compute efficiency, being able to code and being able to optimise like a madman are not necessarily the same thing.

Re: How important is being syncable?

Reply #17
That is often the case yes.
It seems no other lossless audio codec has been subject to the same amount of scientific output, but when I browsed a few of them, only every now and then there were timing tests. And to compare, those should rather be "everything else equal" than madman-optimal.
It is a bit disappointing that they do not give code source. There are sciences where code is supposed to be available, and I am a bit surprised that it is not so much expected in sciences where the algorithms are the topic of research - but then, IEEE papers are full of proprietary this-and-that.

Re: How important is being syncable?

Reply #18
Anyway, here is a paper that may interest the two of you: https://www.tara.tsukuba.ac.jp/~maki/reprint/Makino/amada17gcce577-580.pdf

About choosing optimal frame length - and optimal prediction order. For hi-res, which the codecs are certainly not fine-tuned for.