HydrogenAudio

Lossless Audio Compression => FLAC => Topic started by: andrewfg on 2012-11-22 11:06:12

Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-22 11:06:12
IMHO flac has one technical weakness that will prevent it taking off as THE standard -- namely it only has Variable Bit Rate compression algorithms. Which means that one cannot predict in advance the length of ouput file that will be generated for any input stream.

This is a problem when you want to transcode another format of audio file to flac and stream it over Http. The Http protocol requires sending a Content Length (number of bytes) header at the start of the streaming process. With flac you cannot know the number of bytes until after the full file has been transcoded. So this means that Http must either wait until the full transcoding has been completed, or alternatively if it wants to start streaming the data immediately, then it must provide some guessed value in its Content Length header (and this is not conformant to the Http specifications, and may cause problems in some music players...)

By contrast, if you are (say) transcoding to mp3 and streaming that via Http, then the transcoder can select a Constant Bit Rate compression algorithm. And in such a case, the Http Content Length is easily calculated in advance as being the input audio file's duration (in seconds) multiplied by the CBR rate (in bytes per second).

So if the flac standard were extended to allow both the existing Variable Bit Rate compression algorithm plus a new Constant Bit Rate option, then it would become much better suited for online streaming.
Title: Streaming FLAC over the Internet
Post by: greynol on 2012-11-22 11:15:49
IMHO flac has one technical weakness that will prevent it taking off as THE standard -- namely it only has Variable Bit Rate compression algorithms. Which means that one cannot predict in advance the length of ouput file that will be generated for any input stream.

Name a single compressed lossless format that isn't VBR in nature.

Don't bother trying; one doesn't exist. Designing one would be futile since some signals must grow in size after being processed by a lossless compression algorithm in order for other signals to be compressible and satisfy the necessary requirement that they remain unique after being decompressed.
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-22 11:19:07
Name a single compressed lossless format that isn't VBR in nature.


There aren't any.
It is true: all current compressed lossless formats suck, as far as Http streaming is concerned.
Title: Streaming FLAC over the Internet
Post by: 2Bdecided on 2012-11-22 11:26:05
By definition, lossless compressed formats can't be CBR.

You could "abuse" the lossyWAV algorithm to make a near-lossless CBR-ish (maybe ABR, or at least capped bitrate) encoder fairly easily.

Cheers,
David.
Title: Streaming FLAC over the Internet
Post by: Nekit1234007 on 2012-11-22 11:45:35
The Http protocol requires sending a Content Length (number of bytes) header

Since when? I can’t speak for streaming media over HTTP, but regular data doesn’t require it. Also some radio stations provide flac streams.
Title: Streaming FLAC over the Internet
Post by: [JAZ] on 2012-11-22 18:24:06
It is true: all current compressed lossless formats suck, as far as Http streaming is concerned.


Definitely, this thread has gone offtopic, but your comments require an answer.

First:
Stream: a continuous (once received) set of data. Generally speaking, a set of data that is being transmitted from a source to a destination.

When the set of data has a determined size, then, it rarely is called stream, but a download or upload.
In other words, streams in internet tend to be of data for which the size is not determined. One such example is, obviously, a Live radio stream.

Second:
The HTTP ContentLength header can have the value of 0, which tells the browser that the size of the stream is not known beforehand.

In our job, we generate PDF files in realtime, and in some cases, start sending them before the file is fully generated (Like for files that can have several megabytes). In such cases, the file size is undetermined and, as such, the ContentLength that we send is zero.
Title: Streaming FLAC over the Internet
Post by: Porcus on 2012-11-22 18:48:46
If this is still relevant after [JAZ]'s clarification:

So if the flac standard were extended to allow both the existing Variable Bit Rate compression algorithm plus a new Constant Bit Rate option, then it would become much better suited for online streaming.


FLAC can deliver CBR. (Signal won't be compressed, though.)
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-22 21:28:54
By definition, lossless compressed formats can't be CBR.


I don't agree. What is it about the "definition" of lossless that says it can't be CBR. Please be specific.
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-22 21:31:42
Since when? I can’t speak for streaming media over HTTP, but regular data doesn’t require it. Also some radio stations provide flac streams.


I am speaking of UPnP / DLNA audio renderers. If you don't provide a Content Length, then the player cannot support Seek.


Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-22 21:35:39

First:
Stream: a continuous (once received) set of data. Generally speaking, a set of data that is being transmitted from a source to a destination.

When the set of data has a determined size, then, it rarely is called stream, but a download or upload.
In other words, streams in internet tend to be of data for which the size is not determined. One such example is, obviously, a Live radio stream.


One other example is UPnP / DLNA "streaming". In general that technology delivers pre-existing tracks. But they call it streaming. Don't argue with me. Argue with them...


Second:
The HTTP ContentLength header can have the value of 0, which tells the browser that the size of the stream is not known beforehand.

In our job, we generate PDF files in realtime, and in some cases, start sending them before the file is fully generated (Like for files that can have several megabytes). In such cases, the file size is undetermined and, as such, the ContentLength that we send is zero.


Yes. And I am sure you do...
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-22 21:37:34
FLAC can deliver CBR. (Signal won't be compressed, though.)


I know. And that is the trick that I use...
Title: Streaming FLAC over the Internet
Post by: Porcus on 2012-11-22 22:49:47
By definition, lossless compressed formats can't be CBR.


I don't agree. What is it about the "definition" of lossless that says it can't be CBR. Please be specific.


It's not the “lossless” part, it's the “compression” part. (Edit: and then, I see, you write that you know.)
Title: Streaming FLAC over the Internet
Post by: Soap on 2012-11-22 23:19:30
By definition, lossless compressed formats can't be CBR.


I don't agree. What is it about the "definition" of lossless that says it can't be CBR. Please be specific.


You were answered earlier on this very point.

Name a single compressed lossless format that isn't VBR in nature.

Don't bother trying; one doesn't exist. Designing one would be futile since some signals must grow in size after being processed by a lossless compression algorithm in order for other signals to be compressible and satisfy the necessary requirement that they remain unique after being decompressed.


You can choose not to "agree" all you want, but information theory (http://en.wikipedia.org/wiki/Information_theory) is solid math.
Title: Streaming FLAC over the Internet
Post by: funkyblue on 2012-11-23 09:45:29
By definition, lossless compressed formats can't be CBR.

You could "abuse" the lossyWAV algorithm to make a near-lossless CBR-ish (maybe ABR, or at least capped bitrate) encoder fairly easily.

Cheers,
David.


What about a ZIP format? IE All FLAC would be is a container for the WAV file. Is this possible without VBR?
Title: Streaming FLAC over the Internet
Post by: Porcus on 2012-11-23 10:00:36
What about a ZIP format? IE All FLAC would be is a container for the WAV file. Is this possible without VBR?


You mean, like when you store in a .zip without compressing? FLAC can do that (http://flac.sourceforge.net/format.html#subframe_verbatim). dBpoweramp also even supports conversion to uncompressed FLAC, likely to satisfy placebophiles who think it makes a difference. (http://forums.linn.co.uk/bb/showthread.php?tid=18382)

WAV is a container, by the way.
Title: Streaming FLAC over the Internet
Post by: funkyblue on 2012-11-23 10:19:07
What about a ZIP format? IE All FLAC would be is a container for the WAV file. Is this possible without VBR?


You mean, like when you store in a .zip without compressing? FLAC can do that (http://flac.sourceforge.net/format.html#subframe_verbatim). dBpoweramp also even supports conversion to uncompressed FLAC, likely to satisfy placebophiles who think it makes a difference. (http://forums.linn.co.uk/bb/showthread.php?tid=18382)

WAV is a container, by the way.


Cool. I always forget WAV is a container. Just no error checking.

Personally VBR is fine. Just make sure HTTP can steam the max value.
Title: Streaming FLAC over the Internet
Post by: [JAZ] on 2012-11-23 20:17:08
I am speaking of UPnP / DLNA audio renderers. If you don't provide a Content Length, then the player cannot support Seek.


We mostly agree, but now think about this. If the DLNA renderer allows the user to seek your realtime-generated stream, what will you do if the user seeks backwards?
Also, depending on the DLNA server/renderer, the content might be fully "downloaded" before it starts to play.
Title: Streaming FLAC over the Internet
Post by: includemeout on 2012-11-23 21:28:07
(...) dBpoweramp also even supports conversion to uncompressed FLAC, likely to satisfy placebophiles who think it makes a difference. (http://forums.linn.co.uk/bb/showthread.php?tid=18382)
(...)


Oh my, following that link has really made my day... and wonder where we would be without good olde TOS8. =)
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-23 22:49:25
You can choose not to "agree" all you want, but information theory (http://en.wikipedia.org/wiki/Information_theory) is solid math.


I don't dispute information theory. Indeed I am prepared to bet that I know more about the math of information theory than you do...

Nevertheless information theory has nothing to say about algorithms that can losslessly compress a data stream at a predetermined number of bytes per second. For example let us try the following algorithm: The flac encoder has nine levels of aggressivity in its compression. So let us start compressing a file at (say) grade 5. After (say) one second let us see if the compressed output is smaller or larger than our target CBR BPS. If smaller then let us back off on the compression aggressivity grade by one grade, and if larger then let us tighten the compression aggressivity by one grade. Repeat every second. So long as you are not over aggressive on choosing your target CBR BPS, then statistically speaking, you will end up with an output file that is dead on the target CBR BPS. Of course there will be Gaussian curve outliers. But, statistically this algorithm will center its statistical Gaussian peak on the selected target CBR BPS. And the longer the file, the closer it will get to the target CBR BPS...

If you want to submit some math to prove me wrong, please feel free to do so.
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-23 23:09:12

... now think about this. If the DLNA renderer allows the user to seek your realtime-generated stream, what will you do if the user seeks backwards?


That is absolutely no problem. The server can always seek backwards to an earlier time offset in its cache of what has already been transcoded. It is actually a bit more difficult to seek forwards: In such a case, the server obviously cannot seek forward into a not yet transcoded cache. Instead the server needs to abandon the current running transcoder process and start another transcoder process starting at a forward time offset into the original source file.

This can only work if the server has given a reasonably accurate Content Length value: If the user seeks to (say) half way through the file, then the renderer makes a Byte-Range seek to Content Length x 0.5. Obviously if Content Length is not existing then this cannot work. And if the provided Content Length is wildly inaccurate then the seek may not end up in the expected place. And if the provided Content Length is way too long, then the seek may end up trying to go beyond the end of the final transcoded actual file...


Also, depending on the DLNA server/renderer, the content might be fully "downloaded" before it starts to play.


If the renderer has fully downloaded the track, then indeed it has full knowledge about the actual track length, so it can process seek requests accurately. However most users are not willing to wait until a track has fully downloaded before starting to play it. This therefore requires a predictive approach towards the seek algorithm. And this is why I think that flac needs to provide a predictive method of estimating a Content Length before the full file has actually been transcoded...
Title: Streaming FLAC over the Internet
Post by: funkyblue on 2012-11-23 23:20:18
What about a ZIP format? IE All FLAC would be is a container for the WAV file. Is this possible without VBR?


You mean, like when you store in a .zip without compressing? FLAC can do that (http://flac.sourceforge.net/format.html#subframe_verbatim). dBpoweramp also even supports conversion to uncompressed FLAC, likely to satisfy placebophiles who think it makes a difference. (http://forums.linn.co.uk/bb/showthread.php?tid=18382)

WAV is a container, by the way.


There are some VERY SPECIAL people in this world

For curiosity sake, what are the commands for uncompressed FLAC files in dBpoweramp?
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-23 23:30:51
For curiosity sake, what are the commands for uncompressed FLAC files in dBpoweramp?


I also would be interested to know if there are any commands for uncompressed flac output in either flac.exe or sox.exe? (As far as I can tell, this is not possible in flac.exe or sox.exe and the only way to do it is to either use libflac.dll or to compile your own wrapper around the flac source code libraries. But I would be delighted if anyone can explain otherwise...)


Title: Streaming FLAC over the Internet
Post by: funkyblue on 2012-11-23 23:38:59
It is an option in dBpoweramp.

http://www.audiostream.com/content/dbpower...-wish-come-true (http://www.audiostream.com/content/dbpoweramps-flac-lossless-uncompressed-wish-come-true)

I find it outputs any WAV file to a bitrate of 1412kbps.

There we go a CBR Lossless format
Title: Streaming FLAC over the Internet
Post by: Wombat on 2012-11-23 23:39:09
I also would be interested to know if there are any commands for uncompressed flac output in either flac.exe or sox.exe? (As far as I can tell, this is not possible in flac.exe or sox.exe and the only way to do it is to either use libflac.dll or to compile your own wrapper around the flac source code libraries. But I would be delighted if anyone can explain otherwise...)

We had this here:
http://www.hydrogenaudio.org/forums/index....964&hl=4608 (http://www.hydrogenaudio.org/forums/index.php?showtopic=91964&hl=4608)
Title: Streaming FLAC over the Internet
Post by: greynol on 2012-11-23 23:58:11
If you want to submit some math to prove me wrong, please feel free to do so.

That's a use of the flying spaghetti monster defense we don't typically see with some argument from authority sprinkled in which we do typically see.

Despite your insistence that you're smarter than the rest of us, I think I'll wait until I see your idea put into action and take note of what those CBR bitrates actually are, thanks.  I won't be holding my breath on the bitrates being interesting, let alone the idea being implemented; in other words, I think we'll chalk this up as a fail on your part after it is all said and done.

In the meantime, does anyone know of a program that displays a histogram of bitrates from flac and hopefully some of the other more popular lossless formats?

There we go a CBR Lossless format

Sure, but we already covered that.
Title: Streaming FLAC over the Internet
Post by: funkyblue on 2012-11-24 00:10:53
I realise that but the link posted a bunch of cryptic commands instead of a simple guide or the fact that dBpoweramp has a uncompress mode build in
Title: Streaming FLAC over the Internet
Post by: 73ChargerFan on 2012-11-24 01:02:38
Seeking - store a seek table in the header, say for every half second. When a user seeks backwards, the player seeks to the half second mark prior to that position, and then processes forward. With caching this could work well, and the size of the table would be small in relation to the file size. And if it is a piece of meta data, then tracks with the table wouldn't break older FLAC players.
Title: Streaming FLAC over the Internet
Post by: Porcus on 2012-11-24 01:20:45
Nevertheless information theory has nothing to say about algorithms that can losslessly compress a data stream at a predetermined number of bytes per second.


I wonder what you mean by «a predetermined number» ...
Title: Streaming FLAC over the Internet
Post by: Soap on 2012-11-24 02:44:19
.... But, statistically this algorithm will center its statistical Gaussian peak on the selected target CBR BPS. And the longer the file, the closer it will get to the target CBR BPS...

If you want to submit some math to prove me wrong, please feel free to do so.


Congrats, you just described an ABR procedure.  Nothing about that will provide a Constant Bit Rate unless you pad.

You may know math, but you demonstrably don't know the terminology.
Title: Streaming FLAC over the Internet
Post by: greynol on 2012-11-24 02:51:40
You pad and your compression goes out the window. Provide the math to prove me wrong.
Title: Streaming FLAC over the Internet
Post by: saratoga on 2012-11-24 02:55:49
I guess in principle you'd have to pad for at least as high a bitrate as your uncompressed PCM, since you can't make a format that compresses every possible stream smaller than the source.  Of course if you're ok with hard to encode samples skipping, you could still do fairly well.
Title: Streaming FLAC over the Internet
Post by: greynol on 2012-11-24 03:00:31
If you could manage to ensure that the only data that would grow is undesireale data, then sure. Good luck with that!
Title: Streaming FLAC over the Internet
Post by: Porcus on 2012-11-24 10:36:34
Well if you are encoding the signal in advance, you could – in principle – get a CBR at a bitrate equal to the peak. That would still save bandwidth, provided that the signal doesn't max out the bitrate anywhere.

If you are recording and streaming «live» (bar the encoding delay), that's a different story.
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-24 13:04:25
Despite your insistence that you're smarter than the rest of us,


Hey look, I am certainly NOT insisting that I am smarter than the rest of you.

I was merely reacting (badly) to the post of "soap" in which he wacked a wiki url ("spaghetti monster") at me and thereby claimed that he is smarter than me.

I think I'll wait until I see your idea put into action and take note of what those CBR bitrates actually are, thanks.  I won't be holding my breath on the bitrates being interesting, let alone the idea being implemented; in other words, I think we'll chalk this up as a fail on your part after it is all said and done.


I don't think you should chalk any such "fail" to me. The OP had asked about the future of flac. And I indicated that there is a specific weakness of flac in relation to its use for online streaming that could be detrimental to its future penetration in media servers and renderers. And I have proposed a possible solution. I don't think it is a "fail" on my behalf if that proposed solution would not be implemented. But it would sadly be a "fail" on behalf of the flac community, if the format were to die out because the community failed to look into this weakness.

It would be nice if others on this thread would be a little more humble in listening to criticisms, rather than (try to) shoot the messenger...

Title: Streaming FLAC over the Internet
Post by: Soap on 2012-11-24 13:21:26
I was merely reacting (badly) to the post of "soap" in which he wacked a wiki url ("spaghetti monster") at me and thereby claimed that he is smarter than me.


Well, "andrewfg", I'd love a cite of where I claimed I was smarter than you, but that is beside the point:

And I indicated that there is a specific weakness of flac in relation to its use for online streaming that could be detrimental to its future penetration in media servers and renderers.


You mentioned a specific weakness of VBR encodings where the stream length is undetermined, not a weakness in a particular format but (as was pointed out) a weakness in any VBR compressed format.  You went on to claim (and this is the meat of the issue) that one could make a lossless compressed format which was CBR.  This (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=98015&view=findpost&p=814974)is the claim which was responded to and quoted by me, and the claim which is indefensible.  Curiously it is also the discussion you are currently choosing to ignore.

Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-24 13:52:32
Congrats, you just described an ABR procedure.  Nothing about that will provide a Constant Bit Rate unless you pad.


I stand corrected. ABR would be just fine.
Title: Streaming FLAC over the Internet
Post by: greynol on 2012-11-24 14:24:19
Well if you are encoding the signal in advance, you could – in principle – get a CBR at a bitrate equal to the peak. That would still save bandwidth, provided that the signal doesn't max out the bitrate anywhere.

This was why I was inquiring about bitrate histograms for lossless encodings.

You might be able to shave off a little with some source material, but what do you do when you find frames with bitrates larger than their uncompressed PCM counterparts?
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-24 17:22:36
You mentioned a specific weakness of VBR encodings where the stream length is undetermined, not a weakness in a particular format but (as was pointed out) a weakness in any VBR compressed format.


No that is absolutely is NOT what I said. Please read my OP again. I actually said that a weakness of flac is that it does not offer a CBR option.

You went on to claim (and this is the meat of the issue) that one could make a lossless compressed format which was CBR. This is the claim which was responded to and quoted by me, and the claim which is indefensible.


No. This claim is defensible.

Let us start with some basic information theory. If you have a data stream containing maximal information such that its content appears to be random then indeed by definition such content cannot be losslessly compressed. However if you have a data stream containing to some extent redundant and repeating patterns, then that data can be losslessly compressed; essentially this is done by using short codes and lookup tables to substitute for the original longer repeating patterns. Music contains a lot of redundancy and repeating patterns, and it therefore lends itself well to being losslessly compressed. And flac is a good algorithm for maximally compressing music.

When a lossless compression algorithm processes a data stream, it reads the data block by block, and it creates and writes out the resulting short codes and lookup tables for each block successively. Depending on the redundancy and repetition in the content of each block, the resulting output block will tend to be shorter in size than the input block. But there are two extremes: a) if a particular block contains no redundancy and repetition, (i.e. it is indiscernible from random data), then the output block will be at maximum the same size as the input block plus some overhead due to the compression algorithm's framing mechanics, and b) if a particular block contains 100% redundancy and repetition then the output block will be at minimum zero plus some overhead due to the compression algorithm's framing mechanics. And typically the output block size will be somewhere in between these two maximum and minimum limits i.e. somewhere in the range between the overhead due to the compression algorithm's framing mechanics, and the size of the original input data plus the overhead due to the compression algorithm's framing mechanics. Or in other words, given an input block's size "I", and the overhead due to the compression algorithm's framing mechanics "M" then the output block's size "O" will be as follows:

M <= O <= (I + M)

And the compression factor "V" for each block is defined as:

V := O / I  resulting in  V(minimum) := (I + M) / I  and  V(maximum) := M / I

So indeed V is variable (VBR) within the range V(minimum) to V(maximum), depending on the redundancy and repetition in the content of each block. As the algorithm processes more and more blocks the individual V values for each block will average out to a value such that:

V(minimum) < V(average) < V(maximum)

The actual value of V(average) will be highly specific to the content of the specific data stream. And indeed its value is not predictable in advance.

The flac algorithm has 9 different aggressivity grades of compression, and on the same data stream each grade will result in a different V(average) compression rate. Let us call these V(average_0) through V(average_8), and so:

V(minimum) < V(average_0) < V(average_1) < ... < V(average_7) < V(average_8) < V(maximum)

Now let us say we want to target on a fixed bit rate over the whole data file. Or as another poster mentioned a fixed average bit rate. This is equivalent to targeting a fixed (average) compression factor. Let us call this A(target), and let us choose it so that:

V(average_0) < A(target) < V(average_8)

Then the compression algorithm, as it processes over each individual block in the input data, would start processing at grade 8 compression, and it will probably determine that it is delivering a higher net average compression than A(target). So for the next few blocks it could back off to say grade 0 compression, and those blocks would deliver a lower net average compression than A(target). And then again for the next few blocks it could select grade 8 compression again. And so on. The compression algorithm would juggle the compression grades so that the net-net average compression over all processed blocks would get close to A(target). And indeed the longer the file, the closer the algorithm will get to A(target).

Now obviously, as we have already seen, the V(average) of any track depends on the degree of redundancy and repetition in the content. So it is indeed possible that if one has bad luck in the choice of source material, then the chosen A(target) might not lie within the bandwidth V(average_0) < A(target) < V(average_8) for that particular source material. This means that the actual delivered level of compression could end up less than the selected A(target). The trick would obviously be to choose an A(target) that is closer to V(minimum) than it is to V(maximum). So in other words there will nevertheless always be some outlier cases where A(target) is not achieved. But if you are not too aggressive and don't have too much bad luck on source material choice, it would work fine.

Title: Streaming FLAC over the Internet
Post by: greynol on 2012-11-24 17:32:32
if a particular block contains no redundancy and repetition, (i.e. it is indiscernible from random data), then the output block will be at maximum the same size as the input block plus some overhead due to the compression algorithm's framing mechanics

Here is where we run into problems with information theory.  My suspicion is that problems arise even when dealing with signals that still contain what you casually refer to as redundancy, especially since you have to figure out how to implement something with real-world constraints in terms of things like making a signal easy enough to decompress (and compress for live streaming) and other considerations which need to be taken into account in order to satisfy playback requirements.

Again, a simple histogram will show the feasibilty of creating a codec similar in function as flac that employs a CBR mode.
Title: Streaming FLAC over the Internet
Post by: tuffy on 2012-11-24 17:48:14
Then the compression algorithm, as it processes over each individual block in the input data, would start processing at grade 8 compression, and it will probably determine that it is delivering a higher net average compression than A(target). So for the next few blocks it could back off to say grade 0 compression, and those blocks would deliver a lower net average compression than A(target). And then again for the next few blocks it could select grade 8 compression again. And so on. The compression algorithm would juggle the compression grades so that the net-net average compression over all processed blocks would get close to A(target). And indeed the longer the file, the closer the algorithm will get to A(target).

The types of subframes FLAC will search for at compression level 8 are a superset of those it will search for at compression level 0 (or any other lower compression level).  That's why it takes longer to encode; FLAC is sorting through more combinations trying to find the smallest frame.  For example, level 0 won't attempt to use LPC subframes.  But if you're encoding at level 8 and the encoder sees that not using LPC subframes generates the smallest frame, it won't use them.
Title: Streaming FLAC over the Internet
Post by: greynol on 2012-11-24 17:48:16
@andrewfg:
Read my edit.
Title: Streaming FLAC over the Internet
Post by: Soap on 2012-11-24 18:20:39
No. This claim is defensible.


That is still ABR if you don't pad, and it still fails to address your original concern:

Which means that one cannot predict in advance the length of ouput file that will be generated for any input stream.


because one can not predict the maximum needed bitrate until the entire track is encoded.
Title: Streaming FLAC over the Internet
Post by: andrewfg on 2012-11-25 00:10:45
That is still ABR if you don't pad, and it still fails to address your original concern:


Actually ABR would go a long way in addressing my concern. With the method I described, if you calculated Content Length based on the target ABR then you could be sure that the actual delivered Content Length would always be greater than or equal to the calculated Content Length.

Whilst it does not guarantee totally accurate Byte-Range Seeks, it does at least eliminate the potential for Seek beyond the end of the file.

because one can not predict the maximum needed bitrate until the entire track is encoded.


Perhaps instead of calling it CBR or ABR, we should call MBR (minimum bit rate). As mentioned already this would be a significant improvement.
Title: Streaming FLAC over the Internet
Post by: Porcus on 2012-11-25 00:15:16
Well if you are encoding the signal in advance, you could – in principle – get a CBR at a bitrate equal to the peak. That would still save bandwidth, provided that the signal doesn't max out the bitrate anywhere.

This was why I was inquiring about bitrate histograms for lossless encodings.

You might be able to shave off a little with some source material, but what do you do when you find frames with bitrates larger than their uncompressed PCM counterparts?


Leave that particular file uncompressed? On average, you are likely to be able to produce a proof-of-concept which will be appreciated by the most curious (and maybe, but don't count on it, someone will actually implement while bandwidth is still an issue).
Title: Streaming FLAC over the Internet
Post by: Soap on 2012-11-25 00:29:58
Actually ABR would go a long way in addressing my concern. With the method I described, if you calculated Content Length based on the target ABR then you could be sure that the actual delivered Content Length would always be greater than or equal to the calculated Content Length.


In that situation what you are calling "target ABR" must always be uncompressed bitrate + format overhead.  This is the entire point of discussing information theory.  You do not know the compressibility of the entire track until you reach the end.  And if you're willing to reach the end of the track you can deliver an accurate Content Length on VBR content.  And if you're willing to send a Content Length greater than the actual content length you, again, can live very happily with VBR.
Title: Streaming FLAC over the Internet
Post by: greynol on 2012-11-25 04:39:41
Axon posted a script to generate histograms and included some samples which were not at all surprising to me.  AFAIC, it confirms what has already been suggested: peak bitrates are likely going to dash hopes of worthwhile CBR/ABR implementations of lossless compression.
Title: Streaming FLAC over the Internet
Post by: Axon on 2012-11-25 05:34:05
AFAIC, it confirms what has already been suggested: peak bitrates are likely going to dash hopes of worthwhile CBR/ABR implementations of lossless compression.

Hold that thought for a sec.

The original context of the OP's (poorly communicated) post was specifically about UPnP/DLNA seek compatibility, which demands CBR or well-behaved-ABR behavior of the raw bitstream. Now suppose the following of the streaming server:


TL;DR: CBR, seekable FLAC stream that retains VBR, sub-redbook bandwidth usage.

There, I fixed it for you
Title: Streaming FLAC over the Internet
Post by: 2Bdecided on 2012-11-26 13:09:01
I knew I should have brought popcorn.