Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: LossyFLAC yielding higher bitrate than FLAC alone (Read 10726 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

LossyFLAC yielding higher bitrate than FLAC alone

Hi

I'm not aware of this issue being known so here it goes...
I noticed that LossyWAV + FLAC (with flac block size of 512) may produce larger files than FLAC alone for some simple music material that already encodes at low bitrates with FLAC.

tools used: lossyWAV 1.3.0, FLAC 1.2.1
example bitrates for a track which consists of solo piano (44.1 kHz, 16 bit, stereo):

Code: [Select]
flac -8     375
-q E, -8    393
-q S, -8    390

flac -5     379
-q E, -5    395
-q S, -5    393


Any thoughts?

LossyFLAC yielding higher bitrate than FLAC alone

Reply #1
This is known in the development threads, particularly with clean recordings of simple classical and solo tonal instruments at modest volume levels. It is a result of the shorter block length (512 samples versus 4096 for FLAC which reduces the lossless encoder's efficiency a little) and the relative lack of bits removed by the lossy process compared to loud, spectrally-dense materials (partly because of the volume level, partly because there may be areas of the spectrum with no content, causing lossyWAV to assume there's a very low noise floor and no scope to remove bits.

It's possible that the more aggressive settings such as -C (eConomic) -P (portable) and -X (eXtraportable) would still offer a reduction, especially on the most recent versions that employ adaptive noise shaping to increase the bit-depth reduction and reduce bitrate (-X reaching around 310 kbps typically).

It's not something we worry about unduly as the files are small in any case and the loss of compression on these is far, far smaller than the compression advantage gained on more typical music. This effect doesn't seem to be observed unless the lossless FLAC bitrate is less than about 500 kbps or so.
Dynamic – the artist formerly known as DickD

LossyFLAC yielding higher bitrate than FLAC alone

Reply #2
So what I'm getting is that it's not really LossyWAV's "fault", but FLAC's, with its small block size.
I'm more interested in the higher quality presets, for archiving music, and this thing seems to happen on many of the quieter tracks (for example, using 'Extreme', a Chopin nocturne can grow by 7% while probably not being bit-for-bit lossless). Guess I'll have to be careful.

Thanks for replying.


 

LossyFLAC yielding higher bitrate than FLAC alone

Reply #4
The small FLAC blocksize is a requirement which originates from lossyWAV in order to make lossyWAV useful.
For quiet classical music and very high lossyWAV quality settings the situation described may occur not very seldom.
But you can always compare the file size lossyWAV | FLAC and FLAC, and use the more appropriate files, or decide a priori according to the musical contents as carpman said.
lame3995o -Q1.7 --lowpass 17

LossyFLAC yielding higher bitrate than FLAC alone

Reply #5
Thanks Halb and Carpman.

LossyFLAC yielding higher bitrate than FLAC alone

Reply #6
The small FLAC blocksize is a requirement which originates from lossyWAV in order to make lossyWAV useful.
It's needed to make lossyWAV more efficient, but it still works with a 4096-sample blocklength - and in these rare cases, works better. Whether it works well enough to bother with lossyWAV at all depends on the source material. A solo piano recording with significant background noise (e.g. an old analogue recording) might benefit considerably. A pristine digital recording with low noise would benefit less. It is theoretically possible, even with the same blocksize, for lossyWAV to increase the bitrate - but I've yet to see it.

Even without lossyWAV, FLAC itself could benefit a little from adjusting its blocksize for optimal efficiency depending on the input signal, but no FLAC encoder bothers. FLAC decoders are supposed to handle it.

Cheers,
David.


LossyFLAC yielding higher bitrate than FLAC alone

Reply #7
As a spectator, early on in LossyWAV's development I wondered about the 512 blocksize. Does the noise floor really change that much from one 10ms period to the next? Trawling through the archives I see 2BDecided started off with 4096 sample blocks because it was flac's default, then switched to 1024 sample blocks when that looked substantially better in his testing, but I wonder how extensive the testing was that fixed on 512.*

Might the optimal lossywav blocksize depend on musical content/genre and on lossywav aggressiveness? I think so. With the current lossywav, some classical music I had on hand compressed better with flac -b 512 when using lossywav 'high' quality but did better with -b 1024 when using lossywav 'extreme'. (In both cases the difference was <1% though.)

4096 samples is understandably too big- and regular FLAC doesn't benefit much from it either. Most of the benefit flac can get from >512 sample blocks is already achieved at 1024 (similarly, 512-sample blocks already eliminate most of the overhead from 256-sample blocks).

*Here's the only stuff I found about that in a quick search: this post refers to a post that's not in its thread anymore- maybe this one?

LossyFLAC yielding higher bitrate than FLAC alone

Reply #8
The small FLAC blocksize is a requirement which originates from lossyWAV in order to make lossyWAV useful.
It's needed to make lossyWAV more efficient, but it still works with a 4096-sample blocklength - and in these rare cases, works better.

Thinking about it, it makes sense since lossyWAV does little to no pre-processing. So encoding a few "problem" files using flac's default blocksize led to the same bitrate as the flac original (no big surprise)...

LossyFLAC yielding higher bitrate than FLAC alone

Reply #9
On more thing - I seem to recall that TAK introduced features for improving lossyTAK compression by noticing when the bit-depth stayed the same for consecutive blocks of 512 and increasing the block size for those blocks adaptively, so TAK might not show the same 7% increase. TAK tends to compress a little bit more than FLAC anyway.

(I may be misremembering - it was only from a glance through TAK changelogs or HA forum threads - I never tried it)

The problem lies then with compatibility - TAK is not widely supported, while FLAC is.

I guess, if anyone felt it worthwhile, FLAC encoders could implement an adaptive block-size mode that looks at bit-depth in 512-sample chunks and reduces the block size to suit the consecutive blocks of equal bit-depth, while reverting to 4096 (or the set blocksize) in other cases. This would retain 4096-sample blocks when possible and handle lossyWAV efficiently with relatively little processor overhead. The only potential problem is if we discovered any decoders that fail to handle changing block size properly.
Dynamic – the artist formerly known as DickD

LossyFLAC yielding higher bitrate than FLAC alone

Reply #10
There's also --merge-blocks option in Wavpack.

LossyFLAC yielding higher bitrate than FLAC alone

Reply #11
On more thing - I seem to recall that TAK introduced features for improving lossyTAK compression by noticing when the bit-depth stayed the same for consecutive blocks of 512 and increasing the block size for those blocks adaptively, so TAK might not show the same 7% increase. TAK tends to compress a little bit more than FLAC anyway.

(I may be misremembering - it was only from a glance through TAK changelogs or HA forum threads - I never tried it)

Early beta releases of TAK 2.1.0 contained a dedicated LossyWav-Codec (basically a variation of the main codec). Advantages:
  • About 2 percent better compression for my test corpus.
  • Usually not more than 0.1 percent compression penality for the afore-noted problem files.

This codec had been removed because there seemed to be little interest.

Maybe i will reintegrate it sometime.

LossyFLAC yielding higher bitrate than FLAC alone

Reply #12
Thanks, TBeck. I guess the demand is going to be limited what with lossyWAV's modest use and TAK's limited device support compared to FLAC (and Wavpack which is in Rockbox) so lumbering people with extra features for a small userbase isn't terribly worthwhile unless lossyTAK becomes more widely used.

Thanks, lvqcl. Using Wavpack with --merge-blocks option may be a better option than FLAC for a user with a Rockbox DAP and this type of content. I noticed the switch was absent from one example in the lossyWAV Wiki (now edited) but was present in the fb2k and EAC lossyWV examples, so I'm glad you mentioned it here.
Dynamic – the artist formerly known as DickD