Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: higher compression ratios in FLAC (Read 20776 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

higher compression ratios in FLAC

Hello,

I have used FLAC casually for many years, to archive and to play a modest collection of albums. 

Recently, I have had some problems and experiences that I am hoping some enthusiasts could address.

First, and simplest, I wanted to determine the compression ratio of particular files, that is, the the size used for the compressed audio within a file divided by the size that would be required if uncompressed.  I use the standard command-line tools, running on a Linux desktop, and I found no way to report this simple information.  I am forced to decompress the file to determine the uncompressed size.

Second, and more mysterious, I wanted to utilize the encoder's capabilities for greater compression to reduce the size consumed by the collection, even if only by a few percent of the total size.  I used the command line tool to transcode the source FLAC files to new targets, while including the -8 switch.  Astonishingly, around half of the result files were larger than the corresponding source.  This effect was reported as a warning by the tool, and could be confirmed by examining the results.  In fact, over all albums, the average size of the result file is only %0.6 smaller than the source.  The files were originally encoded with a tool that I assume invoked the standard FLAC library, as I am not aware of any alternative.  Certainly no commercial FLAC encoder, if any such thing exists, was used.  I believe that the files were originally encoded with default parameters, which suggests that encoding them so as to target highest possible compression would make, contrary to the actual effect I observed, every one smaller, and by a wider margin.  What of course is most strange is that even if the original encoding was aggressive, the result of a transcoding operation such as the one I performed should not make them larger, only, in the worst case, fail to make them smaller.   But perhaps I do not understand the issues correctly. Have I missed any relevant details?  Can anyone account for these peculiar observations?

Thanks.

Re: higher compression ratios in FLAC

Reply #1
[...] I wanted to utilize the encoder's capabilities for greater compression to reduce the size consumed by the collection, even if only by a few percent of the total size. [...]
-8 -e -p
but! no "few". only a hundredth of a percent. and! time/power costs are MUCH greater than the benefits of compression.
if you are talking about saving "few" percent - it's time to buy a new hdd.

Re: higher compression ratios in FLAC

Reply #2
I originally included the -e. switch  I am seeing very similar effects even including -p.   The extra option appears to have little practical benefit.

My collection is currently compressed at very close to 50% ratio.  My understanding is that FLAC compression can achieve much higher ratios with more aggressive encoding settings, in some cases less that 40%.  I am surprised that I have not been able to achieve this effect easily.  Meanwhile, I am even more perplexed that my current files are somehow already encoded at better rates than what I can make the encoder do again.

I also would mention that as I watch the encoder run, it is often showing rates 10%-20% higher at the initial part of the an album, though those rates drop substantially around the point that encoding is 20% or 30% complete for that album.  This observation is yet another baffling effect I am seeing.  Does it provide any hints about what might be happening?

Hopefully someone can shine some light on these problems.  Thanks.

Re: higher compression ratios in FLAC

Reply #3
You will not see much difference in compression ratio with slower settings with the FLAC encoder. To reduce file sizes you might need to change the format or downsample if you have high resolution recordings. TAK is a good format, but AFAIK not supported on Linux. Better compression ratios are possible with quiet sound, less high frequency content (not cymbals, or trumpets), and low stereo separation. A quiet intro would compress better, which might explain the better ratio you observed at the start of your example album.

Possible reasons why existing files had slightly better compression:
Encoder tried a list of -A windows.
Used smaller block size.
Used zero padding.
An alternate encoder such as ffmpeg or cuetools was used.

Re: higher compression ratios in FLAC

Reply #4
compress ratio depends on source. in my experience: flac -8
Sabaton - Metalizer - 76.57%
Blackmore's Night - Ghost Of A Rose - 63.51%
Mylene Farmer - L'autre - 59.28%
classical music, maybe, less 50%

" I am surprised that I have not been able to achieve this effect easily."
difference between (flac -0) and (flac -8) approximately 6%...

TAK is better 2-3% than flac...

Re: higher compression ratios in FLAC

Reply #5
A quiet intro would compress better, which might explain the better ratio you observed at the start of your example album.

The effect is consistent though not universal, across a larger set of albums.  It is not limited to one incidental occurrence.


Possible reasons why existing files had slightly better compression:
An alternate encoder such as ffmpeg or cuetools was used.

So other implementations are available that might be more efficient than the reference?  Are you sure that ffmpeg or cuetools doesn't simply embed or link with libflac?  Does any serious possibility exist that I can get substantially superior results with the combination of a different encoder and more aggressive parameters?

Re: higher compression ratios in FLAC

Reply #6
I am aware of Flake, FLACCL (GPU accelerated) and ffmpeg encoders.

Substantially superior results will only be possible in special cases with unusual source material.

As was discussed in the other thread, ffmpeg achieved better results on an upsampled recording lacking high frequencies entirely (high relative to the sampling rate, example: Vangelis - "Nocturne"). TAK will better FLAC on near mono recordings where the difference between channels is small but non-zero (old music, some hip-hop), on surround sound, and WavPack will be superior on rare 18-bit material padded to 24-bit where the LSBs switch together (example: Eagles - "The Studio Albums 1972-1979").

I agree with m14u, that it is not worth the effort to recompress existing FLAC files, except in special circumstances where you know the original used low settings, or has excessive metadata embedded. There is also a chance of a user mistake, error or bug and the source is accidentally deleted during a mass conversion.

 

Re: higher compression ratios in FLAC

Reply #7
FWIW, here's the command line I use:

flac -8 -l 12 -b 4096 -m -e -r 6

That's the way it used to be before FLAC 8 was dumbed down awhile back.

Re: higher compression ratios in FLAC

Reply #8
First, and simplest, I wanted to determine the compression ratio of particular files, that is, the the size used for the compressed audio within a file divided by the size that would be required if uncompressed.  I use the standard command-line tools, running on a Linux desktop, and I found no way to report this simple information.  I am forced to decompress the file to determine the uncompressed size.
I guess that one of your tools would give you the song length, channels, sampling rate and bit depth (at least ffmpeg does). You can take that to calculate the original bitrate and file size (excluding metadata and container). For example for audio CD and 100s song:

            100               *           44100              *                  2                *        16       =  141120000 bit  = 141120000 bit / 8 = 17640000 byte
(song length in sec) * (sampling rate in Hz) * (number of channels) * (bit depth) =     (size in bit)    =   (size in bit) / 8     =  (size in byte)

Re: higher compression ratios in FLAC

Reply #9
My collection is currently compressed at very close to 50% ratio.  My understanding is that FLAC compression can achieve much higher ratios with more aggressive encoding settings, in some cases less that 40%.

No. 50 percent with -8, then forget all about 40 - regardless of codec. For 40, you need different music.
Have a look at the charts here: http://www.audiograaf.nl/downloads.html

The non-classical part of my CDDA collection compresses around 65 percent. The classical music part at 45. "Material is everything" is a slight exaggeration, but you get the idea.

Re: higher compression ratios in FLAC

Reply #10
FWIW, here's the command line I use:
flac -8 -l 12 -b 4096 -m -e -r 6
flac -h
-8, --compression-level-8, --best  Synonymous with -l 12 -b 4096 -m -r 6 -A tukey(0.5) -A partial_tukey(2) -A punchout_tukey(3)

Re: higher compression ratios in FLAC

Reply #11
I originally asked how to find the compression ratio using the basic flac tools because that is all I was aware might be effective for finding that information, even though I failed to determine how.  I have no special affinity for those tools, however, so if ffmpeg has this functionality, then it helps greatly to say how to use it for that purpose in addition to saying simply that it can be used.


Re: higher compression ratios in FLAC

Reply #12
I originally asked how to find the compression ratio using the basic flac tools because that is all I was aware might be effective for finding that information, even though I failed to determine how.  I have no special affinity for those tools, however, so if ffmpeg has this functionality
I'd use foobar200 for this purpose. On Linux it is possible to use it with Wine. Formula for  calculation of compression ratio (in %) will be $muldiv(%__bitrate%,100,$div($mul(%__samplerate%,%__bitspersample%,%__channels%),1000))'%' You can display this info as custom column in playlist view.

Re: higher compression ratios in FLAC

Reply #13
No way to do it without a GUI and without WINE?  I thought it should be a basic feature of a tool set.  Always been able to do it for all kinds of other compression and archive containers.

Re: higher compression ratios in FLAC

Reply #14
for 44.1/16/2ch
(flac`s bitrate) / 1411 * 100 = ratio?
ffmpeg flac less effective than "original" flac

Re: higher compression ratios in FLAC

Reply #15
ffmpeg flac less effective than "original" flac
not exactly :)
ffmpeg 4.1.3 flac compression 12 / flac 1.3.2 -8
Sabaton - Metalizer - 76.76% / 76.57% (-0.19)
Blackmore's Night - Ghost Of A Rose - 63.65% / 63.51% (-0.14)
Mylene Farmer - L'autre - 58.87% / 59.28% (+0.41)
it makes sense to check ffmpeg on your collection

Re: higher compression ratios in FLAC

Reply #16
ffmpeg 4.1.3 flac compression 12

Do not compare that to flac -8. If nothing has changed, ffmpeg creates non-subset files at levels 11 and 12, so normal users would likely stay safe and don't go beyond 10. https://hydrogenaud.io/index.php/topic,108100.msg887080.html#msg887080

If you really feel like going beyond subset, then there's the lax option.

Edit: And if you really, really want another percent, then consider whether FLAC is that sacred. And some people do embed ridiculous-size album art in every track in an album ...

Re: higher compression ratios in FLAC

Reply #17
My collection is currently compressed at very close to 50% ratio.  My understanding is that FLAC compression can achieve much higher ratios with more aggressive encoding settings, in some cases less that 40%.

No. 50 percent with -8, then forget all about 40 - regardless of codec. For 40, you need different music.
Have a look at the charts here: http://www.audiograaf.nl/downloads.html

The understanding I had that FLAC can get ratios much better than 50% was not intended to apply to music that specifically had already been found to compress at 50% with -8 encoding settings.  My music was encoded long ago with unknown settings, likely the default of whatever software package  was used, so it is my guess that they were not particularly aggressive.  The claim of much better ratios was not specific to any particular input.

I understand that if music encodes with -8 at some ratio, then that ratio would be very close to the best possible encoding.  Such is how the -8 option is documented. 

My situation now is one of is trying to determine what amount of saving I can get by re-encoding, and how best to do it.  I was hopeful for much more than a few percent, but I could still benefit from so little, since one of the potential targets is my mobile device.  A few percent difference could completely determine whether another album fits on the device.  For the hard disk only, I agree with the earlier post that a few percent is not a helpful savings.

In my encoding job, the  -8 option produced negligible savings, and overall worse results compared to the source for many albums.  This effect led me to consider that maybe my test was somehow invalid, and that by considering or resolving some other issue, I might somehow realize my hope of non-negligible, even significant, savings.

Re: higher compression ratios in FLAC

Reply #18
If you still want significant further size reduction in flac, you might look into lossyWAV.

Re: higher compression ratios in FLAC

Reply #19
You might be able to detect weak compression preset by scanning a set of files for the block size with metaflac.

$ metaflac --show-max-blocksize test.flac
1152

Or look for uncompressed FLAC (it exists) by the bitrate vs uncompressed bitrate as recommended above using Foobar.
Or look for encoder version <= 1.1.2 with 16-bit or <= 1.2.0 with 24-bit.

As for --lax, I don't think it is useful. It allows to specify a blocksize over 4608, but that leads to worse compression with the reference encoder. Usually 2048 to 4096 (default best) is optimal. ffmpeg can use 8192 blocks though.

I still think it is not worth the effort to scan the entire collection for a small percentage of poorly compressed files. You will not gain much over the default -5 preset and a recent encoder.

Re: higher compression ratios in FLAC

Reply #20
You might be able to detect weak compression preset by scanning a set of files for the block size with metaflac.[...]
"this is wonderful idea, but it did`t the work"(c)
flac -h
0-2 - b 1152
3-8 - b 4096
except detection 0-2 level...

Re: higher compression ratios in FLAC

Reply #21
It would detect the worst settings, where it might be justified to take any action. I suggested based on the fact that I found such files in my collection. Regarding the encoder versions, compression was significantly improved either in 1.1.3 or 1.1.4, and with RICE2 in 1.2.1 for 24-bit. It might be possible to look for uncompressed "verbatim" frames, but I don't know how to automate it.

Re: higher compression ratios in FLAC

Reply #22
I originally asked how to find the compression ratio using the basic flac tools because that is all I was aware might be effective for finding that information, even though I failed to determine how.  I have no special affinity for those tools, however, so if ffmpeg has this functionality, then it helps greatly to say how to use it for that purpose in addition to saying simply that it can be used.
If you have ffmpeg installed you will probably have ffprobe.
Then with
Code: [Select]
ffprobe YourInputFileHere -hide_banner
you'll get something like
Code: [Select]
Input #0, flac, from 'q:\temp\temp.flac':
  Duration: 00:04:16.22, start: 0.000000, bitrate: 939 kb/s
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s16
You can parse that anyway you want or if it's easier:
Code: [Select]
>ffprobe "q:\temp\temp.flac" -hide_banner -show_streams
you'll get
Code: [Select]
Input #0, flac, from 'q:\temp\temp.flac':
  Duration: 00:04:16.22, start: 0.000000, bitrate: 939 kb/s
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s16
[STREAM]
index=0
codec_name=flac
codec_long_name=FLAC (Free Lossless Audio Codec)
profile=unknown
codec_type=audio
codec_time_base=1/44100
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
sample_fmt=s16
sample_rate=44100
channels=2
channel_layout=stereo
bits_per_sample=0
id=N/A
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/44100
start_pts=0
start_time=0.000000
duration_ts=11299439
duration=256.223107
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=16
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]
You can even change the output format with e.g. -print_format json (available formats: default, compact, csv, flat, ini, json, xml).

If you know exactly what values you want you could use something like:
Code: [Select]
ffprobe -v error "q:\temp\temp.flac" -show_entries stream=duration:stream=channels:stream=sample_rate:stream=sample_fmt -of default=noprint_wrappers=1:nokey=1
that will give you just
Code: [Select]
s16
44100
2
256.223107



Re: higher compression ratios in FLAC

Reply #24
I only have MediaInfo GUI installed so I simply used an installed ffmpeg/ffprobe to generate an example that the thread starter asked for.