Skip to main content
Topic: higher compression ratios in FLAC (Read 2731 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: higher compression ratios in FLAC

Reply #25
My music was encoded long ago with unknown settings, likely the default of whatever software package  was used, so it is my guess that they were not particularly aggressive.
OK. Settings may have been -8 originally though.

It is most often possible to find out what FLAC version is used to encode, but not the setting. If you are on Windows, I suggest foobar2000 - not only is it a good player, but it is also one where you get a lot of support here. It can display tool used (see attachment), and it can convert everything too.


I understand that if music encodes with -8 at some ratio, then that ratio would be very close to the best possible encoding.  Such is how the -8 option is documented. 
Yeah, pretty much.
There are limits to what whatever codec can achieve. FLAC did not target the absolute maximum compression level, but rather decoding speed (= battery life!). Imprecisely speaking, a heavier option will search further for patterns to compress, and that takes time; there is a heavier setting than "-8" that can do a "more brute force" search for a fewer-bytes representation, but that is slow. For practical purposes, a you would rather have it perform a "clever search" over functions that are tested to perform well.

-8 was revised some time ago, because there were found functions that seem to do a better job on the average. That does not rule out that new -8 does slightly worse than old -8.

Furthermore, if you compare file size after re-encodings, there is something called "padding". By default, the reference encoding pads up the metadata block with an empty 8 kB to make room for re-tagging; when that is spent, it will have to re-write the entire file, and that takes much more time than just overwriting a small part of a file. You can switch off in the encoding process, but I wouldn't.


My situation now is one of is trying to determine what amount of saving I can get by re-encoding, and how best to do it.

Here is what I would do, assuming that you have hard drive space to spare:
- Use foobar2000 or the audiotester.exe utility to check integrity of your lossless files. (If they are corrupt, you do not want to re-encode; re-encoding will leave the false impression that everything was right.)
- Use foobar2000 to convert with -8 with a naming structure that completely mirrors the one you had. (Any corrupt ones: copy rather than re-encode.)
- Use foobar2000 with foo_bitcompare to compare - bit by bit - old against new. Then you have safeguarded against any wrong overwrites.
Now you have a backup. If you absolutely want to keep the file-by-file smaller one, then use a copy utility that skips overwriting based on size.

(There is a "FLAC frontend", but it has eaten files of mine. Avoid.)

one of the potential targets is my mobile device.
If you cannot fit it all: Go lossy. mp3 or something.
High Voltage socket-nose-avatar

Re: higher compression ratios in FLAC

Reply #26
Once in a while this topic arises and the last time i checked CUEtools flake -8 compressed slightly better here compared to flac -8 -ep on a mixed genre test corpus while being much faster.
It creates standard spec files but uses bigger blocksizes for high samplerate material.
You may use it with -b 4096 then for a blocksize of 4096 for everything like reference flac does.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: higher compression ratios in FLAC

Reply #27
FWIW, here's the command line I use:

flac -8 -l 12 -b 4096 -m -e -r 6

That's the way it used to be before FLAC 8 was dumbed down awhile back.

FLAC 1.3.2 with -8 -e -p

Talking Heads - Stop Making Sense (1999 Re-Issue)

01-Psycho Killer / 874 kbps
02-Heaven / 868 kbps
03-Thank You for Sending Me an Angel / 994 kbps
04-Found a Job / 940 kbps
05-Slippery People / 917 kbps
06-Burning Down The House / 960 kbps
07-Life During Wartime / 977 kbps
08-Making Flippy Floppy / 977 kbps
09-Swamp / 906 kbps
10-What a Day That Was / 953 kbps
11-This Must Be the Place (Naive Melody) / 942 kbps
12-Once in a Lifetime  / 973 kbps
13- Genius of Love / 950 kbps
14-Girlfriend Is Better / 953 kbps
15-Take Me to the River / 933 kbps
16-Crosseyed and Painless / 978 kbps

Average bitrate: 945 kbps

FLAC 1.3.2 with flac -8 -l 12 -b 4096 -m -e -p -r 6

01-Psycho Killer / 874 kbps
02-Heaven / 868 kbps
03-Thank You for Sending Me an Angel / 994 kbps
04-Found a Job / 940 kbps
05-Slippery People / 917 kbps
06-Burning Down The House / 960 kbps
07-Life During Wartime / 977 kbps
08-Making Flippy Floppy / 977 kbps
09-Swamp / 906 kbps
10-What a Day That Was / 953 kbps
11-This Must Be the Place (Naive Melody) / 942 kbps
12-Once in a Lifetime  / 973 kbps
13- Genius of Love / 950 kbps
14-Girlfriend Is Better / 953 kbps
15-Take Me to the River / 933 kbps
16-Crosseyed and Painless / 978 kbps

Average bitrate: 945 kbps

-----

So, basically identical. 527.4 MB for the album either way as reported by Nautilus, with foobar2000 having gone over them with Optimize File Layout and Minimize File Size. Same encoding time either way.

I'd like to point out that running over the files again when someone used an older FLAC encoder, you usually get smaller files and faster encode times with -8 -e -p, or maybe even just setting 8 by itself, and you don't have to deal with unwieldy command lines and dead slow encoding. Recent FLAC releases have improved compression.

Commentary for others, including OP:

WavPack Extra High (even more so with some additional processing, but after Extra High x3 you see severe diminishing returns for the encoding time) can still beat FLAC -8 -ep, but by about 1.25% or less on a typical album. foobar2000 is inaccurate when it says processing time has no effect on file size with WavPack. Extra Processing level 1 will get you the most bang for your buck when going past Extra High. Files might be 100 KB smaller than simply Extra High without severe slowdowns.

Most of the other lossless codecs either do worse than FLAC, have horrible licenses, or take an insane amount of CPU time to decode (sometimes all of the above) and aren't worth screwing around with.

If you need to free space, you could try WavPack, but for the most part FLAC -8 -ep is going to be your best bet because it is open source, has low decode requirements, and is pretty much universally supported.

Basically, with proprietary lossless codecs, you spend weeks encoding your media library over again so you can store 1% more music in a format where you might not ever get your data back out of it and it probably won't play on much of anything. (If it does, it might hog the processor and drain your battery.)  And it may not actually be lossless because we can't tell what the QA process is to ensure that it's not hitting corner cases and silently corrupting your data. If that's your thing, go for it.

FLAC decodes quickly no matter what level you encoded it at while other formats, like Monkey's Audio, get a little extra compression at higher settings at the cost of causing whatever you're trying to play them back on to have a stroke. Someone encoded some files at APE level Insane or whatever and playing them caused my Core i7-6560U laptop to start stuttering. I got them over to FLAC as soon as possible and for like 2% larger files, they take almost no CPU time to play. :P

With compression, the 80/20 rule is very much in play. Almost anything should be fast enough to encode at FLAC level 8 at several hundred times real speed without the -e -p switches now, so if nothing else, just use FLAC 8.

Re: higher compression ratios in FLAC

Reply #28
Once in a while this topic arises and the last time i checked CUEtools flake -8 compressed slightly better here compared to flac -8 -ep on a mixed genre test corpus while being much faster.
It creates standard spec files but uses bigger blocksizes for high samplerate material.
You may use it with -b 4096 then for a blocksize of 4096 for everything like reference flac does.


I found some stuff that was encoded with Flake the other day and ran it through FLAC (1.3.2) -8 -e -p and it compressed better with FLAC. I guess your mileage may vary. I'd be skeptical of alternative FLAC encoders because there's a comprehensive test suite for reference libFLAC and probably not much testing at all with alternative encoders. Since the point is to preserve the original data, you may just want to play it safe and use the official software.

I checked to see if the rips passed AccurateRIP and they did in that case. Granted, it's not a comprehensive test, but I wouldn't make a habit of using software like Flake.

Simply encoding and decoding several files and matching MD5 sums does not a test suite make. ;)

Re: higher compression ratios in FLAC

Reply #29
I found some stuff that was encoded with Flake the other day and ran it through FLAC (1.3.2) -8 -e -p and it compressed better with FLAC. I guess your mileage may vary. I'd be skeptical of alternative FLAC encoders because there's a comprehensive test suite for reference libFLAC and probably not much testing at all with alternative encoders. Since the point is to preserve the original data, you may just want to play it safe and use the official software.

I checked to see if the rips passed AccurateRIP and they did in that case. Granted, it's not a comprehensive test, but I wouldn't make a habit of using software like Flake.

Simply encoding and decoding several files and matching MD5 sums does not a test suite make. ;)
In guess you missed the older threads over here. CUEtools flake is very well tested for several years now and had known minor errors fixed immediately when reported.
It is not about simply encoding several files.
For size tests you should use a recent CUEtools 2.16 or 2.17 compile encoding some of your files and not rely on some flake stuff you found the other day.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: higher compression ratios in FLAC

Reply #30
I found some stuff that was encoded with Flake the other day and ran it through FLAC (1.3.2) -8 -e -p and it compressed better with FLAC. I guess your mileage may vary. I'd be skeptical of alternative FLAC encoders because there's a comprehensive test suite for reference libFLAC and probably not much testing at all with alternative encoders. Since the point is to preserve the original data, you may just want to play it safe and use the official software.

I checked to see if the rips passed AccurateRIP and they did in that case. Granted, it's not a comprehensive test, but I wouldn't make a habit of using software like Flake.

Simply encoding and decoding several files and matching MD5 sums does not a test suite make. ;)
In guess you missed the older threads over here. CUEtools flake is very well tested for several years now and had known minor errors fixed immediately when reported.
It is not about simply encoding several files.
For size tests you should use a recent CUEtools 2.16 or 2.17 compile encoding some of your files and not rely on some flake stuff you found the other day.


I'm sorry for implying that Flake has problems if that's how you took it.

What I meant to say was:

Reference libFLAC has such a comprehensive test battery that must be passed before any release goes out that it takes hundreds of hours for it to run. On top of that, most software that supports FLAC uses reference by incorporation, which means that it has the most users, which means that corner cases are most likely found and fixed already, considering that reference libFLAC goes back 18 years if you go by when the bitstream format for FLAC was frozen and even longer if you count all development time.

It's more complex than some people have banged around on it a little bit and it doesn't seem to do anything horrible.

In Linux distributions, one of the reasons I stick with the default file system is that it's the default for reasons. Sure, there are geewhizbang alternatives (that have weaknesses to go along with their features), but one thing about the default file system is....it just kind of has to work, and if big corporations like Google and others are tossing around exabytes of data on it and it's used internally in billions of Android devices, and this and that, it tells me that squeaky wheels are going to get grease. Perhaps unsurprisingly, some of the worst data loss situations I've ever heard of came about when people went too far off the beaten path and started using some file systems that were "supported", kind of, but not widely used.

As far as the Flake encoder goes, I think Winamp used it, but Winamp ceased being relevant a long time ago, and I had to look up what this CUE Tools thing was, and it looks all Windows and .NET-ey and so my interest level is dropping. My interest level is dropping.

To be honest, after thousands of FLAC albums, I was surprised to see Flake having been used on a few, so that kind of tells me what kind of real world exposure it's getting.

Re: higher compression ratios in FLAC

Reply #31
If you are still interested in testing the encoder performance you may use this stand alone executable encoder over here https://hydrogenaud.io/index.php/topic,106446.msg903939.html#msg903939
I runs under wine and doesn't need net to run.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: higher compression ratios in FLAC

Reply #32
If you are still interested in testing the encoder performance you may use this stand alone executable encoder over here https://hydrogenaud.io/index.php/topic,106446.msg903939.html#msg903939
I runs under wine and doesn't need net to run.

Hooked CueTools Flake up to foobar2000 1.4.4 beta 1 on my Core i7-6560U laptop on Fedora 30 on an nvme SSD and the Ext4 file system. Both codecs used four encoder jobs (2 physical cores with hyperthreading) running at High priority.

Talking Heads - Stop Making Sense (1999 Re-Issue)

FLAC source converted to FLAC 8 using both libFLAC 1.3.2 and CueTools Flake

Reference libFLAC 1.3.2 with git patches up to November of 2018 as provided by Rarewares in their x86-64 bundle with SSE 3 optimizations compiled by the Intel Compiler:

Time: 15 seconds.

Average Bitrate: 946 kbps

Finished size (Reported by Nautilus after Optimize File Layout and Minimize File Size used in foobar2000): 528 MB

CueTools Flake:

Time: 55 seconds

Average Bitrate: 946 kbps

Finished size (Reported by Nautilus after Optimize File Layout and Minimize File Size used in foobar2000): 527.8 MB

-----------

Conclusion: Reference libFLAC is well over 3 times faster than CueTools Flake at setting 8 and the difference in file size and bitrate is negligible between the two. Reference libFLAC is also more thoroughly tested, as I mentioned previously.

I guess the good news is that according to the audio MD5 provided by foobar2000, Flake didn't _corrupt_ the output of the FLAC files it made, but this is one test with one album. Even if Flake doesn't have any potential data corruption issues, it is a very slow encoder.

-----------

Additional:

libFLAC 1.3.2 (same setup) setting -8 -e -p

Time: 6 Minutes 14 Seconds

Average Bitrate: 945 kbps

Finished size (Reported by Nautilus after Optimize File Layout and Minimize File Size used in foobar2000): 527.4 MB

Conclusion: Adding -e -p to libFLAC ended up resulting in 0.06% smaller files. That's 6/100ths of 1%. The cash value of a coupon in most states. And it requires 25 times as much encoding time to get this.

If the math holds up, you'd reclaim 153.6 megabytes or so on a 256 GB FLAC collection by running them through again with -e -p. Worth it?

Well, at the price of SD card space, that's about 2 cents worth of space.


Re: higher compression ratios in FLAC

Reply #33
And some people do embed ridiculous-size album art in every track in an album ...

I hate that so much when I get FLACs like that because the only easy way I've found to get rid of it is to remove the image file by file in Easytag and then toss the files in foobar2000 and hit "Optimize File Layout and Minimize File Size". There's probably an easier way.

Sometimes the difference isn't shocking, but other times you can weed out 7-8 MB or more of useless embedded album art from a given album.

Re: higher compression ratios in FLAC

Reply #34
Can't you do "Remove all pictures" in batch in Foobar2000?
I usually use Metaflac in frontah, but it is a 4 step process anyway. Add some text around 500 bytes, remove metadata block 'picture', remove padding, remove the text to keep a little bit of padding for future tag edits.

Re: higher compression ratios in FLAC

Reply #35
Can't you do "Remove all pictures" in batch in Foobar2000?
I usually use Metaflac in frontah, but it is a 4 step process anyway. Add some text around 500 bytes, remove metadata block 'picture', remove padding, remove the text to keep a little bit of padding for future tag edits.

For some reason, Easytag will remove the pictures in that they're still in the file but Nautilus can't see them anymore (I presume nothing can.). So that's all kinds of unhelpful. After using Easytag to do that, doing an Optimize File Layout and Minimize File Size in foobar2000 removes them for good, along with all of the padding, as it is essentially writing out a new file with a new tag.

But now that you mention it, a remove all pictures was in foobar2000 the entire time. d'oh!.

Re: higher compression ratios in FLAC

Reply #36
Jóhann Jóhannsson (2011) The Miners' Hymns
i8700k
flac 1.32 -8, ~12sec 208.518.631 Bytes
CUEtools flake -8, ~23sec 207.553.190 Bytes
flac 1.32 -ep -8, ~111sec 207.839.232 Bytes

Removed padding and optimized on all with foobar.

CUEtools is not to shabby against -ep here. I found on silent material CUEtools flake can do things more efficient as even flac -ep
You see thats why one should use a test corpus. I normaly drop ~50-100 albums in for conclusions.
Even this only represents its behaviour here with my music that i report so maybe others can compare.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: higher compression ratios in FLAC

Reply #37
Back to removing album art, you have to remove it then Optimize File Layout and Minimize Size in foobar2000.

Two step process. Still.... :P I just tried it. I wish embedded art was never allowed. What's wrong with folder.jpg?

Re: higher compression ratios in FLAC

Reply #38
I wish embedded art was never allowed. What's wrong with folder.jpg?
also: let's put tags into .txt files, who needs more than just plain audio stream in a file?


Re: higher compression ratios in FLAC

Reply #40
Two step process. Still....
But now you have no padding and a spelling correction or replaygain scan will cause rewriting of the entire file. I think the Optimize layout should leave a little bit automatically, but it is what it is.

Tags that apply to the entire release (album artist, cat number) could go into a single file, but the risk of losing them is too great, and a special application or plugin is needed to read such a format. With pictures it is the other way around. Any program can display or replace cover.jpg, its size is a rough indication of the quality. Annoying to see the cover.jpg not displayed because there is a low quality version embedded and forgotten.

Re: higher compression ratios in FLAC

Reply #41
comrades, by comparing flac8 & flake8 you compare warm and soft. if you take flac (max-compress), do it for flake (max-compress - 12).

Re: higher compression ratios in FLAC

Reply #42
comrades, by comparing flac8 & flake8 you compare warm and soft. if you take flac (max-compress), do it for flake (max-compress - 12).
No! CUEtools flake -8 creates standard files. Anything higher creates non-standard files that may not play everywhere. Besides that forcing CUEtools flake to lax settings is not really optimized and more like a leftover from vintage flake.
All explained in old threads over here.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: higher compression ratios in FLAC

Reply #43
[...] may not play everywhere.[...]
on hardwares? yes, in "old threads over here" says that. without "proof". but it's the little things.
there is not a word about it in the original question.
different between flake8/9/10 only in key (-m) - 3/6/5. anybody test this moment?

Re: higher compression ratios in FLAC

Reply #44
I wish embedded art was never allowed. What's wrong with folder.jpg?
also: let's put tags into .txt files, who needs more than just plain audio stream in a file?

A little bit of plaintext is basically nothing compared to the 1.5 MB JPEGs that I've seen certain people put into EACH music file * 100. :) At this point, you're talking 5-8% of a typical FLAC file being the freaking album art. Even if they're only 100 KB each, it's still obnoxious and people should cut it out.

That being said, I find ALAC rips too where freaking iTunes used the tags as an open sewer for a bunch of gibberish. Just the facts, ma'am.

I scoop out all of the iTunes crap from the tag before converting them to FLAC 8, and then optimize the FLAC files.

Any halfway decent music player app should be able to load lyrics in real time from a website without storing them in each and every file. Interestingly, iTunes lyrics tags don't seem to be picked up in any of my players, even though metaflac will happily copy them into an empty field if you let it. And worse, the iTunes lyrics are often wrong.

Re: higher compression ratios in FLAC

Reply #45
comrades, by comparing flac8 & flake8 you compare warm and soft. if you take flac (max-compress), do it for flake (max-compress - 12).
No! CUEtools flake -8 creates standard files. Anything higher creates non-standard files that may not play everywhere. Besides that forcing CUEtools flake to lax settings is not really optimized and more like a leftover from vintage flake.
All explained in old threads over here.

I haven't tried non-Subset in Flake, but when I did it up in FLAC, it brought each file down by about 100 KB at the cost of encoding at 0.8x of real time on my Core i7 6560U with four encoder processes running.

Even if I had the time to deal with non-Subset, the fact that decoders are not required to play them makes the proposition too iffy for my taste.

The point of Subset was mainly to prevent FLAC from turning into something more like Monkey's Audio where hardware couldn't play it back in real time in exchange for like 1 or 2% better compression. By making non-Subset difficult to use, it prevented people from accidentally stumbling into a situation where their FLAC files won't play on something.

Re: higher compression ratios in FLAC

Reply #46
Two step process. Still....
But now you have no padding and a spelling correction or replaygain scan will cause rewriting of the entire file. I think the Optimize layout should leave a little bit automatically, but it is what it is.

Tags that apply to the entire release (album artist, cat number) could go into a single file, but the risk of losing them is too great, and a special application or plugin is needed to read such a format. With pictures it is the other way around. Any program can display or replace cover.jpg, its size is a rough indication of the quality. Annoying to see the cover.jpg not displayed because there is a low quality version embedded and forgotten.

Actually, I've edited the tags after optimizing them and it does seem to leave some, because it did not have to write out a whole new file.

In fact, I went over them again and it said the size did not change. :)

"Tags in a single file", is, I think a reply to someone else? Anyway, that's essentially what you have with a FLAC image of an entire album and a CUE sheet.

I hate finding stuff like this because it means I _have to_ split it up myself. All so they could save like 1 MB and most of the time they didn't even use FLAC 8 along with it. :)

There's too many people out there using CD ripping software and FLAC who very obviously haven't the faintest idea of what they're doing.

Re: higher compression ratios in FLAC

Reply #47
I wish embedded art was never allowed. What's wrong with folder.jpg?
also: let's put tags into .txt files, who needs more than just plain audio stream in a file?

Roll on WAV!

I hope that was a joke. An audio collection in WAV/AIFF files is about the dumbest possible way to store them, but it's your money. :)

In fact, I'm not even sure why Apple ever created a lossless format. Much less one that's shockingly less efficient than something that already existed.  It's not like any of their products have plenty of storage laying around unless it's a Mac, and even then, not guaranteed.

To Wombat: I'm not sure that "most efficient ALAC codec" is worth bragging about since there's essentially no case where ALAC could possibly beat FLAC for interoperability and robustness, except on Apple stuff that has no real storage anyway where no lossless codec could really be useful, where they have chosen not to support real standards, again. Most efficient ALAC codec means you're the best looking horse in the glue factory.

That being said, a simple conversion from ALAC (encoded with Apple's refalac) to FLAC -8 generally nets a reduction of about 15-18 kbps in my experience. Or 4-8 MB across an album. Well worth the 10 seconds to get it out of Apple's format.

ALAC is open source now, so there's no point bickering about software freedom either way, but it's not a good codec. Even if you have found a way to make the bitrate and performance comparable to FLAC, ALAC would still have no error robustness or even a way to detect corruption, thanks to the MP4 container.

With FLAC, I can ask if anything in my library is corrupt and know about it a minute or two later (impressive for such a large library) and then I can stomp anything that is corrupt with a backup copy. With ALAC, you may never know.

Even worse, from what I've read, ALAC files with more than a little corruption generally become entirely unplayable and it takes a lot to completely destroy a FLAC file even if you only had one copy, because there might be a dropout of 1/10th of a second somewhere, but it recovers the stream.

As usual, Apple "engineering" by marketing department and Not Invented Here mentality at its best.

If you have anything in ALAC it's best to use refalac to decompress it and then feed it into FLAC, and then to check again to make sure it matches AccurateRIP or something.

Re: higher compression ratios in FLAC

Reply #48
To Wombat: I'm not sure that "most efficient ALAC codec" is worth bragging about since there's essentially no case where ALAC could possibly beat FLAC for interoperability and robustness, except on Apple stuff that has no real storage anyway where no lossless codec could really be useful, where they have chosen not to support real standards, again. Most efficient ALAC codec means you're the best looking horse in the glue factory.

That being said, a simple conversion from ALAC (encoded with Apple's refalac) to FLAC -8 generally nets a reduction of about 15-18 kbps in my experience. Or 4-8 MB across an album. Well worth the 10 seconds to get it out of Apple's format.
Are you talking to my old upload thread? Back in 2014 when i posted this CUEtools ALAC -10 was almost equal to flac -8. At this time indeed it was the best compressing ALAC encoder for CD material i knew of.
I never used it but ALAC i can imagine is still the main lossless format for many.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: higher compression ratios in FLAC

Reply #49
I wish embedded art was never allowed. What's wrong with folder.jpg?
also: let's put tags into .txt files, who needs more than just plain audio stream in a file?
Roll on WAV!
I hope that was a joke.
Yeah, like that embedded comment ...

I use folder.jpeg quite a lot, but some releases have per-track art. If you don't like embedded, don't use it.


In fact, I'm not even sure why Apple ever created a lossless format. Much less one that's shockingly less efficient than something that already existed.
Neither am I, but Microsoft also wanted their own thing.
One hypothesis: probably because Apple landed on the MP4 container. Which is video ready and ... and Apple had DRM in MP4, maybe that is a reason?

Yeah, it is mediocre at best (and sucks when it comes to integrity check ...) So what? Apple users do as they are told ... and how many would need an extra hard drive anyway? Lossless audio - outside BluRay discs - is a niche product.

High Voltage socket-nose-avatar

 
SimplePortal 1.0.0 RC1 © 2008-2019