Skip to main content

Topic: New FLAC encoder (Read 289888 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • iGold
  • [*][*][*]
New FLAC encoder
Reply #275
Code: [Select]
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)


I thought this was solved?

I submitted bug about it in sourceforce bug tracker after I discovered it was not fixed after 1st try. Justin, are you looking there?

New FLAC encoder
Reply #276
Code: [Select]
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)


I thought this was solved?

I submitted bug about it in sourceforce bug tracker after I discovered it was not fixed after 1st try. Justin, are you looking there? 

I missed that one...  I've changed the settings so I get an email now when someone posts a new bug.  I have also updated the status of your bug report.
Thanks!
-Justin

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
New FLAC encoder
Reply #277
My results for Flake SVN r110 wisodev build P3 optimised with RW corpus.

Code: [Select]
Setting     Ratio     Enc     Dec
=================================
0         65.352%    136x    106x
1         62.822%    114x    104x
2         62.661%     98x    104x
3         59.581%     94x     97x
4         59.392%     92x     93x
5         59.292%     90x     96x
6         59.649%     66x     96x
7         59.307%     54x     96x
8         59.132%     49x     95x
8 -m 1    59.115%     83x     94x
8 -v 1    58.747%     47x     93x
9         59.019%     40x     95x
10        59.006%     28x     94x
11        58.758%     22x     83x
12        58.729%      9x     83x
12 -v 1   58.378%      9x     84x
99        58.336%      2x     72x
Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.

I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...
These results appear to bear yours out completely.  I'm so glad you posted or I would have been really concerned about my -6 results!

I will post results for the other wisodev compiles (normal and org) as well, in the next day or so.  I will also post the spreadsheet used to calculate the summaries (uploaded to Google Docs & Spreadsheets).

Edit: Something else of note.  In fact two somethings.

Firstly, I should mention that the results above are CPU-only times, where my TAK lossless comparison values are CPU+IO times.  I started reporting CPU+IO in that comparison and it's awkward to stop now.

Secondly, the scripts I use to perform my tests use FSUM to check the decoded WAVE files against the MD5 hashes of the source files once all files are decoded.  This normally serves me well, but in this instance nearly all files fail the check.

I have done a little checking and it appears to be down to the WAVE header written, which I think is 2 bytes smaller than the source (bear in mind the source files are decoded WavPack files).  Again, from my limited understanding, this appears to be down to the ExtraParamSize bytes not being written by the FLAC decoder.  From the document I have read it appears that these bytes are not required if no extra parameters are present.  Strangely enough though every fourth file or so the hashes do match!

Edit: It seems that, as expected, with the files that do match MD5 hashes, neither have the ExtraParamSize bytes; therefore FLAC is acting consistently, while the source files are not consistent.  Maybe this is noticable as most decompressors just return the original WAVE header, whereas FLAC creates its own?

I really have no idea what all this means, but it thought it worth noting.  I bit compared a few files that had mismatched MD5s using foobar and the audio is bit-identical, as you would expect.

Here's the report from FSUM for -5 (it's the same for all though).

Code: [Select]
FAILED       MD5          41_30sec.wav
FAILED      MD5          ATrain.wav
FAILED      MD5          Bachpsichord.wav
OK          MD5          Bartok_strings2.wav
FAILED      MD5          BeautySlept.wav
FAILED      MD5          BigYellow.wav
FAILED      MD5          Blackwater.wav
FAILED      MD5          bodyheat.wav
OK          MD5          chanchan.wav
FAILED      MD5          DaFunk.wav
FAILED      MD5          death2.wav
OK          MD5          Debussy.wav
FAILED      MD5          EnolaGay.wav
FAILED      MD5          experiencia.wav
FAILED      MD5          female_speech.wav
FAILED      MD5          FloorEssence.wav
OK          MD5          getiton.wav
FAILED      MD5          gone.wav
OK          MD5          Hongroise.wav
FAILED      MD5          Illinois.wav
FAILED      MD5          ItCouldBeSweet.wav
FAILED      MD5          kraftwerk.wav
FAILED      MD5          Layla.wav
OK          MD5          Leahy.wav
FAILED      MD5          LifeShatters.wav
FAILED      MD5          macabre.wav
FAILED      MD5          Mahler.wav
FAILED      MD5          male_speech.wav
OK          MD5          Mama.wav
FAILED      MD5          MidnightVoyage.wav
FAILED      MD5          mybloodrusts.wav
OK          MD5          NewYorkCity.wav
OK          MD5          OrdinaryWorld.wav
FAILED      MD5          Polonaise.wav
FAILED      MD5          Quizas.wav
FAILED      MD5          riteofspring.wav
OK          MD5          rosemary.wav
FAILED      MD5          Scars.wav
OK          MD5          SinceAlways.wav
FAILED      MD5          thear1.wav
FAILED      MD5          TheSource.wav
OK          MD5          TomsDiner.wav
OK          MD5          trust.wav
OK          MD5          Twelve.wav
FAILED      MD5          velvet.wav
OK          MD5          Waiting.wav

31 checksums failed
  • Last Edit: 18 October, 2006, 11:54:07 AM by Synthetic Soul
I'm on a horse.

  • Wombat
  • [*][*][*][*][*]
New FLAC encoder
Reply #278
I don´t know the story of flake but now while i test around i am not really thrilled to have the extra options.
These options like l32 or v1 simply brake the standard in my eyes. l32 encoded files don´t play on every hardware and v1 is confusing even winamp and my Squeezebox again when i want to fast forward.
What about *.flak for flake encodings? So everyone knows he may have to reencode to use it properly when he gets such a file by accident.
This may have been discudssed before but i somehow missed that.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

  • MedO
  • [*][*][*][*]
New FLAC encoder
Reply #279
These options like l32 or v1 simply brake the standard in my eyes.


As I understand it, files encoded with -v 1 or -l 32 are standard flac files, not a weird format extension introduced by flake.

  • bhoar
  • [*][*][*][*][*]
  • Members (Donating)
New FLAC encoder
Reply #280
My results for Flake SVN r110 wisodev build P3 optimised with RW corpus.

Code: [Select]
Setting	 Ratio	 Enc	 Dec
=================================
0 65.352% 136x 106x
1 62.822% 114x 104x
2 62.661% 98x 104x
3 59.581% 94x 97x
4 59.392% 92x 93x
5 59.292% 90x 96x
6 59.649% 66x 96x
7 59.307% 54x 96x
8 59.132% 49x 95x
8 -m 1 59.115% 83x 94x
8 -v 1 58.747% 47x 93x
9 59.019% 40x 95x
10 59.006% 28x 94x
11 58.758% 22x 83x
12 58.729%   9x 83x
12 -v 1  58.378%   9x 84x
99 58.336%   2x 72x

I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").

Looking at the above, -5 seems to be the sweet spot if your rate performance as the most important and "-8 -v 1" the sweet spot if you rate compression as the most important.

-brendan
Hacking CD Robots & Autoloaders: http://hyperdiscs.pbwiki.com/

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #281
Thanks wisodev.

I have an AMD XP CPU.  Should I be looking at /org/flake.exe, /p3/flake.exe or /flake.exe?


Code: [Select]
.\flake.exe        - ICL Build with PGO Optimizations
.\P3\flake.exe        - ICL Build with PGO & P3 Optimizations
.\ORG\flake.exe        - ICL Build


The P3 is optimized for Pentium III and should work for all processors above P3 specs. It Works very well on Athlon XP processors (I have Athlon XP 2000+). I have in mind to add P4 build but I can't build it on my machine because of PGO optimization that req running the program and my CPU doesn't support P4 builds.

The original (ORG) build is not considered for normal usage only added for speed testing of PGO and PGO&P3 builds. For most CPUs the .\flake.exe build is good and should be used as safest solution.

@wisodev

I would be interested what compiler flags you use (and which compiler). Have you added asm to the sources?


All I have added is build enviroment in win32 directory. No assembly optimizations added by me. I have added some small changes to code to build it cleanly under MSVC with Intel compiler. Maybe Justin can add them to official sources (just diff my sources with SVN revision 110 to see what have been changed).

I am using MSVC++ v7.1 and Intel C++ Compiler v8.0.040. This version of Intel compiler works best for me. I have tried newer versions but I can't get better results.

The compiler flags are main reason for speedups of my builds. I am using PGO optimization provided by Intel C++ compiler and for P3 build additionaly /QxK flag is used to enable even more optimizations. More information is avaible when you check the project settings for eacg configuraion (all witch begin with Release, the Debug builds are obviously not optimized ;-).

  • pest
  • [*][*][*]
New FLAC encoder
Reply #282
I really have no idea what all this means, but it thought it worth noting.  I bit compared a few files that had mismatched MD5s using foobar and the audio is bit-identical, as you would expect.


You're right. FLAC creates a fresh RIFF/WAVE-Header, but Wavpack for example recreates the original one.
  • Last Edit: 18 October, 2006, 01:09:54 PM by pest

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
New FLAC encoder
Reply #283
Code: [Select]
.\flake.exe        - ICL Build with PGO Optimizations
.\P3\flake.exe        - ICL Build with PGO & P3 Optimizations
.\ORG\flake.exe        - ICL Build
The P3 is optimized for Pentium III and should work for all processors above P3 specs. It Works very well on Athlon XP processors (I have Athlon XP 2000+). <snip> The original (ORG) build is not considered for normal usage only added for speed testing of PGO and PGO&P3 builds. For most CPUs the .\flake.exe build is good and should be used as safest solution.
Thanks for the info wisodev. With that in mind I won't bother with the ORG build.  Here's the PGO and PGO+P3 results:

Code: [Select]
                   |   PGO + P3     |   PGO
Setting     Ratio  |   Enc     Dec  |   Enc     Dec
===================+================+==============
0         65.352%  |  136x    106x  |  133x    104x
1         62.822%  |  114x    104x  |  110x    106x
2         62.661%  |   98x    104x  |   95x    104x
3         59.581%  |   94x     97x  |   91x     98x
4         59.392%  |   92x     93x  |   90x     96x
5         59.292%  |   90x     96x  |   88x     96x
6         59.649%  |   66x     96x  |   65x     99x
7         59.307%  |   54x     96x  |   52x     98x
8         59.132%  |   49x     95x  |   47x     95x
8 -m 1    59.115%  |   83x     94x  |   80x     93x
8 -v 1    58.747%  |   47x     93x  |   46x     95x
9         59.019%  |   40x     95x  |   39x     96x
10        59.006%  |   28x     94x  |   27x     95x
11        58.758%  |   22x     83x  |   21x     83x
12        58.729%  |    9x     83x  |    8x     81x
12 -v 1   58.378%  |    9x     84x  |    8x     84x
99        58.336%  |    2x     72x  |    2x     72x

View the Google spreadsheet with all values for all files here.

You're right. FLAC creates a fresh RIFF/WAVE-Header, but Wavpack for example recreates the original one.
Thanks for the confirmation.  This makes sense to me.
I'm on a horse.

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
New FLAC encoder
Reply #284
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Not sure if this fits the bill, but I can't think of any other way of displaying it.

I'm on a horse.

New FLAC encoder
Reply #285
I don´t know the story of flake but now while i test around i am not really thrilled to have the extra options.
These options like l32 or v1 simply brake the standard in my eyes. l32 encoded files don´t play on every hardware and v1 is confusing even winamp and my Squeezebox again when i want to fast forward.
What about *.flak for flake encodings? So everyone knows he may have to reencode to use it properly when he gets such a file by accident.
This may have been discudssed before but i somehow missed that.

The files may confuse some software decoders and skip in some hardware decoders, but I assure you they are 100% valid FLAC files based only on the specification.  The issues that occur are not Flake issues, they are due to there being 2 levels of compliance, the subset specification and the full specification.  Flake gives a warning when the files do not fall within the subset specification, but all generated files fall within the full specification.

If you're worried about getting a FLAC file which may cause problems, I could try to write a small utility program which would do frame analysis and detect whether a file falls within the subset specification.  As for winamp, maybe adding a seektable using metaflac would help?...I don't really know how the winamp plug-in does seeking and I don't use winamp (anymore).  I do use xmms, which completely locks up when seeking with the variable bitrate files...no good.  Personally, I see these as software bugs, but I can't really blame the makers since that feature of the specification is not implemented by the reference encoder.  MPlayer plays & seeks just fine...it uses its own demuxer & FFmpeg's decoder.

On an unrelated note, thanks for all the benchmarks!  The graph is great  Now that I see the results, I'm seriously considering abandoning the "mirror the reference encoder" idea for compression levels.  I think trimming it down to just 7 presets corresponding to  0, 1, 5, 8m1, 10, 11, and 99 should pretty much cover what is needed.

-0 for really-fast encoding
-1 for fast encoding
-5 for normal encoding
"-8 -m 1" for high compression
-10 for best compression within subset
-11 for extra compression outside subset (-12 speed loss isn't worth the tiny gain)
-99 for experimental

I would probably get rid of the numbered presets in favor of named presets.  What do you all think about this idea?

-Justin

  • bhoar
  • [*][*][*][*][*]
  • Members (Donating)
New FLAC encoder
Reply #286
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Not sure if this fits the bill, but I can't think of any other way of displaying it...


Actually, that was quite useful.  The two points on the encoding line that stood out to me numerically after looking up and down the chart a few times also appear to stand out rather obviously in the chart.

-brendan
Hacking CD Robots & Autoloaders: http://hyperdiscs.pbwiki.com/

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #287
@PrakashP

For PGO&P3 Instrumentation build (Release Instrumentation P3|Win32) I am using this options for ICL:

Code: [Select]
/GL /Ox /Og /Ob1 /Ot /G7 /GA /QxK /FD /MT /Qc99 /Qprof_genx /Gd /TC

For PGO&P3 Feedback build (Release Feedback P3|Win32) I am using this options for ICL:

Code: [Select]
/GL /Ox /Og /Ob1 /Ot /G7 /GA /QxK /FD /MT /Qc99 /Qprof_use /Gd /TC

For PGO only build just remove /QxK option.

wisodev
  • Last Edit: 19 October, 2006, 03:03:52 AM by wisodev

  • Wombat
  • [*][*][*][*][*]
New FLAC encoder
Reply #288
The files may confuse some software decoders and skip in some hardware decoders, but I assure you they are 100% valid FLAC files based only on the specification.  The issues that occur are not Flake issues, they are due to there being 2 levels of compliance, the subset specification and the full specification.  Flake gives a warning when the files do not fall within the subset specification, but all generated files fall within the full specification.

If you're worried about getting a FLAC file which may cause problems, I could try to write a small utility program which would do frame analysis and detect whether a file falls within the subset specification.  As for winamp, maybe adding a seektable using metaflac would help?...I don't really know how the winamp plug-in does seeking and I don't use winamp (anymore).  I do use xmms, which completely locks up when seeking with the variable bitrate files...no good.  Personally, I see these as software bugs, but I can't really blame the makers since that feature of the specification is not implemented by the reference encoder.  MPlayer plays & seeks just fine...it uses its own demuxer & FFmpeg's decoder.

Thanks for your answer. I read the first time deeper about flac and switches. I am sure you can penetrate normal flac to the point it doesn´t work on any player also
No need for a tool that tests the frames. This -v1 doesn´t seem to need any more decoding power than without so it is time to "correct" the decoders.
btw. this latest wisodev build is awsome fast!
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

  • MedO
  • [*][*][*][*]
New FLAC encoder
Reply #289
@Synthetic Soul:
Thanks for the tests, it would also be nice to have "-8 -m 1 -v 1" in next time, maybe instead of "-8 -v 1"? This should have a very good compression/encoding time ratio.
  • Last Edit: 19 October, 2006, 08:15:05 AM by MedO

  • kockroach
  • [*][*][*][*]
New FLAC encoder
Reply #290
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Not sure if this fits the bill, but I can't think of any other way of displaying it.



Synthetic Soul,

The plot of the results look great.  I was wondering why it is that "-m 1" is only tested on "8".  Can it not be used on levels 9 and up?  I didn't see anything about that, so I apologize if I missed it.  I have to wonder how much it could potentially increase encoding times of those above 9, especially if it can increase 12's encoding time significantly.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
New FLAC encoder
Reply #291
I tried -10 -m 1 on a single album and it gives a significant boost (~100%) to the encoding speed without decreasing the compression ratio (+1 kbps). This switch looks great!

I also tried -v1, but there's no seeking with foobar2000 (0.9.4.1).
  • Last Edit: 19 October, 2006, 08:32:04 AM by guruboolez

  • MedO
  • [*][*][*][*]
New FLAC encoder
Reply #292
I think "-9 -m 1" ist the same as "-8 -m 1".
"-9" and upward use search methods which are probably better (compression-wise) than "-m 1".

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
New FLAC encoder
Reply #293
@Synthetic Soul:
Thanks for the tests, it would also be nice to have "-8 -m 1 -v 1" in next time, maybe instead of "-8 -v 1"? This should have a very good compression/encoding time ratio.
The plot of the results look great.  I was wondering why it is that "-m 1" is only tested on "8".  Can it not be used on levels 9 and up?  I didn't see anything about that, so I apologize if I missed it.  I have to wonder how much it could potentially increase encoding times of those above 9, especially if it can increase 12's encoding time significantly.
I simply used the settings requested by Justin, and added -8 -m 1 as it was mentioned by MedO shortly after, and it was suggested that more info was required (so I thought I may as well add it to the list).

My wife is possibly beginning labour at the moment, so I kind of have other things on my mind. 

If I get chance I will try to run the tests requested.
I'm on a horse.

  • kockroach
  • [*][*][*][*]
New FLAC encoder
Reply #294
My wife is possibly beginning labour at the moment, so I kind of have other things on my mind. 

That's awesome!  Hope all goes well.   

  • iGold
  • [*][*][*]
New FLAC encoder
Reply #295
I think trimming it down to just 7 presets corresponding to  0, 1, 5, 8m1, 10, 11, and 99 should pretty much cover what is needed.
...
I would probably get rid of the numbered presets in favor of named presets.  What do you all think about this idea?

I think it's a good idea. And if we really want we always can use cryptic -b -t -l -m -r -s combinations

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
New FLAC encoder
Reply #296
That's awesome!  Hope all goes well.   
Thank you.  Things are unfortunately going very slowly, so there's no news yet.

Except...

My results for Flake SVN r110 wisodev build P3 optimised with RW corpus, with -8 -m 1 -v 1 and -9 to -12 with -m 1:

Code: [Select]
Setting          Ratio     Enc     Dec
======================================
0              65.352%    136x    106x
1              62.822%    114x    104x
2              62.661%     98x    104x
3              59.581%     94x     97x
4              59.392%     92x     93x
5              59.292%     90x     96x
6              59.649%     66x     96x
7              59.307%     54x     96x
8              59.132%     49x     95x
8 -m 1         59.115%     83x     94x
8 -v 1         58.747%     47x     93x
8 -m 1 -v 1    58.752%     83x     96x
9              59.019%     40x     95x
9 -m 1         59.115%     85x     96x
10             59.006%     28x     94x
10 -m 1        59.115%     86x    100x
11             58.758%     22x     83x
11 -m 1        58.920%     67x     76x
12             58.729%      9x     83x
12 -m 1        58.920%     67x     76x
12 -v 1        58.378%      9x     84x
99             58.336%      2x     72x

I don't understand the -9 -m 1/-10 -m 1 and -11 -m 1/-12 -m 1 thing, but I'm sure someone does.
  • Last Edit: 20 October, 2006, 04:19:20 AM by Synthetic Soul
I'm on a horse.

  • MedO
  • [*][*][*][*]
New FLAC encoder
Reply #297
I don't understand the -9 -m 1/-10 -m 1 and -11 -m 1/-12 -m 1 thing, but I'm sure someone does.

Well, the only setting that changes between -8, -9 and -10 is the -m switch, and you override that manually, so you get the same results because the settings are identical. The same goes for -11/-12.

Edit: I just saw that -11 -m 1 is bigger than -8 -m 1. I guess the "estimate" search method just isn't optimised for those high predictor orders. Just a guess, I don't know much about FLAC technology...

Edit2: I confused rows in the table... thanks Josef Pohm for the hint. 
  • Last Edit: 21 October, 2006, 01:06:49 PM by MedO

  • JWolf
  • [*][*]
New FLAC encoder
Reply #298
I have a Rio Karma and while it can play FlAKE -12 compressed flac files, it has a lot of trouble seeking with these FLAC files. FLAC 1.1.2_CVS which gets close compression works great with my Rio Karma.

What I'd like to know is what are the best settings for Flake to work with my Rio Karma? I'd like to compare compressions sizes to find out which is better. I was playing around and ended up gettng a larger file that didn't work vs FLAC 1.1.2_CVS.

What setting is it in FLAKE that causes it not to be fully compatible with my Rio? I tried the -v 1 as suggested earlier in the thread to make it compatible with FLAC and that also failed.

Jon
  • Last Edit: 20 October, 2006, 11:18:25 PM by JWolf

New FLAC encoder
Reply #299
I have a Rio Karma and while it can play FlAKE -12 compressed flac files, it has a lot of trouble seeking with these FLAC files. FLAC 1.1.2_CVS which gets close compression works great with my Rio Karma.

What I'd like to know is what are the best settings for Flake to work with my Rio Karma? I'd like to compare compressions sizes to find out which is better. I was playing around and ended up gettng a larger file that didn't work vs FLAC 1.1.2_CVS.

What setting is it in FLAKE that causes it not to be fully compatible with my Rio? I tried the -v 1 as suggested earlier in the thread to make it compatible with FLAC and that also failed.

Jon

For now the best setting for your Rio would probably be "flake -10".  It generates files which are roughly equivalent (compatibility-wise) to "flac -8".  Definitely don't use -v 1 since it is probably the most experimental option and has compatibility issues.

edit: to answer your other question more fully, it's probably the max LPC order which is causing problems on the Rio with flake -12 files.

edit2: Given the many questions, for the next release, I will try to make some documentation devoted to compatibility of various features of the FLAC specification and Flake encoding parameters.

-Justin
  • Last Edit: 21 October, 2006, 12:17:48 AM by Justin Ruggles