Skip to main content

Topic: New FLAC encoder (Read 291225 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
New FLAC encoder
Reply #250
Thank you very much Wombat

May I suggest to Justin that the first official release of flake should come with a small warning (inside the documentation and/or inside flake.exe with -h switch) about possible hardware players' issues with -11 & -12 setting?

New FLAC encoder
Reply #251
Exactly how you predicted. -10 still is playable. From -11 on it stutters.


To my little knoledge decoding speed of lossless codecs mostly depends on the maximum number of predictor order used.

Flake -5 uses 8 max predictor order, like Flac -5 (and Garf version of Flac).
Flake -8,-9 and -10 uses 12 max predictor order like Flac -8 (and Garf version of Flac).
Flake -11 and -12 uses 32, so it requires much more decoding power.

I hope that's correct.

  • jcoalson
  • [*][*][*][*][*]
  • Developer
New FLAC encoder
Reply #252
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this.  with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion.  this goes for flac too which currently allows -l 32 without a warning.

most of the compression advantage of flake over FLAC 1.1.2 has already been incorporated into FLAC 1.1.3.  the extra compression from 32-order LPC is minimal.

Josh

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #253
Flake (Win32) from SVN revision 104 available:

Download from my Website

or use direct links:

Download binarys:
flake-svn-r104-win32-bin.zip

Download sources:
flake-svn-r104-win32-src.zip

NOTE: I have added my small patch to code (quality or speed is not affected), can be found here, the sources are already patched (this patch comes from my local subversion repository and is not related with sf.net subversion).

wisodev

  • fairway
  • [*][*]
New FLAC encoder
Reply #254

Exactly how you predicted. -10 still is playable. From -11 on it stutters.


To my little knoledge decoding speed of lossless codecs mostly depends on the maximum number of predictor order used.

Flake -5 uses 8 max predictor order, like Flac -5 (and Garf version of Flac).
Flake -8,-9 and -10 uses 12 max predictor order like Flac -8 (and Garf version of Flac).


Though seeking with a flake encoded file is still significantly slower than with a FLAC (reference encoder) encoded file (try it in winamp with the default flac plugin).

New FLAC encoder
Reply #255
Thank you very much Wombat 

May I suggest to Justin that the first official release of flake should come with a small warning (inside the documentation and/or inside flake.exe with -h switch) about possible hardware players' issues with -11 & -12 setting?

Okay, that sounds like a good idea.

I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this.  with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion.  this goes for flac too which currently allows -l 32 without a warning.

Cool.  Based on this, and the hardware decoding issues, I'll go ahead and add a separate warning for any lpc order over 12 for any sample rate up to 48kHz.  Once the official subset standard changes, I'll change Flake's warnings accordingly.

-Justin

  • JWolf
  • [*][*]
New FLAC encoder
Reply #256
Ah, oh. I see a problem. Does -12 take that much processing power or isn´t it 100% compatible to flac 1.1.2 decoding anymore?
My Squeezebox can´t play flake -12 on all titles. It stutters around. Flake -5 for example is fine.

Edit:
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.


Using the current version of FLAKE, I was able to play -12 compressed files on my Rio Karma with no hiccups.

Jon

  • jcoalson
  • [*][*][*][*][*]
  • Developer
New FLAC encoder
Reply #257
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this.  with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion.  this goes for flac too which currently allows -l 32 without a warning.

it is done.

the new limits: if the sample rate is <=48kHz, max blocksize = 4608 and max LPC order = 12.  all the tests I did showed that 32-order LPC got <0.1% more compression (usually a lot less).

http://flac.cvs.sourceforge.net/*checkout*...mat.html#subset

Josh

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #258
I have builded a faster FLAKE binary from rev. 104 but it needs some testing.

On one of my test files encoding time has droped from 193.468 seconds (previous r104 build) to 173.906 seconds (test r104 build) using -12 command-line option.

Download test binary:
flake_r104_test.zip

wisodev

  • MedO
  • [*][*][*][*]
New FLAC encoder
Reply #259
I have builded a faster FLAKE binary from rev. 104 but it needs some testing.

On one of my test files encoding time has droped from 193.468 seconds (previous r104 build) to 173.906 seconds (test r104 build) using -12 command-line option.

Download test binary:
flake_r104_test.zip

wisodev


Your test compile is a bit faster than mine on my machine (Celeron M 1,4Ghz). I can't give any good values, though, because I already got several seconds difference when running my own build twice, so the measurement is not very reliable.

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
New FLAC encoder
Reply #260
I have builded a faster FLAKE binary from rev. 104 but it needs some testing.

On one of my test files encoding time has droped from 193.468 seconds (previous r104 build) to 173.906 seconds (test r104 build) using -12 command-line option.
Here's some results for my "FLAC" corpus:

Code: [Select]
Setting       R104    R104 Test   Diff
======================================
-10       286.726s     274.561s    96%
-11       375.687s     347.035s    92%
-12      1002.105s     914.239s    91%

Edit: Here's the figures for my MinGW compile:

Code: [Select]
Setting  MinGW R104   R104 Test   Diff
======================================
-10        394.015s    274.561s    70%
-11        486.764s    347.035s    71%
-12       1322.178s    914.239s    69%

I don't think I'll bother anymore!    Excellent work wisodev.

NB: Timings were recorded using TIMER, and CPU-only times are reported.
  • Last Edit: 07 October, 2006, 03:32:11 PM by Synthetic Soul
I'm on a horse.

New FLAC encoder
Reply #261
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this.  with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion.  this goes for flac too which currently allows -l 32 without a warning.

it is done. 

the new limits: if the sample rate is <=48kHz, max blocksize = 4608 and max LPC order = 12.  all the tests I did showed that 32-order LPC got <0.1% more compression (usually a lot less).

http://flac.cvs.sourceforge.net/*checkout*...mat.html#subset

Josh

Splendid. I have modified the Flake warning message output and the return code of the flake_validate_params function to reflect the new standard.

  • iGold
  • [*][*][*]
New FLAC encoder
Reply #262
I checkout rev.107 from SVN and compile it myself under x86_64 Ubuntu. While encoding disc image I see:
Quote
Signed 16-bit 44100 Hz stereo
samples: 116712120 (0m16.961s)

If I try to compile this test code:
Code: [Select]
int th, tm, ts, tms;
tms = 116712120 * 1000 / 44100;
I get two "integer overflow" warnings.

I think int is too small here (in flake.c at line 447).

New FLAC encoder
Reply #263
I checkout rev.107 from SVN and compile it myself under x86_64 Ubuntu. While encoding disc image I see:
Quote
Signed 16-bit 44100 Hz stereo
samples: 116712120 (0m16.961s)

If I try to compile this test code:
Code: [Select]
int th, tm, ts, tms;
tms = 116712120 * 1000 / 44100;
I get two "integer overflow" warnings.

I think int is too small here (in flake.c at line 447).

Thanks. Fixed.
Also, anyone can submit bug reports on Flake's Sourceforge project page.  Getting bug reports via HA makes me feel a little guilty for taking up space with small fixes.  They are appreciated either way though.

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #264
Flake (Win32) from SVN revision 108 available:

Download from my Website

or use direct links:

Download binarys:
flake-svn-r108-win32-bin.zip

Download sources:
flake-svn-r108-win32-src.zip

wisodev

New FLAC encoder
Reply #265
although i ran into a couple of problems compiling.

Would it be too much of a problem to put my changes into the code, to keep it portable across compilers?

optimize.c line 219 replaced with: uint32_t bits[32]; (where 32 used to be levels. MSVC doesn't like non-constant sizes for these things.) I am sure that 32 is as large as levels gets.

in encode.c flake_set_defaults, msvc doen't like this syntax:
foo = ((int[]){1,2,3})[2];
so i defined all these "nameless" arrays as ones with names, so it becomes:
int sel[] = {1,2,3};
foo = sel[2];

thanks,
will

Both MSVC issues have now been fixed.

Also, I am almost done with a new website for Flake.  The only thing left is API documentation and benchmarks.  I am putting in a link to Synthetic Soul's lossless benchmarks, but it would also be nice to have more extensive tests to better illustrate the differences between the compression levels.  I'm not so worried anymore about detailed comparisons to the reference encoder because Synthetic Soul's tests give a good enough overview.  Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels.  Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful.  Corpus totals would be better than file-by-file results.

Thanks!
Justin

New FLAC encoder
Reply #266
Both MSVC issues have now been fixed.
Excellent, thanks

I have included my enc_flac plugin (sources are attached somewhere in this thread =) built with wisodevs libFlake r108 to the beta branch of Winamp.

If it gets a good response from our beta team and QA department (and I can't see why it wouldn't) maybe we'll put it in a subsequent version of winamp.

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
New FLAC encoder
Reply #267
Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels.  Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful.  Corpus totals would be better than file-by-file results.
Has anyone provided this yet?

I don't mind running the tests if no-one else will.  My machine isn't hugely fast (Athlon XP 2400+, 512MB RAM, Win2K) but it's OK, and I will need a compile from wisodev as I have proved already that my compiles are far inferior to his.
I'm on a horse.

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #268
Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels.  Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful.  Corpus totals would be better than file-by-file results.
Has anyone provided this yet?

I don't mind running the tests if no-one else will.  My machine isn't hugely fast (Athlon XP 2400+, 512MB RAM, Win2K) but it's OK, and I will need a compile from wisodev as I have proved already that my compiles are far inferior to his.


I will provide fresh compile in few hours including new experimental build using P3 optimizations (using only compiler options - not assembly optimized!!!) witch seems to be even faster then the last one.
  • Last Edit: 17 October, 2006, 12:48:20 PM by wisodev

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #269
Flake (Win32) from SVN revision 110 available:

Download from my Website

or use direct links:

Download binarys:
flake-svn-r110-win32-bin.zip

Download sources:
flake-svn-r110-win32-src.zip

wisodev
  • Last Edit: 29 November, 2006, 11:41:54 AM by wisodev

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
New FLAC encoder
Reply #270
Thanks wisodev.

I have an AMD XP CPU.  Should I be looking at /org/flake.exe, /p3/flake.exe or /flake.exe?
I'm on a horse.

  • MedO
  • [*][*][*][*]
New FLAC encoder
Reply #271
Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.

I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...

Edit: I just did a short test with the album "Supertramp - Some Things Never Change", comparing the commandlines "-8" and "-8 -m 1".
"-8 -m 1" was of course always way faster, and compressed 8 of the 12 files better than "-8". The differences in bitrate were quite small though, around 0.1-0.2kbps average.
  • Last Edit: 17 October, 2006, 05:16:37 PM by MedO

  • PrakashP
  • [*]
New FLAC encoder
Reply #272
@wisodev

I would be interested what compiler flags you use (and which compiler). Have you added asm to the sources?

  • MedO
  • [*][*][*][*]
New FLAC encoder
Reply #273
Also, I still get this time bug when encoding with wisodev's latest build (all binaries show this)
Code: [Select]
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)


I thought this was solved?

New FLAC encoder
Reply #274
Also, I still get this time bug when encoding with wisodev's latest build (all binaries show this)
Code: [Select]
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)


I thought this was solved?

I did too...sorry.  I just committed a 2nd attempt, which seems to work for me.  I just tested it on a 11 million sample wav and it correctly reported around 4 minutes.
-Justin


Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.

I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...

Edit: I just did a short test with the album "Supertramp - Some Things Never Change", comparing the commandlines "-8" and "-8 -m 1".
"-8 -m 1" was of course always way faster, and compressed 8 of the 12 files better than "-8". The differences in bitrate were quite small though, around 0.1-0.2kbps average.


This is really interesting.  My tests resulted in not quite as good compression.  It could be coincedence.  More testing would be great.  The order estimate method is a pretty crude 1st attempt at an idea I got from a paper which dealt with speech compression.  It suggested a method using the highest order reflection coefficient above a certain threshold.  I've found that the method varies widely depending on the type of audio...I imagine it works well for speech, but would need some tweaking for other types of signals.  I had once tried to do some testing using gnuplot output, but my statistical analysis skills aren't really advanced enough to know what to look for...so I have left it at a simple threshold for now.

edit: I also want to note that I could almost always pick out the best order or close to it using visual output from gnuplot, but I could never translate that into a mathematical analysis that worked well.  There is definitely potential there though.

-Justin
  • Last Edit: 17 October, 2006, 09:53:19 PM by Justin Ruggles