Skip to main content

Topic: New FLAC encoder (Read 290747 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • jcoalson
  • [*][*][*][*][*]
  • Developer
New FLAC encoder
Reply #50
I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter.  so far it works but it will not be ready for the next release.

I suppose the pseudo-inverse approach (through SVD) is even slower (but more accurate) than the Cholesky approach.

yep, you're right.  I'm mainly interested in establishing the lower bound first to see if other solvers would even be practical.

as for compression improvement, I don't have good numbers yet but it's not dramatic.  the autocorrelation formulation is surpisingly good with the proper window.

Josh

  • HbG
  • [*][*][*][*]
New FLAC encoder
Reply #51
Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.
 
  I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.
 
 
Veni Vidi Vorbis.

New FLAC encoder
Reply #52
Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.
 
  I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.
 
 

Sorry about that.  The Flake development is in a lull right now since I'm working feverishly on the Aften AC-3 encoder.  I'll run valgrind on Flake and see if I can fix that leak.
-Justin

  • HbG
  • [*][*][*][*]
New FLAC encoder
Reply #53
Thanks in advance!
Veni Vidi Vorbis.

New FLAC encoder
Reply #54

Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.
 
  I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.
 
 

Sorry about that.  The Flake development is in a lull right now since I'm working feverishly on the Aften AC-3 encoder.  I'll run valgrind on Flake and see if I can fix that leak.
-Justin


Here is a new bug-fix release to correct the memory leaks.  This download also includes some pre-compiled binaries for Linux and Windows.

http://jbr.homelinux.org/flake/flake-07.tar.gz

-Justin

  • HbG
  • [*][*][*][*]
New FLAC encoder
Reply #55
That fixed it, thanks.

But the speed's now dropped from 29.40x to about 18.5, still much faster than flac (11.9x) but not nearly as impressive.

Only tested the windows binary, the output is correct.
Veni Vidi Vorbis.

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #56
I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).

Download:
http://www.thefrontend.net/flake/flake-0.7-win32.rar
Tests:
http://www.thefrontend.net/flake/TESTS-0.7-win32.zip
Old builds:
http://www.thefrontend.net/flake/

wiso

New FLAC encoder
Reply #57
That fixed it, thanks.

But the speed's now dropped from 29.40x to about 18.5, still much faster than flac (11.9x) but not nearly as impressive.

Only tested the windows binary, the output is correct.

Okay, well that has convinced me to take a leap and release a more up-to-date version of Flake.  I'm not sure if the windows binaries will still be slower than before since I was not the one who created the binary for version 06.  All I can say is that it's faster for me on Linux, and I hope it is on Windows as well.

http://jbr.homelinux.org/flake/

It still doesn't have all the features in the FFmpeg encoder, but I'll get to that eventually.

Enjoy.
-Justin

I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).

Sorry to do two releases back-to-back.  I've had the 0.08 version sitting around and had wanted to tweak it some more before release, but I realized that might be a while, so I went ahead and released it.  I'll probably create a Sourceforge page so that I can do SVN updates and not have to release so often.

Also, my versioning has changed.  I was using "pre-release 01,02,...07".  Now I've switched to more normal versioning.  The latest is version 0.08.

Thanks!
-Justin

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #58

I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).

Sorry to do two releases back-to-back.  I've had the 0.08 version sitting around and had wanted to tweak it some more before release, but I realized that might be a while, so I went ahead and released it.  I'll probably create a Sourceforge page so that I can do SVN updates and not have to release so often.

Also, my versioning has changed.  I was using "pre-release 01,02,...07".  Now I've switched to more normal versioning.  The latest is version 0.08.

Thanks!
-Justin


This is no problem for me. The progress is there and this is more important.

wiso

  • HbG
  • [*][*][*][*]
New FLAC encoder
Reply #59
Oh yeah, wisodev's .07 build clocks in at about 29.1x for me with no memory leaks. Slightly slower than the leaky .06'es, but a major improvement nevertheless. Compression is the same. I should mention that i made some slight changes to my system and a reinstall, so there may be inaccuracies with my old benchmark results, but these should be fairly small, probably no more than .1x.

Pipe support still seems disfunctional, i get the following with both wisodev's .07 and justin's original fixed .06:
Code: [Select]
D:\MP3\Pearl Jam\Pearl Jam - Ten>type test.wav |flake -8 - test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 23965116

blocksize: 4608
max order: 12
order method: 4-level
stereo method: search

| bytes: 4448

The process tried to write to a nonexistent pipe.


0.08's windows binaries do not work for me. It reads the files but never starts encoding, just throws me back to the commandline, like this:
Code: [Select]
D:\MP3\Pearl Jam\Pearl Jam - Ten>flake-win32-i586.exe test.wav test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 23965116

D:\MP3\Pearl Jam\Pearl Jam - Ten>

All three builds do it. It does parse options and respond to -h, it creates a 0 byte output file.
Also, is there any chance of a changelog?

Like wisodev says, there is improvement and that is most important. Flake has a lot of potential, keep up the good work!
Veni Vidi Vorbis.

New FLAC encoder
Reply #60
Oh yeah, wisodev's .07 build clocks in at about 29.1x for me with no memory leaks. Slightly slower than the leaky .06'es, but a major improvement nevertheless. Compression is the same. I should mention that i made some slight changes to my system and a reinstall, so there may be inaccuracies with my old benchmark results, but these should be fairly small, probably no more than .1x.

Pipe support still seems disfunctional, i get the following with both wisodev's .07 and justin's original fixed .06:

Hmm...I'll try to fix that after I get the Sourceforge page up and running.  I'm hoping for approval tomorrow.

Quote
0.08's windows binaries do not work for me. It reads the files but never starts encoding, just throws me back to the commandline, like this:
...
All three builds do it. It does parse options and respond to -h, it creates a 0 byte output file.
Also, is there any chance of a changelog?

Now that is strange.  If I have a chance I'll install Wine so I can do some debugging.  I'm guessing it's related to configure & Makefile.

Ok, I'll create a changelog.  Having the source in an svn repository will help document development as well since there will be commit messages.

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #61
He, I have found too that 0.08 windows binarys are not working (all of them) even my builds are not working.

When it starts it throws stack overflow error on my machine (Windows XP)

But I have found the source of this problem.

in file encode.c in function
Code: [Select]
flake_encode_init(FlakeContext *s)

there was changed since version 07 way of handling context (FlacEncodeContext ctx_local;) memory allocation
and this change to local variable is causing under windows stack overflow
Code: [Select]
    FlacEncodeContext ctx_local;
    FlacEncodeContext *ctx;
    int i, header_len;

    if(s == NULL) {
        return -1;
    }

    ctx = &ctx_local;


I have changed to method used in 07 (allocating memory at the beginning of the function) and it works OK and do not throws any errors.

wiso

  • sundance
  • [*][*][*]
New FLAC encoder
Reply #62
Justin,

would you consider adding wisodev's "patch" to flake.c that he did in v06?
Or is there any catch I can't see?

Code: [Select]
    if(!strncmp(opts.infile, "-", 2)) {
#ifdef _WIN32 // win32 pipes patch by wiso
        _setmode(_fileno(stdin), _O_BINARY);
#endif
        ifp = stdin;
    } else {
        ifp = fopen(opts.infile, "rb");
        if(!ifp) {
            fprintf(stderr, "error opening input file: %s\n", opts.infile);
            return 1;
        }
    }
    if(!strncmp(opts.outfile, "-", 2)) {
#ifdef _WIN32 // win32 pipes patch by wiso
        _setmode(_fileno(stdout), _O_BINARY);
#endif
         ofp = stdout;
    } else {
        ofp = fopen(opts.outfile, "wb");
        if(!ofp) {
            fprintf(stderr, "error opening output file: %s\n", opts.outfile);
            return 1;
        }
    }


It would help a lot to have piping work in WIN32...

Thanks a lot!

.sundance.
  • Last Edit: 11 September, 2006, 05:01:58 PM by sundance

New FLAC encoder
Reply #63
would you consider adding wiso's "patch" to flake.c:

Yeah, I'll apply both of wiso's fixes and put them into SVN as soon as Sourceforge approves the project.  I'll also do a bit of code cleanup and documentation before doing the initial import and then jump up to version 0.10 as the 1st Sourceforge release.

-Justin

  • sundance
  • [*][*][*]
New FLAC encoder
Reply #64
Justin,

that's really good news.
Too bad that there's already a project called FLAKE registered at SF: http://sourceforge.net/projects/flake

.sundance.

New FLAC encoder
Reply #65
Justin,

that's really good news.
Too bad that there's already a project called FLAKE registered at SF: http://sourceforge.net/projects/flake

.sundance.

I know!  I guess I should have looked around first...  I had no idea until I tried to register.  Mine doesn't use all caps though.  The "unix name" will be flake-enc.  Still, I don't know if that will be grounds for rejection?  I may have to rename it, which would really suck!

-Justin

New FLAC encoder
Reply #66
Whew...it was accepted.
http://sourceforge.net/projects/flake-enc/

I'll work on getting everything uploaded and active this afternoon and evening.

  • jcoalson
  • [*][*][*][*][*]
  • Developer
New FLAC encoder
Reply #67
"flache" (High-Efficiency? painful?) is not taken...

New FLAC encoder
Reply #68
Whew...it was accepted.
http://sourceforge.net/projects/flake-enc/

I'll work on getting everything uploaded and active this afternoon and evening.

The SVN repository is up.
Version 0.10 has been released.

For now I have built binaries for Linux, optimized for i586, i686, or k8.  Also I included a couple (probably still slow) cross-compiled binaries for Windows, optimized for i586 or i686.  If wiso decides to be very kind again and builds a faster Windows binary, I would be happy to put it on Sourceforge as well.

Thanks,
Justin

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #69
Quote
For now I have built binaries for Linux, optimized for i586, i686, or k8.  Also I included a couple (probably still slow) cross-compiled binaries for Windows, optimized for i586 or i686.  If wiso decides to be very kind again and builds a faster Windows binary, I would be happy to put it on Sourceforge as well.

Thanks,
Justin

I have downloaded the sources from sourceforge and I will upload binarys as soon as I come back home from work.

  • sundance
  • [*][*][*]
New FLAC encoder
Reply #70
What do you think about adding some version information to the console output like
Code: [Select]
Flake: FLAC audio encoder version 0.10
(c) 2006  Justin Ruggles
to be able to identify which version of flake you're testing more easier.
Great you already included that version information as a vendor string to the flac header!

.sundance.
  • Last Edit: 12 September, 2006, 09:04:20 AM by sundance

  • krmathis
  • [*][*][*][*][*]
New FLAC encoder
Reply #71
Justin. I pulled revision 2 from SVN, and tried to build this on Mac OS 10.47 PowerPC/ GCC 4.0.1 (Xcode Tools 2.4). But it errors out like this:
Quote
$ ./configure
source path      /Users/krmathis/Downloads/flake-enc
C compiler      cc
make            make
CPU              powerpc (generic)
big-endian      yes
inttypes.h      yes
AltiVec enabled  yes
debug symbols    yes
strip symbols    yes
optimize        yes
Creating config.mak and config.h...
$ make
make -C libflake all

** Snip **

cc -O3  -no-cpp-precomp -pipe -force_cpusubtype_ALL -Wno-sign-compare -fomit-frame-pointer -faltivec -g -Wdeclaration-after-statement -Wall -Wno-switch -mdynamic-no-pic -DHAVE_CONFIG_H -I. -I.. -I/Users/krmathis/Downloads/flake-enc/libflake -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -c -o wav.o wav.c
cc -L../libflake -Wl,-dynamic,-search_paths_first  -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2


I successfully compiled flake 0.7 two days ago. But the build system have changed since then (from "make" to "./configure && make")

Any idea whats wrong?

  • wisodev
  • [*][*][*]
  • Developer
New FLAC encoder
Reply #72
FLAKE 0.10 Win32 Build and Sources

Download binarys:
http://www.thefrontend.net/flake/flake-0.10-win32-bin.rar
Download sources:
http://www.thefrontend.net/flake/flake-0.10-win32-src.rar

Alternative ZIP download:

Download binarys:
http://www.thefrontend.net/flake/flake-0.10-win32-bin.zip
Download sources:
http://www.thefrontend.net/flake/flake-0.10-win32-src.zip

I have tested my build and it looks like encoding speed has increased , this is great news!

Test results (using my build, sf.net build i686 and flac 1.1.2 build):
Code: [Select]
TEST FILE: 44100Hz, 16bit, Stereo, 431 916 620 bytes (13 Tracks)

ENCODE_FLAKE_INTEL [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

Flake: FLAC audio encoder
© 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 107972676

blocksize: 4608
max order: 12
order method: 4-level
stereo method: mid-side

progress: 100% | ratio: 0.741 | bitrate: 1045.2 kbps | bytes: 319851517


Kernel Time  =    3.328 = 00:00:03.328 =  3%
User Time    =    93.218 = 00:01:33.218 =  89%
Process Time =    96.546 = 00:01:36.546 =  92%
Global Time  =  104.172 = 00:01:44.172 = 100%
ENCODE_FLAKE_i686 [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

Flake: FLAC audio encoder
© 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 107972676

blocksize: 4608
max order: 12
order method: 4-level
stereo method: mid-side

progress: 100% | ratio: 0.741 | bitrate: 1045.2 kbps | bytes: 319851517


Kernel Time  =    3.078 = 00:00:03.078 =  2%
User Time    =  132.953 = 00:02:12.953 =  94%
Process Time =  136.031 = 00:02:16.031 =  96%
Global Time  =  140.671 = 00:02:20.671 = 100%
ENCODE_FLAC_1_1_2 [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

options: -P 4096 -b 4608 -m -l 12 -e -q 0 -r 0,6
Catamenia - [1999] - Morning Crimson (HQ).wav: wrote 321356528 bytes, ratio=0,74
4

Kernel Time  =    3.828 = 00:00:03.828 =  1%
User Time    =  206.640 = 00:03:26.640 =  96%
Process Time =  210.468 = 00:03:30.468 =  97%
Global Time  =  214.891 = 00:03:34.891 = 100%
DECODE_FROM_FLAKE_INTEL [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

t_INTEL.flac: done

Kernel Time  =    3.328 = 00:00:03.328 =  7%
User Time    =    22.703 = 00:00:22.703 =  53%
Process Time =    26.031 = 00:00:26.031 =  61%
Global Time  =    42.188 = 00:00:42.188 = 100%
DECODE_FROM_i686 [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

t_i686.flac: done

Kernel Time  =    3.140 = 00:00:03.140 =  7%
User Time    =    23.078 = 00:00:23.078 =  56%
Process Time =    26.218 = 00:00:26.218 =  63%
Global Time  =    41.094 = 00:00:41.094 = 100%
DECODE_FROM_FLAC_1_1_2 [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

t_FLAC.flac: done

Kernel Time  =    3.015 = 00:00:03.015 =  7%
User Time    =    22.953 = 00:00:22.953 =  56%
Process Time =    25.968 = 00:00:25.968 =  63%
Global Time  =    40.765 = 00:00:40.765 = 100%

---

FILE SIZE

t_FLAC.flac 321 356 528
t_i686.flac 319 851 517
t_INTEL.flac 319 851 517

  • lexor
  • [*][*][*]
New FLAC encoder
Reply #73
Code: [Select]
 
FILE            SIZE

t_FLAC.flac        321 356 528
t_i686.flac        319 851 517
t_INTEL.flac        319 851 517

wisodev, what's up with the second column?
  • Last Edit: 12 September, 2006, 03:21:06 PM by lexor
The Plan Within Plans

  • HbG
  • [*][*][*][*]
New FLAC encoder
Reply #74
My run went at 37.00x That's more than three times faster than flac at -8 and 7.5x up from 0.7. Memory usage dropped a little, and no leaks.

Pipe support works, i can use it as external encoder in foobar2000 without problems.
What's even better, it seems foobar can automatically emebed a .cue file in the tag, only it doesn't preserve indexes, and it doesn't embed it in the actual flac way. Can't win 'em all.

Still, this is an awesome achievement. I can't believe the reference encoder had so much room for improvement!

This needs to get on rarewares.
  • Last Edit: 12 September, 2006, 08:21:02 PM by HbG
Veni Vidi Vorbis.