HydrogenAudio

Lossless Audio Compression => FLAC => Topic started by: Nayru on 2006-05-26 13:29:22

Title: New FLAC encoder
Post by: Nayru on 2006-05-26 13:29:22
A new FLAC encoder has been developed, which is proposed for inclusion in ffmpeg.  The author claims that in high-compression mode, it can compress certain files better than libFLAC.

http://mplayerhq.hu/pipermail/ffmpeg-devel...May/011385.html (http://mplayerhq.hu/pipermail/ffmpeg-devel/2006-May/011385.html)
Title: New FLAC encoder
Post by: kurtnoise on 2006-05-26 14:40:08
For those who want to test it, I've made (http://www.hydrogenaudio.org/forums/index.php?showtopic=45016#) a compile for win32 plateforms.


booo...it crashes. 
Title: New FLAC encoder
Post by: jcoalson on 2006-05-26 21:57:09
it seems like a nice clean interface.  some notes:

1. at the higher compression settings it is using large block sizes (16k for 44.1kHz) and high lpc order (32) which increase the decoding burden with little improvement (at least in my tests).  (e.g. I'm not sure the squeezebox supports 16k blocks due to buffer requirements.)  also, performance with larger blocksizes in more dependent on the audio characteristics so his results might be biased by his sample.

2. it has an interesting way of selecting the stereo decorrelation method which is faster than the reference encoder; whether it's better on not needs some testing.  same for fixed/lpc subframe selection.

3. it's guessing at the max frame size without supporting verbatim encoding, which might explain it crashing

4. the lpc analysis stage looks just like the reference encoder.

so I think any compression gain is just due to the larger blocksize and high lpc order, which you can do with the reference encoder (though I wouldn't encourage that for general use).

Josh
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-03 06:14:29
Hi,
I am the person writing the encoder mentioned above.  It is in an extremely experimental phase at the moment.  I am fairly new to audio encoding, so most of what I am doing is just learning by trying things out and reading as much as I can.  I do enjoy it though.  I want to say that I greatly admire what Josh has done with the flac format and reference software.  Its openness and good documentation are a rarity and a pleasure.  I am writing an encoder for FFmpeg because one of the goals of that project is to write its own decoders and encoders so as not to rely on too many external libraries, and because of licensing (which I don't think is really much of an issue in the  case of FLAC).  Also, I am doing it for the shear pleasure of learning something new.  My claims of better encoding are more due to excitement at the results than anything else.  Beating the reference encoder is not my goal.  It's just a good motivator, both for the excitement when I succeed and for the challenge when I can't seem to get anywhere close.

it seems like a nice clean interface.  some notes:

1. at the higher compression settings it is using large block sizes (16k for 44.1kHz) and high lpc order (32) which increase the decoding burden with little improvement (at least in my tests).  (e.g. I'm not sure the squeezebox supports 16k blocks due to buffer requirements.)  also, performance with larger blocksizes in more dependent on the audio characteristics so his results might be biased by his sample.

That is true.  I have found some odd results in my tests though.  Many times I have seen my encoder do much better on a particular clip with, say, max lpc order of 32, but do not see the same benefit on the same clip with the reference encoder.  I couldn't say why or how though.  I am mostly just trying to test anything I can and observing what settings work best in which situations.  You do have a point about buffer requirements.  I had to increase the SDL audio buffer size in ffplay to get the high blocksize clips to play without skipping.

The issue with block sizes and different audio characteristics is one I am still struggling with.  It seems to be one of the biggest drawbacks to really good compression quality.  Variable blocksize would be ideal, but I know the compatiblity issues.  Given the limitations, one of my objectives is to have good compatible modes, but also to provide more options for better compression.  The reference encoder seems to go for the middle ground in many cases.  This has its benefits, but personally, I like more options.

Quote
2. it has an interesting way of selecting the stereo decorrelation method which is faster than the reference encoder; whether it's better on not needs some testing.  same for fixed/lpc subframe selection.

Yeah, the stereo decorrelation selection was just a guess that turned out to work okay.  I haven't looked closely enough to see how the reference encoder does it.  As for fixed/lpc subframe selection...I can't really remember how I was doing that on May 26  .  I have had too many abandoned approaches to keep up...  Right now I am using fixed mode (and mostly just order 2) only in the fastest encoding setting and for small frames (usually just the last frame).

Quote
3. it's guessing at the max frame size without supporting verbatim encoding, which might explain it crashing

I have no idea about the crashing on windows, as I use Linux.  As for verbatim encoding, I actually started the program as verbatim only just to get the bitstream right.  I later removed it.  I probably should add it back in as a backup mechanism for audio that really just won't compress...a rarity, but it would save the guessing you mentioned.  For now, I have at least added a warning & skip frames that are too big.  It's not ideal, nor lossless, but at least it gives a warning...

Quote
4. the lpc analysis stage looks just like the reference encoder.

Yeah, it pretty much is.  I have modified the autocorrelation slightly and implemented my own quantizer.  The levinson-durbin function is the same.  I experimented some with other ways of estimating the best predictor order, but it was not as good as the reference encoder.  When I coupled the 2 together and selected the best by counting the bits I got slightly better encoding...that might have just been chance though  I could have probably selected an order at random and done the same thing.  I guess I'll have to keep researching.  It's not that I think the reference encoder has a bad solution...I just like a challenge.  I recently added, and am currently experimenting with, window functions before autocorrelation.  Yesterday I was pointed to a Hydrogenaudio thread on how this is being added to the reference encoder as well.  Very exciting stuff!

Quote
so I think any compression gain is just due to the larger blocksize and high lpc order, which you can do with the reference encoder (though I wouldn't encourage that for general use).

Blocksize I understand, but what's wrong with a higher LPC order?  True, it isn't ideal for every situation, but it does help in many cases.  I haven't noticed much of an impact on decoding, and I have a quite slow computer.

Anyway, I just thought I would respond.  I am new to Hydrogenaudio, but since I seem to have thrown myself into the fray I might as well join in on the discussion.

-Justin
Title: New FLAC encoder
Post by: kurtnoise on 2006-06-03 16:49:49
Hi Justin,

Thank you very much for this work (fork ?)...

I've made a benchmarck on my AMD64 3200+. I've tested the last flake 0.2 and flac from the last cvs entry with a wav sample of 1h11m55s (mostly electro and pop rock).

Here are my results:

1°) Flake 0.2 compiled with gcc 4.1.0 & CFLAGS = -g -O3 -Wall -pedantic -std=c99 -march=k8
Code: [Select]
|  level  |   Real Time   |       Bytes        |  Ratio   |  kbps |
|    0     |  2m11.511s  |  530 951 235  |  0,697  | 984   |
|    1     |  2m34.031s  |  513 566 686  |  0.675  | 952   |
|    2     |  2m34.206s  |  499 679 481  |  0,656  | 926   |
|    3     |  2m36.720s  |  491 472 721  |  0,645  | 911   |
|    4     |  2m35.962s  |  488 370 918  |  0,641  | 905   |
|    5     |  2m50.350s  |  487 493 404  |  0,640  | 904   |
|    6     |  2m51.587s  |  487 565 406  |  0,640  | 904   |
|    7     |  2m50.688s  |  490 601 577  |  0,644  | 910   |
|    8     |  4m13.558s  |  490 820 897  |  0,644  | 910   |



2°) FLAC CVS 20060603 compiled with gcc 4.1.0
Code: [Select]
|  level  |   Real Time   |       Bytes        |  Ratio   |  kbps |
|    0     |  1m16.089s  |  531 245 719  |  0,698  | 984   |
|    1     |  1m20.170s  |  510 802 749  |  0.671  | 947   |
|    2     |  1m25.361s  |  505 186 083  |  0,663  | 940   |
|    3     |  1m32.409s  |  501 361 052  |  0,652  | 933   |
|    4     |  1m35.588s  |  489 589 293  |  0,643  | 908   |
|    5     |  1m50.881s  |  488 073 198  |  0,641  | 905   |
|    6     |  2m04.660s  |  487 691 205  |  0,641  | 904   |
|    7     |  6m59.237s  |  486 851 452  |  0,640  | 903   |
|    8     |  9m00.266s  |  485 704 730  |  0,638  | 901   |



Now, I've a question : why flake crashes during encoding on win32 plateform ? In fact, compilation is fine but running fails. Any idea ?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-04 04:00:18
Thank you very much for this work (fork ?)...

You're welcome.  Fork?  Well, I suppose that if I get the LPC order selection code replaced with my own then I can claim it to be nearly non-derived .  The Levinson-Durbin LPC generation code is nearly identical to that in several other programs I've seen.  I copied the UTF-8 code basically because I'm not 100% clear on how that encoding works.  I suppose I should separate these 3 or 4 functions into a single file with proper copyright notice and attribution...or at least put a small notice above each and include both a COPYING.libFLAC file and COPYING.lgpl file.  It wasn't really a fork though.  I basically started from scratch using the specs and some reused FFmpeg code.  I even tried my own UTF-8 function working backwards from the decoding in FFmpeg...I could never get it to work though, so I copied in that function.  When adding non-fixed LPC I tried my hand at that as well, but fell short.  So it's more of an independent implementation with crutches.  The structure is my own, but the puzzle had to be filled-in in a few places.

Quote
I've made a benchmarck on my AMD64 3200+. I've tested the last flake 0.2 and flac from the last cvs entry with a wav sample of 1h11m55s (mostly electro and pop rock).

Here are my results:

1°) Flake 0.2 compiled with gcc 4.1.0 & CFLAGS = -g -O3 -Wall -pedantic -std=c99 -march=k8

|  level  |  Real Time  |     Bytes      |  Ratio  |  kbps |
[...]
|   5    |  2m50.350s  |  487 493 404  |  0,640  | 904  |

2°) FLAC CVS 20060603 compiled with gcc 4.1.0

|  level  |  Real Time  |     Bytes      |  Ratio  |  kbps |
[...]
|   8    |  9m00.266s  |  485 704 730  |  0,638  | 901  |

I also noticed similar results (not quite as good as libFLAC) on samples where flake did the best with smaller block sizes (i.e. level 5, which is blocksize 2048), but that the reference encoder did best at level 8 (blocksize 4608).  I am still trying to wrap my head around this issue.  It is possibly due to whatever libFLAC does in "exhaustive" mode?  Or maybe my version of stereo decorrelation works well only on audio with certain characteristics?  Testing, testing, and more testing.  I'm working on my own horribly slow "brute-force" mode to help me out in my tests.

Quote
Now, I've a question : why flake crashes during encoding on win32 plateform ? In fact, compilation is fine but running fails. Any idea ?

I really have no clue here.  What does the crash look like?  Is it a segfault?  Any gdb info?  If I had Windows I would test it myself.  If MinGW is being used, maybe it needs "--enable-memalign-hack"...?  It has been several..5..6.. years since I have done any gcc compilation in Windows.

-Justin
Title: New FLAC encoder
Post by: kurtnoise on 2006-06-04 07:55:11
I really have no clue here.  What does the crash look like?  Is it a segfault?  Any gdb info?  If I had Windows I would test it myself.  If MinGW is being used, maybe it needs "--enable-memalign-hack"...?  It has been several..5..6.. years since I have done any gcc compilation in Windows.

Through wine, here is the debug :

Quote
lionel@kubuntu:~/flake$ wine ./flake.exe -2 /home/lionel/wav/input.wav /home/lionel/wav/output.flac
Flake: FLAC audio encoder
Signed 16-bit 44100 Hz stereo
samples: 190288560
wine: Unhandled page fault on write access to 0x7f9a0dd4 at address 0x405703 (thread 0009), starting debugger...
WineDbg starting on pid 0x8
Unhandled exception: page fault on write access to 0x7f9a0dd4 in 32-bit code (0x00405703).
Register dump:
CS:0073 SS:007b DS:007b ES:007b FS:1007 GS:0033
EIP:00405703 ESP:7fbafdcc EBP:7fbafdd8 EFLAGS:00210206(  - 00      - RIP1)
EAX:001f4514 EBX:ffffffff ECX:7f9a0dd4 EDX:7fd085e0
ESI:00401210 EDI:7ffdf760
Stack dump:
0x7fbafdcc:  7fcf0020 004018f6 ffffffff 7fbafeb8
0x7fbafddc:  00403306 7fd085e0 7fbafe60 0b5792b0
0x7fbafdec:  00402fd4 7fcf0020 00000000 7fcf0020
0x7fbafdfc:  00402fcf 7fd08c38 7fd0945c 7fbafe28
0x7fbafe0c:  00000010 7fcf0000 00000000 7fd084b0
0x7fbafe1c:  00000000 7fcf0000 7fc946dc 7fbafe48
0200: sel=1007 base=7fe6c000 limit=00001fff 32-bit rw-
Backtrace:
=>1 0x00405703 in flake (+0x5703) (0x00405703)
  2 0x00403306 in flake (+0x3306) (0x00403306)
  3 0x004011d9 __mingw_CRTStartup+0xc9 [/build/buildd/mingw32-runtime-3.8/build_dir/src/mingw-runtime-3.8/crt1.c:226] in flake (0x004011d9)
  4 0x00401223 in flake (+0x1223) (0x00401223)
  5 0x7fc5b311 in kernel32 (+0x4b311) (0x7fc5b311)
  6 0xb7ed8ddb wine_switch_to_stack+0x17 in libwine.so.1 (0xb7ed8ddb)
0x00405703: orl $0,0x0(%ecx)
Modules:
Module  Address                Debug info      Name (19 modules)
PE      0x00400000-00438000    DIA            flake
ELF    0x7bf00000-7bf03000    Deferred        <wine-loader>
ELF    0x7f93f000-7f9a0000    Deferred        msvcrt<elf>
  \-PE  0x7f950000-7f9a0000    \              msvcrt
ELF    0x7fbee000-7fcf0000    Export          kernel32<elf>
  \-PE  0x7fc10000-7fcf0000    \              kernel32
ELF    0x7fe07000-7fe11000    Deferred        libnss_files.so.2
ELF    0x7fe11000-7fe26000    Deferred        libnsl.so.1
ELF    0x7fe3a000-7fe5c000    Deferred        libm.so.6
ELF    0x7fe63000-7fe6c000    Deferred        libnss_nis.so.2
ELF    0x7fe70000-7ff66000    Deferred        libwine_unicode.so.1
ELF    0x7ff66000-7ffe0000    Deferred        ntdll<elf>
  \-PE  0x7ff80000-7ffe0000    \              ntdll
ELF    0xb7d71000-b7d7a000    Deferred        libnss_compat.so.2
ELF    0xb7d7b000-b7d7e000    Deferred        libdl.so.2
ELF    0xb7d7e000-b7ead000    Deferred        libc.so.6
ELF    0xb7ead000-b7ebf000    Deferred        libpthread.so.0
ELF    0xb7ed4000-b7eee000    Export          libwine.so.1
ELF    0xb7ef1000-b7f07000    Deferred        ld-linux.so.2
Threads:
process  tid      prio (all id:s are in hex)
00000008 (D) Z:\home\lionel\flake\flake.exe
        00000009    0 <==

Hope this helps....


About fork. It was just a joking.  Keep up the good work.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-04 08:27:10
Through wine, here is the debug :

lionel@kubuntu:~/flake$ wine ./flake.exe -2 /home/lionel/wav/input.wav /home/lionel/wav/output.flac
Flake: FLAC audio encoder
Signed 16-bit 44100 Hz stereo
samples: 190288560
wine: Unhandled page fault on write access to 0x7f9a0dd4 at address 0x405703 (thread 0009), starting debugger...
WineDbg starting on pid 0x8
Unhandled exception: page fault on write access to 0x7f9a0dd4 in 32-bit code (0x00405703).

It's kinda hard to tell without more symbol names, but my thoughts still go to memory alignment in the structs.  I tried rearranging the structs so that non-DWORD values are always at the end.  Maybe that will help?  If that's even the issue.  It seems to be happening in the small section in flake.c around flake_encode_init.  This is when the flac encoder structures are first accessed.  Some of those values are pointers, so if the alignment is off, the pointers could end up pointing to unaccessible memory, which could cause a page fault.

-Justin
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-14 08:56:53
Hi,
I have made a small webpage to post updated versions and various test results from my FLAC encoder.  The page will be updated frequently.  I have included a Linux i386 binary along with the source code, and I hope I can get some help providing a Windows binary.  Hopefully, the more people I can get to test it out, the more I can improve it.

http://home.earthlink.net/~jruggle/flake/ (http://home.earthlink.net/~jruggle/flake/)

Please note that this pre-release is not intended for general use, only testing.  It does not currently calculate MD5 sums, do any tagging, or add any padding.  The wav parser is not very robust either.

Thanks,
-Justin
Title: New FLAC encoder
Post by: kurtnoise on 2006-06-14 09:39:44
Hi Justin,

Great stuff...Now, flake 0.4 works fine too on win32 os.  I uploaded a windows build here (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=35&t=45016&st=0#entry396129).
Title: New FLAC encoder
Post by: krmathis on 2006-06-14 18:42:16
flake dont seem to work correctly on Mac OS X (PowerPC).

The source compiles fine, and the flake executable does it work to my source .wav file.
But it dont compress the file at all (actually it is larger than the original), and the output is just noice.

flake 0.4:
Code: [Select]
Flake: FLAC audio encoder
© 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 28563276

blocksize: 4608
max order: 8
order method: estimate
stereo method: estimate

progress: 100% | ratio: 1.025 | bitrate: 1447.0 kbps | bytes: 117148084
flac 1.1.2:
Code: [Select]
flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

options: -P 4096 -b 4608 -m -l 8 -q 0 -r 3,3
wrote 78408059 bytes, ratio=0.686
I have tried multiple input files, from different sources, but the same happends every time.
Is this a known problem?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-14 18:49:18
flake dont seem to work correctly on Mac OS X (PowerPC).

The source compiles fine, and the flake executable does it work to my source .wav file.
But it dont compress the file at all (actually it is larger than the original), and the output is just noice.

...

I have tried multiple input files, from different sources, but the same happends every time.
Is this a known problem?


Dang. I thought I had fixed the endian issue.  I'll keep trying though.  It's hard to test when I don't have a big-endian machine.
-Justin
Title: New FLAC encoder
Post by: krmathis on 2006-06-14 19:34:22
Hope you nail this, cause this looks like an interresting application.

If you need a test dummy, just give me a hint.
Title: New FLAC encoder
Post by: jcoalson on 2006-06-14 19:46:45
Yesterday I was pointed to a Hydrogenaudio thread on how this is being added to the reference encoder as well.  Very exciting stuff!

btw in the FLAC code in CVS there is a src/libFLAC/window.c that may save you some time.

Blocksize I understand, but what's wrong with a higher LPC order?  True, it isn't ideal for every situation, but it does help in many cases.  I haven't noticed much of an impact on decoding, and I have a quite slow computer.

the number of multiply-adds in the FIR filter equals the order.  flac -8 uses max 12, so order 32 requires 20 more multiplies+adds per sample than what is widely used.

Josh
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-14 20:33:53
Yesterday I was pointed to a Hydrogenaudio thread on how this is being added to the reference encoder as well.  Very exciting stuff!

btw in the FLAC code in CVS there is a src/libFLAC/window.c that may save you some time.

Thanks.  I did see that.  I chose to go with the Welch window.  It is simple to calculate and gives above-average results most of the time.

Quote
Blocksize I understand, but what's wrong with a higher LPC order?  True, it isn't ideal for every situation, but it does help in many cases.  I haven't noticed much of an impact on decoding, and I have a quite slow computer.

the number of multiply-adds in the FIR filter equals the order.  flac -8 uses max 12, so order 32 requires 20 more multiplies+adds per sample than what is widely used.

Once I started benchmarking the decoding, I saw what you were talking about.  Using max order 32 seems to increase decoding time about 30%.  I didn't notice until I actually timed it because I was using short clips.  So, I moved the higher block sizes and orders to extended levels 9-15 and modified 0-8 to use similar output parameters to the reference encoder.  The higher block sizes don't seem to do much now that I've changed the encoder quite a bit.  I'm strongly considering removing the preset levels for 16384 and maybe even 8192.

-Justin
Title: New FLAC encoder
Post by: ggf31416 on 2006-06-15 04:06:03
40.964.828 bytes 16-bit Stereo 44100Hz WAV 3:52 "La cosecha de mujeres"

Code: [Select]
FLAKE 0.4
Level | Ratio |  Bitrate    |  Time
 0    | 0.788 | 1112.6 kbps |  6.484
 1    | 0.747 | 1054.7 kbps |  7.328
 2    | 0.741 | 1045.1 kbps |  7.250
 3    | 0.733 | 1034.5 kbps |  7.468
 4    | 0.729 | 1029.2 kbps |  7.500
 5    | 0.726 | 1024.1 kbps |  7.906
 6    | 0.721 | 1017.5 kbps |  11.250
 7    | 0.721 | 1017.5 kbps |  12.718
 8    | 0.719 | 1015.0 kbps |  14.343
 9    | 0.719 | 1014.8 kbps |  29.046
10    | 0.719 | 1015.2 kbps |  20.671
11    | 0.719 | 1014.3 kbps | 120.562
12    | 0.720 | 1016.7 kbps |  21.093
13    | 0.720 | 1016.1 kbps | 119.875
14    | 0.722 | 1019.5 kbps |  21.703
15    | 0.722 | 1019.1 kbps | 120.828

FLAC 1.1.2 CVS
Level | Ratio |  Time
0    | 0,782 |  2.687
1    | 0,778 |  2.890
2    | 0,776 |  3.609
3    | 0,731 |  3.750
4    | 0,723 |  4.812
5    | 0,722 |  6.031
6    | 0,722 |  6.625
7    | 0,721 | 21.531
8    | 0,720 | 28.140
[/font]

In some samples a large block size can reduce the compression ratio.
WAV 111.266.108 bytes 16-bit Stereo 44100Hz  10:30 "La navidad de los pobres"

Code: [Select]
Parameters       | Ratio  |   Bitrate   |   Bytes   |  Time
FLAKE -8 -b 1152 |  0.645 |  909.7 kbps |  71722600 | 39.640
FLAKE -8 -b 2048 |  0.642 |  906.0 kbps |  71436942 | 40.031
FLAKE -8 -b 2304 |  0.642 |  906.2 kbps |  71446745 | 39.656
FLAKE -8 -b 4096 |  0.644 |  909.3 kbps |  71697454 | 39.609
FLAKE -8 -b 4608 |  0.645 |  910.6 kbps |  71796426 | 38.843
FLAKE -8 -b 8192 |  0.652 |  920.1 kbps |  72549094 | 37.812
FLAC CVS -8      |  0.647 |  ~913 kbps |  71971927 | 78.421
[/font]

I suggest testing several block sizes in the high levels, using a lower compression level. Then use the best value to compress the sample:
Example:

Code: [Select]
-b x,[n]

-b x Block size [16 - 65535] (default: 4608)
-b 1 Try -b 2048 and -b 4608
-b 2 Try -b 2048, -b 2304, -b 4096, -b 4608 and -b 8192.
-b 3 Try -b 192, -b 576, -b 1152, -b 2304, -b 4608, -b 256, -b 512, -b 1024, -b 2048, -b 4096, -b 8192 and -b 16384.

n Use compression level n for the tries. Default: 5 for -b 1, -b 2 or -b 3.

Compression Levels:
[1-9] Without changes
10 = -l 12 -o 4 -s 2 -b 1,5  Estimated Time: ~  45 (2* level 5 + level 9)
11 = -l 12 -o 4 -s 2 -b 2,5  Estimated Time: ~  69 (5*5+9)
12 = -l 32 -o 4 -s 2 -b 2,5  Estimated Time: ~ 160 (5*5+11)
13 = -l 32 -o 4 -s 2 -b 3,5  Estimated Time: ~ 215 (12*5+11)
14 = -l 32 -o 4 -s 2 -b 3,8  Estimated Time: ~ 293 (12*8+11)

All Times are Process Time (Timer 3.01). Celeron D325 2.53 Ghz under Windows XP SP2.
Estimates times are based on the first sample ("La cosecha de mujeres").[/font]
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-15 07:43:32
I suggest testing several block sizes in the high levels, using a lower compression level. Then use the best value to compress the sample:
Example:

-b x Block size [16 - 65535] (default: 4608)
-b 1 Try -b 2048 and -b 4608
-b 2 Try -b 2048, -b 2304, -b 4096, -b 4608 and -b 8192.
-b 3 Try -b 192, -b 576, -b 1152, -b 2304, -b 4608, -b 256, -b 512, -b 1024, -b 2048, -b 4096, -b 8192 and -b 16384.

Along those same lines, I am working on implementing block switching.  The resulting files would not be Subset compliant, but they would still decode with at least the reference decoder and with FFmpeg/MPlayer, which are the only decoders I have tested with at this point.  I have found that 8192 only rarely increases compression and 16384 almost never does.  I have now removed them as presets (pending pre-release 5) in favor of an 8-level order search.

The advantage of block switching would be with audio which compresses better at 2048 or 1024 blocksize.  What you're suggesting seems like it could be accomplished with a 2-pass encoding, which I guess I could try to implement as another option.

Thanks,
Justin
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-15 09:47:27
Hi,
Pre-release 5 is now out.

* I hope the PPC endian issue is fixed.  If not, please let me know.

* Added support for streamed input and output.  For example,
cat test.wav | flake - - | flac -d -o test2.wav -

* Added 8-bit input support.  24-bit input is also supported, but only converts to 16-bit for now, so it's not lossless.

* Changed up the extended level options.  8192 and 16384 block sizes have now been removed as preset options.  Also, I added an 8-level prediction order search (levels 9 & 11).

-Justin
Title: New FLAC encoder
Post by: krmathis on 2006-06-15 17:00:17
Pre-release 5 is now out.
* I hope the PPC endian issue is fixed.  If not, please let me know.

It seems to work fine now. The track encodes, and the resulting FLAC files play fine.
Thanks!

* now on to testing this application *
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-17 03:31:51
Hi all,

Sorry to keep posting updates so often.  The online version of my FLAC encoder is now pre-release 6.

Changes are mostly header-related.
* added MD5 support! yay!
* added padding in the header to make room for metadata
* added a placeholder for a future vendor tag

I am at a good stopping point for now.  My wishlist is 90% done.  I will finally start work on an FFmpeg patch to integrate the current version.

Also, it would be really great to see some other peoples' test results.  My computer is pretty darn slow, so I have only been able to test small files when doing everyday sort of benchmarking.  I would love to see how it performs, say, on a whole CD.

Thanks,
Justin
Title: New FLAC encoder
Post by: krmathis on 2006-06-17 08:45:19
Album: Toto - Legend (in a single file)
Playing time: 72m 35s
Original File Size: 768284896 bytes
Computer specs: Apple PowerBook G4, 1.5GHz, 1GB ram, 7200rpm HDD.
Encoders: flac 1.1.2 and flake 06

Code: [Select]
flac:

level   enc time     bytes       ratio     kbps      dec time
-----  ---------   --------      -----    ------    ---------
  0     1m 50s     518214304     0.675               1m 15s
  5     3m 34s     496252171     0.646               1m 14s
  8    13m 46s     494499692     0.644               1m 26s

flake:

level   enc time     bytes       ratio     kbps      dec time
-----  ---------   --------      -----    ------    ---------
  0     3m 21s     526551866     0.685     967.2     1m 21s
  5     6m 53s     495548928     0.645     910.3     1m 30s
  8    13m 37s     492004491     0.640     903.8     1m 14s

The compression rate is comparable to flac, but it encodes noticeable slower at -0 and -5.
Anyway, you have  done great work so far!

Download the Mac OS X univeral binary here: direct link (http://www.hydrogenaudio.org/forums/index.php?act=Attach&type=post&id=2294)
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-18 04:20:06
Album: Toto - Legend (in a single file)
Playing time: 72m 35s
Original File Size: 768284896 bytes
Computer specs: Apple PowerBook G4, 1.5GHz, 1GB ram, 7200rpm HDD.
Encoders: flac 1.1.2 and flake 06

Code: [Select]
flac:

level   enc time     bytes       ratio     kbps      dec time
-----  ---------   --------      -----    ------    ---------
  0     1m 50s     518214304     0.675               1m 15s
  5     3m 34s     496252171     0.646               1m 14s
  8    13m 46s     494499692     0.644               1m 26s

flake:

level   enc time     bytes       ratio     kbps      dec time
-----  ---------   --------      -----    ------    ---------
  0     3m 21s     526551866     0.685     967.2     1m 21s
  5     6m 53s     495548928     0.645     910.3     1m 30s
  8    13m 37s     492004491     0.640     903.8     1m 14s

The compression rate is comparable to flac, but it encodes noticeable slower at -0 and -5.
Anyway, you have  done great work so far!


Thank you for the feedback.
It looks like I have a long way to go to make level 5 comparable to the reference encoder on PPC.  At least level 8 is on par.

I finally let my system run for a while to test a whole album.  Here is what I got at level 8.

Smashing Pumpkins - Siamese Dream
playing time: 1h2m15.8s

flac: encode time=1h1m35.42s  ratio=0.583
flake: encode time=35m43.75s  ratio=0.582

I know...it really is about time for a major upgrade...I'm still holding out for 10 good years out of my '97 IBM.

Thanks,
Justin
Title: New FLAC encoder
Post by: krmathis on 2006-06-18 07:39:50
Thank you for the feedback.
It looks like I have a long way to go to make level 5 comparable to the reference encoder on PPC.  At least level 8 is on par.
You are welcome!
Looks like FLAC have optimized their code for PowerPC, while you of course have consentrated on i386 only.
I guess it would be hard for you to improve the performance on PowerPC as well, since you dont have access to one. But that might not be that important since Apple are moving their line to Intel, and at the current stage they only have two PowerPC computers left.

Quote
I finally let my system run for a while to test a whole album.  Here is what I got at level 8.

Smashing Pumpkins - Siamese Dream
playing time: 1h2m15.8s

flac: encode time=1h1m35.42s  ratio=0.583
flake: encode time=35m43.75s  ratio=0.582

I know...it really is about time for a major upgrade...I'm still holding out for 10 good years out of my '97 IBM.
Wow, you have cut the -8 encoding time on your i386 machine to half.

That you still use a 300MHz computer from '97 are just insane. My computer are only two (2) years old, and I am seriously looking for a replacement (probably next generation MacBook Pro)! 
Title: New FLAC encoder
Post by: miros on 2006-06-18 08:13:49
Here are my results for four albums lying around. CPU AMD K8 1.8GHz, 64-bit OS, everything cached in memory, so the harddisk wouldn't be a bottleneck. Included is also decoding speed of another decoder that is very sensitive to higher LPC order. Versions used: flake-06 and flac CVS.

Code: [Select]
garbage / beautiful garbage

lvl|  ratio       speed (x realtime)    |  ratio       speed (x realtime)  
   |           flake   flac -d  flacdec |            flac   flac -d  flacdec
0  |  0.713     52.1    202.9    418.1  |  0.727    135.5    210.0    424.3
1  |  0.678     50.7    184.7    391.6  |  0.703    137.6    205.9    408.5
2  |  0.678     50.6    184.8    391.2  |  0.701     90.4    206.4    416.7
3  |  0.670     50.0    179.7    382.4  |  0.694     90.9    179.5    385.5
4  |  0.668     51.1    176.0    373.5  |  0.668     77.6    170.7    351.5
5  |  0.666     46.5    170.4    352.9  |  0.667     58.5    172.3    354.6
6  |  0.664     31.2    169.0    344.0  |  0.666     43.9    172.0    355.2
7  |  0.664     27.9    170.8    348.9  |  0.665      9.7    172.8    359.0
8  |  0.661     24.1    163.6    309.1  |  0.663      7.2    168.4    329.1
9  |  0.661     18.4    164.2    311.9  |  
10 |  0.661     14.6    165.3    314.6  |  
11 |  0.657     11.8    110.0    118.7  |  
12 |  0.657      4.5    114.4    122.4  |  

u2 / the joshua tree

lvl|  ratio       speed (x realtime)    |  ratio       speed (x realtime)  
   |           flake   flac -d  flacdec |            flac   flac -d  flacdec
0  |  0.589     57.0    217.9    450.5  |  0.585    141.0    223.1    448.4
1  |  0.553     55.8    194.2    422.7  |  0.573    142.7    218.2    430.5
2  |  0.553     55.3    195.3    419.7  |  0.570     92.4    219.2    438.7
3  |  0.545     55.0    189.2    411.7  |  0.554     93.3    192.1    411.7
4  |  0.542     56.4    185.7    404.0  |  0.540     79.1    181.1    374.4
5  |  0.540     50.9    179.3    376.7  |  0.539     59.3    180.7    377.6
6  |  0.535     33.1    178.2    365.3  |  0.538     44.5    181.9    378.1
7  |  0.535     29.4    180.2    372.0  |  0.536      9.7    182.6    384.9
8  |  0.533     25.3    173.5    332.3  |  0.535      7.3    180.2    360.5
9  |  0.533     19.1    175.2    336.3  |  
10 |  0.533     15.0    175.2    337.8  |  
11 |  0.530     12.2    127.7    144.5  |  
12 |  0.530      4.5    131.3    147.8  |  

smashing pumpkins / gish

lvl|  ratio       speed (x realtime)    |  ratio       speed (x realtime)  
   |           flake   flac -d  flacdec |            flac   flac -d  flacdec
0  |  0.654     54.6    211.1    434.1  |  0.635    138.9    216.9    436.2
1  |  0.599     54.1    193.0    411.7  |  0.634    140.2    213.4    425.5
2  |  0.599     54.0    192.5    411.1  |  0.633     91.4    213.4    428.1
3  |  0.582     53.6    188.4    400.4  |  0.581     93.1    190.2    404.5
4  |  0.577     54.9    183.2    392.5  |  0.575     79.0    179.6    370.6
5  |  0.571     49.8    178.0    367.7  |  0.574     59.3    181.3    372.1
6  |  0.566     32.7    178.8    365.3  |  0.573     44.6    180.7    372.1
7  |  0.565     29.0    181.1    369.1  |  0.570      9.9    181.9    378.2
8  |  0.561     25.2    175.4    332.2  |  0.568      7.4    179.6    359.2
9  |  0.560     19.0    175.4    333.8  |  
10 |  0.560     15.0    176.2    336.2  |  
11 |  0.559     12.4    149.8    183.7  |  
12 |  0.559      4.6    155.9    190.2  |  

walk the line OST

lvl|  ratio       speed (x realtime)    |  ratio       speed (x realtime)  
   |           flake   flac -d  flacdec |            flac   flac -d  flacdec
0  |  0.628     54.7    217.1    445.7  |  0.654    137.9    220.3    441.0
1  |  0.608     52.9    193.6    410.6  |  0.621    141.0    216.5    425.3
2  |  0.607     51.9    194.8    409.3  |  0.618     91.8    219.3    440.2
3  |  0.602     52.2    190.5    402.7  |  0.637     91.9    191.5    406.6
4  |  0.602     53.7    186.4    397.6  |  0.603     78.5    183.9    380.1
5  |  0.601     48.7    180.3    377.2  |  0.602     59.0    185.4    387.1
6  |  0.600     31.8    176.9    357.3  |  0.601     44.2    185.7    386.5
7  |  0.599     28.5    183.4    377.8  |  0.599      9.7    187.9    395.1
8  |  0.599     25.1    180.6    354.8  |  0.599      7.2    186.6    383.0
9  |  0.599     18.9    180.5    361.4  |  
10 |  0.599     14.9    180.1    365.1  |  
11 |  0.598     12.4    163.3    255.1  |  
12 |  0.598      4.5    167.8    263.1  |


Justin, are you planning to implement  a variable blocksize support?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-25 02:14:14
Justin, are you planning to implement  a variable blocksize support?


Thanks for the decoding benchmarks.  To answer your question, yes, eventually.  My current order of business is integrating the current version with FFmpeg.  There has already been a very basic version committed to current FFmpeg SVN which is along the lines of using "flac -0" but with even less compression.  It was more of a test-run to get things properly integrated & generating compliant FLAC files.  I will reply here again when FFmpeg has competitive encoding, not just compliant encoding.

The variable blocksize support will mostly likely be after my newest, forthcoming, FFmpeg patch has been applied & optimized by more experienced FFmpeg developers.

-Justin
Title: New FLAC encoder
Post by: HbG on 2006-06-28 00:58:01
Code: [Select]
Pearl Jam - Ten (new european version) / Track 11: Release / Length: 543.426 seconds
System: Athlon XP 3200 - 1GB DDR400 - NForce2

Flac version:  libFLAC CVS 20060425
Flake version: flake-06

encode   | ratio | bitrate | speed  | dec. speed
---------|-------|---------|--------|-------------
flac   8 | 0.534 | 754.?   | 11.93x | 192.30x
flake  8 | 0.533 | 752.1   | 23.31x | 187.11x
flake 12 | 0.531 | 749.8   |  4.27x | 162.23x

Commandlines for encoding:
timethis "start /realtime /b /wait flac -8 test.wav -o NUL: -f"
timethis "start /realtime /b /wait flake -8 test.wav -o NUL:"
timethis "start /realtime /b /wait flake -12 test.wav -o NUL:"

Encodes were done three times, fastest run counts. Spread between runs was small, no more than 0.1s. Speeds are rounded.

Decoding speed was measured with Foobar2000 0.9.1, 10 passes, no dsp, high priority, buffer in memory.
Flake -8 file decodes about 44% faster with flacdec.


A 95.5% speedup. Impressive. Flake -8 is my new god.

I still miss some options, most important to me is -V as i like to overclock my hardware. Veryfying doesn't slow encoding down much anyway. Also reading from stdin doesn't seem to work on windows, it will say "invalid wav file: -" and then proceeds create a 0 byte flac output file, another feature it could do without
For the rest it's nothing metaflac can't handle.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-28 04:26:46
Code: [Select]
Pearl Jam - Ten (new european version) / Track 11: Release / Length: 543.426 seconds
System: Athlon XP 3200 - 1GB DDR400 - NForce2

Flac version:  libFLAC CVS 20060425
Flake version: flake-06

encode   | ratio | bitrate | speed  | dec. speed
---------|-------|---------|--------|-------------
flac   8 | 0.534 | 754.?   | 11.93x | 192.30x
flake  8 | 0.533 | 752.1   | 23.31x | 187.11x
flake 12 | 0.531 | 749.8   |  4.27x | 162.23x

Wonderful.  This is typical of the results I get as well.  I'm glad to see the relative encoding times are comparable on a fast, modern system.

Quote
A 95.5% speedup. Impressive. Flake -8 is my new god.

  I'm glad you like it.  Keep in mind that it is experimental at this point, but there will eventually be another release which will use FFmpeg's wav decoder & FLAC encoder.  So far it seems it might end up being faster as well.

Quote
I still miss some options, most important to me is -V as i like to overclock my hardware. Veryfying doesn't slow encoding down much anyway.

Oddly enough, I was just thinking about this today.  I noticed many people on Hydrogenaudio use this option.  I will definitely add something similar when I start back working on Flake.  I might have to use a modified version of FFmpeg's FLAC decoder instead of doing it directly through libavcodec.  But you're right that this is an important feature of the reference encoder.

Quote
Also reading from stdin doesn't seem to work on windows, it will say "invalid wav file: -" and then proceeds create a 0 byte flac output file, another feature it could do without

Hmm.  Maybe that has to do with having to set stdin to binary mode in windows...maybe.  This is good to know though.  Thanks for the feedback.

Quote
For the rest it's nothing metaflac can't handle.

I might even add metadata/cuesheet/seektable support someday.  Personally, I'm not a fan of the vorbis/flac comment specification, but it works.

All of this is probably at least a month or two (or three) away.  I have a lot going on in the next couple months.

Thanks,
-Justin
Title: New FLAC encoder
Post by: surf on 2006-06-28 07:56:41
hello.
does this new encoder supports unicode?
i can't connect to the page...404
Title: New FLAC encoder
Post by: sundance on 2006-06-28 08:47:12
Justin,

I'd also like to join in saying "thank-you" for your efforts to improve FLAC encoding. Highly appreciated!
I also integrated it into fb2k; the only drawback (= speed penalty) here is that reading from stdin doesn't work here either so temp. WAVs have to be uses with the converter...

Keep it up!

-sundance-


Btw, why didn't you use another vendor string than "[put libavcodec/libavformat versions here]"?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-28 14:38:20
Hello,

Btw, why didn't you use another vendor string than "[put libavcodec/libavformat versions here]"?


Because one of my main purposes for writing the encoder was to integrate it with FFmpeg.  I just needed a frontend to test it with before integrating it, so I made Flake.  The next release will have the lavc version as the vendor string.

Thanks,
-Justin
Title: New FLAC encoder
Post by: HbG on 2006-06-28 14:57:01
I just needed a frontend to test it with before integrating it, so I made Flake.  The next release will have the lavc version as the vendor string.

Thanks,
-Justin


I can't speak for the rest of us, but i really like the standalone, dropin flac replacement that flake is. I'm not very familiar with the ffmpeg project, but i fear it won't be as easy to use from the commandline or as encoder for eac or foobar when integrated into ffmpeg.
Title: New FLAC encoder
Post by: sundance on 2006-06-28 15:25:07
...but i really like the standalone, dropin flac replacement that flake is.


Second that!
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-06-29 03:06:36
I just needed a frontend to test it with before integrating it, so I made Flake.  The next release will have the lavc version as the vendor string.

Thanks,
-Justin


I can't speak for the rest of us, but i really like the standalone, dropin flac replacement that flake is. I'm not very familiar with the ffmpeg project, but i fear it won't be as easy to use from the commandline or as encoder for eac or foobar when integrated into ffmpeg.


Not to worry.  I will still re-release Flake as a standalone encoder once the code is part of FFmpeg.  I'll just use the relevant code from FFmpeg and might have to restructure it a little bit for use with Flake.  FFmpeg has/is a commandline frontend to its libavcodec and libavformat libraries.  When using FFmpeg, you'll most likely only be able to specify the compression level, not each individual option.  Flake will use the FFmpeg code, but be specifically only for FLAC encoding and will provide more detailed options along the lines of what it has currently.

As far as familiarity, you can find out more about FFmpeg at
http://ffmpeg.mplayerhq.hu/ (http://ffmpeg.mplayerhq.hu/)
It is a wonderful project with very good developers.  If you don't mind the 50 or so emails a day, subscribe to ffmpeg-devel and watch for a while...or just browse through the archives.

-Justin
Title: New FLAC encoder
Post by: wisodev on 2006-07-04 19:19:01
Well I am propobly too late but I have builded some binarys for win32 using intel compiler. If someone want to try them here is link http://thefrontend.sourceforge.net/flake/ (http://thefrontend.sourceforge.net/flake/). All versions compiled be me where faster then kurtnoise builds on my machine (Athlon XP 2000+). Archives include full sources, binarys and project files. In file TESTS-0.4-win32.zip are some speed tests results just to prove me claims ;-)
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-07-04 23:12:30
Well I am propobly too late but I have builded some binarys for win32 using intel compiler. If someone want to try them here is link http://thefrontend.sourceforge.net/flake/ (http://thefrontend.sourceforge.net/flake/). All versions compiled be me where faster then kurtnoise builds on my machine (Athlon XP 2000+). Archives include full sources, binarys and project files. In file TESTS-0.4-win32.zip are some speed tests results just to prove me claims ;-)


Thank you!  Especially for the win32 build environment stuff!  I am currently working on a simple configure/build system for unix environments.  I'll try to use some of what you have done as a basis to improve Windows compatibility in the next release.

The current development version of Flake has about a 15% speed improvement over pre-release 06 (on x86 linux gcc 3.4.4).  Compression has generally improved only slightly so far, except at compression level 0, which has a significant improvement.  FFmpeg integration of the full encoder is only about 1/2 done right now, but is going very well I think.  Right now it has levels 0 to 5, and you can change specific options on the commandline.  If you build FFmpeg from current SVN (http://ffmpeg.mplayerhq.hu/download.html), here are some of the FLAC encoding options.

Encode at level 5:
ffmpeg -i test.wav test.flac

Encode at level 0:
ffmpeg -i test.wav -compression_level 0 test.flac

Encode using individual options:
ffmpeg -i test.wav -frame_size 2048 -use_lpc 1 -min_prediction_order 6 -max_prediction_order 12 -min_partition_order 0 -max_partition_order 6 -lpc_coeff_precision 15 test.flac

Quickly convert ogg-flac to raw flac:
ffmpeg -i test.ogg -acodec copy test.flac

Also, the most updated Flake webpage is at another location.  I will be able to post some test samples of my own instead of linking to other sites since I now have more storage space.  I am limited to only 10MB on my Earthlink account.
http://jbr.homelinux.org/flake/ (http://jbr.homelinux.org/flake/)

-Justin
Title: New FLAC encoder
Post by: wisodev on 2006-07-05 06:59:59
Quote
Thank you!  Especially for the win32 build environment stuff!  I am currently working on a simple configure/build system for unix environments.

Quick note about the win32 project files (VC++ 7.1 and ICL 8.0 compatible): the FLAKE code compiles only with ICL not CL because of C99 option in intel compiler settings (without this option code does not compile). So the build environment works only with visual studio and intel compiler integrated. But it works very well without any nasty errors and produce great binarys ;-). I will in near future update the build environment to Intel 9.1 (actually I have done this locally and only need to update online stuff, but before that some tests will be committed to check if new version works good as prevoius).
Title: New FLAC encoder
Post by: HbG on 2006-07-07 21:36:14
Well I am propobly too late but I have builded some binarys for win32 using intel compiler. If someone want to try them here is link http://thefrontend.sourceforge.net/flake/ (http://thefrontend.sourceforge.net/flake/). All versions compiled be me where faster then kurtnoise builds on my machine (Athlon XP 2000+). Archives include full sources, binarys and project files. In file TESTS-0.4-win32.zip are some speed tests results just to prove me claims ;-)


Nice, for version 0.6 at -8 it's 26% faster here, with 29.40x vs 23.31x.

That makes it 146% faster than flac!

Oh, one more thing that just comes to mind, i miss a -q option for flake, but that's a minor issue. It does boost performance a tiny bit.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-07-08 00:29:37
Oh, one more thing that just comes to mind, i miss a -q option for flake, but that's a minor issue. It does boost performance a tiny bit.


That just never crossed my mind.  I'll add it to the next release.

Thanks,
Justin
Title: New FLAC encoder
Post by: jcoalson on 2006-07-08 00:51:39
another thing I would suggest is a big fat warning if the compression setting used is not in the FLAC subset (http://flac.sourceforge.net/format.html#subset).  not sure, looks like currently all settings are in the subset.  but I'm considering restricting the subset further (to lpc order<=12 and blocksize <=4608) for the next release, at least for <=48kHz.

Josh
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-07-08 21:44:08
another thing I would suggest is a big fat warning if the compression setting used is not in the FLAC subset (http://flac.sourceforge.net/format.html#subset).  not sure, looks like currently all settings are in the subset.  but I'm considering restricting the subset further (to lpc order<=12 and blocksize <=4608) for the next release, at least for <=48kHz.


Thanks Josh.  The 0-8 presets in Flake are within the subset, but the block size can be specified on the commandline to be any valid size within the spec, not just limited to the predefined ones required for subset.

I agree with your intentions to further restrict the subset requirements.  Block size over 4608 is rarely beneficial anyway unless it is a quite stationary signal, but the affect on decoding speed is substantial.  Maybe this would also make it easier to standardize hardware support.  There are products which say that they support FLAC levels 0 to 8.  This doesn't really say much about what else is truly supported outside the predefined levels.  If subset limits were restricted to the settings of level 8 it would be simpler and more definitive to just say that a product supports the FLAC Subset format or the full FLAC format (or FLAC 2 ).

-Justin
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-07-17 02:36:58
Hello,

I'm posting again here because the FFmpeg FLAC encoder has some significant changes.  Nearly all of Flake-06 has been integrated, along with some pretty great improvements by Michael Niedermayer (lead FFmpeg developer).  It is quite a bit faster than Flake (level 8 is about 60% faster than the reference encoder). Some of the key improvements are:
I have done a few benchmark comparisons (http://jbr.homelinux.org/flake/ffmpeg.html).

The FFmpeg homepage (http://ffmpeg.mplayerhq.hu/) has instructions for downloading/compiling the current SVN version.

One major thing left to do is to work out how to encode the MD5 sum in the global header.  Since FFmpeg supports many container formats, it is proving to be a little tricky.

-Justin
Title: New FLAC encoder
Post by: krmathis on 2006-07-17 08:02:31
Great work!
Hope you plan to "back-port" these improvements to the standalone flake encoder as well?
Title: New FLAC encoder
Post by: Garf on 2006-07-17 09:43:22
  • optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion


What's the advantage of the Cholesky method? I'm not familiar with it.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-07-17 10:36:03
  • optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion


What's the advantage of the Cholesky method? I'm not familiar with it.

Michael Niedermayer explains it better than I ever could here (http://article.gmane.org/gmane.comp.video.ffmpeg.cvs/3599). It took me a while to understand it even partially.

You probably have a better math background than me, so these might help.
Choleski decomposition (http://en.wikipedia.org/wiki/Choleski_decomposition)
Gaussian elimination (http://en.wikipedia.org/wiki/Gaussian_elimination)
Toeplitz matrix (http://en.wikipedia.org/wiki/Toeplitz_matrix)
Autocorrelation (http://en.wikipedia.org/wiki/Autocorrelation)
Levinson recursion (http://en.wikipedia.org/wiki/Levinson_recursion)

Quote
Great work!
Hope you plan to "back-port" these improvements to the standalone flake encoder as well?

Yes I do.  Some have been already, but I don't want to release anything new until the FFmpeg encoder gets to a good stable state.  (i.e. until the MD5/extradata/other containers issue has been solved).

-Justin
Title: New FLAC encoder
Post by: Garf on 2006-07-17 10:51:24
Okay, I understand the what and why

What's the speed impact?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-07-17 21:45:17
Okay, I understand the what and why

What's the speed impact?

Very big.  1-pass is only a little slower, but it does not give as good results as Levinson-Durbin.  The two seem to be equivilant in compression at around 3-4 pass.  So the real benefit is if you have a fast computer and/or lots of time to spare to wait for it to squeeze out a few more tenths of a percent.  Decoding speed is not affected.

example:
Levinson-Durbin level 12 - 80sec
Cholesky 10-pass level 12 - 1500sec
compression difference 0.3%

A good thing to add in the future that might give similar compression results would be to adaptively select or brute-force search for an optimal window function to go along with the Levinson-Durbin recursion.  I think Josh might be adding something like that to the reference encoder, but I'm not sure.  I know it currently is user-selectable, but it would be great if the best window was chosen by the encoder for each frame.  FFmpeg currently uses only a Welch window function.  I just chose that because it is a simple calculation and seems to work well most of the time.

-Justin
Title: New FLAC encoder
Post by: SebastianG on 2006-07-18 09:14:20
  • optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion

Heh! This is actually what I was suggesting during the "best window search for FLAC" period.
Too bad it's that slow.

Sebastian
Title: New FLAC encoder
Post by: jcoalson on 2006-07-24 07:04:38
I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter.  so far it works but it will not be ready for the next release.

Josh

(edit: lower bound on the least-squares solution I mean; I have some other experiments solving the system another way but that's too new.)
Title: New FLAC encoder
Post by: SebastianG on 2006-07-24 08:27:54
I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter.  so far it works but it will not be ready for the next release.

I suppose the pseudo-inverse (http://en.wikipedia.org/wiki/Pseudo_inverse#Finding_the_pseudoinverse_of_a_matrix) approach (through SVD) is even slower (but more accurate) than the Cholesky approach.
Title: New FLAC encoder
Post by: TBeck on 2006-07-24 16:25:05
I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter.  so far it works but it will not be ready for the next release.

Can you please tell which compression improvement can be achieved?
Title: New FLAC encoder
Post by: jcoalson on 2006-07-25 04:49:47
I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter.  so far it works but it will not be ready for the next release.

I suppose the pseudo-inverse (http://en.wikipedia.org/wiki/Pseudo_inverse#Finding_the_pseudoinverse_of_a_matrix) approach (through SVD) is even slower (but more accurate) than the Cholesky approach.

yep, you're right.  I'm mainly interested in establishing the lower bound first to see if other solvers would even be practical.

as for compression improvement, I don't have good numbers yet but it's not dramatic.  the autocorrelation formulation is surpisingly good with the proper window.

Josh
Title: New FLAC encoder
Post by: HbG on 2006-09-09 21:08:32
Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.
 
  I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.
 
 
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-09 23:21:46
Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.
 
  I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.
 
 

Sorry about that.  The Flake development is in a lull right now since I'm working feverishly on the Aften AC-3 encoder.  I'll run valgrind on Flake and see if I can fix that leak.
-Justin
Title: New FLAC encoder
Post by: HbG on 2006-09-09 23:34:30
Thanks in advance!
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-10 00:29:52

Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.
 
  I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.
 
 

Sorry about that.  The Flake development is in a lull right now since I'm working feverishly on the Aften AC-3 encoder.  I'll run valgrind on Flake and see if I can fix that leak.
-Justin


Here is a new bug-fix release to correct the memory leaks.  This download also includes some pre-compiled binaries for Linux and Windows.

http://jbr.homelinux.org/flake/flake-07.tar.gz (http://jbr.homelinux.org/flake/flake-07.tar.gz)

-Justin
Title: New FLAC encoder
Post by: HbG on 2006-09-10 12:02:27
That fixed it, thanks.

But the speed's now dropped from 29.40x to about 18.5, still much faster than flac (11.9x) but not nearly as impressive.

Only tested the windows binary, the output is correct.
Title: New FLAC encoder
Post by: wisodev on 2006-09-10 18:35:35
I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).

Download:
http://www.thefrontend.net/flake/flake-0.7-win32.rar (http://www.thefrontend.net/flake/flake-0.7-win32.rar)
Tests:
http://www.thefrontend.net/flake/TESTS-0.7-win32.zip (http://www.thefrontend.net/flake/TESTS-0.7-win32.zip)
Old builds:
http://www.thefrontend.net/flake/ (http://www.thefrontend.net/flake/)

wiso
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-10 19:31:31
That fixed it, thanks.

But the speed's now dropped from 29.40x to about 18.5, still much faster than flac (11.9x) but not nearly as impressive.

Only tested the windows binary, the output is correct.

Okay, well that has convinced me to take a leap and release a more up-to-date version of Flake.  I'm not sure if the windows binaries will still be slower than before since I was not the one who created the binary for version 06.  All I can say is that it's faster for me on Linux, and I hope it is on Windows as well.

http://jbr.homelinux.org/flake/ (http://jbr.homelinux.org/flake/)

It still doesn't have all the features in the FFmpeg encoder, but I'll get to that eventually.

Enjoy.
-Justin

I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).

Sorry to do two releases back-to-back.  I've had the 0.08 version sitting around and had wanted to tweak it some more before release, but I realized that might be a while, so I went ahead and released it.  I'll probably create a Sourceforge page so that I can do SVN updates and not have to release so often.

Also, my versioning has changed.  I was using "pre-release 01,02,...07".  Now I've switched to more normal versioning.  The latest is version 0.08.

Thanks!
-Justin
Title: New FLAC encoder
Post by: wisodev on 2006-09-10 19:46:58

I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).

Sorry to do two releases back-to-back.  I've had the 0.08 version sitting around and had wanted to tweak it some more before release, but I realized that might be a while, so I went ahead and released it.  I'll probably create a Sourceforge page so that I can do SVN updates and not have to release so often.

Also, my versioning has changed.  I was using "pre-release 01,02,...07".  Now I've switched to more normal versioning.  The latest is version 0.08.

Thanks!
-Justin


This is no problem for me. The progress is there and this is more important.

wiso
Title: New FLAC encoder
Post by: HbG on 2006-09-11 01:35:53
Oh yeah, wisodev's .07 build clocks in at about 29.1x for me with no memory leaks. Slightly slower than the leaky .06'es, but a major improvement nevertheless. Compression is the same. I should mention that i made some slight changes to my system and a reinstall, so there may be inaccuracies with my old benchmark results, but these should be fairly small, probably no more than .1x.

Pipe support still seems disfunctional, i get the following with both wisodev's .07 and justin's original fixed .06:
Code: [Select]
D:\MP3\Pearl Jam\Pearl Jam - Ten>type test.wav |flake -8 - test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 23965116

blocksize: 4608
max order: 12
order method: 4-level
stereo method: search

| bytes: 4448

The process tried to write to a nonexistent pipe.


0.08's windows binaries do not work for me. It reads the files but never starts encoding, just throws me back to the commandline, like this:
Code: [Select]
D:\MP3\Pearl Jam\Pearl Jam - Ten>flake-win32-i586.exe test.wav test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 23965116

D:\MP3\Pearl Jam\Pearl Jam - Ten>

All three builds do it. It does parse options and respond to -h, it creates a 0 byte output file.
Also, is there any chance of a changelog?

Like wisodev says, there is improvement and that is most important. Flake has a lot of potential, keep up the good work!
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-11 03:28:08
Oh yeah, wisodev's .07 build clocks in at about 29.1x for me with no memory leaks. Slightly slower than the leaky .06'es, but a major improvement nevertheless. Compression is the same. I should mention that i made some slight changes to my system and a reinstall, so there may be inaccuracies with my old benchmark results, but these should be fairly small, probably no more than .1x.

Pipe support still seems disfunctional, i get the following with both wisodev's .07 and justin's original fixed .06:

Hmm...I'll try to fix that after I get the Sourceforge page up and running.  I'm hoping for approval tomorrow.

Quote
0.08's windows binaries do not work for me. It reads the files but never starts encoding, just throws me back to the commandline, like this:
...
All three builds do it. It does parse options and respond to -h, it creates a 0 byte output file.
Also, is there any chance of a changelog?

Now that is strange.  If I have a chance I'll install Wine so I can do some debugging.  I'm guessing it's related to configure & Makefile.

Ok, I'll create a changelog.  Having the source in an svn repository will help document development as well since there will be commit messages.
Title: New FLAC encoder
Post by: wisodev on 2006-09-11 07:38:14
He, I have found too that 0.08 windows binarys are not working (all of them) even my builds are not working.

When it starts it throws stack overflow error on my machine (Windows XP)

But I have found the source of this problem.

in file encode.c in function
Code: [Select]
flake_encode_init(FlakeContext *s)

there was changed since version 07 way of handling context (FlacEncodeContext ctx_local;) memory allocation
and this change to local variable is causing under windows stack overflow
Code: [Select]
    FlacEncodeContext ctx_local;
    FlacEncodeContext *ctx;
    int i, header_len;

    if(s == NULL) {
        return -1;
    }

    ctx = &ctx_local;


I have changed to method used in 07 (allocating memory at the beginning of the function) and it works OK and do not throws any errors.

wiso
Title: New FLAC encoder
Post by: sundance on 2006-09-11 21:55:35
Justin,

would you consider adding wisodev's "patch" to flake.c that he did in v06?
Or is there any catch I can't see?

Code: [Select]
    if(!strncmp(opts.infile, "-", 2)) {
#ifdef _WIN32 // win32 pipes patch by wiso
        _setmode(_fileno(stdin), _O_BINARY);
#endif
        ifp = stdin;
    } else {
        ifp = fopen(opts.infile, "rb");
        if(!ifp) {
            fprintf(stderr, "error opening input file: %s\n", opts.infile);
            return 1;
        }
    }
    if(!strncmp(opts.outfile, "-", 2)) {
#ifdef _WIN32 // win32 pipes patch by wiso
        _setmode(_fileno(stdout), _O_BINARY);
#endif
         ofp = stdout;
    } else {
        ofp = fopen(opts.outfile, "wb");
        if(!ofp) {
            fprintf(stderr, "error opening output file: %s\n", opts.outfile);
            return 1;
        }
    }


It would help a lot to have piping work in WIN32...

Thanks a lot!

.sundance.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-11 22:03:50
would you consider adding wiso's "patch" to flake.c:

Yeah, I'll apply both of wiso's fixes and put them into SVN as soon as Sourceforge approves the project.  I'll also do a bit of code cleanup and documentation before doing the initial import and then jump up to version 0.10 as the 1st Sourceforge release.

-Justin
Title: New FLAC encoder
Post by: sundance on 2006-09-11 22:08:15
Justin,

that's really good news.
Too bad that there's already a project called FLAKE registered at SF: http://sourceforge.net/projects/flake (http://sourceforge.net/projects/flake)

.sundance.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-11 22:21:50
Justin,

that's really good news.
Too bad that there's already a project called FLAKE registered at SF: http://sourceforge.net/projects/flake (http://sourceforge.net/projects/flake)

.sundance.

I know!  I guess I should have looked around first...  I had no idea until I tried to register.  Mine doesn't use all caps though.  The "unix name" will be flake-enc.  Still, I don't know if that will be grounds for rejection?  I may have to rename it, which would really suck!

-Justin
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-11 22:49:32
Whew...it was accepted.
http://sourceforge.net/projects/flake-enc/ (http://sourceforge.net/projects/flake-enc/)

I'll work on getting everything uploaded and active this afternoon and evening.
Title: New FLAC encoder
Post by: jcoalson on 2006-09-11 23:25:10
"flache" (High-Efficiency? painful?) is not taken...
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-12 03:27:36
Whew...it was accepted.
http://sourceforge.net/projects/flake-enc/ (http://sourceforge.net/projects/flake-enc/)

I'll work on getting everything uploaded and active this afternoon and evening.

The SVN repository is up.
Version 0.10 has been released.

For now I have built binaries for Linux, optimized for i586, i686, or k8.  Also I included a couple (probably still slow) cross-compiled binaries for Windows, optimized for i586 or i686.  If wiso decides to be very kind again and builds a faster Windows binary, I would be happy to put it on Sourceforge as well.

Thanks,
Justin
Title: New FLAC encoder
Post by: wisodev on 2006-09-12 07:19:58
Quote
For now I have built binaries for Linux, optimized for i586, i686, or k8.  Also I included a couple (probably still slow) cross-compiled binaries for Windows, optimized for i586 or i686.  If wiso decides to be very kind again and builds a faster Windows binary, I would be happy to put it on Sourceforge as well.

Thanks,
Justin

I have downloaded the sources from sourceforge and I will upload binarys as soon as I come back home from work.
Title: New FLAC encoder
Post by: sundance on 2006-09-12 14:03:14
What do you think about adding some version information to the console output like
Code: [Select]
Flake: FLAC audio encoder version 0.10
(c) 2006  Justin Ruggles
to be able to identify which version of flake you're testing more easier.
Great you already included that version information as a vendor string to the flac header!

.sundance.
Title: New FLAC encoder
Post by: krmathis on 2006-09-12 16:14:11
Justin. I pulled revision 2 from SVN, and tried to build this on Mac OS 10.47 PowerPC/ GCC 4.0.1 (Xcode Tools 2.4). But it errors out like this:
Quote
$ ./configure
source path      /Users/krmathis/Downloads/flake-enc
C compiler      cc
make            make
CPU              powerpc (generic)
big-endian      yes
inttypes.h      yes
AltiVec enabled  yes
debug symbols    yes
strip symbols    yes
optimize        yes
Creating config.mak and config.h...
$ make
make -C libflake all

** Snip **

cc -O3  -no-cpp-precomp -pipe -force_cpusubtype_ALL -Wno-sign-compare -fomit-frame-pointer -faltivec -g -Wdeclaration-after-statement -Wall -Wno-switch -mdynamic-no-pic -DHAVE_CONFIG_H -I. -I.. -I/Users/krmathis/Downloads/flake-enc/libflake -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -c -o wav.o wav.c
cc -L../libflake -Wl,-dynamic,-search_paths_first  -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2


I successfully compiled flake 0.7 two days ago. But the build system have changed since then (from "make" to "./configure && make")

Any idea whats wrong?
Title: New FLAC encoder
Post by: wisodev on 2006-09-12 17:28:15
FLAKE 0.10 Win32 Build and Sources

Download binarys:
http://www.thefrontend.net/flake/flake-0.10-win32-bin.rar (http://www.thefrontend.net/flake/flake-0.10-win32-bin.rar)
Download sources:
http://www.thefrontend.net/flake/flake-0.10-win32-src.rar (http://www.thefrontend.net/flake/flake-0.10-win32-src.rar)

Alternative ZIP download:

Download binarys:
http://www.thefrontend.net/flake/flake-0.10-win32-bin.zip (http://www.thefrontend.net/flake/flake-0.10-win32-bin.zip)
Download sources:
http://www.thefrontend.net/flake/flake-0.10-win32-src.zip (http://www.thefrontend.net/flake/flake-0.10-win32-src.zip)

I have tested my build and it looks like encoding speed has increased , this is great news!

Test results (using my build, sf.net build i686 and flac 1.1.2 build):
Code: [Select]
TEST FILE: 44100Hz, 16bit, Stereo, 431 916 620 bytes (13 Tracks)

ENCODE_FLAKE_INTEL [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

Flake: FLAC audio encoder
© 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 107972676

blocksize: 4608
max order: 12
order method: 4-level
stereo method: mid-side

progress: 100% | ratio: 0.741 | bitrate: 1045.2 kbps | bytes: 319851517


Kernel Time  =    3.328 = 00:00:03.328 =  3%
User Time    =    93.218 = 00:01:33.218 =  89%
Process Time =    96.546 = 00:01:36.546 =  92%
Global Time  =  104.172 = 00:01:44.172 = 100%
ENCODE_FLAKE_i686 [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

Flake: FLAC audio encoder
© 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 107972676

blocksize: 4608
max order: 12
order method: 4-level
stereo method: mid-side

progress: 100% | ratio: 0.741 | bitrate: 1045.2 kbps | bytes: 319851517


Kernel Time  =    3.078 = 00:00:03.078 =  2%
User Time    =  132.953 = 00:02:12.953 =  94%
Process Time =  136.031 = 00:02:16.031 =  96%
Global Time  =  140.671 = 00:02:20.671 = 100%
ENCODE_FLAC_1_1_2 [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

options: -P 4096 -b 4608 -m -l 12 -e -q 0 -r 0,6
Catamenia - [1999] - Morning Crimson (HQ).wav: wrote 321356528 bytes, ratio=0,74
4

Kernel Time  =    3.828 = 00:00:03.828 =  1%
User Time    =  206.640 = 00:03:26.640 =  96%
Process Time =  210.468 = 00:03:30.468 =  97%
Global Time  =  214.891 = 00:03:34.891 = 100%
DECODE_FROM_FLAKE_INTEL [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

t_INTEL.flac: done

Kernel Time  =    3.328 = 00:00:03.328 =  7%
User Time    =    22.703 = 00:00:22.703 =  53%
Process Time =    26.031 = 00:00:26.031 =  61%
Global Time  =    42.188 = 00:00:42.188 = 100%
DECODE_FROM_i686 [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

t_i686.flac: done

Kernel Time  =    3.140 = 00:00:03.140 =  7%
User Time    =    23.078 = 00:00:23.078 =  56%
Process Time =    26.218 = 00:00:26.218 =  63%
Global Time  =    41.094 = 00:00:41.094 = 100%
DECODE_FROM_FLAC_1_1_2 [-8]

Timer 3.01  Copyright © 2002-2003 Igor Pavlov  2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

t_FLAC.flac: done

Kernel Time  =    3.015 = 00:00:03.015 =  7%
User Time    =    22.953 = 00:00:22.953 =  56%
Process Time =    25.968 = 00:00:25.968 =  63%
Global Time  =    40.765 = 00:00:40.765 = 100%

---

FILE SIZE

t_FLAC.flac 321 356 528
t_i686.flac 319 851 517
t_INTEL.flac 319 851 517
Title: New FLAC encoder
Post by: lexor on 2006-09-12 20:19:41
Code: [Select]
 
FILE            SIZE

t_FLAC.flac        321 356 528
t_i686.flac        319 851 517
t_INTEL.flac        319 851 517

wisodev, what's up with the second column?
Title: New FLAC encoder
Post by: HbG on 2006-09-13 01:20:10
My run went at 37.00x That's more than three times faster than flac at -8 and 7.5x up from 0.7. Memory usage dropped a little, and no leaks.

Pipe support works, i can use it as external encoder in foobar2000 without problems.
What's even better, it seems foobar can automatically emebed a .cue file in the tag, only it doesn't preserve indexes, and it doesn't embed it in the actual flac way. Can't win 'em all.

Still, this is an awesome achievement. I can't believe the reference encoder had so much room for improvement!

This needs to get on rarewares.
Title: New FLAC encoder
Post by: wisodev on 2006-09-13 06:48:30
Quote
Code: [Select]
 
FILE            SIZE

t_FLAC.flac        321 356 528
t_i686.flac        319 851 517
t_INTEL.flac        319 851 517

wisodev, whats up with the second column?


This are output file sizes in bytes that are results of runned tests above. t_FLAC.flac is output file from original Flac encoder at version 1.1.2. The t_i686.flac and t_INTEL.flac are output file from Flake encoder version 0.10. t_i686.flac encoded using binary from Justin build and t_INTEL.flac is encoding output from my build. (It is called INTEL because I use Intel C++ Compiler to build my binarys). The mystery is solved ;-)
Title: New FLAC encoder
Post by: sundance on 2006-09-13 08:29:40
wisodev,

still one mystery remains: 
Code: [Select]
DECODE_FROM_FLAKE_INTEL...
Kernel Time  =     3.328 = 00:00:03.328 =   7%
User Time    =    22.703 = 00:00:22.703 =  53%
Process Time =    26.031 = 00:00:26.031 =  61%
Global Time  =    42.188 = 00:00:42.188 = 100%

DECODE_FROM_i686...
Kernel Time  =     3.140 = 00:00:03.140 =   7%
User Time    =    23.078 = 00:00:23.078 =  56%
Process Time =    26.218 = 00:00:26.218 =  63%
Global Time  =    41.094 = 00:00:41.094 = 100%


Although both compilations are expected to generate bit-identical output, the decoding times vary.
Guess you would have to run those test multiple times to average out those variations.

.sundance.
Title: New FLAC encoder
Post by: wisodev on 2006-09-13 08:52:03
Quote
wisodev,

still one mystery remains:
Code: [Select]
DECODE_FROM_FLAKE_INTEL...
Kernel Time  =     3.328 = 00:00:03.328 =   7%
User Time    =    22.703 = 00:00:22.703 =  53%
Process Time =    26.031 = 00:00:26.031 =  61%
Global Time  =    42.188 = 00:00:42.188 = 100%

DECODE_FROM_i686...
Kernel Time  =     3.140 = 00:00:03.140 =   7%
User Time    =    23.078 = 00:00:23.078 =  56%
Process Time =    26.218 = 00:00:26.218 =  63%
Global Time  =    41.094 = 00:00:41.094 = 100%


Although both compilations are expected to generate bit-identical output, the decoding times vary.
Guess you would have to run those test multiple times to average out those variations.

.sundance.


Well, the test file is more then 400 MB and testing it in right way using more runs is time consuming. I only needed some simple estimate of speed, not precise testing. But the difference in decoding time is marginal and in range of measure error I think.
Title: New FLAC encoder
Post by: lexor on 2006-09-13 14:43:36
This are output file sizes in bytes that are results of runned tests above. t_FLAC.flac is output file from original Flac encoder at version 1.1.2. The t_i686.flac and t_INTEL.flac are output file from Flake encoder version 0.10. t_i686.flac encoded using binary from Justin build and t_INTEL.flac is encoding output from my build. (It is called INTEL because I use Intel C++ Compiler to build my binarys). The mystery is solved ;-)

bleh put commas next time, I thought those were 3 different columns not a single number. out of curiosity where have you seen a number with spaces in it, that you didn't think the table looks wrong?
Title: New FLAC encoder
Post by: wisodev on 2006-09-13 19:07:26

This are output file sizes in bytes that are results of runned tests above. t_FLAC.flac is output file from original Flac encoder at version 1.1.2. The t_i686.flac and t_INTEL.flac are output file from Flake encoder version 0.10. t_i686.flac encoded using binary from Justin build and t_INTEL.flac is encoding output from my build. (It is called INTEL because I use Intel C++ Compiler to build my binarys). The mystery is solved ;-)

bleh put commas next time, I thought those were 3 different columns not a single number. out of curiosity where have you seen a number with spaces in it, that you didn't think the table looks wrong?


Oh the spaces are common error in windows ;-)

Try right clicking on some file in explorer and you can see this spaces everywhere or better try in windows console DIR command and the spaces comes in file sizes. This is funny.
Title: New FLAC encoder
Post by: iGold on 2006-09-14 08:18:40
I want to write my notes on lack of some features in current flake version.

In windows world, especially for on-the-fly ripping, it may be problem, but if you have flac/metaflac utilites and able to write simple script/batch file you can:So, metaflac is a swiss knife for .flac files when you have already compressed .flac file with lack of some features (which flac executable can add at compress time!). Referenced flac executable is great and featureful program but flake generates file smaller in size and complitely compatible with FLAC specification.

And now I'm using flake for (the best, ATM) compression (with -8, for compatibility reasons) and flac/metaflac to test/process .flac files.
Title: New FLAC encoder
Post by: Egor on 2006-09-14 08:46:55
Oh the spaces are common error in windows ;-)

"It's not a bug, it's a feature."

Digit grouping symbol depends on your locale, and can be specified explicitly in Control Panel -> Regional options -> Numbers. BTW, a space is a correct grouping symbol in the Russian locale, and a comma is not. A comma is a decimal point symbol here.
Title: New FLAC encoder
Post by: wisodev on 2006-09-14 09:18:45
Quote
Digit grouping symbol depends on your locale, and can be specified explicitly in Control Panel -> Regional options -> Numbers. BTW, a space is a correct grouping symbol in Russian locale, and a comma is not. A comma is a decimal point symbol here.


Yes thanks for clarifying this issue. In Polish locale space is also valid as you can see in screenshot attached below and yes it is feature of OS. I was bit too sarcastic I suppose.

(http://img241.imageshack.us/img241/2084/clipboard1ex3.th.jpg) (http://img241.imageshack.us/my.php?image=clipboard1ex3.jpg)
Title: New FLAC encoder
Post by: lexor on 2006-09-14 19:27:19
hi, I have a somewhat strange request  a few pages back there was a discussion about linear solvers and pseudo-inverses, can someone possibly post actual matrix that would appear on the left hand side (taken from encoding a sample of your choice, during algorithm run) and a right hand side vector? any one will do. obviously as attachement in a text file and not plaintext in the reply (since I'm assuming the dimensions will be quite large)

thank you very much, I know it's weired
Title: New FLAC encoder
Post by: Madman2003 on 2006-09-14 19:32:41
Is this encoder (flake 0.10) trustworthy for "production" use?
Title: New FLAC encoder
Post by: bukem on 2006-09-14 23:17:30
FYI:

I have just encoded my favorite album of Maria Peszek called "Miasto Mania" with flake -12 and it plays perfectly on my iAudio U3 2GB flash player even though the specification states that it does support only flac 0, 1 & 2. Isn't that great! 
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-15 00:07:01
Is this encoder (flake 0.10) trustworthy for "production" use?

It depends on what you're looking for.  The testing has not been extremely exhaustive, but on the other hand, I have yet to produce a FLAC file with Flake which will not decode losslessly using the reference decoder.  Also, it embeds an MD5 sum for checking losslessness.  One thing it lacks which may be important to some is verification during encoding.

FYI:

I have just encoded my favorite album of Maria Peszek called "Miasto Mania" with flake -12 and it plays perfectly on my iAudio U3 2GB flash player even though the specification states that it does support only flac 0, 1 & 2. Isn't that great! 

Now that is a surprise!  It would be interesting to make some test cases to put hardware players through a FLAC compatibility test.
Title: New FLAC encoder
Post by: guruboolez on 2006-09-15 00:10:03
I have just encoded my favorite album of Maria Peszek called "Miasto Mania" with flake -12 and it plays perfectly on my iAudio U3 2GB flash player even though the specification states that it does support only flac 0, 1 & 2. Isn't that great! 

This is exactly the kind of feedback I was looking for! Thank you for this test
Title: New FLAC encoder
Post by: bukem on 2006-09-15 12:20:20

I have just encoded my favorite album of Maria Peszek called "Miasto Mania" with flake -12 and it plays perfectly on my iAudio U3 2GB flash player even though the specification states that it does support only flac 0, 1 & 2. Isn't that great! 

This is exactly the kind of feedback I was looking for! Thank you for this test


I'm glad that I could help a bit. BTW, I've checked the iAudio site again, and to my surprise, I've found updated specification of U3:
Quote
MP3 : MPEG 1/2/2.5 Layer 3, ~320kbps, ~48khz, mono/stereo
WMA : ~256kbps, ~48khz, mono/stereo
OGG : ~Q10, ~44.1khz, mono/stereo
FLAC : compression level 0~8, ~44.1khz, mono/stereo
WAV : ~48khz, 16bit, mono/stereo
Video File up to 160x128
JPG, TXT
Title: New FLAC encoder
Post by: jcoalson on 2006-09-15 16:32:28
yes, that level 0-2 limitation was from the early days and I'm pretty sure all the iAUDIOs that support FLAC will play 0-8.  their forums are the best place to ask though if you're about to buy.

Josh
Title: New FLAC encoder
Post by: HbG on 2006-09-16 00:53:17
I've encoded over a dozen cd's with flake -8 so far and never encountered a problem apart from the memory leaks in older versions.

Don't have a DAP that plays flacs though.

Because it has relatively few switches there's not much to go wrong either. Up to -8 it should output valid flac files that should be indistinguishable from the reference encoder output (save the encoder version string of course). But don't quote me on that.

Indeed a -V option that uses the reference decoder for verification and a warning for levels over -8 (if there isn't one already) would come in handy.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-18 11:31:59
I have tried running my scripts with wisodev's flake 0.10 and both times now flake has crashed.  The first time a memory allocation error was reported.  My wife has just GTalked me to tell me that it has happened today (I left the script running before I went to work) so I'm afraid I can't tell you the exact error message.

I will try running the script again and post the exact (standard Windows) message, but I was wondering if anyone else had experienced this?
Title: New FLAC encoder
Post by: sundance on 2006-09-18 12:01:00
I've been using wiso's flake 0.10 several times in foobar's converter without any problems yet...

[edit] Forgot mentioning that I used the Intel compilation...
Title: New FLAC encoder
Post by: wisodev on 2006-09-18 13:00:18
@Synthetic Soul

Can you try running this scripts using flake-0.10-win32-i686.zip binary from flake sourceforge page.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-18 13:39:00
I have run my scripts here at work using the i586, i686 and wisodev builds available at sourceforge.

In the first run, with all three encoders using -5 all encodes (50 files (http://synthetic-soul.co.uk/comparison/lossless/corpus.asp) sent to each encoder) were successful.

I just started another batch file only using the wisodev build using -0, -5, -8 and -12.  The encoder crashed using -0 on file 37.wav (http://synthetic-soul.co.uk/comparison/lossless/file.asp?FilePKID=38).  This leads me to believe that something about the file is causing the issue, as the first crash happened on 37.wav also, although I can't remember the compression setting used.  As previously mentioned I cannot confirm any details about the second crash.

I will perform some more tests with the wisodev build, specifically with my file 37.wav, and let you know if I can pin this down any further.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-18 13:53:24
OK, I have tested 37.wav with the i586, i686 and wisodev builds at compression levels -0, -1, -5, -8, and -12.

All three builds work fine for all levels, except -0 where all three builds crash.

Edit:  Here's Flake's dying words, if it is any use:

Code: [Select]
Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 10465224

blocksize: 1024
max order: 0
order method: maximum
stereo method: mid-side

progress: 100% | ratio: 0.813 | bitrate: 1147.0 kbps | bytes: 33988647

Edit 2: The above tests were taken using my "Yalac" corpus (used to test Yalac/TAK).  I have just run the scripts on my "FLAC" corpus (used to test 1.1.2_CVS) and I had the same happen with file 02.wav, which is the same file as 37.wav.  Therefore, out of 78 files tested only one file can highlight this bug.
Title: New FLAC encoder
Post by: Will Fisher on 2006-09-18 16:15:56
Hey,

I work for Winamp, and I'm very impressed by this encoder, so I wrote a winamp encoder plugin which uses libflake.

Binary + source: http://stashbox.org/uploads/1158591116/enc_flac.zip (http://stashbox.org/uploads/1158591116/enc_flac.zip)

To compile you'll need wisodevs win32 sources, extract them to the flake directory. Feel free to put this into the sourceforge SVN or whatever.

For the moment, I just let the user configure the compression value, everything else is chosen based upon the compression by libflake.

I have replicated the speed differences between flake and the reference flac encoder when comparing with other flac encoding plugins, and for a much smaller binary!

I can't make any guarantees, but hopefully one day this will be considered for inclusion in the winamp installer. 

Thanks,

Will Fisher
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-18 16:58:36
It seems the problem is still evident even in the first 10 seconds of the file, so I have uploaded a Flake -12 encoded version of the first 10 seconds for others to test:

http://synthetic-soul.co.uk/temp/37_0-10_12.flac (http://synthetic-soul.co.uk/temp/37_0-10_12.flac)
Title: New FLAC encoder
Post by: HbG on 2006-09-18 22:59:32
I can encode that sample at -0 just fine, using wisodev's Flake 0.10 build, both through Foobar and the commandline.

I did discover another bug: when trying to read from the following filename: "- 37_0-10_12.wav" or when trying to write a file called "- 37_0-10_12.flac" it fails. For the output i get a "tried to write to nonexistent pipe" error and for the input file, it treats it like an unrecognised commandline and gives the standard message. I tested further and it gives the error for all filenames that start with a dash.
Flac handles it fine.
Title: New FLAC encoder
Post by: bukem on 2006-09-18 23:45:51
@Synthetic Soul:

Flake 0.10 has failed to compress your test file with -0 switch on my Pentium 4 PC BUT it succeeded to compress it on my Intel Centrino laptop. Seems like Flake has some issues with Pentium 4?

Edit: I did my tests with Flake on WinXP SP2
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-19 01:26:04
I can encode that sample at -0 just fine, using wisodev's Flake 0.10 build, both through Foobar and the commandline.

I did discover another bug: when trying to read from the following filename: "- 37_0-10_12.wav" or when trying to write a file called "- 37_0-10_12.flac" it fails. For the output i get a "tried to write to nonexistent pipe" error and for the input file, it treats it like an unrecognised commandline and gives the standard message. I tested further and it gives the error for all filenames that start with a dash.
Flac handles it fine.

I'll try to fix that very soon.  Flake only has very simple commandline parsing right now.  It should be fairly trivial to fix.
Thanks,
Justin


It seems the problem is still evident even in the first 10 seconds of the file, so I have uploaded a Flake -12 encoded version of the first 10 seconds for others to test:

http://synthetic-soul.co.uk/temp/37_0-10_12.flac (http://synthetic-soul.co.uk/temp/37_0-10_12.flac)


Hmm. It works for me, but I'm using Linux.  The "-0" mode has been changed some since the 0.10 release.  Could you try using current SVN?  If you need a binary, I can compile one & post it.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-19 07:29:52
Mine is failing at home on Win2K with Athlon XP 2400+, and at work with XP.  I believe the CPU at work is AMD.  I will confirm later.

I will see if I can compile from SVN today at work.

Edit: I will only be able to compile if there are Visual C++ project files, as I'm not a C++ programmer, I can just hack some code and click a button to compile.  As you are a Linux man I doubt this will be the case so a binary would be nice if someone could provide.  Edit 2: I do have Cygwin installed, but I'm not sure that I have the necessary parts, and I've never compiled using a makefile!  I see from browsing the repository that there are no Visual C++ project files.

Flake 0.10 has failed to compress your test file with -0 switch on my Pentium 4 PC BUT it succeeded to compress it on my Intel Centrino laptop. Seems like Flake has some issues with Pentium 4?
Thanks for proving I'm not mad anyway!
Title: New FLAC encoder
Post by: Mangix on 2006-09-19 08:27:48
i think that you can just use make to compile the sources. if you want to do that, just grab a copy of MinGW(there's a nice installer for it too) and use "mingw32-make" on the directory with all of the source code. i just quickly browsed through the SVN and i saw that there was a makefile in there so theoretically, it should work. I'm a bit lazy to try it out though.

edit: btw, flake is really nice. right now, I'm using the -12 option on build 0.10 and it makes smaller files with faster decoding. i tested it with Josh's optimized FLAC build(the one which he uploaded here) and flake still beats it. I've also heard that FLAC 1.1.3 will have some of flake's improvements so i can't wait till FLAC 1.1.3 releases.

edit2: spelling
Title: New FLAC encoder
Post by: ckjnigel on 2006-09-19 08:47:13
I've also heard that FLAC 1.1.3 will have some of flake's improvements


Are the two developers in contact and working cooperatively rather than competitively?
Title: New FLAC encoder
Post by: Mangix on 2006-09-19 08:53:11
i, personally, am not too sure. i've only seen a post in a yalac thread(or something similar) by Josh saying that some of flake's improvements will be in 1.1.3. if you want to know the answer to the question, then i guess you should contact Josh.
Title: New FLAC encoder
Post by: bukem on 2006-09-19 09:06:14
Guys, sorry for beeing noob, but could someone of you upload the build for win32 platform using latest SVN, please?
Title: New FLAC encoder
Post by: wisodev on 2006-09-19 10:53:05
@bukem
I will upload binarys based on SVN sources after I come home from work (around 16.00 CET).

@Synthetic Soul
You can use project files from flake-0.10-win32-wisodev-src.zip at SourceForge. Copy win32 directory to exported SVN sources root directory and build flake project using Debug setting to provide more information on possible crashes.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-19 11:14:29
I appear to have managed to build flake under cygwin.  I had to download autoconf and make but it finally worked (once I worked out gcc didn't like paths with spaces...).

My binary can be found here (http://synthetic-soul.co.uk/temp/flake_svn_20060919c.7z). Edit: Link updated to bug-free version (see #113)

It appears to work with 37.wav.

@Synthetic Soul
You can use project files from flake-0.10-win32-wisodev-src.zip at SourceForge. Copy win32 directory to exported SVN sources root directory and build flake project using Debug setting to provide more information on possible crashes.
Ah, thanks for that.  I may take a look anyway.

Edit: wisodev.  I'm using Visual C++ 2005 Express Edition and, after I had converted the solution file to the newer format, it complains about the .icproj files... saying the application is not installed...  no idea what it's all about.  I think I've proved for now that the SVN code works with my file, but I will no doubt test again if you release an SVN version.

Justin, out of interest, what is flake_g.exe?  Is it a debug version?  As I said, I'm not used to compiling with make!

i think that you can just use make to compile the sources. if you want to do that, just grab a copy of MinGW(there's a nice installer for it too) and use "mingw32-make" on the directory with all of the source code.
I tried this, but the directory had a configure file which I found out is used with autoconf to create config.mak which is required to build, so I had to go the cygwin route (I think).  Thanks for the info anyway.


Edit 10: Hmm... for some reason my cygwin compile refuses to accept -12 saying 21 is too high.  Using -21 works but I'm yet to check whether it is actually being accepted as 12.  Is this my compile's fault?! 
Title: New FLAC encoder
Post by: bukem on 2006-09-19 13:19:50
@wisodev:
thanks

@Synthetic Soul / Justin:
There's problem with your compile of flake or with that particular SVN version. When I try to use preset -12 I get following error:

Code: [Select]
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

D:\documents\music\.work>flake -12 37_0-10_12.wav test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

invalid compression: 21. must be 0 to 12.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.

It seems that for preset 12 wrong value is hardcoded in the source code -> 21 instead of 12.
Title: New FLAC encoder
Post by: wisodev on 2006-09-19 13:40:43
@Synthetic Soul
You can use project files from flake-0.10-win32-wisodev-src.zip at SourceForge. Copy win32 directory to exported SVN sources root directory and build flake project using Debug setting to provide more information on possible crashes.
Ah, thanks for that.  I may take a look anyway.

Edit: wisodev.  I'm using Visual C++ 2005 Express Edition and, after I had converted the solution file to the newer format, it complains about the .icproj files... saying the application is not installed...  no idea what it's all about.  I think I've proved for now that the SVN code works with my file, but I will no doubt test again if you release an SVN version.


Oh sorry I am using Intel C++ compile at version 8.0.40 and C99 switch (this is req.) witch means it wan't compile Visual Studio without Intel compiler. And version 8 of Intel C++ compiler do not works with Express edition, maybe version 9 works but it produces for me slower binarys (trail version of compiler available on intel website).
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-19 13:50:27
@Synthetic Soul / Justin:
There's problem with your compile of flake or with that particular SVN version. When I try to use preset -12 I get following error:

Code: [Select]
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

D:\documents\music\.work>flake -12 37_0-10_12.wav test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

invalid compression: 21. must be 0 to 12.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.
Yes, I just noticed this (see edit above)!  Very curious.  I can only assume it is my dodgy compiling though.  I would wait for wisodev's.
Title: New FLAC encoder
Post by: pest on 2006-09-19 14:43:53
there is a small bug in flake.c in the parse_number() subroutine

a quick fix

change

Code: [Select]
  for(i=0; i<max; i++)


to

Code: [Select]
  for(i=max-1; i>=0; i--)
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-19 14:54:58
there is a small bug in flake.c in the parse_number() subroutine
Thanks for that.

New binary uploaded

http://synthetic-soul.co.uk/temp/flake_svn_20060919c.7z (http://synthetic-soul.co.uk/temp/flake_svn_20060919c.7z)

Edit: Compile reuploaded using Justin's fix described below.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-19 15:18:03
i, personally, am not too sure. i've only seen a post in a yalac thread(or something similar) by Josh saying that some of flake's improvements will be in 1.1.3. if you want to know the answer to the question, then i guess you should contact Josh.

To answer your question, no we are not in direct collaboration.  It's not that I want to be competitive...open source is collaborative by its very nature.  I do want to continue to make Flake faster and better, but I'll also be quite happy if some of the ideas in Flake are used to help improve the reference encoder.
 
@Synthetic Soul / Justin:
There's problem with your compile of flake or with that particular SVN version. When I try to use preset -12 I get following error:

Code: [Select]
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

D:\documents\music\.work>flake -12 37_0-10_12.wav test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

invalid compression: 21. must be 0 to 12.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.
Yes, I just noticed this (see edit above)! Very curious. I can only assume it is my dodgy compiling though. I would wait for wisodev's.

  It is fixed now in SVN.  And as was just pointed out, it was my dodgy coding, not anyone's dodgy compiling.


there is a small bug in flake.c in the parse_number() subroutine

a quick fix

change

Code: [Select]
  for(i=0; i<max; i++)


to

Code: [Select]
  for(i=max-1; i>=0; i--)

That was the problem, but that solution does not work.  The 'max' parameter is not the actual number of digits, but the maximum number.  If the number of digits is shorter, that code will not work.  I applied a fix which counts the digits first, then works backwards.
Title: New FLAC encoder
Post by: pest on 2006-09-19 16:06:31
Quote
The 'max' parameter is not the actual number of digits, but the maximum number.


the fix was as quick as my thoughts about it, sry!
btw, why do you reinvent the wheel? use atoi()

this also works
Code: [Select]
int ParseNumber(char *arg)
{
  int i;
  int n = 0;
  for(i=0;i<strlen(arg);i++)
  {
    if(arg[i] < '0' || arg[i] > '9') return -1;
    n *= 10;
    n += (arg[i]-48);
  }
  return n;
}
Title: New FLAC encoder
Post by: lexor on 2006-09-19 16:18:49
Hey Justin, I was just looking over the code for Levinson-Durbin recursion, and noticed something odd. In it's very signature you have lpc[][], which is declared in lpc_calc_coefs as lpc[32][32] (I put numbers instead of constant name for readability here), but the algorithm works on the Toeplitz matrix, so storage should be 2N-1. That's a terrible waste of bits, even if it is only a 32x32 array.

what makes even less sense to me is that Levinson algorithm supposed to solve Ax=b for x, given A and b. and that lpc_compute_coefs seems to take 2 vectors and compute the matrix.
Title: New FLAC encoder
Post by: wisodev on 2006-09-19 16:34:04
Wll I am bit late but I have tested this wav file witch caused crushes.
I think the s->max_frame_size is estimated incorrectly.

I have patched sources and I think fixed both bugs (crushing and incorrect command-line argument parsing).
My ParseNumber fix is similiar to posted here.

Download patched sources:
flake-svn-2006-09-19-patched-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-patched-win32-src.zip)
Download patched binarys:
flake-svn-2006-09-19-patched-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-patched-win32-bin.zip)

wiso
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-19 17:15:15
Thanks for the non-cygwin-reliant compile wisodev.

I must admit that I am a little confused about the s->max_frame_size bug.

The version I compiled from the SVN source worked fine with my troublesome 37.wav.

Are you saying that, even so, there was still a bug present that could reproduce the behaviour I had seen?
Title: New FLAC encoder
Post by: wisodev on 2006-09-19 17:23:34
Thanks for the non-cygwin-reliant compile wisodev.

I must admit that I am a little confused about the s->max_frame_size bug.

The version I compiled from the SVN source worked fine with my troublesome 37.wav.

Are you saying that, even so, there was still a bug present that could reproduce the behaviour I had seen?


No problem.

- Well I tested the sources (before ParseNumber fix) and this bug was present. The free(frame) in main.c  was causing this crush.
- When I changed max frame bufer to bigger value (added 4 bytes) then it was all OK.
- I have checked older sources (07 version of flake) and the estimation of max frame buffer (encode.c) was changed since them. So I have patched the svn sources and the bug was gone.
- Please try this flake-svn-2006-09-19-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-win32-bin.zip) binary because it is causing flake to crush with 37.wav file and setting -0.

This becomes little complicated. Maybe Justin can put more light on this bug.
Title: New FLAC encoder
Post by: bukem on 2006-09-19 18:10:19
Thanx wisodev for non-cygwin version! And thank you Synthetic Soul for a cygwin one
Title: New FLAC encoder
Post by: jcoalson on 2006-09-19 19:00:12
I've also heard that FLAC 1.1.3 will have some of flake's improvements

Are the two developers in contact and working cooperatively rather than competitively?

it's not competitive.  it's good to have independent (compliant) implementations.  flake code is simpler and encodes faster, but has less features.

the improvement is in how the signal is windowed before lpc analysis.  we were both working on this at the same time but flake has a shorter release cycle.  the method in the upcoming version of FLAC is slower but gives slightly more compression.

Hey Justin, I was just looking over the code for Levinson-Durbin recursion, and noticed something odd. In it's very signature you have lpc[][], which is declared in lpc_calc_coefs as lpc[32][32] (I put numbers instead of constant name for readability here), but the algorithm works on the Toeplitz matrix, so storage should be 2N-1. That's a terrible waste of bits, even if it is only a 32x32 array.

that 32x32 array is for the output coefficients for all orders.  the max order is 32 so you need 32+31+30+...+1 doubles.  there is some waste putting it in a sparse array but not much, and the code is simpler.

what makes even less sense to me is that Levinson algorithm supposed to solve Ax=b for x, given A and b. and that lpc_compute_coefs seems to take 2 vectors and compute the matrix.

it's computing several solutions for several Ax=b, for several x and b, from the same autocorrelation of the original signal.  that's the magic.  see also http://en.wikipedia.org/wiki/Linear_prediction (http://en.wikipedia.org/wiki/Linear_prediction)
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-19 19:31:27
- Please try this flake-svn-2006-09-19-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-win32-bin.zip) binary because it is causing flake to crush with 37.wav file and setting -0.
I am now at home, on Win 2K AMD XP 2400+.  I have just tried both your compile and the cygwin compile with the full 37.wav and 37_0-10_12.wav.  The cygwin compile is fine, but your compile crashes.

I'm getting really confused now! 
Title: New FLAC encoder
Post by: bukem on 2006-09-19 19:38:38
@Synthetic Soul:
if I'm not wrong wisodev has compiled two versions of flake:

1. patched (with no error) -> flake-svn-2006-09-19-patched-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-patched-win32-bin.zip)
2. and not patched (for testing purpose?) -> flake-svn-2006-09-19-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-win32-bin.zip)
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-19 20:03:27
Yes.  The patched version does work fine here.

However, what is still slightly confusing is that my cygwin compile from the SVN works fine, but wisodev's non-patched compile from the SVN does not.

NB: When I posted above I was confused about wisodev's two compiles, but I realised after posting what he was asking.  However, as you can see, I am still confused.  That said, if wisodev's patch means that all new compiles would work then I suppose that should be an end to the confusion... I don't need to worry why my compile works...  I hope.
Title: New FLAC encoder
Post by: bukem on 2006-09-19 20:52:41
@wisodev:
another problem found -> "p" switch for padding isn't working at all; it seems that is treated as "s" switch for stereo decorrelation 

Code: [Select]
D:\documents\music\.work>flake -p 256 -12 37_0-10_12.wav test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

invalid stereo decorrelation method: 2. must be 0 or 1.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.
Title: New FLAC encoder
Post by: bhoar on 2006-09-19 21:11:59
Yes.  The patched version does work fine here.

However, what is still slightly confusing is that my cygwin compile from the SVN works fine, but wisodev's non-patched compile from the SVN does not.

NB: When I posted above I was confused about wisodev's two compiles, but I realised after posting what he was asking.  However, as you can see, I am still confused.


Sometimes bugs only show up due to compiler related attributes, such as the byte-padding required for certain data types, or the order of items in the heap.  That is to say, the bug exists in both versions, but depending on which compiler compiled it, the bug might appear immediately (data overwritten was important), occassionally (data overwritten might be discarded often, or LSB of some other musical data, etc.) or not at all (due to enforced data-alignment and/or padding with unused bytes).
Title: New FLAC encoder
Post by: lexor on 2006-09-19 21:23:35
what makes even less sense to me is that Levinson algorithm supposed to solve Ax=b for x, given A and b. and that lpc_compute_coefs seems to take 2 vectors and compute the matrix.

it's computing several solutions for several Ax=b, for several x and b, from the same autocorrelation of the original signal.  that's the magic.  see also http://en.wikipedia.org/wiki/Linear_prediction (http://en.wikipedia.org/wiki/Linear_prediction)

that only furthers my confusion, it reiterates that we solving Ax=b for x, given a matrix A and a vector b. but in the function call we only have 1 vector as input, and 1 empty vector and 1 empty matrix:

compute_lpc_coefs(const double *autoc, int max_order, double lpc[][MAX_LPC_ORDER], double *ref)

autoc is the Nx1 vector that is precomputed, max_order is given int, but lpc[][] and *ref are not initialized to anything before they are passed to this function. What is matrix A and vector b then?

I guess my confusion comes from that fact that I don't understand what autoc holds after compute_autocorr is called to fill it. But even then we still apparently trying to solve Ax=b with only 1 of the 3 variables given.
Title: New FLAC encoder
Post by: pest on 2006-09-19 21:35:35
Quote
I guess my confusion comes from that fact that I don't understand what autoc holds after compute_autocorr is called to fill it.


http://en.wikipedia.org/wiki/Autocorrelation (http://en.wikipedia.org/wiki/Autocorrelation)

Quote
But even then we still apparently trying to solve Ax=b with only 1 of the 3 variables given.


the levinson-durbin algo is recursive, so you can model an unknown system as an autoregressive process
Title: New FLAC encoder
Post by: jcoalson on 2006-09-19 21:49:57
it is a two step process.  the autocorrelation step transforms the problem into the Ra=-r form from the article, this is called the autocorrelation formulation of the original Ax=b.

the autocorrelation matrix R is circulant and can be represented efficiently as a vector.  that's the input to the second step that calculates the final filter coefficients for each order with levinson/levinson-durbin recursion.

Josh

p.s. maybe the confusion is on the arguments, autoc is input (vector form of R) and lpc are the output coefficients for each order.  because of the recursive nature of the solver you can get the solution for each higher order with only incremental effort, that's why I calculate them all.  in shorten the loop aborts when the error estimate stops decreasing to save time.
Title: New FLAC encoder
Post by: lexor on 2006-09-19 22:40:42
Thank you for taking time to explain, guys. Right now my greatest confusion comes from notation they use, I can't quite grasp what -r is. From this line in the LPC explanation:

Quote
The above equations are called the normal equations or Yule-Walker equations. In matrix form the equations can be equivalently written as: Ra = -r

Yet the "above" line they are talking about seems to say that -r = -R. So are we solving Ra = -R?
Yet below that line, it says that the system Ra = -r is the same as Ra= [1 0 0 0 ... 0]. Is that why we have no vector b in function parameters, because it's assumed to be first unit vector?

in addition to all that, what the hell is R(-j) mean? what kind of a notation is that, negative can't be a valid index.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-20 02:23:46
@wisodev:
another problem found -> "p" switch for padding isn't working at all; it seems that is treated as "s" switch for stereo decorrelation 

Code: [Select]
D:\documents\music\.work>flake -p 256 -12 37_0-10_12.wav test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

invalid stereo decorrelation method: 2. must be 0 or 1.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.

I fixed that this morning 9:46 EST.  It was missing a break statement after the section which handled the '-p' parsing.
edit: maybe not...I thought I had fixed it.  Will look into further.

edit#2: This is now indeed fixed.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-20 03:04:31
the improvement is in how the signal is windowed before lpc analysis. we were both working on this at the same time but flake has a shorter release cycle. the method in the upcoming version of FLAC is slower but gives slightly more compression.

On this note, I've been considering doing an adaptive selection between several windows based on either a simple predictability measurement (e.g. energy of fixed 2nd order residual) or maybe a very coarse covariance.  Have you done any experimentation with adaptive window selection?  I know that the mp4als reference encoder does something like this, but only a binary is supplied for that part of the encoder, not the source.
Title: New FLAC encoder
Post by: yngwin on 2006-09-20 06:27:08
Users of Gentoo Linux may be interested in the ebuilds I wrote for flake, both the 0.10 release and the subversion one. See http://berkano.net/bits/2006/09/20/gentoo-overlay (http://berkano.net/bits/2006/09/20/gentoo-overlay)
Title: New FLAC encoder
Post by: bukem on 2006-09-20 07:50:03
Justin:
Thank you for the fix.

wisodev:
This may get boring after while but I'm waiting for your new native compile
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-20 07:59:10
While we () are waiting, you could try my cygwin compile.

http://synthetic-soul.co.uk/temp/flake_svn_20060920.7z (http://synthetic-soul.co.uk/temp/flake_svn_20060920.7z)

@Mangix, if you see this, I got the source using the SVN command line (I'm at home), using the command detailed on Sourceforge.  It took less than a minute.  I downloaded the SVN zip (http://subversion.tigris.org/downloads/svn-win32-1.4.0.zip), unzipped (I just unzipped to my desktop for speed), opened a command line in the "bin" ("\svn-win32-1.4.0\svn-win32-1.4.0\bin") directory and then used the command:

svn co https://svn.sourceforge.net/svnroot/flake-enc (https://svn.sourceforge.net/svnroot/flake-enc) flake-enc

This created a folder in my "bin" directory called "flake-enc".  I then moved that to somewhere more convenient (a short path with no spaces) and then ran "./configure" then "make" using a cygwin console.  A few minutes work in all.  I hope this helps.  I would be interested to see your MinGW compile (if that's the way you go).  I couldn't work out how to do it yesterday.
Title: New FLAC encoder
Post by: stephanV on 2006-09-20 12:45:12
'configure && make' is the only thing needed for MingW/MSYS AFAICT.
Title: New FLAC encoder
Post by: PrakashP on 2006-09-20 13:05:19
valgrind confirms that wisodev's patch
Code: [Select]
diff -urd flake-enc/libflake/encode.c flake-svn-2006-09-19-patched-win32-src/libflake/encode.c
--- flake-enc/libflake/encode.c 2006-09-20 10:39:45.000000000 +0200
+++ flake-svn-2006-09-19-patched-win32-src/libflake/encode.c    2006-09-19 16:59:42.000000000 +0200
@@ -341,11 +341,7 @@
     else                             ctx->lpc_precision = 15;

     /* set maximum encoded frame size in verbatim mode */
-    if(ctx->channels == 2) {
-        s->max_frame_size = 14 + ((ctx->blocksize * 33 + 7) >> 3);
-    } else {
-        s->max_frame_size = 14 + (ctx->blocksize * ctx->channels * 2);
-    }
+    s->max_frame_size = 14 + ((ctx->blocksize * ctx->channels * (ctx->bps+1)) >> 3);
     ctx->max_framesize = s->max_frame_size;

     /* select amount of padding to use in header */


fixes the issue. Otherwise I get:

Code: [Select]
==29220== Invalid write of size 4
==29220==    at 0x409EC1: output_residual (bitio.h:103)
==29220==    by 0x40A676: output_subframes (encode.c:729)
==29220==    by 0x40AF62: flake_encode_frame (encode.c:801)
==29220==    by 0x401F21: main (flake.c:378)
==29220==  Address 0x403DAB5 is 4,237 bytes inside a block of size 4,238 alloc'd
==29220==    at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220==    by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid write of size 1
==29220==    at 0x40AF76: flake_encode_frame (bitio.h:70)
==29220==    by 0x401F21: main (flake.c:378)
==29220==  Address 0x403DAB9 is 3 bytes after a block of size 4,238 alloc'd
==29220==    at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220==    by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid read of size 1
==29220==    at 0x40B7C0: calc_crc16 (crc.c:71)
==29220==    by 0x40AFC1: flake_encode_frame (encode.c:767)
==29220==    by 0x401F21: main (flake.c:378)
==29220==  Address 0x403DAB6 is 0 bytes after a block of size 4,238 alloc'd
==29220==    at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220==    by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid read of size 1
==29220==    at 0x40B7C9: calc_crc16 (crc.c:71)
==29220==    by 0x40AFC1: flake_encode_frame (encode.c:767)
==29220==    by 0x401F21: main (flake.c:378)
==29220==  Address 0x403DAB7 is 1 bytes after a block of size 4,238 alloc'd
==29220==    at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220==    by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid read of size 1
==29220==    at 0x40B7EF: calc_crc16 (crc.c:71)
==29220==    by 0x40AFC1: flake_encode_frame (encode.c:767)
==29220==    by 0x401F21: main (flake.c:378)
==29220==  Address 0x403DAB8 is 2 bytes after a block of size 4,238 alloc'd
==29220==    at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220==    by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid read of size 1
==29220==    at 0x40B81D: calc_crc16 (crc.c:71)
==29220==    by 0x40AFC1: flake_encode_frame (encode.c:767)
==29220==    by 0x401F21: main (flake.c:378)
==29220==  Address 0x403DAB9 is 3 bytes after a block of size 4,238 alloc'd
==29220==    at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220==    by 0x401EAD: main (flake.c:369)


BTW, the svn version has been speeded up by quite some amount in conrast to last release. Some minor speed-up will be commited shortly, I think. So maybe someone likes to bench.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-20 14:06:15
valgrind confirms that wisodev's patch
Code: [Select]
diff -urd flake-enc/libflake/encode.c flake-svn-2006-09-19-patched-win32-src/libflake/encode.c
--- flake-enc/libflake/encode.c 2006-09-20 10:39:45.000000000 +0200
+++ flake-svn-2006-09-19-patched-win32-src/libflake/encode.c    2006-09-19 16:59:42.000000000 +0200
@@ -341,11 +341,7 @@
     else                             ctx->lpc_precision = 15;

     /* set maximum encoded frame size in verbatim mode */
-    if(ctx->channels == 2) {
-        s->max_frame_size = 14 + ((ctx->blocksize * 33 + 7) >> 3);
-    } else {
-        s->max_frame_size = 14 + (ctx->blocksize * ctx->channels * 2);
-    }
+    s->max_frame_size = 14 + ((ctx->blocksize * ctx->channels * (ctx->bps+1)) >> 3);
     ctx->max_framesize = s->max_frame_size;

     /* select amount of padding to use in header */

While that works, it is not as accurate and does not fix the root of the problem.  See if this patch helps.
Code: [Select]
Index: libflake/bitio.c
===================================================================
--- libflake/bitio.c    (revision 2)
+++ libflake/bitio.c    (working copy)
@@ -39,7 +39,7 @@
         buf = NULL;
     }
     bw->buffer = buf;
-    bw->buf_end = bw->buffer + len;
+    bw->buf_end = bw->buffer + (((len+3)>>2)<<2); // must be 32-bit aligned
     bw->buf_ptr = bw->buffer;
     bw->bit_left = 32;
     bw->bit_buf = 0;
@@ -77,7 +77,7 @@
         fprintf(stderr, "ERROR: bits=%d val=%u\n", bits, val);
     }*/
    
-    if(bw->eof || bw->buf_ptr >= bw->buf_end) {
+    if(bw->eof || (bw->buf_ptr+3) >= bw->buf_end) {
         bw->eof = 1;
         return;
     }


edit: updated patch
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-20 14:09:11
'configure && make' is the only thing needed for MingW/MSYS AFAICT.
Yeah, no doubt.  The whole thing was new to me yesterday, and I was rushing as I was supposed to be working.  I dunno, I tried a few different things and never sucessfully compiled.  I was getting very confused between my cygwin and my mingw though!  Maybe I'll find some time to get to grips with mingw in the future.

To get this post on topic, I have performed a couple of quick tests using -5 and I would say wisodev's compile takes just over 90% as long to encode as my cygwin compile.  I guess I should have tested with the Sourceforge i686 compile as well... Also, two files were compressed to different sizes, although are bit-identical.

BTW, the svn version has been speeded up by quite some amount in conrast to last release. Some minor speed-up will be commited shortly, I think. So maybe someone likes to bench.
Good Lord it's difficult to get a version to test.  I knew I shouldn't have gotten into this!  If you could let us know when these changes are checked in that would be cool.

I only wanted to test Flake against Yalac, but it's proving a bit of a job to find a starting point.
Title: New FLAC encoder
Post by: PrakashP on 2006-09-20 14:39:09
@Justin

Your patch makes it better, but still not sufficient:
Code: [Select]
==23819== Invalid write of size 1
==23819==    at 0x40B0EC: flake_encode_frame (bitio.h:71)
==23819==    by 0x401F71: main (flake.c:378)
==23819==  Address 0x403DAB6 is 0 bytes after a block of size 4,238 alloc'd
==23819==    at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==23819==    by 0x401EFD: main (flake.c:369)
==23819==
==23819== Invalid read of size 1
==23819==    at 0x40BE29: calc_crc16 (crc.c:71)
==23819==    by 0x40B137: flake_encode_frame (encode.c:767)
==23819==    by 0x401F71: main (flake.c:378)
==23819==  Address 0x403DAB6 is 0 bytes after a block of size 4,238 alloc'd
==23819==    at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==23819==    by 0x401EFD: main (flake.c:369)


Good Lord it's difficult to get a version to test.  I knew I shouldn't have gotten into this!  If you could let us know when these changes are checked in that would be cool.


The major speed-ups are already checked in.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-20 14:57:37
The major speed-ups are already checked in.
OK, thanks.

What about the errors with Justin's patch?  Will they be included?

Edit: K.  I had another go with MinGW and it worked just fine.  I think my earlier mistake was simply not using the MSYS console to run the apps.

I have checked using objdump and get the following:

Code: [Select]
$ objdump -p flake.exe | grep "DLL Name"
        DLL Name: KERNEL32.dll
        DLL Name: msvcrt.dll
        DLL Name: msvcrt.dll
(Note the reliance on msvcrt.dll and not cygwin1.dll.)
       
Compiles using current SVN (rev 15): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_20060920b.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_20060920b_mingw.7z)
Title: New FLAC encoder
Post by: jcoalson on 2006-09-20 21:07:31
the improvement is in how the signal is windowed before lpc analysis. we were both working on this at the same time but flake has a shorter release cycle. the method in the upcoming version of FLAC is slower but gives slightly more compression.

On this note, I've been considering doing an adaptive selection between several windows based on either a simple predictability measurement (e.g. energy of fixed 2nd order residual) or maybe a very coarse covariance.  Have you done any experimentation with adaptive window selection?  I know that the mp4als reference encoder does something like this, but only a binary is supplied for that part of the encoder, not the source.

I haven't tried anything like that.  I did a brute force exhaustive search test (try all windows for each frame, pick best one) to see what the lower bound was and it didn't end up yielding that much more compression.

when it comes to adding fancy like adaptive schemes I start to worry about patents since currently in libFLAC the encoder and decoder ship together.

Josh
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-21 01:47:56
The major speed-ups are already checked in.
OK, thanks.

What about the errors with Justin's patch?  Will they be included?

Based on Prakash's latest test I didn't quite solve the issue.  I'll keep working on it and hopefully be able to apply a definite fix this evening.  I just have to find a test sample which makes the valgrind complaint occur.  I think it's only very noisy samples which have to be encoded in verbatim mode so as not to actually inflate the file size.

Edit: This issue should be completely fixed in SVN now.
 
the improvement is in how the signal is windowed before lpc analysis. we were both working on this at the same time but flake has a shorter release cycle. the method in the upcoming version of FLAC is slower but gives slightly more compression.

On this note, I've been considering doing an adaptive selection between several windows based on either a simple predictability measurement (e.g. energy of fixed 2nd order residual) or maybe a very coarse covariance. Have you done any experimentation with adaptive window selection? I know that the mp4als reference encoder does something like this, but only a binary is supplied for that part of the encoder, not the source.

I haven't tried anything like that. I did a brute force exhaustive search test (try all windows for each frame, pick best one) to see what the lower bound was and it didn't end up yielding that much more compression.

when it comes to adding fancy like adaptive schemes I start to worry about patents since currently in libFLAC the encoder and decoder ship together.

Josh

  Thanks.  That's good to know.  It may not be worth the research/testing time if the gain is only trivial.
   
    -Justin
Title: New FLAC encoder
Post by: HbG on 2006-09-21 02:38:35
Since i don't have much to add to the technical discussion, all i'll say is this: It'd be very useful to have someone providing proper daily builds, also because this makes promoting flake usage much easier and prompts people to give more feedback. Then i'd happily put my Athlon XP to benchmarking.

As flake is not a reference encoder, patents are less of a worry, right?
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-21 07:36:27
Edit: This issue should be completely fixed in SVN now.
Thanks Justin.

Compiles using current SVN (rev 27): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_r27_cygwin.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_r27_mingw.7z)

I should point out that I'm just posting these for interest.  Due to the speed improvements of wisodev's builds I would wait for him to post if you want to do comparison testing.
Title: New FLAC encoder
Post by: wisodev on 2006-09-21 15:21:15
FLAKE SVN REVISION 27

Download binarys:
flake-svn-r27-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r27-win32-bin.zip)

Download sources:
flake-svn-r27-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r27-win32-src.zip)

wiso
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-21 15:31:37
Cool, thanks wisodev.

I think this is the version I'll use to compare with Yalac, unless anyone can tell me of any major changes being checked in in the next day or so?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-21 15:39:49
Cool, thanks wisodev. 

I think this is the version I'll use to compare with Yalac, unless anyone can tell me of any major changes being checked in in the next day or so?

Well, I do have the day off work, so maybe I'll take that as a challenge and work on Flake.
At any rate, if I do implement the next big thing on my list I will probably do another version release several days after.  There is 1 change I'll work on this morning that you may want to wait for though.  The low-compression levels (0 to 3) will probably get faster.

-Justin
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-21 16:16:06
 Sounds like I best hold fire.

Good luck with the day off.
Title: New FLAC encoder
Post by: TBeck on 2006-09-22 00:45:11
Sounds like I best hold fire.

Sorry if i should be asking for too much. You allready have done so much to help me!

Yalac V0.12 will soon be done. After the integration of a seek table and the final tuning of the presets it's performance should not change anymore until the first public release (ok, it may get a bit faster when i disable my debug code).

A test of V0.12 isn't urgent for me, but  if it would be easier for you to perform both tests simultaneously, i could work a bit faster.

Thanks

  Thomas
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-22 06:28:18
Well, I've gotten a lot done today.  I just applied code to SVN for variable block size.  You can test it out using "-v 1" on the commandline.  It has much less of an adverse effect on speed than I had anticipated, which is great.  Hopefully in the future I'll be able to enhance the algorithm for the block splitting decision.  This works pretty well for a first attempt though.

One down-side is that the resulting files are not within the FLAC Subset format.  They do, however, decode just fine with the libFLAC and FFmpeg decoders.  Hardware support may be sporatic though.

I'm not going to add anything big for a few days and just focus on bug fixes and small changes.  Then I'll release a new version.

-Justin
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-22 07:42:57
Compiles using current SVN (rev 42): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_r42_cygwin.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_r42_mingw.7z)

Sorry if i should be asking for too much. You allready have done so much to help me!
Hey, no worries Thomas.  I am doing this for my own interest, although of course I'm hoping it will benefit you (and Justin and Josh perhaps).

A test of V0.12 isn't urgent for me, but  if it would be easier for you to perform both tests simultaneously, i could work a bit faster.
No, that's fine Thomas, I wouldn't want to rush you.  It doesn't make too much difference to me if I perform the tests at different times.  Thanks anyway.

Well, I've gotten a lot done today.  I just applied code to SVN for variable block size.  You can test it out using "-v 1" on the commandline.  It has much less of an adverse effect on speed than I had anticipated, which is great.
...
I'm not going to add anything big for a few days and just focus on bug fixes and small changes.  Then I'll release a new version.
Cool, sounds interesting.  Once I have wisodev's build I'll set some scripts running.
Title: New FLAC encoder
Post by: bukem on 2006-09-22 08:22:20
@Justin:
I'll check new build on my iAudio U3 this evening. I'm just curious if it's able to handle variable block sized flac files.

Edit: Huston we have a problem! I've just downloaded Synth's compile and -v switch IS NOT SUPPORTED

Code: [Select]
D:\documents\music\.work>flake -p 256 -12 -v 1 "01. autechre - kalpol introl.wav" "01. autechre - kalpol introl.flac"

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

invalid option: -v
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.


Edit2: I've checked the SVN (v42) and found -v switch implemented in flake.c - seems that there's something wrong with Synth's compile...
Title: New FLAC encoder
Post by: bukem on 2006-09-22 13:59:23
OK, I've managed to compile flake v42 by myself and to do a quick test with variable block sized flac on my iAudio U3 - and I'm simply astonished because it just WORKS!  There is small glitch though, the player doesn't display remaining play time of the track anymore but that's not a problem for me.

FYI: I've converted Thunderball's album "Scorpio Rising" with flake -p 256 -12 -v 1. I just love my iAudio U3 
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-22 14:35:44
Edit2: I've checked the SVN (v42) and found -v switch implemented in flake.c - seems that there's something wrong Synth's compile...
How bloody annoying!  Only time they may have been of use and they weren't right!  I have recompiled, checked the -v switch, and uploaded, just in case.  I have no idea what happend this morning, I'm sure some files had updated...

Thanks for highlighting the error.

Edit: OK, I've just done some (after the horse has bolted) testing with -v 1 at -0, -5, -8 and -12 and in all cases the -v versions are bigger than the fixed block size versions.  Am I missing something?

E.g.:

Code: [Select]
C:\C\Flake\SVN\flake>flake-svn-cygwin -5 "C:\Documents and Settings\Neil\My Docu
ments\Audio\Flac Corpus\source\20.wav" 5_no.flac

Flake: FLAC audio encoder
© 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 2743608

block size: 4608
variable: no
prediction type: levinson-durbin
prediction order: 1,8
partition order: 0,6
order method: estimate
stereo method: mid-side
header padding: 4096

progress: 100% | ratio: 0.413 | bitrate: 583.2 kbps | bytes: 4525937


C:\C\Flake\SVN\flake>flake-svn-cygwin -5 -v 1 "C:\Documents and Settings\Neil\My
 Documents\Audio\Flac Corpus\source\20.wav" 5_yes.flac

Flake: FLAC audio encoder
© 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 2743608

** WARNING! FLAC file will not be Subset compliant **

block size: 4608
variable: yes
prediction type: levinson-durbin
prediction order: 1,8
partition order: 0,6
order method: estimate
stereo method: mid-side
header padding: 4096

progress: 100% | ratio: 0.414 | bitrate: 583.5 kbps | bytes: 4528149
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-22 15:54:44
Edit: OK, I've just done some (after the horse has bolted) testing with -v 1 at -0, -5, -8 and -12 and in all cases the -v versions are bigger than the fixed block size versions.  Am I missing something?

Hmmm.  It worked for my samples.  I'll do more testing tonight.  I may temporarily add a variable frame-split-threshold until I can figure things out.

Thanks for the feedback,
-Justin
Title: New FLAC encoder
Post by: bukem on 2006-09-22 16:42:12
In my case I saved 1 Mbyte of 319 Mbytes flac album when converted with -v 1 switch.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-22 16:43:36
Justin, my post was a bit premature.  I was just a little surprised at the (minor) increase in size over various compression levels, so posted without really thinking.

What I totaly forgot was that this was just one file.

To rectify I have just tested -5 on my "FLAC corpus" (28 files) and the results are a lot better.

Out of the 28 files 12 files suffer with the -v switch, but on the whole the compression improves from 59.843% to 59.754%, or 1,221,384 bytes in 1,371,283,088 (a 0.001% improvement).

As you can see from the figures, those that suffer from variable blocks (-v 1) are the files at the outer edges - generally those that compress the best (e.g.: Charlotte Church, Andrea Bocelli), but also two Motorhead tracks (-274 and -5,698).  This variation reminds me of the window testing I was involved in for FLAC 1.1.2_CVS (1.1.3), with the two bounds acting differently than the core files... not sure if that's relevant.

When wisodev is kind enough to provide a compile of Rev. 42 I'll soon provide some more figures, using my Yalac corpus, including timings and using various "key" compression levels.

Code: [Select]
Wav            Off                    On
               Filesize     Comp %    Filesize     Comp %       Diff
====================================================================
  44205884     30037774    67.950%    29996667    67.857%     41,107
  40501484     18854486    46.553%    18860045    46.566%     -5,559
  41860940     32194824    76.909%    32179874    76.873%     14,950
  36997004     11961940    32.332%    11965877    32.343%     -3,937
  41849180     22570842    53.934%    22499530    53.763%     71,312
  43095740     31742309    73.655%    31646473    73.433%     95,836
  98031404     52089827    53.136%    52103883    53.150%    -14,056
  29964524     22412071    74.795%    22412345    74.796%       -274
  44963228     18196142    40.469%    18207104    40.493%    -10,962
  37462700     25086008    66.963%    24958122    66.621%    127,886
  35209484     24646149    69.999%    24626419    69.943%     19,730
  39280796     26879523    68.429%    26836293    68.319%     43,230
  35898620     24982661    69.592%    24917519    69.411%     65,142
  44295260     30913592    69.790%    30808634    69.553%    104,958
.134433308     84608716    62.937%    84103316    62.561%    505,400
  46214492     33193196    71.824%    33150151    71.731%     43,045
  70752908     31462354    44.468%    31470864    44.480%     -8,510
  37989548     22986231    60.507%    22940337    60.386%     45,894
  56989004     29822007    52.329%    29827193    52.339%     -5,186
  47973788     28413887    59.228%    28367289    59.131%     46,598
  10974476      4525937    41.241%     4528149    41.261%     -2,212
  45400700     17135222    37.742%    17141711    37.756%     -6,489
  79469420     56785940    71.456%    56754994    71.417%     30,946
  35719868     26702556    74.755%    26675013    74.678%     27,543
  32309468     13942750    43.154%    13947729    43.169%     -4,979
  73810508     30283837    41.029%    30312140    41.068%    -28,303
  55389644     44922899    81.103%    44928597    81.114%     -5,698
  30239708     23258281    76.913%    23224309    76.801%     33,972
====================================================================
1371283088    820611961    59.843%   819390577    59.754%    1221384
Title: New FLAC encoder
Post by: wisodev on 2006-09-22 18:17:22
FLAKE SVN REVISION 42

Download binarys:
flake-svn-r42-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r42-win32-bin.zip)

Download sources:
flake-svn-r42-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r42-win32-src.zip)

wiso
Title: New FLAC encoder
Post by: jcoalson on 2006-09-22 18:46:24
I would caution against encoding variable-blocksize files in native FLAC unless familiar with the implications.  there is a design flaw in the native FLAC container that can make variable blocksize files difficult to handle in some ways (I may fix this later but it's a format change).  this is not a problem when in other containers e.g. Ogg FLAC.

definitely I would say that a fraction of a percent size decrease is not worth the useability tradeoff.

Josh
Title: New FLAC encoder
Post by: TBeck on 2006-09-22 19:46:31
I tested wisodev's build on my two primary file sets as before flake V0.10.

Participiants

Yalac V0.11a  slightly improved over V0.11.
FLAC  1.1.2
Flake 0.10    file flake-0.10-win32-i686.zip from sourceforge.
Flake sv42    flake-svn-r42-win32-bin.zip from wisodev.

Results

Code: [Select]
          Yalac V0.11a |  FLAC  |  Flake 0.10   |  Flake sv42   |
          Turbo  Fast  |  -8    |  -8     -12   |  -8     -12   |
-----------------------+--------+---------------+---------------+
rw                     |        |               |               |
Ratio:    58.03  57.14 |  59.54 |  59.06  58.74 |  58.88  58.51 |
-----------------------+--------+---------------+---------------+
songs                  |        |               |               |
Ratio:    49.09  48.37 |  51.35 |  50.27  49.59 |  50.27  49.58 |
-----------------------+--------+---------------+---------------+

Ratio is the compression ratio in percent. No speed data this time, because i am not sure, if both flake versions have been bulit with the same compiler optimizations. Subjectively Flake sv42 with -v 1 seems not to be significantly slower than V0.10.

For the test sets:

"rw" contains 46 files from http://www.rarewares.org/test_samples/ (http://www.rarewares.org/test_samples/). Should be the whole dir. I call this my primary test set and prefer to use it for my tuning.

"songs" contains CD-Ripps of the beginning of some songs. This set contains many files that are benefiting from higher predictor orders. It may be less representative than set rw, but i keep it because it has been my primary set for many years.

Some remarks

Up to 0.23 percent better compression for sample set "rw", only 0.01 better compression for "songs".

Not too surprising for me: "rw" contains many samples with very fast changes of the signal characteristics, where variable block sizes should help most.

Possibly Justin could try to use some of the "rw" files for further optimizations of the block length switching. I found them very useful for this.

"songs" contains files, which can quite well be encoded with constant frame sizes of up to 200 ms.

Keep on the great work!

  Thomas
Title: New FLAC encoder
Post by: TBeck on 2006-09-22 21:40:26
Here the results of set "rw" for each file:

Code: [Select]
                     |  FLAC  |  Flake 0.10   | Flake sv42 -v1|
                     |  -8    |  -8     -12   |  -8     -12   |
---------------------+--------+---------------+---------------+
41_30sec.flac          65.56     65.77  65.66    65.33  65.20
ATrain.flac            51.58     50.08  49.66    49.87  49.34
Bachpsichord.flac      64.67     64.20  63.33    63.99  63.08
Bartok_strings2.flac   53.29     51.85  51.16    51.94  51.14
BeautySlept.flac       63.37     63.35  62.81    63.25  62.68
BigYellow.flac         75.40     75.12  74.94    74.81  74.61
Blackwater.flac        60.64     59.43  59.30    59.38  59.21
bodyheat.flac          59.79     59.32  58.86    59.51  58.95
chanchan.flac          64.35     64.08  64.01    63.77  63.69
DaFunk.flac            66.87     66.87  66.79    66.13  66.00
death2.flac            40.31     41.52  41.03    38.94  38.58
Debussy.flac           28.54     27.95  27.83    28.01  27.84
EnolaGay.flac          65.48     65.17  63.54    65.40  63.80
experiencia.flac       71.82     71.65  71.47    71.66  71.46
female_speech.flac     34.15     34.47  34.08    34.06  33.44
FloorEssence.flac      64.68     64.23  64.13    64.12  63.99
getiton.flac           53.79     53.15  53.02    52.70  52.56
gone.flac              65.67     65.12  65.06    65.09  65.01
Hongroise.flac         40.84     36.65  36.45    36.53  36.31
Illinois.flac          59.58     59.19  58.86    59.23  58.84
ItCouldBeSweet.flac    56.86     56.65  56.46    56.95  56.65
kraftwerk.flac         46.50     46.82  46.34    45.84  45.40
Layla.flac             65.93     65.59  65.21    65.18  64.80
Leahy.flac             73.84     73.82  73.75    73.76  73.67
LifeShatters.flac      70.58     69.95  69.78    70.06  69.83
macabre.flac           63.27     62.80  62.02    62.89  62.04
Mahler.flac            49.86     48.91  48.86    48.85  48.80
male_speech.flac       36.28     36.36  36.08    35.94  35.57
Mama.flac              66.85     66.47  66.43    66.47  66.41
MidnightVoyage.flac    62.40     61.92  61.23    61.91  61.13
mybloodrusts.flac      70.14     70.01  69.94    70.08  69.97
NewYorkCity.flac       78.30     78.33  78.14    78.36  78.18
OrdinaryWorld.flac     67.66     67.22  66.59    66.77  66.16
Polonaise.flac         36.50     33.94  33.46    34.05  33.46
Quizas.flac            60.78     60.66  60.55    59.91  59.77
riteofspring.flac      51.95     53.39  53.33    53.32  53.23
rosemary.flac          55.77     55.31  55.16    55.19  55.01
Scars.flac             59.81     59.16  58.62    59.06  58.53
SinceAlways.flac       71.10     70.69  70.64    70.28  70.21
thear1.flac            74.01     73.64  73.61    73.64  73.62
TheSource.flac         57.28     57.05  56.88    56.61  56.43
TomsDiner.flac         53.73     53.81  52.96    53.85  52.82
trust.flac             61.49     60.71  60.64    60.70  60.59
Twelve.flac            72.16     71.85  71.55    71.96  71.53
velvet.flac            68.12     68.12  67.70    67.70  67.48
Waiting.flac           67.67     67.18  66.55    67.19  66.46
Title: New FLAC encoder
Post by: guruboolez on 2006-09-22 21:48:16
Code: [Select]
                     |  FLAC  |  Flake 0.10   | Flake sv42 -v1|
                     |  -8    |  -8     -12   |  -8     -12   |
---------------------+--------+---------------+---------------+
...
riteofspring.flac      51.95     53.39  53.33    53.32  53.23
...


 
I guess that it can't be a typo, but could you confirm these values?
Title: New FLAC encoder
Post by: TBeck on 2006-09-22 21:53:49
Code: [Select]
                     |  FLAC  |  Flake 0.10   | Flake sv42 -v1|
                     |  -8    |  -8     -12   |  -8     -12   |
---------------------+--------+---------------+---------------+
...
riteofspring.flac      51.95     53.39  53.33    53.32  53.23
...


 
I guess that it can't be a typo, but could you confirm these values?

Yes, they are right. This file is one of the very rare cases, where a 16 bit file (or a part of it) contains a constant least significant bit, which can be removed by Flac's "Look for wasted bits" option. Probably in Flake this option is not implemented.
Title: New FLAC encoder
Post by: guruboolez on 2006-09-22 21:57:17
Confirmation... and explanation! Thanks 

N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?
Title: New FLAC encoder
Post by: TBeck on 2006-09-22 22:07:10
N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?

I have not performed a deeper analysis of this file. Possibly one channel has a constant least significant bit (LSB), because compression can be improved by about 3 percent by the wasted bits option. If both channels were affected, it would be about 6 percent.

"That the 16th bit is always the same through the whole file?" Through the whole file or through a whole frame (my notation) or block (Flac's notation). Usually the analysis is beeing performed on a frame basis (implementaion dependend). And theoretically more than 1 bit can be constant.

"In other words 15 bit recording pad to 16 bit?" If someone multiplies each sample with 2 (or shifts it by one place), the least significant bit will become zero:
Code: [Select]
  decimal  binary
    9        1001
   18       10010
Title: New FLAC encoder
Post by: Will Fisher on 2006-09-23 01:01:58
new winamp encoder plugin as per wizos drop: http://stashbox.org/uploads/1158969212/enc_flac.dll (http://stashbox.org/uploads/1158969212/enc_flac.dll)

although i ran into a couple of problems compiling.

Would it be too much of a problem to put my changes into the code, to keep it portable across compilers?

optimize.c line 219 replaced with: uint32_t bits[32]; (where 32 used to be levels. MSVC doesn't like non-constant sizes for these things.) I am sure that 32 is as large as levels gets.

in encode.c flake_set_defaults, msvc doen't like this syntax:
foo = ((int[]){1,2,3})[2];
so i defined all these "nameless" arrays as ones with names, so it becomes:
int sel[] = {1,2,3};
foo = sel[2];

thanks,
will
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-23 02:07:28
I would caution against encoding variable-blocksize files in native FLAC unless familiar with the implications.  there is a design flaw in the native FLAC container that can make variable blocksize files difficult to handle in some ways (I may fix this later but it's a format change).  this is not a problem when in other containers e.g. Ogg FLAC.

definitely I would say that a fraction of a percent size decrease is not worth the useability tradeoff.

Josh

I agree.  The variable block size headers are a bit tricky.  I don't have any issue with using it for my personal files because I know the players I use support it.  If I was sharing the files with other people I would definitely not use that option though.  Perhaps I should change the warning to something less technical like "Warning! The chosen options are not part of the FLAC Subset format and therefore may not be compatible with some FLAC players."


N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?

I have not performed a deeper analysis of this file. Possibly one channel has a constant least significant bit (LSB), because compression can be improved by about 3 percent by the wasted bits option. If both channels were affected, it would be about 6 percent.

"That the 16th bit is always the same through the whole file?" Through the whole file or through a whole frame (my notation) or block (Flac's notation). Usually the analysis is beeing performed on a frame basis (implementaion dependend). And theoretically more than 1 bit can be constant.

"In other words 15 bit recording pad to 16 bit?" If someone multiplies each sample with 2 (or shifts it by one place), the least significant bit will become zero:
Code: [Select]
  decimal  binary
    9        1001
   18       10010

I think I may have removed that feature because I could not find any files which had any wasted bits...I can't really remember.  I at least know I had implemented it at one point, but it's definitely no longer there.  It would be easy enough to add.

As far as the compression gain for variable block size, I have a feeling I can do better.  This was a first attempt and pretty much a shot-in-the-dark.  What I think I'll do next is a really slow brute-force approach so that I will have something to compare results against.

Thanks for all the great results.  It would be interesting to see some speed comparisons.  I know that the current SVN version is quite a bit faster on my machine than 0.10, but my machine is certainly not typical (Linux, AMD K6-2).

-Justin
Title: New FLAC encoder
Post by: TBeck on 2006-09-23 02:42:39
Thanks for all the great results.  It would be interesting to see some speed comparisons.  I know that the current SVN version is quite a bit faster on my machine than 0.10, but my machine is certainly not typical (Linux, AMD K6-2).

Same as before with encoding times.

Participiants

FLAC  1.1.2
Flake 0.10    file flake-0.10-win32-i686.zip from sourceforge.
Flake sv42    flake-svn-r42-win32-bin.zip from wisodev.

Test system

P3-800 with a (slow) 40 GB HD.

Code: [Select]
            FLAC  |  Flake 0.10   | Flake sv42 -v1|
            -8    |  -8     -12   |  -8     -12   |
------------------+---------------+---------------+
rw                |               |               |
Ratio:      59.54 |  59.06  58.74 |  58.88  58.51 |
EncoTime:  280.47 | 133.57 871.84 |  79.03 362.25 |
------------------+---------------+---------------+
songs             |               |               |
Ratio:      51.35 |  50.27  49.59 |  50.27  49.58 |
EncoTime:  185.51 |  86.14 591.69 |  50.76 241.82 |
------------------+---------------+---------------+


Can -12 in combination with -v1 be so much faster now?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-23 05:56:59
Can -12 in combination with -v1 be so much faster now?

Yes indeed.  Level 12 makes more use of 2 functions which were optimized since version 0.10.  The different binaries might be making some of that difference as well.

BTW...where might I find the "rw" corpus?

-Justin
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-23 07:46:18
Hi,
Flake r46 has a new variable block size algorithm, which is selected by using "-v 2".  It gives a lower-bound for even 3-level splitting (splitting frames exactly in half up to 3 times).  Encoding takes about 4x as long.  Hopefully I will be able to use this to help improve -v 1 by comparing the resulting files.  Future developments might also include uneven splitting, which could better isolate changes within a frame with less overhead.
-Justin

Edit: In r47 I modified -v 1 to give better compression.
Title: New FLAC encoder
Post by: guruboolez on 2006-09-23 10:03:30
Justin Ruggles> BTW...where might I find the "rw" corpus?

here (http://www.rjamorim.com/test/samples/).
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-09-24 11:26:45
Compiles using current SVN (rev 48): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_r48_cygwin.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_r48_mingw.7z)


I have added Flake SVN Rev.42 (wisodev build) to my Yalac comparison.  Please see this post on the Yalac thread (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=43494&view=findpost&p=434373) for info.

Here's the FLAC-related results in compression order:

Code: [Select]
Encoder Setting    Compression    Encode    Decode
==================================================
Flake -12 -v 1         65.259%     7.29x    70.41x
Flake -12              65.368%     7.38x    69.31x
Flake -11              65.376%    19.10x    69.19x
FLAC 1.1.2_CVS -9      65.528%     5.73x    71.34x
Flake -8 -v 1          65.530%    37.94x    74.49x
Flake -10              65.555%    24.82x    73.63x
Flake -9               65.559%    35.48x    75.07x
FLAC 1.1.2_CVS -8      65.611%     9.25x    70.96x
Flake -8               65.658%    41.21x    74.93x
Flake -5 -v 1          65.668%    64.70x    74.89x
Flake -5               65.798%    69.44x    75.44x
FLAC 1.1.2_CVS -5      65.916%    36.75x    71.46x
FLAC -8                66.028%     9.43x    71.42x
FLAC -5                66.279%    38.44x    72.05x
FLAC 1.1.2_CVS -0      70.734%    69.34x    72.35x
FLAC -0                70.734%    68.97x    75.84x
Flake -0               71.679%    82.18x    71.13x
Flake -0 -v 1          71.728%    76.70x    72.80x
Title: New FLAC encoder
Post by: wisodev on 2006-09-24 11:40:29
FLAKE SVN REVISION 48

Download binarys:
flake-svn-r48-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r48-win32-bin.zip)

Download sources:
flake-svn-r48-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r48-win32-src.zip)

wiso
Title: New FLAC encoder
Post by: krmathis on 2006-09-24 12:21:19
Justin. I pulled revision 2 from SVN, and tried to build this on Mac OS 10.47 PowerPC/ GCC 4.0.1 (Xcode Tools 2.4). But it errors out like this:
Quote
$ ./configure
source path      /Users/krmathis/Downloads/flake-enc
C compiler      cc
make            make
CPU              powerpc (generic)
big-endian      yes
inttypes.h      yes
AltiVec enabled  yes
debug symbols    yes
strip symbols    yes
optimize        yes
Creating config.mak and config.h...
$ make
make -C libflake all

** Snip **

cc -O3  -no-cpp-precomp -pipe -force_cpusubtype_ALL -Wno-sign-compare -fomit-frame-pointer -faltivec -g -Wdeclaration-after-statement -Wall -Wno-switch -mdynamic-no-pic -DHAVE_CONFIG_H -I. -I.. -I/Users/krmathis/Downloads/flake-enc/libflake -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -c -o wav.o wav.c
cc -L../libflake -Wl,-dynamic,-search_paths_first  -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2


I successfully compiled flake 0.7 two days ago. But the build system have changed since then (from "make" to "./configure && make")

Any idea whats wrong?
B.U.M.P.
Revision 48 still fails with the same error...   
Title: New FLAC encoder
Post by: HbG on 2006-09-24 13:18:10
Code: [Select]
Encoder Setting    Compression    Encode    Decode
==================================================
Flake -0               71.679%    82.18x    71.13x


  I  think flac's decoder is quite inefficient...  With encoding speed so vastly improved decoding speed may start to hold it back in the great league of lossless formats.
Title: New FLAC encoder
Post by: MedO on 2006-09-24 13:50:53
I really like what I see about Flake develpment, great work! However, I don't know if these SVN builds are safe to use and won't produce "mistakes" with some files. Is there a simple and fast way to verify that the produced files will decode bit-identical and without errors? The official Flac encoder has a verify option to check this IIRC.
Title: New FLAC encoder
Post by: TBeck on 2006-09-24 14:10:38
Code: [Select]
Encoder Setting    Compression    Encode    Decode
==================================================
Flake -0               71.679%    82.18x    71.13x


  I  think flac's decoder is quite inefficient...  With encoding speed so vastly improved decoding speed may start to hold it back in the great league of lossless formats.

Am i right? You are wondering, why decoding isn't faster than encoding?

I have to go a bit technical, to explain it.

To simplify it:

The encoder has to perform tree operations:

A) Analyze the signal and select an optimal prediction method.
B) Prediction: Predict the next sample and calculate the difference  (residual) between the prediction and the true sample.
C) BitIO: Write the residual into the bitstream (file).

The decoder has to perform two operations:

A) BitIO: Read the residual from the bitstream (file).
B) Prediction: Predict the next sample and add the residual from (A) to get the true sample value.

Differences between encoding and decoding:

For Flake -0, signal analysis (A) of the encoder is very simple and fast.

The calculations encoder and decoder have to perform for the prediction of the fast -0 mode are absolutely simple, can be performed ultra fast and should be equally fast on the encoder and decoder side.

But the difference of the processing requirements for the prediction in encoder and decoder is quite small compared to the difference for the BitIO!

One important operation, processors have to perform, is a conditional jump. It is needed, if -depending on a certain condition- two different actions have to be performed.

But modern processors like to know as soon as possible what will be going on later. In the case of a conditional jump, they had to wait until the critical condition has been calculated. Because they don't want to wait, they try to predict the result of the condition based upon their knowledge about prior excecutions of the
conditional jump.

This works quite well, if the condition follows a simple (repetitive) pattern, for instance: The condition is 5 times true, than 1 time false, than 5 times true. Then the conditional jump will be called predictable.

The worst case is a conditional jump, which is following an unpredictable pattern. Modern processors may have to wait 20 and more clock cycle, if they have performed a misprediction!

Compare this to the about 0.5 clock cycles needed for simple operations like additions or substractions!

What is important: The BitIO of the encoder needs far less conditional jumps (if well implemented) than the decoder, and the jumps on the decoder side will be unpredictable most of the time.

A simplified example:

On average 1 mispredictions per sample on the encoder side = 20 clock cycles
On average 3 mispredictions per sample on the dncoder side = 60 clock cycles

And possibly 5 simple cpu operations for the prediction: 5 * 0.5 = 2.5 clock cycles.

Sorry, my english is too limited for a good explaination. I hope, you can understand.
Title: New FLAC encoder
Post by: HbG on 2006-09-24 14:15:47
MedO: Use FlacTester (http://www.vuplayer.com/files/flactester.zip). Just give it a directory and it will recursively work it's way through it and all subdirs testing all flacs it finds. Ideal for keeping your flac collection in proper working order.

TBeck: thanks for the explanation, i understand. Indeed it seemingly goes against nature to have faster encoding than decoding..
It would also mean the difference should be even greater for P4 processors.
Title: New FLAC encoder
Post by: MedO on 2006-09-24 14:53:44
MedO: Use FlacTester (http://www.vuplayer.com/files/flactester.zip). Just give it a directory and it will recursively work it's way through it and all subdirs testing all flacs it finds. Ideal for keeping your flac collection in proper working order.


Thanks for the tip. So I can rely on Flake producing correct checksums? Just want to be sure :-)
Title: New FLAC encoder
Post by: HbG on 2006-09-24 15:25:22
It's a md5 hash, if flake manages to produce a corrupt hash that fits the corrupt output, i'd be very impressed.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-24 16:06:30
Justin. I pulled revision 2 from SVN, and tried to build this on Mac OS 10.47 PowerPC/ GCC 4.0.1 (Xcode Tools 2.4). But it errors out like this:$ ./configure
source path      /Users/krmathis/Downloads/flake-enc
C compiler      cc
make            make
CPU              powerpc (generic)
big-endian      yes
inttypes.h      yes
AltiVec enabled  yes
debug symbols    yes
strip symbols    yes
optimize        yes
Creating config.mak and config.h...
$ make
make -C libflake all

** Snip **

cc -O3  -no-cpp-precomp -pipe -force_cpusubtype_ALL -Wno-sign-compare -fomit-frame-pointer -faltivec -g -Wdeclaration-after-statement -Wall -Wno-switch -mdynamic-no-pic -DHAVE_CONFIG_H -I. -I.. -I/Users/krmathis/Downloads/flake-enc/libflake -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -c -o wav.o wav.c
cc -L../libflake -Wl,-dynamic,-search_paths_first  -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2

Sorry...I did not see this originally.  It looks like it might have to do with byteswap.h.  I'll look into it.
Title: New FLAC encoder
Post by: krmathis on 2006-09-24 19:14:42
Sorry...I did not see this originally.  It looks like it might have to do with byteswap.h.  I'll look into it.

Thank you!
Title: New FLAC encoder
Post by: jcoalson on 2006-09-25 06:41:01
It's a md5 hash, if flake manages to produce a corrupt hash that fits the corrupt output, i'd be very impressed.

md5 is only useful as an error-detection mechanism after a verified-good encoding.  there are many ways the encoder can mess up the data going in and still create a file with a correct md5.  that's why there's a --verify mode in flac, and why the flac test suite uses --verify as well as encodes/decodes/compares many thousands of different streams/encoding combinations before a release.

Josh
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-25 07:15:18
It's a md5 hash, if flake manages to produce a corrupt hash that fits the corrupt output, i'd be very impressed.

md5 is only useful as an error-detection mechanism after a verified-good encoding.  there are many ways the encoder can mess up the data going in and still create a file with a correct md5.  that's why there's a --verify mode in flac, and why the flac test suite uses --verify as well as encodes/decodes/compares many thousands of different streams/encoding combinations before a release.

Josh

I definitely agree.  It's not nearly as hard as it may seem to produce a false MD5 which matches decoded output.  This can happen if the wav data gets corrupted (by either program bugs or by machine-related errors) anywhere between the wav file read and the MD5 update.

If I ever have the time and there is a demand for it, I might import FFmpeg's FLAC decoder into Flake for verification during encoding.  For now I'll add it to the TODO list.

-Justin
Title: New FLAC encoder
Post by: sundance on 2006-09-25 07:27:40
HbG,
Quote
I think flac's decoder is quite inefficient... With encoding speed so vastly improved decoding speed may start to hold it back in the great league of lossless formats.

You could be right.
Some time ago I did some tests with flacdec (http://phoenix.inf.upol.cz/~lichvarm/affd/), and despite running it under cygwin, it was faster than flac -d. Since I don't have the necessary tools to build an optimized win32-compilation, I didn't test further...

[edit]
Here's my test results (12 flac files, encoded with flac -8, totalling 447.635.340 bytes) for decoding to nul:
a) flac -d -o nul [file]: 34,31 sec (avg.)
b) flacdec -o nul [file]: 12,21 sec (avg.)
Although flacdec doesn't "check MD5 signature of audio data", I don't think that's the culprit...

.sundance.
Title: New FLAC encoder
Post by: MedO on 2006-09-25 08:43:47
Quote
I definitely agree.  It's not nearly as hard as it may seem to produce a false MD5 which matches decoded output.  This can happen if the wav data gets corrupted (by either program bugs or by machine-related errors) anywhere between the wav file read and the MD5 update.

If I ever have the time and there is a demand for it, I might import FFmpeg's FLAC decoder into Flake for verification during encoding.  For now I'll add it to the TODO list.

-Justin


So the only way I can be sure is checking the decoded audio against the original? I guess I could write a script to do this, but that would probably end any speed gain I got with Flake.
Title: New FLAC encoder
Post by: HbG on 2006-09-25 13:14:36
Whoops, i stand corrected on the MD5's. I didn't think of the wav data getting corrupted before being hashed, although technically that does not indicate a fault in the encoding routines.

Sundance, you're right, i had even used flacdec before and forgot about it. As it's not a reference decoder i'm not using it in combination with flake. Still a better compile would be nice to see just what it can do, it does decode variable blocksize properly and without measurable speed penalty for the sample i used.

I did a little benchmark with a flake -8 encoded file, flac -d gives me 151.88x, foobar 180.23x, flacdec 265.60x
Sundance's flacdec-to-flac ratio is 2.81, mine is 1.75, there is something strange going on here.

Flacdec generates an output stream with a bandwidth of over 45MB/s. I couldn't find a suitable md5 benchmark for windows quickly but i did find this (http://etud.epita.fr/%7Ebevand_m/papers/md5-amd64.html). Looks like the impact of hashing on overal performance is significant, but nowhere near enough to explain the whole difference.

MedO, that's the tradeoff of new software, it needs testing.
Title: New FLAC encoder
Post by: sundance on 2006-09-25 14:14:06
Quote
Sundance's flacdec-to-flac ratio is 2.81, mine is 1.75, there is something strange going on here.

I tested on a P4@2.8 GHz using that "script" along with 4NT (enhanced command shell).
All files resided on a fast local hard drive; the output went to nul: device.
Code: [Select]
timer
for %f in (*.flac) flac -f -d -o nul "%f"
timer
timer
for %f in (*.flac) flacdec -o nul "%f"
timer

Result:
Code: [Select]
Timer 1 on: 15:12:02
Timer 1 off: 15:12:37  Elapsed: 0:00:35,53
Timer 1 on: 15:12:37
Timer 1 off: 15:12:50  Elapsed: 0:00:12,91

I ran this test 5 times to eleminate disc caching related timing differences and didn't use the best and worst result to calculate the average time needed. Would be interested if you acually created wav files or if also redirected to nul...

.sundance.

P.S. Hope this discussion isn't to much OT...
Title: New FLAC encoder
Post by: MedO on 2006-09-25 15:06:18
MedO, that's the tradeoff of new software, it needs testing.


I wrote a small wrapper script using flake, md5sum and flac (for decoding). I can post it here if anyone's interested, but only after I improved it a bit, since this was only for quick testing. Btw, this is still a good deal faster than using the official encoder when both run at -8.
Title: New FLAC encoder
Post by: jcoalson on 2006-09-25 16:24:25
I tested on a P4@2.8 GHz using that "script" along with 4NT (enhanced command shell).
All files resided on a fast local hard drive; the output went to nul: device.
Code: [Select]
timer
for %f in (*.flac) flac -f -d -o nul "%f"
timer
timer
for %f in (*.flac) flacdec -o nul "%f"
timer

I don't know what "-o nul" means to flacdec but to flac that means output to a file called "nul" so you are counting the i/o time, which is probably half the total time.  if you want to eliminate that, use -t instead of -d.

also, flac decoding is fast enough that md5 checking does take a significant amount of time.  but it would still add only 10-20% I think, not double.  you can't turn off md5 checking in the command line decoder (unless you use metaflac to set the md5 sum of the file to all zeroes)

the flac binary for windows that I release is compiled with msvc6 which might generate slower code.  but it has a lot of assembly optimization, so I think this 1.7-2.8x difference is because of something else (like file i/o) not comparing apples to apples.

Josh
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-25 16:43:57
Here's my test results (12 flac files, encoded with flac -8, totalling 447.635.340 bytes) for decoding to nul:
a) flac -d -o nul [file]: 34,31 sec (avg.)
b) flacdec -o nul [file]: 12,21 sec (avg.)
Although flacdec doesn't "check MD5 signature of audio data", I don't think that's the culprit...

In my tests, the MD5 checking accounts for about 25% of the decoding time in flac (using flac -t).
Title: New FLAC encoder
Post by: kjoonlee on 2006-09-25 16:44:12
I think NUL is the DOS/Windows equivalent of /dev/null.
Title: New FLAC encoder
Post by: sundance on 2006-09-25 16:48:31
Josh,

sorry for not comparing objectively.
I assumed that flac also used the nul: device in my commandline since no file was actually generated. But that was of course a wrong assumption since you can't create such a file in dos/win/win32. I'll try to repeat my little "test" with flac -t as you've suggested when I'm back home at my computer.

And please don't get me wrong: Nagging about flac is the least I want to do - I've been a long time flac user and lover and because of that I'm really excited with all the progress of the flac format.

.sundance.
Title: New FLAC encoder
Post by: krmathis on 2006-09-25 17:41:40
Sorry...I did not see this originally.  It looks like it might have to do with byteswap.h.  I'll look into it.

Thank you!

Justin. Can you please let me know when you think you have fixed this?
Title: New FLAC encoder
Post by: jcoalson on 2006-09-25 17:55:26
Josh,

sorry for not comparing objectively.

no problem!

I assumed that flac also used the nul: device in my commandline since no file was actually generated. But that was of course a wrong assumption since you can't create such a file in dos/win/win32. I'll try to repeat my little "test" with flac -t as you've suggested when I'm back home at my computer.

hmm... if it didn't write a file, maybe windows C lib is interpreting nul on its own.  with nul it might still count time copying the output data around, not sure.  the comparison also depends on how exactly flacdec handles '-o nul'

And please don't get me wrong: Nagging about flac is the least I want to do - I've been a long time flac user and lover and because of that I'm really excited with all the progress of the flac format.

no, that's fine, it's just that I really don't know how another decoder could be 2x faster than the reference, that would be a huge improvement since the reference decoder is already optimized.  a faster encoder is easy to believe.

Josh
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-25 17:56:02

Sorry...I did not see this originally.  It looks like it might have to do with byteswap.h.  I'll look into it.

Thank you! 

Justin. Can you please let me know when you think you have fixed this?

Well, I don't have a mac to test on, so I'm kinda at a loss.  I did change some of the configuration options, but I doubt they will help.  I'm wondering if maybe bswap_32 is being redefined somewhere...  Try the patch below.  If it doesn't fix the problem, at least it might point me in the right direction.
-Justin

Code: [Select]
Index: flake/wav.c
===================================================================
--- flake/wav.c    (revision 46)
+++ flake/wav.c    (working copy)
@@ -537,7 +537,7 @@
#ifdef WORDS_BIG_ENDIAN
         uint16_t *buf16 = (uint16_t *)buffer;
         for(i=0; i<nsmp; i++) {
-            buf16[i] = bswap_16(buf16[i]);
+            buf16[i] = flake_bswap_16(buf16[i]);
         }
#endif
         if(wf->source_format != WAV_SAMPLE_FMT_S16) return -1;
@@ -568,7 +568,7 @@
#ifdef WORDS_BIG_ENDIAN
             uint32_t *buf32 = (uint32_t *)buffer;
             for(i=0; i<nsmp; i++) {
-                buf32[i] = bswap_32(buf32[i]);
+                buf32[i] = flake_bswap_32(buf32[i]);
             }
#endif
             if(wf->source_format != WAV_SAMPLE_FMT_FLT) return -1;
@@ -591,7 +591,7 @@
#ifdef WORDS_BIG_ENDIAN
         uint64_t *buf64 = (uint64_t *)buffer;
         for(i=0; i<nsmp; i++) {
-            buf64[i] = bswap_64(buf64[i]);
+            buf64[i] = flake_bswap_64(buf64[i]);
         }
#endif
         if(wf->source_format != WAV_SAMPLE_FMT_DBL) return -1;
Index: bswap.h
===================================================================
--- bswap.h    (revision 54)
+++ bswap.h    (working copy)
@@ -8,24 +8,24 @@

#include "config.h"

-static inline uint16_t bswap_16(uint16_t x){
+static inline uint16_t flake_bswap_16(uint16_t x){
     return (x>>8) | (x<<8);
}

-static inline uint32_t bswap_32(uint32_t x){
+static inline uint32_t flake_bswap_32(uint32_t x){
     x= ((x<<8)&0xFF00FF00) | ((x>>8)&0x00FF00FF);
     return (x>>16) | (x<<16);
}

-static inline uint64_t bswap_64(uint64_t x)
+static inline uint64_t flake_bswap_64(uint64_t x)
{
     union {
         uint64_t ll;
         uint32_t l[2];
     } w, r;
     w.ll = x;
-    r.l[0] = bswap_32(w.l[1]);
-    r.l[1] = bswap_32(w.l[0]);
+    r.l[0] = flake_bswap_32(w.l[1]);
+    r.l[1] = flake_bswap_32(w.l[0]);
     return r.ll;
}

@@ -36,13 +36,13 @@
#define be2me_16(x) (x)
#define be2me_32(x) (x)
#define be2me_64(x) (x)
-#define le2me_16(x) bswap_16(x)
-#define le2me_32(x) bswap_32(x)
-#define le2me_64(x) bswap_64(x)
+#define le2me_16(x) flake_bswap_16(x)
+#define le2me_32(x) flake_bswap_32(x)
+#define le2me_64(x) flake_bswap_64(x)
#else
-#define be2me_16(x) bswap_16(x)
-#define be2me_32(x) bswap_32(x)
-#define be2me_64(x) bswap_64(x)
+#define be2me_16(x) flake_bswap_16(x)
+#define be2me_32(x) flake_bswap_32(x)
+#define be2me_64(x) flake_bswap_64(x)
#define le2me_16(x) (x)
#define le2me_32(x) (x)
#define le2me_64(x) (x)
Title: New FLAC encoder
Post by: krmathis on 2006-09-25 19:26:23
Well, I don't have a mac to test on, so I'm kinda at a loss.  I did change some of the configuration options, but I doubt they will help.  I'm wondering if maybe bswap_32 is being redefined somewhere...  Try the patch below.  If it doesn't fix the problem, at least it might point me in the right direction.
-Justin

I understand that you have no way to test this, since you don't have access to a Mac. But I am more than willing to perform the tests needed. If that helps?

Ok, I had to manually edit the files according to the patch you listed. Because I could not get patch to recognize the patch file I created with nano.

Revision 61 with the patched files failed just like before:
Quote
cc -L/Users/krmathis/.Trash/flake-enc/libflake -Wl,-dynamic,-search paths_first  -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2


The problem seems to be line #277 in flake-enc/libflake/md5.c
Cause if I change "bswap_16" to "flake_bswap_16" the error message change according to it.


Edit: I removed the above mentioned line, and it compiled successfully!
Code: [Select]
-        sig16[i] = bswap_16(sig16[i]);

It launches, but I don't know how to test if it works as expected...


Edit 2:
It compressed a wav file successfully. But the result can't be right!
Quote
40801624 Sep 25 20:33 01 - Speak to Me - Breathe In the Air.flac
22205524 Sep 23 14:27 01 - Speak to Me - Breathe In the Air.m4a
41952016 Sep 25 20:32 01 - Speak to Me - Breathe In the Air.wav
The resulting FLAC file was only compressed to 97% of the original one. While the ALAC (.m4a) file compressed to ~60%.
Quote
Flake: FLAC audio encoder
© 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 10486980

block size: 4608
variable: none
prediction type: levinson-durbin
prediction order: 1,8
partition order: 0,6
order method: estimate
stereo method: mid-side
header padding: 4096

progress: 100% | ratio: 0.973 | bitrate: 1372.5 kbps | bytes: 40801624

Was this caused by the line I edited out?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-25 19:41:00
The problem seems to be line #277 in flake-enc/libflake/md5.c
Cause if I change "bswap_16" to "flake_bswap_16" the error message change according to it.

Fixed.  I forgot to #include "bswap.h" at the top of md5.c.

-Justin
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-25 19:56:36
It compressed a wav file successfully. But the result can't be right!
Quote
40801624 Sep 25 20:33 01 - Speak to Me - Breathe In the Air.flac
22205524 Sep 23 14:27 01 - Speak to Me - Breathe In the Air.m4a
41952016 Sep 25 20:32 01 - Speak to Me - Breathe In the Air.wav
The resulting FLAC file was only compressed to 97% of the original one. While the ALAC (.m4a) file compressed to ~60%.
...
Was this caused by the line I edited out?

I hope so...but I sort of doubt it.  Play back the flac file.  I suspect it's all noise.  If you try rev67 and still have the problem please speak up and I'll investigate.
Title: New FLAC encoder
Post by: krmathis on 2006-09-25 20:12:54

The problem seems to be line #277 in flake-enc/libflake/md5.c
Cause if I change "bswap_16" to "flake_bswap_16" the error message change according to it.

Fixed.  I forgot to #include "bswap.h" at the top of md5.c.

-Justin

Success!
There is still something wrong, cause the compression rate really sucks.
Flake: "ratio: 0.973 | bitrate: 1372.5 kbps | bytes: 40801624 "
FLAC 1.1.2: "22381217 bytes, ratio=0.534"

Same big/little-endian problem as we had earlier (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=403140) maybe?


Edit: Saw you reply after posting this one.
1. Using revision 67
2. The resulting FLAC file is just noice.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-25 20:42:27
Same big/little-endian problem as we had earlier (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=403140) maybe?

Edit: Saw you reply after posting this one.
1. Using revision 67
2. The resulting FLAC file is just noice.

Yes indeed.  It should be fixed now in r69.
Title: New FLAC encoder
Post by: krmathis on 2006-09-25 22:00:18

Same big/little-endian problem as we had earlier (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=403140) maybe?

Edit: Saw you reply after posting this one.
1. Using revision 67
2. The resulting FLAC file is just noice.

Yes indeed.  It should be fixed now in r69.

Sadly it is not.
Revision 69 still output garbled data at a compression ratio of 0.973 (I removed all source traces and pulled a fresh one).

Thanks a lot!
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-25 23:41:25


Same big/little-endian problem as we had earlier (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=403140) maybe?

Edit: Saw you reply after posting this one.
1. Using revision 67
2. The resulting FLAC file is just noice.

Yes indeed.  It should be fixed now in r69.

Sadly it is not.
Revision 69 still output garbled data at a compression ratio of 0.973 (I removed all source traces and pulled a fresh one).

Thanks a lot!

Very odd.  I can't figure it out.  Try hand-editing config.h and remove the line with #define WORDS_BIGENDIAN 1.  And/or put a printf line right before the call to bswap_16 in wav.c.  Those won't fix anything, but might rule out some things.

[edit]changed the instructions slightly

[edit2] btw...i should have sourceforge compile farm access in the next 24-hrs. that way i can test on other platforms.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-09-26 07:08:09
As usual the fix was too simple for me to notice right away...just a spelling error.  *sigh*  I have fixed the problem and tested on a ppc (yay for compile farm).  Thanks for finding the problem so that I could fix it.

-Justin
Title: New FLAC encoder
Post by: krmathis on 2006-09-26 15:41:50
As usual the fix was too simple for me to notice right away...just a spelling error.  *sigh*
You certainly nailed it!

Flake rev69: ratio: 0.973 | bitrate: 1372.5 kbps | bytes: 40801624
Flake rev77: ratio: 0.527 | bitrate: 743.6 kbps | bytes: 22107579
FLAC 1.1.2: 22381217 bytes, ratio=0.534

Quote
Thanks for finding the problem so that I could fix it.
No problem!
..and thank you for fixing the bugs I have found.

** now on to testing this "beast" **
Title: New FLAC encoder
Post by: plinej on 2006-10-02 04:46:39
I just found this thread today and have skimmed thru it. I was happy to see that flac is being put into ffmpeg. I just compiled the latest ffmpeg source from svn to test this out. It seemed to encode fine but playing back the file with xmms does not display the length of the track and will not let you skip ahead. My flac and flake encoded files work fine in xmms though. I just wanted to see if that was a known problem or maybe I encoded wrong. I just used the example posted in this thread:

ffmpeg -i test.wav test.flac
Title: New FLAC encoder
Post by: xmixahlx on 2006-10-02 06:05:58
i packaged a svn snapshot for debian

available at the usual place


later
Title: New FLAC encoder
Post by: Mangix on 2006-10-02 06:22:16
i got a question about the code being used in flake. when i tried to compile flake with Microsoft Visual C++, i noticed that flake uses data types such as int32_7 and uint63_t. i also read on the wikipedia about inttypes.h and found out that they have equivalents which are built in the C++ language itself.

so my question is, are there any main diffrences between, let's say, "int32_t" and "long"?

edit: reason that i'm asking is because the Platform SDK does not give an inttypes.h file.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-02 07:13:22
i got a question about the code being used in flake. when i tried to compile flake with Microsoft Visual C++, i noticed that flake uses data types such as int32_7 and uint63_t. i also read on the wikipedia about inttypes.h and found out that they have equivalents which are built in the C++ language itself.

so my question is, are there any main diffrences between, let's say, "int32_t" and "long"?

edit: reason that i'm asking is because the Platform SDK does not give an inttypes.h file.

The main difference has to do with 64-bit platforms.  Rather than explain it, maybe a snippet from glibc's stdint.h (which is included by inttypes.h) would help.  inttypes.h is part of the ISO C99 standard, which is the baseline standard used in Flake.

Code: [Select]
#ifndef __int8_t_defined
# define __int8_t_defined
typedef signed char        int8_t;
typedef short int        int16_t;
typedef int            int32_t;
# if __WORDSIZE == 64
typedef long int        int64_t;
# else
__extension__
typedef long long int        int64_t;
# endif
#endif

/* Unsigned.  */
typedef unsigned char        uint8_t;
typedef unsigned short int    uint16_t;
#ifndef __uint32_t_defined
typedef unsigned int        uint32_t;
# define __uint32_t_defined
#endif
#if __WORDSIZE == 64
typedef unsigned long int    uint64_t;
#else
__extension__
typedef unsigned long long int    uint64_t;
#endif


For MSVC, you can try adding this bit of code somewhere.  I will add it myself in the near future...I've been meaning to, but just keep putting it off.
Code: [Select]
#if defined(_WIN32) && defined(_MSC_VER)
typedef __int8 int8_t;
typedef unsigned __int8 uint8_t;
typedef __int16 int16_t;
typedef unsigned __int16 uint16_t;
typedef __int32 int32_t;
typedef unsigned __int32 uint32_t;
typedef __int64 int64_t;
typedef unsigned __int64 uint64_t;
#endif


If you need me to explain more I can.
-Justin

edit: added the msvc stuff
Title: New FLAC encoder
Post by: Mangix on 2006-10-02 08:01:55
from looking at the code, i can only guess that on 32-bit platforms, int64_t and uint64_t must be defined diffrently in order to work.

also thanks for that piece of code that you put there. helps a lot. i'm don't have much experiance in c++ so that code helps.
Title: New FLAC encoder
Post by: wisodev on 2006-10-02 18:04:23
I have builded latest Flake (Win32) from SVN revision 95:

Download from my Website (http://www.thefrontend.net/flake/)

or use direct links:

Download binarys:
flake-svn-r95-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r95-win32-bin.zip)

Download sources:
flake-svn-r95-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r95-win32-src.zip)

wisodev
Title: New FLAC encoder
Post by: MedO on 2006-10-02 18:14:18
Thanks wisodev!
What are the general changes from r48?
Title: New FLAC encoder
Post by: wisodev on 2006-10-02 18:33:17
What are the general changes from r48?


Well Flake is in constant development so I decided to build fresh binary from latest SVN sources. General changes (in current SVN version) are listed in Flake Changelog file included in src archive.
Title: New FLAC encoder
Post by: Mangix on 2006-10-02 23:45:39
wisodev: which compiler do you use? MinGW?
Title: New FLAC encoder
Post by: MedO on 2006-10-03 00:05:51
wisodev: which compiler do you use? MinGW?


Quote
I am using Intel C++ compile at version 8.0.40 and C99 switch (this is req.)


But Flake compiles fine with MinGW. I just tried it and, after getting gcc 4.1 to run, it's just as fast as wisodev's build for me.
Title: New FLAC encoder
Post by: Mangix on 2006-10-03 00:12:08
i tried out one of wisodev's builds and his seem to be a few seconds faster at mode -12 for me. the other executable that i tested was one compiled by me.
Title: New FLAC encoder
Post by: MedO on 2006-10-03 00:21:46
i tried out one of wisodev's builds and his seem to be a few seconds faster at mode -12 for me. the other executable that i tested was one compiled by me.


Here's my build:
Flake r100 mingw-build (http://www.uni-koblenz.de/~smaxein/flake.zip)

Only the "naked" binary, though.

Compiler Flags: -s -O3 -fexpensive-optimizations -march=i686
Compiler Version: gcc 4.1.1
Title: New FLAC encoder
Post by: Mangix on 2006-10-03 00:54:35
bah. now i can't get rev95 to work. it keeps saying "error parsing filenames" when i try to encode it.

"flake -12 C:\test\spirit.wav C:\test\spirit.flac" is the command line that i use.

edit: interesting. now i don't need to even specify an output file. pretty interesting...
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-03 01:02:28
But Flake compiles fine with MinGW. I just tried it and, after getting gcc 4.1 to run, it's just as fast as wisodev's build for me.

I have not done any gcc4 testing, much less under MinGW.  Were there any compile issues that I may need to solve or did everything build smoothly?  Since the build system is hand-made, not autotools, I've been trying to put it through more tests.

Thanks,
Justin
Title: New FLAC encoder
Post by: Mangix on 2006-10-03 01:05:26
i just did a short test with 3 encoders. the first was my own compiled one, the second was MedO's one, and the third was wisodev's compiler.

the results sucked because my compile took the longest to encode while MedO's compile took 61 seconds and wisodev's took 64 seconds. mine took 65.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-03 01:07:58
bah. now i can't get rev95 to work. it keeps saying "error parsing filenames" when i try to encode it.

"flake -12 C:\test\spirit.wav C:\test\spirit.flac" is the command line that i use.

edit: interesting. now i don't need to even specify an output file. pretty interesting...

Correct.  Now you don't have to specify the output if you want the same base name, but if you want a specific output file, you have to now use "-o output.flac".  The order method switch was changed to "-m".  I'm working on a new option system right now though, so things may change again before the next full release. I hope to add multiple file input as well.  Everything will be documented better for the release...the current svn version is undergoing pretty constant changes.

edit: typo
edit2: another typo...
Title: New FLAC encoder
Post by: Mangix on 2006-10-03 02:46:22
heh. i just tried out rev95 again with foobar2000's Converter and the thing blew up on me. the command line that i used was "-12 - -o %d" and when i tried it, when it finished the first or second tracked, foobar2000 gave an error saying "Error flushing file". and also, the output file was "filename.flac=". i am not even sure why there was an = at the end.

edit: why the hell do i always mix up line with like...
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-03 03:19:32
heh. i just tried out rev95 again with foobar2000's Converter and the thing blew up on me. the command line that i used was "-12 - -o %d" and when i tried it, when it finished the first or second tracked, foobar2000 gave an error saying "Error flushing file". and also, the output file was "filename.flac=". i am not even sure why there was an = at the end.

edit: why the hell do i always mix up line with like...

I think it was a bug in my code.  Try rev96.

edit: a little explaining.  I had forgotten that you have to include the trailing '\0' in the character count for strncpy.
Title: New FLAC encoder
Post by: Mangix on 2006-10-03 04:01:35
yep. looks like it's fixed. thank you
Title: New FLAC encoder
Post by: MedO on 2006-10-03 10:06:50
Flake r100 mingw-build (http://www.uni-koblenz.de/~smaxein/flake.zip)
Same location as last time.

Building was quite smooth, but I didn't use the makefile (don't know if I even could, I'm quite new to this). I just started CodeBlocks, added all needed files for the library to a project, set the compiler paths so it could find bswap.h and added an empty config.h. Setting up the project for flake was almost the same.

There was only one warning when building:
flake\wav.c:473: warning: enumeration value 'WAV_SAMPLE_FMT_UNKNOWN' not handled in switch

That was already present with the gcc 3 I used before (can't remember the exact version)
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-03 10:33:46
I haven't been able to compile using MinGW since rev 48.  If I run ./configure I get:

/bin/cat: /tmp/flake-conf-10500-6548-17128.c: No such file or directory
gcc is unable to create an executable file.

It still works with cygwin.

I don't need to compile, but I thought it worth pointing out, as it seems relevant at the moment.
Title: New FLAC encoder
Post by: kurtnoise on 2006-10-03 12:25:33
no problem here with mingw compile...Maybe you need to update mingw-runtime.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-03 13:35:33
It was only just over a week ago that it did work.  If I choose "Update" MinGW only allows me to install new apps, and not update current apps.  I've no idea where or how to update.

As I say, no big deal, I was just curious why the change.
Title: New FLAC encoder
Post by: kjoonlee on 2006-10-03 14:11:23
Does flake use autotools? If yes, would a ./autogen.sh generate new ./configure* files?
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-03 14:23:48
I just downloaded Rev. 19 of configure (http://svn.sourceforge.net/viewvc/*checkout*/flake-enc/configure?revision=19) and that works fine. Rev. 50 (http://svn.sourceforge.net/viewvc/*checkout*/flake-enc/configure?revision=50) (the next version) does not.

Code: [Select]
$ gcc -v
Reading specs from c:/mingw/bin/../lib/gcc/mingw32/3.4.2/specs
Configured with: ../gcc/configure --with-gcc --with-gnu-ld --with-gnu-as --host=mingw32 --target=mingw32 --prefix=/mingw --enable-threads --disable-nls --enable-languages=c,c++,f77,ada,objc,java --disable-win32-registry --disable-shared --enable-sjlj-exceptions --enable-libgcj --disable-java-awt --without-x --enable-java-gc=boehm --disable-libgcj-debug --enable-interpreter --enable-hash-synchronization --enable-libstdcxx-debug
Thread model: win32
gcc version 3.4.2 (mingw-special)
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-03 14:30:37
Flake r100 mingw-build (http://www.uni-koblenz.de/%7Esmaxein/flake.zip)
Same location as last time.

Building was quite smooth, but I didn't use the makefile (don't know if I even could, I'm quite new to this). I just started CodeBlocks, added all needed files for the library to a project, set the compiler paths so it could find bswap.h and added an empty config.h. Setting up the project for flake was almost the same.

There was only one warning when building:
flake\wav.c:473: warning: enumeration value 'WAV_SAMPLE_FMT_UNKNOWN' not handled in switch

That was already present with the gcc 3 I used before (can't remember the exact version)

Hmm.  Are you not using ./configure + make because it doesn't work?  If you have msys installed that's all that you should need to do.

I haven't been able to compile using MinGW since rev 48.  If I run ./configure I get:

/bin/cat: /tmp/flake-conf-10500-6548-17128.c: No such file or directory
gcc is unable to create an executable file.

It still works with cygwin.

I don't need to compile, but I thought it worth pointing out, as it seems relevant at the moment.

That is odd.  Can you compile/run a simple hello world program with your MinGW version?
edit: looks like I was too slow. I'm glad it works for you now.

Does flake use autotools? If yes, would a ./autogen.sh generate new ./configure* files?

No, I did not use autotools for Flake, nor do it ever intend to.  I may decide to use some other build system, but it won't be autotools.  make distclean + ./configure + make is all that should be necessary under MinGW.

-Justin
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-03 14:44:15
edit: looks like I was too slow. I'm glad it works for you now.
Well, rev.19 of configure does, but none since, and that's not really much use is it?

I dunno.  I don't really care!  I just thought it may be of interest.  It seems to me that something changed in version 50 that breaks configure with my versions (work and home) of MinGW (installed using MinGW 5.0.3  and MSYS 1.0.11).  I would assume that this would be the same for others, but if not then, please, forget it.  I've been expecting someone else to point this out, but as MedO was releasing versions I thought it was worth piping up.

If it's not an issue for anyone else then please don't pursue it.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-03 14:48:26
I just downloaded Rev. 19 of configure (http://svn.sourceforge.net/viewvc/*checkout*/flake-enc/configure?revision=19) and that works fine. Rev. 50 (http://svn.sourceforge.net/viewvc/*checkout*/flake-enc/configure?revision=50) (the next version) does not.

Could you post or PM me the config.err you get using rev50?  This would help me narrow-down which change caused the issue.  That was quite a big change from r19 to r50.  In retrospect, I probably should have broken it up into smaller commits.

The fact that it doesn't work does concern me, and it may be a simple fix.  If I can't figure it out from your config.err output then I'll maybe give up on it.

edit: typo. and I was too slow again...added response to last post.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-03 15:03:09
Certainly.  [attachment=2576:attachment]
Title: New FLAC encoder
Post by: MedO on 2006-10-03 16:00:10
Hmm.  Are you not using ./configure + make because it doesn't work?  If you have msys installed that's all that you should need to do.


I didn't install msys, no. I have never used it yet. I know ./configure -scripts from building Linux apps, but I never wrote one or used one under Windows. Hmm, if I have no problems compiling Flake without running ./configure first, does CodeBlocks do the necessary stuff that ./configure normally does?

As I said, I'm quite new to this.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-03 19:55:09
I received an email this afternoon (I found it when I got home) from a new user here called Mitch 1 2.  He is waiting for his five day suspension to end before posting, but sent me this:

Quote
I had the same problem. Get this:
http://optusnet.dl.sourceforge.net/sourcef...napshot.tar.bz2 (http://optusnet.dl.sourceforge.net/sourceforge/mingw/bash-3.1-MSYS-1.0.11-snapshot.tar.bz2)
and put the binaries in your msys\1.0\bin directory.

His suggestion worked; I have successfully compiled rev. 101.

So, I assume it was to do with bash not being available?
Title: New FLAC encoder
Post by: kurtnoise on 2006-10-03 20:12:15
most probably an older version...bourne shell 2.4 doesn't work anymore with some apps.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-04 02:40:31

Hmm.  Are you not using ./configure + make because it doesn't work?  If you have msys installed that's all that you should need to do.


I didn't install msys, no. I have never used it yet. I know ./configure -scripts from building Linux apps, but I never wrote one or used one under Windows. Hmm, if I have no problems compiling Flake without running ./configure first, does CodeBlocks do the necessary stuff that ./configure normally does?

As I said, I'm quite new to this.

For building binaries only, in Windows, with gcc/mingw...yes CodeBlocks probably does all that is necessary.  For building a static library or building on other platforms, the configure script is useful because it tries to automatically detect information that is needed to compile and install.  It can also detect system features so it can use compiler flags which will generate faster binaries.  But, if you already know gcc well enough, you can just set the compiler flags yourself.

-Justin
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-04 05:51:25
Current SVN version now supports multiple input files.  So you can encode a whole directory of wav files with 1 command.  It's as simple as running something such as:
flake -8 *.wav
or
flake -12 file1.wav file2.wav file7.wav

This has not been tested in Windows, so I hope it works okay.

Now...no new features until the next release...for real this time.  I'll work on documentation and testing (and my ac3 encoder) while I give Flake a little resting time for bugs to show themselves.  The docs will take a bit of work, but at least it will be a nice change of pace.

Thanks,
-Justin
Title: New FLAC encoder
Post by: Mangix on 2006-10-04 23:27:40
you have #include <inttypes.h> at wav.h . just though i should point that out. just tried building it again with VC++ 2005 and it blew up on syntax errors. i'll try to rebuild it in the future with VC++ 2005 when i get interested in doing that again.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-05 02:53:25
you have #include <inttypes.h> at wav.h  . just though i should point that out. just tried building it again with VC++ 2005 and it blew up on syntax errors. i'll try to rebuild it in the future with VC++ 2005 when i get interested in doing that again.

Thanks. fixed.
Title: New FLAC encoder
Post by: xmixahlx on 2006-10-05 07:44:09
svn r104 at RareWares/Debian now
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-05 14:25:42
Windows x86 compiles using current SVN (rev 104): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_r104_cygwin.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_r104_mingw.7z)
Title: New FLAC encoder
Post by: sundance on 2006-10-05 16:00:24
Just a minor thing:
There's a discrepancy concerning the command line syntax. When you invoke flake w/o parameters, it says:
Code: [Select]
Flake: FLAC audio encoder
© 2006  Justin Ruggles

usage: flake [options] <input.wav> [output.flac]

But when you add -h it's this:
Code: [Select]
Flake: FLAC audio encoder
© 2006  Justin Ruggles

usage: flake [options] <input.wav> [-o output.flac]
options:
       [-h]         Print out list of commandline options
       [-p #]       Padding bytes to put in header (default: 4096)
.
.
.

Guess '-o output.flac' is the one that's correct.
And still no version/revision number to display... 

.sundance.
Title: New FLAC encoder
Post by: Wombat on 2006-10-05 16:50:46
Ah, oh. I see a problem. Does -12 take that much processing power or isn´t it 100% compatible to flac 1.1.2 decoding anymore?
My Squeezebox can´t play flake -12 on all titles. It stutters around. Flake -5 for example is fine.

Edit:
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-05 16:59:49
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.
That has more similarities with the upcoming FLAC 1.1.3, or FLAC 1.1.2_CVS from this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=44229).
Title: New FLAC encoder
Post by: guruboolez on 2006-10-05 17:08:11
Ah, oh. I see a problem. Does -12 take that much processing power or isn´t it 100% compatible to flac 1.1.2 decoding anymore?
My Squeezebox can´t play flake -12 on all titles. It stutters around. Flake -5 for example is fine.

After Bukem (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=433703)'s positive report, your one is the first negative one and is precisely what I feared.
My own decoding tests done with foobar2000 and my old Duron revealed that -11 and -12 are considerably slower than all other modes and than official/Garf's encoder. From memory, it's ~x48 for flake high profile and a bit more than x60 for all other settings/encoders.
As a consequence I expected from some devices with few 'power' headroom to have troubles decoding -q11/12 encodings.
Could you also try with -q10 and -q11? I guess that the first one should be problem free and the latter to sutter as well as -q12.
Title: New FLAC encoder
Post by: Wombat on 2006-10-05 17:10:46
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.
That has more similarities with the upcoming FLAC 1.1.3, or FLAC 1.1.2_CVS from this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=44229).

Thanks. So flake behaves different. As i read over the thread it was like the new ideas of 1.1.3 were in flake already and nothing more.

@Guru
i´ll do
Title: New FLAC encoder
Post by: Wombat on 2006-10-05 17:22:13
After Bukem (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=433703)'s positive report, your one is the first negative one and is precisely what I feared.
My own decoding tests done with foobar2000 and my old Duron revealed that -11 and -12 are considerably slower than all other modes and than official/Garf's encoder. From memory, it's ~x48 for flake high profile and a bit more than x60 for all other settings/encoders.
As a consequence I expected from some devices with few 'power' headroom to have troubles decoding -q11/12 encodings.
Could you also try with -q10 and -q11? I guess that the first one should be problem free and the latter to sutter as well as -q12.

Exactly how you predicted. -10 still is playable. From -11 on it stutters.
Title: New FLAC encoder
Post by: guruboolez on 2006-10-05 17:28:23
Thank you very much Wombat

May I suggest to Justin that the first official release of flake should come with a small warning (inside the documentation and/or inside flake.exe with -h switch) about possible hardware players' issues with -11 & -12 setting?
Title: New FLAC encoder
Post by: Josef Pohm on 2006-10-05 17:32:31
Exactly how you predicted. -10 still is playable. From -11 on it stutters.


To my little knoledge decoding speed of lossless codecs mostly depends on the maximum number of predictor order used.

Flake -5 uses 8 max predictor order, like Flac -5 (and Garf version of Flac).
Flake -8,-9 and -10 uses 12 max predictor order like Flac -8 (and Garf version of Flac).
Flake -11 and -12 uses 32, so it requires much more decoding power.

I hope that's correct.
Title: New FLAC encoder
Post by: jcoalson on 2006-10-05 18:03:06
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this.  with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion.  this goes for flac too which currently allows -l 32 without a warning.

most of the compression advantage of flake over FLAC 1.1.2 has already been incorporated into FLAC 1.1.3.  the extra compression from 32-order LPC is minimal.

Josh
Title: New FLAC encoder
Post by: wisodev on 2006-10-05 20:39:39
Flake (Win32) from SVN revision 104 available:

Download from my Website (http://www.thefrontend.net/flake/)

or use direct links:

Download binarys:
flake-svn-r104-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-bin.zip)

Download sources:
flake-svn-r104-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-src.zip)

NOTE: I have added my small patch to code (quality or speed is not affected), can be found here (http://www.thefrontend.net/flake/svn/flake_rev104_wiso.diff), the sources are already patched (this patch comes from my local subversion repository and is not related with sf.net subversion).

wisodev
Title: New FLAC encoder
Post by: fairway on 2006-10-05 21:50:44

Exactly how you predicted. -10 still is playable. From -11 on it stutters.


To my little knoledge decoding speed of lossless codecs mostly depends on the maximum number of predictor order used.

Flake -5 uses 8 max predictor order, like Flac -5 (and Garf version of Flac).
Flake -8,-9 and -10 uses 12 max predictor order like Flac -8 (and Garf version of Flac).


Though seeking with a flake encoded file is still significantly slower than with a FLAC (reference encoder) encoded file (try it in winamp with the default flac plugin).
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-06 02:07:16
Thank you very much Wombat 

May I suggest to Justin that the first official release of flake should come with a small warning (inside the documentation and/or inside flake.exe with -h switch) about possible hardware players' issues with -11 & -12 setting?

Okay, that sounds like a good idea.

I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this.  with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion.  this goes for flac too which currently allows -l 32 without a warning.

Cool.  Based on this, and the hardware decoding issues, I'll go ahead and add a separate warning for any lpc order over 12 for any sample rate up to 48kHz.  Once the official subset standard changes, I'll change Flake's warnings accordingly.

-Justin
Title: New FLAC encoder
Post by: JWolf on 2006-10-07 05:02:20
Ah, oh. I see a problem. Does -12 take that much processing power or isn´t it 100% compatible to flac 1.1.2 decoding anymore?
My Squeezebox can´t play flake -12 on all titles. It stutters around. Flake -5 for example is fine.

Edit:
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.


Using the current version of FLAKE, I was able to play -12 compressed files on my Rio Karma with no hiccups.

Jon
Title: New FLAC encoder
Post by: jcoalson on 2006-10-07 07:51:34
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this.  with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion.  this goes for flac too which currently allows -l 32 without a warning.

it is done.

the new limits: if the sample rate is <=48kHz, max blocksize = 4608 and max LPC order = 12.  all the tests I did showed that 32-order LPC got <0.1% more compression (usually a lot less).

http://flac.cvs.sourceforge.net/*checkout*...mat.html#subset (http://flac.cvs.sourceforge.net/*checkout*/flac/flac/doc/html/format.html#subset)

Josh
Title: New FLAC encoder
Post by: wisodev on 2006-10-07 10:15:00
I have builded a faster FLAKE binary from rev. 104 but it needs some testing.

On one of my test files encoding time has droped from 193.468 seconds (previous r104 build (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-bin.zip)) to 173.906 seconds (test r104 build (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)) using -12 command-line option.

Download test binary:
flake_r104_test.zip (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)

wisodev
Title: New FLAC encoder
Post by: MedO on 2006-10-07 12:25:56
I have builded a faster FLAKE binary from rev. 104 but it needs some testing.

On one of my test files encoding time has droped from 193.468 seconds (previous r104 build (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-bin.zip)) to 173.906 seconds (test r104 build (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)) using -12 command-line option.

Download test binary:
flake_r104_test.zip (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)

wisodev


Your test compile is a bit faster than mine on my machine (Celeron M 1,4Ghz). I can't give any good values, though, because I already got several seconds difference when running my own build twice, so the measurement is not very reliable.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-07 19:00:31
I have builded a faster FLAKE binary from rev. 104 but it needs some testing.

On one of my test files encoding time has droped from 193.468 seconds (previous r104 build (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-bin.zip)) to 173.906 seconds (test r104 build (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)) using -12 command-line option.
Here's some results for my "FLAC" corpus:

Code: [Select]
Setting       R104    R104 Test   Diff
======================================
-10       286.726s     274.561s    96%
-11       375.687s     347.035s    92%
-12      1002.105s     914.239s    91%

Edit: Here's the figures for my MinGW compile:

Code: [Select]
Setting  MinGW R104   R104 Test   Diff
======================================
-10        394.015s    274.561s    70%
-11        486.764s    347.035s    71%
-12       1322.178s    914.239s    69%

I don't think I'll bother anymore!    Excellent work wisodev.

NB: Timings were recorded using TIMER, and CPU-only times are reported.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-08 04:57:49
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this.  with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion.  this goes for flac too which currently allows -l 32 without a warning.

it is done. 

the new limits: if the sample rate is <=48kHz, max blocksize = 4608 and max LPC order = 12.  all the tests I did showed that 32-order LPC got <0.1% more compression (usually a lot less).

http://flac.cvs.sourceforge.net/*checkout*...mat.html#subset (http://flac.cvs.sourceforge.net/*checkout*/flac/flac/doc/html/format.html#subset)

Josh

Splendid. I have modified the Flake warning message output and the return code of the flake_validate_params function to reflect the new standard.
Title: New FLAC encoder
Post by: iGold on 2006-10-11 21:20:46
I checkout rev.107 from SVN and compile it myself under x86_64 Ubuntu. While encoding disc image I see:
Quote
Signed 16-bit 44100 Hz stereo
samples: 116712120 (0m16.961s)

If I try to compile this test code:
Code: [Select]
int th, tm, ts, tms;
tms = 116712120 * 1000 / 44100;
I get two "integer overflow" warnings.

I think int is too small here (in flake.c at line 447).
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-11 22:14:26
I checkout rev.107 from SVN and compile it myself under x86_64 Ubuntu. While encoding disc image I see:
Quote
Signed 16-bit 44100 Hz stereo
samples: 116712120 (0m16.961s)

If I try to compile this test code:
Code: [Select]
int th, tm, ts, tms;
tms = 116712120 * 1000 / 44100;
I get two "integer overflow" warnings.

I think int is too small here (in flake.c at line 447).

Thanks. Fixed.
Also, anyone can submit bug reports on Flake's Sourceforge project page.  Getting bug reports via HA makes me feel a little guilty for taking up space with small fixes.  They are appreciated either way though.
Title: New FLAC encoder
Post by: wisodev on 2006-10-12 19:16:14
Flake (Win32) from SVN revision 108 available:

Download from my Website (http://www.thefrontend.net/flake/)

or use direct links:

Download binarys:
flake-svn-r108-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r108-win32-bin.zip)

Download sources:
flake-svn-r108-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r108-win32-src.zip)

wisodev
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-15 18:31:34
although i ran into a couple of problems compiling.

Would it be too much of a problem to put my changes into the code, to keep it portable across compilers?

optimize.c line 219 replaced with: uint32_t bits[32]; (where 32 used to be levels. MSVC doesn't like non-constant sizes for these things.) I am sure that 32 is as large as levels gets.

in encode.c flake_set_defaults, msvc doen't like this syntax:
foo = ((int[]){1,2,3})[2];
so i defined all these "nameless" arrays as ones with names, so it becomes:
int sel[] = {1,2,3};
foo = sel[2];

thanks,
will

Both MSVC issues have now been fixed.

Also, I am almost done with a new website for Flake.  The only thing left is API documentation and benchmarks.  I am putting in a link to Synthetic Soul's lossless benchmarks, but it would also be nice to have more extensive tests to better illustrate the differences between the compression levels.  I'm not so worried anymore about detailed comparisons to the reference encoder because Synthetic Soul's tests give a good enough overview.  Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels.  Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful.  Corpus totals would be better than file-by-file results.

Thanks!
Justin
Title: New FLAC encoder
Post by: Will Fisher on 2006-10-16 10:32:09
Both MSVC issues have now been fixed.
Excellent, thanks

I have included my enc_flac plugin (sources are attached somewhere in this thread =) built with wisodevs libFlake r108 to the beta branch of Winamp.

If it gets a good response from our beta team and QA department (and I can't see why it wouldn't) maybe we'll put it in a subsequent version of winamp.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-17 16:38:53
Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels.  Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful.  Corpus totals would be better than file-by-file results.
Has anyone provided this yet?

I don't mind running the tests if no-one else will.  My machine isn't hugely fast (Athlon XP 2400+, 512MB RAM, Win2K) but it's OK, and I will need a compile from wisodev as I have proved already that my compiles are far inferior to his.
Title: New FLAC encoder
Post by: wisodev on 2006-10-17 17:43:31
Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels.  Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful.  Corpus totals would be better than file-by-file results.
Has anyone provided this yet?

I don't mind running the tests if no-one else will.  My machine isn't hugely fast (Athlon XP 2400+, 512MB RAM, Win2K) but it's OK, and I will need a compile from wisodev as I have proved already that my compiles are far inferior to his.


I will provide fresh compile in few hours including new experimental build using P3 optimizations (using only compiler options - not assembly optimized!!!) witch seems to be even faster then the last one.
Title: New FLAC encoder
Post by: wisodev on 2006-10-17 20:34:59
Flake (Win32) from SVN revision 110 available:

Download from my Website (http://win32builds.sourceforge.net/flake/index.html)

or use direct links:

Download binarys:
flake-svn-r110-win32-bin.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r110-win32-bin.zip?download)

Download sources:
flake-svn-r110-win32-src.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r110-win32-src.zip?download)

wisodev
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-17 20:48:33
Thanks wisodev.

I have an AMD XP CPU.  Should I be looking at /org/flake.exe, /p3/flake.exe or /flake.exe?
Title: New FLAC encoder
Post by: MedO on 2006-10-17 21:17:29
Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.

I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...

Edit: I just did a short test with the album "Supertramp - Some Things Never Change", comparing the commandlines "-8" and "-8 -m 1".
"-8 -m 1" was of course always way faster, and compressed 8 of the 12 files better than "-8". The differences in bitrate were quite small though, around 0.1-0.2kbps average.
Title: New FLAC encoder
Post by: PrakashP on 2006-10-17 21:46:17
@wisodev

I would be interested what compiler flags you use (and which compiler). Have you added asm to the sources?
Title: New FLAC encoder
Post by: MedO on 2006-10-17 22:01:16
Also, I still get this time bug when encoding with wisodev's latest build (all binaries show this)
Code: [Select]
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)


I thought this was solved?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-18 02:47:09
Also, I still get this time bug when encoding with wisodev's latest build (all binaries show this)
Code: [Select]
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)


I thought this was solved?

I did too...sorry.  I just committed a 2nd attempt, which seems to work for me.  I just tested it on a 11 million sample wav and it correctly reported around 4 minutes.
-Justin


Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.

I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...

Edit: I just did a short test with the album "Supertramp - Some Things Never Change", comparing the commandlines "-8" and "-8 -m 1".
"-8 -m 1" was of course always way faster, and compressed 8 of the 12 files better than "-8". The differences in bitrate were quite small though, around 0.1-0.2kbps average.


This is really interesting.  My tests resulted in not quite as good compression.  It could be coincedence.  More testing would be great.  The order estimate method is a pretty crude 1st attempt at an idea I got from a paper which dealt with speech compression.  It suggested a method using the highest order reflection coefficient above a certain threshold.  I've found that the method varies widely depending on the type of audio...I imagine it works well for speech, but would need some tweaking for other types of signals.  I had once tried to do some testing using gnuplot output, but my statistical analysis skills aren't really advanced enough to know what to look for...so I have left it at a simple threshold for now.

edit: I also want to note that I could almost always pick out the best order or close to it using visual output from gnuplot, but I could never translate that into a mathematical analysis that worked well.  There is definitely potential there though.

-Justin
Title: New FLAC encoder
Post by: iGold on 2006-10-18 03:10:48
Code: [Select]
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)


I thought this was solved?

I submitted bug about it in sourceforce bug tracker after I discovered it was not fixed after 1st try. Justin, are you looking there?
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-18 03:36:46
Code: [Select]
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)


I thought this was solved?

I submitted bug about it in sourceforce bug tracker after I discovered it was not fixed after 1st try. Justin, are you looking there? 

I missed that one...  I've changed the settings so I get an email now when someone posts a new bug.  I have also updated the status of your bug report.
Thanks!
-Justin
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-18 13:36:51
My results for Flake SVN r110 wisodev build P3 optimised with RW corpus.

Code: [Select]
Setting     Ratio     Enc     Dec
=================================
0         65.352%    136x    106x
1         62.822%    114x    104x
2         62.661%     98x    104x
3         59.581%     94x     97x
4         59.392%     92x     93x
5         59.292%     90x     96x
6         59.649%     66x     96x
7         59.307%     54x     96x
8         59.132%     49x     95x
8 -m 1    59.115%     83x     94x
8 -v 1    58.747%     47x     93x
9         59.019%     40x     95x
10        59.006%     28x     94x
11        58.758%     22x     83x
12        58.729%      9x     83x
12 -v 1   58.378%      9x     84x
99        58.336%      2x     72x
Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.

I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...
These results appear to bear yours out completely.  I'm so glad you posted or I would have been really concerned about my -6 results!

I will post results for the other wisodev compiles (normal and org) as well, in the next day or so.  I will also post the spreadsheet used to calculate the summaries (http://spreadsheets.google.com/ccc?key=pwAdtAz-_urL3FUFUo7Olfw) (uploaded to Google Docs & Spreadsheets).

Edit: Something else of note.  In fact two somethings.

Firstly, I should mention that the results above are CPU-only times, where my TAK lossless comparison values are CPU+IO times.  I started reporting CPU+IO in that comparison and it's awkward to stop now.

Secondly, the scripts I use to perform my tests use FSUM to check the decoded WAVE files against the MD5 hashes of the source files once all files are decoded.  This normally serves me well, but in this instance nearly all files fail the check.

I have done a little checking and it appears to be down to the WAVE header written, which I think is 2 bytes smaller than the source (bear in mind the source files are decoded WavPack files).  Again, from my limited understanding, this appears to be down to the ExtraParamSize bytes not being written by the FLAC decoder.  From the document I have read it appears that these bytes are not required if no extra parameters are present.  Strangely enough though every fourth file or so the hashes do match!

Edit: It seems that, as expected, with the files that do match MD5 hashes, neither have the ExtraParamSize bytes; therefore FLAC is acting consistently, while the source files are not consistent.  Maybe this is noticable as most decompressors just return the original WAVE header, whereas FLAC creates its own?

I really have no idea what all this means, but it thought it worth noting.  I bit compared a few files that had mismatched MD5s using foobar and the audio is bit-identical, as you would expect.

Here's the report from FSUM for -5 (it's the same for all though).

Code: [Select]
FAILED       MD5          41_30sec.wav
FAILED      MD5          ATrain.wav
FAILED      MD5          Bachpsichord.wav
OK          MD5          Bartok_strings2.wav
FAILED      MD5          BeautySlept.wav
FAILED      MD5          BigYellow.wav
FAILED      MD5          Blackwater.wav
FAILED      MD5          bodyheat.wav
OK          MD5          chanchan.wav
FAILED      MD5          DaFunk.wav
FAILED      MD5          death2.wav
OK          MD5          Debussy.wav
FAILED      MD5          EnolaGay.wav
FAILED      MD5          experiencia.wav
FAILED      MD5          female_speech.wav
FAILED      MD5          FloorEssence.wav
OK          MD5          getiton.wav
FAILED      MD5          gone.wav
OK          MD5          Hongroise.wav
FAILED      MD5          Illinois.wav
FAILED      MD5          ItCouldBeSweet.wav
FAILED      MD5          kraftwerk.wav
FAILED      MD5          Layla.wav
OK          MD5          Leahy.wav
FAILED      MD5          LifeShatters.wav
FAILED      MD5          macabre.wav
FAILED      MD5          Mahler.wav
FAILED      MD5          male_speech.wav
OK          MD5          Mama.wav
FAILED      MD5          MidnightVoyage.wav
FAILED      MD5          mybloodrusts.wav
OK          MD5          NewYorkCity.wav
OK          MD5          OrdinaryWorld.wav
FAILED      MD5          Polonaise.wav
FAILED      MD5          Quizas.wav
FAILED      MD5          riteofspring.wav
OK          MD5          rosemary.wav
FAILED      MD5          Scars.wav
OK          MD5          SinceAlways.wav
FAILED      MD5          thear1.wav
FAILED      MD5          TheSource.wav
OK          MD5          TomsDiner.wav
OK          MD5          trust.wav
OK          MD5          Twelve.wav
FAILED      MD5          velvet.wav
OK          MD5          Waiting.wav

31 checksums failed
Title: New FLAC encoder
Post by: Wombat on 2006-10-18 14:17:49
I don´t know the story of flake but now while i test around i am not really thrilled to have the extra options.
These options like l32 or v1 simply brake the standard in my eyes. l32 encoded files don´t play on every hardware and v1 is confusing even winamp and my Squeezebox again when i want to fast forward.
What about *.flak for flake encodings? So everyone knows he may have to reencode to use it properly when he gets such a file by accident.
This may have been discudssed before but i somehow missed that.
Title: New FLAC encoder
Post by: MedO on 2006-10-18 16:06:02
These options like l32 or v1 simply brake the standard in my eyes.


As I understand it, files encoded with -v 1 or -l 32 are standard flac files, not a weird format extension introduced by flake.
Title: New FLAC encoder
Post by: bhoar on 2006-10-18 17:10:26
My results for Flake SVN r110 wisodev build P3 optimised with RW corpus.

Code: [Select]
Setting	 Ratio	 Enc	 Dec
=================================
0 65.352% 136x 106x
1 62.822% 114x 104x
2 62.661% 98x 104x
3 59.581% 94x 97x
4 59.392% 92x 93x
5 59.292% 90x 96x
6 59.649% 66x 96x
7 59.307% 54x 96x
8 59.132% 49x 95x
8 -m 1 59.115% 83x 94x
8 -v 1 58.747% 47x 93x
9 59.019% 40x 95x
10 59.006% 28x 94x
11 58.758% 22x 83x
12 58.729%   9x 83x
12 -v 1  58.378%   9x 84x
99 58.336%   2x 72x

I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").

Looking at the above, -5 seems to be the sweet spot if your rate performance as the most important and "-8 -v 1" the sweet spot if you rate compression as the most important.

-brendan
Title: New FLAC encoder
Post by: wisodev on 2006-10-18 17:39:40
Thanks wisodev.

I have an AMD XP CPU.  Should I be looking at /org/flake.exe, /p3/flake.exe or /flake.exe?


Code: [Select]
.\flake.exe        - ICL Build with PGO Optimizations
.\P3\flake.exe        - ICL Build with PGO & P3 Optimizations
.\ORG\flake.exe        - ICL Build


The P3 is optimized for Pentium III and should work for all processors above P3 specs. It Works very well on Athlon XP processors (I have Athlon XP 2000+). I have in mind to add P4 build but I can't build it on my machine because of PGO optimization that req running the program and my CPU doesn't support P4 builds.

The original (ORG) build is not considered for normal usage only added for speed testing of PGO and PGO&P3 builds. For most CPUs the .\flake.exe build is good and should be used as safest solution.

@wisodev

I would be interested what compiler flags you use (and which compiler). Have you added asm to the sources?


All I have added is build enviroment in win32 directory. No assembly optimizations added by me. I have added some small changes to code to build it cleanly under MSVC with Intel compiler. Maybe Justin can add them to official sources (just diff my sources with SVN revision 110 to see what have been changed).

I am using MSVC++ v7.1 and Intel C++ Compiler v8.0.040. This version of Intel compiler works best for me. I have tried newer versions but I can't get better results.

The compiler flags are main reason for speedups of my builds. I am using PGO optimization provided by Intel C++ compiler and for P3 build additionaly /QxK flag is used to enable even more optimizations. More information is avaible when you check the project settings for eacg configuraion (all witch begin with Release, the Debug builds are obviously not optimized ;-).
Title: New FLAC encoder
Post by: pest on 2006-10-18 18:08:00
I really have no idea what all this means, but it thought it worth noting.  I bit compared a few files that had mismatched MD5s using foobar and the audio is bit-identical, as you would expect.


You're right. FLAC creates a fresh RIFF/WAVE-Header, but Wavpack for example recreates the original one.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-18 20:19:17
Code: [Select]
.\flake.exe        - ICL Build with PGO Optimizations
.\P3\flake.exe        - ICL Build with PGO & P3 Optimizations
.\ORG\flake.exe        - ICL Build
The P3 is optimized for Pentium III and should work for all processors above P3 specs. It Works very well on Athlon XP processors (I have Athlon XP 2000+). <snip> The original (ORG) build is not considered for normal usage only added for speed testing of PGO and PGO&P3 builds. For most CPUs the .\flake.exe build is good and should be used as safest solution.
Thanks for the info wisodev. With that in mind I won't bother with the ORG build.  Here's the PGO and PGO+P3 results:

Code: [Select]
                   |   PGO + P3     |   PGO
Setting     Ratio  |   Enc     Dec  |   Enc     Dec
===================+================+==============
0         65.352%  |  136x    106x  |  133x    104x
1         62.822%  |  114x    104x  |  110x    106x
2         62.661%  |   98x    104x  |   95x    104x
3         59.581%  |   94x     97x  |   91x     98x
4         59.392%  |   92x     93x  |   90x     96x
5         59.292%  |   90x     96x  |   88x     96x
6         59.649%  |   66x     96x  |   65x     99x
7         59.307%  |   54x     96x  |   52x     98x
8         59.132%  |   49x     95x  |   47x     95x
8 -m 1    59.115%  |   83x     94x  |   80x     93x
8 -v 1    58.747%  |   47x     93x  |   46x     95x
9         59.019%  |   40x     95x  |   39x     96x
10        59.006%  |   28x     94x  |   27x     95x
11        58.758%  |   22x     83x  |   21x     83x
12        58.729%  |    9x     83x  |    8x     81x
12 -v 1   58.378%  |    9x     84x  |    8x     84x
99        58.336%  |    2x     72x  |    2x     72x

View the Google spreadsheet with all values for all files here (http://spreadsheets.google.com/ccc?key=pwAdtAz-_urKzEsJW438tFg).

You're right. FLAC creates a fresh RIFF/WAVE-Header, but Wavpack for example recreates the original one.
Thanks for the confirmation.  This makes sense to me.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-18 21:07:49
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Not sure if this fits the bill, but I can't think of any other way of displaying it.

(http://www.synthetic-soul.co.uk/temp/flake-svn-r110.gif)
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-19 02:19:17
I don´t know the story of flake but now while i test around i am not really thrilled to have the extra options.
These options like l32 or v1 simply brake the standard in my eyes. l32 encoded files don´t play on every hardware and v1 is confusing even winamp and my Squeezebox again when i want to fast forward.
What about *.flak for flake encodings? So everyone knows he may have to reencode to use it properly when he gets such a file by accident.
This may have been discudssed before but i somehow missed that.

The files may confuse some software decoders and skip in some hardware decoders, but I assure you they are 100% valid FLAC files based only on the specification.  The issues that occur are not Flake issues, they are due to there being 2 levels of compliance, the subset specification and the full specification.  Flake gives a warning when the files do not fall within the subset specification, but all generated files fall within the full specification.

If you're worried about getting a FLAC file which may cause problems, I could try to write a small utility program which would do frame analysis and detect whether a file falls within the subset specification.  As for winamp, maybe adding a seektable using metaflac would help?...I don't really know how the winamp plug-in does seeking and I don't use winamp (anymore).  I do use xmms, which completely locks up when seeking with the variable bitrate files...no good.  Personally, I see these as software bugs, but I can't really blame the makers since that feature of the specification is not implemented by the reference encoder.  MPlayer plays & seeks just fine...it uses its own demuxer & FFmpeg's decoder.

On an unrelated note, thanks for all the benchmarks!  The graph is great  Now that I see the results, I'm seriously considering abandoning the "mirror the reference encoder" idea for compression levels.  I think trimming it down to just 7 presets corresponding to  0, 1, 5, 8m1, 10, 11, and 99 should pretty much cover what is needed.

-0 for really-fast encoding
-1 for fast encoding
-5 for normal encoding
"-8 -m 1" for high compression
-10 for best compression within subset
-11 for extra compression outside subset (-12 speed loss isn't worth the tiny gain)
-99 for experimental

I would probably get rid of the numbered presets in favor of named presets.  What do you all think about this idea?

-Justin
Title: New FLAC encoder
Post by: bhoar on 2006-10-19 07:34:09
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Not sure if this fits the bill, but I can't think of any other way of displaying it...


Actually, that was quite useful.  The two points on the encoding line that stood out to me numerically after looking up and down the chart a few times also appear to stand out rather obviously in the chart.

-brendan
Title: New FLAC encoder
Post by: wisodev on 2006-10-19 08:01:37
@PrakashP

For PGO&P3 Instrumentation build (Release Instrumentation P3|Win32) I am using this options for ICL:

Code: [Select]
/GL /Ox /Og /Ob1 /Ot /G7 /GA /QxK /FD /MT /Qc99 /Qprof_genx /Gd /TC

For PGO&P3 Feedback build (Release Feedback P3|Win32) I am using this options for ICL:

Code: [Select]
/GL /Ox /Og /Ob1 /Ot /G7 /GA /QxK /FD /MT /Qc99 /Qprof_use /Gd /TC

For PGO only build just remove /QxK option.

wisodev
Title: New FLAC encoder
Post by: Wombat on 2006-10-19 13:03:19
The files may confuse some software decoders and skip in some hardware decoders, but I assure you they are 100% valid FLAC files based only on the specification.  The issues that occur are not Flake issues, they are due to there being 2 levels of compliance, the subset specification and the full specification.  Flake gives a warning when the files do not fall within the subset specification, but all generated files fall within the full specification.

If you're worried about getting a FLAC file which may cause problems, I could try to write a small utility program which would do frame analysis and detect whether a file falls within the subset specification.  As for winamp, maybe adding a seektable using metaflac would help?...I don't really know how the winamp plug-in does seeking and I don't use winamp (anymore).  I do use xmms, which completely locks up when seeking with the variable bitrate files...no good.  Personally, I see these as software bugs, but I can't really blame the makers since that feature of the specification is not implemented by the reference encoder.  MPlayer plays & seeks just fine...it uses its own demuxer & FFmpeg's decoder.

Thanks for your answer. I read the first time deeper about flac and switches. I am sure you can penetrate normal flac to the point it doesn´t work on any player also
No need for a tool that tests the frames. This -v1 doesn´t seem to need any more decoding power than without so it is time to "correct" the decoders.
btw. this latest wisodev build is awsome fast!
Title: New FLAC encoder
Post by: MedO on 2006-10-19 13:14:33
@Synthetic Soul:
Thanks for the tests, it would also be nice to have "-8 -m 1 -v 1" in next time, maybe instead of "-8 -v 1"? This should have a very good compression/encoding time ratio.
Title: New FLAC encoder
Post by: kockroach on 2006-10-19 13:26:18
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Not sure if this fits the bill, but I can't think of any other way of displaying it.

(http://www.synthetic-soul.co.uk/temp/flake-svn-r110.gif)

Synthetic Soul,

The plot of the results look great.  I was wondering why it is that "-m 1" is only tested on "8".  Can it not be used on levels 9 and up?  I didn't see anything about that, so I apologize if I missed it.  I have to wonder how much it could potentially increase encoding times of those above 9, especially if it can increase 12's encoding time significantly.
Title: New FLAC encoder
Post by: guruboolez on 2006-10-19 13:31:29
I tried -10 -m 1 on a single album and it gives a significant boost (~100%) to the encoding speed without decreasing the compression ratio (+1 kbps). This switch looks great!

I also tried -v1, but there's no seeking with foobar2000 (0.9.4.1).
Title: New FLAC encoder
Post by: MedO on 2006-10-19 13:46:43
I think "-9 -m 1" ist the same as "-8 -m 1".
"-9" and upward use search methods which are probably better (compression-wise) than "-m 1".
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-19 14:48:35
@Synthetic Soul:
Thanks for the tests, it would also be nice to have "-8 -m 1 -v 1" in next time, maybe instead of "-8 -v 1"? This should have a very good compression/encoding time ratio.
The plot of the results look great.  I was wondering why it is that "-m 1" is only tested on "8".  Can it not be used on levels 9 and up?  I didn't see anything about that, so I apologize if I missed it.  I have to wonder how much it could potentially increase encoding times of those above 9, especially if it can increase 12's encoding time significantly.
I simply used the settings requested by Justin, and added -8 -m 1 as it was mentioned by MedO shortly after, and it was suggested that more info was required (so I thought I may as well add it to the list).

My wife is possibly beginning labour at the moment, so I kind of have other things on my mind. 

If I get chance I will try to run the tests requested.
Title: New FLAC encoder
Post by: kockroach on 2006-10-19 14:59:43
My wife is possibly beginning labour at the moment, so I kind of have other things on my mind. 

That's awesome!  Hope all goes well.   
Title: New FLAC encoder
Post by: iGold on 2006-10-19 15:29:41
I think trimming it down to just 7 presets corresponding to  0, 1, 5, 8m1, 10, 11, and 99 should pretty much cover what is needed.
...
I would probably get rid of the numbered presets in favor of named presets.  What do you all think about this idea?

I think it's a good idea. And if we really want we always can use cryptic -b -t -l -m -r -s combinations
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-20 09:18:38
That's awesome!  Hope all goes well.   
Thank you.  Things are unfortunately going very slowly, so there's no news yet.

Except...

My results for Flake SVN r110 wisodev build P3 optimised with RW corpus, with -8 -m 1 -v 1 and -9 to -12 with -m 1:

Code: [Select]
Setting          Ratio     Enc     Dec
======================================
0              65.352%    136x    106x
1              62.822%    114x    104x
2              62.661%     98x    104x
3              59.581%     94x     97x
4              59.392%     92x     93x
5              59.292%     90x     96x
6              59.649%     66x     96x
7              59.307%     54x     96x
8              59.132%     49x     95x
8 -m 1         59.115%     83x     94x
8 -v 1         58.747%     47x     93x
8 -m 1 -v 1    58.752%     83x     96x
9              59.019%     40x     95x
9 -m 1         59.115%     85x     96x
10             59.006%     28x     94x
10 -m 1        59.115%     86x    100x
11             58.758%     22x     83x
11 -m 1        58.920%     67x     76x
12             58.729%      9x     83x
12 -m 1        58.920%     67x     76x
12 -v 1        58.378%      9x     84x
99             58.336%      2x     72x

I don't understand the -9 -m 1/-10 -m 1 and -11 -m 1/-12 -m 1 thing, but I'm sure someone does.
Title: New FLAC encoder
Post by: MedO on 2006-10-20 12:10:19
I don't understand the -9 -m 1/-10 -m 1 and -11 -m 1/-12 -m 1 thing, but I'm sure someone does.

Well, the only setting that changes between -8, -9 and -10 is the -m switch, and you override that manually, so you get the same results because the settings are identical. The same goes for -11/-12.

Edit: I just saw that -11 -m 1 is bigger than -8 -m 1. I guess the "estimate" search method just isn't optimised for those high predictor orders. Just a guess, I don't know much about FLAC technology...

Edit2: I confused rows in the table... thanks Josef Pohm for the hint. 
Title: New FLAC encoder
Post by: JWolf on 2006-10-21 04:17:14
I have a Rio Karma and while it can play FlAKE -12 compressed flac files, it has a lot of trouble seeking with these FLAC files. FLAC 1.1.2_CVS which gets close compression works great with my Rio Karma.

What I'd like to know is what are the best settings for Flake to work with my Rio Karma? I'd like to compare compressions sizes to find out which is better. I was playing around and ended up gettng a larger file that didn't work vs FLAC 1.1.2_CVS.

What setting is it in FLAKE that causes it not to be fully compatible with my Rio? I tried the -v 1 as suggested earlier in the thread to make it compatible with FLAC and that also failed.

Jon
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-21 05:10:10
I have a Rio Karma and while it can play FlAKE -12 compressed flac files, it has a lot of trouble seeking with these FLAC files. FLAC 1.1.2_CVS which gets close compression works great with my Rio Karma.

What I'd like to know is what are the best settings for Flake to work with my Rio Karma? I'd like to compare compressions sizes to find out which is better. I was playing around and ended up gettng a larger file that didn't work vs FLAC 1.1.2_CVS.

What setting is it in FLAKE that causes it not to be fully compatible with my Rio? I tried the -v 1 as suggested earlier in the thread to make it compatible with FLAC and that also failed.

Jon

For now the best setting for your Rio would probably be "flake -10".  It generates files which are roughly equivalent (compatibility-wise) to "flac -8".  Definitely don't use -v 1 since it is probably the most experimental option and has compatibility issues.

edit: to answer your other question more fully, it's probably the max LPC order which is causing problems on the Rio with flake -12 files.

edit2: Given the many questions, for the next release, I will try to make some documentation devoted to compatibility of various features of the FLAC specification and Flake encoding parameters.

-Justin
Title: New FLAC encoder
Post by: JWolf on 2006-10-21 10:13:06
I have a Rio Karma and while it can play FlAKE -12 compressed flac files, it has a lot of trouble seeking with these FLAC files. FLAC 1.1.2_CVS which gets close compression works great with my Rio Karma.

What I'd like to know is what are the best settings for Flake to work with my Rio Karma? I'd like to compare compressions sizes to find out which is better. I was playing around and ended up gettng a larger file that didn't work vs FLAC 1.1.2_CVS.

What setting is it in FLAKE that causes it not to be fully compatible with my Rio? I tried the -v 1 as suggested earlier in the thread to make it compatible with FLAC and that also failed.

Jon

For now the best setting for your Rio would probably be "flake -10".  It generates files which are roughly equivalent (compatibility-wise) to "flac -8".  Definitely don't use -v 1 since it is probably the most experimental option and has compatibility issues.

edit: to answer your other question more fully, it's probably the max LPC order which is causing problems on the Rio with flake -12 files.

edit2: Given the many questions, for the next release, I will try to make some documentation devoted to compatibility of various features of the FLAC specification and Flake encoding parameters.

-Justin


I just tested Flake at -10 and found that the compatability with seeking in my Rio Karma still isn't there. I did the same track in Flac 1.1.2_CVS and compatability is perfect. My settings for Flac are "-8 -A tukey(0.25) -A gauss(0.1875) -V --sector-align %1" and the file sizes are very similar between the two. 34,189,547 for Flake and 34,153,250 for Flac. There is something different in the Flake file that the Rio Karma is not liking. I just decided to try Flake at -1 and it still doesn't seek properly. Once it plays it will play allt he way fine. But as for seeking it eaier doesn't seek or takes a long time to seek.
Title: New FLAC encoder
Post by: Madman2003 on 2006-10-21 11:01:13
You do realize flake does not add seek points to the file (at least version 0.10 doesn't)? Try adding seekpoints using metaflac.
Title: New FLAC encoder
Post by: JWolf on 2006-10-21 12:15:24
You do realize flake does not add seek points to the file (at least version 0.10 doesn't)? Try adding seekpoints using metaflac.

I'll have to try that. I didn't realize. Thanks!
Title: New FLAC encoder
Post by: JWolf on 2006-10-21 12:27:53
Hot damn! Adding seek points works. Thank you Madman2003.
Title: New FLAC encoder
Post by: Justin Ruggles on 2006-10-22 19:12:34
My wife is possibly beginning labour at the moment, so I kind of have other things on my mind. 

Great avatar  and congratulations.
Title: New FLAC encoder
Post by: Will Fisher on 2006-10-25 15:31:57
'grats everyone, you made it into Winamp 5.31
http://forums.winamp.com/showthread.php?s=...threadid=258205 (http://forums.winamp.com/showthread.php?s=&threadid=258205)
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-25 15:55:16
Congratulations Justin.  Given both Winamp and FLAC's popularity this seems like an excellent achievement to me.

Great avatar  and congratulations.
Thank you.
Title: New FLAC encoder
Post by: jcoalson on 2006-10-25 20:06:22
'grats everyone, you made it into Winamp 5.31
http://forums.winamp.com/showthread.php?s=...threadid=258205 (http://forums.winamp.com/showthread.php?s=&threadid=258205)

just confirmed, it's using r108.  flake is neat and has promise but I think this is premature to use it in winamp now for the following reasons:

1. no seektable support which can make seeking awkwards.

2. still too easy to create non-subset streams.  the winamp plugin goes up to -12 and there is no warning for -11 or -12.

except in rare cases, non-subset streams are an extremely poor tradeoff of reduced usability for a tiny fraction of compression.  that's why even in libFLAC you have to explicitly turn off subset support when using non-subset options or the encoder init call will fail.

3. most important, flake is not well tested.  I built r108 and ran part of the FLAC test suite against it (stereo 16bps streams), and some files silently failed to generate correct FLAC files, i.e. encoding them appeared to work but the decoded file did not match the input.

users will be more forgiving for a decoder; another decoder can be used to recover a good FLAC file. but a lossless encoder is going to be held to a higher standard.  if it generates FLACs that are not lossless or cause problems on playback, that's more serious.

Josh


edit: I hope "flake is not well tested" doesn't sound critical, what I mean is not tested well-enough as a lossless encoder for production use.
Title: New FLAC encoder
Post by: MedO on 2006-10-25 21:10:28
3. most important, flake is not well tested.  I built r108 and ran part of the FLAC test suite against it (stereo 16bps streams), and some files silently failed to generate correct FLAC files, i.e. encoding them appeared to work but the decoded file did not match the input.


Argh, stupid me. I already used flake in my daily "work", but for security wrote a small wrapper script that would use flac -d and an md5summer to ensure the created flac file was ok. I thought this way I could safely use flake and also give some feedback when I encountered a problem. However, I just found out the script has a bug and let any errors through without warning  . Maybe I should have tested it *before* using... ah well.

In what way were the files damaged, i.e. would flactester catch faulty ones?
Title: New FLAC encoder
Post by: jcoalson on 2006-10-25 21:17:23
In what way were the files damaged, i.e. would flactester catch faulty ones?

I will check it out more tonight and upload a sample, but the FLACs were valid, they just decoded to different audio than the input.  also the MD5 sum was wrong.

Josh
Title: New FLAC encoder
Post by: Egor on 2006-10-27 19:38:16
also the MD5 sum was wrong.

Does it mean that flake stored wrong MD5 of the original audio? I mean, is it secure to test flake implementation with just "flac -t", or is it necessary to decode and compare with the original?


BTW, shouldn't it be "we recommend not to use it for archival" instead of "we recommend to not use it for archival"? 
Title: New FLAC encoder
Post by: jcoalson on 2006-10-27 21:02:07
Does it mean that flake stored wrong MD5 of the original audio? I mean, is it secure to test flake implementation with just "flac -t", or is it necessary to decode and compare with the original?

flac -t does not catch it as the flake file has a correct MD5 sum for its audio, but the audio is not the same as the original wave.

Josh
Title: New FLAC encoder
Post by: MedO on 2006-10-28 09:53:18
Does it mean that flake stored wrong MD5 of the original audio? I mean, is it secure to test flake implementation with just "flac -t", or is it necessary to decode and compare with the original?

flac -t does not catch it as the flake file has a correct MD5 sum for its audio, but the audio is not the same as the original wave.

Josh


Is the difference large or is it just, like, the least significant bit in a few samples changed?
Title: New FLAC encoder
Post by: Egor on 2006-10-28 10:36:20
Is the difference large or is it just, like, the least significant bit in a few samples changed?

He-he    that doesn't matter. The question is whether flake is lossless (no data lost or changed) or not.
Title: New FLAC encoder
Post by: MedO on 2006-10-28 10:57:42
Is the difference large or is it just, like, the least significant bit in a few samples changed?

He-he    that doesn't matter. The question is whether flake is lossless (no data lost or changed) or not.


I know that this is a serious bug, but I'm trying to decide whether to re-rip the CDs I already compressed with flake.
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-10-28 11:54:00
Question regarding setting up foobar with Flake moved to this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=49658&hl=). Issue is foobar-/user-related, not a Flake issue.
Title: New FLAC encoder
Post by: jcoalson on 2006-10-30 06:07:29
ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio.  when I have more time after the next release I will look into the rest of the suite.

Josh
[attachment=2624:attachment]
Title: New FLAC encoder
Post by: Madman2003 on 2006-11-05 10:07:13
ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio.  when I have more time after the next release I will look into the rest of the suite.

Josh
[attachment=2624:attachment]


I'll asume that the problem exists, i'm curious how you made that file (in other words, should i worry about some flac's i created).
Title: New FLAC encoder
Post by: wisodev on 2006-11-06 17:26:59
Flake (Win32) from SVN revision 112 available:

Note:
I've changed way I build binarys. The old method used visual studio project files. Now I'm using simple batch script (build_intel.cmd available in src archive) to directly compile and link binarys using only Intel C++ Compiler. As results you don't need Visual Studio to build flake under windows. You need only Intel C++ Compiler and MS Platform SDK. The old project files are still available. And one more thing it seems that new binarys are just bit faster ;-)

Download from my Website (http://win32builds.sourceforge.net/flake/index.html)

or use direct links:

Download binarys:
flake-svn-r112-win32-bin.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r112-win32-bin.zip?download)

Download sources:
flake-svn-r112-win32-src.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r112-win32-src.zip?download)

wisodev

EDIT: changed url to downloads and my website
Title: New FLAC encoder
Post by: JWolf on 2006-11-06 23:21:55
ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio.  when I have more time after the next release I will look into the rest of the suite.

Josh
[attachment=2624:attachment]

Turns out FLAC 1.1.3 beta 2 also cannot handle this WAV file without an error.

flac 1.1.3beta2, Copyright © 2000,2001,2002,2003,2004,2005,2006  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

options: -P 8192 -b 4608 -m -l 12 -e -q 0 -r 0,6
wacky2.wav: WARNING: skipping unknown sub-chunk 'fact'
wacky2.wav: WARNING: skipping unknown sub-chunk 'pad '
wacky2.wav: WARNING: skipping unknown sub-chunk 'pad '
wacky2.wav: wrote 8328 bytes, ratio=138.800

The resulting WAV from the FLAC file no longer matches. The original is 96 bytes and the decompressed one is 60 bytes. FLAKE 's decompressed WAV file is 72 bytes.
Title: New FLAC encoder
Post by: jcoalson on 2006-11-06 23:38:55
no, that's not an error, it's a warning that it is skipping unsupported subchunks.  the audio is correctly compressed with flac-1.1.3 (and earlier).

with flake, the non-audio subchunk that follows the audio is treated as audio and compressed along with the audio.  that's why when decoded it decodes to more audio samples than are in the original wave file.

Josh
Title: New FLAC encoder
Post by: Will Fisher on 2006-11-07 12:14:20
no, that's not an error, it's a warning that it is skipping unsupported subchunks.  the audio is correctly compressed with flac-1.1.3 (and earlier).

with flake, the non-audio subchunk that follows the audio is treated as audio and compressed along with the audio.  that's why when decoded it decodes to more audio samples than are in the original wave file.

Josh
This would be a bug in the flake wav reading code. Winamp doesn't use this code, our WAV reading is done by libsndfile (which as far as I know, doesn't have this problem).
Title: New FLAC encoder
Post by: mpxplay on 2006-11-08 19:57:24
I've made a FLAKE output plugin for Mpxplay and I've added ID3-info (artist,title,etc.) handling for the vorbis-comment function(s). Something similar would be nice in the original flake lib too.

You can download the modified source of v0.10 here: http://www.freewebtown.com/mpxplay (http://www.freewebtown.com/mpxplay) -> SCFLACS.ZIP

A short list about the modifications (maybe not complete):
- a new struct FlakeComments_t in flake.h, added to the end of struct FlakeContext
- flake_comment_init_utf8(FlakeContext *s,char *codepage) function to enable UTF8 text encoding (you have to call it by standard), the "codepage" is required for DOS version only (set it to NULL)
- flake_comment_add(FlakeContext *s,char *type,char *data), where the "type" can be "artist","title","album","date","genre","comment" or other standard vorbiscomment types
- modified write_vorbis_comment() function in encode.c
- added AnsiToUTF8 function to encode.c, but the the WIN32 version is not tested at all

You have to call flake_comment_init_utf8() and flake_comment_add() functions first,
then the flake_encode_init().

regards
Attila
Title: New FLAC encoder
Post by: Northpole on 2006-11-12 00:22:16
I downloaded the SVN zip (http://subversion.tigris.org/downloads/svn-win32-1.4.0.zip), unzipped (I just unzipped to my desktop for speed), opened a command line in the "bin" ("\svn-win32-1.4.0\svn-win32-1.4.0\bin") directory and then used the command:

svn co https://svn.sourceforge.net/svnroot/flake-enc (https://svn.sourceforge.net/svnroot/flake-enc) flake-enc

This created a folder in my "bin" directory called "flake-enc".  I then moved that to somewhere more convenient (a short path with no spaces) and then ran "./configure" then "make" using a cygwin console.


@Synthetic Soul,

I downloaded and tried to compile flake with mingw like your above instructions, but didn't have any luck.  The download created the flake "bin" directory and I moved it to a easy to get at location as you did.  What command line did you use and did you enter it in the MSYS console?

I have MSYS-1.0.11 installed.  I have tried the "configure&&make" command in the MSYS console window but all I get is "exec: bash: not found

Any pointer would be most welcome.
Title: New FLAC encoder
Post by: Mangix on 2006-11-12 02:10:25
get bash.

http://prdownloads.sf.net/mingw/bash-2.05b...ar.bz2?download (http://prdownloads.sf.net/mingw/bash-2.05b-MSYS.tar.bz2?download)
Title: New FLAC encoder
Post by: Northpole on 2006-11-13 15:56:58
@Mangix

Thanks for the tip.  Worked good.
Title: New FLAC encoder
Post by: JWolf on 2006-12-13 13:10:23
Can someone please post a Win32 compile of FLAKE 112 please? Thanks!
Title: New FLAC encoder
Post by: Synthetic Soul on 2006-12-13 13:40:11
As per post #319:

http://win32builds.sourceforge.net/flake/index.html (http://win32builds.sourceforge.net/flake/index.html)
Title: New FLAC encoder
Post by: JWolf on 2006-12-16 19:27:58
As per post #319:

http://win32builds.sourceforge.net/flake/index.html (http://win32builds.sourceforge.net/flake/index.html)

Nice. Now I can bookmark it. Thank you!
Title: New FLAC encoder
Post by: wisodev on 2006-12-17 16:48:54
I have uploaded new experimental Win64 builds of FLAKE using latest sources. Please try this builds and submit results.

Binarys for 64 bit windows (should work on EM64T and AMD64 compatible processors):

flake-svn-r112-win64-bin.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r112-win64-bin.zip?download)

Other related downloads are placed here (https://sourceforge.net/project/showfiles.php?group_id=183195).
Title: New FLAC encoder
Post by: miros on 2006-12-17 18:06:40
The variable block size mode has a small bug that makes seeking impossible. In frame header is stored frame number instead of sample number. A quick hack:

Code: [Select]
Index: libflake/encode.c
===================================================================
--- libflake/encode.c    (revision 112)
+++ libflake/encode.c    (working copy)
@@ -973,7 +973,10 @@
         assert(!ctx->bw->eof);
     }
     if(frame_buffer != NULL) {
-        ctx->frame_count++;
+        if(ctx->params.variable_block_size)
+            ctx->frame_count += s->params.block_size;
+        else
+            ctx->frame_count++;
     }
     return bitwriter_count(ctx->bw);
}
Title: New FLAC encoder
Post by: jcoalson on 2006-12-18 19:50:46
yes.  actually, storing the frame number in a variable-blocksize file is illegal according to the FLAC spec.

Josh
Title: New FLAC encoder
Post by: Justin Ruggles on 2007-01-09 01:01:18
ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio.  when I have more time after the next release I will look into the rest of the suite.
[attachment=2624:attachment]


You're right about the bug.  It is now fixed.  The wav reader was only checking for end-of-file not end-of-data-chunk.  Thanks for the bug repot...sorry for the slow reply.  I'm back doing development after some unexpected downtime.

yes.  actually, storing the frame number in a variable-blocksize file is illegal according to the FLAC spec.


This issue has been fixed as well.  Thanks miros for the info.

-Justin
Title: New FLAC encoder
Post by: wisodev on 2007-01-09 20:32:22
My Flake SVN builds at revision 114 are available for download.

http://win32builds.sourceforge.net/flake/index.html (http://win32builds.sourceforge.net/flake/index.html)

Download binaries: flake-svn-r114-bin.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r114-bin.zip?download)

Download sources: flake-svn-r114-src.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r114-src.zip?download)

wisodev
Title: New FLAC encoder
Post by: NuSkooler on 2007-01-10 01:07:39
Using the file flake.exe in ~/flake-svn-r114-bin/exe_x64 I get the following error:
Quote
Fatal Error : This program was not built to run on the processor in your system.


I am using a AMD 64 X2 5000+ on Windows XP 64. Am I wrong in thinking this should be the proper executable or is this a bug?
Title: New FLAC encoder
Post by: jcoalson on 2007-01-10 01:20:01
I've reworked the flac test suite to use flake.  it actually took a bit of work since the options are so different but I think the coverage is pretty good for the flac test streams at least.  it could use more tests that exercise specific features of flake.

but, even then the tests crash on a lot of inputs.  also there are places where it doesn't handle odd resolutions correctly; but except for those, I haven't run across any places where it is silently non-lossless yet (the worst kind of bug).  the tests are still running though.

Justin, I'll PM you about the scripts so if you want you can try to reproduce them.

Josh
Title: New FLAC encoder
Post by: jcoalson on 2007-01-10 01:37:53
PS uploaded here: http://www.hydrogenaudio.org/forums/index....showtopic=51745 (http://www.hydrogenaudio.org/forums/index.php?showtopic=51745)
Title: New FLAC encoder
Post by: wisodev on 2007-01-10 12:15:13
Quote
Using the file flake.exe in ~/flake-svn-r114-bin/exe_x64 I get the following error:
Quote

Fatal Error : This program was not built to run on the processor in your system.

I am using a AMD 64 X2 5000+ on Windows XP 64. Am I wrong in thinking this should be the proper executable or is this a bug?

I was using as target system based on a processor with Intel® EM64T running Windows Server 2003 x64 Edition or Windows XP Professional x64 Edition as described in documentation. I was thinking that this should work on Intel and AMD processor running on 64 bit CPU, but now I am not sure. I need to investigate this or maybe someone can put more light here? Sorry for any inconvenience from my side.

Edit:
From compiler documantation (I used /QxP):
Quote
/QxP
Intel® Core™ Duo processors and Intel® Core™ Solo processors, Intel Pentium 4 processors with Streaming SIMD Extensions 3, and compatible Intel processors with Streaming SIMD Extensions 3 (SSE3)

Quote
If a program compiled with -x (Linux) or /Qx (Windows) is executed on a non-compatible processor, it might fail with an illegal instruction exception or display other unexpected behavior. Executing programs compiled with -xN, -xB or -xP (Linux) or /QxN, /QxB or /QxP (Windows) on unsupported processors will display a run-time error similar to the following:

Fatal Error : This program was not built to run on the processor in your system.
Title: New FLAC encoder
Post by: alvaro84 on 2007-01-10 16:45:50
AFAIK Intel's compiler generates such a code that explicitly checks for the 'GenuineIntel' string in the CPUID, and if it started on an AMD processor, it simply quits even though an X2 should run the program without problems. Someone should try to 'crack out' this check from the startup code...
Title: New FLAC encoder
Post by: PrakashP on 2007-01-10 17:09:35
It might not worth the trouble, because usually gcc generates better code for Athlon CPUs - if optimization for specific Athlon CPUs is chosen.
Title: New FLAC encoder
Post by: NuSkooler on 2007-01-10 18:16:38
I was using as target system based on a processor with Intel® EM64T running Windows Server 2003 x64 Edition or Windows XP Professional x64 Edition as described in documentation. I was thinking that this should work on Intel and AMD processor running on 64 bit CPU, but now I am not sure. I need to investigate this or maybe someone can put more light here? Sorry for any inconvenience from my side.


I'm not sure if this is the problem, but Intel does have the Itanium 64 chipset that uses a different instruction base than the "x86" 64's (AMD 64's, etc.). If compiled for said chips, the code would not function on non-Itanium processors. AFAIK, this is only for special builds of Windows XP / 2003 and M$ has started (Vista+) supporting the more standard "x86" instruction set that is used by AMD.
Title: New FLAC encoder
Post by: wisodev on 2007-01-13 12:33:17
I have added new flake builds to my win32builds (http://www.sf.net/projects/win32builds) project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.

Download:

Visual Studio 2005 builds of flake svn revision 114. (http://prdownloads.sourceforge.net/win32builds/flake-vc2005-r114.zip?download)

Visual Studio 2005 and Intel C++ Compiler 9.1 builds of flake svn revision 114. (http://prdownloads.sourceforge.net/win32builds/flake-icl91-r114.zip?download)

Thanks,
wisodev
Title: New FLAC encoder
Post by: NuSkooler on 2007-01-14 02:09:33
I have added new flake builds to my win32builds (http://www.sf.net/projects/win32builds) project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.


I just tested the release binaries of both the M$ and Intel compilers. Good news: They both work great!
Title: New FLAC encoder
Post by: wisodev on 2007-01-14 12:38:33

I have added new flake builds to my win32builds (http://www.sf.net/projects/win32builds) project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.


I just tested the release binaries of both the M$ and Intel compilers. Good news: They both work great!


Great news!

Anyway both releases have some potential for speed up. I didn't used PGO optimizations (VC2005 compiler has also this feature).
Title: New FLAC encoder
Post by: Caroliano on 2007-01-24 21:37:44
I have an MD5 mismach with the "flake-0.10-win32-wisodev-bin" that I downloaded from Flake SourceForge project page. If I compare with foobar, I get: "ERROR (foo_bitcompare) : Files have different length". Indeed,  the generated file is 1211905 samples long, while the original wav file is 1211894.

With the "flake-icl91-r114", also compiled by wisodev, I have an perfect output. So, whatever it was, is already corrected.

BUT not in the Flake SourceForge project page, that is where people expect to find an good binary. We need a new official release, if this error is real.

I can reproduce the error here, and you can find the source file here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=41248&view=findpost&p=363343), to check if it is realy an error. The file for download was encoded by an old version of Flac, not by Flake.
Title: New FLAC encoder
Post by: Justin Ruggles on 2007-01-25 01:46:20
I have an MD5 mismach with the "flake-0.10-win32-wisodev-bin" that I downloaded from Flake SourceForge project page. If I compare with foobar, I get: "ERROR (foo_bitcompare) : Files have different length". Indeed,  the generated file is 1211905 samples long, while the original wav file is 1211894.

With the "flake-icl91-r114", also compiled by wisodev, I have an perfect output. So, whatever it was, is already corrected.

BUT not in the Flake SourceForge project page, that is where people expect to find an good binary. We need a new official release, if this error is real.

I can reproduce the error here, and you can find the source file here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=41248&view=findpost&p=363343), to check if it is realy an error. It was encoded by an old version of Flac, not by Flake.


I could not reproduce, but I believe you and agree with your point.  There have been some critical bug fixes since version 0.10 was released.  I am on vacation next week, so I hope to be able to package up a new release and put it up on Sourceforge.  I also hope to get Flake to pass the FLAC test suite before then.
Title: New FLAC encoder
Post by: jcoalson on 2007-01-26 22:46:09
Justin, I found another bug that may be the culprit.  I've gone back to finish the speedups of the libFLAC bitbuffer I started a long time ago and realized it is similar to flake's.  then I noticed a problem in bitwriter_writebits():
Code: [Select]
static inline void
bitwriter_writebits(BitWriter *bw, int bits, uint32_t val)
{
    assert(bits == 32 || val < (1U << bits));

    if(bits == 0) return;
    if(bw->eof || (bw->buf_ptr+3) >= bw->buf_end) {
        bw->eof = 1;
    } else {
        if(bits < bw->bit_left) {
            bw->bit_buf = (bw->bit_buf << bits) | val;
            bw->bit_left -= bits;
        } else {
            bw->bit_buf <<= bw->bit_left;    ///////////// <-------- HERE
            bw->bit_buf |= val >> (bits - bw->bit_left);
            if(bw->buffer != NULL) {
                *(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
            }
            bw->buf_ptr += 4;
            bw->bit_left += (32 - bits);
            bw->bit_buf = val;
        }
    }
}


if bits==32 and bw->bit_left==32, then it will get to that line and try to shift left by 32 bits.  according to C semantics that is a NOP, so bit_buf will have junk instead of being 0 for the next instruction.

I think it's just lucky that it is rarely (never?) triggered.  most writes are <32bit quantities and you do do 31 bits at a time of the rice unary portion.  but some metadata writes are doing 32 bits and if they are word-aligned there could be trouble.

in my new writer I am doing something like this (trying to translate back into flake-style, may be buggy):

Code: [Select]
        } else if(bw->bit_left<32) {
            bw->bit_buf <<= bw->bit_left;    ///////////// <-------- OK now
            bw->bit_buf |= val >> (bits - bw->bit_left);
            if(bw->buffer != NULL) {
                *(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
            }
            bw->buf_ptr += 4;
            bw->bit_left += (32 - bits);
            bw->bit_buf = val;
        } else {
            assert(bits==32);
            bw->bit_buf = val;
            if(bw->buffer != NULL) {
                *(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
            }
            bw->buf_ptr += 4;
        }

it can actually be optimized more but that's the idea.

Josh
Title: New FLAC encoder
Post by: Justin Ruggles on 2007-01-27 00:21:18
Justin, I found another bug that may be the culprit.  I've gone back to finish the speedups of the libFLAC bitbuffer I started a long time ago and realized it is similar to flake's.  then I noticed a problem in bitwriter_writebits():
...
if bits==32 and bw->bit_left==32, then it will get to that line and try to shift left by 32 bits.  according to C semantics that is a NOP, so bit_buf will have junk instead of being 0 for the next instruction.

I think it's just lucky that it is rarely (never?) triggered.  most writes are <32bit quantities and you do do 31 bits at a time of the rice unary portion.  but some metadata writes are doing 32 bits and if they are word-aligned there could be trouble.
...

Thanks Josh!  I actually did not know that a left shift by 32 was a NOP.  The code which this bit writer was based upon only allowed for doing 31 bits at once, and I did not catch that outside case when trying to adapt it to handle 32 bits.  I applied the fix as you suggested.

-Justin
Title: New FLAC encoder
Post by: Madman2003 on 2007-02-16 15:09:02
What's the state of flake in the sense of trustworthyness?

Also, any info if/when metadata support becomes available?
Title: New FLAC encoder
Post by: Justin Ruggles on 2007-02-16 20:29:38
What's the state of flake in the sense of trustworthyness?

Also, any info if/when metadata support becomes available?

I'll let others answer as to general trustworthyness.  I will say that Flake does generate MD5 checksums, but it does not have the verify-during-encoding option like flac does.

Metadata is far down the TODO list.  For now, the only option is to apply metadata to flake-generated files using metaflac.
Title: New FLAC encoder
Post by: Madman2003 on 2007-02-17 21:42:44
I'm looking forward to a "fully featured" flake, it's better suited to my system.
Flake is considerably(4-5x at -8) faster than flac for me. Anyone who can tell anything about the trustworthyness? (in other words, encountered problems with it)