A new FLAC encoder has been developed, which is proposed for inclusion in ffmpeg. The author claims that in high-compression mode, it can compress certain files better than libFLAC.
http://mplayerhq.hu/pipermail/ffmpeg-devel...May/011385.html (http://mplayerhq.hu/pipermail/ffmpeg-devel/2006-May/011385.html)
For those who want to test it, I've made (http://www.hydrogenaudio.org/forums/index.php?showtopic=45016#) a compile for win32 plateforms.
booo...it crashes.
it seems like a nice clean interface. some notes:
1. at the higher compression settings it is using large block sizes (16k for 44.1kHz) and high lpc order (32) which increase the decoding burden with little improvement (at least in my tests). (e.g. I'm not sure the squeezebox supports 16k blocks due to buffer requirements.) also, performance with larger blocksizes in more dependent on the audio characteristics so his results might be biased by his sample.
2. it has an interesting way of selecting the stereo decorrelation method which is faster than the reference encoder; whether it's better on not needs some testing. same for fixed/lpc subframe selection.
3. it's guessing at the max frame size without supporting verbatim encoding, which might explain it crashing
4. the lpc analysis stage looks just like the reference encoder.
so I think any compression gain is just due to the larger blocksize and high lpc order, which you can do with the reference encoder (though I wouldn't encourage that for general use).
Josh
Hi,
I am the person writing the encoder mentioned above. It is in an extremely experimental phase at the moment. I am fairly new to audio encoding, so most of what I am doing is just learning by trying things out and reading as much as I can. I do enjoy it though. I want to say that I greatly admire what Josh has done with the flac format and reference software. Its openness and good documentation are a rarity and a pleasure. I am writing an encoder for FFmpeg because one of the goals of that project is to write its own decoders and encoders so as not to rely on too many external libraries, and because of licensing (which I don't think is really much of an issue in the case of FLAC). Also, I am doing it for the shear pleasure of learning something new. My claims of better encoding are more due to excitement at the results than anything else. Beating the reference encoder is not my goal. It's just a good motivator, both for the excitement when I succeed and for the challenge when I can't seem to get anywhere close.
it seems like a nice clean interface. some notes:
1. at the higher compression settings it is using large block sizes (16k for 44.1kHz) and high lpc order (32) which increase the decoding burden with little improvement (at least in my tests). (e.g. I'm not sure the squeezebox supports 16k blocks due to buffer requirements.) also, performance with larger blocksizes in more dependent on the audio characteristics so his results might be biased by his sample.
That is true. I have found some odd results in my tests though. Many times I have seen my encoder do much better on a particular clip with, say, max lpc order of 32, but do not see the same benefit on the same clip with the reference encoder. I couldn't say why or how though. I am mostly just trying to test anything I can and observing what settings work best in which situations. You do have a point about buffer requirements. I had to increase the SDL audio buffer size in ffplay to get the high blocksize clips to play without skipping.
The issue with block sizes and different audio characteristics is one I am still struggling with. It seems to be one of the biggest drawbacks to really good compression quality. Variable blocksize would be ideal, but I know the compatiblity issues. Given the limitations, one of my objectives is to have good compatible modes, but also to provide more options for better compression. The reference encoder seems to go for the middle ground in many cases. This has its benefits, but personally, I like more options.
2. it has an interesting way of selecting the stereo decorrelation method which is faster than the reference encoder; whether it's better on not needs some testing. same for fixed/lpc subframe selection.
Yeah, the stereo decorrelation selection was just a guess that turned out to work okay. I haven't looked closely enough to see how the reference encoder does it. As for fixed/lpc subframe selection...I can't really remember how I was doing that on May 26 . I have had too many abandoned approaches to keep up... Right now I am using fixed mode (and mostly just order 2) only in the fastest encoding setting and for small frames (usually just the last frame).
3. it's guessing at the max frame size without supporting verbatim encoding, which might explain it crashing
I have no idea about the crashing on windows, as I use Linux. As for verbatim encoding, I actually started the program as verbatim only just to get the bitstream right. I later removed it. I probably should add it back in as a backup mechanism for audio that really just won't compress...a rarity, but it would save the guessing you mentioned. For now, I have at least added a warning & skip frames that are too big. It's not ideal, nor lossless, but at least it gives a warning...
4. the lpc analysis stage looks just like the reference encoder.
Yeah, it pretty much is. I have modified the autocorrelation slightly and implemented my own quantizer. The levinson-durbin function is the same. I experimented some with other ways of estimating the best predictor order, but it was not as good as the reference encoder. When I coupled the 2 together and selected the best by counting the bits I got slightly better encoding...that might have just been chance though I could have probably selected an order at random and done the same thing. I guess I'll have to keep researching. It's not that I think the reference encoder has a bad solution...I just like a challenge. I recently added, and am currently experimenting with, window functions before autocorrelation. Yesterday I was pointed to a Hydrogenaudio thread on how this is being added to the reference encoder as well. Very exciting stuff!
so I think any compression gain is just due to the larger blocksize and high lpc order, which you can do with the reference encoder (though I wouldn't encourage that for general use).
Blocksize I understand, but what's wrong with a higher LPC order? True, it isn't ideal for every situation, but it does help in many cases. I haven't noticed much of an impact on decoding, and I have a quite slow computer.
Anyway, I just thought I would respond. I am new to Hydrogenaudio, but since I seem to have thrown myself into the fray I might as well join in on the discussion.
-Justin
Hi Justin,
Thank you very much for this work (fork ?)...
I've made a benchmarck on my AMD64 3200+. I've tested the last flake 0.2 and flac from the last cvs entry with a wav sample of 1h11m55s (mostly electro and pop rock).
Here are my results:
1°) Flake 0.2 compiled with gcc 4.1.0 & CFLAGS = -g -O3 -Wall -pedantic -std=c99 -march=k8
| level | Real Time | Bytes | Ratio | kbps |
| 0 | 2m11.511s | 530 951 235 | 0,697 | 984 |
| 1 | 2m34.031s | 513 566 686 | 0.675 | 952 |
| 2 | 2m34.206s | 499 679 481 | 0,656 | 926 |
| 3 | 2m36.720s | 491 472 721 | 0,645 | 911 |
| 4 | 2m35.962s | 488 370 918 | 0,641 | 905 |
| 5 | 2m50.350s | 487 493 404 | 0,640 | 904 |
| 6 | 2m51.587s | 487 565 406 | 0,640 | 904 |
| 7 | 2m50.688s | 490 601 577 | 0,644 | 910 |
| 8 | 4m13.558s | 490 820 897 | 0,644 | 910 |
2°) FLAC CVS 20060603 compiled with gcc 4.1.0
| level | Real Time | Bytes | Ratio | kbps |
| 0 | 1m16.089s | 531 245 719 | 0,698 | 984 |
| 1 | 1m20.170s | 510 802 749 | 0.671 | 947 |
| 2 | 1m25.361s | 505 186 083 | 0,663 | 940 |
| 3 | 1m32.409s | 501 361 052 | 0,652 | 933 |
| 4 | 1m35.588s | 489 589 293 | 0,643 | 908 |
| 5 | 1m50.881s | 488 073 198 | 0,641 | 905 |
| 6 | 2m04.660s | 487 691 205 | 0,641 | 904 |
| 7 | 6m59.237s | 486 851 452 | 0,640 | 903 |
| 8 | 9m00.266s | 485 704 730 | 0,638 | 901 |
Now, I've a question : why flake crashes during encoding on win32 plateform ? In fact, compilation is fine but running fails. Any idea ?
Thank you very much for this work (fork ?)...
You're welcome. Fork? Well, I suppose that if I get the LPC order selection code replaced with my own then I can claim it to be nearly non-derived . The Levinson-Durbin LPC generation code is nearly identical to that in several other programs I've seen. I copied the UTF-8 code basically because I'm not 100% clear on how that encoding works. I suppose I should separate these 3 or 4 functions into a single file with proper copyright notice and attribution...or at least put a small notice above each and include both a COPYING.libFLAC file and COPYING.lgpl file. It wasn't really a fork though. I basically started from scratch using the specs and some reused FFmpeg code. I even tried my own UTF-8 function working backwards from the decoding in FFmpeg...I could never get it to work though, so I copied in that function. When adding non-fixed LPC I tried my hand at that as well, but fell short. So it's more of an independent implementation with crutches. The structure is my own, but the puzzle had to be filled-in in a few places.
I've made a benchmarck on my AMD64 3200+. I've tested the last flake 0.2 and flac from the last cvs entry with a wav sample of 1h11m55s (mostly electro and pop rock).
Here are my results:
1°) Flake 0.2 compiled with gcc 4.1.0 & CFLAGS = -g -O3 -Wall -pedantic -std=c99 -march=k8
| level | Real Time | Bytes | Ratio | kbps |
[...]
| 5 | 2m50.350s | 487 493 404 | 0,640 | 904 |
2°) FLAC CVS 20060603 compiled with gcc 4.1.0
| level | Real Time | Bytes | Ratio | kbps |
[...]
| 8 | 9m00.266s | 485 704 730 | 0,638 | 901 |
I also noticed similar results (not quite as good as libFLAC) on samples where flake did the best with smaller block sizes (i.e. level 5, which is blocksize 2048), but that the reference encoder did best at level 8 (blocksize 4608). I am still trying to wrap my head around this issue. It is possibly due to whatever libFLAC does in "exhaustive" mode? Or maybe my version of stereo decorrelation works well only on audio with certain characteristics? Testing, testing, and more testing. I'm working on my own horribly slow "brute-force" mode to help me out in my tests.
Now, I've a question : why flake crashes during encoding on win32 plateform ? In fact, compilation is fine but running fails. Any idea ?
I really have no clue here. What does the crash look like? Is it a segfault? Any gdb info? If I had Windows I would test it myself. If MinGW is being used, maybe it needs "--enable-memalign-hack"...? It has been several..5..6.. years since I have done any gcc compilation in Windows.
-Justin
I really have no clue here. What does the crash look like? Is it a segfault? Any gdb info? If I had Windows I would test it myself. If MinGW is being used, maybe it needs "--enable-memalign-hack"...? It has been several..5..6.. years since I have done any gcc compilation in Windows.
Through wine, here is the debug :
lionel@kubuntu:~/flake$ wine ./flake.exe -2 /home/lionel/wav/input.wav /home/lionel/wav/output.flac
Flake: FLAC audio encoder
Signed 16-bit 44100 Hz stereo
samples: 190288560
wine: Unhandled page fault on write access to 0x7f9a0dd4 at address 0x405703 (thread 0009), starting debugger...
WineDbg starting on pid 0x8
Unhandled exception: page fault on write access to 0x7f9a0dd4 in 32-bit code (0x00405703).
Register dump:
CS:0073 SS:007b DS:007b ES:007b FS:1007 GS:0033
EIP:00405703 ESP:7fbafdcc EBP:7fbafdd8 EFLAGS:00210206( - 00 - RIP1)
EAX:001f4514 EBX:ffffffff ECX:7f9a0dd4 EDX:7fd085e0
ESI:00401210 EDI:7ffdf760
Stack dump:
0x7fbafdcc: 7fcf0020 004018f6 ffffffff 7fbafeb8
0x7fbafddc: 00403306 7fd085e0 7fbafe60 0b5792b0
0x7fbafdec: 00402fd4 7fcf0020 00000000 7fcf0020
0x7fbafdfc: 00402fcf 7fd08c38 7fd0945c 7fbafe28
0x7fbafe0c: 00000010 7fcf0000 00000000 7fd084b0
0x7fbafe1c: 00000000 7fcf0000 7fc946dc 7fbafe48
0200: sel=1007 base=7fe6c000 limit=00001fff 32-bit rw-
Backtrace:
=>1 0x00405703 in flake (+0x5703) (0x00405703)
2 0x00403306 in flake (+0x3306) (0x00403306)
3 0x004011d9 __mingw_CRTStartup+0xc9 [/build/buildd/mingw32-runtime-3.8/build_dir/src/mingw-runtime-3.8/crt1.c:226] in flake (0x004011d9)
4 0x00401223 in flake (+0x1223) (0x00401223)
5 0x7fc5b311 in kernel32 (+0x4b311) (0x7fc5b311)
6 0xb7ed8ddb wine_switch_to_stack+0x17 in libwine.so.1 (0xb7ed8ddb)
0x00405703: orl $0,0x0(%ecx)
Modules:
Module Address Debug info Name (19 modules)
PE 0x00400000-00438000 DIA flake
ELF 0x7bf00000-7bf03000 Deferred <wine-loader>
ELF 0x7f93f000-7f9a0000 Deferred msvcrt<elf>
\-PE 0x7f950000-7f9a0000 \ msvcrt
ELF 0x7fbee000-7fcf0000 Export kernel32<elf>
\-PE 0x7fc10000-7fcf0000 \ kernel32
ELF 0x7fe07000-7fe11000 Deferred libnss_files.so.2
ELF 0x7fe11000-7fe26000 Deferred libnsl.so.1
ELF 0x7fe3a000-7fe5c000 Deferred libm.so.6
ELF 0x7fe63000-7fe6c000 Deferred libnss_nis.so.2
ELF 0x7fe70000-7ff66000 Deferred libwine_unicode.so.1
ELF 0x7ff66000-7ffe0000 Deferred ntdll<elf>
\-PE 0x7ff80000-7ffe0000 \ ntdll
ELF 0xb7d71000-b7d7a000 Deferred libnss_compat.so.2
ELF 0xb7d7b000-b7d7e000 Deferred libdl.so.2
ELF 0xb7d7e000-b7ead000 Deferred libc.so.6
ELF 0xb7ead000-b7ebf000 Deferred libpthread.so.0
ELF 0xb7ed4000-b7eee000 Export libwine.so.1
ELF 0xb7ef1000-b7f07000 Deferred ld-linux.so.2
Threads:
process tid prio (all id:s are in hex)
00000008 (D) Z:\home\lionel\flake\flake.exe
00000009 0 <==
Hope this helps....
About fork. It was just a joking. Keep up the good work.
Through wine, here is the debug :
lionel@kubuntu:~/flake$ wine ./flake.exe -2 /home/lionel/wav/input.wav /home/lionel/wav/output.flac
Flake: FLAC audio encoder
Signed 16-bit 44100 Hz stereo
samples: 190288560
wine: Unhandled page fault on write access to 0x7f9a0dd4 at address 0x405703 (thread 0009), starting debugger...
WineDbg starting on pid 0x8
Unhandled exception: page fault on write access to 0x7f9a0dd4 in 32-bit code (0x00405703).
It's kinda hard to tell without more symbol names, but my thoughts still go to memory alignment in the structs. I tried rearranging the structs so that non-DWORD values are always at the end. Maybe that will help? If that's even the issue. It seems to be happening in the small section in flake.c around flake_encode_init. This is when the flac encoder structures are first accessed. Some of those values are pointers, so if the alignment is off, the pointers could end up pointing to unaccessible memory, which could cause a page fault.
-Justin
Hi,
I have made a small webpage to post updated versions and various test results from my FLAC encoder. The page will be updated frequently. I have included a Linux i386 binary along with the source code, and I hope I can get some help providing a Windows binary. Hopefully, the more people I can get to test it out, the more I can improve it.
http://home.earthlink.net/~jruggle/flake/ (http://home.earthlink.net/~jruggle/flake/)
Please note that this pre-release is not intended for general use, only testing. It does not currently calculate MD5 sums, do any tagging, or add any padding. The wav parser is not very robust either.
Thanks,
-Justin
Hi Justin,
Great stuff...Now, flake 0.4 works fine too on win32 os. I uploaded a windows build here (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=35&t=45016&st=0#entry396129).
flake dont seem to work correctly on Mac OS X (PowerPC).
The source compiles fine, and the flake executable does it work to my source .wav file.
But it dont compress the file at all (actually it is larger than the original), and the output is just
noice.
flake 0.4:
Flake: FLAC audio encoder
© 2006 Justin Ruggles
Signed 16-bit 44100 Hz stereo
samples: 28563276
blocksize: 4608
max order: 8
order method: estimate
stereo method: estimate
progress: 100% | ratio: 1.025 | bitrate: 1447.0 kbps | bytes: 117148084
flac 1.1.2:
flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.
options: -P 4096 -b 4608 -m -l 8 -q 0 -r 3,3
wrote 78408059 bytes, ratio=0.686
I have tried multiple input files, from different sources, but the same happends every time.
Is this a known problem?
flake dont seem to work correctly on Mac OS X (PowerPC).
The source compiles fine, and the flake executable does it work to my source .wav file.
But it dont compress the file at all (actually it is larger than the original), and the output is just noice.
...
I have tried multiple input files, from different sources, but the same happends every time.
Is this a known problem?
Dang. I thought I had fixed the endian issue. I'll keep trying though. It's hard to test when I don't have a big-endian machine.
-Justin
Hope you nail this, cause this looks like an interresting application.
If you need a test dummy, just give me a hint.
Yesterday I was pointed to a Hydrogenaudio thread on how this is being added to the reference encoder as well. Very exciting stuff!
btw in the FLAC code in CVS there is a src/libFLAC/window.c that may save you some time.
Blocksize I understand, but what's wrong with a higher LPC order? True, it isn't ideal for every situation, but it does help in many cases. I haven't noticed much of an impact on decoding, and I have a quite slow computer.
the number of multiply-adds in the FIR filter equals the order. flac -8 uses max 12, so order 32 requires 20 more multiplies+adds per sample than what is widely used.
Josh
Yesterday I was pointed to a Hydrogenaudio thread on how this is being added to the reference encoder as well. Very exciting stuff!
btw in the FLAC code in CVS there is a src/libFLAC/window.c that may save you some time.
Thanks. I did see that. I chose to go with the Welch window. It is simple to calculate and gives above-average results most of the time.
Blocksize I understand, but what's wrong with a higher LPC order? True, it isn't ideal for every situation, but it does help in many cases. I haven't noticed much of an impact on decoding, and I have a quite slow computer.
the number of multiply-adds in the FIR filter equals the order. flac -8 uses max 12, so order 32 requires 20 more multiplies+adds per sample than what is widely used.
Once I started benchmarking the decoding, I saw what you were talking about. Using max order 32 seems to increase decoding time about 30%. I didn't notice until I actually timed it because I was using short clips. So, I moved the higher block sizes and orders to extended levels 9-15 and modified 0-8 to use similar output parameters to the reference encoder. The higher block sizes don't seem to do much now that I've changed the encoder quite a bit. I'm strongly considering removing the preset levels for 16384 and maybe even 8192.
-Justin
40.964.828 bytes 16-bit Stereo 44100Hz WAV 3:52 "La cosecha de mujeres"
FLAKE 0.4
Level | Ratio | Bitrate | Time
0 | 0.788 | 1112.6 kbps | 6.484
1 | 0.747 | 1054.7 kbps | 7.328
2 | 0.741 | 1045.1 kbps | 7.250
3 | 0.733 | 1034.5 kbps | 7.468
4 | 0.729 | 1029.2 kbps | 7.500
5 | 0.726 | 1024.1 kbps | 7.906
6 | 0.721 | 1017.5 kbps | 11.250
7 | 0.721 | 1017.5 kbps | 12.718
8 | 0.719 | 1015.0 kbps | 14.343
9 | 0.719 | 1014.8 kbps | 29.046
10 | 0.719 | 1015.2 kbps | 20.671
11 | 0.719 | 1014.3 kbps | 120.562
12 | 0.720 | 1016.7 kbps | 21.093
13 | 0.720 | 1016.1 kbps | 119.875
14 | 0.722 | 1019.5 kbps | 21.703
15 | 0.722 | 1019.1 kbps | 120.828
FLAC 1.1.2 CVS
Level | Ratio | Time
0 | 0,782 | 2.687
1 | 0,778 | 2.890
2 | 0,776 | 3.609
3 | 0,731 | 3.750
4 | 0,723 | 4.812
5 | 0,722 | 6.031
6 | 0,722 | 6.625
7 | 0,721 | 21.531
8 | 0,720 | 28.140
[/font]
In some samples a large block size can reduce the compression ratio.
WAV 111.266.108 bytes 16-bit Stereo 44100Hz 10:30 "La navidad de los pobres"
Parameters | Ratio | Bitrate | Bytes | Time
FLAKE -8 -b 1152 | 0.645 | 909.7 kbps | 71722600 | 39.640
FLAKE -8 -b 2048 | 0.642 | 906.0 kbps | 71436942 | 40.031
FLAKE -8 -b 2304 | 0.642 | 906.2 kbps | 71446745 | 39.656
FLAKE -8 -b 4096 | 0.644 | 909.3 kbps | 71697454 | 39.609
FLAKE -8 -b 4608 | 0.645 | 910.6 kbps | 71796426 | 38.843
FLAKE -8 -b 8192 | 0.652 | 920.1 kbps | 72549094 | 37.812
FLAC CVS -8 | 0.647 | ~913 kbps | 71971927 | 78.421
[/font]
I suggest testing several block sizes in the high levels, using a lower compression level. Then use the best value to compress the sample:
Example:
-b x,[n]
-b x Block size [16 - 65535] (default: 4608)
-b 1 Try -b 2048 and -b 4608
-b 2 Try -b 2048, -b 2304, -b 4096, -b 4608 and -b 8192.
-b 3 Try -b 192, -b 576, -b 1152, -b 2304, -b 4608, -b 256, -b 512, -b 1024, -b 2048, -b 4096, -b 8192 and -b 16384.
n Use compression level n for the tries. Default: 5 for -b 1, -b 2 or -b 3.
Compression Levels:
[1-9] Without changes
10 = -l 12 -o 4 -s 2 -b 1,5 Estimated Time: ~ 45 (2* level 5 + level 9)
11 = -l 12 -o 4 -s 2 -b 2,5 Estimated Time: ~ 69 (5*5+9)
12 = -l 32 -o 4 -s 2 -b 2,5 Estimated Time: ~ 160 (5*5+11)
13 = -l 32 -o 4 -s 2 -b 3,5 Estimated Time: ~ 215 (12*5+11)
14 = -l 32 -o 4 -s 2 -b 3,8 Estimated Time: ~ 293 (12*8+11)
All Times are Process Time (Timer 3.01). Celeron D325 2.53 Ghz under Windows XP SP2.
Estimates times are based on the first sample ("La cosecha de mujeres").[/font]
I suggest testing several block sizes in the high levels, using a lower compression level. Then use the best value to compress the sample:
Example:
-b x Block size [16 - 65535] (default: 4608)
-b 1 Try -b 2048 and -b 4608
-b 2 Try -b 2048, -b 2304, -b 4096, -b 4608 and -b 8192.
-b 3 Try -b 192, -b 576, -b 1152, -b 2304, -b 4608, -b 256, -b 512, -b 1024, -b 2048, -b 4096, -b 8192 and -b 16384.
Along those same lines, I am working on implementing block switching. The resulting files would not be Subset compliant, but they would still decode with at least the reference decoder and with FFmpeg/MPlayer, which are the only decoders I have tested with at this point. I have found that 8192 only rarely increases compression and 16384 almost never does. I have now removed them as presets (pending pre-release 5) in favor of an 8-level order search.
The advantage of block switching would be with audio which compresses better at 2048 or 1024 blocksize. What you're suggesting seems like it could be accomplished with a 2-pass encoding, which I guess I could try to implement as another option.
Thanks,
Justin
Hi,
Pre-release 5 is now out.
* I hope the PPC endian issue is fixed. If not, please let me know.
* Added support for streamed input and output. For example,
cat test.wav | flake - - | flac -d -o test2.wav -
* Added 8-bit input support. 24-bit input is also supported, but only converts to 16-bit for now, so it's not lossless.
* Changed up the extended level options. 8192 and 16384 block sizes have now been removed as preset options. Also, I added an 8-level prediction order search (levels 9 & 11).
-Justin
Pre-release 5 is now out.
* I hope the PPC endian issue is fixed. If not, please let me know.
It seems to work fine now. The track encodes, and the resulting FLAC files play fine.
Thanks!
* now on to testing this application *
Hi all,
Sorry to keep posting updates so often. The online version of my FLAC encoder is now pre-release 6.
Changes are mostly header-related.
* added MD5 support! yay!
* added padding in the header to make room for metadata
* added a placeholder for a future vendor tag
I am at a good stopping point for now. My wishlist is 90% done. I will finally start work on an FFmpeg patch to integrate the current version.
Also, it would be really great to see some other peoples' test results. My computer is pretty darn slow, so I have only been able to test small files when doing everyday sort of benchmarking. I would love to see how it performs, say, on a whole CD.
Thanks,
Justin
Album: Toto - Legend (in a single file)
Playing time: 72m 35s
Original File Size: 768284896 bytes
Computer specs: Apple PowerBook G4, 1.5GHz, 1GB ram, 7200rpm HDD.
Encoders: flac 1.1.2 and flake 06
flac:
level enc time bytes ratio kbps dec time
----- --------- -------- ----- ------ ---------
0 1m 50s 518214304 0.675 1m 15s
5 3m 34s 496252171 0.646 1m 14s
8 13m 46s 494499692 0.644 1m 26s
flake:
level enc time bytes ratio kbps dec time
----- --------- -------- ----- ------ ---------
0 3m 21s 526551866 0.685 967.2 1m 21s
5 6m 53s 495548928 0.645 910.3 1m 30s
8 13m 37s 492004491 0.640 903.8 1m 14s
The compression rate is comparable to flac, but it encodes noticeable slower at -0 and -5.
Anyway, you have done great work so far!
Download the Mac OS X univeral binary here: direct link (http://www.hydrogenaudio.org/forums/index.php?act=Attach&type=post&id=2294)
Album: Toto - Legend (in a single file)
Playing time: 72m 35s
Original File Size: 768284896 bytes
Computer specs: Apple PowerBook G4, 1.5GHz, 1GB ram, 7200rpm HDD.
Encoders: flac 1.1.2 and flake 06
flac:
level enc time bytes ratio kbps dec time
----- --------- -------- ----- ------ ---------
0 1m 50s 518214304 0.675 1m 15s
5 3m 34s 496252171 0.646 1m 14s
8 13m 46s 494499692 0.644 1m 26s
flake:
level enc time bytes ratio kbps dec time
----- --------- -------- ----- ------ ---------
0 3m 21s 526551866 0.685 967.2 1m 21s
5 6m 53s 495548928 0.645 910.3 1m 30s
8 13m 37s 492004491 0.640 903.8 1m 14s
The compression rate is comparable to flac, but it encodes noticeable slower at -0 and -5.
Anyway, you have done great work so far!
Thank you for the feedback.
It looks like I have a long way to go to make level 5 comparable to the reference encoder on PPC. At least level 8 is on par.
I finally let my system run for a while to test a whole album. Here is what I got at level 8.
Smashing Pumpkins - Siamese Dream
playing time: 1h2m15.8s
flac: encode time=1h1m35.42s ratio=0.583
flake: encode time=35m43.75s ratio=0.582
I know...it really is about time for a major upgrade...I'm still holding out for 10 good years out of my '97 IBM.
Thanks,
Justin
Thank you for the feedback.
It looks like I have a long way to go to make level 5 comparable to the reference encoder on PPC. At least level 8 is on par.
You are welcome!
Looks like FLAC have optimized their code for PowerPC, while you of course have consentrated on i386 only.
I guess it would be hard for you to improve the performance on PowerPC as well, since you dont have access to one. But that might not be that important since Apple are moving their line to Intel, and at the current stage they only have two PowerPC computers left.
I finally let my system run for a while to test a whole album. Here is what I got at level 8.
Smashing Pumpkins - Siamese Dream
playing time: 1h2m15.8s
flac: encode time=1h1m35.42s ratio=0.583
flake: encode time=35m43.75s ratio=0.582
I know...it really is about time for a major upgrade...I'm still holding out for 10 good years out of my '97 IBM.
Wow, you have cut the -8 encoding time on your i386 machine to half.
That you still use a 300MHz computer from '97 are just insane. My computer are only two (2) years old, and I am seriously looking for a replacement (probably next generation MacBook Pro)!
Here are my results for four albums lying around. CPU AMD K8 1.8GHz, 64-bit OS, everything cached in memory, so the harddisk wouldn't be a bottleneck. Included is also decoding speed of another decoder that is very sensitive to higher LPC order. Versions used: flake-06 and flac CVS.
garbage / beautiful garbage
lvl| ratio speed (x realtime) | ratio speed (x realtime)
| flake flac -d flacdec | flac flac -d flacdec
0 | 0.713 52.1 202.9 418.1 | 0.727 135.5 210.0 424.3
1 | 0.678 50.7 184.7 391.6 | 0.703 137.6 205.9 408.5
2 | 0.678 50.6 184.8 391.2 | 0.701 90.4 206.4 416.7
3 | 0.670 50.0 179.7 382.4 | 0.694 90.9 179.5 385.5
4 | 0.668 51.1 176.0 373.5 | 0.668 77.6 170.7 351.5
5 | 0.666 46.5 170.4 352.9 | 0.667 58.5 172.3 354.6
6 | 0.664 31.2 169.0 344.0 | 0.666 43.9 172.0 355.2
7 | 0.664 27.9 170.8 348.9 | 0.665 9.7 172.8 359.0
8 | 0.661 24.1 163.6 309.1 | 0.663 7.2 168.4 329.1
9 | 0.661 18.4 164.2 311.9 |
10 | 0.661 14.6 165.3 314.6 |
11 | 0.657 11.8 110.0 118.7 |
12 | 0.657 4.5 114.4 122.4 |
u2 / the joshua tree
lvl| ratio speed (x realtime) | ratio speed (x realtime)
| flake flac -d flacdec | flac flac -d flacdec
0 | 0.589 57.0 217.9 450.5 | 0.585 141.0 223.1 448.4
1 | 0.553 55.8 194.2 422.7 | 0.573 142.7 218.2 430.5
2 | 0.553 55.3 195.3 419.7 | 0.570 92.4 219.2 438.7
3 | 0.545 55.0 189.2 411.7 | 0.554 93.3 192.1 411.7
4 | 0.542 56.4 185.7 404.0 | 0.540 79.1 181.1 374.4
5 | 0.540 50.9 179.3 376.7 | 0.539 59.3 180.7 377.6
6 | 0.535 33.1 178.2 365.3 | 0.538 44.5 181.9 378.1
7 | 0.535 29.4 180.2 372.0 | 0.536 9.7 182.6 384.9
8 | 0.533 25.3 173.5 332.3 | 0.535 7.3 180.2 360.5
9 | 0.533 19.1 175.2 336.3 |
10 | 0.533 15.0 175.2 337.8 |
11 | 0.530 12.2 127.7 144.5 |
12 | 0.530 4.5 131.3 147.8 |
smashing pumpkins / gish
lvl| ratio speed (x realtime) | ratio speed (x realtime)
| flake flac -d flacdec | flac flac -d flacdec
0 | 0.654 54.6 211.1 434.1 | 0.635 138.9 216.9 436.2
1 | 0.599 54.1 193.0 411.7 | 0.634 140.2 213.4 425.5
2 | 0.599 54.0 192.5 411.1 | 0.633 91.4 213.4 428.1
3 | 0.582 53.6 188.4 400.4 | 0.581 93.1 190.2 404.5
4 | 0.577 54.9 183.2 392.5 | 0.575 79.0 179.6 370.6
5 | 0.571 49.8 178.0 367.7 | 0.574 59.3 181.3 372.1
6 | 0.566 32.7 178.8 365.3 | 0.573 44.6 180.7 372.1
7 | 0.565 29.0 181.1 369.1 | 0.570 9.9 181.9 378.2
8 | 0.561 25.2 175.4 332.2 | 0.568 7.4 179.6 359.2
9 | 0.560 19.0 175.4 333.8 |
10 | 0.560 15.0 176.2 336.2 |
11 | 0.559 12.4 149.8 183.7 |
12 | 0.559 4.6 155.9 190.2 |
walk the line OST
lvl| ratio speed (x realtime) | ratio speed (x realtime)
| flake flac -d flacdec | flac flac -d flacdec
0 | 0.628 54.7 217.1 445.7 | 0.654 137.9 220.3 441.0
1 | 0.608 52.9 193.6 410.6 | 0.621 141.0 216.5 425.3
2 | 0.607 51.9 194.8 409.3 | 0.618 91.8 219.3 440.2
3 | 0.602 52.2 190.5 402.7 | 0.637 91.9 191.5 406.6
4 | 0.602 53.7 186.4 397.6 | 0.603 78.5 183.9 380.1
5 | 0.601 48.7 180.3 377.2 | 0.602 59.0 185.4 387.1
6 | 0.600 31.8 176.9 357.3 | 0.601 44.2 185.7 386.5
7 | 0.599 28.5 183.4 377.8 | 0.599 9.7 187.9 395.1
8 | 0.599 25.1 180.6 354.8 | 0.599 7.2 186.6 383.0
9 | 0.599 18.9 180.5 361.4 |
10 | 0.599 14.9 180.1 365.1 |
11 | 0.598 12.4 163.3 255.1 |
12 | 0.598 4.5 167.8 263.1 |
Justin, are you planning to implement a variable blocksize support?
Justin, are you planning to implement a variable blocksize support?
Thanks for the decoding benchmarks. To answer your question, yes, eventually. My current order of business is integrating the current version with FFmpeg. There has already been a very basic version committed to current FFmpeg SVN which is along the lines of using "flac -0" but with even less compression. It was more of a test-run to get things properly integrated & generating compliant FLAC files. I will reply here again when FFmpeg has competitive encoding, not just compliant encoding.
The variable blocksize support will mostly likely be after my newest, forthcoming, FFmpeg patch has been applied & optimized by more experienced FFmpeg developers.
-Justin
Pearl Jam - Ten (new european version) / Track 11: Release / Length: 543.426 seconds
System: Athlon XP 3200 - 1GB DDR400 - NForce2
Flac version: libFLAC CVS 20060425
Flake version: flake-06
encode | ratio | bitrate | speed | dec. speed
---------|-------|---------|--------|-------------
flac 8 | 0.534 | 754.? | 11.93x | 192.30x
flake 8 | 0.533 | 752.1 | 23.31x | 187.11x
flake 12 | 0.531 | 749.8 | 4.27x | 162.23x
Commandlines for encoding:
timethis "start /realtime /b /wait flac -8 test.wav -o NUL: -f"
timethis "start /realtime /b /wait flake -8 test.wav -o NUL:"
timethis "start /realtime /b /wait flake -12 test.wav -o NUL:"
Encodes were done three times, fastest run counts. Spread between runs was small, no more than 0.1s. Speeds are rounded.
Decoding speed was measured with Foobar2000 0.9.1, 10 passes, no dsp, high priority, buffer in memory.
Flake -8 file decodes about 44% faster with flacdec.
A 95.5% speedup. Impressive. Flake -8 is my new god.
I still miss some options, most important to me is -V as i like to overclock my hardware. Veryfying doesn't slow encoding down much anyway. Also reading from stdin doesn't seem to work on windows, it will say "invalid wav file: -" and then proceeds create a 0 byte flac output file, another feature it could do without
For the rest it's nothing metaflac can't handle.
Pearl Jam - Ten (new european version) / Track 11: Release / Length: 543.426 seconds
System: Athlon XP 3200 - 1GB DDR400 - NForce2
Flac version: libFLAC CVS 20060425
Flake version: flake-06
encode | ratio | bitrate | speed | dec. speed
---------|-------|---------|--------|-------------
flac 8 | 0.534 | 754.? | 11.93x | 192.30x
flake 8 | 0.533 | 752.1 | 23.31x | 187.11x
flake 12 | 0.531 | 749.8 | 4.27x | 162.23x
Wonderful. This is typical of the results I get as well. I'm glad to see the relative encoding times are comparable on a fast, modern system.
A 95.5% speedup. Impressive. Flake -8 is my new god.
I'm glad you like it. Keep in mind that it is experimental at this point, but there will eventually be another release which will use FFmpeg's wav decoder & FLAC encoder. So far it seems it might end up being faster as well.
I still miss some options, most important to me is -V as i like to overclock my hardware. Veryfying doesn't slow encoding down much anyway.
Oddly enough, I was just thinking about this today. I noticed many people on Hydrogenaudio use this option. I will definitely add something similar when I start back working on Flake. I might have to use a modified version of FFmpeg's FLAC decoder instead of doing it directly through libavcodec. But you're right that this is an important feature of the reference encoder.
Also reading from stdin doesn't seem to work on windows, it will say "invalid wav file: -" and then proceeds create a 0 byte flac output file, another feature it could do without
Hmm. Maybe that has to do with having to set stdin to binary mode in windows...maybe. This is good to know though. Thanks for the feedback.
For the rest it's nothing metaflac can't handle.
I might even add metadata/cuesheet/seektable support someday. Personally, I'm not a fan of the vorbis/flac comment specification, but it works.
All of this is probably at least a month or two (or three) away. I have a lot going on in the next couple months.
Thanks,
-Justin
hello.
does this new encoder supports unicode?
i can't connect to the page...404
Justin,
I'd also like to join in saying "thank-you" for your efforts to improve FLAC encoding. Highly appreciated!
I also integrated it into fb2k; the only drawback (= speed penalty) here is that reading from stdin doesn't work here either so temp. WAVs have to be uses with the converter...
Keep it up!
-sundance-
Btw, why didn't you use another vendor string than "[put libavcodec/libavformat versions here]"?
Hello,
Btw, why didn't you use another vendor string than "[put libavcodec/libavformat versions here]"?
Because one of my main purposes for writing the encoder was to integrate it with FFmpeg. I just needed a frontend to test it with before integrating it, so I made Flake. The next release will have the lavc version as the vendor string.
Thanks,
-Justin
I just needed a frontend to test it with before integrating it, so I made Flake. The next release will have the lavc version as the vendor string.
Thanks,
-Justin
I can't speak for the rest of us, but i really like the standalone, dropin flac replacement that flake is. I'm not very familiar with the ffmpeg project, but i fear it won't be as easy to use from the commandline or as encoder for eac or foobar when integrated into ffmpeg.
...but i really like the standalone, dropin flac replacement that flake is.
Second that!
I just needed a frontend to test it with before integrating it, so I made Flake. The next release will have the lavc version as the vendor string.
Thanks,
-Justin
I can't speak for the rest of us, but i really like the standalone, dropin flac replacement that flake is. I'm not very familiar with the ffmpeg project, but i fear it won't be as easy to use from the commandline or as encoder for eac or foobar when integrated into ffmpeg.
Not to worry. I will still re-release Flake as a standalone encoder once the code is part of FFmpeg. I'll just use the relevant code from FFmpeg and might have to restructure it a little bit for use with Flake. FFmpeg has/is a commandline frontend to its libavcodec and libavformat libraries. When using FFmpeg, you'll most likely only be able to specify the compression level, not each individual option. Flake will use the FFmpeg code, but be specifically only for FLAC encoding and will provide more detailed options along the lines of what it has currently.
As far as familiarity, you can find out more about FFmpeg at
http://ffmpeg.mplayerhq.hu/ (http://ffmpeg.mplayerhq.hu/)
It is a wonderful project with very good developers. If you don't mind the 50 or so emails a day, subscribe to ffmpeg-devel and watch for a while...or just browse through the archives.
-Justin
Well I am propobly too late but I have builded some binarys for win32 using intel compiler. If someone want to try them here is link http://thefrontend.sourceforge.net/flake/ (http://thefrontend.sourceforge.net/flake/). All versions compiled be me where faster then kurtnoise builds on my machine (Athlon XP 2000+). Archives include full sources, binarys and project files. In file TESTS-0.4-win32.zip are some speed tests results just to prove me claims ;-)
Well I am propobly too late but I have builded some binarys for win32 using intel compiler. If someone want to try them here is link http://thefrontend.sourceforge.net/flake/ (http://thefrontend.sourceforge.net/flake/). All versions compiled be me where faster then kurtnoise builds on my machine (Athlon XP 2000+). Archives include full sources, binarys and project files. In file TESTS-0.4-win32.zip are some speed tests results just to prove me claims ;-)
Thank you! Especially for the win32 build environment stuff! I am currently working on a simple configure/build system for unix environments. I'll try to use some of what you have done as a basis to improve Windows compatibility in the next release.
The current development version of Flake has about a 15% speed improvement over pre-release 06 (on x86 linux gcc 3.4.4). Compression has generally improved only slightly so far, except at compression level 0, which has a significant improvement. FFmpeg integration of the full encoder is only about 1/2 done right now, but is going very well I think. Right now it has levels 0 to 5, and you can change specific options on the commandline. If you build FFmpeg from current SVN (http://ffmpeg.mplayerhq.hu/download.html), here are some of the FLAC encoding options.
Encode at level 5:
ffmpeg -i test.wav test.flac
Encode at level 0:
ffmpeg -i test.wav -compression_level 0 test.flac
Encode using individual options:
ffmpeg -i test.wav -frame_size 2048 -use_lpc 1 -min_prediction_order 6 -max_prediction_order 12 -min_partition_order 0 -max_partition_order 6 -lpc_coeff_precision 15 test.flac
Quickly convert ogg-flac to raw flac:
ffmpeg -i test.ogg -acodec copy test.flac
Also, the most updated Flake webpage is at another location. I will be able to post some test samples of my own instead of linking to other sites since I now have more storage space. I am limited to only 10MB on my Earthlink account.
http://jbr.homelinux.org/flake/ (http://jbr.homelinux.org/flake/)
-Justin
Thank you! Especially for the win32 build environment stuff! I am currently working on a simple configure/build system for unix environments.
Quick note about the win32 project files (VC++ 7.1 and ICL 8.0 compatible): the FLAKE code compiles only with ICL not CL because of C99 option in intel compiler settings (without this option code does not compile). So the build environment works only with visual studio and intel compiler integrated. But it works very well without any nasty errors and produce great binarys ;-). I will in near future update the build environment to Intel 9.1 (actually I have done this locally and only need to update online stuff, but before that some tests will be committed to check if new version works good as prevoius).
Well I am propobly too late but I have builded some binarys for win32 using intel compiler. If someone want to try them here is link http://thefrontend.sourceforge.net/flake/ (http://thefrontend.sourceforge.net/flake/). All versions compiled be me where faster then kurtnoise builds on my machine (Athlon XP 2000+). Archives include full sources, binarys and project files. In file TESTS-0.4-win32.zip are some speed tests results just to prove me claims ;-)
Nice, for version 0.6 at -8 it's 26% faster here, with 29.40x vs 23.31x.
That makes it 146% faster than flac!
Oh, one more thing that just comes to mind, i miss a -q option for flake, but that's a minor issue. It does boost performance a tiny bit.
Oh, one more thing that just comes to mind, i miss a -q option for flake, but that's a minor issue. It does boost performance a tiny bit.
That just never crossed my mind. I'll add it to the next release.
Thanks,
Justin
another thing I would suggest is a big fat warning if the compression setting used is not in the FLAC subset (http://flac.sourceforge.net/format.html#subset). not sure, looks like currently all settings are in the subset. but I'm considering restricting the subset further (to lpc order<=12 and blocksize <=4608) for the next release, at least for <=48kHz.
Josh
another thing I would suggest is a big fat warning if the compression setting used is not in the FLAC subset (http://flac.sourceforge.net/format.html#subset). not sure, looks like currently all settings are in the subset. but I'm considering restricting the subset further (to lpc order<=12 and blocksize <=4608) for the next release, at least for <=48kHz.
Thanks Josh. The 0-8 presets in Flake are within the subset, but the block size can be specified on the commandline to be any valid size within the spec, not just limited to the predefined ones required for subset.
I agree with your intentions to further restrict the subset requirements. Block size over 4608 is rarely beneficial anyway unless it is a quite stationary signal, but the affect on decoding speed is substantial. Maybe this would also make it easier to standardize hardware support. There are products which say that they support FLAC levels 0 to 8. This doesn't really say much about what else is truly supported outside the predefined levels. If subset limits were restricted to the settings of level 8 it would be simpler and more definitive to just say that a product supports the FLAC Subset format or the full FLAC format (or FLAC 2 ).
-Justin
Hello,
I'm posting again here because the FFmpeg FLAC encoder has some significant changes. Nearly all of Flake-06 has been integrated, along with some pretty great improvements by Michael Niedermayer (lead FFmpeg developer). It is quite a bit faster than Flake (level 8 is about 60% faster than the reference encoder). Some of the key improvements are:
- encoding speed at lower levels is now on-par with the reference encoder
- better stereo decorrelation
- new, faster log search for finding optimal LPC order
- optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion
I have done a few benchmark comparisons (http://jbr.homelinux.org/flake/ffmpeg.html).
The FFmpeg homepage (http://ffmpeg.mplayerhq.hu/) has instructions for downloading/compiling the current SVN version.
One major thing left to do is to work out how to encode the MD5 sum in the global header. Since FFmpeg supports many container formats, it is proving to be a little tricky.
-Justin
Great work!
Hope you plan to "back-port" these improvements to the standalone flake encoder as well?
- optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion
What's the advantage of the Cholesky method? I'm not familiar with it.
- optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion
What's the advantage of the Cholesky method? I'm not familiar with it.
Michael Niedermayer explains it better than I ever could here (http://article.gmane.org/gmane.comp.video.ffmpeg.cvs/3599). It took me a while to understand it even partially.
You probably have a better math background than me, so these might help.
Choleski decomposition (http://en.wikipedia.org/wiki/Choleski_decomposition)
Gaussian elimination (http://en.wikipedia.org/wiki/Gaussian_elimination)
Toeplitz matrix (http://en.wikipedia.org/wiki/Toeplitz_matrix)
Autocorrelation (http://en.wikipedia.org/wiki/Autocorrelation)
Levinson recursion (http://en.wikipedia.org/wiki/Levinson_recursion)
Great work!
Hope you plan to "back-port" these improvements to the standalone flake encoder as well?
Yes I do. Some have been already, but I don't want to release anything new until the FFmpeg encoder gets to a good stable state. (i.e. until the MD5/extradata/other containers issue has been solved).
-Justin
Okay, I understand the what and why
What's the speed impact?
Okay, I understand the what and why
What's the speed impact?
Very big. 1-pass is only a little slower, but it does not give as good results as Levinson-Durbin. The two seem to be equivilant in compression at around 3-4 pass. So the real benefit is if you have a fast computer and/or lots of time to spare to wait for it to squeeze out a few more tenths of a percent. Decoding speed is not affected.
example:
Levinson-Durbin level 12 - 80sec
Cholesky 10-pass level 12 - 1500sec
compression difference 0.3%
A good thing to add in the future that might give similar compression results would be to adaptively select or brute-force search for an optimal window function to go along with the Levinson-Durbin recursion. I think Josh might be adding something like that to the reference encoder, but I'm not sure. I know it currently is user-selectable, but it would be great if the best window was chosen by the encoder for each frame. FFmpeg currently uses only a Welch window function. I just chose that because it is a simple calculation and seems to work well most of the time.
-Justin
- optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion
Heh! This is actually what I was suggesting during the "best window search for FLAC" period.
Too bad it's
that slow.
Sebastian
I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter. so far it works but it will not be ready for the next release.
Josh
(edit: lower bound on the least-squares solution I mean; I have some other experiments solving the system another way but that's too new.)
I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter. so far it works but it will not be ready for the next release.
I suppose the pseudo-inverse (http://en.wikipedia.org/wiki/Pseudo_inverse#Finding_the_pseudoinverse_of_a_matrix) approach (through SVD) is even slower (but more accurate) than the Cholesky approach.
I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter. so far it works but it will not be ready for the next release.
Can you please tell which compression improvement can be achieved?
I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter. so far it works but it will not be ready for the next release.
I suppose the pseudo-inverse (http://en.wikipedia.org/wiki/Pseudo_inverse#Finding_the_pseudoinverse_of_a_matrix) approach (through SVD) is even slower (but more accurate) than the Cholesky approach.
yep, you're right. I'm mainly interested in establishing the lower bound first to see if other solvers would even be practical.
as for compression improvement, I don't have good numbers yet but it's not dramatic. the autocorrelation formulation is surpisingly good with the proper window.
Josh
Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.
I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.
Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.
I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.
Sorry about that. The Flake development is in a lull right now since I'm working feverishly on the Aften AC-3 encoder. I'll run valgrind on Flake and see if I can fix that leak.
-Justin
Thanks in advance!
Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.
I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.
Sorry about that. The Flake development is in a lull right now since I'm working feverishly on the Aften AC-3 encoder. I'll run valgrind on Flake and see if I can fix that leak.
-Justin
Here is a new bug-fix release to correct the memory leaks. This download also includes some pre-compiled binaries for Linux and Windows.
http://jbr.homelinux.org/flake/flake-07.tar.gz (http://jbr.homelinux.org/flake/flake-07.tar.gz)
-Justin
That fixed it, thanks.
But the speed's now dropped from 29.40x to about 18.5, still much faster than flac (11.9x) but not nearly as impressive.
Only tested the windows binary, the output is correct.
I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).
Download:
http://www.thefrontend.net/flake/flake-0.7-win32.rar (http://www.thefrontend.net/flake/flake-0.7-win32.rar)
Tests:
http://www.thefrontend.net/flake/TESTS-0.7-win32.zip (http://www.thefrontend.net/flake/TESTS-0.7-win32.zip)
Old builds:
http://www.thefrontend.net/flake/ (http://www.thefrontend.net/flake/)
wiso
That fixed it, thanks.
But the speed's now dropped from 29.40x to about 18.5, still much faster than flac (11.9x) but not nearly as impressive.
Only tested the windows binary, the output is correct.
Okay, well that has convinced me to take a leap and release a more up-to-date version of Flake. I'm not sure if the windows binaries will still be slower than before since I was not the one who created the binary for version 06. All I can say is that it's faster for me on Linux, and I hope it is on Windows as well.
http://jbr.homelinux.org/flake/ (http://jbr.homelinux.org/flake/)
It still doesn't have all the features in the FFmpeg encoder, but I'll get to that eventually.
Enjoy.
-Justin
I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).
Sorry to do two releases back-to-back. I've had the 0.08 version sitting around and had wanted to tweak it some more before release, but I realized that might be a while, so I went ahead and released it. I'll probably create a Sourceforge page so that I can do SVN updates and not have to release so often.
Also, my versioning has changed. I was using "pre-release 01,02,...07". Now I've switched to more normal versioning. The latest is version 0.08.
Thanks!
-Justin
I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).
Sorry to do two releases back-to-back. I've had the 0.08 version sitting around and had wanted to tweak it some more before release, but I realized that might be a while, so I went ahead and released it. I'll probably create a Sourceforge page so that I can do SVN updates and not have to release so often.
Also, my versioning has changed. I was using "pre-release 01,02,...07". Now I've switched to more normal versioning. The latest is version 0.08.
Thanks!
-Justin
This is no problem for me. The progress is there and this is more important.
wiso
Oh yeah, wisodev's .07 build clocks in at about 29.1x for me with no memory leaks. Slightly slower than the leaky .06'es, but a major improvement nevertheless. Compression is the same. I should mention that i made some slight changes to my system and a reinstall, so there may be inaccuracies with my old benchmark results, but these should be fairly small, probably no more than .1x.
Pipe support still seems disfunctional, i get the following with both wisodev's .07 and justin's original fixed .06:
D:\MP3\Pearl Jam\Pearl Jam - Ten>type test.wav |flake -8 - test.flac
Flake: FLAC audio encoder
(c) 2006 Justin Ruggles
Signed 16-bit 44100 Hz stereo
samples: 23965116
blocksize: 4608
max order: 12
order method: 4-level
stereo method: search
| bytes: 4448
The process tried to write to a nonexistent pipe.
0.08's windows binaries do not work for me. It reads the files but never starts encoding, just throws me back to the commandline, like this:
D:\MP3\Pearl Jam\Pearl Jam - Ten>flake-win32-i586.exe test.wav test.flac
Flake: FLAC audio encoder
(c) 2006 Justin Ruggles
Signed 16-bit 44100 Hz stereo
samples: 23965116
D:\MP3\Pearl Jam\Pearl Jam - Ten>
All three builds do it. It does parse options and respond to -h, it creates a 0 byte output file.
Also, is there any chance of a changelog?
Like wisodev says, there is improvement and that is most important. Flake has a lot of potential, keep up the good work!
Oh yeah, wisodev's .07 build clocks in at about 29.1x for me with no memory leaks. Slightly slower than the leaky .06'es, but a major improvement nevertheless. Compression is the same. I should mention that i made some slight changes to my system and a reinstall, so there may be inaccuracies with my old benchmark results, but these should be fairly small, probably no more than .1x.
Pipe support still seems disfunctional, i get the following with both wisodev's .07 and justin's original fixed .06:
Hmm...I'll try to fix that after I get the Sourceforge page up and running. I'm hoping for approval tomorrow.
0.08's windows binaries do not work for me. It reads the files but never starts encoding, just throws me back to the commandline, like this:
...
All three builds do it. It does parse options and respond to -h, it creates a 0 byte output file.
Also, is there any chance of a changelog?
Now that is strange. If I have a chance I'll install Wine so I can do some debugging. I'm guessing it's related to configure & Makefile.
Ok, I'll create a changelog. Having the source in an svn repository will help document development as well since there will be commit messages.
He, I have found too that 0.08 windows binarys are not working (all of them) even my builds are not working.
When it starts it throws stack overflow error on my machine (Windows XP)
But I have found the source of this problem.
in file encode.c in function
flake_encode_init(FlakeContext *s)
there was changed since version 07 way of handling context (FlacEncodeContext ctx_local;) memory allocation
and this change to local variable is causing under windows stack overflow
FlacEncodeContext ctx_local;
FlacEncodeContext *ctx;
int i, header_len;
if(s == NULL) {
return -1;
}
ctx = &ctx_local;
I have changed to method used in 07 (allocating memory at the beginning of the function) and it works OK and do not throws any errors.
wiso
Justin,
would you consider adding wisodev's "patch" to flake.c that he did in v06?
Or is there any catch I can't see?
if(!strncmp(opts.infile, "-", 2)) {
#ifdef _WIN32 // win32 pipes patch by wiso
_setmode(_fileno(stdin), _O_BINARY);
#endif
ifp = stdin;
} else {
ifp = fopen(opts.infile, "rb");
if(!ifp) {
fprintf(stderr, "error opening input file: %s\n", opts.infile);
return 1;
}
}
if(!strncmp(opts.outfile, "-", 2)) {
#ifdef _WIN32 // win32 pipes patch by wiso
_setmode(_fileno(stdout), _O_BINARY);
#endif
ofp = stdout;
} else {
ofp = fopen(opts.outfile, "wb");
if(!ofp) {
fprintf(stderr, "error opening output file: %s\n", opts.outfile);
return 1;
}
}
It would help a lot to have piping work in WIN32...
Thanks a lot!
.sundance.
would you consider adding wiso's "patch" to flake.c:
Yeah, I'll apply both of wiso's fixes and put them into SVN as soon as Sourceforge approves the project. I'll also do a bit of code cleanup and documentation before doing the initial import and then jump up to version 0.10 as the 1st Sourceforge release.
-Justin
Justin,
that's really good news.
Too bad that there's already a project called FLAKE registered at SF: http://sourceforge.net/projects/flake (http://sourceforge.net/projects/flake)
.sundance.
Justin,
that's really good news.
Too bad that there's already a project called FLAKE registered at SF: http://sourceforge.net/projects/flake (http://sourceforge.net/projects/flake)
.sundance.
I know! I guess I should have looked around first... I had no idea until I tried to register. Mine doesn't use all caps though. The "unix name" will be flake-enc. Still, I don't know if that will be grounds for rejection? I may have to rename it, which would really suck!
-Justin
Whew...it was accepted.
http://sourceforge.net/projects/flake-enc/ (http://sourceforge.net/projects/flake-enc/)
I'll work on getting everything uploaded and active this afternoon and evening.
"flache" (High-Efficiency? painful?) is not taken...
Whew...it was accepted.
http://sourceforge.net/projects/flake-enc/ (http://sourceforge.net/projects/flake-enc/)
I'll work on getting everything uploaded and active this afternoon and evening.
The SVN repository is up.
Version 0.10 has been released.
For now I have built binaries for Linux, optimized for i586, i686, or k8. Also I included a couple (probably still slow) cross-compiled binaries for Windows, optimized for i586 or i686. If wiso decides to be very kind again and builds a faster Windows binary, I would be happy to put it on Sourceforge as well.
Thanks,
Justin
For now I have built binaries for Linux, optimized for i586, i686, or k8. Also I included a couple (probably still slow) cross-compiled binaries for Windows, optimized for i586 or i686. If wiso decides to be very kind again and builds a faster Windows binary, I would be happy to put it on Sourceforge as well.
Thanks,
Justin
I have downloaded the sources from sourceforge and I will upload binarys as soon as I come back home from work.
What do you think about adding some version information to the console output like
Flake: FLAC audio encoder version 0.10
(c) 2006 Justin Ruggles
to be able to identify which version of flake you're testing more easier.
Great you already included that version information as a vendor string to the flac header!
.sundance.
Justin. I pulled revision 2 from SVN, and tried to build this on Mac OS 10.47 PowerPC/ GCC 4.0.1 (Xcode Tools 2.4). But it errors out like this:
$ ./configure
source path /Users/krmathis/Downloads/flake-enc
C compiler cc
make make
CPU powerpc (generic)
big-endian yes
inttypes.h yes
AltiVec enabled yes
debug symbols yes
strip symbols yes
optimize yes
Creating config.mak and config.h...
$ make
make -C libflake all
** Snip **
cc -O3 -no-cpp-precomp -pipe -force_cpusubtype_ALL -Wno-sign-compare -fomit-frame-pointer -faltivec -g -Wdeclaration-after-statement -Wall -Wno-switch -mdynamic-no-pic -DHAVE_CONFIG_H -I. -I.. -I/Users/krmathis/Downloads/flake-enc/libflake -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -c -o wav.o wav.c
cc -L../libflake -Wl,-dynamic,-search_paths_first -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2
I successfully compiled flake 0.7 two days ago. But the build system have changed since then (from "make" to "./configure && make")
Any idea whats wrong?
FLAKE 0.10 Win32 Build and SourcesDownload binarys:
http://www.thefrontend.net/flake/flake-0.10-win32-bin.rar (http://www.thefrontend.net/flake/flake-0.10-win32-bin.rar)
Download sources:
http://www.thefrontend.net/flake/flake-0.10-win32-src.rar (http://www.thefrontend.net/flake/flake-0.10-win32-src.rar)
Alternative ZIP download:
Download binarys:
http://www.thefrontend.net/flake/flake-0.10-win32-bin.zip (http://www.thefrontend.net/flake/flake-0.10-win32-bin.zip)
Download sources:
http://www.thefrontend.net/flake/flake-0.10-win32-src.zip (http://www.thefrontend.net/flake/flake-0.10-win32-src.zip)
I have tested my build and it looks like encoding speed has increased , this is great news!
Test results (using my build, sf.net build i686 and flac 1.1.2 build):
TEST FILE: 44100Hz, 16bit, Stereo, 431 916 620 bytes (13 Tracks)
ENCODE_FLAKE_INTEL [-8]
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
Flake: FLAC audio encoder
© 2006 Justin Ruggles
Signed 16-bit 44100 Hz stereo
samples: 107972676
blocksize: 4608
max order: 12
order method: 4-level
stereo method: mid-side
progress: 100% | ratio: 0.741 | bitrate: 1045.2 kbps | bytes: 319851517
Kernel Time = 3.328 = 00:00:03.328 = 3%
User Time = 93.218 = 00:01:33.218 = 89%
Process Time = 96.546 = 00:01:36.546 = 92%
Global Time = 104.172 = 00:01:44.172 = 100%
ENCODE_FLAKE_i686 [-8]
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
Flake: FLAC audio encoder
© 2006 Justin Ruggles
Signed 16-bit 44100 Hz stereo
samples: 107972676
blocksize: 4608
max order: 12
order method: 4-level
stereo method: mid-side
progress: 100% | ratio: 0.741 | bitrate: 1045.2 kbps | bytes: 319851517
Kernel Time = 3.078 = 00:00:03.078 = 2%
User Time = 132.953 = 00:02:12.953 = 94%
Process Time = 136.031 = 00:02:16.031 = 96%
Global Time = 140.671 = 00:02:20.671 = 100%
ENCODE_FLAC_1_1_2 [-8]
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.
options: -P 4096 -b 4608 -m -l 12 -e -q 0 -r 0,6
Catamenia - [1999] - Morning Crimson (HQ).wav: wrote 321356528 bytes, ratio=0,74
4
Kernel Time = 3.828 = 00:00:03.828 = 1%
User Time = 206.640 = 00:03:26.640 = 96%
Process Time = 210.468 = 00:03:30.468 = 97%
Global Time = 214.891 = 00:03:34.891 = 100%
DECODE_FROM_FLAKE_INTEL [-8]
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.
t_INTEL.flac: done
Kernel Time = 3.328 = 00:00:03.328 = 7%
User Time = 22.703 = 00:00:22.703 = 53%
Process Time = 26.031 = 00:00:26.031 = 61%
Global Time = 42.188 = 00:00:42.188 = 100%
DECODE_FROM_i686 [-8]
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.
t_i686.flac: done
Kernel Time = 3.140 = 00:00:03.140 = 7%
User Time = 23.078 = 00:00:23.078 = 56%
Process Time = 26.218 = 00:00:26.218 = 63%
Global Time = 41.094 = 00:00:41.094 = 100%
DECODE_FROM_FLAC_1_1_2 [-8]
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.
t_FLAC.flac: done
Kernel Time = 3.015 = 00:00:03.015 = 7%
User Time = 22.953 = 00:00:22.953 = 56%
Process Time = 25.968 = 00:00:25.968 = 63%
Global Time = 40.765 = 00:00:40.765 = 100%
---
FILE SIZE
t_FLAC.flac 321 356 528
t_i686.flac 319 851 517
t_INTEL.flac 319 851 517
FILE SIZE
t_FLAC.flac 321 356 528
t_i686.flac 319 851 517
t_INTEL.flac 319 851 517
wisodev, what's up with the second column?
My run went at 37.00x That's more than three times faster than flac at -8 and 7.5x up from 0.7. Memory usage dropped a little, and no leaks.
Pipe support works, i can use it as external encoder in foobar2000 without problems.
What's even better, it seems foobar can automatically emebed a .cue file in the tag, only it doesn't preserve indexes, and it doesn't embed it in the actual flac way. Can't win 'em all.
Still, this is an awesome achievement. I can't believe the reference encoder had so much room for improvement!
This needs to get on rarewares.
FILE SIZE
t_FLAC.flac 321 356 528
t_i686.flac 319 851 517
t_INTEL.flac 319 851 517
wisodev, whats up with the second column?
This are output file sizes in bytes that are results of runned tests above. t_FLAC.flac is output file from original Flac encoder at version 1.1.2. The t_i686.flac and t_INTEL.flac are output file from Flake encoder version 0.10. t_i686.flac encoded using binary from Justin build and t_INTEL.flac is encoding output from my build. (It is called INTEL because I use Intel C++ Compiler to build my binarys). The mystery is solved ;-)
wisodev,
still one mystery remains:
DECODE_FROM_FLAKE_INTEL...
Kernel Time = 3.328 = 00:00:03.328 = 7%
User Time = 22.703 = 00:00:22.703 = 53%
Process Time = 26.031 = 00:00:26.031 = 61%
Global Time = 42.188 = 00:00:42.188 = 100%
DECODE_FROM_i686...
Kernel Time = 3.140 = 00:00:03.140 = 7%
User Time = 23.078 = 00:00:23.078 = 56%
Process Time = 26.218 = 00:00:26.218 = 63%
Global Time = 41.094 = 00:00:41.094 = 100%
Although both compilations are expected to generate bit-identical output, the decoding times vary.
Guess you would have to run those test multiple times to average out those variations.
.sundance.
wisodev,
still one mystery remains:
DECODE_FROM_FLAKE_INTEL...
Kernel Time = 3.328 = 00:00:03.328 = 7%
User Time = 22.703 = 00:00:22.703 = 53%
Process Time = 26.031 = 00:00:26.031 = 61%
Global Time = 42.188 = 00:00:42.188 = 100%
DECODE_FROM_i686...
Kernel Time = 3.140 = 00:00:03.140 = 7%
User Time = 23.078 = 00:00:23.078 = 56%
Process Time = 26.218 = 00:00:26.218 = 63%
Global Time = 41.094 = 00:00:41.094 = 100%
Although both compilations are expected to generate bit-identical output, the decoding times vary.
Guess you would have to run those test multiple times to average out those variations.
.sundance.
Well, the test file is more then 400 MB and testing it in right way using more runs is time consuming. I only needed some simple estimate of speed, not precise testing. But the difference in decoding time is marginal and in range of measure error I think.
This are output file sizes in bytes that are results of runned tests above. t_FLAC.flac is output file from original Flac encoder at version 1.1.2. The t_i686.flac and t_INTEL.flac are output file from Flake encoder version 0.10. t_i686.flac encoded using binary from Justin build and t_INTEL.flac is encoding output from my build. (It is called INTEL because I use Intel C++ Compiler to build my binarys). The mystery is solved ;-)
bleh put commas next time, I thought those were 3 different columns not a single number. out of curiosity where have you seen a number with spaces in it, that you didn't think the table looks wrong?
This are output file sizes in bytes that are results of runned tests above. t_FLAC.flac is output file from original Flac encoder at version 1.1.2. The t_i686.flac and t_INTEL.flac are output file from Flake encoder version 0.10. t_i686.flac encoded using binary from Justin build and t_INTEL.flac is encoding output from my build. (It is called INTEL because I use Intel C++ Compiler to build my binarys). The mystery is solved ;-)
bleh put commas next time, I thought those were 3 different columns not a single number. out of curiosity where have you seen a number with spaces in it, that you didn't think the table looks wrong?
Oh the spaces are common error in windows ;-)
Try right clicking on some file in explorer and you can see this spaces everywhere or better try in windows console DIR command and the spaces comes in file sizes. This is funny.
I want to write my notes on lack of some features in current flake version.
In windows world, especially for on-the-fly ripping, it may be problem, but if you have flac/metaflac utilites and able to write simple script/batch file you can:
- test .flac file for integrity (after compression) by flac -t <file.flac> (instead of flac's great -V options at compress time)
- add seek points for quick search in .flac file by metaflac --add-seek-points 10s <file.flac> (flac at compress time does this automatically)
- calculate and add ReplayGain tags by metaflac --add-replay-gain *.flac (instead of flac's --replay-gain option when compress a number of .wav files)
- embed cuesheet in metadata by metaflac --import-cuesheet-from <file.cue> <file.flac> (insead of flac's --cuesheet option at compress time)
- add tags by metaflac --set-tag <name=value> <file.flac> (or even better with metaflac --import-tags-from <somefile.txt> <file.flac> where somefile.txt can be created yourself or exported from current .flac by metaflac --export-tags-to <somefile.flac> <file.flac> and edited in your favorite UTF-8 capable editor).
So, metaflac is a swiss knife for .flac files when you have already compressed .flac file with lack of some features (which flac executable can add at compress time!). Referenced flac executable is great and featureful program but flake generates file smaller in size and complitely compatible with FLAC specification.
And now I'm using flake for (the best, ATM) compression (with
-8, for compatibility reasons) and flac/metaflac to test/process .flac files.
Oh the spaces are common error in windows ;-)
"It's not a bug, it's a feature."
Digit grouping symbol depends on your locale, and can be specified explicitly in Control Panel -> Regional options -> Numbers. BTW, a space is a correct grouping symbol in the Russian locale, and a comma is not. A comma is a decimal point symbol here.
Digit grouping symbol depends on your locale, and can be specified explicitly in Control Panel -> Regional options -> Numbers. BTW, a space is a correct grouping symbol in Russian locale, and a comma is not. A comma is a decimal point symbol here.
Yes thanks for clarifying this issue. In Polish locale space is also valid as you can see in screenshot attached below and yes it is feature of OS. I was bit too sarcastic I suppose.
(http://img241.imageshack.us/img241/2084/clipboard1ex3.th.jpg) (http://img241.imageshack.us/my.php?image=clipboard1ex3.jpg)
hi, I have a somewhat strange request a few pages back there was a discussion about linear solvers and pseudo-inverses, can someone possibly post actual matrix that would appear on the left hand side (taken from encoding a sample of your choice, during algorithm run) and a right hand side vector? any one will do. obviously as attachement in a text file and not plaintext in the reply (since I'm assuming the dimensions will be quite large)
thank you very much, I know it's weired
Is this encoder (flake 0.10) trustworthy for "production" use?
FYI:
I have just encoded my favorite album of Maria Peszek called "Miasto Mania" with flake -12 and it plays perfectly on my iAudio U3 2GB flash player even though the specification states that it does support only flac 0, 1 & 2. Isn't that great!
Is this encoder (flake 0.10) trustworthy for "production" use?
It depends on what you're looking for. The testing has not been extremely exhaustive, but on the other hand, I have yet to produce a FLAC file with Flake which will not decode losslessly using the reference decoder. Also, it embeds an MD5 sum for checking losslessness. One thing it lacks which may be important to some is verification during encoding.
FYI:
I have just encoded my favorite album of Maria Peszek called "Miasto Mania" with flake -12 and it plays perfectly on my iAudio U3 2GB flash player even though the specification states that it does support only flac 0, 1 & 2. Isn't that great!
Now that is a surprise! It would be interesting to make some test cases to put hardware players through a FLAC compatibility test.
I have just encoded my favorite album of Maria Peszek called "Miasto Mania" with flake -12 and it plays perfectly on my iAudio U3 2GB flash player even though the specification states that it does support only flac 0, 1 & 2. Isn't that great!
This is exactly the kind of feedback I was looking for! Thank you for this test
I have just encoded my favorite album of Maria Peszek called "Miasto Mania" with flake -12 and it plays perfectly on my iAudio U3 2GB flash player even though the specification states that it does support only flac 0, 1 & 2. Isn't that great!
This is exactly the kind of feedback I was looking for! Thank you for this test
I'm glad that I could help a bit. BTW, I've checked the iAudio site again, and to my surprise, I've found updated specification of U3:
MP3 : MPEG 1/2/2.5 Layer 3, ~320kbps, ~48khz, mono/stereo
WMA : ~256kbps, ~48khz, mono/stereo
OGG : ~Q10, ~44.1khz, mono/stereo
FLAC : compression level 0~8, ~44.1khz, mono/stereo
WAV : ~48khz, 16bit, mono/stereo
Video File up to 160x128
JPG, TXT
yes, that level 0-2 limitation was from the early days and I'm pretty sure all the iAUDIOs that support FLAC will play 0-8. their forums are the best place to ask though if you're about to buy.
Josh
I've encoded over a dozen cd's with flake -8 so far and never encountered a problem apart from the memory leaks in older versions.
Don't have a DAP that plays flacs though.
Because it has relatively few switches there's not much to go wrong either. Up to -8 it should output valid flac files that should be indistinguishable from the reference encoder output (save the encoder version string of course). But don't quote me on that.
Indeed a -V option that uses the reference decoder for verification and a warning for levels over -8 (if there isn't one already) would come in handy.
I have tried running my scripts with wisodev's flake 0.10 and both times now flake has crashed. The first time a memory allocation error was reported. My wife has just GTalked me to tell me that it has happened today (I left the script running before I went to work) so I'm afraid I can't tell you the exact error message.
I will try running the script again and post the exact (standard Windows) message, but I was wondering if anyone else had experienced this?
I've been using wiso's flake 0.10 several times in foobar's converter without any problems yet...
[edit] Forgot mentioning that I used the Intel compilation...
@Synthetic Soul
Can you try running this scripts using flake-0.10-win32-i686.zip binary from flake sourceforge page.
I have run my scripts here at work using the i586, i686 and wisodev builds available at sourceforge.
In the first run, with all three encoders using -5 all encodes (50 files (http://synthetic-soul.co.uk/comparison/lossless/corpus.asp) sent to each encoder) were successful.
I just started another batch file only using the wisodev build using -0, -5, -8 and -12. The encoder crashed using -0 on file 37.wav (http://synthetic-soul.co.uk/comparison/lossless/file.asp?FilePKID=38). This leads me to believe that something about the file is causing the issue, as the first crash happened on 37.wav also, although I can't remember the compression setting used. As previously mentioned I cannot confirm any details about the second crash.
I will perform some more tests with the wisodev build, specifically with my file 37.wav, and let you know if I can pin this down any further.
OK, I have tested 37.wav with the i586, i686 and wisodev builds at compression levels -0, -1, -5, -8, and -12.
All three builds work fine for all levels, except -0 where
all three builds crash.
Edit: Here's Flake's dying words, if it is any use:
Flake: FLAC audio encoder
(c) 2006 Justin Ruggles
Signed 16-bit 44100 Hz stereo
samples: 10465224
blocksize: 1024
max order: 0
order method: maximum
stereo method: mid-side
progress: 100% | ratio: 0.813 | bitrate: 1147.0 kbps | bytes: 33988647
Edit 2: The above tests were taken using my "Yalac" corpus (used to test Yalac/TAK). I have just run the scripts on my "FLAC" corpus (used to test 1.1.2_CVS) and I had the same happen with file 02.wav, which is the same file as 37.wav. Therefore,
out of 78 files tested only one file can highlight this bug.
Hey,
I work for Winamp, and I'm very impressed by this encoder, so I wrote a winamp encoder plugin which uses libflake.
Binary + source: http://stashbox.org/uploads/1158591116/enc_flac.zip (http://stashbox.org/uploads/1158591116/enc_flac.zip)
To compile you'll need wisodevs win32 sources, extract them to the flake directory. Feel free to put this into the sourceforge SVN or whatever.
For the moment, I just let the user configure the compression value, everything else is chosen based upon the compression by libflake.
I have replicated the speed differences between flake and the reference flac encoder when comparing with other flac encoding plugins, and for a much smaller binary!
I can't make any guarantees, but hopefully one day this will be considered for inclusion in the winamp installer.
Thanks,
Will Fisher
It seems the problem is still evident even in the first 10 seconds of the file, so I have uploaded a Flake -12 encoded version of the first 10 seconds for others to test:
http://synthetic-soul.co.uk/temp/37_0-10_12.flac (http://synthetic-soul.co.uk/temp/37_0-10_12.flac)
I can encode that sample at -0 just fine, using wisodev's Flake 0.10 build, both through Foobar and the commandline.
I did discover another bug: when trying to read from the following filename: "- 37_0-10_12.wav" or when trying to write a file called "- 37_0-10_12.flac" it fails. For the output i get a "tried to write to nonexistent pipe" error and for the input file, it treats it like an unrecognised commandline and gives the standard message. I tested further and it gives the error for all filenames that start with a dash.
Flac handles it fine.
@Synthetic Soul:
Flake 0.10 has failed to compress your test file with -0 switch on my Pentium 4 PC BUT it succeeded to compress it on my Intel Centrino laptop. Seems like Flake has some issues with Pentium 4?
Edit: I did my tests with Flake on WinXP SP2
I can encode that sample at -0 just fine, using wisodev's Flake 0.10 build, both through Foobar and the commandline.
I did discover another bug: when trying to read from the following filename: "- 37_0-10_12.wav" or when trying to write a file called "- 37_0-10_12.flac" it fails. For the output i get a "tried to write to nonexistent pipe" error and for the input file, it treats it like an unrecognised commandline and gives the standard message. I tested further and it gives the error for all filenames that start with a dash.
Flac handles it fine.
I'll try to fix that very soon. Flake only has very simple commandline parsing right now. It should be fairly trivial to fix.
Thanks,
Justin
It seems the problem is still evident even in the first 10 seconds of the file, so I have uploaded a Flake -12 encoded version of the first 10 seconds for others to test:
http://synthetic-soul.co.uk/temp/37_0-10_12.flac (http://synthetic-soul.co.uk/temp/37_0-10_12.flac)
Hmm. It works for me, but I'm using Linux. The "-0" mode has been changed some since the 0.10 release. Could you try using current SVN? If you need a binary, I can compile one & post it.
Mine is failing at home on Win2K with Athlon XP 2400+, and at work with XP. I believe the CPU at work is AMD. I will confirm later.
I will see if I can compile from SVN today at work.
Edit: I will only be able to compile if there are Visual C++ project files, as I'm not a C++ programmer, I can just hack some code and click a button to compile. As you are a Linux man I doubt this will be the case so a binary would be nice if someone could provide. Edit 2: I do have Cygwin installed, but I'm not sure that I have the necessary parts, and I've never compiled using a makefile! I see from browsing the repository that there are no Visual C++ project files.
Flake 0.10 has failed to compress your test file with -0 switch on my Pentium 4 PC BUT it succeeded to compress it on my Intel Centrino laptop. Seems like Flake has some issues with Pentium 4?
Thanks for proving I'm not mad anyway!
i think that you can just use make to compile the sources. if you want to do that, just grab a copy of MinGW(there's a nice installer for it too) and use "mingw32-make" on the directory with all of the source code. i just quickly browsed through the SVN and i saw that there was a makefile in there so theoretically, it should work. I'm a bit lazy to try it out though.
edit: btw, flake is really nice. right now, I'm using the -12 option on build 0.10 and it makes smaller files with faster decoding. i tested it with Josh's optimized FLAC build(the one which he uploaded here) and flake still beats it. I've also heard that FLAC 1.1.3 will have some of flake's improvements so i can't wait till FLAC 1.1.3 releases.
edit2: spelling
I've also heard that FLAC 1.1.3 will have some of flake's improvements
Are the two developers in contact and working cooperatively rather than competitively?
i, personally, am not too sure. i've only seen a post in a yalac thread(or something similar) by Josh saying that some of flake's improvements will be in 1.1.3. if you want to know the answer to the question, then i guess you should contact Josh.
Guys, sorry for beeing noob, but could someone of you upload the build for win32 platform using latest SVN, please?
@bukem
I will upload binarys based on SVN sources after I come home from work (around 16.00 CET).
@Synthetic Soul
You can use project files from flake-0.10-win32-wisodev-src.zip at SourceForge. Copy win32 directory to exported SVN sources root directory and build flake project using Debug setting to provide more information on possible crashes.
I appear to have managed to build flake under cygwin. I had to download autoconf and make but it finally worked (once I worked out gcc didn't like paths with spaces...).
My binary can be found here (http://synthetic-soul.co.uk/temp/flake_svn_20060919c.7z).
Edit: Link updated to bug-free version (see #113)It appears to work with 37.wav.
@Synthetic Soul
You can use project files from flake-0.10-win32-wisodev-src.zip at SourceForge. Copy win32 directory to exported SVN sources root directory and build flake project using Debug setting to provide more information on possible crashes.
Ah, thanks for that. I may take a look anyway.
Edit: wisodev. I'm using Visual C++ 2005 Express Edition and, after I had converted the solution file to the newer format, it complains about the .icproj files... saying the application is not installed... no idea what it's all about. I think I've proved for now that the SVN code works with my file, but I will no doubt test again if you release an SVN version.
Justin, out of interest, what is flake_g.exe? Is it a debug version? As I said, I'm not used to compiling with make!
i think that you can just use make to compile the sources. if you want to do that, just grab a copy of MinGW(there's a nice installer for it too) and use "mingw32-make" on the directory with all of the source code.
I tried this, but the directory had a configure file which I found out is used with autoconf to create config.mak which is required to build, so I had to go the cygwin route (I think). Thanks for the info anyway.
Edit 10: Hmm... for some reason my cygwin compile refuses to accept -12 saying 21 is too high. Using -21 works but I'm yet to check whether it is actually being accepted as 12. Is this my compile's fault?!
@wisodev:thanks
@Synthetic Soul / Justin:There's problem with your compile of flake or with that particular SVN version. When I try to use preset
-12 I get following error:
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
D:\documents\music\.work>flake -12 37_0-10_12.wav test.flac
Flake: FLAC audio encoder
(c) 2006 Justin Ruggles
invalid compression: 21. must be 0 to 12.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.
It seems that for preset 12 wrong value is hardcoded in the source code -> 21 instead of 12.
@Synthetic Soul
You can use project files from flake-0.10-win32-wisodev-src.zip at SourceForge. Copy win32 directory to exported SVN sources root directory and build flake project using Debug setting to provide more information on possible crashes.
Ah, thanks for that. I may take a look anyway.
Edit: wisodev. I'm using Visual C++ 2005 Express Edition and, after I had converted the solution file to the newer format, it complains about the .icproj files... saying the application is not installed... no idea what it's all about. I think I've proved for now that the SVN code works with my file, but I will no doubt test again if you release an SVN version.
Oh sorry I am using Intel C++ compile at version 8.0.40 and C99 switch (this is req.) witch means it wan't compile Visual Studio without Intel compiler. And version 8 of Intel C++ compiler do not works with Express edition, maybe version 9 works but it produces for me slower binarys (trail version of compiler available on intel website).
@Synthetic Soul / Justin:
There's problem with your compile of flake or with that particular SVN version. When I try to use preset -12 I get following error:
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
D:\documents\music\.work>flake -12 37_0-10_12.wav test.flac
Flake: FLAC audio encoder
(c) 2006 Justin Ruggles
invalid compression: 21. must be 0 to 12.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.
Yes, I just noticed this (see edit above)! Very curious. I can only assume it is my dodgy compiling though. I would wait for wisodev's.
there is a small bug in flake.c in the parse_number() subroutine
a quick fix
change
for(i=0; i<max; i++)
to
for(i=max-1; i>=0; i--)
there is a small bug in flake.c in the parse_number() subroutine
Thanks for that.
New binary uploaded
http://synthetic-soul.co.uk/temp/flake_svn_20060919c.7z (http://synthetic-soul.co.uk/temp/flake_svn_20060919c.7z)
Edit: Compile reuploaded using Justin's fix described below.
i, personally, am not too sure. i've only seen a post in a yalac thread(or something similar) by Josh saying that some of flake's improvements will be in 1.1.3. if you want to know the answer to the question, then i guess you should contact Josh.
To answer your question, no we are not in direct collaboration. It's not that I want to be competitive...open source is collaborative by its very nature. I do want to continue to make Flake faster and better, but I'll also be quite happy if some of the ideas in Flake are used to help improve the reference encoder.
@Synthetic Soul / Justin:
There's problem with your compile of flake or with that particular SVN version. When I try to use preset -12 I get following error:
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
D:\documents\music\.work>flake -12 37_0-10_12.wav test.flac
Flake: FLAC audio encoder
(c) 2006 Justin Ruggles
invalid compression: 21. must be 0 to 12.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.
Yes, I just noticed this (see edit above)! Very curious. I can only assume it is my dodgy compiling though. I would wait for wisodev's.
It is fixed now in SVN. And as was just pointed out, it was my dodgy coding, not anyone's dodgy compiling.
there is a small bug in flake.c in the parse_number() subroutine
a quick fix
change
for(i=0; i<max; i++)
to
for(i=max-1; i>=0; i--)
That was the problem, but that solution does not work. The 'max' parameter is not the actual number of digits, but the maximum number. If the number of digits is shorter, that code will not work. I applied a fix which counts the digits first, then works backwards.
The 'max' parameter is not the actual number of digits, but the maximum number.
the fix was as quick as my thoughts about it, sry!
btw, why do you reinvent the wheel? use
atoi()this also works
int ParseNumber(char *arg)
{
int i;
int n = 0;
for(i=0;i<strlen(arg);i++)
{
if(arg[i] < '0' || arg[i] > '9') return -1;
n *= 10;
n += (arg[i]-48);
}
return n;
}
Hey Justin, I was just looking over the code for Levinson-Durbin recursion, and noticed something odd. In it's very signature you have lpc[][], which is declared in lpc_calc_coefs as lpc[32][32] (I put numbers instead of constant name for readability here), but the algorithm works on the Toeplitz matrix, so storage should be 2N-1. That's a terrible waste of bits, even if it is only a 32x32 array.
what makes even less sense to me is that Levinson algorithm supposed to solve Ax=b for x, given A and b. and that lpc_compute_coefs seems to take 2 vectors and compute the matrix.
Wll I am bit late but I have tested this wav file witch caused crushes.
I think the s->max_frame_size is estimated incorrectly.
I have patched sources and I think fixed both bugs (crushing and incorrect command-line argument parsing).
My ParseNumber fix is similiar to posted here.
Download patched sources:
flake-svn-2006-09-19-patched-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-patched-win32-src.zip)
Download patched binarys:
flake-svn-2006-09-19-patched-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-patched-win32-bin.zip)
wiso
Thanks for the non-cygwin-reliant compile wisodev.
I must admit that I am a little confused about the s->max_frame_size bug.
The version I compiled from the SVN source worked fine with my troublesome 37.wav.
Are you saying that, even so, there was still a bug present that could reproduce the behaviour I had seen?
Thanks for the non-cygwin-reliant compile wisodev.
I must admit that I am a little confused about the s->max_frame_size bug.
The version I compiled from the SVN source worked fine with my troublesome 37.wav.
Are you saying that, even so, there was still a bug present that could reproduce the behaviour I had seen?
No problem.
- Well I tested the sources (before ParseNumber fix) and this bug was present. The free(frame) in main.c was causing this crush.
- When I changed max frame bufer to bigger value (added 4 bytes) then it was all OK.
- I have checked older sources (07 version of flake) and the estimation of max frame buffer (encode.c) was changed since them. So I have patched the svn sources and the bug was gone.
- Please try this flake-svn-2006-09-19-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-win32-bin.zip) binary because it is causing flake to crush with 37.wav file and setting
-0.
This becomes little complicated. Maybe Justin can put more light on this bug.
Thanx wisodev for non-cygwin version! And thank you Synthetic Soul for a cygwin one
I've also heard that FLAC 1.1.3 will have some of flake's improvements
Are the two developers in contact and working cooperatively rather than competitively?
it's not competitive. it's good to have independent (compliant) implementations. flake code is simpler and encodes faster, but has less features.
the improvement is in how the signal is windowed before lpc analysis. we were both working on this at the same time but flake has a shorter release cycle. the method in the upcoming version of FLAC is slower but gives slightly more compression.
Hey Justin, I was just looking over the code for Levinson-Durbin recursion, and noticed something odd. In it's very signature you have lpc[][], which is declared in lpc_calc_coefs as lpc[32][32] (I put numbers instead of constant name for readability here), but the algorithm works on the Toeplitz matrix, so storage should be 2N-1. That's a terrible waste of bits, even if it is only a 32x32 array.
that 32x32 array is for the output coefficients for all orders. the max order is 32 so you need 32+31+30+...+1 doubles. there is some waste putting it in a sparse array but not much, and the code is simpler.
what makes even less sense to me is that Levinson algorithm supposed to solve Ax=b for x, given A and b. and that lpc_compute_coefs seems to take 2 vectors and compute the matrix.
it's computing several solutions for several Ax=b, for several x and b, from the same autocorrelation of the original signal. that's the magic. see also http://en.wikipedia.org/wiki/Linear_prediction (http://en.wikipedia.org/wiki/Linear_prediction)
- Please try this flake-svn-2006-09-19-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-win32-bin.zip) binary because it is causing flake to crush with 37.wav file and setting -0.
I am now at home, on Win 2K AMD XP 2400+. I have just tried both your compile and the cygwin compile with the full 37.wav and 37_0-10_12.wav. The cygwin compile is fine, but your compile crashes.
I'm getting really confused now!
@Synthetic Soul:
if I'm not wrong wisodev has compiled two versions of flake:
1. patched (with no error) -> flake-svn-2006-09-19-patched-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-patched-win32-bin.zip)
2. and not patched (for testing purpose?) -> flake-svn-2006-09-19-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-2006-09-19-win32-bin.zip)
Yes. The patched version does work fine here.
However, what is still slightly confusing is that my cygwin compile from the SVN works fine, but wisodev's non-patched compile from the SVN does not.
NB: When I posted above I was confused about wisodev's two compiles, but I realised after posting what he was asking. However, as you can see, I am still confused. That said, if wisodev's patch means that all new compiles would work then I suppose that should be an end to the confusion... I don't need to worry why my compile works... I hope.
@wisodev:another problem found -> "p" switch for padding isn't working at all; it seems that is treated as "s" switch for stereo decorrelation
D:\documents\music\.work>flake -p 256 -12 37_0-10_12.wav test.flac
Flake: FLAC audio encoder
(c) 2006 Justin Ruggles
invalid stereo decorrelation method: 2. must be 0 or 1.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.
Yes. The patched version does work fine here.
However, what is still slightly confusing is that my cygwin compile from the SVN works fine, but wisodev's non-patched compile from the SVN does not.
NB: When I posted above I was confused about wisodev's two compiles, but I realised after posting what he was asking. However, as you can see, I am still confused.
Sometimes bugs only show up due to compiler related attributes, such as the byte-padding required for certain data types, or the order of items in the heap. That is to say, the bug exists in both versions, but depending on which compiler compiled it, the bug might appear immediately (data overwritten was important), occassionally (data overwritten might be discarded often, or LSB of some other musical data, etc.) or not at all (due to enforced data-alignment and/or padding with unused bytes).
what makes even less sense to me is that Levinson algorithm supposed to solve Ax=b for x, given A and b. and that lpc_compute_coefs seems to take 2 vectors and compute the matrix.
it's computing several solutions for several Ax=b, for several x and b, from the same autocorrelation of the original signal. that's the magic. see also http://en.wikipedia.org/wiki/Linear_prediction (http://en.wikipedia.org/wiki/Linear_prediction)
that only furthers my confusion, it reiterates that we solving Ax=b for x, given a matrix A and a vector b. but in the function call we only have 1 vector as input, and 1 empty vector and 1 empty matrix:
compute_lpc_coefs(const double *autoc, int max_order, double lpc[][MAX_LPC_ORDER], double *ref)
autoc is the Nx1 vector that is precomputed, max_order is given int, but lpc[][] and *ref are not initialized to anything before they are passed to this function. What is matrix A and vector b then?
I guess my confusion comes from that fact that I don't understand what autoc holds after compute_autocorr is called to fill it. But even then we still apparently trying to solve Ax=b with only 1 of the 3 variables given.
I guess my confusion comes from that fact that I don't understand what autoc holds after compute_autocorr is called to fill it.
http://en.wikipedia.org/wiki/Autocorrelation (http://en.wikipedia.org/wiki/Autocorrelation)
But even then we still apparently trying to solve Ax=b with only 1 of the 3 variables given.
the levinson-durbin algo is recursive, so you can model an unknown system as an autoregressive process
it is a two step process. the autocorrelation step transforms the problem into the Ra=-r form from the article, this is called the autocorrelation formulation of the original Ax=b.
the autocorrelation matrix R is circulant and can be represented efficiently as a vector. that's the input to the second step that calculates the final filter coefficients for each order with levinson/levinson-durbin recursion.
Josh
p.s. maybe the confusion is on the arguments, autoc is input (vector form of R) and lpc are the output coefficients for each order. because of the recursive nature of the solver you can get the solution for each higher order with only incremental effort, that's why I calculate them all. in shorten the loop aborts when the error estimate stops decreasing to save time.
Thank you for taking time to explain, guys. Right now my greatest confusion comes from notation they use, I can't quite grasp what -r is. From this line in the LPC explanation:
The above equations are called the normal equations or Yule-Walker equations. In matrix form the equations can be equivalently written as: Ra = -r
Yet the "above" line they are talking about seems to say that -r = -R. So are we solving Ra = -R?
Yet below that line, it says that the system Ra = -r is the same as Ra= [1 0 0 0 ... 0]. Is that why we have no vector b in function parameters, because it's assumed to be first unit vector?
in addition to all that, what the hell is R(-j) mean? what kind of a notation is that, negative can't be a valid index.
@wisodev:
another problem found -> "p" switch for padding isn't working at all; it seems that is treated as "s" switch for stereo decorrelation
D:\documents\music\.work>flake -p 256 -12 37_0-10_12.wav test.flac
Flake: FLAC audio encoder
(c) 2006 Justin Ruggles
invalid stereo decorrelation method: 2. must be 0 or 1.
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.
I fixed that this morning 9:46 EST. It was missing a break statement after the section which handled the '-p' parsing.
edit: maybe not...I thought I had fixed it. Will look into further.
edit#2: This is now indeed fixed.
the improvement is in how the signal is windowed before lpc analysis. we were both working on this at the same time but flake has a shorter release cycle. the method in the upcoming version of FLAC is slower but gives slightly more compression.
On this note, I've been considering doing an adaptive selection between several windows based on either a simple predictability measurement (e.g. energy of fixed 2nd order residual) or maybe a very coarse covariance. Have you done any experimentation with adaptive window selection? I know that the mp4als reference encoder does something like this, but only a binary is supplied for that part of the encoder, not the source.
Users of Gentoo Linux may be interested in the ebuilds I wrote for flake, both the 0.10 release and the subversion one. See http://berkano.net/bits/2006/09/20/gentoo-overlay (http://berkano.net/bits/2006/09/20/gentoo-overlay)
Justin:
Thank you for the fix.
wisodev:
This may get boring after while but I'm waiting for your new native compile
While we () are waiting, you could try my cygwin compile.
http://synthetic-soul.co.uk/temp/flake_svn_20060920.7z (http://synthetic-soul.co.uk/temp/flake_svn_20060920.7z)
@Mangix, if you see this, I got the source using the SVN command line (I'm at home), using the command detailed on Sourceforge. It took less than a minute. I downloaded the SVN zip (http://subversion.tigris.org/downloads/svn-win32-1.4.0.zip), unzipped (I just unzipped to my desktop for speed), opened a command line in the "bin" ("\svn-win32-1.4.0\svn-win32-1.4.0\bin") directory and then used the command:
svn co https://svn.sourceforge.net/svnroot/flake-enc (https://svn.sourceforge.net/svnroot/flake-enc) flake-enc
This created a folder in my "bin" directory called "flake-enc". I then moved that to somewhere more convenient (a short path with no spaces) and then ran "./configure" then "make" using a cygwin console. A few minutes work in all. I hope this helps. I would be interested to see your MinGW compile (if that's the way you go). I couldn't work out how to do it yesterday.
'configure && make' is the only thing needed for MingW/MSYS AFAICT.
valgrind confirms that wisodev's patch
diff -urd flake-enc/libflake/encode.c flake-svn-2006-09-19-patched-win32-src/libflake/encode.c
--- flake-enc/libflake/encode.c 2006-09-20 10:39:45.000000000 +0200
+++ flake-svn-2006-09-19-patched-win32-src/libflake/encode.c 2006-09-19 16:59:42.000000000 +0200
@@ -341,11 +341,7 @@
else ctx->lpc_precision = 15;
/* set maximum encoded frame size in verbatim mode */
- if(ctx->channels == 2) {
- s->max_frame_size = 14 + ((ctx->blocksize * 33 + 7) >> 3);
- } else {
- s->max_frame_size = 14 + (ctx->blocksize * ctx->channels * 2);
- }
+ s->max_frame_size = 14 + ((ctx->blocksize * ctx->channels * (ctx->bps+1)) >> 3);
ctx->max_framesize = s->max_frame_size;
/* select amount of padding to use in header */
fixes the issue. Otherwise I get:
==29220== Invalid write of size 4
==29220== at 0x409EC1: output_residual (bitio.h:103)
==29220== by 0x40A676: output_subframes (encode.c:729)
==29220== by 0x40AF62: flake_encode_frame (encode.c:801)
==29220== by 0x401F21: main (flake.c:378)
==29220== Address 0x403DAB5 is 4,237 bytes inside a block of size 4,238 alloc'd
==29220== at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220== by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid write of size 1
==29220== at 0x40AF76: flake_encode_frame (bitio.h:70)
==29220== by 0x401F21: main (flake.c:378)
==29220== Address 0x403DAB9 is 3 bytes after a block of size 4,238 alloc'd
==29220== at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220== by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid read of size 1
==29220== at 0x40B7C0: calc_crc16 (crc.c:71)
==29220== by 0x40AFC1: flake_encode_frame (encode.c:767)
==29220== by 0x401F21: main (flake.c:378)
==29220== Address 0x403DAB6 is 0 bytes after a block of size 4,238 alloc'd
==29220== at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220== by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid read of size 1
==29220== at 0x40B7C9: calc_crc16 (crc.c:71)
==29220== by 0x40AFC1: flake_encode_frame (encode.c:767)
==29220== by 0x401F21: main (flake.c:378)
==29220== Address 0x403DAB7 is 1 bytes after a block of size 4,238 alloc'd
==29220== at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220== by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid read of size 1
==29220== at 0x40B7EF: calc_crc16 (crc.c:71)
==29220== by 0x40AFC1: flake_encode_frame (encode.c:767)
==29220== by 0x401F21: main (flake.c:378)
==29220== Address 0x403DAB8 is 2 bytes after a block of size 4,238 alloc'd
==29220== at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220== by 0x401EAD: main (flake.c:369)
==29220==
==29220== Invalid read of size 1
==29220== at 0x40B81D: calc_crc16 (crc.c:71)
==29220== by 0x40AFC1: flake_encode_frame (encode.c:767)
==29220== by 0x401F21: main (flake.c:378)
==29220== Address 0x403DAB9 is 3 bytes after a block of size 4,238 alloc'd
==29220== at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==29220== by 0x401EAD: main (flake.c:369)
BTW, the svn version has been speeded up by quite some amount in conrast to last release. Some minor speed-up will be commited shortly, I think. So maybe someone likes to bench.
valgrind confirms that wisodev's patch
diff -urd flake-enc/libflake/encode.c flake-svn-2006-09-19-patched-win32-src/libflake/encode.c
--- flake-enc/libflake/encode.c 2006-09-20 10:39:45.000000000 +0200
+++ flake-svn-2006-09-19-patched-win32-src/libflake/encode.c 2006-09-19 16:59:42.000000000 +0200
@@ -341,11 +341,7 @@
else ctx->lpc_precision = 15;
/* set maximum encoded frame size in verbatim mode */
- if(ctx->channels == 2) {
- s->max_frame_size = 14 + ((ctx->blocksize * 33 + 7) >> 3);
- } else {
- s->max_frame_size = 14 + (ctx->blocksize * ctx->channels * 2);
- }
+ s->max_frame_size = 14 + ((ctx->blocksize * ctx->channels * (ctx->bps+1)) >> 3);
ctx->max_framesize = s->max_frame_size;
/* select amount of padding to use in header */
While that works, it is not as accurate and does not fix the root of the problem. See if this patch helps.
Index: libflake/bitio.c
===================================================================
--- libflake/bitio.c (revision 2)
+++ libflake/bitio.c (working copy)
@@ -39,7 +39,7 @@
buf = NULL;
}
bw->buffer = buf;
- bw->buf_end = bw->buffer + len;
+ bw->buf_end = bw->buffer + (((len+3)>>2)<<2); // must be 32-bit aligned
bw->buf_ptr = bw->buffer;
bw->bit_left = 32;
bw->bit_buf = 0;
@@ -77,7 +77,7 @@
fprintf(stderr, "ERROR: bits=%d val=%u\n", bits, val);
}*/
- if(bw->eof || bw->buf_ptr >= bw->buf_end) {
+ if(bw->eof || (bw->buf_ptr+3) >= bw->buf_end) {
bw->eof = 1;
return;
}
edit: updated patch
'configure && make' is the only thing needed for MingW/MSYS AFAICT.
Yeah, no doubt. The whole thing was new to me yesterday, and I was rushing as I was supposed to be working. I dunno, I tried a few different things and never sucessfully compiled. I was getting very confused between my cygwin and my mingw though! Maybe I'll find some time to get to grips with mingw in the future.
To get this post on topic, I have performed a couple of quick tests using -5 and I would say wisodev's compile takes just over 90% as long to encode as my cygwin compile. I guess I should have tested with the Sourceforge i686 compile as well... Also, two files were compressed to different sizes, although are bit-identical.
BTW, the svn version has been speeded up by quite some amount in conrast to last release. Some minor speed-up will be commited shortly, I think. So maybe someone likes to bench.
Good Lord it's difficult to get a version to test. I knew I shouldn't have gotten into this! If you could let us know when these changes are checked in that would be cool.
I only wanted to test Flake against Yalac, but it's proving a bit of a job to find a starting point.
@Justin
Your patch makes it better, but still not sufficient:
==23819== Invalid write of size 1
==23819== at 0x40B0EC: flake_encode_frame (bitio.h:71)
==23819== by 0x401F71: main (flake.c:378)
==23819== Address 0x403DAB6 is 0 bytes after a block of size 4,238 alloc'd
==23819== at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==23819== by 0x401EFD: main (flake.c:369)
==23819==
==23819== Invalid read of size 1
==23819== at 0x40BE29: calc_crc16 (crc.c:71)
==23819== by 0x40B137: flake_encode_frame (encode.c:767)
==23819== by 0x401F71: main (flake.c:378)
==23819== Address 0x403DAB6 is 0 bytes after a block of size 4,238 alloc'd
==23819== at 0x4C2006B: malloc (vg_replace_malloc.c:149)
==23819== by 0x401EFD: main (flake.c:369)
Good Lord it's difficult to get a version to test. I knew I shouldn't have gotten into this! If you could let us know when these changes are checked in that would be cool.
The major speed-ups are already checked in.
The major speed-ups are already checked in.
OK, thanks.
What about the errors with Justin's patch? Will they be included?
Edit: K. I had another go with MinGW and it worked just fine. I think my earlier mistake was simply not using the MSYS console to run the apps.
I have checked using objdump and get the following:
$ objdump -p flake.exe | grep "DLL Name"
DLL Name: KERNEL32.dll
DLL Name: msvcrt.dll
DLL Name: msvcrt.dll
(Note the reliance on msvcrt.dll and not cygwin1.dll.)
Compiles using current SVN (rev 15): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_20060920b.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_20060920b_mingw.7z)
the improvement is in how the signal is windowed before lpc analysis. we were both working on this at the same time but flake has a shorter release cycle. the method in the upcoming version of FLAC is slower but gives slightly more compression.
On this note, I've been considering doing an adaptive selection between several windows based on either a simple predictability measurement (e.g. energy of fixed 2nd order residual) or maybe a very coarse covariance. Have you done any experimentation with adaptive window selection? I know that the mp4als reference encoder does something like this, but only a binary is supplied for that part of the encoder, not the source.
I haven't tried anything like that. I did a brute force exhaustive search test (try all windows for each frame, pick best one) to see what the lower bound was and it didn't end up yielding that much more compression.
when it comes to adding fancy like adaptive schemes I start to worry about patents since currently in libFLAC the encoder and decoder ship together.
Josh
The major speed-ups are already checked in.
OK, thanks.
What about the errors with Justin's patch? Will they be included?
Based on Prakash's latest test I didn't quite solve the issue. I'll keep working on it and hopefully be able to apply a definite fix this evening. I just have to find a test sample which makes the valgrind complaint occur. I think it's only very noisy samples which have to be encoded in verbatim mode so as not to actually inflate the file size.
Edit: This issue should be completely fixed in SVN now.
the improvement is in how the signal is windowed before lpc analysis. we were both working on this at the same time but flake has a shorter release cycle. the method in the upcoming version of FLAC is slower but gives slightly more compression.
On this note, I've been considering doing an adaptive selection between several windows based on either a simple predictability measurement (e.g. energy of fixed 2nd order residual) or maybe a very coarse covariance. Have you done any experimentation with adaptive window selection? I know that the mp4als reference encoder does something like this, but only a binary is supplied for that part of the encoder, not the source.
I haven't tried anything like that. I did a brute force exhaustive search test (try all windows for each frame, pick best one) to see what the lower bound was and it didn't end up yielding that much more compression.
when it comes to adding fancy like adaptive schemes I start to worry about patents since currently in libFLAC the encoder and decoder ship together.
Josh
Thanks. That's good to know. It may not be worth the research/testing time if the gain is only trivial.
-Justin
Since i don't have much to add to the technical discussion, all i'll say is this: It'd be very useful to have someone providing proper daily builds, also because this makes promoting flake usage much easier and prompts people to give more feedback. Then i'd happily put my Athlon XP to benchmarking.
As flake is not a reference encoder, patents are less of a worry, right?
Edit: This issue should be completely fixed in SVN now.
Thanks Justin.
Compiles using current SVN (rev 27): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_r27_cygwin.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_r27_mingw.7z)
I should point out that I'm just posting these for interest. Due to the speed improvements of wisodev's builds I would wait for him to post if you want to do comparison testing.
FLAKE SVN REVISION 27
Download binarys:
flake-svn-r27-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r27-win32-bin.zip)
Download sources:
flake-svn-r27-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r27-win32-src.zip)
wiso
Cool, thanks wisodev.
I think this is the version I'll use to compare with Yalac, unless anyone can tell me of any major changes being checked in in the next day or so?
Cool, thanks wisodev.
I think this is the version I'll use to compare with Yalac, unless anyone can tell me of any major changes being checked in in the next day or so?
Well, I do have the day off work, so maybe I'll take that as a challenge and work on Flake.
At any rate, if I do implement the next big thing on my list I will probably do another version release several days after. There is 1 change I'll work on this morning that you may want to wait for though. The low-compression levels (0 to 3) will probably get faster.
-Justin
Sounds like I best hold fire.
Good luck with the day off.
Sounds like I best hold fire.
Sorry if i should be asking for too much. You allready have done so much to help me!
Yalac V0.12 will soon be done. After the integration of a seek table and the final tuning of the presets it's performance should not change anymore until the first public release (ok, it may get a bit faster when i disable my debug code).
A test of V0.12 isn't urgent for me, but if it would be easier for you to perform both tests simultaneously, i could work a bit faster.
Thanks
Thomas
Well, I've gotten a lot done today. I just applied code to SVN for variable block size. You can test it out using "-v 1" on the commandline. It has much less of an adverse effect on speed than I had anticipated, which is great. Hopefully in the future I'll be able to enhance the algorithm for the block splitting decision. This works pretty well for a first attempt though.
One down-side is that the resulting files are not within the FLAC Subset format. They do, however, decode just fine with the libFLAC and FFmpeg decoders. Hardware support may be sporatic though.
I'm not going to add anything big for a few days and just focus on bug fixes and small changes. Then I'll release a new version.
-Justin
Compiles using current SVN (rev 42): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_r42_cygwin.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_r42_mingw.7z)
Sorry if i should be asking for too much. You allready have done so much to help me!
Hey, no worries Thomas. I am doing this for my own interest, although of course I'm hoping it will benefit you (and Justin and Josh perhaps).
A test of V0.12 isn't urgent for me, but if it would be easier for you to perform both tests simultaneously, i could work a bit faster.
No, that's fine Thomas, I wouldn't want to rush you. It doesn't make too much difference to me if I perform the tests at different times. Thanks anyway.
Well, I've gotten a lot done today. I just applied code to SVN for variable block size. You can test it out using "-v 1" on the commandline. It has much less of an adverse effect on speed than I had anticipated, which is great.
...
I'm not going to add anything big for a few days and just focus on bug fixes and small changes. Then I'll release a new version.
Cool, sounds interesting. Once I have wisodev's build I'll set some scripts running.
@Justin:I'll check new build on my iAudio U3 this evening. I'm just curious if it's able to handle variable block sized flac files.
Edit: Huston we have a problem! I've just downloaded Synth's compile and -v switch IS NOT SUPPORTED D:\documents\music\.work>flake -p 256 -12 -v 1 "01. autechre - kalpol introl.wav" "01. autechre - kalpol introl.flac"
Flake: FLAC audio encoder
(c) 2006 Justin Ruggles
invalid option: -v
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.
Edit2: I've checked the SVN (v42) and found -v switch implemented in flake.c - seems that there's something wrong with Synth's compile...
OK, I've managed to compile flake v42 by myself and to do a quick test with variable block sized flac on my iAudio U3 - and I'm simply astonished because it just WORKS! There is small glitch though, the player doesn't display remaining play time of the track anymore but that's not a problem for me.
FYI: I've converted Thunderball's album "Scorpio Rising" with flake -p 256 -12 -v 1. I just love my iAudio U3
Edit2: I've checked the SVN (v42) and found -v switch implemented in flake.c - seems that there's something wrong Synth's compile...
How bloody annoying! Only time they may have been of use and they weren't right! I have recompiled, checked the -v switch, and uploaded, just in case. I have no idea what happend this morning, I'm sure
some files had updated...
Thanks for highlighting the error.
Edit: OK, I've just done some (after the horse has bolted) testing with -v 1 at -0, -5, -8 and -12 and in all cases the -v versions are bigger than the fixed block size versions. Am I missing something?
E.g.:
C:\C\Flake\SVN\flake>flake-svn-cygwin -5 "C:\Documents and Settings\Neil\My Docu
ments\Audio\Flac Corpus\source\20.wav" 5_no.flac
Flake: FLAC audio encoder
© 2006 Justin Ruggles
Signed 16-bit 44100 Hz stereo
samples: 2743608
block size: 4608
variable: no
prediction type: levinson-durbin
prediction order: 1,8
partition order: 0,6
order method: estimate
stereo method: mid-side
header padding: 4096
progress: 100% | ratio: 0.413 | bitrate: 583.2 kbps | bytes: 4525937
C:\C\Flake\SVN\flake>flake-svn-cygwin -5 -v 1 "C:\Documents and Settings\Neil\My
Documents\Audio\Flac Corpus\source\20.wav" 5_yes.flac
Flake: FLAC audio encoder
© 2006 Justin Ruggles
Signed 16-bit 44100 Hz stereo
samples: 2743608
** WARNING! FLAC file will not be Subset compliant **
block size: 4608
variable: yes
prediction type: levinson-durbin
prediction order: 1,8
partition order: 0,6
order method: estimate
stereo method: mid-side
header padding: 4096
progress: 100% | ratio: 0.414 | bitrate: 583.5 kbps | bytes: 4528149
Edit: OK, I've just done some (after the horse has bolted) testing with -v 1 at -0, -5, -8 and -12 and in all cases the -v versions are bigger than the fixed block size versions. Am I missing something?
Hmmm. It worked for my samples. I'll do more testing tonight. I may temporarily add a variable frame-split-threshold until I can figure things out.
Thanks for the feedback,
-Justin
In my case I saved 1 Mbyte of 319 Mbytes flac album when converted with -v 1 switch.
Justin, my post was a bit premature. I was just a little surprised at the (minor) increase in size over various compression levels, so posted without really thinking.
What I totaly forgot was that this was just one file.
To rectify I have just tested -5 on my "FLAC corpus" (28 files) and the results are a lot better.
Out of the 28 files 12 files suffer with the -v switch, but on the whole the compression improves from 59.843% to 59.754%, or 1,221,384 bytes in 1,371,283,088 (a 0.001% improvement).
As you can see from the figures, those that suffer from variable blocks (-v 1) are the files at the outer edges - generally those that compress the best (e.g.: Charlotte Church, Andrea Bocelli), but also two Motorhead tracks (-274 and -5,698). This variation reminds me of the window testing I was involved in for FLAC 1.1.2_CVS (1.1.3), with the two bounds acting differently than the core files... not sure if that's relevant.
When wisodev is kind enough to provide a compile of Rev. 42 I'll soon provide some more figures, using my Yalac corpus, including timings and using various "key" compression levels.
Wav Off On
Filesize Comp % Filesize Comp % Diff
====================================================================
44205884 30037774 67.950% 29996667 67.857% 41,107
40501484 18854486 46.553% 18860045 46.566% -5,559
41860940 32194824 76.909% 32179874 76.873% 14,950
36997004 11961940 32.332% 11965877 32.343% -3,937
41849180 22570842 53.934% 22499530 53.763% 71,312
43095740 31742309 73.655% 31646473 73.433% 95,836
98031404 52089827 53.136% 52103883 53.150% -14,056
29964524 22412071 74.795% 22412345 74.796% -274
44963228 18196142 40.469% 18207104 40.493% -10,962
37462700 25086008 66.963% 24958122 66.621% 127,886
35209484 24646149 69.999% 24626419 69.943% 19,730
39280796 26879523 68.429% 26836293 68.319% 43,230
35898620 24982661 69.592% 24917519 69.411% 65,142
44295260 30913592 69.790% 30808634 69.553% 104,958
.134433308 84608716 62.937% 84103316 62.561% 505,400
46214492 33193196 71.824% 33150151 71.731% 43,045
70752908 31462354 44.468% 31470864 44.480% -8,510
37989548 22986231 60.507% 22940337 60.386% 45,894
56989004 29822007 52.329% 29827193 52.339% -5,186
47973788 28413887 59.228% 28367289 59.131% 46,598
10974476 4525937 41.241% 4528149 41.261% -2,212
45400700 17135222 37.742% 17141711 37.756% -6,489
79469420 56785940 71.456% 56754994 71.417% 30,946
35719868 26702556 74.755% 26675013 74.678% 27,543
32309468 13942750 43.154% 13947729 43.169% -4,979
73810508 30283837 41.029% 30312140 41.068% -28,303
55389644 44922899 81.103% 44928597 81.114% -5,698
30239708 23258281 76.913% 23224309 76.801% 33,972
====================================================================
1371283088 820611961 59.843% 819390577 59.754% 1221384
FLAKE SVN REVISION 42
Download binarys:
flake-svn-r42-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r42-win32-bin.zip)
Download sources:
flake-svn-r42-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r42-win32-src.zip)
wiso
I would caution against encoding variable-blocksize files in native FLAC unless familiar with the implications. there is a design flaw in the native FLAC container that can make variable blocksize files difficult to handle in some ways (I may fix this later but it's a format change). this is not a problem when in other containers e.g. Ogg FLAC.
definitely I would say that a fraction of a percent size decrease is not worth the useability tradeoff.
Josh
I tested wisodev's build on my two primary file sets as before flake V0.10.
ParticipiantsYalac V0.11a slightly improved over V0.11.
FLAC 1.1.2
Flake 0.10 file flake-0.10-win32-i686.zip from sourceforge.
Flake sv42 flake-svn-r42-win32-bin.zip from wisodev.
Results Yalac V0.11a | FLAC | Flake 0.10 | Flake sv42 |
Turbo Fast | -8 | -8 -12 | -8 -12 |
-----------------------+--------+---------------+---------------+
rw | | | |
Ratio: 58.03 57.14 | 59.54 | 59.06 58.74 | 58.88 58.51 |
-----------------------+--------+---------------+---------------+
songs | | | |
Ratio: 49.09 48.37 | 51.35 | 50.27 49.59 | 50.27 49.58 |
-----------------------+--------+---------------+---------------+
Ratio is the compression ratio in percent. No speed data this time, because i am not sure, if both flake versions have been bulit with the same compiler optimizations. Subjectively Flake sv42 with -v 1 seems not to be significantly slower than V0.10.
For the test sets:"rw" contains 46 files from http://www.rarewares.org/test_samples/ (http://www.rarewares.org/test_samples/). Should be the whole dir. I call this my primary test set and prefer to use it for my tuning.
"songs" contains CD-Ripps of the beginning of some songs. This set contains many files that are benefiting from higher predictor orders. It may be less representative than set rw, but i keep it because it has been my primary set for many years.
Some remarksUp to 0.23 percent better compression for sample set "rw", only 0.01 better compression for "songs".
Not too surprising for me: "rw" contains many samples with very fast changes of the signal characteristics, where variable block sizes should help most.
Possibly Justin could try to use some of the "rw" files for further optimizations of the block length switching. I found them very useful for this.
"songs" contains files, which can quite well be encoded with constant frame sizes of up to 200 ms.
Keep on the great work!
Thomas
Here the results of set "rw" for each file:
| FLAC | Flake 0.10 | Flake sv42 -v1|
| -8 | -8 -12 | -8 -12 |
---------------------+--------+---------------+---------------+
41_30sec.flac 65.56 65.77 65.66 65.33 65.20
ATrain.flac 51.58 50.08 49.66 49.87 49.34
Bachpsichord.flac 64.67 64.20 63.33 63.99 63.08
Bartok_strings2.flac 53.29 51.85 51.16 51.94 51.14
BeautySlept.flac 63.37 63.35 62.81 63.25 62.68
BigYellow.flac 75.40 75.12 74.94 74.81 74.61
Blackwater.flac 60.64 59.43 59.30 59.38 59.21
bodyheat.flac 59.79 59.32 58.86 59.51 58.95
chanchan.flac 64.35 64.08 64.01 63.77 63.69
DaFunk.flac 66.87 66.87 66.79 66.13 66.00
death2.flac 40.31 41.52 41.03 38.94 38.58
Debussy.flac 28.54 27.95 27.83 28.01 27.84
EnolaGay.flac 65.48 65.17 63.54 65.40 63.80
experiencia.flac 71.82 71.65 71.47 71.66 71.46
female_speech.flac 34.15 34.47 34.08 34.06 33.44
FloorEssence.flac 64.68 64.23 64.13 64.12 63.99
getiton.flac 53.79 53.15 53.02 52.70 52.56
gone.flac 65.67 65.12 65.06 65.09 65.01
Hongroise.flac 40.84 36.65 36.45 36.53 36.31
Illinois.flac 59.58 59.19 58.86 59.23 58.84
ItCouldBeSweet.flac 56.86 56.65 56.46 56.95 56.65
kraftwerk.flac 46.50 46.82 46.34 45.84 45.40
Layla.flac 65.93 65.59 65.21 65.18 64.80
Leahy.flac 73.84 73.82 73.75 73.76 73.67
LifeShatters.flac 70.58 69.95 69.78 70.06 69.83
macabre.flac 63.27 62.80 62.02 62.89 62.04
Mahler.flac 49.86 48.91 48.86 48.85 48.80
male_speech.flac 36.28 36.36 36.08 35.94 35.57
Mama.flac 66.85 66.47 66.43 66.47 66.41
MidnightVoyage.flac 62.40 61.92 61.23 61.91 61.13
mybloodrusts.flac 70.14 70.01 69.94 70.08 69.97
NewYorkCity.flac 78.30 78.33 78.14 78.36 78.18
OrdinaryWorld.flac 67.66 67.22 66.59 66.77 66.16
Polonaise.flac 36.50 33.94 33.46 34.05 33.46
Quizas.flac 60.78 60.66 60.55 59.91 59.77
riteofspring.flac 51.95 53.39 53.33 53.32 53.23
rosemary.flac 55.77 55.31 55.16 55.19 55.01
Scars.flac 59.81 59.16 58.62 59.06 58.53
SinceAlways.flac 71.10 70.69 70.64 70.28 70.21
thear1.flac 74.01 73.64 73.61 73.64 73.62
TheSource.flac 57.28 57.05 56.88 56.61 56.43
TomsDiner.flac 53.73 53.81 52.96 53.85 52.82
trust.flac 61.49 60.71 60.64 60.70 60.59
Twelve.flac 72.16 71.85 71.55 71.96 71.53
velvet.flac 68.12 68.12 67.70 67.70 67.48
Waiting.flac 67.67 67.18 66.55 67.19 66.46
| FLAC | Flake 0.10 | Flake sv42 -v1|
| -8 | -8 -12 | -8 -12 |
---------------------+--------+---------------+---------------+
...
riteofspring.flac 51.95 53.39 53.33 53.32 53.23
...
I guess that it can't be a typo, but could you confirm these values?
| FLAC | Flake 0.10 | Flake sv42 -v1|
| -8 | -8 -12 | -8 -12 |
---------------------+--------+---------------+---------------+
...
riteofspring.flac 51.95 53.39 53.33 53.32 53.23
...
I guess that it can't be a typo, but could you confirm these values?
Yes, they are right. This file is one of the very rare cases, where a 16 bit file (or a part of it) contains a constant least significant bit, which can be removed by Flac's "Look for wasted bits" option. Probably in Flake this option is not implemented.
Confirmation... and explanation! Thanks
N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?
N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?
I have not performed a deeper analysis of this file. Possibly one channel has a constant least significant bit (LSB), because compression can be improved by about 3 percent by the wasted bits option. If both channels were affected, it would be about 6 percent.
"That the 16th bit is always the same through the whole file?" Through the whole file or through a whole frame (my notation) or block (Flac's notation). Usually the analysis is beeing performed on a frame basis (implementaion dependend). And theoretically more than 1 bit can be constant.
"In other words 15 bit recording pad to 16 bit?" If someone multiplies each sample with 2 (or shifts it by one place), the least significant bit will become zero:
decimal binary
9 1001
18 10010
new winamp encoder plugin as per wizos drop: http://stashbox.org/uploads/1158969212/enc_flac.dll (http://stashbox.org/uploads/1158969212/enc_flac.dll)
although i ran into a couple of problems compiling.
Would it be too much of a problem to put my changes into the code, to keep it portable across compilers?
optimize.c line 219 replaced with: uint32_t bits[32]; (where 32 used to be levels. MSVC doesn't like non-constant sizes for these things.) I am sure that 32 is as large as levels gets.
in encode.c flake_set_defaults, msvc doen't like this syntax:
foo = ((int[]){1,2,3})[2];
so i defined all these "nameless" arrays as ones with names, so it becomes:
int sel[] = {1,2,3};
foo = sel[2];
thanks,
will
I would caution against encoding variable-blocksize files in native FLAC unless familiar with the implications. there is a design flaw in the native FLAC container that can make variable blocksize files difficult to handle in some ways (I may fix this later but it's a format change). this is not a problem when in other containers e.g. Ogg FLAC.
definitely I would say that a fraction of a percent size decrease is not worth the useability tradeoff.
Josh
I agree. The variable block size headers are a bit tricky. I don't have any issue with using it for my personal files because I know the players I use support it. If I was sharing the files with other people I would definitely not use that option though. Perhaps I should change the warning to something less technical like "Warning! The chosen options are not part of the FLAC Subset format and therefore may not be compatible with some FLAC players."
N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?
I have not performed a deeper analysis of this file. Possibly one channel has a constant least significant bit (LSB), because compression can be improved by about 3 percent by the wasted bits option. If both channels were affected, it would be about 6 percent.
"That the 16th bit is always the same through the whole file?" Through the whole file or through a whole frame (my notation) or block (Flac's notation). Usually the analysis is beeing performed on a frame basis (implementaion dependend). And theoretically more than 1 bit can be constant.
"In other words 15 bit recording pad to 16 bit?" If someone multiplies each sample with 2 (or shifts it by one place), the least significant bit will become zero:
decimal binary
9 1001
18 10010
I think I may have removed that feature because I could not find any files which had any wasted bits...I can't really remember. I at least know I had implemented it at one point, but it's definitely no longer there. It would be easy enough to add.
As far as the compression gain for variable block size, I have a feeling I can do better. This was a first attempt and pretty much a shot-in-the-dark. What I think I'll do next is a really slow brute-force approach so that I will have something to compare results against.
Thanks for all the great results. It would be interesting to see some speed comparisons. I know that the current SVN version is quite a bit faster on my machine than 0.10, but my machine is certainly not typical (Linux, AMD K6-2).
-Justin
Thanks for all the great results. It would be interesting to see some speed comparisons. I know that the current SVN version is quite a bit faster on my machine than 0.10, but my machine is certainly not typical (Linux, AMD K6-2).
Same as before with encoding times.
ParticipiantsFLAC 1.1.2
Flake 0.10 file flake-0.10-win32-i686.zip from sourceforge.
Flake sv42 flake-svn-r42-win32-bin.zip from wisodev.
Test systemP3-800 with a (slow) 40 GB HD.
FLAC | Flake 0.10 | Flake sv42 -v1|
-8 | -8 -12 | -8 -12 |
------------------+---------------+---------------+
rw | | |
Ratio: 59.54 | 59.06 58.74 | 58.88 58.51 |
EncoTime: 280.47 | 133.57 871.84 | 79.03 362.25 |
------------------+---------------+---------------+
songs | | |
Ratio: 51.35 | 50.27 49.59 | 50.27 49.58 |
EncoTime: 185.51 | 86.14 591.69 | 50.76 241.82 |
------------------+---------------+---------------+
Can -12 in combination with -v1 be so much faster now?
Can -12 in combination with -v1 be so much faster now?
Yes indeed. Level 12 makes more use of 2 functions which were optimized since version 0.10. The different binaries might be making some of that difference as well.
BTW...where might I find the "rw" corpus?
-Justin
Hi,
Flake r46 has a new variable block size algorithm, which is selected by using "-v 2". It gives a lower-bound for even 3-level splitting (splitting frames exactly in half up to 3 times). Encoding takes about 4x as long. Hopefully I will be able to use this to help improve -v 1 by comparing the resulting files. Future developments might also include uneven splitting, which could better isolate changes within a frame with less overhead.
-Justin
Edit: In r47 I modified -v 1 to give better compression.
Justin Ruggles> BTW...where might I find the "rw" corpus?
here (http://www.rjamorim.com/test/samples/).
Compiles using current SVN (rev 48): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_r48_cygwin.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_r48_mingw.7z)
I have added Flake SVN Rev.42 (wisodev build) to my Yalac comparison. Please see this post on the Yalac thread (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=43494&view=findpost&p=434373) for info.
Here's the FLAC-related results in compression order:
Encoder Setting Compression Encode Decode
==================================================
Flake -12 -v 1 65.259% 7.29x 70.41x
Flake -12 65.368% 7.38x 69.31x
Flake -11 65.376% 19.10x 69.19x
FLAC 1.1.2_CVS -9 65.528% 5.73x 71.34x
Flake -8 -v 1 65.530% 37.94x 74.49x
Flake -10 65.555% 24.82x 73.63x
Flake -9 65.559% 35.48x 75.07x
FLAC 1.1.2_CVS -8 65.611% 9.25x 70.96x
Flake -8 65.658% 41.21x 74.93x
Flake -5 -v 1 65.668% 64.70x 74.89x
Flake -5 65.798% 69.44x 75.44x
FLAC 1.1.2_CVS -5 65.916% 36.75x 71.46x
FLAC -8 66.028% 9.43x 71.42x
FLAC -5 66.279% 38.44x 72.05x
FLAC 1.1.2_CVS -0 70.734% 69.34x 72.35x
FLAC -0 70.734% 68.97x 75.84x
Flake -0 71.679% 82.18x 71.13x
Flake -0 -v 1 71.728% 76.70x 72.80x
FLAKE SVN REVISION 48
Download binarys:
flake-svn-r48-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r48-win32-bin.zip)
Download sources:
flake-svn-r48-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r48-win32-src.zip)
wiso
Justin. I pulled revision 2 from SVN, and tried to build this on Mac OS 10.47 PowerPC/ GCC 4.0.1 (Xcode Tools 2.4). But it errors out like this:$ ./configure
source path /Users/krmathis/Downloads/flake-enc
C compiler cc
make make
CPU powerpc (generic)
big-endian yes
inttypes.h yes
AltiVec enabled yes
debug symbols yes
strip symbols yes
optimize yes
Creating config.mak and config.h...
$ make
make -C libflake all
** Snip **
cc -O3 -no-cpp-precomp -pipe -force_cpusubtype_ALL -Wno-sign-compare -fomit-frame-pointer -faltivec -g -Wdeclaration-after-statement -Wall -Wno-switch -mdynamic-no-pic -DHAVE_CONFIG_H -I. -I.. -I/Users/krmathis/Downloads/flake-enc/libflake -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -c -o wav.o wav.c
cc -L../libflake -Wl,-dynamic,-search_paths_first -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2
I successfully compiled flake 0.7 two days ago. But the build system have changed since then (from "make" to "./configure && make")
Any idea whats wrong?
B.U.M.P.
Revision 48 still fails with the same error...
Encoder Setting Compression Encode Decode
==================================================
Flake -0 71.679% 82.18x 71.13x
I think flac's decoder is quite inefficient... With encoding speed so vastly improved decoding speed may start to hold it back in the great league of lossless formats.
I really like what I see about Flake develpment, great work! However, I don't know if these SVN builds are safe to use and won't produce "mistakes" with some files. Is there a simple and fast way to verify that the produced files will decode bit-identical and without errors? The official Flac encoder has a verify option to check this IIRC.
Encoder Setting Compression Encode Decode
==================================================
Flake -0 71.679% 82.18x 71.13x
I think flac's decoder is quite inefficient... With encoding speed so vastly improved decoding speed may start to hold it back in the great league of lossless formats.
Am i right? You are wondering, why decoding isn't faster than encoding?
I have to go a bit technical, to explain it.
To simplify it:
The encoder has to perform tree operations:
A) Analyze the signal and select an optimal prediction method.
B) Prediction: Predict the next sample and calculate the difference (residual) between the prediction and the true sample.
C) BitIO: Write the residual into the bitstream (file).
The decoder has to perform two operations:
A) BitIO: Read the residual from the bitstream (file).
B) Prediction: Predict the next sample and add the residual from (A) to get the true sample value.
Differences between encoding and decoding:
For Flake -0, signal analysis (A) of the encoder is very simple and fast.
The calculations encoder and decoder have to perform for the prediction of the fast -0 mode are absolutely simple, can be performed ultra fast and should be equally fast on the encoder and decoder side.
But the difference of the processing requirements for the prediction in encoder and decoder is quite small compared to the difference for the BitIO!
One important operation, processors have to perform, is a conditional jump. It is needed, if -depending on a certain condition- two different actions have to be performed.
But modern processors like to know as soon as possible what will be going on later. In the case of a conditional jump, they had to wait until the critical condition has been calculated. Because they don't want to wait, they try to predict the result of the condition based upon their knowledge about prior excecutions of the
conditional jump.
This works quite well, if the condition follows a simple (repetitive) pattern, for instance: The condition is 5 times true, than 1 time false, than 5 times true. Then the conditional jump will be called predictable.
The worst case is a conditional jump, which is following an unpredictable pattern. Modern processors may have to wait 20 and more clock cycle, if they have performed a misprediction!
Compare this to the about 0.5 clock cycles needed for simple operations like additions or substractions!
What is important: The BitIO of the encoder needs far less conditional jumps (if well implemented) than the decoder, and the jumps on the decoder side will be unpredictable most of the time.
A simplified example:
On average 1 mispredictions per sample on the encoder side = 20 clock cycles
On average 3 mispredictions per sample on the dncoder side = 60 clock cycles
And possibly 5 simple cpu operations for the prediction: 5 * 0.5 = 2.5 clock cycles.
Sorry, my english is too limited for a good explaination. I hope, you can understand.
MedO: Use FlacTester (http://www.vuplayer.com/files/flactester.zip). Just give it a directory and it will recursively work it's way through it and all subdirs testing all flacs it finds. Ideal for keeping your flac collection in proper working order.
TBeck: thanks for the explanation, i understand. Indeed it seemingly goes against nature to have faster encoding than decoding..
It would also mean the difference should be even greater for P4 processors.
MedO: Use FlacTester (http://www.vuplayer.com/files/flactester.zip). Just give it a directory and it will recursively work it's way through it and all subdirs testing all flacs it finds. Ideal for keeping your flac collection in proper working order.
Thanks for the tip. So I can rely on Flake producing correct checksums? Just want to be sure :-)
It's a md5 hash, if flake manages to produce a corrupt hash that fits the corrupt output, i'd be very impressed.
Justin. I pulled revision 2 from SVN, and tried to build this on Mac OS 10.47 PowerPC/ GCC 4.0.1 (Xcode Tools 2.4). But it errors out like this:$ ./configure
source path /Users/krmathis/Downloads/flake-enc
C compiler cc
make make
CPU powerpc (generic)
big-endian yes
inttypes.h yes
AltiVec enabled yes
debug symbols yes
strip symbols yes
optimize yes
Creating config.mak and config.h...
$ make
make -C libflake all
** Snip **
cc -O3 -no-cpp-precomp -pipe -force_cpusubtype_ALL -Wno-sign-compare -fomit-frame-pointer -faltivec -g -Wdeclaration-after-statement -Wall -Wno-switch -mdynamic-no-pic -DHAVE_CONFIG_H -I. -I.. -I/Users/krmathis/Downloads/flake-enc/libflake -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -c -o wav.o wav.c
cc -L../libflake -Wl,-dynamic,-search_paths_first -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2
Sorry...I did not see this originally. It looks like it might have to do with byteswap.h. I'll look into it.
Sorry...I did not see this originally. It looks like it might have to do with byteswap.h. I'll look into it.
Thank you!
It's a md5 hash, if flake manages to produce a corrupt hash that fits the corrupt output, i'd be very impressed.
md5 is only useful as an error-detection mechanism after a verified-good encoding. there are many ways the encoder can mess up the data going in and still create a file with a correct md5. that's why there's a --verify mode in flac, and why the flac test suite uses --verify as well as encodes/decodes/compares many thousands of different streams/encoding combinations before a release.
Josh
It's a md5 hash, if flake manages to produce a corrupt hash that fits the corrupt output, i'd be very impressed.
md5 is only useful as an error-detection mechanism after a verified-good encoding. there are many ways the encoder can mess up the data going in and still create a file with a correct md5. that's why there's a --verify mode in flac, and why the flac test suite uses --verify as well as encodes/decodes/compares many thousands of different streams/encoding combinations before a release.
Josh
I definitely agree. It's not nearly as hard as it may seem to produce a false MD5 which matches decoded output. This can happen if the wav data gets corrupted (by either program bugs or by machine-related errors) anywhere between the wav file read and the MD5 update.
If I ever have the time and there is a demand for it, I might import FFmpeg's FLAC decoder into Flake for verification during encoding. For now I'll add it to the TODO list.
-Justin
HbG,
I think flac's decoder is quite inefficient... With encoding speed so vastly improved decoding speed may start to hold it back in the great league of lossless formats.
You could be right.
Some time ago I did some tests with flacdec (http://phoenix.inf.upol.cz/~lichvarm/affd/), and despite running it under cygwin, it was faster than flac -d. Since I don't have the necessary tools to build an optimized win32-compilation, I didn't test further...
[edit]
Here's my test results (12 flac files, encoded with flac -8, totalling 447.635.340 bytes) for decoding to nul:
a) flac -d -o nul [file]: 34,31 sec (avg.)
b) flacdec -o nul [file]: 12,21 sec (avg.)
Although flacdec doesn't "check MD5 signature of audio data", I don't think that's the culprit...
.sundance.
I definitely agree. It's not nearly as hard as it may seem to produce a false MD5 which matches decoded output. This can happen if the wav data gets corrupted (by either program bugs or by machine-related errors) anywhere between the wav file read and the MD5 update.
If I ever have the time and there is a demand for it, I might import FFmpeg's FLAC decoder into Flake for verification during encoding. For now I'll add it to the TODO list.
-Justin
So the only way I can be sure is checking the decoded audio against the original? I guess I could write a script to do this, but that would probably end any speed gain I got with Flake.
Whoops, i stand corrected on the MD5's. I didn't think of the wav data getting corrupted before being hashed, although technically that does not indicate a fault in the encoding routines.
Sundance, you're right, i had even used flacdec before and forgot about it. As it's not a reference decoder i'm not using it in combination with flake. Still a better compile would be nice to see just what it can do, it does decode variable blocksize properly and without measurable speed penalty for the sample i used.
I did a little benchmark with a flake -8 encoded file, flac -d gives me 151.88x, foobar 180.23x, flacdec 265.60x
Sundance's flacdec-to-flac ratio is 2.81, mine is 1.75, there is something strange going on here.
Flacdec generates an output stream with a bandwidth of over 45MB/s. I couldn't find a suitable md5 benchmark for windows quickly but i did find this (http://etud.epita.fr/%7Ebevand_m/papers/md5-amd64.html). Looks like the impact of hashing on overal performance is significant, but nowhere near enough to explain the whole difference.
MedO, that's the tradeoff of new software, it needs testing.
Sundance's flacdec-to-flac ratio is 2.81, mine is 1.75, there is something strange going on here.
I tested on a P4@2.8 GHz using that "script" along with 4NT (enhanced command shell).
All files resided on a fast local hard drive; the output went to nul: device.
timer
for %f in (*.flac) flac -f -d -o nul "%f"
timer
timer
for %f in (*.flac) flacdec -o nul "%f"
timer
Result:
Timer 1 on: 15:12:02
Timer 1 off: 15:12:37 Elapsed: 0:00:35,53
Timer 1 on: 15:12:37
Timer 1 off: 15:12:50 Elapsed: 0:00:12,91
I ran this test 5 times to eleminate disc caching related timing differences and didn't use the best and worst result to calculate the average time needed. Would be interested if you acually created wav files or if also redirected to nul...
.sundance.
P.S. Hope this discussion isn't to much OT...
MedO, that's the tradeoff of new software, it needs testing.
I wrote a small wrapper script using flake, md5sum and flac (for decoding). I can post it here if anyone's interested, but only after I improved it a bit, since this was only for quick testing. Btw, this is still a good deal faster than using the official encoder when both run at -8.
I tested on a P4@2.8 GHz using that "script" along with 4NT (enhanced command shell).
All files resided on a fast local hard drive; the output went to nul: device.
timer
for %f in (*.flac) flac -f -d -o nul "%f"
timer
timer
for %f in (*.flac) flacdec -o nul "%f"
timer
I don't know what "-o nul" means to flacdec but to flac that means output to a file called "nul" so you are counting the i/o time, which is probably half the total time. if you want to eliminate that, use -t instead of -d.
also, flac decoding is fast enough that md5 checking does take a significant amount of time. but it would still add only 10-20% I think, not double. you can't turn off md5 checking in the command line decoder (unless you use metaflac to set the md5 sum of the file to all zeroes)
the flac binary for windows that I release is compiled with msvc6 which might generate slower code. but it has a lot of assembly optimization, so I think this 1.7-2.8x difference is because of something else (like file i/o) not comparing apples to apples.
Josh
Here's my test results (12 flac files, encoded with flac -8, totalling 447.635.340 bytes) for decoding to nul:
a) flac -d -o nul [file]: 34,31 sec (avg.)
b) flacdec -o nul [file]: 12,21 sec (avg.)
Although flacdec doesn't "check MD5 signature of audio data", I don't think that's the culprit...
In my tests, the MD5 checking accounts for about 25% of the decoding time in flac (using flac -t).
I think NUL is the DOS/Windows equivalent of /dev/null.
Josh,
sorry for not comparing objectively.
I assumed that flac also used the nul: device in my commandline since no file was actually generated. But that was of course a wrong assumption since you can't create such a file in dos/win/win32. I'll try to repeat my little "test" with flac -t as you've suggested when I'm back home at my computer.
And please don't get me wrong: Nagging about flac is the least I want to do - I've been a long time flac user and lover and because of that I'm really excited with all the progress of the flac format.
.sundance.
Sorry...I did not see this originally. It looks like it might have to do with byteswap.h. I'll look into it.
Thank you!
Justin. Can you please let me know when you think you have fixed this?
Josh,
sorry for not comparing objectively.
no problem!
I assumed that flac also used the nul: device in my commandline since no file was actually generated. But that was of course a wrong assumption since you can't create such a file in dos/win/win32. I'll try to repeat my little "test" with flac -t as you've suggested when I'm back home at my computer.
hmm... if it didn't write a file, maybe windows C lib is interpreting nul on its own. with nul it might still count time copying the output data around, not sure. the comparison also depends on how exactly flacdec handles '-o nul'
And please don't get me wrong: Nagging about flac is the least I want to do - I've been a long time flac user and lover and because of that I'm really excited with all the progress of the flac format.
no, that's fine, it's just that I really don't know how another decoder could be 2x faster than the reference, that would be a huge improvement since the reference decoder is already optimized. a faster encoder is easy to believe.
Josh
Sorry...I did not see this originally. It looks like it might have to do with byteswap.h. I'll look into it.
Thank you!
Justin. Can you please let me know when you think you have fixed this?
Well, I don't have a mac to test on, so I'm kinda at a loss. I did change some of the configuration options, but I doubt they will help. I'm wondering if maybe bswap_32 is being redefined somewhere... Try the patch below. If it doesn't fix the problem, at least it might point me in the right direction.
-Justin
Index: flake/wav.c
===================================================================
--- flake/wav.c (revision 46)
+++ flake/wav.c (working copy)
@@ -537,7 +537,7 @@
#ifdef WORDS_BIG_ENDIAN
uint16_t *buf16 = (uint16_t *)buffer;
for(i=0; i<nsmp; i++) {
- buf16[i] = bswap_16(buf16[i]);
+ buf16[i] = flake_bswap_16(buf16[i]);
}
#endif
if(wf->source_format != WAV_SAMPLE_FMT_S16) return -1;
@@ -568,7 +568,7 @@
#ifdef WORDS_BIG_ENDIAN
uint32_t *buf32 = (uint32_t *)buffer;
for(i=0; i<nsmp; i++) {
- buf32[i] = bswap_32(buf32[i]);
+ buf32[i] = flake_bswap_32(buf32[i]);
}
#endif
if(wf->source_format != WAV_SAMPLE_FMT_FLT) return -1;
@@ -591,7 +591,7 @@
#ifdef WORDS_BIG_ENDIAN
uint64_t *buf64 = (uint64_t *)buffer;
for(i=0; i<nsmp; i++) {
- buf64[i] = bswap_64(buf64[i]);
+ buf64[i] = flake_bswap_64(buf64[i]);
}
#endif
if(wf->source_format != WAV_SAMPLE_FMT_DBL) return -1;
Index: bswap.h
===================================================================
--- bswap.h (revision 54)
+++ bswap.h (working copy)
@@ -8,24 +8,24 @@
#include "config.h"
-static inline uint16_t bswap_16(uint16_t x){
+static inline uint16_t flake_bswap_16(uint16_t x){
return (x>>8) | (x<<8);
}
-static inline uint32_t bswap_32(uint32_t x){
+static inline uint32_t flake_bswap_32(uint32_t x){
x= ((x<<8)&0xFF00FF00) | ((x>>8)&0x00FF00FF);
return (x>>16) | (x<<16);
}
-static inline uint64_t bswap_64(uint64_t x)
+static inline uint64_t flake_bswap_64(uint64_t x)
{
union {
uint64_t ll;
uint32_t l[2];
} w, r;
w.ll = x;
- r.l[0] = bswap_32(w.l[1]);
- r.l[1] = bswap_32(w.l[0]);
+ r.l[0] = flake_bswap_32(w.l[1]);
+ r.l[1] = flake_bswap_32(w.l[0]);
return r.ll;
}
@@ -36,13 +36,13 @@
#define be2me_16(x) (x)
#define be2me_32(x) (x)
#define be2me_64(x) (x)
-#define le2me_16(x) bswap_16(x)
-#define le2me_32(x) bswap_32(x)
-#define le2me_64(x) bswap_64(x)
+#define le2me_16(x) flake_bswap_16(x)
+#define le2me_32(x) flake_bswap_32(x)
+#define le2me_64(x) flake_bswap_64(x)
#else
-#define be2me_16(x) bswap_16(x)
-#define be2me_32(x) bswap_32(x)
-#define be2me_64(x) bswap_64(x)
+#define be2me_16(x) flake_bswap_16(x)
+#define be2me_32(x) flake_bswap_32(x)
+#define be2me_64(x) flake_bswap_64(x)
#define le2me_16(x) (x)
#define le2me_32(x) (x)
#define le2me_64(x) (x)
Well, I don't have a mac to test on, so I'm kinda at a loss. I did change some of the configuration options, but I doubt they will help. I'm wondering if maybe bswap_32 is being redefined somewhere... Try the patch below. If it doesn't fix the problem, at least it might point me in the right direction.
-Justin
I understand that you have no way to test this, since you don't have access to a Mac. But I am more than willing to perform the tests needed. If that helps?
Ok, I had to manually edit the files according to the patch you listed. Because I could not get patch to recognize the patch file I created with nano.
Revision 61 with the patched files failed just like before:
cc -L/Users/krmathis/.Trash/flake-enc/libflake -Wl,-dynamic,-search paths_first -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2
The problem seems to be line #277 in flake-enc/libflake/md5.c
Cause if I change "bswap_16" to "flake_bswap_16" the error message change according to it.
Edit: I removed the above mentioned line, and it compiled successfully!
- sig16[i] = bswap_16(sig16[i]);
It launches, but I don't know how to test if it works as expected...
Edit 2:
It compressed a wav file successfully. But the result can't be right!
40801624 Sep 25 20:33 01 - Speak to Me - Breathe In the Air.flac
22205524 Sep 23 14:27 01 - Speak to Me - Breathe In the Air.m4a
41952016 Sep 25 20:32 01 - Speak to Me - Breathe In the Air.wav
The resulting FLAC file was only compressed to 97% of the original one. While the ALAC (.m4a) file compressed to ~60%.
Flake: FLAC audio encoder
© 2006 Justin Ruggles
Signed 16-bit 44100 Hz stereo
samples: 10486980
block size: 4608
variable: none
prediction type: levinson-durbin
prediction order: 1,8
partition order: 0,6
order method: estimate
stereo method: mid-side
header padding: 4096
progress: 100% | ratio: 0.973 | bitrate: 1372.5 kbps | bytes: 40801624
Was this caused by the line I edited out?
The problem seems to be line #277 in flake-enc/libflake/md5.c
Cause if I change "bswap_16" to "flake_bswap_16" the error message change according to it.
Fixed. I forgot to #include "bswap.h" at the top of md5.c.
-Justin
It compressed a wav file successfully. But the result can't be right!
40801624 Sep 25 20:33 01 - Speak to Me - Breathe In the Air.flac
22205524 Sep 23 14:27 01 - Speak to Me - Breathe In the Air.m4a
41952016 Sep 25 20:32 01 - Speak to Me - Breathe In the Air.wav
The resulting FLAC file was only compressed to 97% of the original one. While the ALAC (.m4a) file compressed to ~60%.
...
Was this caused by the line I edited out?
I hope so...but I sort of doubt it. Play back the flac file. I suspect it's all noise. If you try rev67 and still have the problem please speak up and I'll investigate.
The problem seems to be line #277 in flake-enc/libflake/md5.c
Cause if I change "bswap_16" to "flake_bswap_16" the error message change according to it.
Fixed. I forgot to #include "bswap.h" at the top of md5.c.
-Justin
Success!
There is still something wrong, cause the compression rate
really sucks.
Flake: "ratio: 0.973 | bitrate: 1372.5 kbps | bytes: 40801624 "
FLAC 1.1.2: "22381217 bytes, ratio=0.534"
Same big/little-endian problem as we had earlier (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=403140) maybe?
Edit: Saw you reply after posting this one.
1. Using revision 67
2. The resulting FLAC file is just noice.
Same big/little-endian problem as we had earlier (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=403140) maybe?
Edit: Saw you reply after posting this one.
1. Using revision 67
2. The resulting FLAC file is just noice.
Yes indeed. It should be fixed now in r69.
Same big/little-endian problem as we had earlier (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=403140) maybe?
Edit: Saw you reply after posting this one.
1. Using revision 67
2. The resulting FLAC file is just noice.
Yes indeed. It should be fixed now in r69.
Sadly it is not.
Revision 69 still output garbled data at a compression ratio of 0.973 (I removed all source traces and pulled a fresh one).
Thanks a lot!
Same big/little-endian problem as we had earlier (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=403140) maybe?
Edit: Saw you reply after posting this one.
1. Using revision 67
2. The resulting FLAC file is just noice.
Yes indeed. It should be fixed now in r69.
Sadly it is not.
Revision 69 still output garbled data at a compression ratio of 0.973 (I removed all source traces and pulled a fresh one).
Thanks a lot!
Very odd. I can't figure it out. Try hand-editing config.h and remove the line with #define WORDS_BIGENDIAN 1. And/or put a printf line right before the call to bswap_16 in wav.c. Those won't fix anything, but might rule out some things.
[edit]changed the instructions slightly
[edit2] btw...i should have sourceforge compile farm access in the next 24-hrs. that way i can test on other platforms.
As usual the fix was too simple for me to notice right away...just a spelling error. *sigh* I have fixed the problem and tested on a ppc (yay for compile farm). Thanks for finding the problem so that I could fix it.
-Justin
As usual the fix was too simple for me to notice right away...just a spelling error. *sigh*
You certainly nailed it!
Flake rev69: ratio: 0.973 | bitrate: 1372.5 kbps | bytes: 40801624
Flake rev77: ratio: 0.527 | bitrate: 743.6 kbps | bytes: 22107579
FLAC 1.1.2: 22381217 bytes, ratio=0.534
Thanks for finding the problem so that I could fix it.
No problem!
..and thank you for fixing the bugs I have found.
** now on to testing this "beast" **
I just found this thread today and have skimmed thru it. I was happy to see that flac is being put into ffmpeg. I just compiled the latest ffmpeg source from svn to test this out. It seemed to encode fine but playing back the file with xmms does not display the length of the track and will not let you skip ahead. My flac and flake encoded files work fine in xmms though. I just wanted to see if that was a known problem or maybe I encoded wrong. I just used the example posted in this thread:
ffmpeg -i test.wav test.flac
i packaged a svn snapshot for debian
available at the usual place
later
i got a question about the code being used in flake. when i tried to compile flake with Microsoft Visual C++, i noticed that flake uses data types such as int32_7 and uint63_t. i also read on the wikipedia about inttypes.h and found out that they have equivalents which are built in the C++ language itself.
so my question is, are there any main diffrences between, let's say, "int32_t" and "long"?
edit: reason that i'm asking is because the Platform SDK does not give an inttypes.h file.
i got a question about the code being used in flake. when i tried to compile flake with Microsoft Visual C++, i noticed that flake uses data types such as int32_7 and uint63_t. i also read on the wikipedia about inttypes.h and found out that they have equivalents which are built in the C++ language itself.
so my question is, are there any main diffrences between, let's say, "int32_t" and "long"?
edit: reason that i'm asking is because the Platform SDK does not give an inttypes.h file.
The main difference has to do with 64-bit platforms. Rather than explain it, maybe a snippet from glibc's stdint.h (which is included by inttypes.h) would help. inttypes.h is part of the ISO C99 standard, which is the baseline standard used in Flake.
#ifndef __int8_t_defined
# define __int8_t_defined
typedef signed char int8_t;
typedef short int int16_t;
typedef int int32_t;
# if __WORDSIZE == 64
typedef long int int64_t;
# else
__extension__
typedef long long int int64_t;
# endif
#endif
/* Unsigned. */
typedef unsigned char uint8_t;
typedef unsigned short int uint16_t;
#ifndef __uint32_t_defined
typedef unsigned int uint32_t;
# define __uint32_t_defined
#endif
#if __WORDSIZE == 64
typedef unsigned long int uint64_t;
#else
__extension__
typedef unsigned long long int uint64_t;
#endif
For MSVC, you can try adding this bit of code somewhere. I will add it myself in the near future...I've been meaning to, but just keep putting it off.
#if defined(_WIN32) && defined(_MSC_VER)
typedef __int8 int8_t;
typedef unsigned __int8 uint8_t;
typedef __int16 int16_t;
typedef unsigned __int16 uint16_t;
typedef __int32 int32_t;
typedef unsigned __int32 uint32_t;
typedef __int64 int64_t;
typedef unsigned __int64 uint64_t;
#endif
If you need me to explain more I can.
-Justin
edit: added the msvc stuff
from looking at the code, i can only guess that on 32-bit platforms, int64_t and uint64_t must be defined diffrently in order to work.
also thanks for that piece of code that you put there. helps a lot. i'm don't have much experiance in c++ so that code helps.
I have builded latest Flake (Win32) from SVN revision 95:
Download from my Website (http://www.thefrontend.net/flake/)
or use direct links:
Download binarys:
flake-svn-r95-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r95-win32-bin.zip)
Download sources:
flake-svn-r95-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r95-win32-src.zip)
wisodev
Thanks wisodev!
What are the general changes from r48?
What are the general changes from r48?
Well Flake is in constant development so I decided to build fresh binary from latest SVN sources. General changes (in current SVN version) are listed in Flake Changelog file included in src archive.
wisodev: which compiler do you use? MinGW?
wisodev: which compiler do you use? MinGW?
I am using Intel C++ compile at version 8.0.40 and C99 switch (this is req.)
But Flake compiles fine with MinGW. I just tried it and, after getting gcc 4.1 to run, it's just as fast as wisodev's build for me.
i tried out one of wisodev's builds and his seem to be a few seconds faster at mode -12 for me. the other executable that i tested was one compiled by me.
i tried out one of wisodev's builds and his seem to be a few seconds faster at mode -12 for me. the other executable that i tested was one compiled by me.
Here's my build:
Flake r100 mingw-build (http://www.uni-koblenz.de/~smaxein/flake.zip)
Only the "naked" binary, though.
Compiler Flags: -s -O3 -fexpensive-optimizations -march=i686
Compiler Version: gcc 4.1.1
bah. now i can't get rev95 to work. it keeps saying "error parsing filenames" when i try to encode it.
"flake -12 C:\test\spirit.wav C:\test\spirit.flac" is the command line that i use.
edit: interesting. now i don't need to even specify an output file. pretty interesting...
But Flake compiles fine with MinGW. I just tried it and, after getting gcc 4.1 to run, it's just as fast as wisodev's build for me.
I have not done any gcc4 testing, much less under MinGW. Were there any compile issues that I may need to solve or did everything build smoothly? Since the build system is hand-made, not autotools, I've been trying to put it through more tests.
Thanks,
Justin
i just did a short test with 3 encoders. the first was my own compiled one, the second was MedO's one, and the third was wisodev's compiler.
the results sucked because my compile took the longest to encode while MedO's compile took 61 seconds and wisodev's took 64 seconds. mine took 65.
bah. now i can't get rev95 to work. it keeps saying "error parsing filenames" when i try to encode it.
"flake -12 C:\test\spirit.wav C:\test\spirit.flac" is the command line that i use.
edit: interesting. now i don't need to even specify an output file. pretty interesting...
Correct. Now you don't have to specify the output if you want the same base name, but if you want a specific output file, you have to now use "-o output.flac". The order method switch was changed to "-m". I'm working on a new option system right now though, so things may change again before the next full release. I hope to add multiple file input as well. Everything will be documented better for the release...the current svn version is undergoing pretty constant changes.
edit: typo
edit2: another typo...
heh. i just tried out rev95 again with foobar2000's Converter and the thing blew up on me. the command line that i used was "-12 - -o %d" and when i tried it, when it finished the first or second tracked, foobar2000 gave an error saying "Error flushing file". and also, the output file was "filename.flac=". i am not even sure why there was an = at the end.
edit: why the hell do i always mix up line with like...
heh. i just tried out rev95 again with foobar2000's Converter and the thing blew up on me. the command line that i used was "-12 - -o %d" and when i tried it, when it finished the first or second tracked, foobar2000 gave an error saying "Error flushing file". and also, the output file was "filename.flac=". i am not even sure why there was an = at the end.
edit: why the hell do i always mix up line with like...
I think it was a bug in my code. Try rev96.
edit: a little explaining. I had forgotten that you have to include the trailing '\0' in the character count for strncpy.
yep. looks like it's fixed. thank you
Flake r100 mingw-build (http://www.uni-koblenz.de/~smaxein/flake.zip)
Same location as last time.
Building was quite smooth, but I didn't use the makefile (don't know if I even could, I'm quite new to this). I just started CodeBlocks, added all needed files for the library to a project, set the compiler paths so it could find bswap.h and added an empty config.h. Setting up the project for flake was almost the same.
There was only one warning when building:
flake\wav.c:473: warning: enumeration value 'WAV_SAMPLE_FMT_UNKNOWN' not handled in switch
That was already present with the gcc 3 I used before (can't remember the exact version)
I haven't been able to compile using MinGW since rev 48. If I run ./configure I get:
/bin/cat: /tmp/flake-conf-10500-6548-17128.c: No such file or directory
gcc is unable to create an executable file.
It still works with cygwin.
I don't need to compile, but I thought it worth pointing out, as it seems relevant at the moment.
no problem here with mingw compile...Maybe you need to update mingw-runtime.
It was only just over a week ago that it did work. If I choose "Update" MinGW only allows me to install new apps, and not update current apps. I've no idea where or how to update.
As I say, no big deal, I was just curious why the change.
Does flake use autotools? If yes, would a ./autogen.sh generate new ./configure* files?
I just downloaded Rev. 19 of configure (http://svn.sourceforge.net/viewvc/*checkout*/flake-enc/configure?revision=19) and that works fine. Rev. 50 (http://svn.sourceforge.net/viewvc/*checkout*/flake-enc/configure?revision=50) (the next version) does not.
$ gcc -v
Reading specs from c:/mingw/bin/../lib/gcc/mingw32/3.4.2/specs
Configured with: ../gcc/configure --with-gcc --with-gnu-ld --with-gnu-as --host=mingw32 --target=mingw32 --prefix=/mingw --enable-threads --disable-nls --enable-languages=c,c++,f77,ada,objc,java --disable-win32-registry --disable-shared --enable-sjlj-exceptions --enable-libgcj --disable-java-awt --without-x --enable-java-gc=boehm --disable-libgcj-debug --enable-interpreter --enable-hash-synchronization --enable-libstdcxx-debug
Thread model: win32
gcc version 3.4.2 (mingw-special)
Flake r100 mingw-build (http://www.uni-koblenz.de/%7Esmaxein/flake.zip)
Same location as last time.
Building was quite smooth, but I didn't use the makefile (don't know if I even could, I'm quite new to this). I just started CodeBlocks, added all needed files for the library to a project, set the compiler paths so it could find bswap.h and added an empty config.h. Setting up the project for flake was almost the same.
There was only one warning when building:
flake\wav.c:473: warning: enumeration value 'WAV_SAMPLE_FMT_UNKNOWN' not handled in switch
That was already present with the gcc 3 I used before (can't remember the exact version)
Hmm. Are you not using ./configure + make because it doesn't work? If you have msys installed that's all that you should need to do.
I haven't been able to compile using MinGW since rev 48. If I run ./configure I get:
/bin/cat: /tmp/flake-conf-10500-6548-17128.c: No such file or directory
gcc is unable to create an executable file.
It still works with cygwin.
I don't need to compile, but I thought it worth pointing out, as it seems relevant at the moment.
That is odd. Can you compile/run a simple hello world program with your MinGW version?
edit: looks like I was too slow. I'm glad it works for you now.
Does flake use autotools? If yes, would a ./autogen.sh generate new ./configure* files?
No, I did not use autotools for Flake, nor do it ever intend to. I may decide to use some other build system, but it won't be autotools. make distclean + ./configure + make is all that should be necessary under MinGW.
-Justin
edit: looks like I was too slow. I'm glad it works for you now.
Well, rev.19 of configure does, but none since, and that's not really much use is it?
I dunno. I don't really care! I just thought it may be of interest. It seems to me that something changed in version 50 that breaks configure with my versions (work and home) of MinGW (installed using MinGW 5.0.3 and MSYS 1.0.11). I would assume that this would be the same for others, but if not then, please, forget it. I've been expecting someone else to point this out, but as MedO was releasing versions I thought it was worth piping up.
If it's not an issue for anyone else then please don't pursue it.
I just downloaded Rev. 19 of configure (http://svn.sourceforge.net/viewvc/*checkout*/flake-enc/configure?revision=19) and that works fine. Rev. 50 (http://svn.sourceforge.net/viewvc/*checkout*/flake-enc/configure?revision=50) (the next version) does not.
Could you post or PM me the config.err you get using rev50? This would help me narrow-down which change caused the issue. That was quite a big change from r19 to r50. In retrospect, I probably should have broken it up into smaller commits.
The fact that it doesn't work does concern me, and it may be a simple fix. If I can't figure it out from your config.err output then I'll maybe give up on it.
edit: typo. and I was too slow again...added response to last post.
Certainly. [attachment=2576:attachment]
Hmm. Are you not using ./configure + make because it doesn't work? If you have msys installed that's all that you should need to do.
I didn't install msys, no. I have never used it yet. I know ./configure -scripts from building Linux apps, but I never wrote one or used one under Windows. Hmm, if I have no problems compiling Flake without running ./configure first, does CodeBlocks do the necessary stuff that ./configure normally does?
As I said, I'm quite new to this.
I received an email this afternoon (I found it when I got home) from a new user here called
Mitch 1 2. He is waiting for his five day suspension to end before posting, but sent me this:
I had the same problem. Get this:
http://optusnet.dl.sourceforge.net/sourcef...napshot.tar.bz2 (http://optusnet.dl.sourceforge.net/sourceforge/mingw/bash-3.1-MSYS-1.0.11-snapshot.tar.bz2)
and put the binaries in your msys\1.0\bin directory.
His suggestion worked; I have successfully compiled rev. 101.
So, I assume it was to do with bash not being available?
most probably an older version...bourne shell 2.4 doesn't work anymore with some apps.
Hmm. Are you not using ./configure + make because it doesn't work? If you have msys installed that's all that you should need to do.
I didn't install msys, no. I have never used it yet. I know ./configure -scripts from building Linux apps, but I never wrote one or used one under Windows. Hmm, if I have no problems compiling Flake without running ./configure first, does CodeBlocks do the necessary stuff that ./configure normally does?
As I said, I'm quite new to this.
For building binaries only, in Windows, with gcc/mingw...yes CodeBlocks probably does all that is necessary. For building a static library or building on other platforms, the configure script is useful because it tries to automatically detect information that is needed to compile and install. It can also detect system features so it can use compiler flags which will generate faster binaries. But, if you already know gcc well enough, you can just set the compiler flags yourself.
-Justin
Current SVN version now supports multiple input files. So you can encode a whole directory of wav files with 1 command. It's as simple as running something such as:
flake -8 *.wav
or
flake -12 file1.wav file2.wav file7.wav
This has not been tested in Windows, so I hope it works okay.
Now...no new features until the next release...for real this time. I'll work on documentation and testing (and my ac3 encoder) while I give Flake a little resting time for bugs to show themselves. The docs will take a bit of work, but at least it will be a nice change of pace.
Thanks,
-Justin
you have #include <inttypes.h> at wav.h . just though i should point that out. just tried building it again with VC++ 2005 and it blew up on syntax errors. i'll try to rebuild it in the future with VC++ 2005 when i get interested in doing that again.
you have #include <inttypes.h> at wav.h . just though i should point that out. just tried building it again with VC++ 2005 and it blew up on syntax errors. i'll try to rebuild it in the future with VC++ 2005 when i get interested in doing that again.
Thanks. fixed.
svn r104 at RareWares/Debian now
Windows x86 compiles using current SVN (rev 104): cygwin version (http://synthetic-soul.co.uk/temp/flake_svn_r104_cygwin.7z) | mingw version (http://synthetic-soul.co.uk/temp/flake_svn_r104_mingw.7z)
Just a minor thing:
There's a discrepancy concerning the command line syntax. When you invoke flake w/o parameters, it says:
Flake: FLAC audio encoder
© 2006 Justin Ruggles
usage: flake [options] <input.wav> [output.flac]
But when you add -h it's this:
Flake: FLAC audio encoder
© 2006 Justin Ruggles
usage: flake [options] <input.wav> [-o output.flac]
options:
[-h] Print out list of commandline options
[-p #] Padding bytes to put in header (default: 4096)
.
.
.
Guess '-o output.flac' is the one that's correct.
And still no version/revision number to display...
.sundance.
Ah, oh. I see a problem. Does -12 take that much processing power or isn´t it 100% compatible to flac 1.1.2 decoding anymore?
My Squeezebox can´t play flake -12 on all titles. It stutters around. Flake -5 for example is fine.
Edit:
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.
That has more similarities with the upcoming FLAC 1.1.3, or FLAC 1.1.2_CVS from this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=44229).
Ah, oh. I see a problem. Does -12 take that much processing power or isn´t it 100% compatible to flac 1.1.2 decoding anymore?
My Squeezebox can´t play flake -12 on all titles. It stutters around. Flake -5 for example is fine.
After Bukem (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=433703)'s positive report, your one is the first negative one and is precisely what I feared.
My own decoding tests done with foobar2000 and my old Duron revealed that -11 and -12 are considerably slower than all other modes and than official/Garf's encoder. From memory, it's ~x48 for flake high profile and a bit more than x60 for all other settings/encoders.
As a consequence I expected from some devices with few 'power' headroom to have troubles decoding -q11/12 encodings.
Could you also try with -q10 and -q11? I guess that the first one should be problem free and the latter to sutter as well as -q12.
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.
That has more similarities with the upcoming FLAC 1.1.3, or FLAC 1.1.2_CVS from this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=44229).
Thanks. So flake behaves different. As i read over the thread it was like the new ideas of 1.1.3 were in flake already and nothing more.
@Guru
i´ll do
After Bukem (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=45013&view=findpost&p=433703)'s positive report, your one is the first negative one and is precisely what I feared.
My own decoding tests done with foobar2000 and my old Duron revealed that -11 and -12 are considerably slower than all other modes and than official/Garf's encoder. From memory, it's ~x48 for flake high profile and a bit more than x60 for all other settings/encoders.
As a consequence I expected from some devices with few 'power' headroom to have troubles decoding -q11/12 encodings.
Could you also try with -q10 and -q11? I guess that the first one should be problem free and the latter to sutter as well as -q12.
Exactly how you predicted. -10 still is playable. From -11 on it stutters.
Thank you very much Wombat
May I suggest to Justin that the first official release of flake should come with a small warning (inside the documentation and/or inside flake.exe with -h switch) about possible hardware players' issues with -11 & -12 setting?
Exactly how you predicted. -10 still is playable. From -11 on it stutters.
To my little knoledge decoding speed of lossless codecs mostly depends on the maximum number of predictor order used.
Flake -5 uses 8 max predictor order, like Flac -5 (and Garf version of Flac).
Flake -8,-9 and -10 uses 12 max predictor order like Flac -8 (and Garf version of Flac).
Flake -11 and -12 uses 32, so it requires much more decoding power.
I hope that's correct.
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this. with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion. this goes for flac too which currently allows -l 32 without a warning.
most of the compression advantage of flake over FLAC 1.1.2 has already been incorporated into FLAC 1.1.3. the extra compression from 32-order LPC is minimal.
Josh
Flake (Win32) from SVN revision 104 available:
Download from my Website (http://www.thefrontend.net/flake/)
or use direct links:
Download binarys:
flake-svn-r104-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-bin.zip)
Download sources:
flake-svn-r104-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-src.zip)
NOTE: I have added my small patch to code (quality or speed is not affected), can be found here (http://www.thefrontend.net/flake/svn/flake_rev104_wiso.diff), the sources are already patched (this patch comes from my local subversion repository and is not related with sf.net subversion).
wisodev
Exactly how you predicted. -10 still is playable. From -11 on it stutters.
To my little knoledge decoding speed of lossless codecs mostly depends on the maximum number of predictor order used.
Flake -5 uses 8 max predictor order, like Flac -5 (and Garf version of Flac).
Flake -8,-9 and -10 uses 12 max predictor order like Flac -8 (and Garf version of Flac).
Though seeking with a flake encoded file is still significantly slower than with a FLAC (reference encoder) encoded file (try it in winamp with the default flac plugin).
Thank you very much Wombat
May I suggest to Justin that the first official release of flake should come with a small warning (inside the documentation and/or inside flake.exe with -h switch) about possible hardware players' issues with -11 & -12 setting?
Okay, that sounds like a good idea.
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this. with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion. this goes for flac too which currently allows -l 32 without a warning.
Cool. Based on this, and the hardware decoding issues, I'll go ahead and add a separate warning for any lpc order over 12 for any sample rate up to 48kHz. Once the official subset standard changes, I'll change Flake's warnings accordingly.
-Justin
Ah, oh. I see a problem. Does -12 take that much processing power or isn´t it 100% compatible to flac 1.1.2 decoding anymore?
My Squeezebox can´t play flake -12 on all titles. It stutters around. Flake -5 for example is fine.
Edit:
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.
Using the current version of FLAKE, I was able to play -12 compressed files on my Rio Karma with no hiccups.
Jon
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this. with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion. this goes for flac too which currently allows -l 32 without a warning.
it is done.
the new limits: if the sample rate is <=48kHz, max blocksize = 4608 and max LPC order = 12. all the tests I did showed that 32-order LPC got <0.1% more compression (usually a lot less).
http://flac.cvs.sourceforge.net/*checkout*...mat.html#subset (http://flac.cvs.sourceforge.net/*checkout*/flac/flac/doc/html/format.html#subset)
Josh
I have builded a faster FLAKE binary from rev. 104 but it needs some testing.
On one of my test files encoding time has droped from 193.468 seconds (previous r104 build (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-bin.zip)) to 173.906 seconds (test r104 build (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)) using -12 command-line option.
Download test binary:
flake_r104_test.zip (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)
wisodev
I have builded a faster FLAKE binary from rev. 104 but it needs some testing.
On one of my test files encoding time has droped from 193.468 seconds (previous r104 build (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-bin.zip)) to 173.906 seconds (test r104 build (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)) using -12 command-line option.
Download test binary:
flake_r104_test.zip (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)
wisodev
Your test compile is a bit faster than mine on my machine (Celeron M 1,4Ghz). I can't give any good values, though, because I already got several seconds difference when running my own build twice, so the measurement is not very reliable.
I have builded a faster FLAKE binary from rev. 104 but it needs some testing.
On one of my test files encoding time has droped from 193.468 seconds (previous r104 build (http://www.thefrontend.net/flake/svn/flake-svn-r104-win32-bin.zip)) to 173.906 seconds (test r104 build (http://www.thefrontend.net/flake/svn/flake_r104_test.zip)) using -12 command-line option.
Here's some results for my "FLAC" corpus:
Setting R104 R104 Test Diff
======================================
-10 286.726s 274.561s 96%
-11 375.687s 347.035s 92%
-12 1002.105s 914.239s 91%
Edit: Here's the figures for my MinGW compile:
Setting MinGW R104 R104 Test Diff
======================================
-10 394.015s 274.561s 70%
-11 486.764s 347.035s 71%
-12 1322.178s 914.239s 69%
I don't think I'll bother anymore! Excellent work wisodev.
NB: Timings were recorded using TIMER, and CPU-only times are reported.
I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this. with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion. this goes for flac too which currently allows -l 32 without a warning.
it is done.
the new limits: if the sample rate is <=48kHz, max blocksize = 4608 and max LPC order = 12. all the tests I did showed that 32-order LPC got <0.1% more compression (usually a lot less).
http://flac.cvs.sourceforge.net/*checkout*...mat.html#subset (http://flac.cvs.sourceforge.net/*checkout*/flac/flac/doc/html/format.html#subset)
Josh
Splendid. I have modified the Flake warning message output and the return code of the flake_validate_params function to reflect the new standard.
I checkout rev.107 from SVN and compile it myself under x86_64 Ubuntu. While encoding disc image I see:
Signed 16-bit 44100 Hz stereo
samples: 116712120 (0m16.961s)
If I try to compile this test code:
int th, tm, ts, tms;
tms = 116712120 * 1000 / 44100;
I get two "integer overflow" warnings.
I think
int is too small here (in flake.c at line 447).
I checkout rev.107 from SVN and compile it myself under x86_64 Ubuntu. While encoding disc image I see:
Signed 16-bit 44100 Hz stereo
samples: 116712120 (0m16.961s)
If I try to compile this test code:
int th, tm, ts, tms;
tms = 116712120 * 1000 / 44100;
I get two "integer overflow" warnings.
I think int is too small here (in flake.c at line 447).
Thanks. Fixed.
Also, anyone can submit bug reports on Flake's Sourceforge project page. Getting bug reports via HA makes me feel a little guilty for taking up space with small fixes. They are appreciated either way though.
Flake (Win32) from SVN revision 108 available:
Download from my Website (http://www.thefrontend.net/flake/)
or use direct links:
Download binarys:
flake-svn-r108-win32-bin.zip (http://www.thefrontend.net/flake/svn/flake-svn-r108-win32-bin.zip)
Download sources:
flake-svn-r108-win32-src.zip (http://www.thefrontend.net/flake/svn/flake-svn-r108-win32-src.zip)
wisodev
although i ran into a couple of problems compiling.
Would it be too much of a problem to put my changes into the code, to keep it portable across compilers?
optimize.c line 219 replaced with: uint32_t bits[32]; (where 32 used to be levels. MSVC doesn't like non-constant sizes for these things.) I am sure that 32 is as large as levels gets.
in encode.c flake_set_defaults, msvc doen't like this syntax:
foo = ((int[]){1,2,3})[2];
so i defined all these "nameless" arrays as ones with names, so it becomes:
int sel[] = {1,2,3};
foo = sel[2];
thanks,
will
Both MSVC issues have now been fixed.
Also, I am almost done with a new website for Flake. The only thing left is API documentation and benchmarks. I am putting in a link to Synthetic Soul's lossless benchmarks, but it would also be nice to have more extensive tests to better illustrate the differences between the compression levels. I'm not so worried anymore about detailed comparisons to the reference encoder because Synthetic Soul's tests give a good enough overview. Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels. Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful. Corpus totals would be better than file-by-file results.
Thanks!
Justin
Both MSVC issues have now been fixed.
Excellent, thanks
I have included my enc_flac plugin (sources are attached somewhere in this thread =) built with wisodevs libFlake r108 to the beta branch of Winamp.
If it gets a good response from our beta team and QA department (and I can't see why it wouldn't) maybe we'll put it in a subsequent version of winamp.
Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels. Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful. Corpus totals would be better than file-by-file results.
Has anyone provided this yet?
I don't mind running the tests if no-one else will. My machine isn't hugely fast (Athlon XP 2400+, 512MB RAM, Win2K) but it's OK, and I will need a compile from wisodev as I have proved already that my compiles are far inferior to his.
Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels. Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful. Corpus totals would be better than file-by-file results.
Has anyone provided this yet?
I don't mind running the tests if no-one else will. My machine isn't hugely fast (Athlon XP 2400+, 512MB RAM, Win2K) but it's OK, and I will need a compile from wisodev as I have proved already that my compiles are far inferior to his.
I will provide fresh compile in few hours including new experimental build using P3 optimizations (using only compiler options - not assembly optimized!!!) witch seems to be even faster then the last one.
Flake (Win32) from SVN revision 110 available:
Download from my Website (http://win32builds.sourceforge.net/flake/index.html)
or use direct links:
Download binarys:
flake-svn-r110-win32-bin.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r110-win32-bin.zip?download)
Download sources:
flake-svn-r110-win32-src.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r110-win32-src.zip?download)
wisodev
Thanks wisodev.
I have an AMD XP CPU. Should I be looking at /org/flake.exe, /p3/flake.exe or /flake.exe?
Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.
I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...
Edit: I just did a short test with the album "Supertramp - Some Things Never Change", comparing the commandlines "-8" and "-8 -m 1".
"-8 -m 1" was of course always way faster, and compressed 8 of the 12 files better than "-8". The differences in bitrate were quite small though, around 0.1-0.2kbps average.
@wisodev
I would be interested what compiler flags you use (and which compiler). Have you added asm to the sources?
Also, I still get this time bug when encoding with wisodev's latest build (all binaries show this)
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)
I thought this was solved?
Also, I still get this time bug when encoding with wisodev's latest build (all binaries show this)
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)
I thought this was solved?
I did too...sorry. I just committed a 2nd attempt, which seems to work for me. I just tested it on a 11 million sample wav and it correctly reported around 4 minutes.
-Justin
Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.
I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...
Edit: I just did a short test with the album "Supertramp - Some Things Never Change", comparing the commandlines "-8" and "-8 -m 1".
"-8 -m 1" was of course always way faster, and compressed 8 of the 12 files better than "-8". The differences in bitrate were quite small though, around 0.1-0.2kbps average.
This is really interesting. My tests resulted in not quite as good compression. It could be coincedence. More testing would be great. The order estimate method is a pretty crude 1st attempt at an idea I got from a paper which dealt with speech compression. It suggested a method using the highest order reflection coefficient above a certain threshold. I've found that the method varies widely depending on the type of audio...I imagine it works well for speech, but would need some tweaking for other types of signals. I had once tried to do some testing using gnuplot output, but my statistical analysis skills aren't really advanced enough to know what to look for...so I have left it at a simple threshold for now.
edit: I also want to note that I could almost always pick out the best order or close to it using visual output from gnuplot, but I could never translate that into a mathematical analysis that worked well. There is definitely potential there though.
-Justin
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)
I thought this was solved?
I submitted bug about it in sourceforce bug tracker after I discovered it was not fixed after 1st try. Justin, are you looking there?
Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)
I thought this was solved?
I submitted bug about it in sourceforce bug tracker after I discovered it was not fixed after 1st try. Justin, are you looking there?
I missed that one... I've changed the settings so I get an email now when someone posts a new bug. I have also updated the status of your bug report.
Thanks!
-Justin
My results for Flake SVN r110 wisodev build P3 optimised with RW corpus.
Setting Ratio Enc Dec
=================================
0 65.352% 136x 106x
1 62.822% 114x 104x
2 62.661% 98x 104x
3 59.581% 94x 97x
4 59.392% 92x 93x
5 59.292% 90x 96x
6 59.649% 66x 96x
7 59.307% 54x 96x
8 59.132% 49x 95x
8 -m 1 59.115% 83x 94x
8 -v 1 58.747% 47x 93x
9 59.019% 40x 95x
10 59.006% 28x 94x
11 58.758% 22x 83x
12 58.729% 9x 83x
12 -v 1 58.378% 9x 84x
99 58.336% 2x 72x
Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.
I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...
These results appear to bear yours out completely. I'm so glad you posted or I would have been really concerned about my -6 results!
I will post results for the other wisodev compiles (normal and org) as well, in the next day or so. I will also post the spreadsheet used to calculate the summaries (http://spreadsheets.google.com/ccc?key=pwAdtAz-_urL3FUFUo7Olfw) (uploaded to Google Docs & Spreadsheets).
Edit: Something else of note. In fact two somethings.
Firstly, I should mention that the results above are CPU-only times, where my TAK lossless comparison values are CPU+IO times. I started reporting CPU+IO in that comparison and it's awkward to stop now.
Secondly, the scripts I use to perform my tests use FSUM to check the decoded WAVE files against the MD5 hashes of the source files once all files are decoded. This normally serves me well, but in this instance nearly all files fail the check.
I have done a little checking and it appears to be down to the WAVE header written, which I think is 2 bytes smaller than the source (bear in mind the source files are decoded WavPack files). Again, from my limited understanding, this appears to be down to the ExtraParamSize bytes not being written by the FLAC decoder. From the document I have read it appears that these bytes are not required if no extra parameters are present. Strangely enough though every fourth file or so the hashes do match!
Edit: It seems that, as expected, with the files that
do match MD5 hashes,
neither have the ExtraParamSize bytes; therefore FLAC is acting consistently, while the source files are not consistent. Maybe this is noticable as most decompressors just return the original WAVE header, whereas FLAC creates its own?
I really have no idea what all this means, but it thought it worth noting. I bit compared a few files that had mismatched MD5s using foobar and the audio is bit-identical, as you would expect.
Here's the report from FSUM for -5 (it's the same for all though).
FAILED MD5 41_30sec.wav
FAILED MD5 ATrain.wav
FAILED MD5 Bachpsichord.wav
OK MD5 Bartok_strings2.wav
FAILED MD5 BeautySlept.wav
FAILED MD5 BigYellow.wav
FAILED MD5 Blackwater.wav
FAILED MD5 bodyheat.wav
OK MD5 chanchan.wav
FAILED MD5 DaFunk.wav
FAILED MD5 death2.wav
OK MD5 Debussy.wav
FAILED MD5 EnolaGay.wav
FAILED MD5 experiencia.wav
FAILED MD5 female_speech.wav
FAILED MD5 FloorEssence.wav
OK MD5 getiton.wav
FAILED MD5 gone.wav
OK MD5 Hongroise.wav
FAILED MD5 Illinois.wav
FAILED MD5 ItCouldBeSweet.wav
FAILED MD5 kraftwerk.wav
FAILED MD5 Layla.wav
OK MD5 Leahy.wav
FAILED MD5 LifeShatters.wav
FAILED MD5 macabre.wav
FAILED MD5 Mahler.wav
FAILED MD5 male_speech.wav
OK MD5 Mama.wav
FAILED MD5 MidnightVoyage.wav
FAILED MD5 mybloodrusts.wav
OK MD5 NewYorkCity.wav
OK MD5 OrdinaryWorld.wav
FAILED MD5 Polonaise.wav
FAILED MD5 Quizas.wav
FAILED MD5 riteofspring.wav
OK MD5 rosemary.wav
FAILED MD5 Scars.wav
OK MD5 SinceAlways.wav
FAILED MD5 thear1.wav
FAILED MD5 TheSource.wav
OK MD5 TomsDiner.wav
OK MD5 trust.wav
OK MD5 Twelve.wav
FAILED MD5 velvet.wav
OK MD5 Waiting.wav
31 checksums failed
I don´t know the story of flake but now while i test around i am not really thrilled to have the extra options.
These options like l32 or v1 simply brake the standard in my eyes. l32 encoded files don´t play on every hardware and v1 is confusing even winamp and my Squeezebox again when i want to fast forward.
What about *.flak for flake encodings? So everyone knows he may have to reencode to use it properly when he gets such a file by accident.
This may have been discudssed before but i somehow missed that.
These options like l32 or v1 simply brake the standard in my eyes.
As I understand it, files encoded with -v 1 or -l 32 are standard flac files, not a weird format extension introduced by flake.
My results for Flake SVN r110 wisodev build P3 optimised with RW corpus.
Setting Ratio Enc Dec
=================================
0 65.352% 136x 106x
1 62.822% 114x 104x
2 62.661% 98x 104x
3 59.581% 94x 97x
4 59.392% 92x 93x
5 59.292% 90x 96x
6 59.649% 66x 96x
7 59.307% 54x 96x
8 59.132% 49x 95x
8 -m 1 59.115% 83x 94x
8 -v 1 58.747% 47x 93x
9 59.019% 40x 95x
10 59.006% 28x 94x
11 58.758% 22x 83x
12 58.729% 9x 83x
12 -v 1 58.378% 9x 84x
99 58.336% 2x 72x
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Looking at the above, -5 seems to be the sweet spot if your rate performance as the most important and "-8 -v 1" the sweet spot if you rate compression as the most important.
-brendan
Thanks wisodev.
I have an AMD XP CPU. Should I be looking at /org/flake.exe, /p3/flake.exe or /flake.exe?
.\flake.exe - ICL Build with PGO Optimizations
.\P3\flake.exe - ICL Build with PGO & P3 Optimizations
.\ORG\flake.exe - ICL Build
The P3 is optimized for Pentium III and should work for all processors above P3 specs. It Works very well on Athlon XP processors (I have Athlon XP 2000+). I have in mind to add P4 build but I can't build it on my machine because of PGO optimization that req running the program and my CPU doesn't support P4 builds.
The original (ORG) build is not considered for normal usage only added for speed testing of PGO and PGO&P3 builds. For most CPUs the .\flake.exe build is good and should be used as safest solution.
@wisodev
I would be interested what compiler flags you use (and which compiler). Have you added asm to the sources?
All I have added is build enviroment in win32 directory. No assembly optimizations added by me. I have added some small changes to code to build it cleanly under MSVC with Intel compiler. Maybe Justin can add them to official sources (just diff my sources with SVN revision 110 to see what have been changed).
I am using MSVC++
v7.1 and Intel C++ Compiler
v8.0.040. This version of Intel compiler works best for me. I have tried newer versions but I can't get better results.
The compiler flags are main reason for speedups of my builds. I am using PGO optimization provided by Intel C++ compiler and for P3 build additionaly /QxK flag is used to enable even more optimizations. More information is avaible when you check the project settings for eacg configuraion (all witch begin with Release, the Debug builds are obviously not optimized ;-).
I really have no idea what all this means, but it thought it worth noting. I bit compared a few files that had mismatched MD5s using foobar and the audio is bit-identical, as you would expect.
You're right. FLAC creates a fresh RIFF/WAVE-Header, but Wavpack for example recreates the original one.
.\flake.exe - ICL Build with PGO Optimizations
.\P3\flake.exe - ICL Build with PGO & P3 Optimizations
.\ORG\flake.exe - ICL Build
The P3 is optimized for Pentium III and should work for all processors above P3 specs. It Works very well on Athlon XP processors (I have Athlon XP 2000+). <snip> The original (ORG) build is not considered for normal usage only added for speed testing of PGO and PGO&P3 builds. For most CPUs the .\flake.exe build is good and should be used as safest solution.
Thanks for the info wisodev. With that in mind I won't bother with the ORG build. Here's the PGO and PGO+P3 results:
| PGO + P3 | PGO
Setting Ratio | Enc Dec | Enc Dec
===================+================+==============
0 65.352% | 136x 106x | 133x 104x
1 62.822% | 114x 104x | 110x 106x
2 62.661% | 98x 104x | 95x 104x
3 59.581% | 94x 97x | 91x 98x
4 59.392% | 92x 93x | 90x 96x
5 59.292% | 90x 96x | 88x 96x
6 59.649% | 66x 96x | 65x 99x
7 59.307% | 54x 96x | 52x 98x
8 59.132% | 49x 95x | 47x 95x
8 -m 1 59.115% | 83x 94x | 80x 93x
8 -v 1 58.747% | 47x 93x | 46x 95x
9 59.019% | 40x 95x | 39x 96x
10 59.006% | 28x 94x | 27x 95x
11 58.758% | 22x 83x | 21x 83x
12 58.729% | 9x 83x | 8x 81x
12 -v 1 58.378% | 9x 84x | 8x 84x
99 58.336% | 2x 72x | 2x 72x
View the Google spreadsheet with all values for all files here (http://spreadsheets.google.com/ccc?key=pwAdtAz-_urKzEsJW438tFg).
You're right. FLAC creates a fresh RIFF/WAVE-Header, but Wavpack for example recreates the original one.
Thanks for the confirmation. This makes sense to me.
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Not sure if this fits the bill, but I can't think of any other way of displaying it.
(http://www.synthetic-soul.co.uk/temp/flake-svn-r110.gif)
I don´t know the story of flake but now while i test around i am not really thrilled to have the extra options.
These options like l32 or v1 simply brake the standard in my eyes. l32 encoded files don´t play on every hardware and v1 is confusing even winamp and my Squeezebox again when i want to fast forward.
What about *.flak for flake encodings? So everyone knows he may have to reencode to use it properly when he gets such a file by accident.
This may have been discudssed before but i somehow missed that.
The files may confuse some software decoders and skip in some hardware decoders, but I assure you they are 100% valid FLAC files based only on the specification. The issues that occur are not Flake issues, they are due to there being 2 levels of compliance, the subset specification and the full specification. Flake gives a warning when the files do not fall within the subset specification, but all generated files fall within the full specification.
If you're worried about getting a FLAC file which may cause problems, I could try to write a small utility program which would do frame analysis and detect whether a file falls within the subset specification. As for winamp, maybe adding a seektable using metaflac would help?...I don't really know how the winamp plug-in does seeking and I don't use winamp (anymore). I do use xmms, which completely locks up when seeking with the variable bitrate files...no good. Personally, I see these as software bugs, but I can't really blame the makers since that feature of the specification is not implemented by the reference encoder. MPlayer plays & seeks just fine...it uses its own demuxer & FFmpeg's decoder.
On an unrelated note, thanks for all the benchmarks! The graph is great Now that I see the results, I'm seriously considering abandoning the "mirror the reference encoder" idea for compression levels. I think trimming it down to just 7 presets corresponding to 0, 1, 5, 8m1, 10, 11, and 99 should pretty much cover what is needed.
-0 for really-fast encoding
-1 for fast encoding
-5 for normal encoding
"-8 -m 1" for high compression
-10 for best compression within subset
-11 for extra compression outside subset (-12 speed loss isn't worth the tiny gain)
-99 for experimental
I would probably get rid of the numbered presets in favor of named presets. What do you all think about this idea?
-Justin
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Not sure if this fits the bill, but I can't think of any other way of displaying it...
Actually, that was quite useful. The two points on the encoding line that stood out to me numerically after looking up and down the chart a few times also appear to stand out rather obviously in the chart.
-brendan
@PrakashPFor PGO&P3 Instrumentation build (Release Instrumentation P3|Win32) I am using this options for ICL:
/GL /Ox /Og /Ob1 /Ot /G7 /GA /QxK /FD /MT /Qc99 /Qprof_genx /Gd /TC
For PGO&P3 Feedback build (Release Feedback P3|Win32) I am using this options for ICL:
/GL /Ox /Og /Ob1 /Ot /G7 /GA /QxK /FD /MT /Qc99 /Qprof_use /Gd /TC
For PGO only build just remove /QxK option.
wisodev
The files may confuse some software decoders and skip in some hardware decoders, but I assure you they are 100% valid FLAC files based only on the specification. The issues that occur are not Flake issues, they are due to there being 2 levels of compliance, the subset specification and the full specification. Flake gives a warning when the files do not fall within the subset specification, but all generated files fall within the full specification.
If you're worried about getting a FLAC file which may cause problems, I could try to write a small utility program which would do frame analysis and detect whether a file falls within the subset specification. As for winamp, maybe adding a seektable using metaflac would help?...I don't really know how the winamp plug-in does seeking and I don't use winamp (anymore). I do use xmms, which completely locks up when seeking with the variable bitrate files...no good. Personally, I see these as software bugs, but I can't really blame the makers since that feature of the specification is not implemented by the reference encoder. MPlayer plays & seeks just fine...it uses its own demuxer & FFmpeg's decoder.
Thanks for your answer. I read the first time deeper about flac and switches. I am sure you can penetrate normal flac to the point it doesn´t work on any player also
No need for a tool that tests the frames. This -v1 doesn´t seem to need any more decoding power than without so it is time to "correct" the decoders.
btw. this latest wisodev build is awsome fast!
@Synthetic Soul:
Thanks for the tests, it would also be nice to have "-8 -m 1 -v 1" in next time, maybe instead of "-8 -v 1"? This should have a very good compression/encoding time ratio.
I'd love to see some curves plotted of effort vs. reward (aka price/performance aka "sweet spot").
Not sure if this fits the bill, but I can't think of any other way of displaying it.
(http://www.synthetic-soul.co.uk/temp/flake-svn-r110.gif)
Synthetic Soul,
The plot of the results look great. I was wondering why it is that "-m 1" is only tested on "8". Can it not be used on levels 9 and up? I didn't see anything about that, so I apologize if I missed it. I have to wonder how much it could potentially increase encoding times of those above 9, especially if it can increase 12's encoding time significantly.
I tried -10 -m 1 on a single album and it gives a significant boost (~100%) to the encoding speed without decreasing the compression ratio (+1 kbps). This switch looks great!
I also tried -v1, but there's no seeking with foobar2000 (0.9.4.1).
I think "-9 -m 1" ist the same as "-8 -m 1".
"-9" and upward use search methods which are probably better (compression-wise) than "-m 1".
@Synthetic Soul:
Thanks for the tests, it would also be nice to have "-8 -m 1 -v 1" in next time, maybe instead of "-8 -v 1"? This should have a very good compression/encoding time ratio.
The plot of the results look great. I was wondering why it is that "-m 1" is only tested on "8". Can it not be used on levels 9 and up? I didn't see anything about that, so I apologize if I missed it. I have to wonder how much it could potentially increase encoding times of those above 9, especially if it can increase 12's encoding time significantly.
I simply used the settings requested by Justin, and added -8 -m 1 as it was mentioned by MedO shortly after, and it was suggested that more info was required (so I thought I may as well add it to the list).
My wife is possibly beginning labour at the moment, so I kind of have other things on my mind.
If I get chance I will try to run the tests requested.
My wife is possibly beginning labour at the moment, so I kind of have other things on my mind.
That's awesome! Hope all goes well.
I think trimming it down to just 7 presets corresponding to 0, 1, 5, 8m1, 10, 11, and 99 should pretty much cover what is needed.
...
I would probably get rid of the numbered presets in favor of named presets. What do you all think about this idea?
I think it's a good idea. And if we really want we always can use cryptic -b -t -l -m -r -s combinations
That's awesome! Hope all goes well.
Thank you. Things are unfortunately going very slowly, so there's no news yet.
Except...
My results for Flake SVN r110 wisodev build P3 optimised with RW corpus, with -8 -m 1 -v 1 and -9 to -12 with -m 1:
Setting Ratio Enc Dec
======================================
0 65.352% 136x 106x
1 62.822% 114x 104x
2 62.661% 98x 104x
3 59.581% 94x 97x
4 59.392% 92x 93x
5 59.292% 90x 96x
6 59.649% 66x 96x
7 59.307% 54x 96x
8 59.132% 49x 95x
8 -m 1 59.115% 83x 94x
8 -v 1 58.747% 47x 93x
8 -m 1 -v 1 58.752% 83x 96x
9 59.019% 40x 95x
9 -m 1 59.115% 85x 96x
10 59.006% 28x 94x
10 -m 1 59.115% 86x 100x
11 58.758% 22x 83x
11 -m 1 58.920% 67x 76x
12 58.729% 9x 83x
12 -m 1 58.920% 67x 76x
12 -v 1 58.378% 9x 84x
99 58.336% 2x 72x
I don't understand the -9 -m 1/-10 -m 1 and -11 -m 1/-12 -m 1 thing, but I'm sure someone does.
I don't understand the -9 -m 1/-10 -m 1 and -11 -m 1/-12 -m 1 thing, but I'm sure someone does.
Well, the only setting that changes between -8, -9 and -10 is the -m switch, and you override that manually, so you get the same results because the settings are identical. The same goes for -11/-12.
Edit:
I just saw that -11 -m 1 is bigger than -8 -m 1. I guess the "estimate" search method just isn't optimised for those high predictor orders. Just a guess, I don't know much about FLAC technology...Edit2: I confused rows in the table... thanks Josef Pohm for the hint.
I have a Rio Karma and while it can play FlAKE -12 compressed flac files, it has a lot of trouble seeking with these FLAC files. FLAC 1.1.2_CVS which gets close compression works great with my Rio Karma.
What I'd like to know is what are the best settings for Flake to work with my Rio Karma? I'd like to compare compressions sizes to find out which is better. I was playing around and ended up gettng a larger file that didn't work vs FLAC 1.1.2_CVS.
What setting is it in FLAKE that causes it not to be fully compatible with my Rio? I tried the -v 1 as suggested earlier in the thread to make it compatible with FLAC and that also failed.
Jon
I have a Rio Karma and while it can play FlAKE -12 compressed flac files, it has a lot of trouble seeking with these FLAC files. FLAC 1.1.2_CVS which gets close compression works great with my Rio Karma.
What I'd like to know is what are the best settings for Flake to work with my Rio Karma? I'd like to compare compressions sizes to find out which is better. I was playing around and ended up gettng a larger file that didn't work vs FLAC 1.1.2_CVS.
What setting is it in FLAKE that causes it not to be fully compatible with my Rio? I tried the -v 1 as suggested earlier in the thread to make it compatible with FLAC and that also failed.
Jon
For now the best setting for your Rio would probably be "flake -10". It generates files which are roughly equivalent (compatibility-wise) to "flac -8". Definitely don't use -v 1 since it is probably the most experimental option and has compatibility issues.
edit: to answer your other question more fully, it's probably the max LPC order which is causing problems on the Rio with flake -12 files.
edit2: Given the many questions, for the next release, I will try to make some documentation devoted to compatibility of various features of the FLAC specification and Flake encoding parameters.
-Justin
I have a Rio Karma and while it can play FlAKE -12 compressed flac files, it has a lot of trouble seeking with these FLAC files. FLAC 1.1.2_CVS which gets close compression works great with my Rio Karma.
What I'd like to know is what are the best settings for Flake to work with my Rio Karma? I'd like to compare compressions sizes to find out which is better. I was playing around and ended up gettng a larger file that didn't work vs FLAC 1.1.2_CVS.
What setting is it in FLAKE that causes it not to be fully compatible with my Rio? I tried the -v 1 as suggested earlier in the thread to make it compatible with FLAC and that also failed.
Jon
For now the best setting for your Rio would probably be "flake -10". It generates files which are roughly equivalent (compatibility-wise) to "flac -8". Definitely don't use -v 1 since it is probably the most experimental option and has compatibility issues.
edit: to answer your other question more fully, it's probably the max LPC order which is causing problems on the Rio with flake -12 files.
edit2: Given the many questions, for the next release, I will try to make some documentation devoted to compatibility of various features of the FLAC specification and Flake encoding parameters.
-Justin
I just tested Flake at -10 and found that the compatability with seeking in my Rio Karma still isn't there. I did the same track in Flac 1.1.2_CVS and compatability is perfect. My settings for Flac are "-8 -A tukey(0.25) -A gauss(0.1875) -V --sector-align %1" and the file sizes are very similar between the two. 34,189,547 for Flake and 34,153,250 for Flac. There is something different in the Flake file that the Rio Karma is not liking. I just decided to try Flake at -1 and it still doesn't seek properly. Once it plays it will play allt he way fine. But as for seeking it eaier doesn't seek or takes a long time to seek.
You do realize flake does not add seek points to the file (at least version 0.10 doesn't)? Try adding seekpoints using metaflac.
You do realize flake does not add seek points to the file (at least version 0.10 doesn't)? Try adding seekpoints using metaflac.
I'll have to try that. I didn't realize. Thanks!
Hot damn! Adding seek points works. Thank you Madman2003.
My wife is possibly beginning labour at the moment, so I kind of have other things on my mind.
Great avatar and congratulations.
'grats everyone, you made it into Winamp 5.31
http://forums.winamp.com/showthread.php?s=...threadid=258205 (http://forums.winamp.com/showthread.php?s=&threadid=258205)
Congratulations Justin. Given both Winamp and FLAC's popularity this seems like an excellent achievement to me.
Great avatar and congratulations.
Thank you.
'grats everyone, you made it into Winamp 5.31
http://forums.winamp.com/showthread.php?s=...threadid=258205 (http://forums.winamp.com/showthread.php?s=&threadid=258205)
just confirmed, it's using r108. flake is neat and has promise but I think this is premature to use it in winamp now for the following reasons:
1. no seektable support which can make seeking awkwards.
2. still too easy to create non-subset streams. the winamp plugin goes up to -12 and there is no warning for -11 or -12.
except in rare cases, non-subset streams are an extremely poor tradeoff of reduced usability for a tiny fraction of compression. that's why even in libFLAC you have to explicitly turn off subset support when using non-subset options or the encoder init call will fail.
3.
most important, flake is not well tested. I built r108 and ran part of the FLAC test suite against it (stereo 16bps streams), and some files silently failed to generate correct FLAC files, i.e. encoding them appeared to work but the decoded file did not match the input.
users will be more forgiving for a decoder; another decoder can be used to recover a good FLAC file. but a lossless encoder is going to be held to a higher standard. if it generates FLACs that are not lossless or cause problems on playback, that's more serious.
Josh
edit: I hope "flake is not well tested" doesn't sound critical, what I mean is not tested well-enough as a lossless encoder for production use.
3. most important, flake is not well tested. I built r108 and ran part of the FLAC test suite against it (stereo 16bps streams), and some files silently failed to generate correct FLAC files, i.e. encoding them appeared to work but the decoded file did not match the input.
Argh, stupid me. I already used flake in my daily "work", but for security wrote a small wrapper script that would use flac -d and an md5summer to ensure the created flac file was ok. I thought this way I could safely use flake and also give some feedback when I encountered a problem. However, I just found out the script has a bug and let any errors through without warning . Maybe I should have tested it *before* using... ah well.
In what way were the files damaged, i.e. would flactester catch faulty ones?
In what way were the files damaged, i.e. would flactester catch faulty ones?
I will check it out more tonight and upload a sample, but the FLACs were valid, they just decoded to different audio than the input. also the MD5 sum was wrong.
Josh
also the MD5 sum was wrong.
Does it mean that flake stored wrong MD5 of the original audio? I mean, is it secure to test flake implementation with just "flac -t", or is it necessary to decode and compare with the original?
BTW, shouldn't it be "we recommend not to use it for archival" instead of "we recommend to not use it for archival"?
Does it mean that flake stored wrong MD5 of the original audio? I mean, is it secure to test flake implementation with just "flac -t", or is it necessary to decode and compare with the original?
flac -t does not catch it as the flake file has a correct MD5 sum for its audio, but the audio is not the same as the original wave.
Josh
Does it mean that flake stored wrong MD5 of the original audio? I mean, is it secure to test flake implementation with just "flac -t", or is it necessary to decode and compare with the original?
flac -t does not catch it as the flake file has a correct MD5 sum for its audio, but the audio is not the same as the original wave.
Josh
Is the difference large or is it just, like, the least significant bit in a few samples changed?
Is the difference large or is it just, like, the least significant bit in a few samples changed?
He-he that doesn't matter. The question is whether flake is lossless (no data lost or changed) or not.
Is the difference large or is it just, like, the least significant bit in a few samples changed?
He-he that doesn't matter. The question is whether flake is lossless (no data lost or changed) or not.
I know that this is a serious bug, but I'm trying to decide whether to re-rip the CDs I already compressed with flake.
Question regarding setting up foobar with Flake moved to this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=49658&hl=). Issue is foobar-/user-related, not a Flake issue.
ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio. when I have more time after the next release I will look into the rest of the suite.
Josh
[attachment=2624:attachment]
ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio. when I have more time after the next release I will look into the rest of the suite.
Josh
[attachment=2624:attachment]
I'll asume that the problem exists, i'm curious how you made that file (in other words, should i worry about some flac's i created).
Flake (Win32) from SVN revision 112 available:
Note:
I've changed way I build binarys. The old method used visual studio project files. Now I'm using simple batch script (build_intel.cmd available in src archive) to directly compile and link binarys using only Intel C++ Compiler. As results you don't need Visual Studio to build flake under windows. You need only Intel C++ Compiler and MS Platform SDK. The old project files are still available. And one more thing it seems that new binarys are just bit faster ;-)
Download from my Website (http://win32builds.sourceforge.net/flake/index.html)
or use direct links:
Download binarys:
flake-svn-r112-win32-bin.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r112-win32-bin.zip?download)
Download sources:
flake-svn-r112-win32-src.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r112-win32-src.zip?download)
wisodev
EDIT: changed url to downloads and my website
ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio. when I have more time after the next release I will look into the rest of the suite.
Josh
[attachment=2624:attachment]
Turns out FLAC 1.1.3 beta 2 also cannot handle this WAV file without an error.
flac 1.1.3beta2, Copyright © 2000,2001,2002,2003,2004,2005,2006 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.
options: -P 8192 -b 4608 -m -l 12 -e -q 0 -r 0,6
wacky2.wav: WARNING: skipping unknown sub-chunk 'fact'
wacky2.wav: WARNING: skipping unknown sub-chunk 'pad '
wacky2.wav: WARNING: skipping unknown sub-chunk 'pad '
wacky2.wav: wrote 8328 bytes, ratio=138.800
The resulting WAV from the FLAC file no longer matches. The original is 96 bytes and the decompressed one is 60 bytes. FLAKE 's decompressed WAV file is 72 bytes.
no, that's not an error, it's a warning that it is skipping unsupported subchunks. the audio is correctly compressed with flac-1.1.3 (and earlier).
with flake, the non-audio subchunk that follows the audio is treated as audio and compressed along with the audio. that's why when decoded it decodes to more audio samples than are in the original wave file.
Josh
no, that's not an error, it's a warning that it is skipping unsupported subchunks. the audio is correctly compressed with flac-1.1.3 (and earlier).
with flake, the non-audio subchunk that follows the audio is treated as audio and compressed along with the audio. that's why when decoded it decodes to more audio samples than are in the original wave file.
Josh
This would be a bug in the flake wav reading code. Winamp doesn't use this code, our WAV reading is done by libsndfile (which as far as I know, doesn't have this problem).
I've made a FLAKE output plugin for Mpxplay and I've added ID3-info (artist,title,etc.) handling for the vorbis-comment function(s). Something similar would be nice in the original flake lib too.
You can download the modified source of v0.10 here: http://www.freewebtown.com/mpxplay (http://www.freewebtown.com/mpxplay) -> SCFLACS.ZIP
A short list about the modifications (maybe not complete):
- a new struct FlakeComments_t in flake.h, added to the end of struct FlakeContext
- flake_comment_init_utf8(FlakeContext *s,char *codepage) function to enable UTF8 text encoding (you have to call it by standard), the "codepage" is required for DOS version only (set it to NULL)
- flake_comment_add(FlakeContext *s,char *type,char *data), where the "type" can be "artist","title","album","date","genre","comment" or other standard vorbiscomment types
- modified write_vorbis_comment() function in encode.c
- added AnsiToUTF8 function to encode.c, but the the WIN32 version is not tested at all
You have to call flake_comment_init_utf8() and flake_comment_add() functions first,
then the flake_encode_init().
regards
Attila
I downloaded the SVN zip (http://subversion.tigris.org/downloads/svn-win32-1.4.0.zip), unzipped (I just unzipped to my desktop for speed), opened a command line in the "bin" ("\svn-win32-1.4.0\svn-win32-1.4.0\bin") directory and then used the command:
svn co https://svn.sourceforge.net/svnroot/flake-enc (https://svn.sourceforge.net/svnroot/flake-enc) flake-enc
This created a folder in my "bin" directory called "flake-enc". I then moved that to somewhere more convenient (a short path with no spaces) and then ran "./configure" then "make" using a cygwin console.
@Synthetic Soul,
I downloaded and tried to compile flake with mingw like your above instructions, but didn't have any luck. The download created the flake "bin" directory and I moved it to a easy to get at location as you did. What command line did you use and did you enter it in the MSYS console?
I have MSYS-1.0.11 installed. I have tried the "configure&&make" command in the MSYS console window but all I get is "exec: bash: not found
Any pointer would be most welcome.
get bash.
http://prdownloads.sf.net/mingw/bash-2.05b...ar.bz2?download (http://prdownloads.sf.net/mingw/bash-2.05b-MSYS.tar.bz2?download)
@Mangix
Thanks for the tip. Worked good.
Can someone please post a Win32 compile of FLAKE 112 please? Thanks!
As per post #319:
http://win32builds.sourceforge.net/flake/index.html (http://win32builds.sourceforge.net/flake/index.html)
As per post #319:
http://win32builds.sourceforge.net/flake/index.html (http://win32builds.sourceforge.net/flake/index.html)
Nice. Now I can bookmark it. Thank you!
I have uploaded new experimental Win64 builds of FLAKE using latest sources. Please try this builds and submit results.
Binarys for 64 bit windows (should work on EM64T and AMD64 compatible processors):
flake-svn-r112-win64-bin.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r112-win64-bin.zip?download)
Other related downloads are placed here (https://sourceforge.net/project/showfiles.php?group_id=183195).
The variable block size mode has a small bug that makes seeking impossible. In frame header is stored frame number instead of sample number. A quick hack:
Index: libflake/encode.c
===================================================================
--- libflake/encode.c (revision 112)
+++ libflake/encode.c (working copy)
@@ -973,7 +973,10 @@
assert(!ctx->bw->eof);
}
if(frame_buffer != NULL) {
- ctx->frame_count++;
+ if(ctx->params.variable_block_size)
+ ctx->frame_count += s->params.block_size;
+ else
+ ctx->frame_count++;
}
return bitwriter_count(ctx->bw);
}
yes. actually, storing the frame number in a variable-blocksize file is illegal according to the FLAC spec.
Josh
ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio. when I have more time after the next release I will look into the rest of the suite.
[attachment=2624:attachment]
You're right about the bug. It is now fixed. The wav reader was only checking for end-of-file not end-of-data-chunk. Thanks for the bug repot...sorry for the slow reply. I'm back doing development after some unexpected downtime.
yes. actually, storing the frame number in a variable-blocksize file is illegal according to the FLAC spec.
This issue has been fixed as well. Thanks miros for the info.
-Justin
My Flake SVN builds at revision 114 are available for download.
http://win32builds.sourceforge.net/flake/index.html (http://win32builds.sourceforge.net/flake/index.html)
Download binaries: flake-svn-r114-bin.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r114-bin.zip?download)
Download sources: flake-svn-r114-src.zip (http://prdownloads.sourceforge.net/win32builds/flake-svn-r114-src.zip?download)
wisodev
Using the file flake.exe in ~/flake-svn-r114-bin/exe_x64 I get the following error:
Fatal Error : This program was not built to run on the processor in your system.
I am using a AMD 64 X2 5000+ on Windows XP 64. Am I wrong in thinking this should be the proper executable or is this a bug?
I've reworked the flac test suite to use flake. it actually took a bit of work since the options are so different but I think the coverage is pretty good for the flac test streams at least. it could use more tests that exercise specific features of flake.
but, even then the tests crash on a lot of inputs. also there are places where it doesn't handle odd resolutions correctly; but except for those, I haven't run across any places where it is silently non-lossless yet (the worst kind of bug). the tests are still running though.
Justin, I'll PM you about the scripts so if you want you can try to reproduce them.
Josh
PS uploaded here: http://www.hydrogenaudio.org/forums/index....showtopic=51745 (http://www.hydrogenaudio.org/forums/index.php?showtopic=51745)
Using the file flake.exe in ~/flake-svn-r114-bin/exe_x64 I get the following error:
Fatal Error : This program was not built to run on the processor in your system.
I am using a AMD 64 X2 5000+ on Windows XP 64. Am I wrong in thinking this should be the proper executable or is this a bug?
I was using as target system based on a processor with Intel® EM64T running Windows Server 2003 x64 Edition or Windows XP Professional x64 Edition as described in documentation. I was thinking that this should work on Intel and AMD processor running on 64 bit CPU, but now I am not sure. I need to investigate this or maybe someone can put more light here? Sorry for any inconvenience from my side.
Edit:
From compiler documantation (I used /QxP):
/QxP
Intel® Core™ Duo processors and Intel® Core™ Solo processors, Intel Pentium 4 processors with Streaming SIMD Extensions 3, and compatible Intel processors with Streaming SIMD Extensions 3 (SSE3)
If a program compiled with -x (Linux) or /Qx (Windows) is executed on a non-compatible processor, it might fail with an illegal instruction exception or display other unexpected behavior. Executing programs compiled with -xN, -xB or -xP (Linux) or /QxN, /QxB or /QxP (Windows) on unsupported processors will display a run-time error similar to the following:
Fatal Error : This program was not built to run on the processor in your system.
AFAIK Intel's compiler generates such a code that explicitly checks for the 'GenuineIntel' string in the CPUID, and if it started on an AMD processor, it simply quits even though an X2 should run the program without problems. Someone should try to 'crack out' this check from the startup code...
It might not worth the trouble, because usually gcc generates better code for Athlon CPUs - if optimization for specific Athlon CPUs is chosen.
I was using as target system based on a processor with Intel® EM64T running Windows Server 2003 x64 Edition or Windows XP Professional x64 Edition as described in documentation. I was thinking that this should work on Intel and AMD processor running on 64 bit CPU, but now I am not sure. I need to investigate this or maybe someone can put more light here? Sorry for any inconvenience from my side.
I'm not sure if this is the problem, but Intel does have the Itanium 64 chipset that uses a different instruction base than the "x86" 64's (AMD 64's, etc.). If compiled for said chips, the code would not function on non-Itanium processors. AFAIK, this is only for special builds of Windows XP / 2003 and M$ has started (Vista+) supporting the more standard "x86" instruction set that is used by AMD.
I have added new flake builds to my win32builds (http://www.sf.net/projects/win32builds) project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.
Download:
Visual Studio 2005 builds of flake svn revision 114. (http://prdownloads.sourceforge.net/win32builds/flake-vc2005-r114.zip?download)
Visual Studio 2005 and Intel C++ Compiler 9.1 builds of flake svn revision 114. (http://prdownloads.sourceforge.net/win32builds/flake-icl91-r114.zip?download)
Thanks,
wisodev
I have added new flake builds to my win32builds (http://www.sf.net/projects/win32builds) project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.
I just tested the release binaries of both the M$ and Intel compilers. Good news: They both work great!
I have added new flake builds to my win32builds (http://www.sf.net/projects/win32builds) project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.
I just tested the release binaries of both the M$ and Intel compilers. Good news: They both work great!
Great news!
Anyway both releases have some potential for speed up. I didn't used PGO optimizations (VC2005 compiler has also this feature).
I have an MD5 mismach with the "flake-0.10-win32-wisodev-bin" that I downloaded from Flake SourceForge project page. If I compare with foobar, I get: "ERROR (foo_bitcompare) : Files have different length". Indeed, the generated file is 1211905 samples long, while the original wav file is 1211894.
With the "flake-icl91-r114", also compiled by wisodev, I have an perfect output. So, whatever it was, is already corrected.
BUT not in the Flake SourceForge project page, that is where people expect to find an good binary. We need a new official release, if this error is real.
I can reproduce the error here, and you can find the source file here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=41248&view=findpost&p=363343), to check if it is realy an error. The file for download was encoded by an old version of Flac, not by Flake.
I have an MD5 mismach with the "flake-0.10-win32-wisodev-bin" that I downloaded from Flake SourceForge project page. If I compare with foobar, I get: "ERROR (foo_bitcompare) : Files have different length". Indeed, the generated file is 1211905 samples long, while the original wav file is 1211894.
With the "flake-icl91-r114", also compiled by wisodev, I have an perfect output. So, whatever it was, is already corrected.
BUT not in the Flake SourceForge project page, that is where people expect to find an good binary. We need a new official release, if this error is real.
I can reproduce the error here, and you can find the source file here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=41248&view=findpost&p=363343), to check if it is realy an error. It was encoded by an old version of Flac, not by Flake.
I could not reproduce, but I believe you and agree with your point. There have been some critical bug fixes since version 0.10 was released. I am on vacation next week, so I hope to be able to package up a new release and put it up on Sourceforge. I also hope to get Flake to pass the FLAC test suite before then.
Justin, I found another bug that may be the culprit. I've gone back to finish the speedups of the libFLAC bitbuffer I started a long time ago and realized it is similar to flake's. then I noticed a problem in bitwriter_writebits():
static inline void
bitwriter_writebits(BitWriter *bw, int bits, uint32_t val)
{
assert(bits == 32 || val < (1U << bits));
if(bits == 0) return;
if(bw->eof || (bw->buf_ptr+3) >= bw->buf_end) {
bw->eof = 1;
} else {
if(bits < bw->bit_left) {
bw->bit_buf = (bw->bit_buf << bits) | val;
bw->bit_left -= bits;
} else {
bw->bit_buf <<= bw->bit_left; ///////////// <-------- HERE
bw->bit_buf |= val >> (bits - bw->bit_left);
if(bw->buffer != NULL) {
*(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
}
bw->buf_ptr += 4;
bw->bit_left += (32 - bits);
bw->bit_buf = val;
}
}
}
if bits==32 and bw->bit_left==32, then it will get to that line and try to shift left by 32 bits. according to C semantics that is a NOP, so bit_buf will have junk instead of being 0 for the next instruction.
I think it's just lucky that it is rarely (never?) triggered. most writes are <32bit quantities and you do do 31 bits at a time of the rice unary portion. but some metadata writes are doing 32 bits and if they are word-aligned there could be trouble.
in my new writer I am doing something like this (trying to translate back into flake-style, may be buggy):
} else if(bw->bit_left<32) {
bw->bit_buf <<= bw->bit_left; ///////////// <-------- OK now
bw->bit_buf |= val >> (bits - bw->bit_left);
if(bw->buffer != NULL) {
*(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
}
bw->buf_ptr += 4;
bw->bit_left += (32 - bits);
bw->bit_buf = val;
} else {
assert(bits==32);
bw->bit_buf = val;
if(bw->buffer != NULL) {
*(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
}
bw->buf_ptr += 4;
}
it can actually be optimized more but that's the idea.
Josh
Justin, I found another bug that may be the culprit. I've gone back to finish the speedups of the libFLAC bitbuffer I started a long time ago and realized it is similar to flake's. then I noticed a problem in bitwriter_writebits():
...
if bits==32 and bw->bit_left==32, then it will get to that line and try to shift left by 32 bits. according to C semantics that is a NOP, so bit_buf will have junk instead of being 0 for the next instruction.
I think it's just lucky that it is rarely (never?) triggered. most writes are <32bit quantities and you do do 31 bits at a time of the rice unary portion. but some metadata writes are doing 32 bits and if they are word-aligned there could be trouble.
...
Thanks Josh! I actually did not know that a left shift by 32 was a NOP. The code which this bit writer was based upon only allowed for doing 31 bits at once, and I did not catch that outside case when trying to adapt it to handle 32 bits. I applied the fix as you suggested.
-Justin
What's the state of flake in the sense of trustworthyness?
Also, any info if/when metadata support becomes available?
What's the state of flake in the sense of trustworthyness?
Also, any info if/when metadata support becomes available?
I'll let others answer as to general trustworthyness. I will say that Flake does generate MD5 checksums, but it does not have the verify-during-encoding option like flac does.
Metadata is far down the TODO list. For now, the only option is to apply metadata to flake-generated files using metaflac.
I'm looking forward to a "fully featured" flake, it's better suited to my system.
Flake is considerably(4-5x at -8) faster than flac for me. Anyone who can tell anything about the trustworthyness? (in other words, encountered problems with it)