Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: New FLAC encoder (Read 374461 times) previous topic - next topic
0 Members and 3 Guests are viewing this topic.

New FLAC encoder

Reply #325
@Mangix

Thanks for the tip.  Worked good.

New FLAC encoder

Reply #326
Can someone please post a Win32 compile of FLAKE 112 please? Thanks!


 

New FLAC encoder

Reply #329
I have uploaded new experimental Win64 builds of FLAKE using latest sources. Please try this builds and submit results.

Binarys for 64 bit windows (should work on EM64T and AMD64 compatible processors):

flake-svn-r112-win64-bin.zip

Other related downloads are placed here.

New FLAC encoder

Reply #330
The variable block size mode has a small bug that makes seeking impossible. In frame header is stored frame number instead of sample number. A quick hack:

Code: [Select]
Index: libflake/encode.c
===================================================================
--- libflake/encode.c    (revision 112)
+++ libflake/encode.c    (working copy)
@@ -973,7 +973,10 @@
         assert(!ctx->bw->eof);
     }
     if(frame_buffer != NULL) {
-        ctx->frame_count++;
+        if(ctx->params.variable_block_size)
+            ctx->frame_count += s->params.block_size;
+        else
+            ctx->frame_count++;
     }
     return bitwriter_count(ctx->bw);
}

New FLAC encoder

Reply #331
yes.  actually, storing the frame number in a variable-blocksize file is illegal according to the FLAC spec.

Josh

New FLAC encoder

Reply #332
ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio.  when I have more time after the next release I will look into the rest of the suite.
[attachment=2624:attachment]


You're right about the bug.  It is now fixed.  The wav reader was only checking for end-of-file not end-of-data-chunk.  Thanks for the bug repot...sorry for the slow reply.  I'm back doing development after some unexpected downtime.

yes.  actually, storing the frame number in a variable-blocksize file is illegal according to the FLAC spec.


This issue has been fixed as well.  Thanks miros for the info.

-Justin

New FLAC encoder

Reply #333
My Flake SVN builds at revision 114 are available for download.

http://win32builds.sourceforge.net/flake/index.html

Download binaries: flake-svn-r114-bin.zip

Download sources: flake-svn-r114-src.zip

wisodev

New FLAC encoder

Reply #334
Using the file flake.exe in ~/flake-svn-r114-bin/exe_x64 I get the following error:
Quote
Fatal Error : This program was not built to run on the processor in your system.


I am using a AMD 64 X2 5000+ on Windows XP 64. Am I wrong in thinking this should be the proper executable or is this a bug?

New FLAC encoder

Reply #335
I've reworked the flac test suite to use flake.  it actually took a bit of work since the options are so different but I think the coverage is pretty good for the flac test streams at least.  it could use more tests that exercise specific features of flake.

but, even then the tests crash on a lot of inputs.  also there are places where it doesn't handle odd resolutions correctly; but except for those, I haven't run across any places where it is silently non-lossless yet (the worst kind of bug).  the tests are still running though.

Justin, I'll PM you about the scripts so if you want you can try to reproduce them.

Josh


New FLAC encoder

Reply #337
Quote
Using the file flake.exe in ~/flake-svn-r114-bin/exe_x64 I get the following error:
Quote

Fatal Error : This program was not built to run on the processor in your system.

I am using a AMD 64 X2 5000+ on Windows XP 64. Am I wrong in thinking this should be the proper executable or is this a bug?

I was using as target system based on a processor with Intel® EM64T running Windows Server 2003 x64 Edition or Windows XP Professional x64 Edition as described in documentation. I was thinking that this should work on Intel and AMD processor running on 64 bit CPU, but now I am not sure. I need to investigate this or maybe someone can put more light here? Sorry for any inconvenience from my side.

Edit:
From compiler documantation (I used /QxP):
Quote
/QxP
Intel® Core™ Duo processors and Intel® Core™ Solo processors, Intel Pentium 4 processors with Streaming SIMD Extensions 3, and compatible Intel processors with Streaming SIMD Extensions 3 (SSE3)

Quote
If a program compiled with -x (Linux) or /Qx (Windows) is executed on a non-compatible processor, it might fail with an illegal instruction exception or display other unexpected behavior. Executing programs compiled with -xN, -xB or -xP (Linux) or /QxN, /QxB or /QxP (Windows) on unsupported processors will display a run-time error similar to the following:

Fatal Error : This program was not built to run on the processor in your system.

New FLAC encoder

Reply #338
AFAIK Intel's compiler generates such a code that explicitly checks for the 'GenuineIntel' string in the CPUID, and if it started on an AMD processor, it simply quits even though an X2 should run the program without problems. Someone should try to 'crack out' this check from the startup code...

New FLAC encoder

Reply #339
It might not worth the trouble, because usually gcc generates better code for Athlon CPUs - if optimization for specific Athlon CPUs is chosen.

New FLAC encoder

Reply #340
I was using as target system based on a processor with Intel® EM64T running Windows Server 2003 x64 Edition or Windows XP Professional x64 Edition as described in documentation. I was thinking that this should work on Intel and AMD processor running on 64 bit CPU, but now I am not sure. I need to investigate this or maybe someone can put more light here? Sorry for any inconvenience from my side.


I'm not sure if this is the problem, but Intel does have the Itanium 64 chipset that uses a different instruction base than the "x86" 64's (AMD 64's, etc.). If compiled for said chips, the code would not function on non-Itanium processors. AFAIK, this is only for special builds of Windows XP / 2003 and M$ has started (Vista+) supporting the more standard "x86" instruction set that is used by AMD.

New FLAC encoder

Reply #341
I have added new flake builds to my win32builds project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.

Download:

Visual Studio 2005 builds of flake svn revision 114.

Visual Studio 2005 and Intel C++ Compiler 9.1 builds of flake svn revision 114.

Thanks,
wisodev

New FLAC encoder

Reply #342
I have added new flake builds to my win32builds project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.


I just tested the release binaries of both the M$ and Intel compilers. Good news: They both work great!

New FLAC encoder

Reply #343

I have added new flake builds to my win32builds project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.


I just tested the release binaries of both the M$ and Intel compilers. Good news: They both work great!


Great news!

Anyway both releases have some potential for speed up. I didn't used PGO optimizations (VC2005 compiler has also this feature).

New FLAC encoder

Reply #344
I have an MD5 mismach with the "flake-0.10-win32-wisodev-bin" that I downloaded from Flake SourceForge project page. If I compare with foobar, I get: "ERROR (foo_bitcompare) : Files have different length". Indeed,  the generated file is 1211905 samples long, while the original wav file is 1211894.

With the "flake-icl91-r114", also compiled by wisodev, I have an perfect output. So, whatever it was, is already corrected.

BUT not in the Flake SourceForge project page, that is where people expect to find an good binary. We need a new official release, if this error is real.

I can reproduce the error here, and you can find the source file here, to check if it is realy an error. The file for download was encoded by an old version of Flac, not by Flake.

New FLAC encoder

Reply #345
I have an MD5 mismach with the "flake-0.10-win32-wisodev-bin" that I downloaded from Flake SourceForge project page. If I compare with foobar, I get: "ERROR (foo_bitcompare) : Files have different length". Indeed,  the generated file is 1211905 samples long, while the original wav file is 1211894.

With the "flake-icl91-r114", also compiled by wisodev, I have an perfect output. So, whatever it was, is already corrected.

BUT not in the Flake SourceForge project page, that is where people expect to find an good binary. We need a new official release, if this error is real.

I can reproduce the error here, and you can find the source file here, to check if it is realy an error. It was encoded by an old version of Flac, not by Flake.


I could not reproduce, but I believe you and agree with your point.  There have been some critical bug fixes since version 0.10 was released.  I am on vacation next week, so I hope to be able to package up a new release and put it up on Sourceforge.  I also hope to get Flake to pass the FLAC test suite before then.

New FLAC encoder

Reply #346
Justin, I found another bug that may be the culprit.  I've gone back to finish the speedups of the libFLAC bitbuffer I started a long time ago and realized it is similar to flake's.  then I noticed a problem in bitwriter_writebits():
Code: [Select]
static inline void
bitwriter_writebits(BitWriter *bw, int bits, uint32_t val)
{
    assert(bits == 32 || val < (1U << bits));

    if(bits == 0) return;
    if(bw->eof || (bw->buf_ptr+3) >= bw->buf_end) {
        bw->eof = 1;
    } else {
        if(bits < bw->bit_left) {
            bw->bit_buf = (bw->bit_buf << bits) | val;
            bw->bit_left -= bits;
        } else {
            bw->bit_buf <<= bw->bit_left;    ///////////// <-------- HERE
            bw->bit_buf |= val >> (bits - bw->bit_left);
            if(bw->buffer != NULL) {
                *(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
            }
            bw->buf_ptr += 4;
            bw->bit_left += (32 - bits);
            bw->bit_buf = val;
        }
    }
}


if bits==32 and bw->bit_left==32, then it will get to that line and try to shift left by 32 bits.  according to C semantics that is a NOP, so bit_buf will have junk instead of being 0 for the next instruction.

I think it's just lucky that it is rarely (never?) triggered.  most writes are <32bit quantities and you do do 31 bits at a time of the rice unary portion.  but some metadata writes are doing 32 bits and if they are word-aligned there could be trouble.

in my new writer I am doing something like this (trying to translate back into flake-style, may be buggy):

Code: [Select]
        } else if(bw->bit_left<32) {
            bw->bit_buf <<= bw->bit_left;    ///////////// <-------- OK now
            bw->bit_buf |= val >> (bits - bw->bit_left);
            if(bw->buffer != NULL) {
                *(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
            }
            bw->buf_ptr += 4;
            bw->bit_left += (32 - bits);
            bw->bit_buf = val;
        } else {
            assert(bits==32);
            bw->bit_buf = val;
            if(bw->buffer != NULL) {
                *(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
            }
            bw->buf_ptr += 4;
        }

it can actually be optimized more but that's the idea.

Josh

New FLAC encoder

Reply #347
Justin, I found another bug that may be the culprit.  I've gone back to finish the speedups of the libFLAC bitbuffer I started a long time ago and realized it is similar to flake's.  then I noticed a problem in bitwriter_writebits():
...
if bits==32 and bw->bit_left==32, then it will get to that line and try to shift left by 32 bits.  according to C semantics that is a NOP, so bit_buf will have junk instead of being 0 for the next instruction.

I think it's just lucky that it is rarely (never?) triggered.  most writes are <32bit quantities and you do do 31 bits at a time of the rice unary portion.  but some metadata writes are doing 32 bits and if they are word-aligned there could be trouble.
...

Thanks Josh!  I actually did not know that a left shift by 32 was a NOP.  The code which this bit writer was based upon only allowed for doing 31 bits at once, and I did not catch that outside case when trying to adapt it to handle 32 bits.  I applied the fix as you suggested.

-Justin

New FLAC encoder

Reply #348
What's the state of flake in the sense of trustworthyness?

Also, any info if/when metadata support becomes available?

New FLAC encoder

Reply #349
What's the state of flake in the sense of trustworthyness?

Also, any info if/when metadata support becomes available?

I'll let others answer as to general trustworthyness.  I will say that Flake does generate MD5 checksums, but it does not have the verify-during-encoding option like flac does.

Metadata is far down the TODO list.  For now, the only option is to apply metadata to flake-generated files using metaflac.