HydrogenAudio

I'd like to report… something… about FLAC.

Sample: Nick Dolle - Doctor quotes (https://img1.wsimg.com/blobby/go/687bd300-dd9d-4622-ad17-33cd13503495/downloads/1c2apncj5_839618.wav) (it comes from there (https://legendaryvoiceprints.com/demos) in case WAV link in temporary)

$ powershell -c "(Get-Item in.wav).Length"
3058192

$ powershell -c "(Get-FileHash in.wav -a SHA256).Hash.ToLower()"
abab6a1762e450e76cf53889e4f7d4f4c4b27e8d81bbf441c3ad15ce37041506

$ flac -7 in.wav
flac 1.4.3
in.wav: WARNING: skipping unknown chunk 'JUNK' (use --keep-foreign-metadata to keep)
in.wav: WARNING: skipping unknown chunk 'bext' (use --keep-foreign-metadata to keep)
in.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=24
in.wav: WARNING: skipping unknown chunk 'minf' (use --keep-foreign-metadata to keep)
in.wav: WARNING: skipping unknown chunk 'elm1' (use --keep-foreign-metadata to keep)
in.wav: WARNING: there is data trailing the audio data. Use --keep-foreign-metadata or --ignore-chunk-sizes to keep it
in.wav: wrote 1686579 bytes, ratio=0,555

$ flac -7f --keep-foreign-metadata in.wav
flac 1.4.3
in.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=24
in.wav: wrote 1708403 bytes, ratio=0,563

$ flac -df --keep-foreign-metadata in.flac
in.flac: done
ERROR verifying foreign metadata restore from in.flac to in.wav: stored main chunk length differs from written length

$ powershell -c "(Get-Item in.wav).Length"
3058168

$ powershell -c "(Get-FileHash in.wav -a SHA256).Hash.ToLower()"
bd45039bcc5b4b99341e1b62710e5167086ec37d941a622c1521df7dc27879a8

The same happens with the latest FLAC git-66152791 20240301.

But WavPack 5.7.0, Monkey Audio 10.53, TAK 2.3.3 compress it without issues, then decompress bit-perfect.

ProTools generated BWF.

The new "--force-legacy-wave-format" doesn't help either.

I'd like to report… something… about FLAC.

Thanks, I'll have a look at that.

Not sure what to do here. This file does not comply with any WAVE specification I know of, but I guess ProTools is not something I can just ignore. The problem is that the fmt chunk is too large, it contains padding where it should not have any.

To be honest, that file is so far out of spec that I'm surprised Avid has the nerve to generate something like that unless, of course, it is never meant to be used outside of their own environment.

Unless it is not actually straight from ProTools of course. It would also be strange that I haven't had a bug report about this earlier.-

The file at https://hydrogenaud.io/index.php/topic,121311.0.html seems also to be generated by ProTools - at least MediaInfo says so. No idea whether it is related.

Further googling leads me to https://duc.avid.com/showthread.php?t=92522 . Still no idea whether it is related.


(Edit: In any case, wouldn't it be better to err out in the encoding stage, than finding out only upon decoding that the file cannot be reconstructed?)

Not sure what to do here. This file does not comply with any WAVE specification I know of, but I guess ProTools is not something I can just ignore. The problem is that the fmt chunk is too large, it contains padding where it should not have any.

But WavPack 5.7.0, Monkey Audio 10.53, TAK 2.3.3 compress it without issues, then decompress bit-perfect.

https://www.youtube.com/watch?v=5-sfG8BV8wU

After making sure that WavPack 5.7.0, Monkey Audio 10.53, TAK 2.3.3 encode dolle.wav (https://hydrogenaud.io/index.php/topic,123176.msg1040976.html#msg1040976) and then decode it back bit-perfect (BLAKE3 checksums were compared), I decided to observe the behavior of less popular lossless encoders.

$ la.exe dolle.wav
Lossless Audio Compressor Version 0.4b, copyright Michael Bevin 2002-2004
Error: invalid .wav header for file - dolle.wav

$ mp4alsRM23.exe dolle.wav
mp4alsRM23 - MPEG-4 Audio Lossless Coding (ALS), Reference Model Codec Version 23 for Win32
File type of dolle.wav is not supported!

$ ofr.exe dolle.wav
OptimFROG Lossless Audio Codec (Win/x64) v5.100 [2016-09-02]
srcFile: <dolle.wav>
dstFile: <dolle.ofr>
Exception UNKNOWN in file unknown, line 0
description: cbSize is incorrect or not an EXTENSIBLE format
Errors occurred compressing <dolle.wav>

$ sac.exe dolle.wav dolle.sac
Sac v0.6.6 - Lossless Audio Coder (c) Sebastian Lehmann
Open: 'dolle.wav': ok (3058192 Bytes)
warning: invalid fmt-chunk size
warning: input is not a valid .wav file
Total time: [00:00:00]

$ shorten.exe dolle.wav
shorten version 3.6.1: (c) 1992-1999 Tony Robinson and SoftSound Ltd
shorten.exe: input file is not a valid RIFF WAVE file

$ tta.exe -e dolle.wav dolle.tta
TTA1 lossless audio encoder/decoder, version 2.3
Encoding: "dolle.wav" to "dolle.tta"
tta.exe: can't read from input file

ALAC is a special case. Both qAAC --alac and FFMPEG -acodec alac encode dolle.wav without warnings or errors, but the decoded WAV is different from the original.

HALAC 0.2.6 MT SSE2 (https://hydrogenaud.io/index.php/topic,125248.msg1039348.html#msg1039348) just crashed.

@NetRanger, WARNING, cannot set number of threads: multithreading was not enabled during compilation of this binary.

While analyzing storm.wav (https://hydrogenaud.io/index.php/topic,113324.msg1041175.html#msg1041175), a problematic WAV from another topic, using WAVfix (https://github.com/agfline/wavfix) utility, I came across a mention of the oddity of Pro Tools generated files. However, the analysis of Pro Tools generated dolle.wav (https://hydrogenaud.io/index.php/topic,123176.msg1040976.html#msg1040976) says that it does not require repair, although data chunk is odd w/ pad.

$ wavfix storm.wav

> Processing 'storm.wav'
| [w] Wrong RIFF size: 0030203932 B + 8 [file size: 0030203996 B;]
| Current file structure :
| ======================
|     <fmt > chunk [offset: 0000000012; size: 0000000016 + 4 + 4;]
|     <data> chunk [offset: 0000000036; size: 0030203896 + 4 + 4;]
|     [NULL] chunk [offset: 0030203940; size: 0000000056 + 0 + 0;]
|
| [i] Checking <data> chunk..  ok.
|
| Recovered file structure :
| ========================
|     <fmt > chunk [offset: 0000000012; size: 0000000016 + 4 + 4;]
|     <data> chunk [offset: 0000000036; size: 0030203896 + 4 + 4;]
|     [NULL] chunk [offset: 0030203940; size: 0000000056 + 0 + 0;]
| [w] 1 unknwon bytes block remains.
|     Some programs like  Pro Tools use wrong formated files
|     like this ( <DIGI> chunk size missing  1 byte in PT ),
|     so we wont correct it. Yet it should play fine anyway.
|
| [i] Saving repaired file to 'storm_REPAIRED.wav'
| File successfully recovered.

$ wavfix dolle.wav

> Processing 'dolle.wav'
| Current file structure :
| ======================
|     <JUNK> chunk [offset: 0000000012; size: 0000000092 + 4 + 4;]
|     <bext> chunk [offset: 0000000112; size: 0000000602 + 4 + 4;]
|     <fmt > chunk [offset: 0000000722; size: 0000000040 + 4 + 4;]
|     <minf> chunk [offset: 0000000770; size: 0000000016 + 4 + 4;]
|     <elm1> chunk [offset: 0000000794; size: 0000000214 + 4 + 4;]
|     <data> chunk [offset: 0000001016; size: 0003036447 + 4 + 4;] odd w/ pad
|     <regn> chunk [offset: 0003037471; size: 0000000092 + 4 + 4;]
|     <umid> chunk [offset: 0003037571; size: 0000000024 + 4 + 4;]
|     <DGDA> chunk [offset: 0003037603; size: 0000020580 + 4 + 4;]
|
| [i] Checking <data> chunk..  ok.
|
| File ok.

SoX refuses to read dolle.wav at all, even with --ignore-length flag, throwing an error: “WAVE header error: cbSize inconsistent with fmt size”. FFMPEG comes to rescue in a way.

$ ffmpeg -i dolle.wav -c:a pcm_s24le dolle.ffmpeg.wav

$ wavfix dolle.ffmpeg.wav

> Processing 'dolle.ffmpeg.wav'
| Current file structure :
| ======================
|     <fmt > chunk [offset: 0000000012; size: 0000000040 + 4 + 4;]
|     <LIST> chunk [offset: 0000000060; size: 0000000062 + 4 + 4;]
|     <data> chunk [offset: 0000000130; size: 0003036447 + 4 + 4;] odd w/ pad
|
| [i] Checking <data> chunk..  ok.
|
| File ok.

$ flac -7 --keep-foreign-metadata dolle.ffmpeg.wav
dolle.ffmpeg.wav: wrote 1686800 bytes, ratio=0,556

$ flac -d --keep-foreign-metadata dolle.ffmpeg.flac -o dolle.ffmpeg.flac-d.wav
dolle.ffmpeg.flac: done

$ b3sum dolle.ffmpeg.wav dolle.ffmpeg.flac-d.wav
620af01885809c2b11eef54b71ae3d863e9427400a040f616a37f295e70da091  dolle.ffmpeg.wav
620af01885809c2b11eef54b71ae3d863e9427400a040f616a37f295e70da091  dolle.ffmpeg.flac-d.wav

Do I read this correctly?

* ProTools does weird things
* WAVfix chooses not to touch it because the mistake is made by a big player and sure as hell they are not going to take the blame for doing any fixes that could make ProTools reject the file
* SoX refuses to read them
?

A quick googling finds this: https://stackoverflow.com/questions/26169678/why-certain-wav-files-cannot-be-decoded-in-firefox
Some issues with a ProTools .wav file, but here SoX reads it.

Not sure what to do here. This file does not comply with any WAVE specification I know of, but I guess ProTools is not something I can just ignore.

I agree.
But I think it is better to reject files upon encoding.

Better yet, reject them with explanation why. Big players, huh, not adhering to standards, I wonder if they are the first...

I believe the problem is not with Pro Tools generated files, but with FLAC itself, which processes non-audio data in a picky way rather than storing it.

Perhaps, there was a reason for this design (https://hydrogenaud.io/index.php/topic,113324.msg1041202.html#msg1041202), but the ordinary user expects FLAC to be something like playable ZIP. As we can see, WavPack, Monkey Audio and TAK are more in line with this expectation. Over time, FLAC got --keep-foreign-metadata flag to preserve non-audio data. But a) it is still considered experimental: “NOTE: --keep-foreign-metadata is a new feature; make sure to test the output file before deleting the original”, and b) it causes confusion, because some files can be compressed without it, but not with it (see the log below). In particular, I stumbled upon this ticket (https://trac.ffmpeg.org/ticket/1158) on FFMPEG issue tracker created 12 years ago (in 2012) mentioning Gallery Metacorder (http://www.gallery.co.uk/metacorder/intro.html) app, which also stores non-audio data in WAV. Guess what? FLAC refuses to process it. Guess how the ordinary user reacts? “Aggrrhh!”

Is there really no way to teach FLAC to be less picky and just store non-audio data?

$ ffprobe -loglevel error -show_entries format_tags -of json metacorder.wav
{
    "format": {
        "tags": {
            "comment": "gSCENE=66a\r\ngTAKE=002\r\ngTAPE=007\r\ngNOTE=Circle \r\ngUBITS=00000000\r\n",
            "encoded_by": "Gallery Metacorder",
            "date": "2005:08:04",
            "creation_time": "16:55:54",
            "time_reference": "4090608000",
            "coding_history": ""
        }
    }
}

$ wavfix metacorder.wav
> Processing 'metacorder.wav'
| Current file structure :
| ======================
|     <bext> chunk [offset: 0000000012; size: 0000000858 + 4 + 4;]
|     <fmt > chunk [offset: 0000000878; size: 0000000016 + 4 + 4;]
|     <data> chunk [offset: 0000000902; size: 0000210154 + 4 + 4;]
|     <iXML> chunk [offset: 0000211064; size: 0000002839 + 4 + 4;] odd w/ pad
|     <iXTC> chunk [offset: 0000213911; size: 0000000020 + 4 + 4;]
|
| [i] Checking <data> chunk..  ok.
|
| File ok.

$ flac metacorder.wav
flac 1.4.3
metacorder.wav: WARNING: skipping unknown chunk 'bext' (use --keep-foreign-metadata to keep)
metacorder.wav: WARNING: there is data trailing the audio data. Use --keep-foreign-metadata or --ignore-chunk-sizes to keep it
metacorder.wav: wrote 34136 bytes, ratio=0,162

$ flac -f --keep-foreign-metadata metacorder.wav
metacorder.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

I believe the problem is not with Pro Tools generated files, but with FLAC itself, which processes non-audio data in a picky way rather than storing it.
....

Not so. There is no requirement for tools such as FLAC to store non-standard chunks created by applications such as ProTools. For chunks that are non-standard an application is only required to skip over them. The ordinary user does not expect FLAC to perform as you suggest since the ordinary user is unlikely to be using such files. The non-standard chunks created by ProTools are for ProTools consumption and nothing else simply because nothing, or little, else knows what to do with them. The fact that WavPack and Monkeys Audio choose to preserve all the chunk information is a generous feature that is neither required nor expected. You are suggesting a problem where none exists. FLAC performs, in this regard, as required and expected.

The ordinary user does not know that there are standard and non-standard chunks. He is told that there is a new way to make an audio file not only take up less space and decompress it back while preserving its properties (as in the case of omnivorous ZIP-like tools (https://hydrogenaud.io/index.php/topic,113324.msg1041222.html#msg1041222)), but also be playable on various devices. “That's so cool! Let's start by squeezing those audio lectures.” However, at present FLAC takes advantage of people's ignorance, keeping only audio data lossless, the rest is a game of chance. I break into a cold sweat thinking about how many times over the years FLAC has skipped non-standard chunks that might be helpful to me or to someone else in the future (author's comments, voice recorder tags, software helpers, coordinates, Easter eggs, etc). I can vividly imagine how the announcer Nick Dolle writes to me and asks, "Buddy, my website is down, but I heard you still have a demo of my voice, could you share it, please?" I send him FLAC, then he raises an eyebrow in surprise: “Where did the tag with the recording date go?” Oh, it was washed away.

(https://i4.imageban.ru/out/2024/03/16/b8025f918bffdbb482ab7c95661f8780.webp)

Is there really no way to teach FLAC to be less picky and just store non-audio data?

Absolutely. Download the source code and do it yourself. Nobody is here to cater for your every little whim.

FLAC was designed as a compressor for audio, not for files. The FLAC FAQ has more than one item about that. Quoting from https://xiph.org/flac/faq.html :

I compressed a WAVE file to FLAC, then decompressed to WAVE, and the two weren't identical. Why?
I compressed a WAVE file to FLAC and it said "warning: skipping unknown sub-chunk LIST". Why?
I decoded a FLAC file and the WAVE is 2 bytes shorter than the original. Why?

WavPack was designed to compress files (and so was Monkey's, which took its inspiration from WavPack) - FLAC was not, that is an auxilliary feature for power users. Even if the reference implementation has been able to store RIFF chunks for many years, it seems the documentation is even more recent than the most official release.

Quite possibly I should not be the first to knock other users over obsession about corner cases, but put bluntly: using FLAC to restore anything but audio is for power users who are willing to read manuals (and at least the FAQ).
My issue with the reference implementation's handling, is that by now it pretends to do something it cannot.

@Porcus, I understand that the origins of FLAC and the trio of other popular lossless encoders have different ideas. My question is how much this is set in stone and prevents FLAC from becoming more flexible with respect to non-audio data. For example, Dominik Reichl, the developer of Keepass password manager, added support for other encryption algorithms without insisting on using Rijndael (AES), which is still considered a standard somewhere in North America, thereby showing flexibility towards users. So I posted metacorder.wav (https://hydrogenaud.io/index.php/topic,123176.msg1041237.html#msg1041237) that uses “non-standard” iXML chunk. What is it? Wikipedia article (https://en.wikipedia.org/wiki/IXML) and official specification (http://www.gallery.co.uk/ixml/) claim it popped up not yesterday, but in the mid-2000s. Will something terrible happen if FLAC learns to store it? Are there any technical or ideological obstacles? Or the will to progress gave way to laziness? Surely there is someone here who is familiar with how decisions about further development are made.

Powerful and intuitive Metadata functionality, including Project, SoundRoll, scene, take, track labels and indexes, recording notes. User friendly system automatically updates scene and take using US, UK or generic slating systems. Metadata can be edited before, during or after recording. Metadata stored in several formats including bext and the revolutionary new iXML format for seamless post production..

My question is how much this is set in stone and prevents FLAC from becoming more flexible with respect to non-audio data.

First of all, can you please stop hijacking threads? If you have a problem, start a new thread instead of necroposting or posting something completely irrelevant for the current thread?

Second, can you please stop calling developers lazy? For most of use this is unpaid work. Especially what you said about FSLAC sounded particularly condescending.

Back to the matter at hand, the FLAC reference command line tool can store and restore foreign metadata. It currently has the limitation that it will not restore a non-standard fmt chunk. Why not? Because that contains data necessary to parse the file, and FLAC currently writes its own based on its own metadata, to assure the file stays valid. This might loosen in the future, but I think mindlessly writing invalid data unchecked is bad behaviour. As you have seen, many tools simply reject this weird Protools fmt chunk.

You seem to have a problem with FLAC not storing foreign metadata by default. FLAC warns the user when such metadata is present, suggesting the use of an extra option. Recently FLAC got the option --keep-foreign-metadata-if-present that you can leave always on without getting an error. But yes, it is not on by default.

The ordinary user does not know that there are standard and non-standard chunks.

Standards are for users' own good, even if they don't realize that. What you are arguing for is applying Postel's law (https://en.wikipedia.org/wiki/Robustness_principle): "be conservative in what you do, be liberal in what you accept from others", which has a fair amount of criticism. Yes, short-term the user may be satisfied that his file was processed without errors but eventually they will start complaining again that app A interpreted their broken file differently than app B.

I can vividly imagine how the announcer Nick Dolle writes to me and asks, "Buddy, my website is down, but I heard you still have a demo of my voice, could you share it, please?" I send him FLAC, then he raises an eyebrow in surprise: “Where did the tag with the recording date go?”

Then you should raise your eyebrow and say: "Nick, buddy, where are your backups? Oh, you haven't done any? So maybe lower your eyebrow and be glad that you were able to recover anything at all."

For example, Dominik Reichl, the developer of Keepass password manager, added support for other encryption algorithms

Would he add support for files produced by apps that implemented those other algorithms incorrectly?

So I posted metacorder.wav (https://hydrogenaud.io/index.php/topic,123176.msg1041237.html#msg1041237) that uses “non-standard” iXML chunk. What is it? Wikipedia article (https://en.wikipedia.org/wiki/IXML) and official specification (http://www.gallery.co.uk/ixml/) claim it popped up not yesterday, but in the mid-2000s. Will something terrible happen if FLAC learns to store it? Are there any technical or ideological obstacles?

The problem with your file is not that it has an unknown iXML chunk. iXML chunks, or any other chunks, are fine and that's what --keep-foreign-metadata is for.

The problem with your file is that the chunk is stored incorrectly. The standard for RIFF says that if the size of the data in the chunk is odd then it needs a pad-byte (https://en.wikipedia.org/wiki/Resource_Interchange_File_Format#Explanation) to make it even. The iXML chunk in your file is missing that byte:

00034390  20 20 20 20 20 20 20 69  58 54 43 14 00 00 00 00  |       iXTC.....|

You see that the next chunk after "iXML" starts at odd position. That makes it an invalid RIFF file. Insert a byte there:

00034390  20 20 20 20 20 20 20 00  69 58 54 43 14 00 00 00  |       .iXTC....|

and it will make it less invalid RIFF file (in attachment). The size in the RIFF chunk is still/now wrong, 213'931 instead of 213'932, but apparently that's something flac can deal with:

]$ flac -f --keep-foreign-metadata metacorder.fix.wav
NOTE: --keep-foreign-metadata is a new feature; make sure to test the output file before deleting the original.

flac 1.4.3
Copyright (C) 2000-2009  Josh Coalson, 2011-2023  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

WARNING: RIFF chunk size of file metacorder.fix.wav does not agree with filesize
metacorder.fix.wav: wrote 37970 bytes, ratio=0.181

Or the will to progress gave way to laziness?

To you it would be progress, to others it would be regression.

You seem to have a problem with FLAC not storing foreign metadata by default.

To you it would be progress, to others it would be regression.

Me? I had problems in my youth, and now, following Jack Ma's advice (https://www.youtube.com/watch?v=DByuo1UBiW0), I am concerned with what is of interest to a wider range of users. So I report not my problems so much as I report missed opportunities to make tools better overall. It is quite common to miss such things due to resting on your laurels, déformation professionnelle (https://en.wikipedia.org/wiki/D%C3%A9formation_professionnelle) and the echo chamber (https://en.wikipedia.org/wiki/Echo_chamber_(media)). In this case, if the competitors such as WavPack, Monkey Audio and TAK proved that it is possible to become even more lossless by storing not only the audio, but also the accompanying data without warnings and errors, then why not go this route? Tell me about the obstacles, if any, as I don't see how this can be a regression.

Look, MP3 encoder Helix (https://hydrogenaud.io/index.php/topic,113324.msg1041258.html#msg1041258), which processed 16-bit yesterday, now supports up to 32-bit floating-point and Unicode.

First of all, can you please stop hijacking threads? Second, can you please stop calling developers lazy?

Spoiler (click to show/hide)

First, there is a difference between necromancing the relevant thread that is old - and just picking one that happens to be about the same software.

As for software ... there are different tools for different uses.

In this case, if the competitors such as WavPack, Monkey Audio and TAK proved that it is possible to become even more lossless by storing not only the audio, but also the accompanying data without warnings and errors, then why not go this route?

Because well, after twenty-something years of both WavPack, Monkey's and FLAC, there hasn't been enough need for FLAC to do that job? Proof: FLAC has become the most dominant lossless format still.

FLAC doesn't support floating-point signals either. For those very few ones (I think there should be more of them, better do float in processing ...), I use WavPack. And if I want to preserve at file - WavPack it is, it is designed to do that. To the extent that old wvunpack decoders can decode to a file format it knows nothing about.

What I think is troublesome about reference flac's handling here, is not that it cannot handle these files, it is that it let itself be fooled to thinking it can. "I am not designed to do this!" is acceptable ... but annoying when the files come from a big player and you must expect there to be a lot of them around.

ALAC is a special case. Both qAAC --alac and FFMPEG -acodec alac encode dolle.wav without warnings or errors, but the decoded WAV is different from the original.

Was there any difference PCM-wise?
ALAC doesn't keep "foreign metadata" (non-PCM chunks in the WAV file), so WAV file shouldn't be identical and that's normal.

Tell me about the obstacles, if any, as I don't see how this can be a regression.

I'll summarize what happens
1. Protools generates an invalid WAVE file.
2. FLAC stores this invalid metadata when invoked with --keep-foreign-metadata
3. FLAC restores a valid/fixed WAVE file when invoked with -d --keep-foreign-metadata
4. FLAC returns an error that the restored file is not identical to the original file. All foreign metadata is there, but the fmt chunk has been corrected

I could do a few things:
1. Do the same thing as WavPack/Monkeys/TAK, but then the restored file is invalid. I would consider this a regression, FLAC should not output invalid files
2. Return a more comprehensible warning instead of an error, explaining that the the fmt chunk has been altered but all other metadata is correct
3. Warn users on encoding that the file is invalid and restoring might not give back an identical file

So, I do not think fixing this is straightforward and haven't yet made a decision

Your solution item #3 - the warning - can be done independently of the others I think?


Also, concerning 4:

4. FLAC returns an error that the restored file is not identical to the original file. All foreign metadata is there, but the fmt chunk has been corrected

Will all foreign metadata be the same? I understood @danadam's reply #19 that there was at least one chunk of odd byte length, and so the writing application should have byte-padded it. But if the flac decoder does that, then it alters not only <fmt >, but the metadata ...
... in a way that likely doesn't matter, because likely the writing application is just too stupid to know the padding byte, and so likely it won't notice that it is there - but I'm not sure one can trust that.


As goes Pro Tools specifically ... https://avidtech.my.salesforce-sites.com/pkb/articles/en_US/faq/en388231

@ktf You've summarized the situation with FLAC in relation to dolle.wav (https://hydrogenaud.io/index.php/topic,125651.0.html). But to illustrate the scale of the problem of dealing with “invalid” sources, I additionally published metacorder.wav (https://hydrogenaud.io/index.php/topic,125651.msg1041237.html#msg1041237) (analyzed later by @danadam) that is not compressed at all with the --keep-foreign-metadata flag. Both of these sources are not created for the sake of chicanery, but are taken from real life apps (ProTools and Metacorder respectively).

WavPack, Monkey Audio and TAK find a way to store the user's belongings in a more compact form, optimizing the space they occupy, while FLAC also tries to evaluate and fix them, effectively becoming a warden. Imagine that, instead of putting the scattered pages of the King James Bible into one pile, you get these pages with omissions due to the latest findings like the Dead Sea Scrolls (https://en.wikipedia.org/wiki/Dead_Sea_Scrolls) after compression. And the works of Dostoevsky in Russian language are not compressed at all due to US government sanctions. I believe the compression should be FLAC priority, leaving non-audio inconsistencies with the fleeting standards at the user's discretion, of which he should be unobtrusively informed.

Was there any difference PCM-wise? ALAC doesn't keep "foreign metadata".

I don't know, sir. My intention was to describe the behavior of apps that claim to be lossless encoders. Accordingly, if the restored audio file is not identical to the original (which is confirmed by verification of checksums), I consider it a loss.

*sighs* ...

Behind all this pseudo-intellectualism ... go get yourself a torx key AND a hex key and use them as appropriate, rather than trying to force one into being the other.

And now you have wasted the refalac dev's time here too, you could in the very least confirm that the audio is lossless.
(ALAC doesn't even come with a file format, it uses container formats that are already around.)

Quote from: Kraeved on 2024-03-16 22:59:40

Tell me about the obstacles, if any, as I don't see how this can be a regression.

I'll summarize what happens
1. Protools generates an invalid WAVE file.
2. FLAC stores this invalid metadata when invoked with --keep-foreign-metadata
3. FLAC restores a valid/fixed WAVE file when invoked with -d --keep-foreign-metadata
4. FLAC returns an error that the restored file is not identical to the original file. All foreign metadata is there, but the fmt chunk has been corrected

I could do a few things:
1. Do the same thing as WavPack/Monkeys/TAK, but then the restored file is invalid. I would consider this a regression, FLAC should not output invalid files
2. Return a more comprehensible warning instead of an error, explaining that the the fmt chunk has been altered but all other metadata is correct
3. Warn users on encoding that the file is invalid and restoring might not give back an identical file

So, I do not think fixing this is straightforward and haven't yet made a decision

My suggestion, FWIW, would be to do (1.), above. Protools, and any other apps that create non-standard chunks, including mal-formed fmt chunks, have been doing so for some considerable period of time apparantly with no objections from users of those tools, or any other apps that may use those files. It could be argued that it is not the job of FLAC to correct what may be incorrect but clearly seems not to cause a problem. That may hurt intellectually and aesthetically but is surely the pragmatic solution? Just my thoughts. ;)

Quote from: nu774 on 2024-03-17 08:22:27

Was there any difference PCM-wise? ALAC doesn't keep "foreign metadata".

I don't know, sir. My intention was to describe the behavior of apps that claim to be lossless encoders. Accordingly, if the restored audio file is not identical to the original (which is confirmed by verification of checksums), I consider it a loss.

Thank you for an indirect answer.
Since you have mentioned ffmpeg + ALAC and you don't seem to know this trivial fact, I have to tell you that ffmpeg NEVER bother to preserve source container chunks on conversion no matter what audio codec you choose.
Same goes for "lossless" audio format conversion via foobar2000 or other general audio converters that decode from one format and encode to other.
Lossless audio codec simply means encoding audio data (usually PCM) without loss.  Input format can be anything... it may be RTP packets coming from the internet, or audio track in MMT/TLV broadcast.  Lossless coder can transcode them without loss in terms of PCM data and not more than that. What you want can be achieved in VERY limited scenarios or use cases. Outside of there, it's simply impossible or nonsense.

I believe the compression should be FLAC priority, leaving non-audio inconsistencies with the fleeting standards at the user's discretion, of which he should be unobtrusively informed.

This isn't really fleeting. The requirement of IFF chunks being an even number of bytes (which the metacorder.wav file violates) dates back to 1985.

I will consider adding handling to the flac command line tool for this kind of malformed files. Maybe I'll add an option to disable 'fixing' of files. But it is rather low on my priority list. I've opened an issue here: https://github.com/xiph/flac/issues/680

I will consider adding handling to the flac command line tool for this kind of malformed files. Maybe I'll add an option to disable 'fixing' of files.

Thanks for taking a step from tyranny to freedom, @ktf.
I stand with @john33, “it is not the job of FLAC to correct what may be incorrect”.

I have to tell you that ffmpeg NEVER bother to preserve source container chunks on conversion

Without a doubt, FFMPEG has a room for improvement as well. At this stage, it at least preserves dolle.wav metadata and allows to compress metacorder.wav, which FLAC itself does not do with --keep-foreign-metadata flag. We can talk about how to bring that data back to WAV intact on their issue tracker, when the expected change happens in the reference FLAC encoder. For example, in 2011 they implemented -write_bext 1 flag to preserve that chunk.

Right now my heart and mind are here.

 $ ffprobe -loglevel error -show_entries format_tags -of json dolle.wav
{
    "format": {
        "tags": {
            "encoded_by": "Pro Tools",
            "originator_reference": "4ut5zBTw#CdaaaGk",
            "date": "2017-12-1",
            "creation_time": "22:06:31",
            "time_reference": "158760000"
        }
    }
}

$ flac --keep-foreign-metadata dolle.wav
flac 1.4.3
dolle.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=24
dolle.wav: wrote 1717062 bytes, ratio=0,565

$ ffprobe -loglevel error -show_entries format_tags -of json dolle.flac
{
    "format": {
    }
}

$ ffmpeg -loglevel error -i dolle.wav -bitexact -map_metadata 0 dolle.flac -y

$ ffprobe -loglevel error -show_entries format_tags -of json dolle.flac
{
    "format": {
        "tags": {
            "encoded_by": "Pro Tools",
            "originator_reference": "4ut5zBTw#CdaaaGk",
            "date": "2017-12-1",
            "creation_time": "22:06:31",
            "time_reference": "158760000"
        }
    }
}

-----------------------------------------------------------------------------------

$ mediainfo metacorder.wav
General
Complete name                            : metacorder.wav
Format                                   : Wave
Format settings                          : PcmWaveformat
File size                                : 209 KiB
Duration                                 : 2 s 189 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 782 kb/s
Producer                                 : Gallery Metacorder
Description                              : gSCENE=66a / gTAKE=002 / gTAPE=007 / gNOTE=Circle  / gUBITS=00000000
Encoded date                             : 2005:08:04 16:55:54

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 2 s 189 ms
Bit rate mode                            : Constant
Bit rate                                 : 768 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 205 KiB (98%)

$ flac --keep-foreign-metadata metacorder.wav
flac 1.4.3
metacorder.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

$ ffmpeg -loglevel error -i metacorder.wav -bitexact -map_metadata 0 metacorder.flac -y

$ mediainfo metacorder.flac
General
Complete name                            : metacorder.flac
Format                                   : FLAC
Format/Info                              : Free Lossless Audio Codec
File size                                : 42.0 KiB
Duration                                 : 2 s 189 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 157 kb/s
Description                              : gSCENE=66a / gTAKE=002 / gTAPE=007 / gNOTE=Circle  / gUBITS=00000000
Recorded date                            : 2005:08:04
encoded_by                               : Gallery Metacorder
creation_time                            : 16:55:54
time_reference                           : 4090608000

Audio
Format                                   : FLAC
Format/Info                              : Free Lossless Audio Codec
Duration                                 : 2 s 189 ms
Bit rate mode                            : Variable
Bit rate                                 : 126 kb/s
Channel(s)                               : 1 channel
Channel layout                           : M
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Compression mode                         : Lossless
Stream size                              : 33.7 KiB (80%)
Writing library                          : ffmpeg
MD5 of the unencoded content             : 81A6AD258A5C39DE55DF2041727B9E3B

I’d like to clarify a couple things. In the case of metacoder.wav the reason that WavPack is able to handle this file without complaint is because WavPack only parses RIFF chunks up to the audio chunk. For any data past the audio, WavPack simply copies it verbatim without even parsing it or checking the length.

As for the dolle.wav file, the problem is that the cbSize field is wrong (as was said) and in previous versions of WavPack I basically ignored that field (it’s really kind of redundant). I ran into an issue recently where I was mistakenly using information in the format header even though the cbSize indicated that it wasn’t there, so I fixed that here (https://github.com/dbry/WavPack/commit/07ffb331d23c16e2d2979370441f63ea722760bc). But, if the format header was any bigger than the standard extended version, I would have errored out on that too just like FLAC does. We can only anticipate so many surprises.

In summary, for both of these cases, WavPack handles them “correctly” because I am lazy, not because I’m somehow more concerned about the user’s experience. And it should be a lesson that what appears initially to be an obvious shortcoming may, when examined more closely, actually be the most reasonable compromise given the intended application (Josh never intended FLAC to be a file compressor, whereas WAV is right in WavPack’s name).

Things are often more complex and subtle than they at first appear, and the true lazy developers are the ones that create files that so blatantly violate the specs they are purporting to follow. I mean, that metacoder.wav file has an odd length, which makes it an invalid WAV file right out of the gate. It would not be completely unreasonable for a file reader to see that and refuse to even open it!

Dear @MonkeysAudio and @enzo, could you please shed some light, like @bryant aka WavPack author did, on how Monkey Audio manages to process such non-standard audio files as dolle.wav (https://hydrogenaud.io/index.php/topic,125651.0.html) and metacorder.wav (https://hydrogenaud.io/index.php/topic,125651.msg1041237.html#msg1041237) without errors and return them bit-perfect?

$ mac dolle.wav auto -c2000
--- Monkey's Audio Console Front End (v 10.55) (c) Matthew T. Ashland ---
Compressing (normal)...
Progress: 100.0% (0.0 seconds remaining, 0.3 seconds total)
Success

$ mac dolle.ape dolle.restored.wav -d
Decompressing...
Progress: 100.0% (0.0 seconds remaining, 0.4 seconds total)
Success

$ mac metacorder.wav auto -c2000
Compressing (normal)...
Progress: 100.0% (0.0 seconds remaining, 0.0 seconds total)
Success

$ mac metacorder.ape metacorder.restored.wav -d
Decompressing...
Progress: 100.0% (0.0 seconds remaining, 0.0 seconds total)
Success

$ b3sum *.wav
5bc927c245396131cf6e4dccd46c6399423c05ed0bd67d2209446217e0bb93d2  dolle.restored.wav
5bc927c245396131cf6e4dccd46c6399423c05ed0bd67d2209446217e0bb93d2  dolle.wav
4accf01292e8d2ed9ef9b2047c13841cebaeec59b22570ae8694ef88232304c4  metacorder.restored.wav
4accf01292e8d2ed9ef9b2047c13841cebaeec59b22570ae8694ef88232304c4  metacorder.wav

It isn't so hard to create a lossless file compressor. But for an audio file format, you also want to serve a couple of more purposes, like for example playing it.
And so you want to know precisely where the audio starts and ends, and what format it is. That is in the headers of the [WAVE or AIFF or whatever] source file, and that means you not only have to compress the file, but to understand part of its content.
Headers have to be processed into the correct information - which isn't straightforwards if they don't conform to If the WAVE or AIFF specification. If those file headers are not reliable, you could at worst get that wrong, and the following could happen:

* Your compression algorithm, which heavily depends on audio samples being serially correlated, would not give smaller sizes because it doesn't know where an individual sample starts and ends.
* If you try to play back the audio part, it will sound like noise.

Hypothetical you say? No, I've had Monkey's do that to weirdo AIFF files. Monkey's would uncompress them bit-exactly, but they would sound like noise. From memory: For one of them, the information was in the headers in a way that it could be processed in the next Monkey's release. For the other, the next Monkey's release did the right thing and rejected it.

You should also be aware that e.g. WAVE is a container format that need not contain PCM. It can for example contain MP3. The encoder should reject those files - not because they are non-compliant, but because they are "out of scope". This is not what I was constructed to do, and I won't just pretend.

Non-compliant files can fool the encoder into processing something it should reject. The files can also be flawed in such a way that everything goes well - not only reversing the algorithm to restore the file, which is in some sense the easier part.


With that said, the following is above my paygrade: I simply don't have the knowledge to tell why this approach has not been chosen, though I know it has been "attempted":
Upon playback, the decoder serves not the audio, but the unpacked file. It is a WAVE file, maybe non-compliant, maybe the encoder was wrong about the content, whatever - that is the player's business, not the compessed format's. Player has to treat it as if it were the original file. Any other information - like seekpoints - are an auxilliary service.

A simple illustration: Try to encode wrong.wav to flac/ape/wavpack, then don't just decompress to .wav and compare checksum, but play/analyze/process the encoded files via different software (e.g. foobar, MPC-HC, Audacity, SoX, Spek) or hardware like DAPs and streamers. Also, listen to wrong.wav in compressed and uncompressed form via stereo headphones, don't just look at graph and stats.

Use correct.wav as a reference, but keep in mind that such a reference won't exist in real-life situations.

I use flac, wavpack and sometimes ape, apart from format support like DSD and float, if for some reasons I need to preserve the whole file, I will use wavpack, for example, keeping a bit-perfect specimen of a potentially malformed .wav file for debugging some programs I write.

Another reason to use wavpack is for example, Audition and Reaper can show the stored markers and regions within a wavpack file without using wvunpack. With flac, even with --keep-foreign-metadata, I still need an intermediate .wav file to get the markers and regions back. Opening a flac file directly in Audition and Reaper these info won't show up.

Apart from the observed behaviour of wrong.wav posted above when using the APE/FLAC/WavPack command line tools and various 3rd party decoder/players, I also found that even though there is only one byte of difference between correct.wav and wrong.wav, 7z achieved a much smaller file size when compressing correct.wav and wrong.wav separately (652KB vs 713KB). Also, I cannot reduce file size by putting these two files into a solid 7z. It seems that 7z used different methods to compress correct.wav and wrong.wav so solid compression doesn't work. I tried different settings on the 7z GUI but still cannot find a way to defeat this behaviour, don't know if it can be overridden by using the 7z command line tool or not.

Anyway, good job on flac.exe to reject wrong.wav.

7-zip has special treatment for .wav : https://www.7-zip.org/7z.html
Putting my money on it not recognizing the "wrong" as .wav

Putting my money on it not recognizing the "wrong" as .wav

Well, It's quite natural that 7z compressed "correct.wav" more efficiently even if 7z recognized both of them as WAV, since it can exploit the similarity of channels on correct.wav (which looks like a stereo file), but not on wrong.wav (which looks like a mono file).

Now it works ;)

Quote from: Porcus on 2024-03-19 11:42:39

Putting my money on it not recognizing the "wrong" as .wav

Well, It's quite natural that 7z compressed "correct.wav" more efficiently even if 7z recognized both of them as WAV

And it seems it does. Looking at the compression model used in test.7z, it uses delta:2 on one and delta:4 on the other. Apparently that means two bytes per sample and four bytes per sample (the latter meaning, stereo and four bytes per stereo sample).

@ktf
Is it true that flac's raw mode guarantees bit-perfectness at file level?

How about this? When the command-line tool encounters files with potentially malformed metadata, as long as the audio data encoding format can be automatically detected, offer an option to encode as raw and automatically insert the required raw mode parameters and suggests the required decoder version (e.g. for 32-bit int support), and shows the required commands to decode the file properly.

This mechanism can force users to manually invoke the encoding and decoding processes, to ensure they understand what they are doing and what risks they should face, for example, compatibility with 3rd party tools.

For example, --keep foreign-metadata can preserve the attached .wav file bit-perfect, but putting this .wav file into "SoX Spectogram.cmd" below will cause SoX to crash. So the mechanism mentioned above can make sure that users understand the crash is not flac's fault.

Updated 'SoX Spectogram' folder with the SoX binary from above
https://www.mediafire.com/file/u9uhy0nbotrp8hx/SoX+Spectogram.7z/file

@ktf
Is it true that flac's raw mode guarantees bit-perfectness at file level?

No, flac's raw mode is for reading headerless PCM files. You could of course use it to store WAV/AIFF/W64, but that is not what it was meant to do.

FLAC can store 'foreign metadata'. When this is activated on both encoding and decoding, the tool will try to store and restore the file (WAV/AIFF/W64) as it was encoded. As you probably know, FLAC tells you the following:

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

So no, the keep foreign metadata option does not guarantee anything.

How about this? When the command-line tool encounters files with potentially malformed metadata, as long as the audio data encoding format can be automatically detected, offer an option to encode as raw and automatically insert the required raw mode parameters and suggests the required decoder version (e.g. for 32-bit int support), and shows the required commands to decode the file properly.

No. FLAC needs to know what is metadata and what is audio, otherwise you can have noise playback. So, FLAC will not store files in which it cannot detect what is audio and what isn't.

@bennetng Pushing helicopter parenting (https://en.wikipedia.org/wiki/Helicopter_parent) makes no good, because the number of apps that create audio files in a non-standard way and new specifications that extend or reinterpret previous ones is only growing, leaving FLAC heuristic behind. For example, read a story about the mess that was made, trying to make WAV larger than 4 GB, told by the user (http://blog.bjornroche.com/2009/11/wave64-vs-rf64-vs-caf.html) and the developer (http://www.mega-nerd.com/erikd/Blog/CodeHacking/libsndfile/rf64_specs.html). The fact that it is possible to craft a file that causes troubles, like a ZIP bomb (https://en.wikipedia.org/wiki/Zip_bomb), does not mean that compressors must have a mechanism to reject files — this is a task for watchdog apps, such as antiviruses, which regularly annoy with false positives (remember all those noobs running around yelling there is a red spot in the VirusTotal report, which calls into question the credibility of the well-known developer). Compression resembles vacuum packaging (https://en.wikipedia.org/wiki/Vacuum_packing) that pumps out excess air and enables you to store the contents in a more compact form, but the packaging is a transport medium (https://en.wikipedia.org/wiki/Viral_transport_medium) and is not responsible for the contents: you can put ordinary cheese (cheddar, gouda, parmesan), blue cheese with edible mold (dorblu, gorgonzola, roquefort) or poisoned cheese in it.

As for SoX, which has not been updated for more than 8 years, although it would be worth it (say, do you know that internal operations in it are still carried out without floating point (https://hydrogenaud.io/index.php/topic,101850.msg840016.html#msg840016)?), it draws a spectrogram without issues on my end, but I use RareWares version (https://www.rarewares.org/others.php) 14.4.2 x64 built on June 24, 2023,. Try updating yours.

Quote from: bennetng on 2024-03-20 04:30:12

@ktf
Is it true that flac's raw mode guarantees bit-perfectness at file level?

No, flac's raw mode is for reading headerless PCM files. You could of course use it to store WAV/AIFF/W64, but that is not what it was meant to do.

Quote

How about this? When the command-line tool encounters files with potentially malformed metadata, as long as the audio data encoding format can be automatically detected, offer an option to encode as raw and automatically insert the required raw mode parameters and suggests the required decoder version (e.g. for 32-bit int support), and shows the required commands to decode the file properly.

No. FLAC needs to know what is metadata and what is audio, otherwise you can have noise playback. So, FLAC will not store files in which it cannot detect what is audio and what isn't.

I am talking about using flac in the same way as 7z, so it means the files are not meant to be played in compressed form and they must be decoded. For example I attached a SoundFont file which flac does not understand but it contains compressible data:

encode:
flac woodwinds.sf2 --force-raw-format --endian=little --channels=1 --bps=16 --sample-rate=44100 --sign=signed

decode:
flac woodwinds.flac -d --force-raw-format --endian=little --sign=signed -o woodwinds-decode.sf2

In the case of .wav input, flac.exe should be able to parse what raw encoding and decoding settings to use when receiving the file.

For example, --keep foreign-metadata can preserve the attached .wav file bit-perfect, but putting this .wav file into "SoX Spectogram.cmd" below will cause SoX to crash. So the mechanism mentioned above can make sure that users understand the crash is not flac's fault.

Quote from: NetRanger on 2024-02-18 20:27:20

Updated 'SoX Spectogram' folder with the SoX binary from above
https://www.mediafire.com/file/u9uhy0nbotrp8hx/SoX+Spectogram.7z/file

I should also mention that foobar2000 shows strange duration when reading this file.

@bennetng Pushing helicopter parenting (https://en.wikipedia.org/wiki/Helicopter_parent) makes no good, because the number of apps that create files in a non-standard way and new specifications that extend or reinterpret previous ones is only growing, leaving FLAC heuristic behind. For example, read a story about the mess that was made, trying to make WAV larger than 4 GB, told by the user (http://blog.bjornroche.com/2009/11/wave64-vs-rf64-vs-caf.html) and the developer (http://www.mega-nerd.com/erikd/Blog/CodeHacking/libsndfile/rf64_specs.html). The fact that it is possible to craft a file that causes troubles, like a ZIP bomb (https://en.wikipedia.org/wiki/Zip_bomb), does not mean that compressors must have a mechanism to reject files — this is a task for a separate class of watchdog apps, such as antiviruses, which regularly annoy with false positives (remember all those noobs running around yelling there is a red spot in the VirusTotal report, which calls into question the credibility of the well-known developer). Compression resembles vacuum packaging (https://en.wikipedia.org/wiki/Vacuum_packing) that pumps out excess air and enables you to store the contents in a more compact form, but the packaging is a transport medium (https://en.wikipedia.org/wiki/Viral_transport_medium) and is not responsible for the contents: you can put ordinary cheese (cheddar, gouda, parmesan), blue cheese with edible mold (dorblu, gorgonzola, roquefort) or poisoned cheese in it.

First of all, generic archive formats like zip, rar and 7z are meant for generic purposes regardless of compression efficiency (e.g. they can even use the "store" method). These formats are designed to keep the data unchanged even if they contain malware etc.

For media formats, keep in mind that not everything is upgradable, or upgradable for free, especially hardware. Even on the software side, not everything is FOSS, and even in the FOSS scene, not every software receive update. Always remember that as long as you cannot code it yourself, you always rely on others to fulfill your wishes. You can wait indefinitely though, but probably not everyone can wait.

(say, do you know that internal operations in it are still carried out without floating point (https://hydrogenaud.io/index.php/topic,101850.msg840016.html#msg840016)?)

This is off topic and especially when flac also does not support float. But yeah, why don't you code it yourself?

@bennetng Code what? You came here with a WAV file (https://hydrogenaud.io/index.php/topic,125651.msg1041488.html#msg1041488) that supposedly causes SoX to crash, so I suggested you get the SoX version that might not crash. Not only suggested, but also provided a download link to save your time. After all, the less obsessed you are with issues not related to FLAC, the more attention you can pay to this topic about audio files created by well-known apps that are good enough to be played and edited for decades, but not good enough to be processed by FLAC in one or both directions. This topic has already been assessed as worthy of attention and a request for improvement (https://hydrogenaud.io/index.php/topic,125651.msg1041313.html#msg1041313) was created. Now all that remains is to pray and wish the developers strength and good spirits.

@bennetng Code what?

As for SoX, which has not been updated for more than 8 years, although it would be worth it (say, do you know that internal operations in it are still carried out without floating point (https://hydrogenaud.io/index.php/topic,101850.msg840016.html#msg840016)?)

Then why you mentioned the above? Why floating point support has anything to do with the crash when generating a spectrogram with that file? If you want floating point support, code it yourself.

You came here with a WAV file (https://hydrogenaud.io/index.php/topic,125651.msg1041488.html#msg1041488) that supposedly causes SoX to crash, so I suggested you update the version of SoX that might not crash. Not only suggested, but also provided a download link to save your time.

You have to read my previous post. The compile of SoX that Netranger provided is newer than the rarewares one you mentioned.
https://hydrogenaud.io/index.php?msg=1039602

I also tried to use the rarewares version you mentioned to generate a spectrogram with that file, no crash, but still cannot get any useful output, I got this:

Input File     : 'E:\download\junglede.wav'
Channels       : 2
Sample Rate    : 22050
Precision      : 16-bit
Duration       : 50:30:48.27 = 4009754427 samples ~ 1.36386e+07 CDDA sectors
File Size      : 140k
Bit Rate       : 6.17
Sample Encoding: 16-bit Signed Integer PCM

In:0.00% 00:00:00.00 [50:30:48.27] Out:0     [      |      ]        Clip:0    sox WARN wav: Premature EOF on .wav input file
In:0.00% 00:00:01.59 [50:30:46.68] Out:35.1k [!=====|=====!]        Clip:0

Don't ignore that I also mentioned foobar2000 shows strange duration with this file.

Prime example of garbage in garbage out.

@bennetng , did you mean for an option to specify the headers to be deleted and then specifying the signal like you specify raw? I think OptimFROG has such an option.
I don't see the point in using FLAC to raw-compress a header'ed WAVE file. Sure the format allows compression as raw, and for material that is not intended to be playable one can put sample rate = 0 in streaminfo - but I don't think it should be encouraged as a solution. It would just disguise a corrupted .wav as a valid but unplayable .flac, wouldn't it?

On to the wrong.wav, after having looked at it ... I don't know that WAVE specification, but https://learn.microsoft.com/en-us/windows/win32/api/mmeapi/ns-mmeapi-waveformat doesn't say anything about what information takes precedent in case of inconsistencies. Here we have nChannels=1, nBlockAlign=4 and wBitsPerSample=16, and so if we for the sake of discussion assume that no more than one of these is wrong: Is there anything that says that this file should be interpreted as
* mono with 4 bytes per channel, i.e. wBitsPerSample is wrong and should be replaced by 32?
* 4 bytes per channel and 16 bits, i.e. nChannels is wrong and should be replaced by 2?
* mono with 16 wBitsPerSample, i.e. nBlockAlign is wrong and should be replaced by 2?
 
As far as I can tell,
flac, TAK and OptimFROG reject it
Monkey's roundtrip it to bit-exactly the original
WavPack is fooled by it, into unpacking it to something else
ffmpeg thinks it is ten seconds long, I assume then it disregards nBlockAlign although I didn't bother to listen
foobar2000 seems to do the same thing when playing it.

@bennetng , did you mean for an option to specify the headers to be deleted and then specifying the signal like you specify raw? I think OptimFROG has such an option.
I don't see the point in using FLAC to raw-compress a header'ed WAVE file. Sure the format allows compression as raw, and for material that is not intended to be playable one can put sample rate = 0 in streaminfo - but I don't think it should be encouraged as a solution. It would just disguise a corrupted .wav as a valid but unplayable .flac, wouldn't it?

For SoundFonts, it is what I can do without writing any code, so a proof of concept. The actual implementation of course can be improved for .wav (or even BW64!) depends on the will of developers. Take this as an idea instead of feature request. The important thing is to make sure users really understand what they are doing.

On to the wrong.wav, after having looked at it ... I don't know that WAVE specification, but https://learn.microsoft.com/en-us/windows/win32/api/mmeapi/ns-mmeapi-waveformat doesn't say anything about what information takes precedent in case of inconsistencies. Here we have nChannels=1, nBlockAlign=4 and wBitsPerSample=16, and so if we for the sake of discussion assume that no more than one of these is wrong: Is there anything that says that this file should be interpreted as
* mono with 4 bytes per channel, i.e. wBitsPerSample is wrong and should be replaced by 32?
* 4 bytes per channel and 16 bits, i.e. nChannels is wrong and should be replaced by 2?
* mono with 16 wBitsPerSample, i.e. nBlockAlign is wrong and should be replaced by 2?

There is also a "bytes per second" field to consider.

On to the wrong.wav, after having looked at it ... I don't know that WAVE specification, but https://learn.microsoft.com/en-us/windows/win32/api/mmeapi/ns-mmeapi-waveformat doesn't say anything about what information takes precedent in case of inconsistencies. Here we have nChannels=1, nBlockAlign=4 and wBitsPerSample=16, and so if we for the sake of discussion assume that no more than one of these is wrong: Is there anything that says that this file should be interpreted as
* mono with 4 bytes per channel, i.e. wBitsPerSample is wrong and should be replaced by 32?
* 4 bytes per channel and 16 bits, i.e. nChannels is wrong and should be replaced by 2?
* mono with 16 wBitsPerSample, i.e. nBlockAlign is wrong and should be replaced by 2?

There is a consistent interpretation (i.e., assuming no values are "wrong"), which is mono 16-bit samples in 32-bit containers. That's obviously kind of weird, but 24-bit samples in 32-bit containers is a thing (https://patchwork.kernel.org/project/alsa-devel/patch/61B6C8BF61481342973655BC8AB0FF355AE2F4@PGSMSX101.gar.corp.intel.com/), and this is how WavPack interprets it because I initially thought it would be a good idea to handle those cases. Unfortunately, since there are no files like this and the spec kinda says that it's not a thing (or how the sample would be justified, if it was a thing), this "feature" doesn't get exercised and doesn't work now (if it ever did).

  
As far as I can tell,
flac, TAK and OptimFROG reject it
Monkey's roundtrip it to bit-exactly the original
WavPack is fooled by it, into unpacking it to something else
ffmpeg thinks it is ten seconds long, I assume then it disregards nBlockAlign although I didn't bother to listen
foobar2000 seems to do the same thing when playing it.

I will start rejecting these as well, rather than trying to make my interpretation above work correctly.

Another reminder to always use the -v (verify) option when encoding with WavPack (it catches this one of course).   ;D

There is a consistent interpretation (i.e., assuming no values are "wrong"), which is mono 16-bit samples in 32-bit containers.

Oh. Yes this is old WAVE. And, *sighs*, my memory isn't the best ... This thread: https://hydrogenaud.io/index.php/topic,121447.msg1008257.html#msg1008257
Anyway, a stereo 16 in 32 from that collection attached, and it completely beats me why that works with FLAC/WavPack/OptimFROG and the mono doesn't ... rookie mistakes, moi?

Fixing broken WAV files with SoX
https://langdoc.github.io/2017-05-28-sox-trick.html

PS E:\download> .\sox junglede.wav junglede.flac
E:\download\sox.exe WARN wav: Premature EOF on .wav input file
PS E:\download> .\sox --ignore-length junglede.wav ignore.flac
PS E:\download> dir *.flac|select length,name

Length Name
------ ----
 44203 ignore.flac
371537 junglede.flac

The fun thing is, without using --ignore-length, the encoded flac file is even larger than the input .wav file (https://hydrogenaud.io/index.php/topic,125651.msg1041488.html#msg1041488), in both Rarewares and NetRanger's versions.

On to the wrong.wav
...
As far as I can tell,
flac, TAK and OptimFROG reject it
...

Add refalac to the list, and thanks Bryant for the attention.

Add refalac to the list, and thanks Bryant for the attention.

Although qaac/refalac's own WAV parser rejects wrong.wav, refalac will try with libsndfile when it is available, and libsndfile happily load it.
In case of qaac, it will be loaded by CoreAudio's ExtAudioFile API even when libsndfile is not available.

Thanks for the tips. I extracted refalac.exe from foobar's free encoder package without using other libraries, so didn't know about these details.

Well, it's just a shortcoming of how qaac/refalac handles input files. It just tries each handler one by one until it succeeds.
Usually it works, but occasionally behavior becomes less predictable to the user.

Quote from: Porcus on 2024-03-20 11:14:10

On to the wrong.wav
...
As far as I can tell,
flac, TAK and OptimFROG reject it
...

Add refalac to the list, and thanks Bryant for the attention.

I have fixed this in WavPack with this commit (https://github.com/dbry/WavPack/commit/428bc7dfd583df3d1d31213882faa0b09c4793de) by rejecting such WAV files.

While I was fixing this I looked at other formats and see that CAF also has this redundancy and, unlike WAV, the format documentation clearly states that having unpacked samples is valid and gives a detailed example of the 24-bit sample in 32-bit container case (values are shifted "up" for both BE and LE).

I created a few sample files like this and found that WavPack handled them fine, except for not verifying that the proper bytes were zeroed. I tried other programs and FFmpeg and Audacity (via libsndfile) both fail on these files. The only other program that handled them correctly was (no surprise) @nu774 's Foobar2000 CAF component. Good job!

If anyone is interested, I can attach them to the thread.
 

I have fixed this in WavPack with this commit (https://github.com/dbry/WavPack/commit/428bc7dfd583df3d1d31213882faa0b09c4793de) by rejecting such WAV files.

Thanks. Cool Edit / Audition 24.0 float files use 4 bytes block align and 24 bits per sample in header, will these files be affected? Here are some sample files saved with Audition 1.5:
https://hydrogenaud.io/index.php?action=dlattach;topic=114816.0;attach=22034
[edit]Warning to innocent lurkers: Some of these files are not safe to play outside of Cool Edit / Audition, beware of loud noise!

Quote from: bryant on 2024-03-29 00:37:35

I have fixed this in WavPack with this commit (https://github.com/dbry/WavPack/commit/428bc7dfd583df3d1d31213882faa0b09c4793de) by rejecting such WAV files.

Thanks. Cool Edit / Audition 24.0 float files use 4 bytes block align and 24 bits per sample in header, will these files be affected? Here are some sample files saved with Audition 1.5:
https://hydrogenaud.io/index.php?action=dlattach;topic=114816.0;attach=22034
[edit]Warning to innocent lurkers: Some of these files are not safe to play outside of Cool Edit / Audition, beware of loud noise!

Yes, thanks for reminding me of these! The 24.0 float files are indeed broken now. Will need to come up with a more complete fix.

Perhaps only accept type 1, 24 bits per sample, 4 bytes block align per channel files when -a is being used. Otherwise signal analysis on the data chuck would be required to guess the audio encoding format, for example, if it is really unpacked 24-bit int.

This should be relevant:
https://hydrogenaud.io/index.php/topic,121447.msg1009022.html#msg1009022

I have updated this again with this commit (https://github.com/dbry/WavPack/commit/874be36d4babd4d36b380758914023454d3a50d3).

For now I've decided to re-allow unpacked samples in WAV (they are already valid in CAF), however I now reject the file if all the required padding bits are not zero, and suggest the --pre-quantize option to "fix" the files, and if it's appropriate I suggest the Adobe Audition / Cool Edit option.

I realize that these files are probably invalid WAVs, but if the data is consistent (i.e., all the correct bytes are zero) then why not? I also prefer the behavior that if someone tries to compress a Audition 24.0 float file without the -a flag they'll get an error message with a reminder about the option instead of generating a corrupt WavPack file (which obviously was less than ideal).

And so, the validity in Reply 50 of that test file ... with that interpretation?


(BTW, I am not much fan of overmoderation, but @korth as mod: This thread has become more about peculiar WAVE files than a wishlist on one particular encoder/decoder. Or maybe wishful thinking on my part. Maybe it does no harm staying in this subforum.)

And so, the validity in Reply 50 of that test file ... with that interpretation?


(BTW, I am not much fan of overmoderation, but @korth as mod: This thread has become more about peculiar WAVE files than a wishlist on one particular encoder/decoder. Or maybe wishful thinking on my part. Maybe it does no harm staying in this subforum.)

Yes, the header is valid with that interpretation. However, the audio does not have the appropriate NULL padding (which would be half the bytes) and so you get an error message, which is more informative than "unsupported WAV format".

I agree that this thread has become about lossless compressors' handling of non-conforming WAV files in general, not just FLAC's handling of one particular non-conforming file. Perhaps a rename and move to General Audio or Lossless / Other Codecs would do?

I have updated this again with this commit (https://github.com/dbry/WavPack/commit/874be36d4babd4d36b380758914023454d3a50d3).

Thanks. The treatment looks very comprehensive so even uninformed users will get alarmed by the error messages and discourage them from creating problematic files.

The design goal of Monkey's Audio is to perfectly restore files to their original state.  All the header, data, and footer should be included.  I suppose you could argue whether an audio compressor really needs to do this, but it was one of my design goals.

If anyone finds an exception to this, please provide a sample and I'll fix it.

Thanks.