Reasonable handling of non-compliant source files in lossless audio compressors

Topic: Reasonable handling of non-compliant source files in lossless audio compressors (Read 16936 times) previous topic - next topic - Topic derived from FLAC-git Releases (Co...

0 Members and 1 Guest are viewing this topic.

Reasonable handling of non-compliant source files in lossless audio compressors

2024-03-11 17:25:39

I'd like to report… something… about FLAC.

Sample: Nick Dolle - Doctor quotes (it comes from there in case WAV link in temporary)

Code: [Select]

$ powershell -c "(Get-Item in.wav).Length"
3058192

$ powershell -c "(Get-FileHash in.wav -a SHA256).Hash.ToLower()"
abab6a1762e450e76cf53889e4f7d4f4c4b27e8d81bbf441c3ad15ce37041506

$ flac -7 in.wav
flac 1.4.3
in.wav: WARNING: skipping unknown chunk 'JUNK' (use --keep-foreign-metadata to keep)
in.wav: WARNING: skipping unknown chunk 'bext' (use --keep-foreign-metadata to keep)
in.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=24
in.wav: WARNING: skipping unknown chunk 'minf' (use --keep-foreign-metadata to keep)
in.wav: WARNING: skipping unknown chunk 'elm1' (use --keep-foreign-metadata to keep)
in.wav: WARNING: there is data trailing the audio data. Use --keep-foreign-metadata or --ignore-chunk-sizes to keep it
in.wav: wrote 1686579 bytes, ratio=0,555

$ flac -7f --keep-foreign-metadata in.wav
flac 1.4.3
in.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=24
in.wav: wrote 1708403 bytes, ratio=0,563

$ flac -df --keep-foreign-metadata in.flac
in.flac: done
ERROR verifying foreign metadata restore from in.flac to in.wav: stored main chunk length differs from written length

$ powershell -c "(Get-Item in.wav).Length"
3058168

$ powershell -c "(Get-FileHash in.wav -a SHA256).Hash.ToLower()"
bd45039bcc5b4b99341e1b62710e5167086ec37d941a622c1521df7dc27879a8

The same happens with the latest FLAC git-66152791 20240301.

But WavPack 5.7.0, Monkey Audio 10.53, TAK 2.3.3 compress it without issues, then decompress bit-perfect.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #1 – 2024-03-11 18:35:37

ProTools generated BWF.

The new "--force-legacy-wave-format" doesn't help either.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #2 – 2024-03-11 18:44:36

Quote from: Kraeved on 2024-03-11 17:25:39

I'd like to report… something… about FLAC.

Thanks, I'll have a look at that.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #3 – 2024-03-11 20:14:32

Not sure what to do here. This file does not comply with any WAVE specification I know of, but I guess ProTools is not something I can just ignore. The problem is that the fmt chunk is too large, it contains padding where it should not have any.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #4 – 2024-03-11 20:19:43

To be honest, that file is so far out of spec that I'm surprised Avid has the nerve to generate something like that unless, of course, it is never meant to be used outside of their own environment.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #5 – 2024-03-11 20:25:22

Unless it is not actually straight from ProTools of course. It would also be strange that I haven't had a bug report about this earlier.-

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #6 – 2024-03-11 21:15:39

The file at https://hydrogenaud.io/index.php/topic,121311.0.html seems also to be generated by ProTools - at least MediaInfo says so. No idea whether it is related.

Further googling leads me to https://duc.avid.com/showthread.php?t=92522 . Still no idea whether it is related.

(Edit: In any case, wouldn't it be better to err out in the encoding stage, than finding out only upon decoding that the file cannot be reconstructed?)

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #7 – 2024-03-12 22:00:16

Quote from: ktf on 2024-03-11 20:14:32

Not sure what to do here. This file does not comply with any WAVE specification I know of, but I guess ProTools is not something I can just ignore. The problem is that the fmt chunk is too large, it contains padding where it should not have any.

Quote from: Kraeved on 2024-03-11 17:25:39

But WavPack 5.7.0, Monkey Audio 10.53, TAK 2.3.3 compress it without issues, then decompress bit-perfect.

https://www.youtube.com/watch?v=5-sfG8BV8wU

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #8 – 2024-03-15 21:04:06

After making sure that WavPack 5.7.0, Monkey Audio 10.53, TAK 2.3.3 encode dolle.wav and then decode it back bit-perfect (BLAKE3 checksums were compared), I decided to observe the behavior of less popular lossless encoders.

Code: [Select]

$ la.exe dolle.wav
Lossless Audio Compressor Version 0.4b, copyright Michael Bevin 2002-2004
Error: invalid .wav header for file - dolle.wav

$ mp4alsRM23.exe dolle.wav
mp4alsRM23 - MPEG-4 Audio Lossless Coding (ALS), Reference Model Codec Version 23 for Win32
File type of dolle.wav is not supported!

$ ofr.exe dolle.wav
OptimFROG Lossless Audio Codec (Win/x64) v5.100 [2016-09-02]
srcFile: <dolle.wav>
dstFile: <dolle.ofr>
Exception UNKNOWN in file unknown, line 0
description: cbSize is incorrect or not an EXTENSIBLE format
Errors occurred compressing <dolle.wav>

$ sac.exe dolle.wav dolle.sac
Sac v0.6.6 - Lossless Audio Coder (c) Sebastian Lehmann
Open: 'dolle.wav': ok (3058192 Bytes)
warning: invalid fmt-chunk size
warning: input is not a valid .wav file
Total time: [00:00:00]

$ shorten.exe dolle.wav
shorten version 3.6.1: (c) 1992-1999 Tony Robinson and SoftSound Ltd
shorten.exe: input file is not a valid RIFF WAVE file

$ tta.exe -e dolle.wav dolle.tta
TTA1 lossless audio encoder/decoder, version 2.3
Encoding: "dolle.wav" to "dolle.tta"
tta.exe: can't read from input file

ALAC is a special case. Both qAAC --alac and FFMPEG -acodec alac encode dolle.wav without warnings or errors, but the decoded WAV is different from the original.

HALAC 0.2.6 MT SSE2 just crashed.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #9 – 2024-03-16 00:48:22

@NetRanger, WARNING, cannot set number of threads: multithreading was not enabled during compilation of this binary.

While analyzing storm.wav, a problematic WAV from another topic, using WAVfix utility, I came across a mention of the oddity of Pro Tools generated files. However, the analysis of Pro Tools generated dolle.wav says that it does not require repair, although data chunk is odd w/ pad.

Code: [Select]

$ wavfix storm.wav

> Processing 'storm.wav'
| [w] Wrong RIFF size: 0030203932 B + 8 [file size: 0030203996 B;]
| Current file structure :
| ======================
|     <fmt > chunk [offset: 0000000012; size: 0000000016 + 4 + 4;]
|     <data> chunk [offset: 0000000036; size: 0030203896 + 4 + 4;]
|     [NULL] chunk [offset: 0030203940; size: 0000000056 + 0 + 0;]
|
| [i] Checking <data> chunk..  ok.
|
| Recovered file structure :
| ========================
|     <fmt > chunk [offset: 0000000012; size: 0000000016 + 4 + 4;]
|     <data> chunk [offset: 0000000036; size: 0030203896 + 4 + 4;]
|     [NULL] chunk [offset: 0030203940; size: 0000000056 + 0 + 0;]
| [w] 1 unknwon bytes block remains.
|     Some programs like  Pro Tools use wrong formated files
|     like this ( <DIGI> chunk size missing  1 byte in PT ),
|     so we wont correct it. Yet it should play fine anyway.
|
| [i] Saving repaired file to 'storm_REPAIRED.wav'
| File successfully recovered.

$ wavfix dolle.wav

> Processing 'dolle.wav'
| Current file structure :
| ======================
|     <JUNK> chunk [offset: 0000000012; size: 0000000092 + 4 + 4;]
|     <bext> chunk [offset: 0000000112; size: 0000000602 + 4 + 4;]
|     <fmt > chunk [offset: 0000000722; size: 0000000040 + 4 + 4;]
|     <minf> chunk [offset: 0000000770; size: 0000000016 + 4 + 4;]
|     <elm1> chunk [offset: 0000000794; size: 0000000214 + 4 + 4;]
|     <data> chunk [offset: 0000001016; size: 0003036447 + 4 + 4;] odd w/ pad
|     <regn> chunk [offset: 0003037471; size: 0000000092 + 4 + 4;]
|     <umid> chunk [offset: 0003037571; size: 0000000024 + 4 + 4;]
|     <DGDA> chunk [offset: 0003037603; size: 0000020580 + 4 + 4;]
|
| [i] Checking <data> chunk..  ok.
|
| File ok.

SoX refuses to read dolle.wav at all, even with --ignore-length flag, throwing an error: “WAVE header error: cbSize inconsistent with fmt size”. FFMPEG comes to rescue in a way.

Code: [Select]

$ ffmpeg -i dolle.wav -c:a pcm_s24le dolle.ffmpeg.wav

$ wavfix dolle.ffmpeg.wav

> Processing 'dolle.ffmpeg.wav'
| Current file structure :
| ======================
|     <fmt > chunk [offset: 0000000012; size: 0000000040 + 4 + 4;]
|     <LIST> chunk [offset: 0000000060; size: 0000000062 + 4 + 4;]
|     <data> chunk [offset: 0000000130; size: 0003036447 + 4 + 4;] odd w/ pad
|
| [i] Checking <data> chunk..  ok.
|
| File ok.

$ flac -7 --keep-foreign-metadata dolle.ffmpeg.wav
dolle.ffmpeg.wav: wrote 1686800 bytes, ratio=0,556

$ flac -d --keep-foreign-metadata dolle.ffmpeg.flac -o dolle.ffmpeg.flac-d.wav
dolle.ffmpeg.flac: done

$ b3sum dolle.ffmpeg.wav dolle.ffmpeg.flac-d.wav
620af01885809c2b11eef54b71ae3d863e9427400a040f616a37f295e70da091  dolle.ffmpeg.wav
620af01885809c2b11eef54b71ae3d863e9427400a040f616a37f295e70da091  dolle.ffmpeg.flac-d.wav

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #10 – 2024-03-16 09:53:20

Do I read this correctly?

* ProTools does weird things
* WAVfix chooses not to touch it because the mistake is made by a big player and sure as hell they are not going to take the blame for doing any fixes that could make ProTools reject the file
* SoX refuses to read them
?

A quick googling finds this: https://stackoverflow.com/questions/26169678/why-certain-wav-files-cannot-be-decoded-in-firefox
Some issues with a ProTools .wav file, but here SoX reads it.

Quote from: ktf on 2024-03-11 20:14:32

Not sure what to do here. This file does not comply with any WAVE specification I know of, but I guess ProTools is not something I can just ignore.

I agree.
But I think it is better to reject files upon encoding.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #11 – 2024-03-16 11:42:07

Better yet, reject them with explanation why. Big players, huh, not adhering to standards, I wonder if they are the first...

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #12 – 2024-03-16 11:53:52

I believe the problem is not with Pro Tools generated files, but with FLAC itself, which processes non-audio data in a picky way rather than storing it.

Perhaps, there was a reason for this design, but the ordinary user expects FLAC to be something like playable ZIP. As we can see, WavPack, Monkey Audio and TAK are more in line with this expectation. Over time, FLAC got --keep-foreign-metadata flag to preserve non-audio data. But a) it is still considered experimental: “NOTE: --keep-foreign-metadata is a new feature; make sure to test the output file before deleting the original”, and b) it causes confusion, because some files can be compressed without it, but not with it (see the log below). In particular, I stumbled upon this ticket on FFMPEG issue tracker created 12 years ago (in 2012) mentioning Gallery Metacorder app, which also stores non-audio data in WAV. Guess what? FLAC refuses to process it. Guess how the ordinary user reacts? “Aggrrhh!”

Is there really no way to teach FLAC to be less picky and just store non-audio data?

Code: [Select]

$ ffprobe -loglevel error -show_entries format_tags -of json metacorder.wav
{
    "format": {
        "tags": {
            "comment": "gSCENE=66a\r\ngTAKE=002\r\ngTAPE=007\r\ngNOTE=Circle \r\ngUBITS=00000000\r\n",
            "encoded_by": "Gallery Metacorder",
            "date": "2005:08:04",
            "creation_time": "16:55:54",
            "time_reference": "4090608000",
            "coding_history": ""
        }
    }
}

$ wavfix metacorder.wav
> Processing 'metacorder.wav'
| Current file structure :
| ======================
|     <bext> chunk [offset: 0000000012; size: 0000000858 + 4 + 4;]
|     <fmt > chunk [offset: 0000000878; size: 0000000016 + 4 + 4;]
|     <data> chunk [offset: 0000000902; size: 0000210154 + 4 + 4;]
|     <iXML> chunk [offset: 0000211064; size: 0000002839 + 4 + 4;] odd w/ pad
|     <iXTC> chunk [offset: 0000213911; size: 0000000020 + 4 + 4;]
|
| [i] Checking <data> chunk..  ok.
|
| File ok.

$ flac metacorder.wav
flac 1.4.3
metacorder.wav: WARNING: skipping unknown chunk 'bext' (use --keep-foreign-metadata to keep)
metacorder.wav: WARNING: there is data trailing the audio data. Use --keep-foreign-metadata or --ignore-chunk-sizes to keep it
metacorder.wav: wrote 34136 bytes, ratio=0,162

$ flac -f --keep-foreign-metadata metacorder.wav
metacorder.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #13 – 2024-03-16 12:19:48

Quote from: Kraeved on 2024-03-16 11:53:52

I believe the problem is not with Pro Tools generated files, but with FLAC itself, which processes non-audio data in a picky way rather than storing it.
....

Not so. There is no requirement for tools such as FLAC to store non-standard chunks created by applications such as ProTools. For chunks that are non-standard an application is only required to skip over them. The ordinary user does not expect FLAC to perform as you suggest since the ordinary user is unlikely to be using such files. The non-standard chunks created by ProTools are for ProTools consumption and nothing else simply because nothing, or little, else knows what to do with them. The fact that WavPack and Monkeys Audio choose to preserve all the chunk information is a generous feature that is neither required nor expected. You are suggesting a problem where none exists. FLAC performs, in this regard, as required and expected.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #14 – 2024-03-16 13:26:36

The ordinary user does not know that there are standard and non-standard chunks. He is told that there is a new way to make an audio file not only take up less space and decompress it back while preserving its properties (as in the case of omnivorous ZIP-like tools), but also be playable on various devices. “That's so cool! Let's start by squeezing those audio lectures.” However, at present FLAC takes advantage of people's ignorance, keeping only audio data lossless, the rest is a game of chance. I break into a cold sweat thinking about how many times over the years FLAC has skipped non-standard chunks that might be helpful to me or to someone else in the future (author's comments, voice recorder tags, software helpers, coordinates, Easter eggs, etc). I can vividly imagine how the announcer Nick Dolle writes to me and asks, "Buddy, my website is down, but I heard you still have a demo of my voice, could you share it, please?" I send him FLAC, then he raises an eyebrow in surprise: “Where did the tag with the recording date go?” Oh, it was washed away.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #15 – 2024-03-16 14:24:51

Quote from: Kraeved on 2024-03-16 11:53:52

Is there really no way to teach FLAC to be less picky and just store non-audio data?

Absolutely. Download the source code and do it yourself. Nobody is here to cater for your every little whim.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #16 – 2024-03-16 14:40:15

FLAC was designed as a compressor for audio, not for files. The FLAC FAQ has more than one item about that. Quoting from https://xiph.org/flac/faq.html :

Quote

I compressed a WAVE file to FLAC, then decompressed to WAVE, and the two weren't identical. Why?
I compressed a WAVE file to FLAC and it said "warning: skipping unknown sub-chunk LIST". Why?
I decoded a FLAC file and the WAVE is 2 bytes shorter than the original. Why?

WavPack was designed to compress files (and so was Monkey's, which took its inspiration from WavPack) - FLAC was not, that is an auxilliary feature for power users. Even if the reference implementation has been able to store RIFF chunks for many years, it seems the documentation is even more recent than the most official release.

Quite possibly I should not be the first to knock other users over obsession about corner cases, but put bluntly: using FLAC to restore anything but audio is for power users who are willing to read manuals (and at least the FAQ).
My issue with the reference implementation's handling, is that by now it pretends to do something it cannot.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #17 – 2024-03-16 15:36:56

@Porcus, I understand that the origins of FLAC and the trio of other popular lossless encoders have different ideas. My question is how much this is set in stone and prevents FLAC from becoming more flexible with respect to non-audio data. For example, Dominik Reichl, the developer of Keepass password manager, added support for other encryption algorithms without insisting on using Rijndael (AES), which is still considered a standard somewhere in North America, thereby showing flexibility towards users. So I posted metacorder.wav that uses “non-standard” iXML chunk. What is it? Wikipedia article and official specification claim it popped up not yesterday, but in the mid-2000s. Will something terrible happen if FLAC learns to store it? Are there any technical or ideological obstacles? Or the will to progress gave way to laziness? Surely there is someone here who is familiar with how decisions about further development are made.

Quote

Powerful and intuitive Metadata functionality, including Project, SoundRoll, scene, take, track labels and indexes, recording notes. User friendly system automatically updates scene and take using US, UK or generic slating systems. Metadata can be edited before, during or after recording. Metadata stored in several formats including bext and the revolutionary new iXML format for seamless post production..

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #18 – 2024-03-16 16:35:29

Quote from: Kraeved on 2024-03-16 15:36:56

My question is how much this is set in stone and prevents FLAC from becoming more flexible with respect to non-audio data.

First of all, can you please stop hijacking threads? If you have a problem, start a new thread instead of necroposting or posting something completely irrelevant for the current thread?

Second, can you please stop calling developers lazy? For most of use this is unpaid work. Especially what you said about FSLAC sounded particularly condescending.

Back to the matter at hand, the FLAC reference command line tool can store and restore foreign metadata. It currently has the limitation that it will not restore a non-standard fmt chunk. Why not? Because that contains data necessary to parse the file, and FLAC currently writes its own based on its own metadata, to assure the file stays valid. This might loosen in the future, but I think mindlessly writing invalid data unchecked is bad behaviour. As you have seen, many tools simply reject this weird Protools fmt chunk.

You seem to have a problem with FLAC not storing foreign metadata by default. FLAC warns the user when such metadata is present, suggesting the use of an extra option. Recently FLAC got the option --keep-foreign-metadata-if-present that you can leave always on without getting an error. But yes, it is not on by default.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #19 – 2024-03-16 21:30:35

Quote from: Kraeved on 2024-03-16 13:26:36

The ordinary user does not know that there are standard and non-standard chunks.

Standards are for users' own good, even if they don't realize that. What you are arguing for is applying Postel's law: "be conservative in what you do, be liberal in what you accept from others", which has a fair amount of criticism. Yes, short-term the user may be satisfied that his file was processed without errors but eventually they will start complaining again that app A interpreted their broken file differently than app B.

Quote from: Kraeved on 2024-03-16 13:26:36

I can vividly imagine how the announcer Nick Dolle writes to me and asks, "Buddy, my website is down, but I heard you still have a demo of my voice, could you share it, please?" I send him FLAC, then he raises an eyebrow in surprise: “Where did the tag with the recording date go?”

Then you should raise your eyebrow and say: "Nick, buddy, where are your backups? Oh, you haven't done any? So maybe lower your eyebrow and be glad that you were able to recover anything at all."

Quote from: Kraeved on 2024-03-16 15:36:56

For example, Dominik Reichl, the developer of Keepass password manager, added support for other encryption algorithms

Would he add support for files produced by apps that implemented those other algorithms incorrectly?

Quote from: Kraeved on 2024-03-16 15:36:56

So I posted metacorder.wav that uses “non-standard” iXML chunk. What is it? Wikipedia article and official specification claim it popped up not yesterday, but in the mid-2000s. Will something terrible happen if FLAC learns to store it? Are there any technical or ideological obstacles?

The problem with your file is not that it has an unknown iXML chunk. iXML chunks, or any other chunks, are fine and that's what --keep-foreign-metadata is for.

The problem with your file is that the chunk is stored incorrectly. The standard for RIFF says that if the size of the data in the chunk is odd then it needs a pad-byte to make it even. The iXML chunk in your file is missing that byte:

Code: [Select]

00034390  20 20 20 20 20 20 20 69  58 54 43 14 00 00 00 00  |       iXTC.....|

You see that the next chunk after "iXML" starts at odd position. That makes it an invalid RIFF file. Insert a byte there:

Code: [Select]

00034390  20 20 20 20 20 20 20 00  69 58 54 43 14 00 00 00  |       .iXTC....|

and it will make it less invalid RIFF file (in attachment). The size in the RIFF chunk is still/now wrong, 213'931 instead of 213'932, but apparently that's something flac can deal with:

Code: [Select]

]$ flac -f --keep-foreign-metadata metacorder.fix.wav
NOTE: --keep-foreign-metadata is a new feature; make sure to test the output file before deleting the original.

flac 1.4.3
Copyright (C) 2000-2009  Josh Coalson, 2011-2023  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

WARNING: RIFF chunk size of file metacorder.fix.wav does not agree with filesize
metacorder.fix.wav: wrote 37970 bytes, ratio=0.181

Quote from: Kraeved on 2024-03-16 15:36:56

Or the will to progress gave way to laziness?

To you it would be progress, to others it would be regression.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #20 – 2024-03-16 22:59:40

Quote from: ktf on 2024-03-16 16:35:29

You seem to have a problem with FLAC not storing foreign metadata by default.

Quote from: danadam on 2024-03-16 21:30:35

To you it would be progress, to others it would be regression.

Me? I had problems in my youth, and now, following Jack Ma's advice, I am concerned with what is of interest to a wider range of users. So I report not my problems so much as I report missed opportunities to make tools better overall. It is quite common to miss such things due to resting on your laurels, déformation professionnelle and the echo chamber. In this case, if the competitors such as WavPack, Monkey Audio and TAK proved that it is possible to become even more lossless by storing not only the audio, but also the accompanying data without warnings and errors, then why not go this route? Tell me about the obstacles, if any, as I don't see how this can be a regression.

Look, MP3 encoder Helix, which processed 16-bit yesterday, now supports up to 32-bit floating-point and Unicode.

Quote from: ktf on 2024-03-16 16:35:29

First of all, can you please stop hijacking threads? Second, can you please stop calling developers lazy?

Spoiler (click to show/hide)

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #21 – 2024-03-17 00:19:24

First, there is a difference between necromancing the relevant thread that is old - and just picking one that happens to be about the same software.

As for software ... there are different tools for different uses.

Quote from: Kraeved on 2024-03-16 22:59:40

In this case, if the competitors such as WavPack, Monkey Audio and TAK proved that it is possible to become even more lossless by storing not only the audio, but also the accompanying data without warnings and errors, then why not go this route?

Because well, after twenty-something years of both WavPack, Monkey's and FLAC, there hasn't been enough need for FLAC to do that job? Proof: FLAC has become the most dominant lossless format still.

FLAC doesn't support floating-point signals either. For those very few ones (I think there should be more of them, better do float in processing ...), I use WavPack. And if I want to preserve at file - WavPack it is, it is designed to do that. To the extent that old wvunpack decoders can decode to a file format it knows nothing about.

What I think is troublesome about reference flac's handling here, is not that it cannot handle these files, it is that it let itself be fooled to thinking it can. "I am not designed to do this!" is acceptable ... but annoying when the files come from a big player and you must expect there to be a lot of them around.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #22 – 2024-03-17 08:22:27

Quote

ALAC is a special case. Both qAAC --alac and FFMPEG -acodec alac encode dolle.wav without warnings or errors, but the decoded WAV is different from the original.

Was there any difference PCM-wise?
ALAC doesn't keep "foreign metadata" (non-PCM chunks in the WAV file), so WAV file shouldn't be identical and that's normal.

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #23 – 2024-03-17 08:35:39

Quote from: Kraeved on 2024-03-16 22:59:40

Tell me about the obstacles, if any, as I don't see how this can be a regression.

I'll summarize what happens
1. Protools generates an invalid WAVE file.
2. FLAC stores this invalid metadata when invoked with --keep-foreign-metadata
3. FLAC restores a valid/fixed WAVE file when invoked with -d --keep-foreign-metadata
4. FLAC returns an error that the restored file is not identical to the original file. All foreign metadata is there, but the fmt chunk has been corrected

I could do a few things:
1. Do the same thing as WavPack/Monkeys/TAK, but then the restored file is invalid. I would consider this a regression, FLAC should not output invalid files
2. Return a more comprehensible warning instead of an error, explaining that the the fmt chunk has been altered but all other metadata is correct
3. Warn users on encoding that the file is invalid and restoring might not give back an identical file

So, I do not think fixing this is straightforward and haven't yet made a decision

Re: Reasonable handling of non-compliant source files in lossless audio compressors

Reply #24 – 2024-03-17 09:39:47

Your solution item #3 - the warning - can be done independently of the others I think?

Also, concerning 4:

Quote from: ktf on 2024-03-17 08:35:39

4. FLAC returns an error that the restored file is not identical to the original file. All foreign metadata is there, but the fmt chunk has been corrected

Will all foreign metadata be the same? I understood @danadam's reply #19 that there was at least one chunk of odd byte length, and so the writing application should have byte-padded it. But if the flac decoder does that, then it alters not only <fmt >, but the metadata ...
... in a way that likely doesn't matter, because likely the writing application is just too stupid to know the padding byte, and so likely it won't notice that it is there - but I'm not sure one can trust that.

As goes Pro Tools specifically ... https://avidtech.my.salesforce-sites.com/pkb/articles/en_US/faq/en388231

Notice