Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Stopping Foobar2000 transcoding lossy to lossless  (Read 6765 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Stopping Foobar2000 transcoding lossy to lossless

I am a bit stumped.  I  noticed after some re encoding albums (wv+wvc) that foobar
left some .tmp.wvc files instead of .wvc.  Not a lot maybe half a dozen out of 200 albums.
I fiddled with some settings and turned on 'encode to temporary location, move when done..'
That seems to have fixed the .tmp issue.

However, Foobar will transcode WV+.tmp.wvc which is NOT lossless to lossless - as if all is totally normal !
No warnings nothing. 

I tried to replicate this using the CLI  (rename .wvc to .wvcc) and get
"can't transcode lossy file Enya - Watermark.wv to lossless...not allowed!"
Which is CORRECT and what i want foobar to do - stop encoding and issue an error.
I played around with various foobar options but can't seem to stop it.

So i am totally lost.

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #1 –
Pardon me if I don't understand, but for wavpack filename.wv + filename.wvc = lossless and it can be encoded to lossless.
Is there a problem where you don't expect foobar to use filename.tmp.wvc as the correction file, as it might be unfinished or something?
TAPE LOADING ERROR

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #2 –
Quote from: itisljar on
Pardon me if I don't understand, but for wavpack filename.wv + filename.wvc = lossless and it can be encoded to lossless.
Is there a problem where you don't expect foobar to use filename.tmp.wvc as the correction file, as it might be unfinished or something?

Yes, I am still somewhat scared of foobar since it left several .tmp.wvc 's previously .  It didn't alert me.
Only caught it by looking at an album subfolder prior to encoding. Then I noticed a .wvc.tmp. Turns out
this occured several times. I feared i have converted several tracks with .tmp.wvc to wv+wvc thinking all
was OK. 

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #3 –
Can confirm with both 1.6.12 and v2x64beta12.

For the problem:
This is a foobar2000 problem (if fb2k takes on support for correction files, it should IMHO apply the lossy-to-lossless warning), so ... nah f**c it, instead of saying "post there", I invoke @Peter .

I note that under the "General" info window, neither of the following change when I delete the .wvc file:
Codec Profile: WavPack normal, hybrid
Codec: hybrid

It might be a good thing to let the user know whether it is lossy or has a correction file. Presumably there is a speed penalty in it, two files to consider every time time the user drags a .wv into a playlist, but ... I still think it is a good thing. Maybe as an advanced option, for that matter - users who don't use .wvc at all could switch off.
A suggestion:
* keep "Codec Profile"
* alter "Codec" to either "lossy" or "lossless"
... because (1) "Codec Profile" says something about choices made for encoding, and "Codec" says something about what you got in there; and (2) Queries %__encoding% HAS e and %__encoding% HAS y will get the right hits. (*)

Now if it is "Codec: lossy" that triggers the warning when transcoding to lossless, that gives even more a case for that suggestion.
(*) Uh but ... isn't there a Codec: synthetic as well?



For a solution:
Is there a way to repair if you transcoded the .wv only, but still have a .wvc? I don't see anything in the manual to "force .wv that presents itself as lossless to still consider correctionfilename.wvc", but I could be wrong.
@bryant, is it possible to hack this or that byte in a .wv file? If not, that might for sudden have become someone's wishlist item.
(And if the "suggestion" above is stupid for WavPack-internal reasons I have not thought of ... please shoot it down.)

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #4 –
Yes, Foobar profile info would be ideal.  Interestingly, I have old versions 1.03 and 0.8.3.
1.03 still displays only a hybrid flag.  0.8.3 goes further to display lossless or lossy:

samplerate = 44100
channels = 2
bitspersample = 16
format = Integer
codec = WavPack
bitrate = 1088
compression = Hybrid Lossless
replaygain_album_gain = -10.53 dB
replaygain_album_peak = 1.000000
replaygain_track_gain = -11.20 dB
replaygain_track_peak = 1.000000
 ----------
9146928 samples @ 44100Hz
File size: 8 529 916 bytes

Renaming / deleting .wvc file;

samplerate = 44100
channels = 2
bitspersample = 16
format = Integer
codec = WavPack
bitrate = 329
compression = Hybrid Lossy
replaygain_album_gain = -10.53 dB
replaygain_album_peak = 1.000000
replaygain_track_gain = -11.20 dB
replaygain_track_peak = 1.000000
 ----------
9146928 samples @ 44100Hz
File size: 8 529 916 bytes

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #5 –
There are a few things going on here. The first is that actually it’s not Foobar2000 that's failing to delete the tmp files. That is a WavPack CLI thing, which creates tmp files when it’s overwriting an existing wv file or any time it’s generating correction files (even when not overwriting). It does the remove + rename at the end and should return an error if either operation fails. In the console you’ll see a message indicating what failed, but of course not when using Foobar (but Foobar should report the error and I’ve verified before that it does detect other errors).

Now I don’t think I’ve ever tried an overwriting transcode of files using Foobar, especially with correction files present (which I assume is what we’re talking about). In that case maybe Foobar uses it’s own temporary file names so all the output files get renamed twice (first by WavPack, then Foobar). If it doesn’t, then WavPack might be trying to delete the source files that Foobar still has open unless Foobar intentionally (or accidentally) closes them before sending EOF to WavPack; not sure how that would work out (and hope that isn’t how it happens). I'll experiment with this a little.

As for the other questions, if I understood correctly, when decoding a lossless WavPack file, WavPack may try to open a corresponding correction file before it figures out it can’t use it, but won’t do anything with it and everything will just work losslessly (no error generated).

But if a lossy WavPack file is opened and the wrong correction file is used (say, from a different encoding session with different settings) then it will try to use it, but would generate errors which would then be muted (resulting in just silence). I have thought of putting a UID in both files on encoding and throw an error up if an attempt is made to do that so at least the user would know why the decode was broken and know to delete the correction file.

Which brings up the final point: wv + wvc files are a little like entangled photons. If the wv file is deleted, the wvc file instantly becomes useless garbage, even if it’s stored on a server across the globe.  :D

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #6 –
Quote from: bryant on
As for the other questions, if I understood correctly, when decoding a lossless WavPack file, WavPack may try to open a corresponding correction file before it figures out it can’t use it, but won’t do anything with it and everything will just work losslessly (no error generated).

I think, here is the situation.
1) wavpack -cb192 Origfile.wav -o File.wv
2) ... but it is stuck with File.wv and File.tmp.wvc
3) Convert to some lossless format using foobar2000. Say, I have E:\music-encoded-hybrid-with-cb192\folderstructure\filenames.wv and found that ... uh, I don't need these as hybrid, I'm not using that, so let's try to convert to f:\ordinary-wv\folderstructure\filenames.wv using -hx4 or something.
But File.wv has no accompanying .wvc, due to the .wvc not being renamed properly.
Expected: foobar2000 to scream when trying to transcode lossless to lossy - because it usually does!
But it does not in this case. It happily decodes File.wv without correction, and encodes it.
4) Well then, it is successfully done. Copy back with overwrite.
5) UH-OH!


And that is where this question comes in: Since there is a
File.wv which now identifies as lossless, but contains the same audio as the old File.wv that identified as hybrid
and still the old
File.tmp.wvc
I would hope to to rename the latter to File.wvc and get my lossless original back. But because of the re-encoding of the lossy part, the new File.wv identifies as lossless and I don't find a way to convince it to view itself as hybrid and take the correction on board.

Is anything such even feasible? For all that I know, it could be that the hybrid format includes block by block information about how to interpret the .wvc, so that the audio-only simply isn't enough information.


Quote from: bryant on
But if a lossy WavPack file is opened and the wrong correction file is used (say, from a different encoding session with different settings) then it will try to use it, but would generate errors which would then be muted (resulting in just silence). I have thought of putting a UID in both files on encoding and throw an error up if an attempt is made to do that so at least the user would know why the decode was broken and know to delete the correction file.
It could store the MD5 of the lossy part of the .wv, and that would be enough? No idea whether you can fit it in the format ...

New WavPack Freemium model: pay a modest ran$$omware fee to unlock the magical
wvunpack thiswassupposedtobelosslesshybridbutImessedup.wv --force-treat-as-hybrid-test-every-wvc-you-can-find-recurse-subdirectories
 O:)

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #7 –
Ah, I see what you’re asking now. You just have to explain it slowly and loudly enough!

If the correction file was simply a lossless encoding of the difference between the lossy version and the original, then you could do such a thing. I think this is how the correction file of lossyWAV works, and doing it this way would have certainly simplified the WavPack hybrid mode.

The problem with that method is that there is some additional redundancy in having independent data streams because each stream needs to delineate one sample from the next. The way I decided to implement it, when the decoder is decoding the lossy samples, it knows how many bits it needs to read from the correction file to complete each sample. Therefore, the information required to delineate each sample does not go into the correction file, which saves space. It’s just a stream of bits and the decoder knows how many to use for each sample that comes along.

I just did a quick experiment to show the savings. I took a sample WAV file with a mix of music and encoded it several ways. Here are the results in increasing compression order (note that except for the “default” all are maximum compression -hhx6):

Code: [Select]
original WAV file                26551140 bytes    ---
lossy (.wv) + difference (.wv)   14847284 bytes   44.08%
lossy (.wv) + correction (.wvc)  14181522 bytes   46.59%
default lossless                 14121542 bytes   46.81%
maximum lossless                 13738398 bytes   48.26%

So in this case the correction file is 2.5% better than encoding the difference, and gets you very close to the same overall compression as the lossless default.

But you can also see that without knowing where one sample ends and the next begin, the correction file is meaningless by itself.

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #8 –
How can I get a total audio checksum for 10 tracks instead of checking one by one ?
Or easier to use single file + cue ?

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #9 –
How many are there in total? You can get all in a folder into one output by wvunpack.exe -f7 *.wv
If there is just one, use your fave text tool. Everyone needs Notepad++ I guess.




Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #10 –
Quote from: bryant on
It’s just a stream of bits and the decoder knows how many to use for each sample that comes along.

So, you can put ANY correction file and get... some results? Filename is the only check?

I know I am jumping ahead, but what if there would be some sort of file tag that would identify it's lossy + correction? Like, writing random ID at the end of files (or wherever) and then decoder knows that if there is ID, there should be another file that goes with it with same ID, if there is not, treat as lossy. Without ID, it would be treated as lossless or lossy, I am guessing that it's reading some info in file where that is written.

It wouldn't be backward compatible, tho, but going forward it would be of some help in these cases.
TAPE LOADING ERROR

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #11 –
Quote from: itisljar on
So, you can put ANY correction file and get... some results? Filename is the only check?

I just tried, and got
missing data or crc errors detected in 73 block(s)!
... and a .wav file that is the same as omitting the .wvc altogether.


Quote from: itisljar on
what if there would be some sort of file tag that would identify it's lossy + correction?
wvunpack -ss filename

There is a line that says
modalities:        hybrid lossless, dns
and if you delete the .wvc, it becomes
modalities:        hybrid lossy, dns

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #12 –
There is a slightly puzzling behaviour to this. With -b999, the bit rate is high enough for this particular file not to have any loss. Therefore,
wavpack -b999 harpe.wav
is lossless for this particular file, and reports so when it concludes encoding. That is what the documentation says too: "If the track can be losslessly compressed without exceeding the specified bitrate, then it will be and WavPack will report the compression as lossless."
But wvunpack -ss reports it as hybrid lossy, because it sees that it was encoded in hybrid mode and it finds no .wvc.

Forcing a correction file with -cb999, it does indeed create it. But it contains no audio: deleting the .wvc and bit-comparing to the original, it is identical. But the .wv files are not the same, so the encoder does make a difference.


An alternative would be to generate an ordinary .wv file and report it as
modalities:        lossless, high, extra-1, b-999
(or just "b" or "as hybrid"). I have a hunch that "would be" is a "could have been".

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #13 –
Before I updated to (then) foobar 1.4, I never encountered .tmp files. Used 1.0.3 for ages.
I ts happend with mp3's as well.  Not sure how to reproduce but read something about mutlithreading
or 'copy files to temp location then move back' .
 

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #14 –
Quote from: shadowking on
How can I get a total audio checksum for 10 tracks instead of checking one by one ?
Or easier to use single file + cue ?
On Linux this is easy:
Code: [Select]
wvunpack *.wv --raw -o - | md5sum -
I was able to make it work on Windows too after downloading md5.exe from the web (which is always a little scary):
Code: [Select]
wvunpack *.wv --raw - | md5.exe
One problem on Windows is that files don’t always come up in alphabetical order, and obviously the order matters for the MD5.

As for Foobar2000, I did some experimentation. First, I discovered that it prevents you from overwriting the source files, so that danger is avoided. I also verified that it does report a failure return status from WavPack and I checked to make sure that the code that renames the tmp file really does trigger an error return on failure. So, I don’t know how the tmp files could be left around without Foobar reporting it.

I did run into a bug in Foobar though. When you attempt to overwrite existing files (that are not the source files) Foobar will delete (or move?) the target file before invoking WavPack, but not the correction file. So WavPack gets invoked and sees the existing correction file and prompts for permission to overwrite, but since there’s no console window this actually just hangs everything. It would be possible to use the -y option of WavPack to overwrite any existing file, or the new --no-overwrite to generate an error if an overwrite is attempted instead of hanging (although this is only from 5.5.0), but probably the best and easiest would be to just treat the correction file the same as the WavPack file. @Peter ?

In the end though, I don’t know how the tmp files might not get renamed. The fact that it happens so infrequently means it’s probably some weird timing issue. And of course, it wouldn’t hurt to use the latest version of Foobar2000…   :)

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #15 –
Quote from: itisljar on
Quote from: bryant on
It’s just a stream of bits and the decoder knows how many to use for each sample that comes along.

So, you can put ANY correction file and get... some results? Filename is the only check?

I know I am jumping ahead, but what if there would be some sort of file tag that would identify it's lossy + correction? Like, writing random ID at the end of files (or wherever) and then decoder knows that if there is ID, there should be another file that goes with it with same ID, if there is not, treat as lossy. Without ID, it would be treated as lossless or lossy, I am guessing that it's reading some info in file where that is written.

It wouldn't be backward compatible, tho, but going forward it would be of some help in these cases.
Each frame of a WavPack file has a checksum for the decoded data which works whether it’s lossless or lossy. The correction files are also divided into frames that match one-to-one with the lossy file’s frames, and each of those frames also has a checksum that represents the lossless decoded data (so yeah, I exaggerate when I say they’re just a stream of bits).

So if a mismatched correction file is used then the checksums won’t verify and the blocks will be muted. Or, if DNS is used, the frames won’t even line up (because they're all different lengths) and will just be skipped altogether (which it what I think you saw @Porcus ). In both cases the user is notified.

And yes, I considered adding a UID and it could be done in a backward-compatible way. If the .wv file had an UID then the correction file would be ignored if it didn’t have the matching UID. Old decoders would ignore the UID. But the fact that we won’t ever do anything nasty (like bursts of noise) and we flag the errors when a mismatched correction file is used, I don’t think this is a huge priority.

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #16 –
Quote from: Porcus on
There is a slightly puzzling behaviour to this. With -b999, the bit rate is high enough for this particular file not to have any loss. Therefore,
wavpack -b999 harpe.wav
is lossless for this particular file, and reports so when it concludes encoding. That is what the documentation says too: "If the track can be losslessly compressed without exceeding the specified bitrate, then it will be and WavPack will report the compression as lossless."
But wvunpack -ss reports it as hybrid lossy, because it sees that it was encoded in hybrid mode and it finds no .wvc.

Forcing a correction file with -cb999, it does indeed create it. But it contains no audio: deleting the .wvc and bit-comparing to the original, it is identical. But the .wv files are not the same, so the encoder does make a difference.


An alternative would be to generate an ordinary .wv file and report it as
modalities:        lossless, high, extra-1, b-999
(or just "b" or "as hybrid"). I have a hunch that "would be" is a "could have been".

The case of super high bitrates generating lossless files is something I considered quite a bit (certainly more than its rarity would justify). The reason the encoder displays “lossless” is because it just encoded the whole file without a single sample being “lossy” and it keeps track of that. But, there’s currently no way to store that information in the header, so the file will always appear as lossy (because it will always be hybrid mode...they’re different) and only by decoding the whole file is it possible to know it was actually lossless. If you create a correction file you’ll get counter-intuitive results like this (as you saw):
Code: [Select]
david@david-HP-Laptop-15-bs1xx:~/WavPack$ ls -l 24bit.wv*
-rw-rw-r-- 1 david david 2912902 Nov 12 10:05 24bit.wv
-rw-rw-r-- 1 david david    8400 Nov 12 10:05 24bit.wvc
david@david-HP-Laptop-15-bs1xx:~/WavPack$ cli/wvunpack -vm 24bit.wv

 WVUNPACK  Hybrid Lossless Audio Decompressor  linux-gnu Version 5.5.2
 Copyright (c) 1998 - 2022 David Bryant.  All Rights Reserved.

original md5:  d526d5fb2682675c713be47c62e40341                                
unpacked md5:  d526d5fb2682675c713be47c62e40341                                
verified 24bit.wv (+.wvc) in 0.33 secs (lossless, 44.80%)                                
david@david-HP-Laptop-15-bs1xx:~/WavPack$ cli/wvunpack -vmi 24bit.wv

 WVUNPACK  Hybrid Lossless Audio Decompressor  linux-gnu Version 5.5.2
 Copyright (c) 1998 - 2022 David Bryant.  All Rights Reserved.

original md5:  d526d5fb2682675c713be47c62e40341                                
unpacked md5:  d526d5fb2682675c713be47c62e40341                                
verified 24bit.wv in 0.25 secs (lossy, 776 kbps)

I could store a flag at the end to indicate this (or rewrite the header with a new flag), and I thought of deleting the correction file in this case (which might confuse the user or an application invoking WavPack), but in the end decided that this edge-case is rare enough that it’s not worth worrying about. This may be the first time it’s come up in almost 20 years since WavPack 4!

As for the difference between the WavPack files themselves when creating the correction file, that’s sort of superficial. They both could have correction files, but the dynamic noise shaping works better without the correction file requirement and so you get different results. Killing DNS with -s0 should make the files identical whether a correction is created or not, and it’s also why it's a good idea to not create a correction file if you’re planning to delete it anyway.

What’s funny though is sometimes I want to get around the “can’t transcode lossy to lossless” warning (probably for testing) and so use this trick. It really shouldn’t even allow going to a higher lossy bitrate, but that sounds a little too nanny-state, even for me.

But I do agree that Foobar2000 should at least warn about transcoding WavPack lossy to lossless.

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #17 –
Quote from: bryant on
this edge-case is rare enough that it’s not worth worrying about. This may be the first time it’s come up in almost 20 years since WavPack 4!
And then it was provoked out of curiosity by someone who started reading the manual from top (-b comes early) - not found in the wild. So ... no worries.

(Aside, I'd be all too scared to rely on two files that could be messed up with move and rename. Had I ever created something like this, I would have surely nanny-state'd myself by making a kind of .zip-alike format where the user could open the archive and copy out the lossy part.)

Re: Stopping Foobar2000 transcoding lossy to lossless

Reply #18 –
Quote from: Porcus on
Quote from: bryant on
this edge-case is rare enough that it’s not worth worrying about. This may be the first time it’s come up in almost 20 years since WavPack 4!
And then it was provoked out of curiosity by someone who started reading the manual from top (-b comes early) - not found in the wild. So ... no worries.

(Aside, I'd be all too scared to rely on two files that could be messed up with move and rename. Had I ever created something like this, I would have surely nanny-state'd myself by making a kind of .zip-alike format where the user could open the archive and copy out the lossy part.)


Nothing on the source folders ever got messed , it was the destination i wasn't aware of it initially.
I then deleted the sources assuming all went well (they were in recycle bin). Luckily I did a few albums manually rather than one big job.
In any case I have an older backup on another drive. Thinking back now, I should have copied the source folder tree to a tmp location as
a precaution.

Another way is moving .wvc files to  'artist-album'.ZIP  etc. I tried it before and seemed to work well for lossy
use with the option of full restoration (just unzip to the source and its lossless again). Can also then use wvunpack -s *
or -vm as extra verification.