Skip to main content

Topic: So I migrated my whole FLAC collection to WV. (Read 1250 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
So I migrated my whole FLAC collection to WV.
Hello.

First of all, I would like to say thanks to the developer(s) of this codec. It's a little inefficient than FLAC, has worse hardware support and encodes/decodes slower (note: these observations are based on hybrid mode), but the hybrid mode is really convenient. Please know that I'm not insulting the codec or the developer--it's awesome :D Besides, I like using less mainstream formats, so I like WavPack :D

Anyway, I have several questions, and I hope that someone can put me at ease :D

So I converted my whole FLAC music archive to hybrid lossless WavPack. All 7000 of them. At x6. It took more than a week to finish. WavPack won't copy the tags from FLAC so I had to convert first to WAV then to x1 WavPack while manually copying the tags using Mp3tag, then I made a Python program to reencode all .WV files to x6. No errors ocurred (my Python program checks for WavPack's return/exit code, and doesn't consider a file reencoded if WavPack didn't return zero.

So here's my question. What are the chances that I have a corrupted WV file? I used verify option and added MD5 checksum while encoding. Is this enough to make sure that corrupted WV does not overwrite non-corrupted WVs?
Also, what are the chances that the audio has been modified when encoded to WV but was not reported as an error, so it is not an exact copy of the FLAC?
Also, I noticed that there are times when audio encoded with x8 is larger by up to around 250kb compared to the same audio encoded with x1. Is this a bug or something?
Also, I noticed that the gain setting wvgain gives is very much different from the ones that Foobar2000 uses. Which is correct, and which should I use?
One last question, sorry for being annoying XD
What is the quality of hybrid lossless when played lossy compared to other lossy codecs? Like, is the 210kbps lossy quality transparent, or is it noticeably lossy compared to a 210kbps LAME encoded MP3? I'm not sure, since I don't have really good headphones, but I think I heard "roughness" on tracks that are primarily vocal. I haven't ABXed, though, which is why I'm asking here and hoping it was just my headphones :D I know I shouldn't use spectrographs (?) to judge audio quality, but 210kbps WV's look awesome (just higher noise floor which shouldn't be noticeable, I think) compared to 320kbps CBR MP3 which is full of holes and lowpass.

Thank you for reading this, and please have a nice day :D
  • Last Edit: 11 September, 2017, 01:52:29 AM by OrthographicCube

  • Porcus
  • [*][*][*][*][*]
Re: So I migrated my whole FLAC collection to WV.
Reply #1
You could have used foobar2000 for conversion and gotten the tags transferred?

Anyway, even WavPack hybrid uses an audio MD5? (You used the format's internal audio-MD5, not your own file-checksums?)
If so, you can "easily" verify that they are the same as the FLAC files with foobar2000 and the foo_bitcompare component, assuming that you have not yet added any more files to the collection - and assuming that none of your FLACs were corrupted before you converted (important!):
- In fb2k, open (or index) both the FLAC set and the WavPack set
- In the view, create a checksum column.  Should contain %__md5%
- Mark the FLAC part and take note of number of files, length and number of samples. Make sure they match the ones for WavPack.
- With one of the sets highlighted (say, FLAC), sort by the checksum column. (They should now be every other FLAC and WavPack.)
- Cut the highlighted ones, and paste it at the top. Now you have FLAC-by-checksum on top and WavPack-by-checksum at the bottom.
- Select all, bitcompare.
- Wait for some days, because you chose x6 ;)

If everything is fine, that will be reported. But if as much as one file has an issue, that could mess up the order on everything subsequent ...  By the way, if the sets are the same folder/file structure and, e.g., two drives d:\ and e:\, you could sort by structure of course.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: So I migrated my whole FLAC collection to WV.
Reply #2
First of all, thanks for trusting your audio to WavPack!

You should not worry about the audio getting corrupted based on what you're done. When the -v option is used, WavPack will definitely not delete the original files until it has read the transcoded files and verified that the MD5 sums exactly match the source, and the command-line program will return 1 if even a single file fails (although the program will not stop until all files are done, assuming multiple files were specified).

I am surprised that any files would get bigger when going from -x1 to -x6, but I would need to see the other parameters. In pure lossless mode that would be very unlikely, but in hybrid mode it is not obvious to me how the improvement of -x6 would be distributed between the sizes of the lossy file and the correction file and the quality of the lossy file. The only thing certain is that there should be an overall improvement with the extra computing.

The gain values are different because WavPack and Foobar2000 use different algorithms, although I'm surprised you say they're “very much” different. This was actually recently discussed right here because FLAC also still uses the older ReplayGain algorithm.

As for the quality, a bitrate of 210 kbps would definitely not be transparent most of the time. It might sound fine with some [already noisy] music, but yeah, on solo vocal or instrumental music it would sound rough compared to MP3 (or more modern lossy codecs) at that bitrate. Even at 256 kbps defects are often noticeable, but between 320 kbps and 384 kbps they become pretty rare. I use 360 kbps most of the time, and have been happy.

I'm sorry if you were lead to believe that 210 kbps would be transparent, but fortunately if you have kept the correction files (which I assume you have) it should be easy to transcode again to a higher bitrate if you decide you don't like the noise and still want to stick with WavPack. The good news is that the overall lossless compression will get a little better with higher bitrates.

And yes, you shouldn't use spectrographs to judge audio quality :), because that higher noise floor is probably what you are hearing as “roughness”. If it was constant noise you would hear it as constant background hiss, but because it modulates with the music it creates that other effect.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: So I migrated my whole FLAC collection to WV.
Reply #3
- Wait for some days, because you chose x6 ;)
Well, the advantage of the -x modes is that they don't affect decode speed (which I assume is all we're talking about here).

But yes, if the FLAC files are still available then verifying the integrity of the conversion would be pretty straightforward. And of course one could try a few albums as a sample just to gain some confidence.

Re: So I migrated my whole FLAC collection to WV.
Reply #4
Thank you VERY much for the replies, and to have bryant, the developer, answer my question, I feel really happy XD

Here are my replies:
"You could have used foobar2000 for conversion and gotten the tags transferred?"
> I tried that at first, but Foobar2000 shows errors after conversion (like, converted file not found or something.) I believe this is because wavpack creates file with .tmp.wv extension, but when Foobar attempts to migrate the tags, it tries to find the .wv file, but it doesn't find it (maybe because wavpack still hasn't finished and hasn't renamed the file?) Anyway, I have issues with Foobar conversion, so I made a Windows batch program to convert the FLAC to WAV, the WAV to WV, then open Mp3tag on the directory so I can copy-paste the tags, then delete the WAV and FLAC once I close Mp3tag. Then, another Python program to reencode x1 WV to x6 WV (this took LONG on an Intel Celeron-based laptop LOL )

"Anyway, even WavPack hybrid uses an audio MD5? (You used the format's internal audio-MD5, not your own file-checksums?)"
> It seems so. The MD5 is for the lossless audio only, though, as decoding lossy doesn't match the stored MD5. Also, yep, I just added the -m switch, which should have wavpack compute the MD5 internally.

"If so, you can "easily" verify that they are the same as the FLAC files with foobar2000 and the foo_bitcompare component, assuming that you have not yet added any more files to the collection - and assuming that none of your FLACs were corrupted before you converted (important!):"
> I tried converting an album, and yes, they do have exactly the same MD5s. Also, funny thing, I did discover a corrupted FLAC in my collection XD

"First of all, thanks for trusting your audio to WavPack!"
> It's a relatively mature format, after all :D I trusted it enough to convert FLAC to WV while deleting converted FLAC XD I didn't have a choice, as I have no hard disk space left.

"although the program will not stop until all files are done, assuming multiple files were specified"
> My program doesn't use wildcards at all, just a simple SQL database that lists all WV files in my hard disk, and marks converted ones if they converted successfully. Of course, I made way for the return code 255, which is returned when I cancel execution using Ctrl+C.

"The only thing certain is that there should be an overall improvement with the extra computing."
> Unfortunately, no. My Python program treats the file size as the combined .WV and .WVC sizes. For example,
K:\Kylefile\Music\Sampled Music\Rock\before light\06 weep.wv
inflated 2.34 megabytes when I reencoded it (was it x2 for this album?) to x6.
This album is 96kHz/24bit, so it's a bit special.
I still haven't verified by converting it to x2 (or x1) to see if it will get smaller again, but I'm very certain that it got bigger after using x6 :) Not a big problem, though. I'm sorry to say that WavPack compresses a tiny bit worse and a lot slower than FLAC 8 but I really love the hybrid mode, so it's no biggie :D It's just a few megabytes after all. In the end, I saved about 1GB by converting from x2 to x6. (although I didn't know if I saved space by converting from FLAC to WavPack x6, but honestly, I don't care :D )

"although I'm surprised you say they're “very much” different"
> I meant, a difference of about 1 to 3 dB, if I remember correctly. For example, MusicBee and Foobar gives different results, but only a difference of 0.01 dB.

"As for the quality, a bitrate of 210 kbps would definitely not be transparent most of the time."
> I see. I don't mind, really :D I intend to use the lossy WV for my phone and Bluetooth headset, so the quality will be affected by the really bad lossy Bluetooth audio codec named "SBR" anyway, so I honestly don't mind. I still have the .WVC so I didn't lose anything :D See? How the hybrid mode really does the magic for me :D

"It might sound fine with some [already noisy] music"
> That's what I tested the quality with at first (I listen mostly to metal and rock, but I also listen to a very wide range of music, from EDM to video game music to chiptune), it was noisy so I didn't notice any difference :D

"I'm sorry if you were lead to believe that 210 kbps would be transparent"
> Nah, it's okay, I'm the one who convinced myself that it was transparent by testing the lossy mode with a metal track anyway :D Plus, I don't mind, really :D I dunno, I'm the kind of guy who likes to hear "imperfections" from time to time. I mean, I used to use 48kbps Opus before I transitioned to lossy WavPack so, yeah XD

"but fortunately if you have kept the correction files (which I assume you have) it should be easy to transcode again to a higher bitrate if you decide you don't like the noise and still want to stick with WavPack"
> Nah, the quality doesn't annoy me at all, it's fine :D

"Well, the advantage of the -x modes is that they don't affect decode speed (which I assume is all we're talking about here)."
> Yep, I avoided any -h modes because I considered mobile playback of the lossy WV. Didn't want to use -f either because I need to squeeze every bit of space possible since my 2TB hard drive is running out of space XD

"But yes, if the FLAC files are still available then verifying the integrity of the conversion would be pretty straightforward. And of course one could try a few albums as a sample just to gain some confidence."
> Well, not a single FLAC file remain, but when I did test conversions on a couple of albums, they never resulted in wrong MD5 checksums, which helped deveop trust :D Those test albums never gave a different MD5, but as for the REAL conversion, I have no way to check since I no longer have the FLACs, but I trust WV enough to just accept that it converted perfectly. Still, I'm running Foobar's "Verify inegrity" on all tracks now, with the hope of finding corrupted MP3s/AACs (I didn't touch the lossy files that I have) and if I'm unlucky, corrupted WVs.

Again, thanks for developing this codec, and I wish you more success in the future!

  • Porcus
  • [*][*][*][*][*]
Re: So I migrated my whole FLAC collection to WV.
Reply #5
Well, the advantage of the -x modes
Thanks for the correction, sorry for confusing -x with -h ...

(I've had enough unexpected issues with encoding (to FLAC, none to WavPack I should add), so I always verify afterwards. And in particular, if a file is indeed corrupted - for whatever reason - I do not want to build a new "valid file with corrupted audio".)

Re: So I migrated my whole FLAC collection to WV.
Reply #6
Yeah, fortunately, FLAC reported an error while decoding the one corrupted FLAC to WAV, so it didn't produce a WAV file, causing WavPack to fail when it tried to convert a non-existent WAV to WV--I didn't consider corrupt FLACs while making the Windows batch file  :D

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: So I migrated my whole FLAC collection to WV.
Reply #7
I tried a half dozen of my 24/96 files and could not duplicate what you saw. The -x6 files always got around 0.5% better compression than -x1 or -x2. I believe you saw this, but without having the file is difficult to understand why it would happen, although I can certainly imagine things that might cause it.

I'm glad that you say the ReplayGain difference is 1-3 dB, because that's in the same range as other people report, and is not really that big (a 1 dB difference is so small you can only hear it when switching immediately back and forth).

I am a little curious about the Foobar2000 problem (even though you got around it). WavPack only uses those temp file names when the target file already exists or any time dual-file mode is used (I got a little lazy there). I will try this with a recent Foobar2000 and see if I can duplicate it (next time I'm on Windows).

Of course it's too late now, but what you could have done is use Foobar2000 to convert to just plain single-file lossless WavPack (which should have worked) and then use the WavPack command-line tool to transcode to dual-file mode (it can convert back and forth between single file and dual-file lossless too and I use this to get around, for example, CD rippers that don't handle the dual-file stuff). Oh well...

Glad everything seems to be okay now anyway!

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: So I migrated my whole FLAC collection to WV.
Reply #8
OrthographicCube sent me a sample of the files that were resulting in worse compression with -x6 (thanks!) and I have performed several experiments and think I have a good idea why this is happening.

Yes, the -x6 mode is a little worse than -x1 (or even no -x), but that is actually a minor issue (and one I don't really understand). A much bigger issue is just how bad the compressor is working here, and I have figured out that it is caused by a combination of the very low target bitrate being used and the fact that it's a 96 kHz file.

When WavPack encodes hybrid at 96 kHz it uses aggressive noise shaping to move noise up out of the audible range. This normally works okay; it results in slightly worse overall compression and noise, but reduces the audible noise a lot. However, at this low bitrate the noise is being amplified by the decorrelation filters resulting in huge amounts of extra noise and really bad compression. I have seen this before in some special circumstances, but never really thought that it would happen this bad with just regular 96 kHz audio.

Fortunately, the fix is pretty simple. There is an option -cc that replaces the regular -c. This eliminates the heavy noise shaping that is causing this issue and on this test file reduces the noise by almost 14 dB, and improves the compression by almost 10%! I know it's time consuming, but I think it would definitely be worth reëncoding the 96 kHz files like this. They'll be a huge amount of space saved in the correction files while leaving the lossy files about the same (except with less noise). Another fix would be to keep the aggressive noise shaping and increase the target lossy bitrate to a more reasonable value, say -b3.5. Of course, that will result in bigger lossy files.

I'll do some more investigation, but at this point I'm not sure what to do about this. It's not really a bug, but I can imagine other people having this issue and not even really knowing it's happening. After all, this was only noticed because of the odd -x6 behavior (which does go away with either of the above fixes), but the real problem is the really bad compression with low bitrates and heavy noise shaping.

Re: So I migrated my whole FLAC collection to WV.
Reply #9
Thank you very much for the technical explanation--as a fellow (amateur) programmer, I like to know technical details about how stuff works (or doens't work  :D )

I have thoroughly read the user manual before finally deciding on the parameters that I would use, and I did see the -cc option. In the manual, it says that it "tells WavPack to optimize for the overall compression ratio instead, even if this means some possible degradation of lossy quality". Honestly, I would have used it only if it didn't say that "the effect of this option is not too significant either way"  :D

Sure, I'd reencode all the WVs with -cc if it reduces file size significantly. (It'd take me a bit more than a week of waiting as my Intel Celeron laptop converts 24 hours a day, but I will reencode if I need too--I really need HD space :D ) As I said, I don't mind hearing lossy artifacts during lossy decoding--in fact, I kinda love hearing those, I don't know why, but I guess it reminds me of how complex audio codecs are and it makes me impressed that this works  :D

However, what do you recommend? Do I reencode my whole collection to -ccx6 or, do you think only these high-resolution files are worth reencoding? What is the probability that a normal CD-quality file will get huge benefits from using the -cc option?

Again, thanks for your response, I really appreciate it :)
  • Last Edit: 20 September, 2017, 01:47:18 AM by OrthographicCube

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: So I migrated my whole FLAC collection to WV.
Reply #10
However, what do you recommend? Do I reencode my whole collection to -ccx6 or, do you think only these high-resolution files are worth reencoding? What is the probability that a normal CD-quality file will get huge benefits from using the -cc option?
I recommend this for only the high sampling rate material. You might see a little improvement with the regular stuff but, like I said, normally it won't make much difference. With the 96 kHz stuff and the low bitrate, the difference will be huge.

Of course, you can always try a few albums and see what happens, that's the advantage of lossless!  :)

Re: So I migrated my whole FLAC collection to WV.
Reply #11
Thanks!

I decided to reencode everything. It should be done in 2 weeks.

I noticed that in some (or most?) 96/24 audio, WavPack compresses better than FLAC, up to more than a megabyte difference. That's awesome. I only have few hi-res music, but I do have gigabytes of DSD files, and I'm starting to migrate those files to WavPack too.

Thanks for this codec!