In my own tests, lossyWAV 0.8.8 is significantly faster than version 0.8.7.

Would it make sense to drop all of the extra FFT analysis options (-Xa/b/c), in favour of an "-exhaustive" (or "-e") parameter, which would perform analysis passes using all suitable FFT sizes?

|======|==================|==================|
|  QS  | Time/Rate v0.8.8 | Time/Rate v0.8.7 |
|======|==================|==================|
|  -7  | 15.71s / 47.34x  | 20.47s / 36.33x  |
|  -7a | 21.29s / 34.93x  | 27.84s / 26.71x  |
|  -7b | 27.13s / 27.41x  | 35.97s / 20.67x  |
|  -7c | 32.44s / 22.92x  | 43.44s / 17.12x  |
|======|==================|==================|

... As an aside (and I know that looking at the spectrum in foobar is not any way to evaluate anything....) I looked at the spectral output for a lossyWAV correction file (replaygained +45dB or so) and almost all of the signal was in the high end of the spectrum - so it "looks" like my implementation of your noise shaping filter works!

Some proposals on the bitrate bloat issue:

a) on a per block basis decide which shaping yields the higher number of bits to remove:
shaping 0 or shaping 1 (or shaping x, y, z, ....).
I came to this idea because shaping 1 does not always yield a bitrate bloat. For dither_noise_test shaping 0 yields 705 kbps, whereas shaping 1 yields 295 kbps (when used together with -7), and I couldn't hear a problem.
Sure this means at least doubling the encoding time. Moreover maybe the changing of the noise shaping is audible (but the same argument applies as towards the sudden noise increase and decrease when the anti-clipping strategy goes to work: as long as the noise is hidden we shouldn't mind).
Maybe an autoshaping strategy like this is most adequate: start for the first block with a low shaping value like shaping = 0.2.
For any current block: try the shaping done with the last block, as well as a with two shaping values that add resp. subtract a certain delta from the last shaping value. Always use the shaping from the three possible values which maximizes the number of bits to remove.
This is the basic principle. In order to save some work, checking for changing the shaping value is not necessarily done on every block or with both directions. For instance with the current block it is checked only whether an increase of the shaping value is useful. On the next block the check goes in the opposite direction: check only whether decreasing the shaping value is useful, and so on interchangingly possibly increasing resp. decreasing shaping. Things like that. Maybe the frequency of the changes can trigger pauses for the shaping checking. When changes are rare it's not useful to do the checking with every block.

b) Maybe a more direct approach is more efficient: decide on the spectrum of the signal which shaping to use. When there's a lot of HF in the music, putting the noise into the HF region should be a good thing. Not quite so when there's no HF present in the music to hide the noise.

c) Maybe giving the potential to shift the noise also towards low frequencies instead of only shifting up may be useful with the approaches of a) or b).

d) Think of quality level -7 as of targeting at rather low bitrate lovers accepting some compromise. So for -7 a soften the accuracy requirements for the highest spf frequency zone. For instance don't check this zone at all other than for the 64 sample FFTs (so far this shouldn't seriosly hurt). If necessary: use a spreading of 5 or even 6 instead of 4 for the highest frequency zone in the 64 sample FFT analyses (this hurts - a bit for a spreading of 5, more so when going higher).

Unfortunately, the -shaping parameter barely changes the bits-to-remove as calculated by lossyWAV, however as SebastianG said earlier it makes the predictors in the lossless codec work less efficiently. ...

Quote from: Nick.C on 2008-03-29 17:12:47

Unfortunately, the -shaping parameter barely changes the bits-to-remove as calculated by lossyWAV, however as SebastianG said earlier it makes the predictors in the lossless codec work less efficiently. ...

I see, and now that you write it I remember SebastianG's remark. If this is so: maybe shifting noise downwards in a controlled way makes things easier for the predictor. It's often helpful with wavPack lossy when using rather low bitrate.

[That would take a new noise shaping function. SebastianG very kindly donated his 44.1kHz and 48kHz functions to get noise shaping working in lossyWAV, but I have no idea how to derive a new one which ideally would push noise above 20kHz and below, say, 10Hz.

I have a question for Nick C.

The resulting processed WAV file is smaller than the original ?
Eg. I could burn an entire lossyWAV album in WAV/PCM format without taking the whole space it would with the standard WAV?

so it's the lossless codec that takes advantage over the processed WAV...

Thanks for the tests halb27. Good to hear we have a strong 300..350k range. Bitrates are also looking good with -2 giving archive quality at half the normal bitrate of lossless.

... Using the -7 -autoshape with my 53 sample set I get 366.8kbps compared to 348.1kbps for -7 and 399.2kbps for -7 -shaping 1.0.

Quote from: Nick.C on 2008-03-29 21:14:05

... Using the -7 -autoshape with my 53 sample set I get 366.8kbps compared to 348.1kbps for -7 and 399.2kbps for -7 -shaping 1.0.

Sounds promising. I'm curious what it looks like with regular music.

As it is now, lossyFLAC + lwcdfFLAC is still smaller than a plain FLAC, especially using lossyWAV -1 preset.

Great work Nick.

Re: the bitrate bloat due to "shaping" - surely the whole point of shaping the noise is so that you can add more of it while maintaining the same level in the audible band? So when you enable shaping, you should also add a threshold shift. (Sorry if you're doing this already!).

Re: lower bitrate than normal FLAC: lossyFLAC uses a different default block size. Sometimes the output is smaller than the default FLAC blocksize, sometimes it's larger - maybe this is what's happening? Overall, what you say is correct: lossy+correction should be larger than lossless, but it would be nice if it wasn't!

Re: transcodability: I tested with mp3 problem samples very early on. It would be worth re-testing with the current version. I found you can't be too aggressive if you want to avoid any audible difference (e.g. -1!), but if "different but not worse" is good enough, you can use normal settings (which at the time was -2!). that horrible "trumpet" sample was quite revealing.

Cheers,
David.

  quality_noise_threshold_shifts    : array[1..Quality_Levels] of Integer = (-3,-0,3,6,9,12,15);

  quality_signal_to_noise_ratio     : array[1..Quality_Levels] of Integer = (24,22,20,19,18,17,16);

|======|==================|==================|
|  QS  | Time/Rate v0.8.8 | Time/Rate v0.9.0 |
|======|==================|==================|
|  -7  | 15.71s / 47.34x  | 14.34s / 51.86x  |
|  -7a | 21.29s / 34.93x  | 19.30s / 38.54x  |
|  -7b | 27.13s / 27.41x  | 24.56s / 30.27x  |
|  -7c | 32.44s / 22.92x  | 29.47s / 25.23x  |
|======|==================|==================|

Thank you, Nick.
I'd like to do some abxing using -autoshape, but because abxing isn't so much fun I've been waiting for your version which takes the RMS value into account.
Does v0.9.0 contain this feature?

OK, I'll try to abx my usual problem samples with v0.9.0 -7 -autoshape. I'll also search for other tracks looking for hiss or other HF problems.