Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: lossyWAV Development (Read 573671 times) previous topic - next topic
0 Members and 5 Guests are viewing this topic.

lossyWAV Development

Reply #1025 –
Quote from: Nick.C on
I thought that the problem of hiss you were encountering would be white noise, i.e. shaping too low?

Is full -shaping 1.0 better than -autoshape for the problem samples you identified (bruhns, bibilolo, badvilbel)?

And finally (as if you had nothing else better to do  ) is the v0.9.1 -autoshape any better (if full shaping is better than autoshape v0.9.0)?

I tried v0.9.0 -autoshape, v0.9.0 -shaping 1, v0.9.1 -autoshape on bruhns.
To me the v0.9.0 -autoshape and v0.9.1 -autoshape quality is identical (both from subjective impression as well as the abx results: sec. 9.3-10.2: 7/10 with both versions, sec. 2.3-4.4: 10/10 with v0.9.0 and 9/10 with v0.9.1).
Judgement about -shaping 1 isn't so easy. After listening to -autoshape I had problems identifying the problem with -shaping 1 and scored badly. Once more used to it however I could recognize it and with some trials the problem was even more pronounced to me than with the autoshape versions. Guess my hearing abilities are very much at their limits with these very high frequency problems, but from time to time they become apparent even to me.

I also wanted to try bibilolo sec. 4.3-5.5 but didn't succeed at all in abxing it today (fatigue? bad constitution?)
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #1026 –
Why lossyWAV ?

I think that Musepack can do it better...

lossyWAV Development

Reply #1027 –
Kwevej, read the lossyWAV FAQ.
lossyFLAC (lossyWAV -q 0; FLAC -b 512 -e)

lossyWAV Development

Reply #1028 –
Quote from: Kwevej on
Why lossyWAV ?

I think that Musepack can do it better...

Musepack, as well as Vorbis, AAC, mp3, does it better at a bitrate in the 200 kbps area (very roughly speaking). Most people don't have the need for anything else (apart from maybe lossless archiving).

The special thing about lossyWAV is that when used the way the initial purpose was (achieved by using quality level -2 or -1) we get an extremely high quality.
There is no guarantee that things can't go wrong (after all it's lossy), but according to experience there's nothing to be afraid of. Codecs like mp3 and the others mentioned above have a complicated signal path which changes the original technical description of the music enormously, and there is a lot of heuristic decision making. As a result there's always the risk of artefacts and inaccuracies in the changed technical description though we all know codecs like Vorbis do a great job at pretty low bitrate, and it's very rare that music isn't transparent at a bitrate of ~200 kbps (usually that's overkill already).
LossyWAV in contrary doesn't change the structure of the technical description of the music at all, it uses the usual 16 bit PCM description of the wave samples and only reduces the accuracy of the samples (by rounding and thus zeroing those least significant bits which it thinks it can safely do so. The 16 bits of the PCM format are needed to accurately describe the full dynamics of loud as well as quiet music. For quiet music we can't save a lot of bits because quiet music is described with rather few bits (many of the most significant bits are zero) which we usually need for an accurate description, but for loud music we can - the full 16 bit accuracy usually isn't needed here.)
The downside is that with this approach we can't get the efficiency of Musepack etc. Using lossyWAV the secure way yields a bitrate of ~420 kbps for the -2 setting (or ~470 kbps for the -1 setting which is considered to be overkill, just for the very cautious minded or those who like to use extreme quality lossyWAV as a replacement for lossless archiving.).

There is a wish for going lower in bitrate, and due to additional internal quality assuring mechanisms we can do so without going very risky. This however isn't backed up by the initial idea anymore. To me the -4 quality setting still is transparent and yields a bitrate of ~350 kbps. Even when going lower like with -6 (~310 kbps) the rare and subtle inaccuracies are easily acceptable to me.
Noise shaping is the current theme of further development, and maybe it's possible this way to improve transparancy in the 300-350 kbps range and/or achieve extremely high though not transparent quality a bit below 300 kbps.
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #1029 –
Nick, is there any chance of seeing piping support (both in and out), before lossyWAV 1.0?
lossyFLAC (lossyWAV -q 0; FLAC -b 512 -e)

lossyWAV Development

Reply #1030 –
Quote from: Mitch 1 2 on
Nick, is there any chance of seeing piping support (both in and out), before lossyWAV 1.0?
Unless I were to know where to start, i.e. how *does* foobar2000 pipe WAV data into a program, then how do I pipe that out to FLAC / WavPack / TaK / etc to produce the encoded processed output?

I think it may be better to park piping beside noise shaping and attempt to include it in v1.1.0 rather than to delay the work up to release of v1.0.0 any further.

I have been working on tidying up the code and have shaved another second off the processing time for my 53 problem sample set at preset -7:
Code: [Select]
|======|==================|==================|
|  QS  | Time/Rate v0.9.2 | Time/Rate v0.9.0 |
|======|==================|==================|
|  -7  | 13.14s / 56.60x  | 14.34s / 51.86x  |
|  -7a | 18.26s / 40.73x  | 19.30s / 38.54x  |
|  -7b | 24.25s / 30.66x  | 24.56s / 30.27x  |
|  -7c | 28.62s / 25.98x  | 29.47s / 25.23x  |
|======|==================|==================|
All tests were carried out with the input files cached in memory to ignore read latency.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #1031 –
Quote from: Nick.C on
Unless I were to know where to start, i.e. how *does* foobar2000 pipe WAV data into a program, then how do I pipe that out to FLAC / WavPack / TaK / etc to produce the encoded processed output?
Now you're getting ahead of yourself.  I was simply asking about stdin/stdout support.
To support foobar2000, however, I suppose you could launch (with parameters) an external encoder, e.g. "lossyWAV.exe - -o -enc flac.exe -f -b 512 -e -o %d -".
lossyFLAC (lossyWAV -q 0; FLAC -b 512 -e)

lossyWAV Development

Reply #1032 –
Hi all,

I was messing around with an audio recording from a YouTube video. I got rid of everything > 14kHz. So this isn't your high quality CD audio here (i.e. perhaps not lossyWAVs intended use), but I was surprised that when I ran it through LossyWav (-2) the resultant lossy.flac file was larger (747kbps) than the lossless FLAC file (737 kbps), both were encoded with the latest FLAC using -5.

Is that odd? Seems odd to me.

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

lossyWAV Development

Reply #1033 –
Not really odd.  The noise-floor above your filter was very very low so there's very little if any bits for lossyWAV to remove, and combined with the difference in block size used in FLAC (assuming that you didn't force them both to 512) then... that makes sense.

lossyWAV Development

Reply #1034 –
Quote from: carpman on
Hi all,

I was messing around with an audio recording from a YouTube video. I got rid of everything > 14kHz. So this isn't your high quality CD audio here (i.e. perhaps not lossyWAVs intended use), but I was surprised that when I ran it through LossyWav (-2) the resultant lossy.flac file was larger (747kbps) than the lossless FLAC file (737 kbps), both were encoded with the latest FLAC using -5.

Is that odd? Seems odd to me.

C.
The upper limit for lossyWAV is 16kHz - so you had a 2kHz zone which would not allow any bits to remove.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #1035 –
Nick, jesseg

Thanks for your replies and patience.

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

lossyWAV Development

Reply #1037 –
Quote from: Kwevej on
Quote from: Mitch 1 2 on
Kwevej, read the lossyWAV FAQ.


Someone will send me a FLAC. How would I recognize, that it is really lossless?
I don't like the "Lossy Lossless" idea.
If you rip your own FLAC files, you will always know whether they are lossless or not.

Also, how do you know that the FLAC file you have has not been created from a decoded MP3 file?
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #1038 –
Good point Nick.

Re: knowing if a FLAC is a lossyFLAC...  if --keep-foreign-metadata was used in the FLAC command-line, then you will be able to know if the source file was a lossyWAV or not (but saying nothing of the pre-lossyWAV source).  It's a little round-about way because you would (as far as i know) have to decode it to wav, and then use the lossyWAV -check method.  But...  if there's a way to view metadata in a FLAC file directly, then that would be easier.

And as Nick said, if a FLAC is encoded from an mp3, you have no way at all of knowing without a doubt through a purely technological means, unless the decoder saves the information as meta-data in the WAV or passes it through to a FLAC tag via a transcoder.  And again, FLAC would have to be set to save the meta-data.  Otherwise, it's all judgmental (and inaccurate) at that point, when you're trying to spot coding/decoding artifacts and decide what is what.

lossyWAV Development

Reply #1039 –
Quote from: Nick.C on
Quote from: carpman on
Hi all,

I was messing around with an audio recording from a YouTube video. I got rid of everything > 14kHz. So this isn't your high quality CD audio here (i.e. perhaps not lossyWAVs intended use), but I was surprised that when I ran it through LossyWav (-2) the resultant lossy.flac file was larger (747kbps) than the lossless FLAC file (737 kbps), both were encoded with the latest FLAC using -5.

Is that odd? Seems odd to me.

C.
The upper limit for lossyWAV is 16kHz - so you had a 2kHz zone which would not allow any bits to remove.

Nick would you mind expanding on "The upper limit for lossyWAV is 16kHz"?
I assumed you meant that it was the HF cut-off point so that anything above that frequenecy would be "ignored" and hence missing from the output. However, having done a brief test on a piece recorded from FM radio the frequency plots from the original wav file and the lossywav file look the same, with no reduction in >16k levels. Especially noticeable is that the 19khz pilot tone is still there and not reduced in level.

Sorry if this is a dum question

lossyWAV Development

Reply #1040 –
Quote from: botface on
Quote from: Nick.C on
Quote from: carpman on
Hi all,

I was messing around with an audio recording from a YouTube video. I got rid of everything > 14kHz. So this isn't your high quality CD audio here (i.e. perhaps not lossyWAVs intended use), but I was surprised that when I ran it through LossyWav (-2) the resultant lossy.flac file was larger (747kbps) than the lossless FLAC file (737 kbps), both were encoded with the latest FLAC using -5.

Is that odd? Seems odd to me.

C.
The upper limit for lossyWAV is 16kHz - so you had a 2kHz zone which would not allow any bits to remove.
Nick would you mind expanding on "The upper limit for lossyWAV is 16kHz"?
I assumed you meant that it was the HF cut-off point so that anything above that frequenecy would be "ignored" and hence missing from the output. However, having done a brief test on a piece recorded from FM radio the frequency plots from the original wav file and the lossywav file look the same, with no reduction in >16k levels. Especially noticeable is that the 19khz pilot tone is still there and not reduced in level.

Sorry if this is a dum question
Not a dumb question at all - I am guilty of giving a truncated explanation of what I should have elaborated on.....

When the FFT analyses are carried out on each codec_block in lossyWAV, the results between 20Hz and 16kHz are taken into account when determining bits_to_remove for that FFT (and ultimately that codec_block). The only process applied to the actual audio data is the remove_bits routine, i.e. revised_sample:=round(original_sample / (2^bits_to_remove))*(2^bits_to_remove) which sets the lowest bits_to_remove lsb's to zero. No frequencies are intentionally removed from the output samples.

Anyway, due to the lack of problematic feedback for beta v0.9.1, lossyWAV v0.9.2 RC3 is attached to post #1 in this thread.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #1041 –
Quote from: Kwevej on
Someone will send me a FLAC. How would I recognize, that it is really lossless?
I don't like the "Lossy Lossless" idea.
If it came from an mp3 file, you can often spot this in the spectrogram.

If it came from a lossyWAV file, you can count the number of "wasted bits" in the FLAC file, or the number of (512-sample) blocks of LSBs set to zero (where some MSBs are non-zero) in the decoded .wav. If you find several of either, it's probably from lossyWAV (or some even rarer perversion of the audio).


If you add noise to either of the above, these methods won't work.

Cheers,
David.
 

lossyWAV Development

Reply #1042 –
Thanks to unfortunateson for the 96khzsample FLAC file - when I tried to process the contained WAV file lossyWAV crashed. It turned out to be a divide by zero error in the preparation of the skewing factors.

This led me to re-assess the skewing factor preparation and I quickly found a simple fix (which also improved the methodology) - however, the fix reduces the bitrate of all the quality presets by around 20kbps.

I had already made an unrelated change to the spreading function which increased the bitrate for -3 to -7 by between 2kbps and 4kbps.

However, the amendment to the skewing function preparation has reduced the difference in bitrate between the 3 existing spreading functions.

So, I have amended the skewing function preparation and there is now only one spreading function (that for quality preset -1).

lossyWAV beta v0.9.3 attached to post #1 in this thread.

As this beta has changed some longstanding "constants" of the method, I will be extremely grateful if some of our more acutely eared members could ABX some of the more problematic samples and post feedback.

I am fairly sure that quality *should* not have suffered, but my ears are not good enough to perform the critical evaluation required.

Thanks,

Nick.

I have processed my 53 problem sample set using beta v0.9.3 and the change in spreading function has changed the variation in bitrate somewhat:

Code: [Select]
|-------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
|   Version   |lossyWAV -1|lossyWAV -2|lossyWAV -3|lossyWAV -4|lossyWAV -5|lossyWAV -6|lossyWAV -7|
|-------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| v0.9.2 RC3  |  543kbps  |  494kbps  |  433kbps  |  408kbps  |  385kbps  |  365kbps  |  348kbps  |  
|-------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| beta v0.9.3 |  505kbps  |  467kbps  |  435kbps  |  406kbps  |  381kbps  |  357kbps  |  337kbps  |
|-------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
From this, I feel that maybe -1 should become (even) more conservative in its -snr and -nts settings (currently 24 and -3 respectively). However, the change in change in bitrate between quality presets is now significantly more linear than it has ever been.

Also, as all quality presets now use the same spreading function I could (nearly) implement a fractional quality preset (like oggenc) between -1.000 and -7.000.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #1043 –
Quote from: Nick.C on
... I had already made an unrelated change to the spreading function which increased the bitrate for -3 to -7 by between 2kbps and 4kbps. ...

Very welcome IMO as this important frequency range didn't get much of the skewing effect so far. But why don't you do it for -2 and -1 as well especially as bitrate seems to have dropped by your changes?
As far as I can see 2Bdecided's basic principle is still totally taken care of when using -2 and -1, and that's the important thing.
I wouldn't care much about the bitrate drop. Maybe the lowest quality settings aren't acceptable any more, but IMO a quality scale from -1 to, say, -5 is sufficient.

I will do my usual tests as soon as possible, but my father has died so at the moment I will only seldom look up HA.
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #1044 –
Quote from: Nick.C on
there is now only one spreading function (that for quality preset -1).

Does that mean a performance hit? (I remember that -1 uses the most CPU but I'm not sure if that was partly because of the spreading).
In theory, there is no difference between theory and practice. In practice there is.

lossyWAV Development

Reply #1045 –
Quote from: GeSomeone on
Quote from: Nick.C on
there is now only one spreading function (that for quality preset -1).
Does that mean a performance hit? (I remember that -1 uses the most CPU but I'm not sure if that was partly because of the spreading).
Not at all, the spreading is actually quicker for -1 as fewer bins are averaged across the frequency ranges. The performance hit for -1 is related to the extra 256 sample FFT's which are calculated.

However, I have implemented the floating point quality presets from -1.0000 to -7.0000 and am considering removing the 256 sample FFT from the -1 quality preset (it can still be added using -1a to -1.9999a) which would make all quality presets default to 2 FFT analysis lengths (64 sample and 1024 sample) with 128, 256 and 512 sample FFTs remaining optional.

I will post beta v0.9.4 later today with the FP quality presets enabled and -1 defaulting to 2 FFT analysis lengths.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #1046 –
Quote from: halb27 on
There is a wish for going lower in bitrate, and due to additional internal quality assuring mechanisms we can do so without going very risky. This however isn't backed up by the initial idea anymore. To me the -4 quality setting still is transparent and yields a bitrate of ~350 kbps. Even when going lower like with -6 (~310 kbps) the rare and subtle inaccuracies are easily acceptable to me.


Thanks for detailed explanations. But is there any difference between lossyWAV and f.i. WavPack lossy ? At high bitrates (> 320-350 kbps) WavPack lossy seems to sound transparent too. Is there any advantages for lossyWAV over WavPack in lossy mode ?

lossyWAV Development

Reply #1047 –
Quote from: SokilOff on
Quote from: halb27 on
There is a wish for going lower in bitrate, and due to additional internal quality assuring mechanisms we can do so without going very risky. This however isn't backed up by the initial idea anymore. To me the -4 quality setting still is transparent and yields a bitrate of ~350 kbps. Even when going lower like with -6 (~310 kbps) the rare and subtle inaccuracies are easily acceptable to me.
Thanks for detailed explanations. But is there any difference between lossyWAV and f.i. WavPack lossy ? At high bitrates (> 320-350 kbps) WavPack lossy seems to sound transparent too. Is there any advantages for lossyWAV over WavPack in lossy mode ?
Only really that lossyWAV is compatible with a number of lossless codecs which make use of the wasted-bits approach so is in one sense codec independent. I don't know if anyone has carried out any comparisons between WavPack lossy and lossyWAV output - it would be interesting though....
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #1048 –
IIRC the lossy version of Wavpack on stable release supports CBR only. I can't remember if there's VBR in beta test.

lossyWAV is pure VBR, and does not support CBR.

Cheers,
David.

lossyWAV Development

Reply #1049 –
Quote from: SokilOff on
Quote from: halb27 on


There is a wish for going lower in bitrate, and due to additional internal quality assuring mechanisms we can do so without going very risky. This however isn't backed up by the initial idea anymore. To me the -4 quality setting still is transparent and yields a bitrate of ~350 kbps. Even when going lower like with -6 (~310 kbps) the rare and subtle inaccuracies are easily acceptable to me.


Thanks for detailed explanations. But is there any difference between lossyWAV and f.i. WavPack lossy ? At high bitrates (> 320-350 kbps) WavPack lossy seems to sound transparent too. Is there any advantages for lossyWAV over WavPack in lossy mode ?

lossyWAV has a quality control, whereas wavPack lossy hasn't. With wavPack lossy you give a target bitrate which is internally converted to an accuracy demand for the predictor error. This is not directly related to overall accuracy (cause in case the predictor is seriously wrong, it takes a high degree of accuracy for the predictor error to get at a good overall accuracy). There's more to it which takes into account special problems, but roughly speaking it's like that.
The disadvantage of lossyWAV as compared to wavPack lossy is that for the lossless part a small blocksize of 512 samples is necessary to make good use of the varying bits-to-remove. This however makes the lossless codec less efficient. Moreover David Bryant has implemented an effective noise shaping in wvPack losssy. Noise shaping in lossyWAV is work in progress and at the moment is problematic as it blows up bitrate of the lossless codec because of added high frequency hiss of rather high volume.

So at the moment I think with an average bitrate of >~ 400 kbps lossyWAV is to be preferred (using -2 or -1) because of the better accuracy control.
At a bitrate of roughly 350 kbps I think both codecs' quality is comparable. They are both expected to be transparent with few exceptions. Maybe the number of exceptions is a bit fewer with lossyWAV, but that's speculation, and I think we can be very content with both codecs.
At a bitrate below ~300 kbps I think wavPack lossy is preferable because of it's more efficient coding which becomes more and more important the lower we go with bitrate.
lame3995o -Q1.7 --lowpass 17