lossyWAV Development

Topic: lossyWAV Development (Read 573696 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

lossyWAV Development

Reply #850 – 2008-01-31 17:00:18

Quote from: halb27 on 2008-01-25 18:05:19

In the end: do you think with the two FFT windows -448:575, -64:959 for the 0:511 block the edges are not covered well by these?

Yes, I already said...

Quote from: 2Bdecided on 2008-01-25 13:07:05

As long as there is at least 50% overlap, and all the blocks is covered by a 0.5 or higher parts of the window function, it really doesn't matter which of the two or three proposed schemes you use.

...what I mean was that it is good enough - I am happy with it.
(Even one in the centre is good enough!)

Think about it the opposite way:
1. forget the blocks!
2. consider that the moment with the lowest noise floor could be anywhere
3. pick an amount of window overlap that you're happy will catch this moment adequately, wherever it is relative to the window
4. now remember the blocks again, and use that lowest noise floor to set the bits_to_remove in the appropriate block.

Your suggestion increases the block overlap slightly, in a non-uniform way. It's fine. It may be beneficial (either because it overlaps more in the current block, or ignores more of the adjacent blocks), or it may be wasteful (because the existing method is fine already and efficient). I don't know.

Quote

I guess we have the same thing in mind: accuracy at the edges

No, I'm happy with 50% overlap and centred anywhere. But if you're going to centre it anywhere, it might as well be at the edges.

Quote

but for that IMO the centre point needn't be exactly at the edge but can be a little bit interior to the block.

That's true - but it's only useful if 50% overlap isn't good enough - i.e. if it's too little for within the block, or too much for outside the block. I prefer the solution (if there's a problem) which adjusts the threasholds etc so that 50% overlap is sufficient and resilient to wherever the minimum happens to be relative to the window, but that may be because 50% overlap gives equal and efficient coverage over time, and I like that.

Quote

The advantage is that with such a choice the centre region is taken better care of which is a bit underexposed with the center of the 2 FFT windows situated exactly at the edges.

That's the thing though: if it is underexposed (i.e. it ever causes a problem), I would conclude that the algorithm is wrong and the thresholds or overlap need to be adjusted to compensate. I might do what you've proposed, or something different, but I'd want to find something where it went wrong to decide what's most appropriate ti fix it. The sample I provided, if no one can hear any difference, seems to indicate that there's nothing to fix.

But I'll say it again - any solution with 50% or more overlap is fine by me wherever the windows fall. (unless a problem sample due to this crops up! if deleting or adding half a block of silence to any sample causes a dramatic change, then it really needs to be looked at).

Cheers,
David.

lossyWAV Development

Reply #851 – 2008-02-01 09:39:37

Essentially you say that everything should be fine as long as each sample is off the centre of the corresponding FFT window to a maximum of 50% the FFT length.

Guess it's due to my paranoid nature towards audio that I would prefer a lower value than 50%, but you certainly are more experienced. And as for practical experience you're right: everything looks fine with the 50% overlap.

I see we're getting a lot of variations and options and have arrived at that point already. As far as this is due to me: let's forget about my personal preferences. Now that we're close to the final version it's more important to have the options clean and simple.

lossyWAV Development

Reply #852 – 2008-02-01 21:50:13

My preference vis-a-vis FFT end_overlap and FFT_overlap is 50%/50%, i.e. -512:511;0:1023 for 1024 sample FFT. Yes, this takes into account 3 codec blocks for the 1024 sample analysis, but I feel that this works better than simply using -256:767 as we will have the block ends at 100% not 50%.

I'm still hurting my head on the -merge parameter.....

lossyWAV Development

Reply #853 – 2008-02-01 22:11:56

With -256:767 the block's edges are 50% the FFT length away from the centre, so the general 50% strategy applies.

lossyWAV Development

Reply #854 – 2008-02-01 22:43:50

Quote from: halb27 on 2008-02-01 22:11:56

With -256:767 the block's edges are 50% the FFT length away from the centre, so the general 50% strategy applies.

I know, but as more bits are removed, I am worried that quality might be suffering.... -overlap 16 (-256:767) = 455.1kbps for my 53 sample set vs 461.5kbps for the existing -512:511;0:1023 processing.

lossyWAV Development

Reply #855 – 2008-02-24 12:50:20

What about new beta versions? Or development is stopped?

lossyWAV Development

Reply #856 – 2008-02-24 13:54:28

Quote from: amors on 2008-02-24 12:50:20

What about new beta versions? Or development is stopped?

I've been working (slowly) on the -merge parameter - it's taking some time.

The settings for each of the quality presets are pretty much cast in stone now (pending identification of new problem samples). However.....

Reading back through the thread, I'm almost tempted to include a "-4" parameter which would be the same as -3 was at v0.6.4 RC1 but with 5 allowable clips per channel per codec_block - as was said at the time, to re-write the settings for -3 due to only one problem sample might be considered to be a knee jerk reaction.

lossyWAV beta v0.7.6 attached to post #1 in this thread.

lossyWAV Development

Reply #857 – 2008-02-24 15:22:14

Thank you for the answer and your work.

lossyWAV Development

Reply #858 – 2008-02-25 08:05:01

I for one, appreciate your -4 option. Currently testing, but I doubt that I will hear any problems.
I'm another one saying thanks to everyone who's been involved in this project.
Great sound quality and an increase in battery life on my DAP what more could my ears ask for...

lossyWAV Development

Reply #859 – 2008-02-27 07:57:30

Quote from: stel on 2008-02-25 08:05:01

I for one, appreciate your -4 option. Currently testing, but I doubt that I will hear any problems.
I'm another one saying thanks to everyone who's been involved in this project.
Great sound quality and an increase in battery life on my DAP what more could my ears ask for...

Thinking about the -4 preset and bearing in mind the following table which shows the processed sizes for each of the current presets (and a/b/c variants):

Code: [Select]

53 "Problem" Sample Set Processing Results
==========================================
WAV 131,183,096 bytes 1411.2kbps
FLAC 72,652,785 bytes  781.6kbps (-8)
-1a  52,746,167 bytes  567.4kbps (-5)
-1   51,856,977 bytes  557.9kbps (-5)
-2b  49,032,764 bytes  527.5kbps (-5)
-2a  48,851,896 bytes  525.5kbps (-5)
-2   47,865,987 bytes  514.9kbps (-5)
-3c  43,742,164 bytes  470.6kbps (-5)
-3b  43,497,733 bytes  467.9kbps (-5)
-3a  43,235,774 bytes  465.1kbps (-5)
-3   42,976,155 bytes  462.3kbps (-5)
-4c  39,396,622 bytes  423.8kbps (-5)
-4b  39,238,991 bytes  422.1kbps (-5)
-4a  39,016,370 bytes  419.7kbps (-5)
-4   38,821,415 bytes  417.6kbps (-5)

I am tempted to make -4 equivalent to -4c (accepting the performance hit as a trade off for less likelihood of lack of transparency). A delta of 6.3kbps (-4c compared to -4) is not a large increase in bitrate for increased confidence......

lossyWAV Development

Reply #860 – 2008-02-27 08:01:52

Hi Nick,
pretty high bitrate in the table - I'm used to ~400 kbps on average with -3. Has anything changed for -3?

lossyWAV Development

Reply #861 – 2008-02-27 08:04:52

Quote from: halb27 on 2008-02-27 08:01:52

Hi Nick,
pretty high bitrate in the table - I'm used to ~400 kbps on average with -3. Has anything changed for -3?

My bad, I should have prefaced the table with "following results from my 53 sample set".... Nothing has changed, except I've "improved" the maximum_bits_to_remove process to take into account the actual RMS value of the codec_block.

[edit] I've found an oversight in the codec_block RMS calculation (and amended it), -3 now 42,976,155 bytes, 462.3kbps, -4 now 38,821,415 bytes, 417.6kbps. [edit2] Table above corrected.[/edit2]
The -merge parameter will now add a .lossy.wav and corresponding .lwcdf.wav file to re-create the lossless original.

lossyWAV beta v0.7.7 attached to the first post in this thread.
[/edit]

lossyWAV Development

Reply #862 – 2008-02-27 10:45:13

Quote from: Nick.C on 2008-02-27 07:57:30

Code: [Select]
-4c  39,396,622 bytes  423.8kbps (-5)
-4b  39,238,991 bytes  422.1kbps (-5)
-4a  39,016,370 bytes  419.7kbps (-5)
-4   38,821,415 bytes  417.6kbps (-5)
I am tempted to make -4 equivalent to -4c (accepting the performance hit as a trade off for less likelihood of lack of transparency). A delta of 6.3kbps (-4c compared to -4) is not a large increase in bitrate for increased confidence......

Quote from: Nick.C on 2008-02-27 08:04:52

The -merge parameter will now add a .lossy.wav and corresponding .lwcdf.wav file to re-create the lossless original.

Thanks Nick,
it's appreciated that you're still tying the "loose" ends while the fun wore off a bit , that rebuild with correction file kept you busy for a while.

As for -4 becomes -4c ... I don't see the point.
At first the goal was to make -2 transparent and -3 is great for just listening,
next -3 had to be transparent and -4 is (re)created for slightly lower bit rates.
Now you want -4 transparent too. Where does it end?

The nice thing about your results is that the settings scale so nicely. This give users a chance to pick a sweetspot (size/chances for audible noise) according to their needs.
Don't be too anxious about the lowest settings being not totally transparent all of the time, it's supposed to be lossy after all. lossyWav with such settings might even be atractive too another group of users that need <400k bit rates and find the (possible) artifacts introduced with these settings to be preferred over those that normal lossy codec might give.

The main reason for someone not adding the a-c variants would be speed, so worse bit rate together with worse speed (-4 -> -4c) doesn't seem right for a default.

lossyWAV Development

Reply #863 – 2008-02-27 11:05:28

Quote from: GeSomeone on 2008-02-27 10:45:13

Don't be too anxious about the lowest settings being not totally transparent all of the time, it's supposed to be lossy after all.

I agree. While I appreciate all the tuning work by Nick.C and halb27, I think that the lowest setting is being held to too high a standard. I only casually listen to music, so there's no point exhaustively ABX'ing a preset which is not supposed to be perfect anyway. I would also like for lossyWAV to have a -4 preset, which would, of course, entail more of a risk.

lossyWAV Development

Reply #864 – 2008-02-27 11:20:08

Quote from: GeSomeone on 2008-02-27 10:45:13

Now you want -4 transparent too. Where does it end?

The nice thing about your results is that the settings scale so nicely. This give users a chance to pick a sweetspot (size/chances for audible noise) according to their needs.
Don't be too anxious about the lowest settings being not totally transparent all of the time, it's supposed to be lossy after all. lossyWav with such settings might even be atractive too another group of users that need <400k bit rates and find the (possible) artifacts introduced with these settings to be preferred over those that normal lossy codec might give.

The main reason for someone not adding the a-c variants would be speed, so worse bit rate together with worse speed (-4 -> -4c) doesn't seem right for a default.

I hear you! I will leave -4 as is and allow the more paranoid user ( ) to use the extra FFT's.

Quote from: Mitch 1 2 on 2008-02-27 11:05:28

Quote from: GeSomeone on 2008-02-27 10:45:13
Don't be too anxious about the lowest settings being not totally transparent all of the time, it's supposed to be lossy after all.
I agree. While I appreciate all the tuning work by Nick.C and halb27, I think that the lowest setting is being held to too high a standard. I only casually listen to music, so there's no point exhaustively ABX'ing a preset which is not supposed to be perfect anyway. I would also like for lossyWAV to have a -4 preset, which would, of course, entail more of a risk.

Do you mean keep the existing -4 or go even further? You can still use -nts to increase the bits to remove, however this makes the -snr limiter kick in more often, so you would have to change both at once.

[edit] A quick check shows that -4 -nts 12 -snr 15 yields 33,168,675 bytes, 356.8kbps for the same sample set. [/edit]

lossyWAV Development

Reply #865 – 2008-02-27 12:01:52

Although its not a major issue, I am still slightly bothered by the positive abx of Alex B at > 400 k bitrate vbr. i know its a minute difference and hard to define, but you can't just grab ordinary music (Springsteen drums) and abx at 400 k unless it was a fluke sample (even so its very very tough chance).. The fact that it was the solo intro section means that again the is not enough masking of HF stuff despite such high bitrate. I think that there might not be advantages over Bryants new Wavpack --dns which can outperform lossywav in its current state using much lower bitrate - maybe even under 300k using --dns on some samples like Alex B's. The other question is what chances would Alex B have at pulling of random abx using vorbis, aac, mpc etc @ 320k let alone 400 k ??

I know its like comparing apples to oranges and not everyone using the format is interested in 500k or total transparency, but my head just says @ 400 k I don't want to see people pulling off abx tricks.

I am thinking maybe only -1 and -2 should have been available as fully transparent. But I would like much more options - 256 k would be plenty for some people. -3 has maybe too high expectations ? personaly wavpack --dns at 270 k lossy + correction files looks attractive to me.

I think the scale should be flexible and direct:

-1 - For mutli-transcoding ++ overkill
-2 - Transparent suitable for archiving (Default)
-3 - High quality .Normaly undistinguishable from original.
-4 - medium
-5 - portable

Or a starters guide for lossywav settings:

+Highest quality: Archiving / editing (-1 .. -2)
+High quality / Hifi (-3 .. -4)
+Medium (-5 .. -6)
+Portable / outdoor (-7 ...-8....)

lossyWAV Development

Reply #866 – 2008-02-27 13:00:16

Quote from: shadowking on 2008-02-27 12:01:52

The fact that it was the solo intro section means that again the is not enough masking of HF stuff despite such high bitrate.

There is no masking of any frequency - bit reduction will add noise across the whole spectrum.

Quote from: shadowking on 2008-02-27 12:01:52

I think that there might not be advantages over Bryants new Wavpack --dns which can outperform lossywav in its current state using much lower bitrate - maybe even under 300k using --dns on some samples like Alex B's. The other question is what chances would Alex B have at pulling of random abx using vorbis, aac, mpc etc @ 320k let alone 400 k ??

I know its like comparing apples to oranges and not everyone using the format is interested in 500k or total transparency, but my head just says @ 400 k I don't want to see people pulling off abx tricks.

I am thinking maybe only -1 and -2 should have been available as fully transparent. But I would like much more options - 256 k would be plenty for some people. -3 has maybe too high expectations ? personaly wavpack --dns at 270 k lossy + correction files looks attractive to me.

lossyWAV is and always has been pure VBR. The sample set I use for testing purposes will produce a higher bitrate of output than any real music I've found so far. Previous testing at -3 had my sample set at 462kbps and my 10 album test set at 402kbps. I will process my 10 album test set this evening and post the results.

Quote from: shadowking on 2008-02-27 12:01:52

I think the scale should be flexible and direct:

-1 - For mutli-transcoding ++ overkill
-2 - Transparent suitable for archiving (Default)
-3 - High quality .Normaly undistinguishable from original.
-4 - medium
-5 - portable

Or a starters guide for lossywav settings:

+Highest quality: Archiving / editing (-1 .. -2)
+High quality / Hifi (-3 .. -4)
+Medium (-5 .. -6)
+Portable / outdoor (-7 ...-8....)

Using the settings at the end of my last post, I will add a "-5" parameter which might yield about 310 to 320kbps. This will require to be listened to in order to validate it as a meaningful / acceptable preset, as forcing down the bitrate is meaningless unless the quality of the output remains fit for its intended use.

In the interim, I will post beta v0.7.8 which includes the -5 preset. I will also process my 10 album test set at -5 this evening and post the results.

[edit] Thinking about Alex_B's livin_in_the_future_sample, could someone with good ears try to ABX it against v0.7.8 -4? This would let me know if the "active" maximum_bits_to_remove recently introduced has any beneficial effect on this sample. [/edit]

lossyWAV Development

Reply #867 – 2008-02-27 13:35:48

Quote from: shadowking on 2008-02-27 12:01:52

... I am still slightly bothered by the positive abx of Alex B at > 400 k bitrate vbr. i know its a minute difference and hard to define, but you can't just grab ordinary music (Springsteen drums) and abx at 400 k unless it was a fluke sample (even so its very very tough chance).. The fact that it was the solo intro section means that again the is not enough masking of HF stuff despite such high bitrate. I think that there might not be advantages over Bryants new Wavpack --dns which can outperform lossywav in its current state using much lower bitrate - maybe even under 300k using --dns on some samples like Alex B's.

IIRC there had been two changes after Alex B's abxing: one which made the mechanism more sensitive to the HF area, and one which reduced the noise a bit in an overall sense. After that AlexB couldn't abx the problem any more with -3 IIRC.
As for the comparison with wavPack lossy IMO it's true that when targeting at a relatively low bitrate, say 300 kbps or below, wavPack lossy --dns is the more appropriate choice. With lossyWAV + a lossless codec we have the issue that a small codec's blocksize usually is best in an overall sense which however makes the lossless codec a bit inefficient usually. wavPack lossy doesn't suffer from this. That's why I personally woldn't target at a bitrate like 300 kbps with lossyWAV.
lossyWAV's advantage is it's quality reinsuring mechanism which however needs the current quality setting of at least -3. Anyway loosening it a bit like with the current -4 or -5 approach is a good option for those people who don't need transparency but a very high quality while having bitrate in the 350 kbps or even a bit below that area.

Quote from: Nick.C on 2008-02-27 13:00:16

Previous testing at -3 had my sample set at 462kbps and my 10 album test set at 402kbps. I will process my 10 album test set this evening and post the results.

Would it hurt a lot if you skipped your 52 sample set (with a lot of problem samples where a high bitrate is welcome) in favor of a regular music set? It's not necessery to encode 10 complete albums (a lot of work), a hopefully represantative sample set from these albums will do it. IMO it's more important to have the result of regular tracks even when not very representative than the result of problem sample snippets.

lossyWAV Development

Reply #868 – 2008-02-27 14:26:30

Quote from: Nick.C on 2008-02-27 13:00:16

lossyWAV is and always has been pure VBR.

Technically it's FLAC, WavPack, TAK etc. that are VBR. lossyWav is fixed bit rate because wav's have fixed bit rate.
(Of course what lossyWav does is influence the bitrate of the lossless part by making the wav easier to compress.)

lossyWAV Development

Reply #869 – 2008-02-27 15:21:26

No, it's conceptually VBR, but packed into a CBR linear PCM bitstream for output because that's how the world works.

Can I just say - "preset 5" - please, no!

The lossyWAV principle works well, but it goes from "fine" to "poor" to "useless" over a range of 6-12dB (1 to 2 bits).

It's splitting hairs to define 3 presets between "fine" and "useless". Unlike mp3, I don't believe there's that amount of useful room to play with. You very quickly hit something with a bitrate far higher than mp3, and an audio quality far lower.

Still, I guess it's a good thing if people are asking for lower quality!

Cheers,
David.

Quote from: shadowking on 2008-02-27 12:01:52

Although its not a major issue, I am still slightly bothered by the positive abx of Alex B at > 400 k bitrate vbr. i know its a minute difference and hard to define

...and it was at the preset that's not supposed to be transparent. IIRC it wasn't ABXed at the transparent preset, and was subsequently fixed on the non-transparent preset.

I'm not being defensive. I'm very keen for people to find genuine problem samples. This wasn't one IIRC (it's been 38 pages - I'm sorry if I'm thinking of the wrong one!).

Quote

but you can't just grab ordinary music (Springsteen drums) and abx at 400 k unless it was a fluke sample (even so its very very tough chance).. The fact that it was the solo intro section means that again the is not enough masking of HF stuff despite such high bitrate. I think that there might not be advantages over Bryants new Wavpack --dns which can outperform lossywav in its current state using much lower bitrate - maybe even under 300k using --dns on some samples like Alex B's.

You should see the bitrate of lossyWAV if the noise is allowed to be non-flat!

Cheers,
David.

lossyWAV Development

Reply #870 – 2008-02-27 16:05:22

Quote from: 2Bdecided on 2008-02-27 15:21:26

...
Can I just say - "preset 5" - please, no!

The lossyWAV principle works well, but it goes from "fine" to "poor" to "useless" over a range of 6-12dB (1 to 2 bits). ...

From the lossyWAV principle: yes, but with the added skew and snr mechanism there is a certain room for this IMO.

I once tried -3 with -nts 10 and higher, and to me quality was still good with -nts 10. That was before the HF sensitivity increase due to AlexB's sample, and I arrived at a bitrate ~330 kbps on average. I think something like this can make sense for -5. -4 can be -nts 3 or similar (more attractive to me than -5, but I'll stick with -3).

lossyWAV Development

Reply #871 – 2008-02-27 16:11:02

okay guys, thanks for the explanations. I just don't know where lossywav quality drops sharply (wavpack falls over below 235 k ).. so go for whatever you think is the max point for non-offensive losswav listening .

lossyWAV Development

Reply #872 – 2008-02-27 17:11:58

I'll do some processing of my 10 album test set tonight and post the results. For my 53 sample set and a slightly modified set of quality presets (-4=-3.5; -5=-4; -6=-4.5; -7=-5, but all slightly changed) which may feature in beta v0.7.9:

Code: [Select]

Preset  [Equiv. Settings]    Total Size      Bitrate  [Delta.BR]     10 Album Test Set
==========================================================================================
  FLAC  [---------------] 72,652,785 bytes, 781.6kbps [--------] 3.35GB, 854kbps (-------)
   -1   [-nts -4 -snr 25] 52,138,258 bytes, 560.9kbps [--------] 1.94GB, 496kbps (-65kbps)
   -2   [-nts -2 -snr 23] 48,177,581 bytes, 518.3kbps [42.6kbps] 1.78GB, 453kbps (-65kbps)
   -3   [-nts  0 -snr 21] 42,976,155 bytes, 462.3kbps [56.0kbps] 1.58GB, 403kbps (-59kbps)
   -4   [-nts  3 -snr 20] 40,324,698 bytes, 433.8kbps [28.5kbps] 1.47GB, 375kbps (-59kbps)
   -5   [-nts  6 -snr 19] 37,934,855 bytes, 408.1kbps [25.7kbps] 1.38GB, 352kbps (-56kbps)
   -6   [-nts  9 -snr 18] 35,826,396 bytes, 385.4kbps [22.7kbps] 1.31GB, 333kbps (-52kbps)
   -7   [-nts 12 -snr 17] 33,950,736 bytes, 365.2kbps [20.2kbps] 1.25GB, 318kbps (-47kbps)

lossyWAV Development

Reply #873 – 2008-02-27 19:27:22

It was your target so far to have the -snr value constant. So I quickly checked my productive collection I reincoded recently using -3, and thanks to the lossy.flac embedded meta-information -snr value is -snr 21, so you kept this value for -3. Fine.
The fact that you increase the -snr value for -2 and -1 is in congruence with the increasing defensiveness of these settings, but as -snr affects mainly the quality of the lower frequency range which is already covered particularly by the values we had so far, I personally would prefer a higher -nts value when it is about sacrificing a little bit of bitrate. No big thing to me however.

As for the lower bitrate settings: not my world, just a suggestion:
in case it turs out that too much quality is sacrifcied an alternative is not to lower -snr that much but instead use a larger spreading length for the highest and - to a minor degree - the second highest frequency zone.
When it's about sacrificing quality I think it's perceptually the least offensive to do it in the very high frequency range. With -nts 12 -snr 17 I'm afraid chance isn't very low to get a modest quality in the frequency range of the fundamentals where it will be more disturbing.
Just a suggestion in case this should happen.

lossyWAV Development

Reply #874 – 2008-02-27 20:26:44

Quote from: halb27 on 2008-02-27 19:27:22

It was your target so far to have the -snr value constant. So I quickly checked my productive collection I reincoded recently using -3, and thanks to the lossy.flac embedded meta-information -snr value is -snr 21, so you kept this value for -3. Fine.
The fact that you increase the -snr value for -2 and -1 is in congruence with the increasing defensiveness of these settings, but as -snr affects mainly the quality of the lower frequency range which is already covered particularly by the values we had so far, I personally would prefer a higher -nts value when it is about sacrificing a little bit of bitrate. No big thing to me however.

As for the lower bitrate settings: not my world, just a suggestion:
in case it turs out that too much quality is sacrifcied an alternative is not to lower -snr that much but instead use a larger spreading length for the highest and - to a minor degree - the second highest frequency zone.
When it's about sacrificing quality I think it's perceptually the least offensive to do it in the very high frequency range. With -nts 12 -snr 17 I'm afraid chance isn't very low to get a modest quality in the frequency range of the fundamentals where it will be more disturbing.
Just a suggestion in case this should happen.

I'll see what effect keeping the -snr constant has on the lower quality presets. Table above amended to include results of (ongoing) 10 Album Test Set processing.

Notice