Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: lossyWAV Development (Read 573683 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

lossyWAV Development

Reply #800 –
I wasn't aware that there is already a minimum_bits_to_keep of 5 out of 16.
Based on this I second your suggestion and prefer a minimum_bits_to_keep of 5 out of number of bits used in the block.
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #801 –
Quote from: halb27 on
I wasn't aware that there is already a minimum_bits_to_keep of 5 out of 16.
Based on this I second your suggestion and prefer a minimum_bits_to_keep of 5 out of number of bits used in the block.
Thanks, I will keep this then - however, it may be that we would want to "tweak" the value for different quality presets, in the same way as has been done for allowable clips.

On the allowable clips front, if you were to process livin_in_the_future at -3 and ABX it, looking for clipping artefacts, I would be very grateful, as it did suffer a bit from lost bits due to clipping reduction with v0.6.7 RC2 and doesn't with beta v0.6.8.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)
 

lossyWAV Development

Reply #802 –
Quote from: 2Bdecided on
Are you using minimum_bits_to_keep in a defensive way already? Sorry, I'm not keeping up. If it's key to maintaining quality as it is, then maybe you should add what you propose. If it's not, then extending downwards to help quieter blocks doesn't seem necesary. If it is necesary, it would be better to keep the nosie floor at least x dB below the peaks in the spectral domain, rather than in the time domain - which is what I was trying to get at in a post on the last page.

I agree.

Quote from: Nick.C on
On the minimum_bits_to_keep front, at present maximum_bits_to_remove=bits_per_sample-minimum_bits_to_keep = 16-5 = 11 for *all* codec_blocks. With the new proposal, if the highest filled bit (taking the ABS of -ve numbers first) is the 8th then at most 3 bits would be removed, regardless of what the algorithm produced.

I do have faith in the method, I just like belt, braces and hands in pockets keeping trousers up......

I don't...

If 2Bdecided's approach is working right, you will gain nothing except probably worse compression.

Long time ago i was using old fashioned logarithmic quantization to compress audio: High resolution for low levels, little resolution for high levels. That's very similar to your proposal.

Unfortunately this often doesn't work well. Think about a combination of a low frequency signal with high amplitude (bass guitar) and a higher frequncy signal with low volume. Your approach will calculate the bits_per_sample from the low frequency signal and introduce distortion for the high frequency signal.

It's also likely to fail with a pure low frequency signal of high amplitude. Here i always got annoying distortions with the logarithmic approach.

I don't think it would be a good idea to sacrifice compression for a very questionable improvement of the sound quality.

Sorry: Bad explaination because of my very limited english...

lossyWAV Development

Reply #803 –
Quote from: Nick.C on
Quote from: 2Bdecided on
Are you using minimum_bits_to_keep in a defensive way already? Sorry, I'm not keeping up. If it's key to maintaining quality as it is, then maybe you should add what you propose. If it's not, then extending downwards to help quieter blocks doesn't seem necesary. If it is necesary, it would be better to keep the nosie floor at least x dB below the peaks in the spectral domain, rather than in the time domain - which is what I was trying to get at in a post on the last page.

I'm not sure what you're setting at with the clipping. If you let one sample clip in a block, then there are no wasted bits in that block, surely? The sample is 1111111111111111 so no zeros, so wasted_bits=0. Not sure how other codecs handle it - I remember Bryant saying wavpack was different.

Cheers,
David.
On the clipping front, if bits_to_remove=6 then what would have been 10000000 00000000 (assuming there was no sign bit - that bit is done with floats) would be clipped to 01111111 11000000, i.e. as if it had been rounded down not up.

On the minimum_bits_to_keep front, at present maximum_bits_to_remove=bits_per_sample-minimum_bits_to_keep = 16-5 = 11 for *all* codec_blocks. With the new proposal, if the highest filled bit (taking the ABS of -ve numbers first) is the 8th then at most 3 bits would be removed, regardless of what the algorithm produced.

I do have faith in the method, I just like belt, braces and hands in pockets keeping trousers up......


Would that introduce a varying noise floor for same volume samples? If so, it may be detrimental to perceived audio quality.

lossyWAV Development

Reply #804 –
TBeck,

What Nick is proposing won't make anything sound worse, because it's just extending a safety net to lower amplitudes. It's never used to throw away more bits than the "find the noise floor and quantise below it" approach - only to keep at least 5 bits when fewer than 5 bits were going to be kept.

Nick: does this kick in very often?


Where I do agree with you TBeck is that it's not a great safety catch, for exactly the reasons you've explained. It needs to be done in a spectral domain, not the time domain.

Whether it's worth doing in either domain is open to question. It might be, but it's extra complexity. It's heading even further down the route of having a "psychoacoustic" model. There will come a point when it's better to "borrow" someone else's.

For now, I'm more inclined to be happy with what we have.

Cheers,
David.

lossyWAV Development

Reply #805 –
Quote from: TBeck on
.. Unfortunately this often doesn't work well. Think about a combination of a low frequency signal with high amplitude (bass guitar) and a higher frequncy signal with low volume. ...

Yes, but it's just an additional safety action - not really necessary IMO, but as it's done already Nick.C's new approach is just more consequent than the old approach.
It's low level signals that benefit from the new approach with respect to this safety action.

EDIT: 2Bdecided was faster.
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #806 –
Quote from: 2Bdecided on
What Nick is proposing won't make anything sound worse, because it's just extending a safety net to lower amplitudes. It's never used to throw away more bits than the "find the noise floor and quantise below it" approach - only to keep at least 5 bits when fewer than 5 bits were going to be kept.


Quote from: halb27 on
Yes, but it's just an additional safety action - not really necessary IMO, but as it's done already Nick.C's new approach is just more consequent than the old approach.
It's low level signals that benefit from the new approach with respect to this safety action.

I was aware of this but failed to express it right. 

lossyWAV Development

Reply #807 –
So, reading the last few posts:

I will remove the revised minimum_bits_to_keep method (no, it doesn't kick in very often at all, and it slows down the processing slightly).

Seeking concensus:

Should we retain the recent implementation of allowing a few clipped samples to be "rounded the other way"?
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #808 –
I don't care much about it with a slight negative feeling towards letting 5 samples clip though I don't think that would be audible.
I feel positive about letting isolated samples clip, but that's only because of AlexB's provided track where it happens that bits removed changed abruptly due to only 1 clipped sample.

For differentiating -3 from -2 my favorite is: let 1 sample per block clip for -2, let 2 samples clip for -3.
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #809 –
Quote from: halb27 on
I don't care much about it with a slight negative feeling towards letting 5 samples clip though I don't think that would be audible.
I feel positive about letting isolated samples clip, but that's only because of AlexB's provided track where it happens that bits removed changed abruptly due to only 1 clipped sample.

For differentiating -3 from -2 my favorite is: let 1 sample per block clip for -2, let 2 samples clip for -3.
I was thinking more: -1 = 0; -2 = 1; -3 = 5;

Code has speeded up yet again - now approaching 50% faster than v0.6.4 RC1....
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #810 –
Quote from: Nick.C on
On the clipping front, if bits_to_remove=6 then what would have been 10000000 00000000 (assuming there was no sign bit - that bit is done with floats) would be clipped to 01111111 11000000, i.e. as if it had been rounded down not up.
If you're normally rounding to the nearest number, then rounding down when you should be rounding up means you're jumping further away from the wanted value on this sample than on any other, doesn't it? I.e. you're adding more noise - potentially 50% more.

As it's level dependent, it's not strictly noise - it's distortion.

With my apologies in advance if I've misunderstood!

Cheers,
David.

lossyWAV Development

Reply #811 –
Quote from: 2Bdecided on
If you're normally rounding to the nearest number, then rounding down when you should be rounding up means you're jumping further away from the wanted value on this sample than on any other, doesn't it? I.e. you're adding more noise - potentially 50% more.

As it's level dependent, it's not strictly noise - it's distortion.

With my apologies in advance if I've misunderstood!

Cheers,
David.
No, exactly right - but given the duration - does the benefit not outweight the potential cost?
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #812 –
It would be good to see the resulting difference in bitrate due to allowing this kind of restricted clipping for entire tracks. I think the difference is small. I will try with the only seriously clipping album in my collection.
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #813 –
Quote from: halb27 on
It would be good to see the resulting difference in bitrate due to allowing this kind of restricted clipping for entire tracks. I think the difference is small. I will try with the only seriously clipping album in my collection.
Using livin_in_the_future, there is a 3.5% reduction at -2 (1 clip per channel per codec_block allowed) and 5% reduction at -3 (5 clips...)

lossyWAV beta v0.6.9 attached to post #1 in this thread.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #814 –
I guess that's the result for the 30 sec part - but maybe it's representative for the entire track.
3.5% resp. 5% isn't bad, but it also shows for this case that we get most of the effect with less then 5 samples allowed to clip which is a more cautious approach. Do you mind trying 2 allowed clipped samples per block?
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #815 –
Quote from: halb27 on
I guess that's the result for the 30 sec part - but maybe it's representative for the entire track.
3.5% resp. 5% isn't bad, but it also shows for this case that we get most of the effect with less then 5 samples allowed to clip which is a more cautious approach. Do you mind trying 2 allowed clipped samples per block?
Tried -2 and -3 at 2 allowable clips.
-2 : the FLAC file decreases from 1715683 to 1712152 bytes (no clips 1777874) : -3.50% to -3.70%;
-3 : the FLAC file increases from 1523876 to 1524359 bytes (no clips 1603204) : -4.95% to -4.92%

I think that -1 = 0; -2 = 1; -3 = 2 may be optimal.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #816 –
Looks good. Looking at the -2 result there's only a negligible difference between allowing 1 or 2 samples to clip. So allowing for just 1 sample to clip may be preferable also with -3.

But that's only for AlexB's sample.
I just encoded 7 full length tracks (those in my selective collection) from Francoise Hardy's Album 'Le temps des souvenirs' which I know has a lot of clipping. 0.6.9 (5 allowed clipping samples) provides for a decrease in total filesize of -19,4% (against using 0.6.7RC2).
So for clipped tracks your suggestion yields a significant improvement.
I'd like to try these tracks with less samples allowed clipping. Can you provide such an experimental version, please?
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #817 –
Will do - should be up in an hour or so......
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #818 –
Meanwhile I can report about my listening test with 0.6.9 on AlexB's sample and 2 of my 7 tracks. It's alright to me - I could not abx any of those spots I had a suspicion that there's a slightly audible issue. What I considered 'wrong' in the encoding was also 'wrong' in the orginal -  Francoise Hardy's album has a pretty bad quality.

So from this even 5 allowed clipped samples per block are ok to use. Anyway allowing for only a lower amount of clipped samples provides for a higher degree of safety and might be the better choice in case file size remains similar.
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #819 –
lossyWAV beta v0.7.0 attached to post #1 in this thread.

Very pleased with the speed now - beta v0.7.0 processes 125MB of WAV in 14 seconds (average 53.1x) on an Intel C2D E6600 @ 3.0GHz, 2 x 80GB HDD in RAID0, Windows XP SP2
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #820 –
My 0.7.0 result for AlexB's track:

-clips 3: -5,0%
-clips 2: -5,0% (file size with my FLAC setting: 1523674)
-clips 1: -4,7%

My 0.7.0 result for my 7 Francoise Hardy tracks:

-clips 3: -19,2%
-clips 2: -18,4%
-clips 1: -14,2%


So from this the essential reduction in filesize is achieved already with just 1 allowed sample per block to clip.
2 allowed samples to clip is attractive to some extent, but to a minor degree.
3 allowed samples to clip brings only an insignificant advantage and is not attractive.
More than 3 allowed samples to clip is useless in a practical sense.

So I think we have 4 useful choices:

a) 1 allowed clipped sample per block for -2, 2 allowed clipped samples per block for -3.
b) 1 allowed clipped sample per block for -2 and -3.
c) full clipping prevention with -2, 1 allowed clipped sample per block for -3.
d) full clipping prevention with -2, 2 allowed clipped samples per block for -3.

I personally don't care much about whether we should allow for 1 or 2 clipped samples with -3. I think both choices are fully in congruence with what we want to achieve with -3.
I'm more worried about whether or not we should allow for clipped samples with -2. I feel a bit uncomfortable due to the nature of -2 and the distortion 2Bdecided mentioned when allowing clipping to occur though I don't think that this can be audible.
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #821 –
Quote from: halb27 on
So I think we have 4 useful choices:

a) 1 allowed clipped sample per block for -2, 2 allowed clipped samples per block for -3.
b) 1 allowed clipped sample per block for -2 and -3.
c) full clipping prevention with -2, 1 allowed clipped sample per block for -3.
d) full clipping prevention with -2, 2 allowed clipped samples per block for -3.

I personally don't care much about whether we should allow for 1 or 2 clipped samples with -3. I think both choices are fully in congruence with what we want to achieve with -3.
I'm more worried about whether or not we should allow for clipped samples with -2. I feel a bit uncomfortable due to the nature of -2 and the distortion 2Bdecided mentioned when allowing clipping to occur though I don't think that this can be audible.
Is *one* "distorted" sample (22.68 microseconds) going to be of any real significance? Personally, unless over-ruled by someone with more expert knowledge, is -1=0; -2=1; -3=2, i.e. as per beta v0.7.0.

I'm really pleased about the reduction on your badly clipping tracks : -18.4% is excellent!

This modification may bring down the average bitrate for -3 a bit to bring is a bit closer to that for v0.6.4 RC1.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #822 –
Quote from: Nick.C on
Is *one* "distorted" sample (22.68 microseconds) going to be of any real significance? Personally, unless over-ruled by someone with more expert knowledge, is -1=0; -2=1; -3=2, i.e. as per beta v0.7.0.

So OK then.
Quote from: Nick.C on
This modification may bring down the average bitrate for -3 a bit to bring is a bit closer to that for v0.6.4 RC1.

This is another story and depends heavily on the degree of clipped tracks in the user's collection.

For a short impression I encoded the small regular track set I used so often to find out about the average bitrate for a lossyWAV version:

0.6.7RC2 -3: Total filesize: 141231970
0.7.0 -3 -clips 2: Total filesize: 141227370

Difference: -0,003%
lame3995o -Q1.7 --lowpass 17

lossyWAV Development

Reply #823 –
Quote from: halb27 on
For a short impression I encoded the small regular track set I used so often to find out about the average bitrate for a lossyWAV version:

0.6.7RC2 -3: Total filesize: 141231970
0.7.0 -3 -clips 2: Total filesize: 141227370

Difference: -0,003%
Hehehe..... there is a small penalty for including the -clips parameter : " -clips n" is added to the parameter string in the "fact" chunk in the wav file.....

I just transcoded my Mike Oldfield collection (261 tracks, 24h30m12s) in 40m48s : an average throughput (FLAC [from NAS] > WAV [local]> lossyWAV > FLAC) of 36x - and lo and behold, there was *no* difference at all in the total filesize (from v0.6.7 RC2 and beta v0.7.0) that couldn't be explained by the extra 9 bytes per file!
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV Development

Reply #824 –
When it was up to introducing our current clipping prevention scheme I searched hard for clipped tracks with the result that my collection has clipping next to nothing.

IMO we should stick to your suggestion for -3. After all clipping exists. But I think my clipping album is an extreme case of clipping, and AlexB's sample is more representative for clipped tracks.
Because of this and the fact that clipping is very rare I suggest to allow only 1 clipped sample in a block for -3, and keep the clipping prevention scheme in full action with -2.
lame3995o -Q1.7 --lowpass 17