HydrogenAudio

Lossy Audio Compression => Other Lossy Codecs => Topic started by: Nick.C on 2008-08-25 12:16:05

Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-08-25 12:16:05
Following the release of lossyWAV 1.1.0b, I feel it is (again) time to kick off development of the next minor release.

Items currently on the list for inclusion in 1.x.0:

[blockquote]1.32.0: Implementation of SG's new noise shaping method;
1.2.0: Checking of S (=L-R) channel for matrix surround content;
1.2.0: Revisit the spreading function;[/blockquote]If you have any ideas, suggestions, code optimisations, etc, please post them here.

Link to the hydrogenaudio wiki article (http://wiki.hydrogenaudio.org/index.php?title=LossyWAV)

lossyFLAC resultant bitrates:
Code: [Select]
10 Album Test Set
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|Version| Settings | FLAC -5  |--insane  |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.0.0b| default  | 854kbit/s| 626kbit/s| 539kbit/s| 452kbit/s| 365kbit/s| 295kbit/s| -------- | -------- |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.1.0c| default  | 854kbit/s| 632kbit/s| 548kbit/s| 463kbit/s| 376kbit/s| 285kbit/s| -------- | -------- |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.2.0 | default  | 854kbit/s| 627kbit/s| 544kbit/s| 460kbit/s| 376kbit/s| 288kbit/s| -------- | -------- |
|v1.2.0 |    -t    | 854kbit/s| 582kbit/s| 514kbit/s| 450kbit/s| 385kbit/s| 341kbit/s| 310kbit/s| 283kbit/s|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+

55 Problem Sample Set
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|Version| Settings | FLAC -5  |--insane  |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.0.0b| default  | 780kbit/s| 655kbit/s| 582kbit/s| 503kbit/s| 417kbit/s| 330kbit/s| -------- | -------- |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.1.0c| default  | 780kbit/s| 654kbit/s| 583kbit/s| 508kbit/s| 425kbit/s| 321kbit/s| -------- | -------- |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.2.0 | default  | 780kbit/s| 654kbit/s| 585kbit/s| 510kbit/s| 427kbit/s| 325kbit/s| -------- | -------- |
|v1.2.0 |    -t    | 780kbit/s| 623kbit/s| 565kbit/s| 506kbit/s| 441kbit/s| 391kbit/s| 354kbit/s| 322kbit/s|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+

Suggested foobar2000 converter setup:

lossyFLAC:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.flac
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - -b 512 -5 -f -o%d
Format is: lossless or hybrid
Highest BPS mode supported: 24
lossyTAK:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24
lossyWV:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.wv
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24
There is a known problem within foobar2000 (although more likely to do with cmd.exe itself) when running an executable within the cmd.exe command line from a path which includes spaces. The suggested fix for this is to enclose the element of the path which contains spaces within double quotation marks ("), e.g. c:\"program files"\directory_where_executable_is\executable_name

Change log 1.2.0: 16/12/09
Code optimisation;
Removal of negative -q values in default mode. Quality range for --altpreset remains -4 to 10 (--quality -4 --altpreset == --quality 0 --limit 15159).

[!--sizeo:1--][span style=\"font-size:8pt;line-height:100%\"][!--/sizeo--]Change log 1.1.5c: 21/11/09
Minor revision to internal setting for --altpreset.

Change log 1.1.5b: 20/11/09
Major revision to internal setting for --altpreset.

Change log 1.1.5a: 18/11/09
Bugfix: Correction to high sample-rate processing.

Change log 1.1.4s: 07/11/09
Bugfix: manual --limit setting not working as it should.

Change log 1.1.4r: 03/11/09
Bugfix: shaping in altpreset mode was artificially limited to 50% (only affected -q 6.5 and above).

Change log 1.1.4q: 02/11/09
Reversion to use of previous noise pre-calculated constant;
Shaping now OFF by default. To enable shaping use -s or --shaping, without a parameter for automatic shaping or with a value 0<=n<=1 for user specified shaping.

Change log 1.1.4p: 22/10/09
Mutual exclusivity of shaping, hilimit and altpreset removed;
Added noise pre-calculated constant removed in favour of improved derived formula;
--altpreset parameter now also -t.

Change log 1.1.4n: 27/09/09
Mutual exclusivity of shaping, hilimit and altpreset corrected.

Change log 1.1.4m: 26/09/09
--postanalyse function removed;
--limit changed to --hilimit and --lolimit;
--altpreset parameter introduced which changes default behaviour for shaping and hilimit.
[shaping = 0.5*(max(0,q/10)+max(0,q/10)^2.584962)) -q 0 = 0; -q 5 = 0.3333; -q 10 = 1]
[hilimit = round(14000 + 2000 * max(0,q/10)) / samplerate * 64) * (64/samplerate)]

Change log 1.1.4k: 24/08/09
--postanalyse function modified to use existing spreading function.

Change log 1.1.4j: 23/08/09
--limit lower range changed to 10000Hz.

Change log 1.1.4h: 22/08/09
--limit lower range changed to 14500Hz.

Change log 1.1.4g: 20/08/09
--maxsnr removed. -p or --postanalyse parameter implemented. Using this parameter checks the noise level of the correction data and compares to the low value derived from the associated source audio. If the correction noise (i.e. that of the difference signal) is greater than the source audio low value then the bits_to_remove value is reduced for the codec-block until the added noise is lower. Code further tidied. -F or --fftw parameter removed as FFTW dll is now automatically used if found (slight speed-up makes this the fastest way to go). Stack error fixed which occurs when libfftw3-3.dll v3.2.2 is used (newly released).

Change log 1.1.4f: 24/07/09
Bug in --maxsnr parameter fixed. Bug in pure Delphi compile fixed.

Change log 1.1.4e: 22/07/09
Major code redevelopment - more units, hopefully clearer. New parameter: -Y, --maxsnr <n> which allows specification of difference between maximum FFT result and added noise. Maxsnr works with both default spreading and --sortspread. Link to FFTW Windows DLL download page (http://www.fftw.org/install/windows.html).

Change log 1.1.4d: 07/06/09
Bug fixed whereby lossyWAV would crash if 'libfftw3-3.dll' could not be initialised. If --fftw parameter is used and the DLL cannot be found then lossyWAV will revert to the existing FFT routines and output a warning. Link to FFTW Windows DLL download page (http://www.fftw.org/install/windows.html).

Change log 1.1.4c: 05/06/09
FFTW can now be optionally used for FFT analyses in lossyWAV. Use of FFTW requires the presence of "libfftw3-3.dll" on the host computer, somewhere on the path and the addition of -F or --fftw to the lossyWAV command line. FFT (Delphi and assembler) further optimised. General code tidy-up. Link to FFTW Windows DLL download page (http://www.fftw.org/install/windows.html).

Change log 1.1.4b: 14/05/09
FFT (Delphi and assembler) further optimised. Radix-4 FFT implemented in assembler and Delphi and Radix-8 in Delphi. Significant speedup of Delphi FFT throughput.
General code tidy-up.

Change log 1.1.4a: 05/05/09
--sorspread parameter no longer takes an additional parameter, now on/off;
spreading function changed slightly - now properly computes old and new averages separately;
FFT Real routine corrected as was giving wrong signs of some complex output values (did not affect magnitude of results);

Change log 1.1.3k: 30/04/09
Fault-finding release #1 to attempt to determine cause of WINE incompatibility. (Successful!! )

Change log 1.1.3j: 15/04/09
--sortspread parameter modified (again), now takes a parameter between 0 and 7, 2 is equivalent to beta 1.1.3i.
--centre parameter removed.
Reference_threshold tables removed in favour of direct calculation of the level of added noise due to bitdepth reduction using derived formula.

Change log 1.1.3i: 07/04/09
New --sortspread parameter modified (again). Bitrate matched with default spreading for my 55 problem sample set. Will revise table for my 10 Album Test Set.

Change log 1.1.3h: 05/04/09
New --sortspread parameter modified.
Removed - bug found.

Change log 1.1.3g: 02/04/09
New --sortspread parameter introduced for testing purposes.

Change log 1.1.3f: 31/03/09
New --centre and --underlap <n> parameters introduced for testing purposes; Revised source.

Change log 1.1.3e: 18/03/09
Removal of old and new spreading functions in favour of variant; Code tidy up - speed improvements for pure delphi compile; Revised source.

Change log 1.1.3d: 05/03/09
Bug fix (would crash with a range error sometimes); Speedup of --varspread code. Revised source.

Change log 1.1.3c: 24/02/09
Introduction of -V or --varspread parameter to enable variant spreading function - a hybrid between the old and the new. Revised source.

Change log 1.1.3b: 23/02/09
Bug-fix: high sample rates with 1.1.3 would cause a range-check error or random results. Revised source.

Change log 1.1.3: 22/02/09
Integration of data structures used in new and old spreading functions. Source release.

Change log 1.1.2j: 18/02/09
Implementation of -O or --oldspread parameter to enable the use of the spreading function used in v1.1.0b instead of the revised version currently under development. This gives very slightly different results to v1.1.0b as is to be expected due to the revision of the reference-threshold constants at beta v1.1.1d.

Change log 1.1.2i: 12/02/09
Addition of a -N or --nasty (-q -2.0) and -A or --awful (-q -4.0) to allow extremely low quality levels to be explored.

Change log 1.1.2h: 12/02/09
Addition of a -N or --nasty (-q -2.0) to allow extremely low quality levels to be explored.
Removed: Bug to be fixed.

Change log 1.1.2g: 10/02/09
Addition of a -r or --randombits parameter to randomise the zeroed lsbs.

Change log 1.1.2f: 09/02/09
Further modification to the spreading_function.

Change log 1.1.2e: 06/02/09
Further modification to the noise shaping process - first attempt to attenuate noise-shaping where bits_to_remove is zero for a particular codec block.

Change log 1.1.2d: 05/02/09
Further modification to the noise shaping process - audio data now no longer scaled prior to noiseshaping.

Change log 1.1.2c: 04/02/09
Further modification to the noise shaping process - noise shaping performed even when no bits removed.

Change log 1.1.2b: 03/02/09
Repair of the noise shaping process - now continuous for each channel rather than treating each codec-block totally separately;

Change log 1.1.2: 28/01/09
Code optimisations and data optimisations;
Revisions to the spreading function;

Change log 1.1.1e: 30/09/08
Interim beta, with source as reversion to Delphi complete (with conditional define to re-enable all IA-32/x87 code).

Change log 1.1.1d: 10/09/08
Further revision to the simplified spreading function - slightly higher bitrates than 1.1.1c but I'm happier with the method;
Reference-threshold constants re-calculated using more iterations (2^(32-fft-bit-length) iterations, i.e. 512K iterations for 8192 sample FFT and 128M iterations for 32 sample FFT) and for the first time taking into account FFT-result values less than 1. This only really affects bits-to-remove values between 1 and 7, which is in line with my expectation when I made the change to the noise-calculation method;

Change log 1.1.1c: 02/09/08
Further revision to the simplified spreading function;
Dither removed;

Change log 1.1.1b: 26/08/08
Revision to the simplified spreading function. All bin "averages" now calculated taking into account a variable proportion of bins to either side, i.e. "average" = (fft_result+(fft_result[i-1]+fft_result[i+1])*factor)/(1+2*factor), where factor = 0.0 at 20Hz and 1.0 at 16kHz, with linear interpolation for intermediate values.

Change log 1.1.1a: 25/08/08
Fundamental simplification of spreading function methodology put forward for comment. All bin "averages" now calculated taking into account a fixed proportion of bins to either side, i.e. "average" = (fft_result+(fft_result[i-1]+fft_result[i+1])*factor)/(1+2*factor), where factor = 0.26 in this case;
FFT result overall averaging now carried out prior to the spreading function rather than at the same time;
Reference_threshold constants revised slightly.

Change log 1.1.0b: 03/08/08
FFT lengths will now increase for higher bitrate audio, i.e. 88.2/96kHz, 176.4/192kHz and 352.8/384kHz;
improved logfile output and --detail output;
reference threshold constants for rectangular dither and triangular dither have been calculated so added noise should be the same for dither off and any dither level between 0 and 1 - the number of bits-to-remove will however reduce with "increasing" dither.[/size]
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-08-25 13:36:37
Some questions:I ask as these all add to the time taken to process files (even if the options themselves are not selected).

Comments / criticisms / brickbats welcomed as before.

I will acknowledge the usefulness of the correction file as a quick and automatic way of generating the difference signal between the lossless original and processed output (for scaling=1 only).
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2008-08-25 15:32:01
Some questions:
  • Do we need dither?
  • Do we need 32-bit integer processing?
  • Do we need the capability to create correction files?
I ask as these all add to the time taken to process files (even if the options themselves are not selected).

Comments / criticisms / brickbats welcomed as before.

I will acknowledge the usefulness of the correction file as a quick and automatic way of generating the difference signal between the lossless original and processed output (for scaling=1 only).

Nick,
      I have never been sure when/if dither should be used and if so, how much. I've tried with and without and can't hear any difference so from my perspective it isn't needed

Again personally, I can't see a need for 32 bit integer. As a CoolEdit user I would be more interested in 32 bit float but in any event I'd actually be unlikely to use it.

Similarly correction files. I've never used them and don't suppose I ever will as I get transparent (to me) output from lossyWav.

I actually like lossyWav as it is. It does exactly what I need. I can see that other's needs might be different to mine though.

The only thing I'd like to see is a nice GUI front end - I've mentioned that a couple of times before. I don't mean to go on about it but it would make life a bit easier for me as a non-techie.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2008-08-25 15:43:29
Some questions:
  • Do we need dither?
  • Do we need 32-bit integer processing?
  • Do we need the capability to create correction files?
I ask as these all add to the time taken to process files (even if the options themselves are not selected).

Comments / criticisms / brickbats welcomed as before.

I will acknowledge the usefulness of the correction file as a quick and automatic way of generating the difference signal between the lossless original and processed output (for scaling=1 only).

My opinion on the questions:

a) dithering is not needed
b) I like to see further support for 24 bit depth input files (I replaygain with foobar using 24 bit WAV output files). I cannot imagine anybody needs a bit depth of 32 bit. In case that's what your question is about.
c) I like to be able to listen to the error signal. I don't need the correction file for reconstructing the original signal.

Great that you're still struggling so much to improve lossyWAV.
I'm just a bit sceptical about the new spreading approach. It changes the machinery in a quite significant way at the low frequency end, and I think we can be very content with the current machinery. Changing the machinery would mean we throw away the experience we have so far with lossyWAV's quality (and though the experience situation isn't optimal we do have some experience), and start experiencing quality again from the zero point - more or less). Doesn't spreading just mean averaging over a certain number of bins? With this in mind I wouldn't care whether or not the virtual center of the bins involved is identical with one of the real bins. At least this is how I understand the new spreading idea. Sure there are numerous ways of doing the averaging, but are there expectations for a real benefit when going the new way?
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2008-08-25 19:14:03
 I was waiting for this thread

I wish I could answer your questions but:
1: I don't know what lossywav can gain from dithering (the same as mp3 ? supposed more "natural" background noise ? I always thought dithering was made to soften frequency destruction effect so I don't get the use with lossywav) 
2: I don't get what you meant but my CPU is 32bits & my audio is 16/24Bits 
3: I already said I didn't use correction files in the other thread

I disagree with halb27 on the spreading function, if you must refrain experimentation because you fear to break the machine lossywav will never progress, you just need to be sure it worth it before making it a full release.
Also I don't need a gui personnaly, even if it would exist, I would use F2K. I agree it would be better than F2K for noobs allergic to command-line but it should be a lossywav/flac gui or the noob will end with a big wav file asking himself the purpose of such a useless codec  so maybe a fork of speek' flac frontend ... but not a lossywav gui alone ...

Note: I will most likely convert my whole lossless collection to lossywav after the 1.2.0 release, so I hope it will be VERY good
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2008-08-25 21:05:21
.. you just need to be sure it worth it before making it a full release. ...

That's the problem.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-08-25 21:18:43
One possible variant is to use 1 as a spreading value where 1.1.0b did and for all the values which exceed 1 use something else (1<value<2), i.e.
((2,2,2,2,2,2,2,2),(2,2,2,2,2,2,2,2),(2,2,2,2,2,2,2,2),(1,2,2,2,2,2,2,3),(1,1,2,2,2,2,2,3),(1,1,2,2,
2,2,3,4))
goes to
((SC,SC,SC,SC,SC,SC,SC,SC),(SC,SC,SC,SC,SC,SC,SC,SC),(SC,SC,SC,SC,SC,SC,SC,SC),(1,SC,SC,SC,SC,SC,SC,
SC),(1,SC,SC,SC,SC,SC,SC,SC),(1,1,SC,SC,SC,SC,SC,SC))
this should go some way to alleviate any concerns with respect to reducing quality as less averaging = lower minima.
Title: lossyWAV 1.2.0 Development Thread
Post by: Axon on 2008-08-26 00:28:30
I'll second a request for 32-bit float.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2008-08-26 07:37:03
One possible variant is to use 1 as a spreading value where 1.1.0b did and for all the values which exceed 1 use something else (1<value<2), i.e.
((2,2,2,2,2,2,2,2),(2,2,2,2,2,2,2,2),(2,2,2,2,2,2,2,2),(1,2,2,2,2,2,2,3),(1,1,2,2,2,2,2,3),(1,1,2,2,
2,2,3,4))
goes to
((SC,SC,SC,SC,SC,SC,SC,SC),(SC,SC,SC,SC,SC,SC,SC,SC),(SC,SC,SC,SC,SC,SC,SC,SC),(1,SC,SC,SC,SC,SC,SC,
SC),(1,SC,SC,SC,SC,SC,SC,SC),(1,1,SC,SC,SC,SC,SC,SC))
this should go some way to alleviate any concerns with respect to reducing quality as less averaging = lower minima.

If I understand it correctly you want to stay conservative when including more bins in the averaging compared to what we have now by applying a considerably smaller weight to the bins which are off-center to the highest degree (so a weight of >0.5 for the center bin of the 3 bin spreading replacing current 2 bin spreading?). Sounds good though I still can't see the potential advantage and why you aren't content with current spreading.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-08-26 08:26:02
I am re-examining each major processing component in turn - it's the turn of the spreading function....

I've modified the spreading function so that at the bin corresponding to 20Hz the range is 1.0 and at 16kHz it is 2.0, with linear interpolation for intermediate bins.

lossyWAV beta 1.1.1b attached to post #1 in this thread.

[edit]
The concensus (and what David and SebastianG said earlier) seems to be that dither is not required within lossyWAV.

On the processing of 32-bit integer samples, I'll leave it in at the moment, but I don't think that there are many packages that would output them in favour of 32-bit float samples. I don't know if the method would work on 32-bit float samples - I have a feeling it would be difficult to determine how many bits precision to remove from a float value - unless it was a simple "reduce a 32-bit float value (23-bit mantissa) to a 24-bit float value (15-bit mantissa) by brute force...." process.

It seems that some people like the correction file for analysis rather than reversion to lossless - maybe the --merge parameter can go?
[/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2008-08-26 09:43:27
... maybe the --merge parameter can go?

As for my needs: yes.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2008-08-26 12:33:20
The noise floor already "floats" in 32-bit float.

What do FLAC and other lossless encoders do wrt floating point data and "wasted bits"?

Depending on that, the appropriate lossyWAV processing could be tricky but useful, or pointless.

I only ever use 32-bit float files as intermediate files. Sometimes I archive them as-is, so I can re-work the project later. lossyWAV might be useful here, though TBH I haven't even bothered FLACing them because it's so rare that I do this. Other people might do this on a daily basis!

I have no experience of 32-bit integer audio files. 48-bit integer is common in audio processing (DSP IIR filtering etc), but never as an output.


IMO dither can go, if having the option available is slowing down processing even when it's not used.


If "Implementation of SG's new noise shaping method" means dynamic noise shaping, then depending on how aggressively you do this, it might be worth changing from rectangular spreading functions to something else entirely. I'm pointing this out because you might spend a long time playing with the current spreading function, only to dump it soon after. What you have is a narrow (fractional) version of something vaguely related to the ERB (equivalent rectangular bandwidth) scale - I reckon one day you'll end up with something which is a narrow (fractional) version of something vaguely related to overlapping critical band filters.

I can't help feeling that there's no more or less reason to have reconstruction with lossyWAV than with wavpack lossy, apart from the currently inevitable clunkiness of it. However, if the concept is there, it's another "tick" in the format comparison table, and someone can always come along and implement a more graceful re-uniting of the lossy and correction files later, if they feed the need. If you drop this ability entirely, this possibility is removed. Whether you support the merging in lossyWAV itself is up to you - having that available can't slow down encoding though, can it?

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: krmathis on 2008-08-26 18:38:18
Suggestion: Cross-platform code, allowing Mac OS X and GNU/Linux users to take part of the fun...
But that may be too huge of a task?
Title: lossyWAV 1.2.0 Development Thread
Post by: Axon on 2008-08-26 23:42:01
The noise floor already "floats" in 32-bit float. What do FLAC and other lossless encoders do wrt floating point data and "wasted bits"? Depending on that, the appropriate lossyWAV processing could be tricky but useful, or pointless.


FLAC doesn't even support floating point IIRC.

One could go both ways with that, and say that either lossyWAV has no need for floating point support, or that it provides a very nice way to gracefully encode the floating-point data. I'm leaning towards the latter.

The only issue I'd see otherwise is how to handle +0dbFS samples. I have no suggestions on how to handle that except perhaps to optionally right-shift the output by a few bits and scribble the gain down in the tags.

Quote
I only ever use 32-bit float files as intermediate files. Sometimes I archive them as-is, so I can re-work the project later. lossyWAV might be useful here, though TBH I haven't even bothered FLACing them because it's so rare that I do this. Other people might do this on a daily basis!
I would preferrably record my vinyl transcriptions in floating point on principle alone.

Quote
I have no experience of 32-bit integer audio files. 48-bit integer is common in audio processing (DSP IIR filtering etc), but never as an output.
Heh, if 32-bit float is going to be supported, why not 64-bit floating point too? It's a negligible code change.


In principle, the binning process used to establish critical band responses could be circumvented through clever frequency mapping. For instance, doing a frequency shift from 10khz down to say 100hz would mean that quantization noise that originally fit inside one bin could now fit in several. This could kick something into audibility.

Right?

What I'm getting at here is that perhaps more work should be put into tuning lossyWAV so that virtually all DSP effects/manipulations could not possibly cause an audibility difference, rather than merely ensuring that straight listening will not tease out a difference.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-08-27 07:40:12
It would be possible to read 32-bit float values and write 32-bit integer values (having suitably scaled the output) - this would not change the file-size but only some of the fmt chunk information.

I'll look into it....

To fit in the range -2,147,483,648..2,147,483,647 the 32-bit float value would require to be scaled by a factor of 2^-97.

[edit] I've just been reading about the draft IEEE-754r standard and there will be a 16-bit float value in the range +/-1.## x 2^-15 to +/-1.## x 2^14 with a mantissa of 10 bits. This seems to open up the possibility of 11 bit precision in a 2^30 range, or taking what we know about lossyWAV into account effectively stores a 32-bit integer in a 16-bit float (albeit with reduced precision - but reduced precision is not proving to be a problem ).
Title: lossyWAV 1.2.0 Development Thread
Post by: Axon on 2008-08-27 07:49:12
A complete mapping of the floating point domain is unnecessary unless HDR techniques start creeping in from the video realm to the audio realm (which is rather unlikely). All I'd anticipate would be desired would be a bit shift from 0-4 bits if that.

... Not like I have any kind of valid use for that feature, so feel free to ignore it.
Title: lossyWAV 1.2.0 Development Thread
Post by: SebastianG on 2008-08-27 14:06:34
1.2.0: Implementation of SG's new noise shaping method

Yay!
If you like to get a Matlab version of the code I sent you click here (http://www.badongo.com/file/11074415).

1.2.0: Revisit the spreading function

Can you shed some more light on what's currently happening in this regard? What exactly is fft_result[k], why is there an averaging and what happens after the averaging?

To fit in the range -2,147,483,648..2,147,483,647 the 32-bit float value would require to be scaled by a factor of 2^-97.

IIRC digital full scale is usually +/- 1.0 in float formats. So, in case you want to convert it to 24 bit ints, you should scale the floats by 2^23.

Cheers,
SG
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2008-08-27 14:49:01
I fail to see how anyone that goes through the lengths to save audio in floating point format would want to use lossyWav on it 
The first step would be to convert to something like 24bit integer IMO.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-08-27 22:18:22
Can you shed some more light on what's currently happening in this regard? What exactly is fft_result[k], why is there an averaging and what happens after the averaging?
The FFT_result array is created by taking the magnitude of the raw results of the complex fft analysis and multiplying by the corresponding skewing value.

These results have always been averaged over a number of bins to remove zero or very low single bins. The most recent method now only takes into account a proportion of the bins on either side of the target bin rather than bins one or two bins away from the target bin. I feel that this will still remove single low bins but will possibly be better than the former method.

btw, thanks very much for the Matlab method - I can read matlab, not C!

I see what you mean about 32-bit floats. However the easiest way would be to convert to 32-bit integers (in the first instance) - maybe 24-bit integers later.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2008-08-29 14:18:22
I fail to see how anyone that goes through the lengths to save audio in floating point format would want to use lossyWav on it 
The first step would be to convert to something like 24bit integer IMO.
Not at all - often when I use 32-bit floats (in CEP), it's so I don't have to worry about clipping. It's not always about the quality at all - sometimes it's simple convenience.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-02 12:53:53
lossyWAV beta 1.1.1c attached to post #1 in this thread.

Further minor changes to the spreading-function, resulting in the following bitrates for my 10 album test set:
Code: [Select]
|===============|==========|==========|==========|==========|==========|==========|
|    Version    |   FLAC   | --insane |--extreme |--standard|--portable|  --zero  |
|===============|==========|==========|==========|==========|==========|==========|
|lossyWAV 1.1.0b| 854kbit/s| 632kbit/s| 548kbit/s| 463kbit/s| 376kbit/s| 285kbit/s|
|---------------|----------|----------|----------|----------|----------|----------|
|lossyWAV 1.1.1c| 854kbit/s| 627kbit/s| 542kbit/s| 457kbit/s| 373kbit/s| 281kbit/s|
|===============|==========|==========|==========|==========|==========|==========|
Title: lossyWAV 1.2.0 Development Thread
Post by: krmathis on 2008-09-02 14:22:13
Guess nobody liked my suggestion.
Oh well! ...at least it was worth a try... 
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-02 14:27:38
I love the idea, I just don't have the development platforms or the experience to carry out the conversion.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2008-09-03 17:32:32
I just spend the last 15 min testing LossyWAV V1.1.0b Vs. LossyWAV V1.1.1c Beta at -q 1 (Ginnungagap), in order to see if there was any regression before the big noise shaping jump, personnaly I couldn't hear any major regression/progressions so I guess the serious things for 1.2 can start now

foo_abx 1.3.3 report
foobar2000 v0.9.5.5
2008/09/03 17:58:44

File A: C:\Documents and Settings\Sauvage.S\Bureau\Nouveau dossier\02- Ginnungagap Test Sample (Lossywav)b.lossy.tak
File B: C:\Documents and Settings\Sauvage.S\Bureau\Nouveau dossier\02- Ginnungagap Test Sample (Lossywav)o.lossy.tak

17:58:44 : Test started.
18:00:35 : 01/01  50.0%
18:01:43 : 02/02  25.0%
18:03:09 : 02/03  50.0%
18:03:54 : 03/04  31.3%
18:05:52 : 04/05  18.8%
18:07:42 : 05/06  10.9%
18:10:19 : 05/07  22.7%
18:11:52 : 06/08  14.5%
18:22:29 : 06/09  25.4%
18:22:57 : Test finished.

----------
Total: 6/9 (25.4%)

Edit1: Even if I reached 5/6 at the beginning I couldn't really tell what I was listening to (I mean inside the area of the usual artefact) ... so I think it was lucky guessing ...

Edit2: To be 100% sure it was lucky guessing I spend 10 min this morning to redo the test at -q 0 & I failed even more clearly, so for me there is no regression/improvement except the very small but welcomed kbps gain.

foo_abx 1.3.3 report
foobar2000 v0.9.5.5
2008/09/04 16:27:22

File A: C:\Documents and Settings\Sauvage.S\Bureau\Nouveau dossier\02- Ginnungagap Test Sample (Lossywav)b.lossy.tak
File B: C:\Documents and Settings\Sauvage.S\Bureau\Nouveau dossier\02- Ginnungagap Test Sample (Lossywav)o.lossy.tak

16:27:22 : Test started.
16:29:22 : 01/01  50.0%
16:30:29 : 02/02  25.0%
16:34:43 : 03/03  12.5%
16:35:57 : 03/04  31.3%
16:36:35 : 03/05  50.0%
16:38:30 : 03/06  65.6%
16:38:42 : Test finished.

----------
Total: 3/6 (65.6%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Antonski on 2008-09-03 22:35:01
Suggested foobar2000 converter setup:

lossyFLAC:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.flac
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - -b 512 -5 -f -o%d
Format is: lossless or hybrid
Highest BPS mode supported: 24
lossyTAK:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24
lossyWV:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.wv
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24


Just out of curiosity, why there is no example for lossyAPE (MonkeyAudio)?
Sorry if this has already been mentioned somewhere, the lossyWAV threads are a bit too long

~
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-03 22:57:11
Monkey's Audio does not make use of a "wasted-bits" feature as FLAC, TAK, Wavpack, WMA-Lossless, etc do. Therefore there is no space-saving benefit in using lossyWAV with Monkey's Audio, ALAC, etc.
Title: lossyWAV 1.2.0 Development Thread
Post by: krmathis on 2008-09-04 16:13:30
I love the idea, I just don't have the development platforms or the experience to carry out the conversion.

Fair enough, especially the lack of experience part.
But regarding development platform I am quite sure all you need is a GNU/Linux distro with GCC, ...
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-04 16:25:24
And a knowledge of C which I don't have - lossyWAV is Delphi & IA-32 assembler....
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2008-09-04 19:34:10
And a knowledge of C which I don't have - lossyWAV is Delphi & IA-32 assembler....


http://www.lazarus.freepascal.org/ (http://www.lazarus.freepascal.org/)

I briefly tried it and one of the first problems is dealing with the "uses Windows" import.
But you should be able to workaround that.

If i've read it correctly, it runs in several platforms and *is able to crosscompile* (as in compiling in windows for linux).
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-04 22:20:41
Thanks [JAZ], that's certainly worth a look - if it means that other platforms can be accessed simply by changing my compiler I'll give it a try!
Title: lossyWAV 1.2.0 Development Thread
Post by: servimo on 2008-09-05 04:04:09
Quote
lossyFLAC:
CODE
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.flac
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - -b 512 -5 -f -o%d
Format is: lossless or hybrid
Highest BPS mode supported: 24
What happen if I use lossywav - --standard and flac -8 ? will I have more compression? and why not use it in this default config for foobar2000? if there is some problem.

I did some tests in a little file (an acoustic guitar flac file), this is what happen:

original file:
7.741.294 bytes bitrate 572kbps

lossywav - --standard flac -5:
5.722.908 bytes bitrate 423kbps

lossywav - --standard flac -8: (here I lose the tags(?)) don't know if I did something wrong here, but in the next conversion the tags are all there.
5.713.914 bytes bitrate 422kbps

lossywav - --insane flac -8: (the size is increased)
8.029.224 bytes bitrate 593kbps
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2008-09-05 05:23:36
servimo:
RTFM (http://en.wikipedia.org/wiki/RTFM)
-b 512
Title: lossyWAV 1.2.0 Development Thread
Post by: servimo on 2008-09-05 06:04:22
I should have did it this way: FLAC -> WAV -> lossyFLAC
Is this?
Sorry  I didn't see it is a Development thread and there is others threads. I was just looking for some explanation about lossyWAV
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-05 08:19:30
servimo, the foobar2000 settings work perfectly - if you're having problems, check the converter settings carefully. Please ensure that the "-b 512" remains in the flac element of the command line as this is what ensures optimal flac block length during encoding.

Alternatively, if you create a batch file containing the following:
Code: [Select]
@if exist "%1" flac -d "%1" --stdout --silent|lossywav - --stdout --standard|flac - -b 512 -o "%~n1.lossy.flac" --silent && tag --fromfile "%1" "%~n1.lossy.flac"
and drag-n-drop single files onto it then that should also work.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2008-09-05 08:26:09
...lossywav - --insane flac -8: (the size is increased)...


a) -b 512 usually is essential as sauvage78 said.

b) You encoded a solo instrument. This is the situation where lossyWAV  + FLAC doesn't come out well.
    If you can use WavPack or TAK instead of FLAC the situation is better.
    I have a series of tracks like that, but as they form a very minor portion of my total collection I don't care
    and keep using FLAC. It doesn't sound good that in these cases lossless wavPack yieds smaller files than
    lossyWAV + FLAC, but in practice it's insignificant. Moreover though in theory lossyWAV + FLAC is lossy in
    these cases the error of the procedure is extremely close to zero if not really zero.
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2008-09-05 11:25:47
What happen if I use lossywav - --standard and flac -8 ?


The usage of flac -5 vs any higher compression was chosen because of the nature of FLAC.
Higher settings usually improve on compression, just because they can work on a bigger chunk of samples. But since lossyWav needs to work on a small chunk (so that it can maximize the reduction of bits. That's why the -b 512 setting is required for optimum results), the gains are few, while the encoding time increase.
Title: lossyWAV 1.2.0 Development Thread
Post by: servimo on 2008-09-05 21:56:56
In all those tests I'll keep the -b 512 I didn't change anything alse excepts the parts bold in my post above, --standard and -5. It just happen that when I used -8 for FLAC I didn't have the tags copied between. Nothing more...
I have a DVD with various albums in wav format and I encoded some of them to lossyFLAC and I like the result. The compression and what I hear is very good. The only thing, I think I could notice is that mp3 is more muffled(Google translation) than the lossyFLAC and it reminds me a little of the AAC convertion when I hear it.
As i said above I will keep using these defaults settings, not much gain at all if I use FLAC best compression.
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-09-06 17:01:57
The only thing, I think I could notice is that mp3 is more muffled(Google translation) than the lossyFLAC and it reminds me a little of the AAC convertion when I hear it.


So far only one person has posted transcode listening tests and they had trouble hearing any difference in the mp3's even on problem samples and with "lossyway -P" setting. See http://www.hydrogenaudio.org/forums/index....showtopic=65637 (http://www.hydrogenaudio.org/forums/index.php?showtopic=65637)

My understanding was that (theoretically at least) the lack of psychoacoustics should make it a very good format for transcoding. Can anyone confirm is that correct?
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2008-09-06 17:08:29
...My understanding was that (theoretically at least) the lack of psychoacoustics should make it a very good format for transcoding. Can anyone confirm is that correct?

It is expected to be a very good format for transcoding though just 1 test backs this up. This test was done at low quality setting -q 1, and as people with the target of transcoding are expected to use a higher quality setting this should give some confidence regarding lossyWAV for this one test.
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-09-06 17:14:30
I'm new to lossywav and I've just tested started testing it with a few files. So far I really like it. With the "lossywav - S" setting I'm getting about one half the files size of my previous "monkey audio -extreme" files.

A couple of questions though.

1. When I make a correction file it only seems to contain a small amount of noise without even vestige of the original music. I'm guessing this is a good thing. Is that small noise in the correction file exactly equal to the added noise in the lossywav file or is correspondence between the two more indirect.

2. I see that currently people are mostly interested in the correction file for the purpose of inspection only. Say however that I wanted to keep the correction file for archiving purposes, the correction file seems to compress much more poorly than the actual lossywav so the total storage is larger then with lossless (tak or monkeyaudio). Does anyone know if it would be possible (at least in theory) to make the correction file more compressible, or perhaps for a compressor that "understood it" to compress it more efficiently?
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-09-06 19:16:40
HELP I'm getting double file extentions in all my lossy.tak's converted from foobar (latest 0.9.5.5).

I used the following guide to set up a custom command line encoder in foobar  : http://wiki.hydrogenaudio.org/index.php?title=LossyWAV (http://wiki.hydrogenaudio.org/index.php?title=LossyWAV)

It works perfectly except that the output files are named like "my_song_title.lossy.tak.lossy.tak" and I cant figure out why the double extention. Can anybody help?

BTW, here's the exact code as per the guide. I have exactly this except for different path names where appropriate.

Code: [Select]
lossyTAK settings:

Encoder: c:\windows\system32\cmd.exe
Extension  : lossy.tak
Parameters : /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|
             c:\"program files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-06 20:14:01
uart:

1: The correction file is made up of the difference between the lossless original and the bit-removed samples. Essentially it is white noise, louder where more bits have been removed.

2: The compressibility of the correction file has long been an issue - I suppose if we could find a compressor which would handle it better we might not need lossyWAV at all!

Transcoding problem: try deleting the contents of the "extension" box in the converter settings in foobar2000, then retype "lossy.tak".
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-09-06 20:35:04
Thanks for the info Nick.

Transcoding problem: try deleting the contents of the "extension" box in the converter settings in foobar2000, then retype "".


No that didn't fix it.

BTW, if I remove the "lossy.tak" from the extention box then the conversion fails. When I put it back in it works but with the double extentions.

When I right-click (in foobars playlist) the file I want to convert and select "convert to..." etc, just before it does the conversion it pops up a "save as" dialog and the filename in the dialog is "my_song_name.lossy.tak. However when I click save and it start converting the "converter" states that the destination is "my_song_name.lossy.tak.lossy.tak

If I edit the save dialog and delete the ".lossy.tak" extention before I hit save then it names the file correctly. That is, if make the save file dialog read just "my_song_name" without any extention, then when I hit save it adds the correct extentions just once. This works ok but I have to do it manually every time.

BTW. I have my file system settings (in winxP) set to display file extentions (the default windows XP setting is not to do so, but I'm guessing that many others like myself enable it). Could that be a problem?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-06 20:38:05
I have all file extensions visible by choice too.

How are you naming your files in the converter?

I use:
Code: [Select]
[[%album artist% - ][$char(91)%date%$char(93)] %album%\][%discnumber%-]%tracknumber% - %artist% - %title%
for single tracks and:
Code: [Select]
[%album artist% - ][$char(91)%date%$char(93) ]%album%[ - CD%discnumber%]
for albums.
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-09-06 20:48:18
UPDATE.

Previously I've just been testing this with a single file. Just now I tested it with multiple files selected for conversion. Whereas a single files displays the actual filename in the save dialog box in the case of multiple files it only display the destination folder name. Well what do you know, it names the multiple files correctly.

I'd still be interested to know if there's a fix, but at least the work-around is not so bad now that I know I only need to do it (that is, to delete the extention in the save dialog box before clicking save) for the case of converting single files.

I have all file extensions visible by choice too.

How are you naming your files in the converter?

I use:
Code: [Select]
[[%album artist% - ][$char(91)%date%$char(93)] %album%\][%discnumber%-]%tracknumber% - %artist% - %title%
for single tracks and:
Code: [Select]
[%album artist% - ][$char(91)%date%$char(93) ]%album%[ - CD%discnumber%]
for albums.


I haven't edited those fields. They currently read,

Single track
Code: [Select]
[%list_index% ]%title%


Album Images
Code: [Select]
[%album artist% - ]%album%


Is that a problem?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-06 20:55:20
Your track / album naming strings shouldn't be a problem. I am at a loss with respect to a solution as I have never encountered this phenomenon using foobar2000.
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-09-06 21:03:29
Thanks anyway Nick. I've just cut and pasted your filename settings into foobar and they work fine. With your settings it no longer puts the filename in the save dialog box, it just puts the folder name and everything works fine. I guess I'll have to learn the syntax of those foobar settings if I want to change anything there, otherwise I'll just keep your setting.
Title: lossyWAV 1.2.0 Development Thread
Post by: foosion on 2008-09-06 21:52:41
My theory is that some part of the software checks if you have set the file name has already the extension .lossy.tak, but compares it against .tak only. Of course, that doesn't match, so it will helpfully append .lossy.tak to the file name. Since this occurs only when converting single files, I suspect that the culprit may be the standard Windows "Save As" dialog, but I'm not sure.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2008-09-08 10:30:15
As I wanted to know if it was usefull to include a lowpass filter to lossywav (with the hope to save some space) I did a quick test with Adobe Audition:

Album: Darkness, The - 2003 - Permission To Land:

CDImage Original            + Tak -p2e ==> 286mo
CDImage Lowpass 20Khz + Tak -p2e ==> 280mo
CDImage Original            + Lossywav Portable + Tak -p2e ==> 97.3mo (102 092 800 octets)
CDImage Lowpass 20Khz + Lossywav Portable + Tak -p2e ==> 97.3mo (102 054 995 octets)

Album: Fantômas - 2001 - The Director's Cut

CDImage Original            + Tak -p2e ==> 246mo
CDImage Lowpass 20Khz + Tak -p2e ==> 243mo
CDImage Original            + Lossywav Portable + Tak -p2e ==> 97.3mo (102 080 296 octets)
CDImage Lowpass 20Khz + Lossywav Portable + Tak -p2e ==> 97.3mo (102 090 392 octets)

needless to say it is completely useless  ... but I still wanted to share the result with everyone ...
Title: lossyWAV 1.2.0 Development Thread
Post by: Gow on 2008-09-09 04:11:43
Anyone know of a way to set up up a Lossy WMA-L string for foobar2000?  Zune 80 supports WMA-Lossless and I was reading WMA-L also works with Lossy WAV, so I figured I would go Lossy WMA-L and replace the Mp3/m4a library that I sync my Zune with. 

Though so far it looks like I am going to have to convert to another lossy WAV format and then to WMA-Lossless if I use just foobar2000.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-10 07:55:50
Sorry, I don't know how to setup WMA-L in foobar2000 - good luck!

lossyWAV 1.1.1d attached to post #1 in this thread.
Code: [Select]
|===============|==========|==========|==========|==========|==========|==========|
|    Version    |   FLAC   | --insane |--extreme |--standard|--portable|  --zero  |
|===============|==========|==========|==========|==========|==========|==========|
|lossyWAV 1.1.0b| 854kbit/s| 632kbit/s| 548kbit/s| 463kbit/s| 376kbit/s| 285kbit/s|
|---------------|----------|----------|----------|----------|----------|----------|
|lossyWAV 1.1.1c| 854kbit/s| 627kbit/s| 542kbit/s| 457kbit/s| 373kbit/s| 281kbit/s|
|---------------|----------|----------|----------|----------|----------|----------|
|lossyWAV 1.1.1d| 854kbit/s| 631kbit/s| 547kbit/s| 462kbit/s| 377kbit/s| 283kbit/s|
|===============|==========|==========|==========|==========|==========|==========|
Title: lossyWAV 1.2.0 Development Thread
Post by: Gow on 2008-09-11 00:47:51
Well I have tried and failed...I haven't pinned down the right sequence yet...

What I have used so far...

Code: [Select]
/d /c C:\"Program Files (x86)"\foobar2000\codec\lossywav - --portable --silent --stdout | "C:\Program Files (x86)\Windows Media Components\Encoder\WMCmd.vbs" -silent -a_codec WMA9LSL -a_mode 2 -a_setting Q100_44_2_16 -input %s -output %d


Problem is it gives back a WSHShell / undefined variable...probably has to do something with the piping or some other little variable.

I will have to stick with the workaround method I have setup at the moment.  Convert to Lossy Portable FLAC and then convert that to WMA-Lossless.

I will keep trying every once in a while to try to figure some tweak or some trick to get it to work as a command line.
Title: lossyWAV 1.2.0 Development Thread
Post by: rbrito on 2008-09-16 23:48:29
And a knowledge of C which I don't have - lossyWAV is Delphi & IA-32 assembler....


As I mentioned in another post, I think that I can help you here a bit, so that the program truly becomes cross-platform. I'm dying to know about how it is used and how it works on my files.

Regards, Rogério Brito.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-24 21:38:25
I've been busy transcoding the IA-32/x87 assembler routines back to vanilla Delphi. Only a few more routines to go (nmath &  nwav units) and I'll post the source of 1.1.1e. At present, the processing speed has decreased by a factor of approximately 3 for the (much) reduced assembler version - shows the advantages of hand-coded assembler....
Title: lossyWAV 1.2.0 Development Thread
Post by: rbrito on 2008-09-24 23:39:25
Hi, Nick.

I've been busy transcoding the IA-32/x87 assembler routines back to vanilla Delphi. Only a few more routines to go (nmath &  nwav units) and I'll post the source of 1.1.1e.


Thanks for the information. I was already going to transcode those very units, but seeing that you are working on them, I don't want to deal with a moving target.

Quote
At present, the processing speed has decreased by a factor of approximately 3 for the (much) reduced assembler version - shows the advantages of hand-coded assembler....


Well, that's one price to pay for those which are interested in having the tool be cross-platform. Also, I think that Delphi would have the ability to do conditional compiles, doesn't it? And as compilers mature, we can reach the speed of hand-coded assembly asymptotically...

In the worst case scenario, we can use some already optimized build-dependency like "liboil" (for "optimized inner loops").


Regards, Rogério Brito.

P.S.: BTW, it would be nice if you used the SVN repository of sourceforge, since, this way, other people could see the conversion and learn from it. I am sure that I would appreciate it.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-25 07:15:41
All of the IA-32 / x87 source is still in the project, as you surmised "hidden" behind compiler conditional defines so that I can check that I haven't broken the "fast" version while re-implementing the Delphi version. There is a one-to-one correspondance of routines, so far, between Assembler and Delphi - this should allow people to examine and compare the original Delphi and the optimised assembler versions.

I haven't worked out how to do anything with the SourceForge project - it's an empty folder.
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2008-09-25 11:42:05
I haven't worked out how to do anything with the SourceForge project - it's an empty folder.


You currently have the project configured to use CVS. I personally recommend using the SVN option instead*. Here you have information of both cases

CVS: How to install tortoiseCVS and configure it to use with sourceforge. Do *not* configure it for anonymous user. You wouldn't be able to commit (upload) the changes, as that cannot be changed afterwards.
http://alexandria.wiki.sourceforge.net/CVS...eCVS+with+PuTTY (http://alexandria.wiki.sourceforge.net/CVS+Client+-+TortoiseCVS+with+PuTTY)


SVN: Enable the SVN option in sourceforge:
Login into sourceforge, and go to this page (else you won't have access)
https://sourceforge.net/project/admin/svn.p...group_id=228679 (https://sourceforge.net/project/admin/svn.php?group_id=228679)

Enable the checkbox for subversion, and press "Update" button.

Similarly, go to
https://sourceforge.net/project/admin/cvs.p...group_id=228679 (https://sourceforge.net/project/admin/cvs.php?group_id=228679)

and uncheck the CVS option and press update
(These options are under the Admin menu in the project main page. Now they hide it under the "more" option)

How to install TortoiseSVN and configure it:

http://alexandria.wiki.sourceforge.net/Sub...t+-+TortoiseSVN (http://alexandria.wiki.sourceforge.net/Subversion+Client+-+TortoiseSVN)


Once you have any of the options configured, you have to checkout the repository (which will create an empty directory in your machine with either a CVS or a .svn subfolder) where you can copy the source files, and then do a commit of this directory to upload the changes. (If using SVN, it will ask for your username and password at this point)


The source can always be seen on these url's. It is a great tool to follow the changes.

if CVS: http://lossywav.cvs.sourceforge.net/lossywav/ (http://lossywav.cvs.sourceforge.net/lossywav/)
if SVN: http://lossywav.svn.sourceforge.net/viewvc/lossywav/ (http://lossywav.svn.sourceforge.net/viewvc/lossywav/)




* SVN offers several advantages over CVS, like checking differences with the previous release without the need to contact the repository, renaming of directories and can connect directly to sourceforge, without the need of configuring a public key.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-30 08:18:34
Reversion to Delphi complete (with conditional define to re-enable all of the existing IA-32/x87 code).

lossyWAV beta 1.1.1e attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: rbrito on 2008-09-30 14:53:14
Reversion to Delphi complete (with conditional define to re-enable all of the existing IA-32/x87 code).

lossyWAV beta 1.1.1e attached to post #1 in this thread.


Hummm, I downloaded the zip files that you posted on the first post and I see only Windows executables. Perhaps I'm blind, but I would like to import the sources soon to our SVN repository. Where is the source code? (sorry for the lame question).

A note to other people: we have already established a project on sourceforge and I removed the CVS repository and enabled the SVN one.

I will keep two main parts there: one for Nick's development in Delphi and one for porting issutes. I welcome anybody to send patches (unified format largely preferred) to the code that will be there.

I intend to code things in plain C and create a Debian package of it latter (so many Unix distributions---FreeBSD, Debian, OpenBSD, Ubuntu, Gentoo etc) can benefit from it.

Regards, Rogério Brito.


Sorry, sorry. Just found out the sources. They will be checked into the repository.


Regards, Rogério Brito.
Title: lossyWAV 1.2.0 Development Thread
Post by: rbrito on 2008-09-30 16:11:05
I just checked in the new sources of Nick into the repository. They can be seen here:

http://lossywav.svn.sourceforge.net/viewvc/lossywav/ (http://lossywav.svn.sourceforge.net/viewvc/lossywav/)

Regards, Rogério Brito.
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-09-30 17:47:01
Hi Nick. I was just wondering what version of Delphi are you using on this project? I use Borland Delphi 7 (probably a bit old) and it freezes if I try to open the project file.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-09-30 19:53:11
I downloaded Turbo Delphi 2006 from the Codegear site. It comes with a free 100 year licence when you register to be able to download it.

The reason it is crashing may be that I have done most of the development on a USB stick so that the code is suitably portable - that may cause the IDE to crash when you open the project. I would try creating a new console application and pasting lossywav.dpr into the main code window.
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-10-01 18:29:14
Ok thanks Nick, I'll give that a try
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-10-02 15:08:15
Ok that worked. After a little bit of fiddling I've now got it compiling properly under the older Borland Delphi-7.
Title: lossyWAV 1.2.0 Development Thread
Post by: hellokeith on 2008-10-04 23:34:53
Well I have tried and failed...I haven't pinned down the right sequence yet...

What I have used so far...

Code: [Select]
/d /c C:\"Program Files (x86)"\foobar2000\codec\lossywav - --portable --silent --stdout | "C:\Program Files (x86)\Windows Media Components\Encoder\WMCmd.vbs" -silent -a_codec WMA9LSL -a_mode 2 -a_setting Q100_44_2_16 -input %s -output %d


Problem is it gives back a WSHShell / undefined variable...probably has to do something with the piping or some other little variable.

I will have to stick with the workaround method I have setup at the moment.  Convert to Lossy Portable FLAC and then convert that to WMA-Lossless.

I will keep trying every once in a while to try to figure some tweak or some trick to get it to work as a command line.


Cscript.exe is required to run .vbs files.  I believe it is in C:\Windows\System32\ and put the .vbs as an argument after it with full path in quotes.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-10-22 08:25:06
My implementation of SG's new noise-shaping method having stalled (only for a while - my head hurts....), I am turning my attention to the examination of 2ch. audio for M/S content.

What I really need are a few samples which have such content so that I can get a feel for what I'm looking for.

I can, of course, develop the algorithm without samples but any output cannot be meaningfully examined without them.
Title: lossyWAV 1.2.0 Development Thread
Post by: uart on 2008-10-22 16:39:04
What exactly do you need Nick? Do you mean you simply need some material that has a lot more content in the mid compared to the side channel?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-10-22 20:27:17
I would like some content which has some form of matrix surround encoding (as David mentions here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=63254&view=findpost&p=565434)) to make sure that it doesn't suffer too much when processed by lossyWAV.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-18 16:31:40
Greetings.

I was thinking about ways to gracefully compress HDCD-encoded CDs to a lossy format. There's is a nice HDCD decoder by Christopher Key, which turns HDCD-encoded wav into 24-bit wav with 4 bits wasted (actually containing 20 meaningful bits per sample).

So my questions are:
1) Does lossyWAV support 20-bit input, and would it make any difference if the file was 20-bit or 24-bit with 4 bits wasted?
2) Is there a way to force lossyWAV to get rid of at least 4 bits in every frame to be able to store the output as a regular 16-bit wav?
Title: lossyWAV 1.2.0 Development Thread
Post by: pdq on 2008-11-18 16:42:14
Greetings.

I was thinking about ways to gracefully compress HDCD-encoded CDs to a lossy format. There's is a nice HDCD decoder by Christopher Key, which turns HDCD-encoded wav into 24-bit wav with 4 bits wasted (actually containing 20 meaningful bits per sample).

So my questions are:
1) Does lossyWAV support 20-bit input, and would it make any difference if the file was 20-bit or 24-bit with 4 bits wasted?
2) Is there a way to force lossyWAV to get rid of at least 4 bits in every frame to be able to store the output as a regular 16-bit wav?

1) I think you don't need lossy wav to encode your 20 bits out of 24. Most lossless encoders will account for the low 4 bits always being zero.

2) There are other ways to truncate (or preferrable dither) the data to 16 bits, then pass that on to lossy wav as a regular 16 bit wav file.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-18 16:47:14
1) I think you don't need lossy wav to encode your 20 bits out of 24. Most lossless encoders will account for the low 4 bits always being zero.

2) There are other ways to truncate (or preferrable dither) the data to 16 bits, then pass that on to lossy wav as a regular 16 bit wav file.


1) I know, but i want to encode to lossy wav, not just compress to flac, so the question is how does it handle such input.

2) There are, but i'm afraid that the quality would suffer more after two subsequent lossy processes.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-18 18:38:43
If you decode HDCD to 24-bit WAV, process that output with lossyWAV and pipe to FLAC you will get a 24-bit lossyFLAC file. This will have undergone 1 lossy process (lossyWAV). The difference in filesize between a HDCD>24-bit>16-bit>lossyWAV>FLAC and HDCD>24-bit>lossyWAV>FLAC *should* be a small percentage.
Title: lossyWAV 1.2.0 Development Thread
Post by: pdq on 2008-11-18 19:11:10

1) I think you don't need lossy wav to encode your 20 bits out of 24. Most lossless encoders will account for the low 4 bits always being zero.


1) I know, but i want to encode to lossy wav, not just compress to flac, so the question is how does it handle such input.

A lossy wav file is the same size as the original, just with a little bit more noise. The only possible use that I can see for it is that it gets much smaller when you compress it losslessly.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-18 19:34:33
If you decode HDCD to 24-bit WAV, process that output with lossyWAV and pipe to FLAC you will get a 24-bit lossyFLAC file. This will have undergone 1 lossy process (lossyWAV). The difference in filesize between a HDCD>24-bit>16-bit>lossyWAV>FLAC and HDCD>24-bit>lossyWAV>FLAC *should* be a small percentage.


Thanks for an answer, but that i understand.

The question was, how does lossyWAV handle 20 bit input or 24 bit input with 4 wasted bits and is there any difference in quality between those cases.

And also, is it possible to force lossyWAV to destroy _at least_ given number of bits (e.g. 8 bits for 24 bit input). The reason i'm asking this is not that i'm hoping for a smaller filesize, but because a) some lossless codecs only support 16-bit audio, b) some hardware players only support 16-bit audio. And i don't want to have two lossy processes involved, because this obviously is bad for sound quality.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-18 20:05:26
lossyWAV does not recognise HDCD input - Christopher's decoder uses API calls to the Windows library - thus making it platform dependent. I am unwilling to tie lossyWAV to one platform (especially just after transcoding all of the IA-32 assembly language routines back to Delphi).

Christopher's decoder outputs 24-bit WAV files (for decoded HDCD output).

No option is available to force (at least) a certain number of bits to be removed - it goes against the principle of lossyWAV. Also, lossyWAV, as pdq said above, outputs a file which is almost identical in size to the original (the only addition is a 'fact' chunk and even then not if the output is piped to STDOUT) at the same bit-depth as the original.

I can only suggest you try the following out and see if it sounds palatable:

HDCD > 24-bit WAV > 16-bit WAV (foobar2000?) > lossyWAV > FLAC.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-18 20:20:57
Thanks.

Christopher's decoder outputs 24-bit WAV files (for decoded HDCD output).

In fact, it outputs 24-bit WAV files with 4 lower bits set to zero.

I still don't understand, can (or should) i somehow instruct lossyWAV that 4 lower bits are zero, and will it make any difference? In other words, does lossyWAV support wierd bitsPerSample values, or only fixed set of 16/24?
Title: lossyWAV 1.2.0 Development Thread
Post by: pdq on 2008-11-18 21:09:18
Any low bits that are already zero will remain zero. What lossy wav does is, when appropriate, zero out additional lower bits to make lossless compression more effective.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-18 21:39:32
Any low bits that are already zero will remain zero. What lossy wav does is, when appropriate, zero out additional lower bits to make lossless compression more effective.

Thanks a lot, i really appreciate the answers but those are the things that i already know. What i wanted to know, is does lossyWAV support 20-bit input natively.

Let me explain:
If i take 16 bit wav, and turn in into 24 bit wav by adding 8 zero bits, then process with lossyWAV, then turn it in 16 bit wav again by removing 8 lower bits, i'm not sure i will get the same result as if i processed the 16 bit input directly.

16 bit WAV -> add 8 bits -> lossyWAV -> remove 8 bits == 16 bit WAV -> lossyWAV?

If yes, than it makes no difference if the input is 20 bit or 24 bit with 4 wasted bits.
But if no, than it makes a difference if 20 bit input is supported natively.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-19 07:17:31
A 20-bit sample in a 24-bit container will already be "left justified" as required by the WAV specification, i.e. effectively multiplied by 16, therefore lowest 4 bits are going to be zero.

16>24>lossyWAV>16 will probably not equal 16>lossyWAV, additionally, if the lossyWAV>16 step includes dither then the carefully zeroed lsb's will disappear and the wasted-bits compression advantage will be lost.

[edit] lossyWAV supports 9-bit to 32-bit integer samples in 16, 24 and 32-bit signed integer "containers", exactly the same as the WAV specification. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-20 14:42:18
A 20-bit sample in a 24-bit container will already be "left justified" as required by the WAV specification, i.e. effectively multiplied by 16, therefore lowest 4 bits are going to be zero.

16>24>lossyWAV>16 will probably not equal 16>lossyWAV, additionally, if the lossyWAV>16 step includes dither then the carefully zeroed lsb's will disappear and the wasted-bits compression advantage will be lost.

[edit] lossyWAV supports 9-bit to 32-bit integer samples in 16, 24 and 32-bit signed integer "containers", exactly the same as the WAV specification. [/edit]


Thanks.
Ok, i ported lossyWAV to C#, and will try to fool around with 20-bit input.
I can also send my code when i tidy it up a little, if you are interested.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-20 15:06:55
Yes, thanks.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-20 22:14:01
Here it is:

http://www.hydrogenaudio.org/forums/index....showtopic=67418 (http://www.hydrogenaudio.org/forums/index.php?showtopic=67418)

Tested to be bit identical with 1.1.1e on at least one file, at least with default quality setting
Title: lossyWAV 1.2.0 Development Thread
Post by: lvqcl on 2008-11-21 00:18:45
Processing 1-hour wav (16/44):
LossyWav.exe: 73 sec;
LossyWavSlow.exe: 239 sec;
LossyWavSharp.exe: 144 sec;

Cute. And yes, LossyWavSharp and LossyWavSlow produced identical output.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-21 07:47:58
Here it is:

http://www.hydrogenaudio.org/forums/index....showtopic=67418 (http://www.hydrogenaudio.org/forums/index.php?showtopic=67418)

Tested to be bit identical with 1.1.1e on at least one file, at least with default quality setting
Gregory,

Many thanks for taking the time to port the core functionality of lossyWAV from Delphi to C#. The speed of this first version is impressive!

Looking at the code for the FFT functions - are you using a trick to only have to create one array of A(Re,Im) values by shifting the index by the bitlength difference of the FFT length in question and the maximum FFT bitlength?

Rógerio Brito (rbrito on these forums) is interested in a C port - how different is C# to C (I *really* don't know anything about C)?

Well done again!

Nick.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-21 09:47:42
Looking at the code for the FFT functions - are you using a trick to only have to create one array of A(Re,Im) values by shifting the index by the bitlength difference of the FFT length in question and the maximum FFT bitlength?

Yep. I'm also using a similar trick for reversedbits table.

Rógerio Brito (rbrito on these forums) is interested in a C port - how different is C# to C (I *really* don't know anything about C)?

It's still a lot of work, but about twice easier now - the basic operators are the same in C and C#, as are the basic concepts, such as arrays indexing. Most of the problems i had at first were caused by pascal-style arrays indexing (often starting at 1 instead of 0). With these obstacles removed, it's now only a matter of time to create a C port. Anyone who is familiar with C can read C# code. I hope someone will find a time to do this

UPD: found a small bug in my WriteChunk function from AudioCodecsDotNet.cs, the chunks were not aligned on the word boundary. Corrected version is in CUETools SVN.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-22 15:29:07
I added lossyWAV support to CUETools.

New CUETools also include command line lossyWAV utility.

Binaries can be found here (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=35&t=66233&st=0#entry591203), and source code is here (http://cuetoolsnet.svn.sourceforge.net/viewvc/cuetoolsnet/).

LossyWAVDotNet is a C# class library which implements the algorithm,
AudioCodecsDotNet is a C# WAV reader/writer library
LossyWAVSharp is a command line utility which uses them.

Those 3 projects can be built independently of other projects in CUETools SVN.

Changes to the first version: class library interface improved, fft very slightly optimized and command line utility supports some basic command line options.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-22 18:12:18
It looks like I should "hang up" Delphi and get my head around C....

Well done again!
Title: lossyWAV 1.2.0 Development Thread
Post by: Neasden on 2008-11-22 18:24:47
Did people give up on ABX'ing LossyWAV? I didn't see usual ABX tests anymore. I hope it's that good!
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-22 20:11:15
--standard (-S or -q 5) appears, from all of the individual testers' responses appears to be transparent.

--portable (-P or -q 2.5) also appears to be transparent from the limited testing done on it so far.

My HTPC local copy of my full collection is --portable lossyFLAC, and comes in at about 372kbps, down from 880kbps for lossless FLAC. I have not yet heard an artefact in that collection - however I am not an ABX tester....
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2008-11-22 22:39:08
Did people give up on ABX'ing LossyWAV? I didn't see usual ABX tests anymore. I hope it's that good!

New ears are always welcome!
I did a lot of ABX testing before, but now that lossyWAV has been in a mature state since a long time there is no sense to continue testing. To me and those samples I tested lossyWAV is perfect. But this doesn't mean that in the universe of music there may be problems showing up. That's why it would be nice if we had new testers.

In case you like to distribute I suggest you do it with the --portable quality. In theory at least there should be samples with --portable being not transparent. If such samples should show up it is important to check whether the --standard quality is fine with them or not.
Title: lossyWAV 1.2.0 Development Thread
Post by: singaiya on 2008-11-22 23:10:57
I gave up ABX testing it. It was really difficult on --portable, even though I came very close on a few samples in my transcoding test.
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2008-11-23 12:08:01
LossyWAVDotNet is a C# class library which implements the algorithm,
AudioCodecsDotNet is a C# WAV reader/writer library
LossyWAVSharp is a command line utility which uses them.


I want to add that i've started a Java port from those C# sources.
The performance will be lower, for sure, and i'll have to recheck limits here and there ( for java, everything is signed, unsigned values do not exist. multi-level arrays are not matrixes, but arrays with references to other arrays. Casting can not be freely done like in C/C++/C#.)

I am trying to write it in the most correct way, as being more a reference than a best-of. (I've extracted the FFT parts into its own class, for example. I'm making constants in its own file, I'm removing redundant information and try to simplify the workflow).

I guess i will need still two or three days to complete the port. I'll post it here when done.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-23 20:09:14
Thanks for your effort [JAZ], it's appreciated.
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2008-11-30 19:33:54
Some updates on the port to Java:

I have a working code already, based mostly on the C# sources and to a lesser extent to the delphi sources or the web when something didn't look quite right/clear enough.

I don't have yet a .jar that could be tested, but i'll look into making one soon.

Currently:

It is a lot slower than the C# sources. ( by two orders of magnitude). This is because i've removed some of the lookup tables and implemented the FFT methods as they would be by the book.
Of course, i am in the process to add those tables back, in a way that could reflect what they really are and why they are there.

Another thing that i've discovered so far is that while the output seems to produce similar results, it does not produce the same ones than lossyWavSharp. 
I've processed a .wav with both programs, and they play fine in foobar, but the one generated by jLossyWav, if encoded with .flac, is smaller.
I have no idea yet why and if this is good or bad. I hope to get a clearer idea after finishing the source separation.

Code: [Select]
jCD1_02.wav.lossy.wav: wrote 995611 bytes, ratio=0,342

CD1_02.wav.lossy.wav: wrote 1026802 bytes, ratio=0,353
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-30 19:51:15
Sounds good [JAZ] - keep up the good work!

The lookup tables, especially in the FFT save a lot of time.

I have used Gregory's "trick" of shifting the lookup index by the appropriate number of bits to reduce (by about half) the memory requirements of some of the lookup tables in what will be lossyWAV 1.1.1f (Delphi).
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-11-30 21:00:15
Some updates on the port to Java:

I have a working code already, based mostly on the C# sources and to a lesser extent to the delphi sources or the web when something didn't look quite right/clear enough.

I don't have yet a .jar that could be tested, but i'll look into making one soon.

Currently:

It is a lot slower than the C# sources. ( by two orders of magnitude). This is because i've removed some of the lookup tables and implemented the FFT methods as they would be by the book.
Of course, i am in the process to add those tables back, in a way that could reflect what they really are and why they are there.

Another thing that i've discovered so far is that while the output seems to produce similar results, it does not produce the same ones than lossyWavSharp. 
I've processed a .wav with both programs, and they play fine in foobar, but the one generated by jLossyWav, if encoded with .flac, is smaller.
I have no idea yet why and if this is good or bad. I hope to get a clearer idea after finishing the source separation.

Code: [Select]
jCD1_02.wav.lossy.wav: wrote 995611 bytes, ratio=0,342

CD1_02.wav.lossy.wav: wrote 1026802 bytes, ratio=0,353

To my experience, FFT is the most CPU consuming part of the algorithm, practicly the only one that should be really optimized.

At the early stages i also had quite different results from the delphi project. I ended up debugging them simultaniously and comparing internal values at each step. Tedious work, but had to be done. I recently found out that even to this day there are audio files, on which C# version removes one bit more or one bit less on a block or two. I blame the rounding errors, due to the different floating point precision. Delphi used float, double and extended values. C# port only uses float and double. I tried to preserve the float values, even though the temptation was strong to do everything in double. Changing the precision changes the algorithm output. Not that there's something bad about it, the higher the precision - the more accurate the results are, obviously, but in that case i would not be able to compare result to the reference delphi implementation.

I don't really think there can be 100% match, each language does it's arithmetic slightly different when it comes to rounding and such, but the difference in ratio that you quoted (0,342 vs 0,353) indicates that it's not the only issue here right now.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-11-30 21:04:18
The pure Delphi and Delphi / IA-32 / x87 versions of 1.1.1e produce *slightly* different results with my 54 test sample set (less than 512 bytes in 30MBytes of resultant FLAC).
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2008-12-01 22:47:29
I've uploaded a .jar file with the implementation of the LossyWav algorithm in java.

The file is in this thread:

http://www.hydrogenaudio.org/forums/index....=67697&st=0 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=35&t=67697&st=0)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-12-02 07:47:07
Thanks again [JAZ] - more potential platforms for lossyWAV processing is certainly a good thing.
Title: lossyWAV 1.2.0 Development Thread
Post by: ExUser on 2008-12-09 04:34:59
I too would like 32-bit float support, if that's even possible. My reasons are weird: I'm planning on releasing an EP at some point in the not-so-distant future. I want to amplify the release at +15dB past its ReplayGain value to protest the Loudness War. ReplayGain users won't run into the issue, naturally, just everyone else. However, bitrates get kinda crazy with 32-bit float. So I want to use LossyWAV to chop the bitrate down to something sane for release in FLAC.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-12-09 07:21:05
lossyWAV doesn't support 32-bit float values (9 to 32-bit signed integers only) - and I don't think that FLAC does either, unfortunately.

It is *possible* to include 32-bit float compatibility in lossyWAV, however a target lossless codec would need to be found which could handle floats and also make use of the wasted-bits feature.

[edit] integer compatibility added. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: ExUser on 2008-12-09 08:02:12
Ah right, it's WavPack that supports 32-bit float...
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2008-12-19 21:34:31
I have uploaded a new version of jLossyWav. This time it actually works


The compression is slow, around 0.5X realtime. I hope it can be improved some still, but probably by not much. As i said, this is more an educational port than anything else.


It is producing smaller files than the sharp version of lossywav, but this time the differences are small:

cd1_02.wav_sharp.lossy.wav: wrote 866856 bytes, ratio=0,298
cd1_02.wav_j.lossy.wav: wrote 865984 bytes, ratio=0,298

This probably comes from a different way of calculating the bitsToRemove value, and i've seen the difference file to be quite similar to the one of sharp, just removing more bits in some places. (i.e. what should be expected)

I still want to work the code a bit more, concretely subdivide the Analyzer.process() function, and make more obvious what it is doing.



Download was at :
http://www.hydrogenaudio.org/forums/index....=67697&st=0 (http://www.hydrogenaudio.org/forums/index.php?showtopic=67697&st=0)

[edit: Woops! corrected the link]
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-12-22 08:01:53
Excellent work, [JAZ] - it seems to be coming along nicely.

I've been thinking about removing the FACT chunk for some time now - reasons:

1) Is it really necessary?
2) If the output is piped then the FACT chunk is missing anyway;
3) If a file is re-processed then either no bits will be removed (new q > old q) or the appropriate number of bits (new q < old q) or nearly no bits at all (new q = old q);
4) It's a bit of a pain and requires much more WAV chunk processing than would otherwise be needed;
5) Using the --sampledist parameter the distribution of LSBs and MSBs for the input and output can be examined for signs of pre-processed audio.

Feedback gratefully received.

I have been using the shifted array index trick introduced by Gregory and have now applied it to all of the relevant arrays. This required the Window_Function to be slightly re-written to make it compatible.

Overall memory footprint has reduced quite a bit, but to reduce it further will require modifications to the WAV handling unit as it has quite a large memory requirement (currently....).

Thinking of releasing a 1.2.0 soon - although obviously without the adaptive noise shaping method (a very interesting, complex and baffling proposition that will remain firmly on the to-do list).

[edit] Clarity regarding memory footprint [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-12-22 12:20:23
I agree that the FACT chunk is quite useless.

The whole point of lossyWAV is that we can compress the output with the lossless format of our choice, but with most formats the FACT chunk will be lost.

Some indication that the file is actually lossy is required though. Naming the files as *.lossy.* is not enough, because there's a lot of software out there which automaticly renames files according to the tags.

However, all lossless formats that i know of support some kind of tagging. Tags are usually preserved when transcoding from one format to another. So i think the best way to do this is to choose a tag, e.g. "LOSSY", and add it when compressing to lossless format - we need special options for each lossless format anyway to make sure the block size is right, etc.

For example, lossyFLAC command line would be:
Code: [Select]
c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - --tag=LOSSY=yes -b 512 -5 -f -o%d


Also we might want to store scaling_factor in correction file to be able to merge it with lossy part.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2008-12-22 14:01:40
I agree that the FACT chunk doesn't serve much purpose as it's easilly lost. I was religiously putting "LossyWav" in the comment tag to avoid accidental reprocessing until the realisation dawned that it really didn't matter anyway.
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2008-12-23 22:54:32
About the fact chunk:

With a self compiled version of lossywavSharp, and my system settings, I get errors with that chunk and have to remove it in order for FLAC to encode it. I believe the reason is because the lossywavSharp implementation does not pad the chunk. (A chunk has to be always even. If it is odd, a zero has to be appended).
Probably it doesn't matter, since the compiled version seems to work.

About its use, it helps identify a .wav file, not a losslesly compressed counterpart.

I don't see wav chunks as being something difficult, from the software side of the fence. As for programs/end user, I believe it doesn't matter.



About the memory usage:

Since Java does arrays in a different way, I haven't worried about optimizing them. Yet, I saw several of those that can be changed as to reduce their size.
Most notably, these:

public int[] spreading_averages_int;
public float[] spreading_averages_rem;
public float[] spreading_averages_rec;
public float[] skewing_function;

Their size can be calculated when hi_bins and lo_bins are calculated. (After knowing the bitdepth and sampleRate of the input file)

spreading_averages_int = new int[hi_bins + 1];
spreading_averages_rem = new float[hi_bins + 1];
spreading_averages_rec = new float[hi_bins + 1];
skewing_function = new float[hi_bins + 2]
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2008-12-23 22:55:57
Quote
I believe the reason is because the lossywavSharp implementation does not pad the chunk.

This was a bug in the earliest version, fixed long time ago. Funny thing, this bug revealed itself only past 12AM - when the date string became one byte longer
Title: lossyWAV 1.2.0 Development Thread
Post by: Hancoque on 2008-12-25 02:43:42
Thinking of releasing a 1.2.0 soon - although obviously without the adaptive noise shaping method (a very interesting, complex and baffling proposition that will remain firmly on the to-do list).

What about the checking for matrix surround content? I know, nobody has provided any test samples to analyse. I looked for a Pro Logic II encoder to create some material from 5.1 files but couldn't get one. I wonder how many music albums there are that feature matrix surround encoding. Do they justify any further investigation?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2008-12-25 20:24:16
I have (sort of) implemented M/S checking - I'll go back to it and make sure that it makes sense....
Title: lossyWAV 1.2.0 Development Thread
Post by: Brent on 2009-01-06 21:39:37
With enduring hope that ALAC somehow someday would be compatible with lossywav, I tried the ALAC encoder thats been newly added to ffmpeg a few months ago. Same results as with iTunes: no reduction in filesize...

Just thought I'd mention it.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-07 19:44:08
Didn't find that. Thanks Nick.

Could sombody tell me the correct code to transcode to LossyWAV with correction file?

At the moment I'm using foobar200 with following comandlines:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c C:\Programme\"Audio Tools"\foobar2000\lossyWAV.exe - --standard --silent --stdout|C:\Programme\"Audio Tools"\foobar2000\Takc.exe -e -p4 -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

What do I have to add to get the correction file? It should have the same file name just with the ".lwcdf.tak" extension. I know it has to do with the Command "-C" but I don't get it to work.

thanks in advance.
lossyWAV correction file creation is incompatible with piped output. You would have to write a batch file and run it instead of the "/d /c .... etc...."
Title: lossyWAV 1.2.0 Development Thread
Post by: sidewalking on 2009-01-16 05:28:42

Well I have tried and failed...I haven't pinned down the right sequence yet...

What I have used so far...

Code: [Select]
/d /c C:\"Program Files (x86)"\foobar2000\codec\lossywav - --portable --silent --stdout | "C:\Program Files (x86)\Windows Media Components\Encoder\WMCmd.vbs" -silent -a_codec WMA9LSL -a_mode 2 -a_setting Q100_44_2_16 -input %s -output %d


Problem is it gives back a WSHShell / undefined variable...probably has to do something with the piping or some other little variable.

I will have to stick with the workaround method I have setup at the moment.  Convert to Lossy Portable FLAC and then convert that to WMA-Lossless.

I will keep trying every once in a while to try to figure some tweak or some trick to get it to work as a command line.


Cscript.exe is required to run .vbs files.  I believe it is in C:\Windows\System32\ and put the .vbs as an argument after it with full path in quotes.

I can't get it to work, either.  I can get the original WAV to encode to WMA Lossless, but I can't get all three processes into the converter string to work.  Maybe if there were a batch to call from foobar to handle the lossyWAV conversion and then input that file into the WMALS encoder for a lossyWMALSL file, not just a WWMA Lossless file.

The wiki code here doesn't work, even if you allow for a cscript.exe prompt first:

Code: [Select]
Encoder: c:\program files\windows media components\encoder\WMCmd.vbs
Extension : lossy.wma
Parameters: -input %s -output %d -a_codec WMA9LSL -a_mode 0 -a_setting Q100_44_2_16
Format is : lossless or hybrid
Highest BPS mode supported: 24


It addresses the encoding of a regular WAV file, but doesn't include the lossyWAV processing.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-16 08:54:40
It addresses the encoding of a regular WAV file, but doesn't include the lossyWAV processing.
You're right, of course - I'll work up a batch file to carry out the lossyWAV processing then encode to WMALSL.
Title: lossyWAV 1.2.0 Development Thread
Post by: UED77 on 2009-01-18 02:33:23
You're right, of course - I'll work up a batch file to carry out the lossyWAV processing then encode to WMALSL.


Actually, a batch file would be a good idea for the popular formats: FLAC, TAK, WavPack, and WMALSL. That would ensure that the compressors are called with the proper blocksizes, and could also place a custom tag in the file that would replace the current (and unwieldy) FACT chunk. 
Title: lossyWAV 1.2.0 Development Thread
Post by: sidewalking on 2009-01-23 03:36:49
The only thing I'd like to see is a nice GUI front end - I've mentioned that a couple of times before. I don't mean to go on about it but it would make life a bit easier for me as a non-techie.

I am working on something similar to the Speek/Flac front ends right now.  It won't be fancy, but will relieve the CLI anxiety that some people get (and cause them to pass over or check out newer things).

I agree that Foobar is going to be the most useful, but if there is a need for a basic front end like the ones mentioned - I will throw something together.

(...and I will integrate Flac and probably WavPack, since I am a WavPack junkie).
Title: lossyWAV 1.2.0 Development Thread
Post by: Rigapada on 2009-01-23 05:40:39
Quote
I am working on something similar to the Speek/Flac front ends right now.  It won't be fancy, but will relieve the CLI anxiety that some people get (and cause them to pass over or check out newer things).

I agree that Foobar is going to be the most useful, but if there is a need for a basic front end like the ones mentioned - I will throw something together.

(...and I will integrate Flac and probably WavPack, since I am a WavPack junkie).


I have a small number of folders of Monkeys audio and Flac files, carefully ripped and tagged. I would like to convert them to Lossy Tak files with tags. Can your program help me do it in a single step?

--Rigapada
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-23 07:25:09
.... that would replace the current (and unwieldy) FACT chunk. 
Unfortunately, the FACT chunk is a necessary evil - if the user creates correction files when processing and wishes to revert to lossless at any point. The FACT chunks of the .lossy.wav and .lwcdf.wav files are compared and, if identical, the audio data is merged to reinstate the lossless original.

However, when using piped encoding, the FACT chunk cannot be used (and neither can correction files be created) as FLAC (for one) does not accept additional chunks when piping input.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-01-23 09:22:54

The only thing I'd like to see is a nice GUI front end - I've mentioned that a couple of times before. I don't mean to go on about it but it would make life a bit easier for me as a non-techie.

I am working on something similar to the Speek/Flac front ends right now.  It won't be fancy, but will relieve the CLI anxiety that some people get (and cause them to pass over or check out newer things).

I agree that Foobar is going to be the most useful, but if there is a need for a basic front end like the ones mentioned - I will throw something together.

(...and I will integrate Flac and probably WavPack, since I am a WavPack junkie).

That's jolly decent of you,old chap!

Actually, I've been using Foobar and am quite happy to do my conversions that way - or directly from EAC. But I think you're right in that it might encourage people even less techie than me to use LossyWav.

If you want any testing done just give me a shout

Thinking of releasing a 1.2.0 soon - although obviously without the adaptive noise shaping method (a very interesting, complex and baffling proposition that will remain firmly on the to-do list).

Nick, if the ideal noise-shaping is proving troublesome is there any mileage in introducing a simpler noise-shaping method as a short term option? I have absolutely no idea whether that's a viable proposition but if it were it might make LossyWav useable at lower "q" settings and hence a viable alternative to lossy codecs at the higher end of their range.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-23 10:56:57
There is noise shaping already implemented in lossyWAV (at v1.1.0). This uses SG's published cofficients which are optimised for 44.1kHz and 48kHz.

As an aside, I've been continuing my development of the spreading function - I will post a beta v1.1.1f later today.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-01-23 14:23:22
There is noise shaping already implemented in lossyWAV (at v1.1.0). This uses SG's published cofficients which are optimised for 44.1kHz and 48kHz.

As an aside, I've been continuing my development of the spreading function - I will post a beta v1.1.1f later today.

Nick,
      Sorry. Must've missed that in the change log
Title: lossyWAV 1.2.0 Development Thread
Post by: sidewalking on 2009-01-23 17:01:13
That's jolly decent of you,old chap!

Actually, I've been using Foobar and am quite happy to do my conversions that way - or directly from EAC. But I think you're right in that it might encourage people even less techie than me to use LossyWav.

If you want any testing done just give me a shout

No worries, it isn't much to write home about right now.  I am just learning programming and am not that good with batch files, but using examples of the Flac/Speek frontend as inspiration, as well as the batch file examples here, I am able to get something basic that will write and execute the batch file for processing.  I can handle a front end that will write the files, but I may need help from others on what the batch files should most efficiently contain.

You are more than welcome to help test - I would appreciate the help!  Thanks for offering.

I have a small number of folders of Monkeys audio and Flac files, carefully ripped and tagged. I would like to convert them to Lossy Tak files with tags. Can your program help me do it in a single step?

I can handle the input/output, but so far haven't gotten to tagging.  With some more time and research I am sure I can figure it out.  Just give me some time and I will tackle it as soon as I can.


Nick.C:  Is an option to specify output file name in the plans for the future, or do you want to stay with just the output directory option?
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2009-01-23 22:35:44
I have a small number of folders of Monkeys audio and Flac files, carefully ripped and tagged. I would like to convert them to Lossy Tak files with tags. Can your program help me do it in a single step?

--Rigapada

CUETools (http://www.hydrogenaudio.org/forums/index.php?showtopic=66233) might help, although it's not exactly designed for that, so there are some limitations, e.g. files must be CDDA (44.1khz 16bit stereo). If your files are not accompanied by .cue sheets, use 'CUESheet creator' button first.

Then turn on the LossyWAV checkbox, select the desired codec, gaps appended cue style, press 'batch' button and browse to the desired folder.
Title: lossyWAV 1.2.0 Development Thread
Post by: UED77 on 2009-01-23 22:45:33
Unfortunately, the FACT chunk is a necessary evil - if the user creates correction files when processing and wishes to revert to lossless at any point. The FACT chunks of the .lossy.wav and .lwcdf.wav files are compared and, if identical, the audio data is merged to reinstate the lossless original.


Yes, I understand the need for the FACT chunk (see this post of yours (http://www.hydrogenaudio.org/forums/index.php?showtopic=56129&st=650&p=535138&#entry535138))    but I'm thinking there could be a way of noting in some tag that the file was processed by LossyWAV.

The FACT chunk should still be added to the lossyWAV-processed WAV file.
But upon losslessly compressing the lossyWAV, perhaps add a tag to the output file that contains the same information as the FACT chunk would. Of course, there is nothing to stop the user from removing this tag later, but that's his fault, so to speak.

Of course, it could be more trouble than it's worth, and ultimately you never know how "lossless original" a PCM file is, but if the technical capability exists, we might decide to let the user know in good faith that his file is no longer "pristinely original".
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-01-24 08:21:18
Hi,

Is it possible yet to use lossywav with multichannel lossless Dolby True HD & DTS Master ?
I haven't tried personnaly but I am more & more interested in it ... I am specially scarred by this:

1.2.0: Checking of S (=L-R) channel for matrix surround content

should i wait for 1.2.0 to give it a try ?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-24 08:58:31
Nick.C:  Is an option to specify output file name in the plans for the future, or do you want to stay with just the output directory option?
It was never an intention as I prefer to automatically have the output file have the .lossy.wav extension. However with the advent of piped encoding it might be an idea. I'll have a think about it.

Yes, I understand the need for the FACT chunk (see this post of yours (http://www.hydrogenaudio.org/forums/index.php?showtopic=56129&st=650&p=535138&#entry535138))    but I'm thinking there could be a way of noting in some tag that the file was processed by LossyWAV.

The FACT chunk should still be added to the lossyWAV-processed WAV file.
But upon losslessly compressing the lossyWAV, perhaps add a tag to the output file that contains the same information as the FACT chunk would. Of course, there is nothing to stop the user from removing this tag later, but that's his fault, so to speak.

Of course, it could be more trouble than it's worth, and ultimately you never know how "lossless original" a PCM file is, but if the technical capability exists, we might decide to let the user know in good faith that his file is no longer "pristinely original".
If the FACT chunk is stored as a tag on the file instead then it can more easily (i.e. deliberately) be removed. The FACT chunk can of course be removed by simply transcoding a lossyFLAC file to FLAC without keeping the additional metadata.

Is it possible yet to use lossywav with multichannel lossless Dolby True HD & DTS Master ?
I haven't tried personnaly but I am more & more interested in it ... I am specially scarred by this:

1.2.0: Checking of S (=L-R) channel for matrix surround content

should i wait for 1.2.0 to give it a try ?
I would imagine that if any Dolby True HD or DTS Master content was contained within an integer multi-channel WAV file then lossyWAV would be able to cope with it.

I have made no progress with matrix surround content due to a total lack of information as to how it works.

Obviously no beta last night - time availability problems - should be some time next week.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-01-24 09:25:02
ok np, I asked because nowadays a wonderfull soft called eac3to demux & transcode these streams directly from blu-ray to flac ... it already strips useless 0 bytes from 24bits to 16bits automatically so I had the idea of trying lossywav on it to do various bitdepth directly but I didn't knew if it would work.

One day I would like to try to put together these streams (x264 1080p+lossyflac 5.1+sup) in an mkv, it could rock !
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-28 07:47:43
lossyWAV 1.1.2 attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2009-01-28 12:19:21
lossyWAV 1.1.2 attached to post #1 in this thread.

Am i the only one who's getting an error trying to download it?
Seems like the new forum engine has some unclear new rules concerning attachments.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-28 13:10:56
I have managed to download both of them - however those were the first download for each, so maybe others are indeed having issues.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-01-28 13:24:51
lossyWAV 1.1.2 attached to post #1 in this thread.

Am i the only one who's getting an error trying to download it?
Seems like the new forum engine has some unclear new rules concerning attachments.

I haven't been able to download either. I used the email link to report a problem and that bounced back as an invalid address. As a final attempt to report the problem I PM'd Garf on the asumption that it's to do with the server upgrade. I notice that the number of downloads is "1" so it looks like Nick is the only person to have any luck so far (not that I have any idea how many people may have tried)
Title: lossyWAV 1.2.0 Development Thread
Post by: Dynamic on 2009-01-28 18:50:14
I haven't been able to download either.


It worked for me with Flashgot in Firefox 3.x and thew downloads was already up to 12, so it appears to be resolved.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-01-28 19:05:25
I haven't been able to download either.


It worked for me with Flashgot in Firefox 3.x and thew downloads was already up to 12, so it appears to be resolved.

Yep, I've got it now.

Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-01-28 19:08:04
lossyWAV 1.1.2 attached to post #1 in this thread.

Well done Nick. Another smooth release. It's working a treat
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-01-29 04:21:10
Some questions:
  • Do we need dither?
  • Do we need 32-bit integer processing?
  • Do we need the capability to create correction files?
I ask as these all add to the time taken to process files (even if the options themselves are not selected).


Hello.

Thank you for the great idea & Work youre doing here, I have downloaded 1.1.1 and did some tests but first:
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-29 06:09:56
lossyWAV does indeed pipe output so there is no need for an intermediate WAV file when encoding to a lossless codec which accepts piped input.

--insane (I assume) is the upper quality limit (-q 10). Please supply ABX logs. And samples which you were able to ABX.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-01-29 07:43:31
... About quality ,I did a few blind tests here ,& could tell the difference, ( with 90% success), ...

What was your lossyWAV setting? As Nick.C wrote samples which you can ABX 9/10 or similar are of very high interest. Please share them.
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-01-29 13:45:06
Since v1.1x I cannot run lossywav with WINE, v1.0 works:

@p550:~/downloads/win32$ wine lossywav
wine: Unhandled exception 0x0eedfade at address 0x0000:0x7b845c60 (thread 002d), starting debugger...
First chance exception: 0xc0000025 in 32-bit code (0x7bc3c91e).
Register dump:
CS:0073 SS:007b DS:007b ES:007b FS:0033 GS:003b
EIP:7bc3c91e ESP:0032fa34 EBP:0032fa98 EFLAGS:00000282(  - 00      - -IS1)
EAX:0032fa40 EBX:7bc8f4c4 ECX:00110054 EDX:00000000
ESI:0032fe20 EDI:0032faa4

Backtrace:
=>0 0x7bc3c91e __regs_RtlRaiseException+0x4e() in ntdll (0x0032fa98)
  1 0x7bc7d79a in ntdll (+0x6d79a) (0x0032fdfc)
  2 0x7bc3badc RtlUnwind() in ntdll (0x0032fe78)
  3 0x0041629b in lossywav (+0x1629b) (0x0032fec0)
  4 0x00404c4f in lossywav (+0x4c4f) (0x0032fee4)
  5 0x00404cb7 in lossywav (+0x4cb7) (0x0032ff08)
  6 0x7b878b58 in kernel32 (+0x58b58) (0x0032ffe8)
  7 0xb7e35b07 wine_switch_to_stack+0x17() in libwine.so.1 (0x00000000)
Runtime error 216 at 00ADC248
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-01-29 19:48:54
lossyWAV does indeed pipe output so there is no need for an intermediate WAV file when encoding to a lossless codec which accepts piped input.

--insane (I assume) is the upper quality limit (-q 10). Please supply ABX logs. And samples which you were able to ABX.


yeah I guessed this ability exists but I failed to figure it out ,
was eager to try this .

If anyone can tell me what's the proper way of doing it that is acceptable ,
please lmk.

I want to make sure my output files are of the same standard / quality as yours.

Also - if there is an agreed upon ABX test method here , please direct me to it so I can help.

My background is in Pro Audio Retail , so have access to different testing Hardware & environments , so I think I need to first Line up with the methods used here ,
If anyone can advise Id be grateful.

Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-01-29 19:54:04
... About quality ,I did a few blind tests here ,& could tell the difference, ( with 90% success), ...

What was your lossyWAV setting? As Nick.C wrote samples which you can ABX 9/10 or similar are of very high interest. Please share them.


This format Is new for me, So the only switch I used was "-I".
As I said if there's a 'Correct' or better settings I need to use, please advise.

once I line up with the acceptable standard methods here, I will supply my Test files if theyre needed.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-29 19:55:16
-I == --insane == -q 10 = the highest available quality setting.

If you go to post #1 in this thread you will find the foobar2000 converter settings for lossyWAV processing piped to FLAC, Tak & Wavpack.

foobar2000 also has an ABX capability.

Others are more knowledgeable regarding acceptable ABX methodology.

If you could upload some representative samples (up to 30 seconds each) of the track(s) which you managed to ABX in the Uploads forum it would be greatly appreciated.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-01-29 20:02:29
-I == --insane == -q 10 = the highest available quality setting.

If you go to post #1 in this thread you will find the foobar2000 converter settings for lossyWAV processing piped to FLAC, Tak & Wavpack.

foobar2000 also has an ABX capability.

Others are more knowledgeable regarding acceptable ABX methodology.

If you could upload some representative samples (up to 30 seconds each) of the track(s) which you managed to ABX in the Uploads forum it would be greatly appreciated.


Thanks Nick C.
will try & use it ,( I do not use Foobar , but will delve into it.)
I will look for the Upload Forums & upload files as you request. (did not know they exist).
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-01-29 20:34:11
... the only switch I used was "-I". ...

'-I' means 'insane quality' (the best quality available, 2 steps better than 'standard quality' ['-S'] which so far was considered transparent).
So your sample(s) are of very high interest, cause if you can really ABX the -I results it means there has more development to be done.

As Nick.C wrote the probably easiest way to do ABXing is to use foobar2000. Just take care when installing to have the ABX tool marked.
You add the two files to be abxed to the playlist, mark them both, use the context menu 'Utils' > 'ABX Two Tracks'.
In the dialogue that pops up you can listen to the two files by pressing the 'A' resp. 'B' button.
Listen to them several times in order to get familiar with the supposed differences.
Then start the blind test which means pressing the 'X' and 'Y' button. 'X' can be 'A' (in whch case 'Y' is 'B') or vice versa.
You are allowed to press the 'X', 'Y', 'A' and 'B' button as often as you like. Finally you have an opinion whether 'X is A, Y is B' or 'Y is A, X is B'.
You press the corresponding button. At this time you can still change your mind in case you pressed the wrong button.
You finish this trial by pressing 'Next Trial' which will show you your result and brings you the next trial, that is a new 'A' or 'B' mapping to 'X' resp. 'Y'.
You continue until you finisghed 10 trials.

Someone who is only guessing will arive at a total score of ~5/10 (5 hits out of 10). So in order to have really identified a problem your score should be a lot better.
If you arrive at 10/10 you have defintely abxed a problem. When your score is 9/10 probality still is very high that there is an issue. A result of 8/10 still yields evidence for
this though the result cannot totally be trusted. In this case (and I'd say also in the case of a 7/10 result) best thing to do is to do more ABX tests. In case you always get at a 7/10
or better result evidence is very strong that there is an issue.
Opinion about the right way to do ABXing varies, but in order to get a strong evidence for lossyWAV insane being not transparent IMO this should do.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-29 20:37:16
.... or, in the event of 7/10 or 8/10, try dropping down to --extreme or even --standard and ABX that output against the original instead.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-01-29 20:58:33
Thanks for the informative replies !
I will get to it , this should prove to be a more reliable test !
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-01-29 22:54:09
Thanks for the informative replies !
I will get to it , this should prove to be a more reliable test !


OK Finally set it up , the Foobar conversion did not work for me ,
but I have found the script version.

now trying to get to the ABX test ,  Is A always A & only mapping to X Y changes >?
or the whole mapping changes ?? (turning it into an ABCD test)
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-01-30 00:01:43
nm took me a while (new to this Foobar app ..)
kinda figured this out now , ran a 1st 20 try test here, for the 1st (hardest for me)
single test file, so fatigue will become a factor as well, log included in archive.

Didnt continue testing the 2 others with Foobar as my 1st test file provided enough input for me,have a look.
(& I will be the first to say my ears are not what they used to be :/ )

put the file on rapidshare here:
http://rapidshare.com/files/191350600/TEST_FILES.rar.html (http://rapidshare.com/files/191350600/TEST_FILES.rar.html)
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-01-30 04:54:54
Very interesting. I can't test lossywav ATM, but I failed with wavpack lossy --dns  256k (hx). abx 6/8 then 3/8
I concentrated on the cymbal crash. I had the feeling of a slight hf noise on the 1st abx run but its a bit noisy in my apartment.


@bork - The problem should be magnified with lower quality setting like --portable - Do you mind testing ?
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-01-30 08:13:13
... my 1st test file provided enough input ...

Really, especially as you were using '-I', didn`t you?
What come to my mind is that '-I' uses noise shaping to the max, meaning all the added noise is collected in the 13+ kHz range.
This is not expected to be a problem, but who knows for sure?
Do you mind testing with the additional switch '-s 0' which disables noise shaping, or a low noise shaping setting like '-s 0.3'?
As '-S' is wanted to yield transparent results I'd prefer to have this setting tested. shadowking's proposal of using '-P' is of course intersting too. As a positive side effect for you using '-S' and '-P' testing should be easier.

Can you tell a bit about what is wrong with the lossyWAV result?
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-01-30 23:36:39
... my 1st test file provided enough input ...

Really, especially as you were using '-I', didn`t you?
What come to my mind is that '-I' uses noise shaping to the max, meaning all the added noise is collected in the 13+ kHz range.
This is not expected to be a problem, but who knows for sure?
Do you mind testing with the additional switch '-s 0' which disables noise shaping, or a low noise shaping setting like '-s 0.3'?
As '-S' is wanted to yield transparent results I'd prefer to have this setting tested. shadowking's proposal of using '-P' is of course intersting too. As a positive side effect for you using '-S' and '-P' testing should be easier.

Can you tell a bit about what is wrong with the lossyWAV result?


halb27 - Thanks again for helping me set this up , would not be able to without your help.

As I hinted above , Im the first to say that I do NOT have Reference ears anymore.
On the other hand , I have been a musician most of my life, & have had  experience with sound systems of different levels throughout, (& a music fan of course).

with the last test (log included with the test files above)
I used the script Nick C. provided that uses --standard.

I cannot tell what's wrong with lossywav itself,
but I can highlight key points in my uploaded test files, that helped me tell them apart.


In the track called 'Country Roads' , what gave it away for me was noise.
Once locked in on it ,you can notice it quite clearly , until fatigue gets you (me).

on The 'Rock With You' Track what gave it away for me was the reduced overtones in the ambience of the lead synth line.

On The 'Oleo' track , the one with the log included in the test files,
it was the  midrange thickness of the organ.

Sharp-eared experienced users (with some level of decent  gear),
(not to mention experienced pros /true Audiophiles)
should have a way higher percentage of noticing the differences,
more then my somewhat tired although experienced ears can hope for.

Here is an example:

I saved the log for 'Rock With You' - one look is enough to tell when fatigue has set in on me ,(or maybe i lost my focus ?!) & how long it took me to snap out of it ,
but the results still have meaning.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/01/30 03:22:51

File A: F:\MP3\Favourite\TEST FILES\Michael Jackson - Rock With You.flac
File B: F:\MP3\Favourite\TEST FILES\Michael Jackson - Rock With You.lossy.flac

03:22:51 : Test started.
03:23:54 : 01/01  50.0%
03:24:27 : 02/02  25.0%
03:25:55 : 03/03  12.5%
03:26:43 : 04/04  6.3%
03:27:25 : 05/05  3.1%
03:27:59 : 06/06  1.6%
03:29:53 : 06/07  6.3%
03:32:13 : 07/08  3.5%
03:33:25 : 08/09  2.0%
03:34:21 : 09/10  1.1%
03:35:03 : 10/11  0.6%
03:36:10 : 11/12  0.3%
03:37:25 : 11/13  1.1%
03:39:02 : 12/14  0.6%
03:41:12 : 12/15  1.8%
03:41:57 : 12/16  3.8%
03:43:20 : 12/17  7.2%
03:44:31 : 12/18  11.9%
03:46:05 : 13/19  8.4%
03:47:28 : 13/20  13.2%
03:49:05 : 14/21  9.5%
03:49:28 : 15/22  6.7%
03:50:07 : 16/23  4.7%
03:51:37 : 17/24  3.2%
03:52:22 : 17/25  5.4%
03:52:31 : 17/26  8.4%
03:53:15 : 18/27  6.1%
03:54:34 : 19/28  4.4%
03:55:42 : 20/29  3.1%
03:57:12 : 20/30  4.9%
03:57:41 : 21/31  3.5%
04:00:12 : 21/32  5.5%
04:01:43 : 21/33  8.1%
04:01:57 : 22/34  6.1%
04:04:18 : 23/35  4.5%
04:05:48 : 23/36  6.6%
04:07:36 : 23/37  9.4%
04:09:56 : 24/38  7.2%
04:10:49 : 25/39  5.4%
04:11:55 : 26/40  4.0%
04:12:04 : Test finished.

----------
Total: 26/40 (4.0%)
Title: lossyWAV 1.2.0 Development Thread
Post by: singaiya on 2009-01-31 20:06:10
Wow, nice going Bork. I'm downloading the files and will see if I can ABX one of them.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-01-31 21:05:43
As I hinted above , Im the first to say that I do NOT have Reference ears anymore. ...

A 9/10 result for standard quality is real impressive, so you don't have to blame your ears. You are the first to have found an issue with standard.
Thanks again for your test.
What's most frightening is that you come up even with three samples and various kind of problems as it looks.
I've tried the sample you abxed first and could not find an issue, but this tells us nothing because I've caught a cold and my hearing is suffering.
Thanks for your problem description. Can you add some hints about the spots (second range) of your tracks where the problems are most obvious to you?

Do you mind trying with noise shaping switched off? To do so you add -s 0 after -S or --standard.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-01-31 23:01:38
Oh well, back to the drawing board  .

B0rk, thanks for the extensive ABXing and provision of samples.

As halb27 requested, could you please try --standard with --shaping 0 (-s 0) to attempt to determine whether the additional noise at high frequencies due to noise shaping is part of the problem.

Thanks again,

Nick.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-01-31 23:23:51
Due to exams I do not currently have the time to test the sample but I downloaded the files package & I will test it within a month I hope.
The transparency of portable is of the highest importance for me. I will not let anyone tell that standard is not transparent if I am not 100% sure of his ABXing skills.
standard is a very high quality level IMHO so I need to test by myself. For me, it's either a big problem always hearable by golden heared people or a fake.
I am not telling you are lying, but in this forum the only guy telling me that standard is not transparent that I would blindly trust is Guruboolez.
I am just shocked by a guy that don't know how to ABX but ABX everything at first try, even a musician.
That said, your logs are impressive.
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-02-01 00:11:36
We need near perfect abx result for such high qulity levels  pval < 3 % even though Bork's abx results are statisticaly valid.

If there is a quality issue then [-P] should be abxable with very high confidence 8/8 or 14/16 - easier than -standard.
If this isn't happening there is likely some other problem not related to lossywav.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-01 08:34:35
There is a variation between --portable and  --standard - the shaping factor increases from 0.25 (-P) to 0.50 (-S). If this issue is shaping noise related then changing from --standard to --portable is changing more than one "variable". Using --portable --shaping 0.5 would be consistent.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-02 06:19:23
I will not let anyone tell that standard is not transparent if I am not 100% sure of his ABXing skills.
standard is a very high quality level IMHO so I need to test by myself. For me, it's either a big problem always hearable by golden heared people or a fake.
I am not telling you are lying, but in this forum the only guy telling me that standard is not transparent that I would blindly trust is Guruboolez.
I am just shocked by a guy that don't know how to ABX but ABX everything at first try, even a musician.
That said, your logs are impressive.


Hello again.
I agree , TBH I doubted Ill get any results with this test,
& definitley had to focus to do it , not to mention I was not used to foobar's method,
but Halb27 helped me with it.

Please let me clarify a few things about myself.

I am here because I saw The LossyWav project - & just had to try it, & thank the author for the great work.

I am thankful for the development of it , and would love to see it take off.
My interest in it is very selfish - I have a big music collection,
and I just do not have the space for it all - & this project sounds like the only hope.

I am not a naysayer.

Now about me being new to ABXing ...
nothing can be further from the truth ..
what I AM new to is abxing using apps like Foobar.
(I was (& still am) scared that ABX tests can have LOGS lol).

To make a long boring story short, I have been in music basically all my life & I make
my living from music , in more then one form.

so please ,do not mistake me for a beginner  .. I never said I was.
I am sorry I have not been clearer on my background, I can see now how it might have looked ..
in fact Im pretty shocked my first tests were that 'positive',was not sure theyll show anything that I felt & was not all that successful later on a less well known (to me) tune as youll see later down this post.

while anything can be faked , Im here for the right reasons,
and since 35 is in my rear view mirror , I think am pretty much done with pissing contests.

More testing - Im totally with you on this.

we cannot trust any single person's tests.
I don't trust these ears 50% of the time nowadays - but I sure as hell used to, & they put food on my table & still do.

so getting a Pro Tester base for this should probably high up on the priority list.
& I do not mean myself.

As I hinted above , Im the first to say that I do NOT have Reference ears anymore. ...

A 9/10 result for standard quality is real impressive, so you don't have to blame your ears. You are the first to have found an issue with standard.
Thanks again for your test.
What's most frightening is that you come up even with three samples and various kind of problems as it looks.
I've tried the sample you abxed first and could not find an issue, but this tells us nothing because I've caught a cold and my hearing is suffering.
Thanks for your problem description. Can you add some hints about the spots (second range) of your tracks where the problems are most obvious to you?

Do you mind trying with noise shaping switched off? To do so you add -s 0 after -S or --standard.


Halb27 - Hello again.
Sure Ill try to find the time to do that as well.

Although These were 'good' results , I do KNOW these tracks VERY well,
(& beginners luck with foobar has probably helped ..)


Ill try to explain The spots (If I got your question), but first how I did it.
had a problem getting getting over how confusing that Foobr test is.
didnt quite get it - seemed to me like an ABCD thing.

first I to narrowed it down to first NOT PLAYING the B button for example,
mostly it helped me label the A button as Reference.

although as you'll see in a bit it failed me as well, for me it's simpler.

The other thing that helped me is to set the start and end for the tracks to quite short lengths 3 seconds max I think , or Medium (actually that's as long as Ill ever try with this ABX tests) - 10 seconds max.

Then I made sure I stop (have silence) between the longer bits,
and no gaps or very short ones on the shorter ones.

I made sure I do not move my head at all,
as the slightest movement will cause phase shifts, amplitude differences, etc etc.
eveytime I move - I fuckup as this AB test is not about feeling , it's about standing on your toes & very alert ( Yep I pretty much hate these).

finally , there comes a time when your focus just escapes you, you have ear fatigue,
or just got burned out on this specific slice of music - time to move on to a new slice,
because at this point - there is just no way I can tell anything , as my brain says Im bored with this shit - and just shuts down on me.

Here's another MJ song I tried later , Off The wall, an older song that I Do not know as well - which was a pretty miserable experience:

I'll walk you through it :

foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/01/31 05:20:11

File A: F:\MP3\Favourite\TEST FILES\1mine TRIM.flac
File B: F:\MP3\Favourite\TEST FILES\1mine TRIM.lossy.flac

Locked in on a short slice - Bulls Eye ... erm NOT
05:20:11 : Test started.
05:20:45 : 00/01  100.0%

turns out Id better be launching only the B button , as it seems I somehow related to it more clearly (?!), welcome to Vegas

05:21:29 : 01/02  75.0%
05:21:46 : 02/03  50.0%
05:22:14 : 03/04  31.3%
05:22:36 : 04/05  18.8%
05:22:56 : 05/06  10.9%
05:23:26 : 06/07  6.3%

So far so good 6 bulls eyes - 2 minutes & by this time I start drifting
05:23:43 : 06/08  14.5%
Nope ..
05:24:23 : 07/09  9.0%
Aye
05:24:52 : 07/10  17.2%
slight recovery
05:25:53 : 08/11  11.3%
Nope !:::::::: I am DEAD here & it's getting worse
05:26:27 : 08/12  19.4%
05:27:12 : 08/13  29.1%
05:28:02 : 08/14  39.5%
Trigger A , as Im Fucked , let's try to lock on it
05:28:21 : 09/15  30.4%
05:28:54 : 10/16  22.7%
05:29:40 : 11/17  16.6%
05:30:51 : 12/18  11.9%
switched to another  slice
05:31:36 : 12/19  18.0%
05:32:48 : 13/20  13.2%
05:34:16 : 13/21  19.2%
05:34:46 : 14/22  14.3%
05:35:46 : 15/23  10.5%
Focus lasts for 4 mins - with longer pauses then goes away
05:36:07 : 15/24  15.4%
05:36:16 : 15/25  21.2%
05:37:17 : 15/26  27.9%
05:37:50 : 15/27  35.1%
short break - Fresh Off on it once again
05:39:21 : 16/28  28.6%
05:41:00 : 17/29  22.9%
05:42:45 : 18/30  18.1%
05:43:50 : 19/31  14.1%
05:44:26 : 20/32  10.8%
05:45:37 : 21/33  8.1%
05:46:04 : 22/34  6.1%
This 6 minutes spree couldnt last now could it
05:47:07 : 22/35  8.8%
05:49:11 : 23/36  6.6%
05:50:07 : 23/37  9.4%
05:51:28 : 23/38  12.8%
05:52:24 : 23/39  16.8%
Last Hard Effort
05:53:38 : 24/40  13.4%
05:55:25 : 25/41  10.6%
05:56:24 : 25/42  14.0%
05:58:03 : 26/43  11.1%
Should have really stopped here ..
05:59:24 : 26/44  14.6%
06:01:28 : 26/45  18.6%
06:02:42 : 26/46  23.1%
06:06:29 : 26/47  28.0%
06:07:40 : 27/48  23.5%
06:08:44 : 27/49  28.4%
06:09:46 : 28/50  24.0%
THAT's IT - Im OUT !
06:11:40 : Test finished.
----------
Total: 28/50 (24.0%)

By the end of this - I want to seriously hurt someone.
So some tests can also end this way ...
maybe this result has some value as well.

I feel these kind of tests are so hard on you mentally ,
& maybe even need some strategy  ...
so maybe they do not reflect the truth as well.

The way I see sound was always like I see shoes or a car.
You gotta walk in em for awhile , before you know that something's not as comfy as it should have been.

But this approach will never work on any AB tests, & cannot be 'proven'.

Nick C. please carry on with this great project !
I was & still am very excited about it !

I have some pro audio disciples with younger ears that I can try & test,
if that's welcome/needed at all please lmk.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-02 07:38:53
Thanks again for another round of testing. Would it be reasonable to say that although not transparent, --standard is at least difficult to ABX?

I would appreciate very much if you could try one of these samples having been processed with --shaping 0 appended to the lossyWAV command line. This would allow a clearer determination of whether the noise shaping method is involved in the differences that you are hearing.

[edit] For reference - David Robinson (2Bdecided here) is the author of the method, my implementation of lossyWAV is just that. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-02 08:47:56
... By the end of this - I want to seriously hurt someone.
So some tests can also end this way ...
maybe this result has some value as well. ...

You describe very well the pain of ABXing.

While ABXing you concentrated on a few second range. Can you please tell us the second range (hopefully a narrow range) for the 3 tracks you ABXed successfully?

The background for the question is this:
While nobody doubts your ABX results it would be good if they were confirmed by at least one other independent tester. It makes a difference if a problem is heard only by a golden eared specialist on tracks he knows very well or if there is a problem, even if subtle, which can be heard by several people.

With this in mind it is helpful to concentrate on your three existing problem tracks, help us confirm them, and help finding the source of the problem by trying encoding variations.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-02 08:47:56
Sorry, post was doubled.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-02 20:52:35
Thanks again for another round of testing. Would it be reasonable to say that although not transparent, --standard is at least difficult to ABX?

I would appreciate very much if you could try one of these samples having been processed with --shaping 0 appended to the lossyWAV command line. This would allow a clearer determination of whether the noise shaping method is involved in the differences that you are hearing.


I have not done enough testing to have enough confidence in my ability to tell the difference betwen the modes themselves in relation to the differences being more or less audible, (& I doubt I will ,maybe you will need people with the highest caliber of gear & experience for that.)

I did learn however that in these short tests I do get better results if i really KNOW a piece of music , so my full attention is to the Sound & not the Music.

I will try and test with these parameters,
so the full parameters will be --standard --shaping 0 ?


@ halb27 :
not sure as I did not have exact pin points locations,  it might turn out to be different in a different day , maybe I should keep note of it , wish foobar could do that for me.

There's just so many variables that can change from one tester to another.
For example if your ears/speakers have a good range in the highs,
then a ride cymbal  evolving into a crash cymbal for example ,
will show as more pronounced sometimes (distortion ..) at it's lower highs (10khz-14khz) , but if you use speakers that have full bodied midrange to low range for example, a loss of weight on some sounds can be your cue.

I find that for me,  NOT starting any selection in a sharp attack truly helps me focus instead of just recover from the the aftershock , only to miss the differences during that time.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-02 21:05:14
Yes, I would be very interested to learn if there is a perceptible difference, i.e. is --standard --shaping 0 easier or harder to ABX than --standard.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-02 22:15:59
Yes, I would be very interested to learn if there is a perceptible difference, i.e. is --standard --shaping 0 easier or harder to ABX than --standard.


Got it , will try & report.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-02 22:22:37
Many thanks!
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-03 11:58:25
Nick,

I very much doubt this is the reason for the ABX results, but the noise shaping isn't working quite right. Sorry for not looking at this before...

When you switch the number of bits to remove, the noise shaping breaks at the block boundary - you get a non-noise-shaped click. I was going to post a picture of the CEP spectral view of the difference signal, but I can't see the "attach" box in the new forum view!

The click is still much quieter than what you'd get without noise shaping - and even without noise shaping, we'd assumed content at that level was inaudible - but it would still be better not to have the click.

To my ears, the added noise with noise shaping is far far lower than the added noise without (as it should be!). Of course moving from --insane --shaping 0 to --standard --shaping 0 makes the noise even higher. (Still inaudible to me, though I'm not in a position to play the tracks at a decent level to make sure!).

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-03 13:24:50
I think I understand what you mean - although, how would the noise shaping be "continued" across codec-blocks given that they may have different bits-to-remove values?

Could it be as simple as saving the last N samples / fifo values per channel and scale the fifo values by the 2^(new_bits_to_remove - old_bits_to_remove) and feeding this into the noise shaping calculations?

Currently the first 4 fifo values are zero (which is probably causing the problem). Maybe these should be random numbers (although determining the "correct" magnitude limit might be a challenge.....).
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-03 15:17:57
Is there a problem with "continuing" it? Or, to put it another way, is there a reason to re-set it? I'm really sorry Nick I can't program, otherwise I could read your code and offer useful suggestions!

I think when I did it in MATLAB (this is a long time ago now!) it was quite easy. It depends what you're doing with the numbers - I'm not doing anything clever between blocks, just changing the number of bits to be removed (i.e. the number of bits that are being set to zero) - there's no reason to re-set the calculation though - the noise shaping will happily work across this change.

Are you doing something that makes this difficult, e.g. are you doing scaling based on the bits_to_remove, or running a noise shaping loop with the amplitude referenced/scaled to the lowest remaining non-zero bit? That would make this trickier.

I don't think you should initialise it with random numbers.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-03 18:56:00
I think that I've solved it - lossyWAV 1.1.2b attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-04 01:47:32
thanks for the update Nick.C & 2Bdecided , I will redo the test files with the new version and try some more testing.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-04 05:19:08
Got to testing after a real long day - probably was a mistake hehe ..
but had to check this out.

Did give it some serious listening & time, despite the yawns. 

First off just did some random listening & very quick testing which did not show much in my part , I could not lock onto the differences that much ,
but then I had a smoke & gave it a serious effort in two batches.

Note on the second batch I reset it after realizing Im just betting for a few tries & lost my focus , then did another 80 run to makeup for it ,so the final stats actually should be better for the 80 run session.

I did the OLEO track again - from the Test Files.

Run 1:
Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/04 05:44:11

File A: F:\MP3\Favourite\TEST FILES\Stnadard No Noise Shaping\Bob Berg - Oleo.flac
File B: F:\MP3\Favourite\TEST FILES\Stnadard No Noise Shaping\Bob Berg - Oleo.lossy.flac

05:44:11 : Test started.
05:44:28 : 01/01  50.0%
05:45:14 : 01/02  75.0%
05:45:28 : 02/03  50.0%
05:45:42 : 03/04  31.3%
05:45:52 : 04/05  18.8%
05:46:04 : 05/06  10.9%
05:47:22 : 05/07  22.7%
05:47:35 : 05/08  36.3%
05:47:46 : 06/09  25.4%
05:47:58 : 06/10  37.7%
05:48:22 : 06/11  50.0%
05:48:34 : 07/12  38.7%
05:48:40 : 08/13  29.1%
05:49:34 : 09/14  21.2%
05:50:27 : 09/15  30.4%
05:50:57 : 10/16  22.7%
05:51:34 : 10/17  31.5%
05:51:45 : 11/18  24.0%
05:52:12 : 12/19  18.0%
05:52:30 : 12/20  25.2%
05:53:44 : 13/21  19.2%
05:54:04 : 13/22  26.2%
05:54:51 : 14/23  20.2%
05:55:07 : 14/24  27.1%
05:55:40 : 15/25  21.2%
05:56:35 : 15/26  27.9%
05:57:03 : 16/27  22.1%
05:57:19 : 16/28  28.6%
05:58:25 : 16/29  35.6%
05:59:01 : 17/30  29.2%
05:59:12 : 17/31  36.0%
05:59:32 : 18/32  29.8%
06:00:03 : 19/33  24.3%
06:00:42 : 20/34  19.6%
06:01:57 : 20/35  25.0%
06:02:19 : 21/36  20.3%
06:02:47 : 22/37  16.2%
06:03:31 : 22/38  20.9%
06:04:17 : 22/39  26.1%
06:05:22 : 23/40  21.5%
06:06:29 : 23/41  26.6%
06:07:16 : 24/42  22.0%
06:08:02 : 25/43  18.0%
06:08:25 : 26/44  14.6%
06:08:54 : 27/45  11.6%
06:09:28 : 28/46  9.2%
06:09:38 : 28/47  12.1%
06:10:14 : 29/48  9.7%
06:11:24 : 30/49  7.6%
06:11:49 : 31/50  5.9%
06:11:55 : Test finished.

----------
Total: 31/50 (5.9%)


Run 2:
Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/04 06:19:09

File A: F:\MP3\Favourite\TEST FILES\Stnadard No Noise Shaping\Bob Berg - Oleo.flac
File B: F:\MP3\Favourite\TEST FILES\Stnadard No Noise Shaping\Bob Berg - Oleo.lossy.flac

06:19:09 : Test started.
06:19:25 : 01/01  50.0%
06:19:32 : 02/02  25.0%
06:19:39 : 02/03  50.0%
06:20:11 : 03/04  31.3%
06:20:46 : 03/05  50.0%
06:21:16 : 04/06  34.4%
06:21:36 : 05/07  22.7%
06:21:44 : 05/08  36.3%
06:21:57 : 05/09  50.0%
06:22:16 : 05/10  62.3%
06:22:41 : 05/11  72.6%
06:23:04 : 05/12  80.6%
06:23:26 : 05/13  86.7%
06:23:41 : 05/14  91.0%
06:23:45 : Trial reset.
06:23:57 : 01/01  50.0%
06:24:04 : 01/02  75.0%
06:24:10 : 02/03  50.0%
06:24:20 : 02/04  68.8%
06:24:51 : 02/05  81.3%
06:25:40 : 02/06  89.1%
06:25:49 : 03/07  77.3%
06:26:06 : 04/08  63.7%
06:26:20 : 05/09  50.0%
06:26:29 : 06/10  37.7%
06:26:39 : 07/11  27.4%
06:26:47 : 08/12  19.4%
06:26:57 : 08/13  29.1%
06:27:18 : 09/14  21.2%
06:27:38 : 10/15  15.1%
06:28:00 : 11/16  10.5%
06:28:20 : 12/17  7.2%
06:28:40 : 12/18  11.9%
06:28:59 : 12/19  18.0%
06:29:27 : 13/20  13.2%
06:30:35 : 14/21  9.5%
06:30:45 : 14/22  14.3%
06:31:06 : 14/23  20.2%
06:31:56 : 15/24  15.4%
06:32:09 : 15/25  21.2%
06:32:17 : 16/26  16.3%
06:32:34 : 17/27  12.4%
06:33:21 : 18/28  9.2%
06:34:22 : 18/29  13.2%
06:34:41 : 19/30  10.0%
06:35:05 : 19/31  14.1%
06:35:53 : 19/32  18.9%
06:37:10 : 20/33  14.8%
06:37:22 : 21/34  11.5%
06:37:30 : 22/35  8.8%
06:37:42 : 22/36  12.1%
06:37:54 : 23/37  9.4%
06:38:05 : 23/38  12.8%
06:38:24 : 23/39  16.8%
06:39:06 : 23/40  21.5%
06:39:22 : 23/41  26.6%
06:39:32 : 24/42  22.0%
06:39:50 : 24/43  27.1%
06:40:28 : 25/44  22.6%
06:40:40 : 25/45  27.6%
06:40:54 : 25/46  32.9%
06:41:30 : 26/47  28.0%
06:41:58 : 26/48  33.3%
06:42:40 : 27/49  28.4%
06:43:50 : 27/50  33.6%
06:44:06 : 27/51  39.0%
06:44:32 : 28/52  33.9%
06:44:50 : 29/53  29.2%
06:45:05 : 30/54  24.8%
06:45:23 : 31/55  20.9%
06:45:39 : 32/56  17.5%
06:45:57 : 32/57  21.4%
06:46:08 : 32/58  25.6%
06:46:26 : 33/59  21.7%
06:46:36 : 33/60  25.9%
06:47:10 : 34/61  22.1%
06:47:35 : 35/62  18.7%
06:47:43 : 36/63  15.7%
06:48:08 : 36/64  19.1%
06:48:47 : 37/65  16.1%
06:49:01 : 37/66  19.5%
06:49:44 : 37/67  23.2%
06:50:31 : 38/68  19.8%
06:50:46 : 39/69  16.8%
06:51:22 : 39/70  20.1%
06:51:35 : 39/71  23.8%
06:51:46 : 40/72  20.5%
06:51:58 : 41/73  17.5%
06:52:11 : 42/74  14.8%
06:52:24 : 43/75  12.4%
06:52:42 : 43/76  15.1%
06:52:50 : 43/77  18.1%
06:52:58 : 43/78  21.4%
06:54:18 : 44/79  18.4%
06:54:28 : 45/80  15.7%
06:54:38 : Test finished.

----------
Total: 50/94 (30.3%)
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-04 09:06:34
Thanks for testing lossyWAV 1.1.2b. Now things look pretty fine again. Did you use just -S resp. --standard with noise shaping as defaulted (no -s or --shaping option)?

BTW it seems that when describing how to ABX with foobar I wasn't precise enough in one point: You decide before testing how many trials you do, for instance 8 (minimum) or 10 or 16. After making up your mind the number of trials is fixed. In case you fail ABXing with the test you are free of course to do another ABX test, but when telling about your ABX results you should mention these failed ABX tests.

So far it looks like the noise shaping 'clicks' have caused the issue. Thanks, 2Bdecided, for showing up this problem, and, Nick.C, for immediate fixing.
Title: lossyWAV 1.2.0 Development Thread
Post by: carpman on 2009-02-04 09:55:26
Just out of interest will LossyWav 1.0.1.0 be affected by this issue. I've not been following recent developments and I'm not sure if this is to do with a post 1.0.1.0 noise shaping issue or if it relates to prior versions too.

Thanks and thanks to Bork, Nick, 2Bdecided and halb27 for work on this issue.

C.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-04 10:01:48
This particular problem only affects v1.1.0 onwards.

I'm waiting for David to try the revised version 1.1.2b and see if the "clicking" has been removed / modified.

In testing last night, I couldn't hear anything different. However when using ReplayGain to determine how "quiet" the correction file was the RG value for some files in my problem sample set required about 1.5dB more amplification for 1.1.2b compared to 1.1.2 - not exactly scientific but an indicator that something had changed (hopefully for the better).
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-02-04 11:03:08
Bork,
      In an earlier post you mentioned "reduced overtones" and "midrange thickness" as well as noise. Does your latest listening test and ABX results indicate that these problems have gone away too?
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-04 13:51:50
I'm waiting for David to try the revised version 1.1.2b and see if the "clicking" has been removed / modified.
Definitely different, almost certainly better, but maybe not perfect.

This was the problem...
http://www.david.robinson.org/pics/lw1.1.2_bb_-I.jpg (http://www.david.robinson.org/pics/lw1.1.2_bb_-I.jpg)
(see the vertical lines at block boundaries?)

This is 1.1.2b...
http://www.david.robinson.org/pics/lw1.1.2b_bb_-I.jpg (http://www.david.robinson.org/pics/lw1.1.2b_bb_-I.jpg)
(see most of the vertical lines have gone, but there are still a few?)

I don't think there should be any vertical lines.

The way I assume you've coded it, I bet when you transition from a block with some bits to remove, into a block with zero bits to remove, I think you'll still get a vertical line (click) - unless you carry the noise shaping on into that block. You could try that - it should hopefully have zero effect after the first four samples, but you'd need to check.

In the part shown, there are a few vertical lines even when transition between blocks which both have some bits removed. I'm not sure why that would happen.


This is really splitting hairs. I can hear the difference if I listen to the difference/correction signal at full volume, but in the presence of actual music...! No.


B0RK - please be really careful to report exactly what you're testing when you report an ABX. Exactly what software version, and exactly what settings. I don't know if you've done it yet, but -I --shaping 0 seems the most useful test to me.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-04 13:57:23
I don't think there should be any vertical lines.

The way I assume you've coded it, I bet when you transition from a block with some bits to remove, into a block with zero bits to remove, I think you'll still get a vertical line (click) - unless you carry the noise shaping on into that block. You could try that - it should hopefully have zero effect after the first four samples, but you'd need to check.
Will attempt - may take some time.
In the part shown, there are a few vertical lines even when transition between blocks which both have some bits removed. I'm not sure why that would happen.
Probably due to me scaling before noise-shaping - I'll work on that.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-04 14:37:23
You are a perfectionist!

(I view that as a good thing).

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-04 14:42:09
It's been put to me that it's more of a form of Obsessive Compulsive Disorder.... Probably why I'm an Engineer by profession.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-04 15:56:54
lossyWAV beta 1.1.2c attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-04 20:27:24
Thanks for testing lossyWAV 1.1.2b. Now things look pretty fine again. Did you use just -S resp. --standard with noise shaping as defaulted (no -s or --shaping option)?

BTW it seems that when describing how to ABX with foobar I wasn't precise enough in one point: You decide before testing how many trials you do, for instance 8 (minimum) or 10 or 16. After making up your mind the number of trials is fixed. In case you fail ABXing with the test you are free of course to do another ABX test, but when telling about your ABX results you should mention these failed ABX tests.

So far it looks like the noise shaping 'clicks' have caused the issue. Thanks, 2Bdecided, for showing up this problem, and, Nick.C, for immediate fixing.


Halb27 - thanks again.
I used --standard --shaping 0.
Both runs lossy file is Lossywav 1.1.2b
Im concerned if I am slightly not on the same page on this.

Yes, maybe we should pre determine an agreed upon fixed trials per testing session,
& the weight & need of a 2nd or more session, and 'rules' for them.

although that should provide some balance - deciding what that balance IS, is not that easy.

I just realized that even interpreting these results can go different ways, & needs attention, so here is my view on it:

As you see fom the SECOND run log,even the 14 or so first launches ,that were pretty bad on my part, which were obviously the result of fatigue from a prior 50(!) run,
did not influence the global result , even though they have Halved my 'score' on the 2nd run, still

45/80  15.7% [Pure 80 run] Vs.  50/94 (30.3%)

(This is my fault for being new to foobar , and not realizing that RESET is NOT a new testing session, but did include the failed result in the outcome)


So Now comes the part of how you interpret the results.
I might be totally wrong in my interpretation ,so please feel free to correct me.

As I told you , AB tests are not new to me, only Foobar tests.
So Ill explain my view of it, from my experience , just to be clear & make sure we are all on the same page here:

The Sole Reason for me to up the trials per session from 20 , to 50 , to 80,is to have Fatigue & Focus play a bigger role, & indeed they do.

Also the logs show it better then I can explain in words.

From real hardware shootouts that I used to take part in ,
Keeping a positive score will be harder and harder when fatigue,lost of focus, etc, sets in.

So we have to factor testing 'sessions' , as a time period when you truly can evaluate.

The 1st session posted is pretty clear , Im quite in sync with things , & focused enough to tell these apart.

Total: 31/50 (5.9%)

But actually this is not the important result , the 2nd run is the important one:

45/80  15.7% OR 50/94 (30.3%)

On the surface ,These Numbers show a lower score.
In reality these are equal or better as they still show a positive identification after a prior ~25 minutes 50 run test , that should be enough to cause a serious dent in a tester's ability to differentiate between the 2 with any success at all.

So a 2nd session should have at least a 80% failure rate to begin threatening the validity of the first session.

This is obviously not the case here.
The 2nd Run only says that the result of the 1st run , is indeed valid.

Now about doing a 2nd session at all:

This is NOT a good idea in general as the odds for you to detect a difference at all,
have already shrunk considerably , because the 2nd session is conducted 10 or less minutes apart.

on top of that I have almost doubled the tests per session on the 2nd session as you can see.

(If anyone has a different interpretation of the results I'd like to hear it & learn,
as Statistics is not my field.)

Quote
botface Posted Today, 06:03
Bork,
In an earlier post you mentioned "reduced overtones" and "midrange thickness" as well as noise. Does your latest listening test and ABX results indicate that these problems have gone away too?


although I tend to assume that the extra distortion in the highs was not that obvious (to me) this time, so either the --shaping 0 &/or the corrections made to loassywav were the reason.

the "reduced overtones" and reduced "midrange thickness" & weight are still here.
Title: lossyWAV 1.2.0 Development Thread
Post by: Axon on 2009-02-04 21:19:38
You know, when you add those two results together, you get p<0.078...
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-05 02:45:51
You know, when you add those two results together, you get p<0.078...


I dont understand what that means , so Ill have to take your word for it 
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-05 08:08:04
lossyWAV beta 1.1.2d attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-05 08:11:20
... The Sole Reason for me to up the trials per session from 20 , to 50 , to 80,is to have Fatigue & Focus play a bigger role, & indeed they do. ...

Because of these things and the extreme pain of such an ABX test I think it should be left to the tester to a large extent how to conduct the test.
But even with this in mind I think it's not totally correct the way you do it. Your last tests look a bit like continuing the test so far until you have reached a rather long sequence of more or less uninterrupted hits providing a rather low percentage of probabability for testing. While this still gives information that you can hear a difference (and may be enough for our needs here - but I'm not sure) it is not totally convincing. The longer your test the more likely is that such a sequence of hits happens by chance. You shouldn't try and try just until a guessing probability is reached you consider low enough. Using this as a test stopping criterion is a method for giving your test results a positive bias.

I'd feel more comfortable if you would make sure good listening conditions for you in another way, for instance doing a long warm up listening to just A and B before starting the real X/Y gueeses. And/or - as you seem to feel more comfortable with long test sessions - decide for a long test sequence of say 40 trials or whatever you like best. Use whatever listening condition is best for you. But the number of trials should be fixed.

Other than that thanks again for testing. We now know you didn't use noise shaping to get at your last results. And this brings at least better results than the noise shaping of 1.1.2.
We don't know however whether Nick.C's noise shaping fix improves the situation.
I know it's hard but can you please test your samples using Nick.C's current version (EDITED:) 1.1.2d and leave noise shaping as defaulted?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-05 08:51:31
I know it's hard but can you please test your samples using Nick.C's current version 1.1.2c and leave noise shaping as defaulted?
v1.1.2d, please.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-05 09:57:15
Wow, you're so fast. Sorry I missed your post.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-05 17:59:26
Halb27 , no need to ask me twice to give up on the longer sessions
I doubt they can become a 'standard' anyway ,& now that you mention it,
do think they are a bad idea anyway (.. my reasons are the opposite though , I think they tend to have a negative bias ,& too painful but nm)

Maybe a range of 20 minimum to 40 maximum run per session should be the agreed upon standard ? , you tell me.

Will try the new v1.1.2d version (Thanks Nick !) with default Noise --standard.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-05 18:27:18
Maybe a range of 20 minimum to 40 maximum run per session should be the agreed upon standard ? , you tell me.

You can choose whatever number of trials is most appropriate to you (except for the minimum of 8 trials there is no other restiction), but please choose a discrete number in advance and go through exactly this number of trials.
Will try the new v1.1.2d version (Thanks Nick !) with default Noise --standard.

Thanks a lot. You really bring things forward.
Title: lossyWAV 1.2.0 Development Thread
Post by: singaiya on 2009-02-05 20:17:50
IIRC, it's also valid to instruct foobar to hide ABX results until finished. So each time you make a choice on a trial, you don't know whether it was correct until you tell it you are done with the whole session. This way, you can end the session whenever you like (after 8 of course) and are not suspected of stopping only after a satisfactory p value is achieved.
Title: lossyWAV 1.2.0 Development Thread
Post by: Leo_White on 2009-02-06 10:12:48
I have made simple experiment:
Has compressed a file pcm.wav (the size 60Mb) using wavpack (switch -b200) and has received compressed lossy file in the size 8,5Mb. Further I have unpacked the compressed file, having received unpacked.wav and again have compressed it using wavpack with lossless compression. The received file has occupied 40Mb! It that for compression both time the same compressor was used and it is absolute the same not compressed information contained in both files! Answer please as it is possible is easier:
1. Whence there is such difference in the sizes of the compressed files.
2. As I understand, in lossyWAV the second variant of compression, i.e. lossless compression of preprocessed wav file is always used? Whether probably to create as much as possible optimized and integrated compressor which will provide compression directly to lossy compressed file with so high quality as now and thus much higher compression ratio (as at compression with wavpack lossy).

In my opinion for popularization of lossyWAV it is absolutely necessary to make the elementary GUI and place it on the same page of wikipedia as lossyWAV for downloading! In GUI it is possible to create following options:
make *.lwc (Lossy Wav Compressed)
make *.lws (Lossy Wav Stored)  and chekbox "preserve *.lossy.wav. extension for lossyWAV stored file"
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-06 10:40:07
.. As I understand, in lossyWAV the second variant of compression, i.e. lossless compression of preprocessed wav file is always used? ...

Yes.
... and thus much higher compression ratio (as at compression with wavpack lossy). ...

At the high efficiency end (~320 kbps and below) wavPack lossy is expected to provide the better quality.
A disadvantage of lossyWAV is that the lossless codec has to use short blocks of 512 samples which has a negative impact on the lossless codecs' efficiency.
At very high quality settings (~450 kbps and above) lossyWAV currently provides the better theoretical justification for quality though this does not necessarily mean that qualty is really better. In fact at such a high bitrate wavPack lossy provides excellent results as well.
A real advantage of lossyWAV over wavPack lossy is that it's safe for the future in the sense that the final lossless codec's results can be losslessly transcoded later to any other lossless codec.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-06 13:06:41
Nick,

1.1.2d solves the "clicks" completely.

Unfortunately, running the noise shaping across the zero bits_to_remove blocks doesn't work properly. It's not a bug in the code - the problem is that the noise shaping gets stuck in limit cycles, where the output values get stuck in a loop for a given set of input values, rather than decaying to zero. This means that the difference between the original signal and lossy version is a 1-LSB 22.05kHz tone (for example).

There are a few solutions. The easiest is to skip the zero bits_to_remove blocks; the downside is a "click" at the end of the previous block. The best solution I can think of is gradually transitioning away from the noise shaping during the first 10 or so samples of the zero block - either by crossign fading the coefficients (or the output), to zero. Re-set and re-start it at the next non-zero block.

I still think none of this has anything to do with the ABX results posted here.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-06 13:13:32
Yay!

Hmmm.... This would presumably also happen if a block had some signal followed by digital silence?

Would it still happen if --shaping 0.9 (for example) is used as this may(?) gradually attenuate the noise-shaping related differences?

Ah - problem - the FIFO values are not in the range -1..1 - they will be in the range -(2^bits_to_remove)/2..+(2^bits_to_remove)/2-1. This could cause strange effects when a block with a large bits_to_remove value is immediately followed by a block with zero bits_to_remove!
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-06 13:50:36
lossyWAV beta 1.1.2e attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: Leo_White on 2009-02-06 14:44:37
halb27, thanks for the answer! But you say only that lossyWAV always uses blocksize=512Kb (I understand it). But my main question, is why the sizes of two files, compressed by wavpack so much differs:
1. Wavpack lossy, received from the original (pcm.wav - rip from CD-DA)
2. Wavpack lossless, received by compression of wav file, which has been unpacked from wavpack lossy.

Matter here not only in blocksize! I was just convinced this:
1.   First I have taken the same pcm.wav (60Mb), and have compressed it in wavpack with desirable bitrate 200kbps, but this time have specified fixed blocksize for all file. For this purpose I used the following command line:
wavpack -b200  --blocksize=512  pcm.wav  bs=512.wv
The real bitrate has thus turned out 262kbps, and the size of a file 11Mb
2.   Then I unpacked bs=512.wv into bs=512unpack.wav
3.   After that I compress a file bs=512unpack.wav by use of lossless wavpack, again having specified the fixed size of the block:
wavpack - blocksize=512  bs=512unpack.wav  bs=512_lossless.wv
This time the size of a file has made 46Mb, that in both cases blocksize=512. Why it four times larger?

If Wavpack with fixed blocksize=512Kb makes so essential difference of file size depending on that, lossless or lossy compression is used, than and lossyWAV if it could at once directly not only process, but also to compress a file, then for the first time at absolute the same quality of a file (bit identical, if unpack to *.wav) we could receive the compressed file in the size 4 times less, than now (when we use wavpack, flac, tak, etc. with lossyWAV)! Thus nobody prevents to unpack in the future this compressed file without any loss of quality and then again to pack it by any other new lossless compressor, compatible with lossyWAV.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-06 15:08:30
halb27, thanks for the answer! But you say only that lossyWAV always uses blocksize=512Kb (I understand it)...

512 wave samples, not 512 kb.
... I used the following command line:
wavpack -b200  --blocksize=512  pcm.wav  bs=512.wv ...

wavPack works inefficiently with a blocksize of 512 samples. You should let wavPack decide on the block size.
... But my main question, is why the sizes of two files, compressed by wavpack so much differs:
1. Wavpack lossy, received from the original (pcm.wav - rip from CD-DA)
2. Wavpack lossless, received by compression of wav file, which has been unpacked from wavpack lossy. ...

Lossless codecs work like this: They make a prediction on the next sample based on the previous samples. The prediction follows a fixed scheme and so doesn't need any information to be stored (except for blockwise prediction parameters depending on codec). The prediction error usually is small and is stored exactly in an efficient way. With tle lossy variant of wavPack the prediction error is not stored exactly but is only approximated, with your given bitrate as a control parameter for the size of the prediction error allowed. When you pass your wavPack lossy result to wavPack lossless, wavPack has to store the exact prediction error of your wavPack lossy source, and there is no principle that makes this a more efficient prcedure than storing the prediction error of the original source.
wavPack lossy + wavPack lossless is a useless procedure which is totally different from lossyWAV + a lossless codec which makes efficient use of a reduced number of most significant bits per block.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-02-06 15:11:24
@Nickc.......

It seems that one or two people have recently discovered Lossywav and raised questions. There's also been a bit of development activity lately. That has refreshed a thought I had some time ago but never got round to raising.

Basically I am a big fan of lossywav and would like to see it more widely used. However, in the thread it is describes itself as "Added noise WAV bitdepth reduction method". Now, I know this is a true and accurate description but I'm sure that if I came accross Lossywav now and saw it described thus I would think it would be of no interest to me. Why would I possibly want to add noise to my music?.

I was wondering if changing the desription to something like "Dynamic WAV bitdepth reduction method" or "Variable WAV bitdepth reduction method" might encourage some people to delve a little bit deeper and hopefully find that Lossywav is a good solution for them rather than perhaps being put off by the not very flattering current description. I don't think descriptions like the ones I'm suggesting are inaccuarte or misleading they just don't point out the only negative aspect of Lossywav. In any case I've never heard any additional noise from Lossywav at --portable or above so why mention it?
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-06 15:16:33
Hi Nick,

You're so quick!

There are still a few limit cycles. Some have gone (I guess they were caused by internal values hanging around that were too big?), but some remain.

Like I said, I don't think it's a bug - it's what happens when you put low levels through high order recursive filters.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-06 15:24:35
@botface

A reasonable description is "near lossless audio coding".

Of course it begs the reasonable question "what is near lossless", or even the comment "it's either lossless or lossy - you can't have near lossless" - that's fine. It's a reasonable thing to ask/say, and the answer is also reasonable:

lossless (e.g. FLAC, ZIP) = mathematically lossless
lossy (e.g. mp3) = hopefully sounds similar / the same but might not stand further processing, nowhere near mathematically lossless
near lossless (e.g. lossyWAV) = hopefully sounds the same and stands further processing, is as near to mathematically lossless as the algorithms thinks is necessary (e.g. very quiet sections = mathematically lossless, louder/complex sections = nowhere near mathematically lossless)

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-06 15:26:48
@BORK,

Do you have time to try -I --shaping 0?

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: carpman on 2009-02-06 16:30:38
@botface

A reasonable description is "near lossless audio coding".

How about this snappy title?:

Transparent [until proven otherwise] Variable Bitdepth WAV Pre-processor for Lossless Encoders

C.

EDIT:

The reason I mention this is that the problem is that LossyWAV occupies an intermediary position, and any attempt to position it as an end result confuses the issue.

Lossy: WAV > MP3 (1 step)
Lossless: WAV > FLAC (1 step)
WAV > LossyWAV > FLAC (2 steps)

It's the intermediary step that needs to be stated.

Without that idea in the title, something like "near lossless audio coding" can lead to someone saying "Oh, like very high quality MP3"; near lossless = not lossless = lossy. You can end up with a mental image of High Quality MP3 in FLAC's clothing.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-02-06 18:43:10
@botface

A reasonable description is "near lossless audio coding".

How about this snappy title?:

Transparent [until proven otherwise] Variable Bitdepth WAV Pre-processor for Lossless Encoders

C.

EDIT:

The reason I mention this is that the problem is that LossyWAV occupies an intermediary position, and any attempt to position it as an end result confuses the issue.

Lossy: WAV > MP3 (1 step)
Lossless: WAV > FLAC (1 step)
WAV > LossyWAV > FLAC (2 steps)

It's the intermediary step that needs to be stated.

Without that idea in the title, something like "near lossless audio coding" can lead to someone saying "Oh, like very high quality MP3"; near lossless = not lossless = lossy. You can end up with a mental image of High Quality MP3 in FLAC's clothing.

I have no particular feelings about a name I just think that if we can avoid using "added noise" it might make it more attractive to casual browsers
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-06 18:48:21
I have no particular feelings about a name I just think that if we can avoid using "added noise" it might make it more attractive to casual browsers

I second your suggestion. Sounds reasonable to me.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-06 19:29:22
Lossy: WAV > MP3 (1 step)
Lossless: WAV > FLAC (1 step)
WAV > LossyWAV > FLAC (2 steps)

It's the intermediary step that needs to be stated.
If it's integrated properly, I don't know if many people would care about that. They might read about it, if they were interested, but I'm guessing most people would want to put a .wav, .flac, or CD in and get a lossy.flac file out. Nobody actually wants the intermediate .wav file* - just like, when people pop a CD in and run EAC to generate mp3s, they don't want the intermediate .wav file then either.

* - OK, for debugging.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-07 00:04:43
@BORK,

Do you have time to try -I --shaping 0?

Cheers,
David.


Hmm I see there's an even newer version so Ill get it ,(Thanks Nick & 2BDecided !)
I will redo the Test files , & will try when I can get some more time.

(Im a tiny bit frustrated that the only time I found since the first test is after Ive been exposed to sound for min. 8 hours, so Ill try to change that & not rush these, will get to it.)
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-07 00:42:52
@BORK,

Do you have time to try -I --shaping 0?

Cheers,
David.



Nick C & 2Bdecided ,ok, ran this file Before I go to sleep, because it's a good candidate for some noise, (short 20 run session) can you please check if you can see just what it is that 112e does in this mode, just converted the file using 1.1.2.e --insane --shaping 0 (Log and test files attached).

(Will do the other three files in a later date , so much for not rushing things ..  hmm)

http://rapidshare.com/files/194915589/112e...ping_0.rar.html (http://rapidshare.com/files/194915589/112e_Insane_Shaping_0.rar.html)
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-07 08:36:43
... ran this file ... using 1.1.2.e --insane --shaping 0 (Log and test files attached). ...

Thanks a lot for your test. You can really hear a problem with --insane !!! Hopefully there is a specific problem with lossyWAV that can be fixed.
As I recovered from my cold I'll try your sample tonight when it's quiet at my home though my hearing certainly can't compete with yours.

Something comes to my mind:
Maybe it's a  bit egocentric but I never moved up from version 1.1.0 because this was the last version I tested intensively and no real reason came up to me after 1.1.0 to upgrade to a newer version (things have changed right now because of the way noise shaping is being improved).
There had been significant changes in the software however, and may be there is a specific implementation issue which emerged after 1.1.0.

BORK, do you mind testing again your last sample using --insane --shaping 0 with lossyWAV 1.1.0 (http://www.hydrogenaudio.org/forums/index.php?showtopic=56129&view=findpost&p=613528), which I just uploaded in the UPLOAD section?
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-07 11:03:53
@halb27,

I think you're on the wrong track. Take a listen to the correction file - the noise is barely audible most of the time even without the music playing.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-07 13:53:20
Maybe I'm on the wrong track.
But I've just listened to this sample's error signal of both the 1.1.0 and the 1.1.2e results using --insane --shaping 0. There is a spot in the 4.5...6.0 second range where the error is clearly more audible with the 1.1.2e result.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-07 17:58:28
@halb27,

I think you're on the wrong track. Take a listen to the correction file - the noise is barely audible most of the time even without the music playing.

Cheers,
David.


2Bdecided ,I probably need better understanding of the internals of how lossy wav works, but I cannot understand why/how Isolating the noise in the correction file,
can help undertsand the effect of it removed, as it's isolated from it's effect combined in the music.

Please , No offense ,but ,I see you said that you do not see any real relation to the Full Music ABX results ,& I respect that, but then you say listening to the correction file convinced you ? , that is a valid way to determine the effect ?


for what it's worth, the way I see it, It's like taking a thin yellow layer off a green.

looking at the yellow layer removed & saying all you see is a real pale of yellow, -nothing that seems of any importance.

But the real effect is in the main file, the Music, the green now looks blue instead of green. This was quite obvious to me here.

Obviously more testers are needed At this point, the more the better (calling all users to help out !), so at least you'll know what percentage of (hopefully experienced) users
share similar results & experiences.

If anyone reading this can please test with these file and post his results I would be grateful:
http://rapidshare.com/files/194915589/112e...ping_0.rar.html (http://rapidshare.com/files/194915589/112e_Insane_Shaping_0.rar.html)
Title: lossyWAV 1.2.0 Development Thread
Post by: Hancoque on 2009-02-07 18:26:03
2Bdecided ,I probably need better understanding of the internals of how lossy wav works, but I cannot understand why/how Isolating the noise in the correction file,
can help undertsand the effect of it removed, as it's isolated from it's effect combined in the music.

Please , No offense ,but ,I see you said that you do not see any real relation to the Full Music ABX results ,& I respect that, but then you say listening to the correction file convinced you ? , that is a valid way to determine the effect ?

Well, the correction file is the difference between the original and the processed file. When you listen to it you hear what alterations have been made to the original file. It is easier to listen to that file to determine the differences because they are not masked by the rest and you can also turn up the volume to extreme proportions to make it even easier. lossyWAV takes away certain fine detail of the original waveform. This missing fine detail is audible as white noise that fluctuates in intensity in accordance to the original waveform.


for what it's worth, the way I see it, It's like taking a thin yellow layer off a green.

looking at the yellow layer removed & saying all you see is a real pale of yellow, -nothing that seems of any importance.

But the real effect is in the main file, the Music, the green now looks blue instead of green. This was quite obvious to me here.

I think that's a very valid analogy. It's amazing that you feel music this way, but I now wonder if there is any way at all to reduce the file size in a lossy way that you don't notice at least to a tiny extent. I don't believe it's possible to maintain current bitrates while also achieving full transparency for you. We now know that you can ABX the differences but how does that affect the validity of the process itself for you? Would you rather stick with lossless files or would the differences be acceptable for your personal taste?
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-07 18:54:05
... I now wonder if there is any way at all to reduce the file size in a lossy way that you don't notice at least to a tiny extent. I don't believe it's possible to maintain current bitrates while also achieving full transparency for you. ...

I think that's a bit too early a conclusion though it may come out like that. At the moment there is hope that it's all about an implementation issue.
Title: lossyWAV 1.2.0 Development Thread
Post by: Brent on 2009-02-07 19:15:35
What happens when I process an already 'lossywavved' file again? If I take a lossywav processed at say -q 7, and process it once more at -q 3, will this turn out similar to a that file processed straight from untouched source to -q 3?

In other words, will these give me the same, or very similar results:
source -> -q 3
source -> -q 7 -> 3
?
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-07 20:28:36
If anyone reading this can please test with these file and post his results I would be grateful:
http://rapidshare.com/files/194915589/112e...ping_0.rar.html (http://rapidshare.com/files/194915589/112e_Insane_Shaping_0.rar.html)

I tried your sample. With my first ABX test (10 trials) I got at 5/6, then lost the problem and ended up 6/10.
My second ABX test was a total miss.
I took a rest, had a light evening meal, did some work at the house, and returned to ABXing, but again without success.

Though my results are bad at least the 5/6 result I got from scratch with my first 6 guesses confirm the suspicion that there's something wrong.
Strange enough I tried to ABX the lossyWAV1.1.2e --portable --shaping 0 result, but to me it was as hard as with your supplied --insane result and I did not succeed.
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2009-02-07 22:18:15
What happens when I process an already 'lossywavved' file again? If I take a lossywav processed at say -q 7, and process it once more at -q 3, will this turn out similar to a that file processed straight from untouched source to -q 3?

In other words, will these give me the same, or very similar results:
source -> -q 3
source -> -q 7 -> 3
?



Yes, the results would be very similar. Remember that lossyWav works by reducing the bit precision (which is why it adds noise). It also applies other mechanisms like dither, which is why i don't say "the same", but remember that dither only modifies the LSB (in this case, of the reduced bitdepth, not of the original bitdepth)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-07 22:24:10
Dither has been removed, however noise-shaping is standard since 1.1.0, with effectiveness at q/10, i.e. -q 0 > 0% effective noise-shaping; -q 5 == --standard > 50% effective noise-shaping.

Alternatively, try it - see how big the -q 7 followed by -q 3 file is in relation to a one step -q 3 file.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-08 00:47:27
I think that's a very valid analogy. It's amazing that you feel music this way, but I now wonder if there is any way at all to reduce the file size in a lossy way that you don't notice at least to a tiny extent. I don't believe it's possible to maintain current bitrates while also achieving full transparency for you. We now know that you can ABX the differences but how does that affect the validity of the process itself for you? Would you rather stick with lossless files or would the differences be acceptable for your personal taste?


I would like to first point out that I Like LossyWav a lot (or I would not be here) from the start, & it's already very good.

The only reason to these tests is my attempt to help with improvements, not discredit it in any way.

Since I am not a coder, I just try & help in ways I can.

I would stick to Lossless files for archiving if I'll have to as I have done for long,
but I'd really rather not on my media server / portable etc.

I assume/hope implementation & methods used could be improved as already demonstrated by Nick & 2BDecided , (Thanks!) so I think/hope more testers with a single chosen compression mode, & further improvements (We probably need to choose the best mode the develeper/s think is of the utmost importance & get as many tests & improvements done with it, as testing all will be impossible,at least for me) can bring it to the point that the difference is extremely hard to point at , even with professionals/reference ears/gear etc.


I tried your sample. With my first ABX test (10 trials) I got at 5/6, then lost the problem and ended up 6/10.

Though my results are bad at least the 5/6 result I got from scratch with my first 6 guesses confirm the suspicion that there's something wrong.


That's the fatigue + boredom effect for you, that I mentioned earlier & will always be our enemy .. it always amazes me how quickly it sets in, and how hard it is to shake off once it hits you (hence my obsessive long session attempts).

Ill have to look through the logs but If I am not mistaken, I did not score better then 5 repetitive 'wins' on the tests either, so you heard it as clean as possible if you ask me.

Slightly off topic (but indeed entertaining) I am surely no expert at this, but at least from my experience in Retail Pro Audio Sales using AB tests for clients (50% of the time against their own gear), I learnt the following:

-When people cannot tell a difference, they get it wrong ~70% percent MINIMUM.
-I have yet to see anyone get 5/6 in real life AB tests by chance.
-When they did get something right , they usually managed to do it it when they were 'fresh'.
-For those who persisted to continue testing, the stats were cruel, & by the 15th try few would try again.

People who managed to get something along the lines of a Fifth attempt right listening to something &/or liking the result,usually had an open wallet in their hand the next second,or a day later 
Title: lossyWAV 1.2.0 Development Thread
Post by: carpman on 2009-02-08 01:04:19
-When people cannot tell a difference, they get it wrong ~70% percent MINIMUM.
-I have yet to see anyone get 5/6 in real life AB tests by chance.

I don't understand this.
Are you saying you've never known anyone toss a coin and it come up heads 5 times out of 6? Or is the probability different in A/B tests? I would have thought if people cannot tell the difference, then they have to guess and then wouldn't they have a 50% chance of being right (rather than 30% or less)?
Have I misunderstood A/B tests?

C.
Title: lossyWAV 1.2.0 Development Thread
Post by: Cavaille on 2009-02-08 01:32:07
Forgive my interruption but I have a question. I´m using 24/96 material a lot, for listening, remastering and other stuff. For now I used WavPack lossy a lot but I also tried LossyWAV. As I understood this, LossyWAV dynamically decreases bit-depth, the resulting quantization noise is moved to higher frequencies afterwards via noise-shaping (depends on the quality setting). Right? So far my opinion about LossyWAV is that it is a very ingenious, clever and effective method to reduce needed space to save audio material.

Would it be possible to have a stronger noise-shaping for 24/96 material since frequencies beyond 20.000 Hz are not audible to us (at least not by ear - but the hypersonic effect is still controversial) in order to shift ALL quantization noise to the frequency area from 20.000 to 40.000? Or would then the sheer amount of quantization noise be too "loud", effectively producing clipping?

I followed LossyWAV development for some time now and it always reminded me of SACD where the quantization noise is completely shifted to ultrasonic frequencies. Is this possible? Correct me if I´m wrong, because I´m no technical expert.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-02-08 08:11:47
Dither has been removed, however noise-shaping is standard since 1.1.0, with effectiveness at q/10, i.e. -q 0 > 0% effective noise-shaping; -q 5 == --standard > 50% effective noise-shaping.

This is probably a stupid question but I'm intrigued. ..........

As quality levels are increased (from q0 towards q10) the number of bits removed decreases. Hence less noise is present in the output file. Conversely the lower the quality setting the more bits are remove and more noise is present in the output. So why does noise-shaping increase with quality setting? Shouldn't it be the other way round so that noise-shaping is applied more strongly to the lower quality settings to mask the increased noise?
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2009-02-08 13:15:15
Would it be possible to have a stronger noise-shaping for 24/96 material since frequencies beyond 20.000 Hz are not audible to us (at least not by ear - but the hypersonic effect is still controversial) in order to shift ALL quantization noise to the frequency area from 20.000 to 40.000? Or would then the sheer amount of quantization noise be too "loud", effectively producing clipping?


The higher you shift the noise, the harder it is to compress for a lossless compressor, since more variation exists in the time domain. As such, the benefits of using lossywav could decrease.


That also sort of replies to botface, although I would preffer a better answer from Nick, 2bDecided or others.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-08 13:44:46
... Shouldn't it be the other way round so that noise-shaping is applied more strongly to the lower quality settings to mask the increased noise?

As [JAZ] wrote already there is an efficiency penalty for the lossless codec when applying noise shaping. With low quality settings the file size increase is very remarkable, with high quality settings file size increase is more or less negligible. That's why lossyWAV increases noise shaping with quality setting.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-08 13:46:04
@BORK:

I'd welcome very much if you could try version 1.1.0 with your last sample.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-08 14:57:44
As [JAZ] has already indicated, the decision to apply more noise-shaping to the higher quality levels has everything to do with added bitrate due to noise-shaped audio being inherently more difficult to compress than non-noise-shaped audio.

In this post (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=63254&view=findpost&p=566790) the link between quality preset and shaping level was made - as you can see the trend is quite marked.

@B0rk: I would be interested if you notice any difference by applying a higher upper frequency limit (--limit <n>, where 16000<=n<=20000), say --limit 17500?

@Cavaille: The noise-shaping coefficients were published by SebastianG some time ago and have been calculated for 44.1kHz and 48kHz. The 48kHz coefficients are used for any frequency above 48kHz. This will tend to force the quantization noise energy out of the audible range and into the ultrasonic range as you have already surmised.

In general, noise-shaping does not decrease the quantization noise energy, merely shifts it about the spectrum a bit.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-02-08 17:09:09
As [JAZ] has already indicated, the decision to apply more noise-shaping to the higher quality levels has everything to do with added bitrate due to noise-shaped audio being inherently more difficult to compress than non-noise-shaped audio.

In this post (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=63254&view=findpost&p=566790) the link between quality preset and shaping level was made - as you can see the trend is quite marked.

Yes, I see. But in that example the -q0 -s1.00 Lossy.Flac file is still 10% smaller than the --portable one with its default shaping (q0.25). Or is that too simplistic a way of looking at it?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-08 18:16:33
It is a bit simplistic because you need to look at the other parameters that change with quality-level - they are what make up the bit-rate changes along each line.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-08 19:40:07
I just finished 3 10 trial ABX tests to see whether I can find something wrong with the lossyWAV 1.1.0 --insane --shaping 0 result of Samba e Amor.
I could not find any hint that something maybe wrong and my ABX results were total misses.
I know it's of very limited value cause my ABX results of losyyWAV 1.1.2e were bad as well though my first ABX test then started off fairly well.
Maybe I'm too much inclined to the clearly more audible noise in the error file of 1.1.2e, but I can't withstand the idea that things may have gone wrong after 1.1.0.

BTW the lossyWAV1.1.0 and 1.1.0b results for Samba e Amor are identical when using --insane --shaping 0 (probably they are always identical). So 1.1.0b can be tried instead of 1.1.0. 1.1.0b is still available in the lossyWAV thread on 1.1.0 (including source).
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-08 19:45:54
The *big* difference after v1.1.0b is the fundamental change in the spreading function. I think that I need to re-visit the spreading function in light of mixed reports regarding the apparent quality of v1.1.2e vs v1.1.0b.

I am, as always, grateful for the time and effort expended by those users who find new difficult samples and choose to ABX them.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-08 21:43:37
Some general comments...

I'm always delighted that people are willing to spend their time listening and testing in order to provide ABX results.

The correction file tells you nothing about what the noise will sound like on top of the music, but it does show you exactly what lossyWAV is doing. That's its use. Without knowledge of the original music, it's meaningless.


I think it would be worth looking further at -I 1.1.2e, -P 1.1.2e, and -I 1.1.0b (all with --shaping 0).

As you know only too well halb27, it makes little sense (on the face of it) to think there's a problem with -I, but completely fail to hear a problem with -P. In investigating this, it makes more sense to listen to the correction file first, rather than the bit-reduced output - if lossyWAV is working as expected then how can -I be audible but -P be inaudible? (I don't say it's impossible - I'm saying, if that's what's happening, let's find out why). If lossyWAV is not working as expected (e.g. there's an important part where -P adds less noise than -I) then it's hopefully an easy fix (but I'm not hopeful that it's this simple).

As for 1.1.0b vs 1.1.2e (both -I --shaping 0) - a moment by moment comparison of added noise would be interesting.


If you haven't done already, I'd encourage you to read the end of the "24-bit vs 16-bit - any samples that work" thread. lossyWAV 1.1.2 -I --shaping 0 wants to keep 19-bits of the 24-bit recording (the same recording which someone can ABX when converted to 16-bits vs the original 24-bits). So it's not like it's being universally careless!

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-08 22:45:58
@BORK:

I'd welcome very much if you could try version 1.1.0 with your last sample.


Sorry for late reply , will dl it & try soon , I have been following in between work,
will try it, thanks.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-08 23:37:51
@B0rk: I would be interested if you notice any difference by applying a higher upper frequency limit (--limit <n>, where 16000<=n<=20000), say --limit 17500?


missed this post as well , sorry about that,
3 questions :

-what does that switch do ?
-please give me a Full (including all other needed switches) commandline example for it.
-what version to use ?

Thanks.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-02-09 10:55:45
For what it's worth I've tried ABXing Samba e Amor and a solo guitar piece from my own library with and without shaping and with 1.1.0 and with 1.1.2e. I can't hear any difference between any of them. Having said that ABXing is not my forte
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-09 13:21:01
3 questions :

-what does that switch do ?
-please give me a Full (including all other needed switches) commandline example for it.
-what version to use ?
1) It raises the upper frequency limit taken into account when determining the average and lowest bin values during the FFT analyses carried out on the audio data;
2) "lossywav <filename> -S --limit 17500 --shaping 0" would process using --standard and with no noise shaping, taking into account frequencies up to 17.5kHz;
3) 1.1.2e in the first instance, 1.1.0b as well if you can spare the time.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-09 22:52:53
lossyWAV beta 1.1.2f attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-09 23:22:31
lossyWAV beta 1.1.2f attached to post #1 in this thread.


Nick Thanks !
Ill do it with this version instead.

Quote from: botface link=msg=0 date=
For what it's worth I've tried ABXing Samba e Amor and a solo guitar piece from my own library with and without shaping and with 1.1.0 and with 1.1.2e. I can't hear any difference between any of them. Having said that ABXing is not my forte


botface thanks.
You & other testers please note - 'failing' this ABX Test means nothing.
it dependes on what you focus on , how fast you switch between samples,
& cannot be done without some practice, even if the differences do exist.

Look at this Log I just tested,just got a new dac here to test  - had I stopped in a 11 run, it would have probably been the best result I ever got.
Since I insist to stick with at least a 20 run , & fatigue kicked on me from there on,
I 'failed' it.

but note the 1st 11 first/fresh results.
00:52:33 : 09/11  3.3%

so even though the score is 'failed' that's all I need to know 2 things:
1 - There's a difference , that can be pointed at.
2 - This DAC's Nice

anyone can fail these kind of tests - many times over - it still does not mean that thre is no difference or that long term listening won't hint at it.

Also try different kinds of music - & try to listen to shortish lengths for this test.

foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/10 00:48:51

File A: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Country Roads - Pat Metheny.flac
File B: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Country Roads - Pat Metheny.lossy.flac

00:48:51 : Test started.
00:49:01 : 00/01  100.0%
00:49:25 : 01/02  75.0%
00:49:48 : 02/03  50.0%
00:50:16 : 03/04  31.3%
00:50:31 : 04/05  18.8%
00:50:53 : 05/06  10.9%
00:51:10 : 06/07  6.3%
00:51:25 : 06/08  14.5%
00:51:47 : 07/09  9.0%
00:52:25 : 08/10  5.5%
00:52:33 : 09/11  3.3%
00:52:46 : 09/12  7.3%
00:53:00 : 09/13  13.3%
00:53:25 : 09/14  21.2%
00:53:56 : 09/15  30.4%
00:54:08 : 09/16  40.2%
00:54:25 : 09/17  50.0%
00:54:32 : 10/18  40.7%
00:54:53 : 10/19  50.0%
00:54:59 : 11/20  41.2%
00:55:06 : 11/21  50.0%
00:55:12 : 12/22  41.6%
00:55:23 : 12/23  50.0%
00:55:35 : 13/24  41.9%
00:55:50 : 13/25  50.0%
00:56:17 : 13/26  57.7%
00:56:21 : Test finished.

----------
Total: 13/26 (57.7%)

From Riches To Rags 
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-10 00:47:02
Nick C. , again thanks a lot for the snappy updates,
I really cannot keep up with you ! ( That is NOT a complaint ! You Rock !)

OK I am done testing the old 112e version shaping 0, I just tested one too many of these.

I got some weird results with this DAC I brought in to test with (just to see if it will make any difference for me at all), but some wide spread results,
from results like the above , to 80% wrong on some new file I am not familiar with,
back & forth.

So I just I went back to the OLEO track as I really know it quite well,
wondering If trying an unknown (although better) DAC was a good idea ..
this is probably the only/last time Ill ever see this again, so Im posting it.. :

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/10 01:51:34

File A: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bob Berg - Oleo.flac
File B: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bob Berg - Oleo.lossy.flac

01:51:34 : Test started.
01:51:41 : 01/01  50.0%
01:51:47 : 02/02  25.0%
01:51:57 : 02/03  50.0%
01:52:04 : 03/04  31.3%
01:52:16 : 04/05  18.8%
01:52:28 : 05/06  10.9%
01:52:35 : 05/07  22.7%
01:52:56 : 06/08  14.5%
01:53:09 : 07/09  9.0%
01:53:31 : 08/10  5.5%
01:53:48 : 08/11  11.3%
01:54:13 : 08/12  19.4%
01:54:43 : 08/13  29.1%
01:55:01 : 09/14  21.2%
01:55:13 : 10/15  15.1%
01:55:23 : 11/16  10.5%
01:55:39 : 12/17  7.2%
01:55:49 : 13/18  4.8%
01:56:07 : 14/19  3.2%
01:56:28 : 14/20  5.8%
01:56:41 : 15/21  3.9%
01:56:50 : 16/22  2.6%
01:57:00 : 16/23  4.7%
01:57:14 : 17/24  3.2%
01:57:31 : 18/25  2.2%
01:57:44 : 18/26  3.8%
01:58:35 : 19/27  2.6%
01:59:11 : 20/28  1.8%
01:59:33 : 21/29  1.2%
01:59:55 : 22/30  0.8%
02:00:01 : Test finished.

----------
Total: 22/30 (0.8%)


OK, Time for me to move on to the new version & your new commandline instructions,
will continue tomorrw hopefully, Thanks for all the Work You Do !! it's appreciated !
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-10 10:42:19
Nick,

I don't suppose there's a hidden option in lossyWAV to dump the bits_to_remove list into a text file is there?

I'm not asking you to add it in a new version since I'd need it in older versions too to compare, and that would be a mad amount of work for little benefit. I'm just wondering if it's been there all along?

If not, I'll just keep looking at the correction files in Cool Edit. That shows differences easily enough.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-10 11:25:50
These results are a little hard to understand.

An ABX of -S --shaping 0 was reported here:
http://www.hydrogenaudio.org/forums/index....st&p=613046 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=65499&view=findpost&p=613046)
...it wasn't that conclusive.

This is the noise added by lossyWAV -S:
[attachment=4882:lossyWAV_S.jpg]


An ABX of -I --shaping 0 has been reported several times.
Some of these ABX results are conclusive.

This is the noise added by lossyWAV -I:
[attachment=4883:lossyWAV_I.jpg]


Given that lossyWAV does absolutely nothing "clever" with the noise when you add "--shaping 0", logic would suggest that -S would be much easier to ABX than -I, since the noise is 13.5dB louder (that's 4x as much noise!).

So what's happening here?

I think it would help greatly if BORG could say more about what he hears in these two situations. So please do!


On the general problem of explaining how 13.5dB more noise may be harder to detect...

It's either a poorly understood properly of human hearing, a poorly understood property of the replay equipment, or some less than rigorous application of ABX statistics.

To have a hope of tracing this problem, we need to attack each possibility in turn, in reverse order I think.


ABX stats

So, with the ABXing, as has already been said, pick a number of trials and stick to it. I doesn't matter what the number is (as long as it's greater than 7), but if you're picking then testing, I'd expect all tests to include the same number of trials.

Also, to save conjecture, take a break when fatigued - don't say "I'd nailed it at 6 trials, but then it went wrong", or "around the 16th trial you'll see I'd got it, but it had gone by the 32nd" etc etc - this is of no more use than tossing a coin, getting 3 heads in a row, and declaring it's a biased coin. If you don't use the statistic properly, they're worthless. It may reflect the reality of testing - people do get fatigued - but somehow we've got to find a way of not breaking the statistics or the tests become worthless.

The idea of doing lots of separate runs of tests because at some point you'll nail it is also worthless. If you head for p=5%, you still have a 5% chance of getting a result by chance. If you repeat the test until you get a positive result, then the (statistical) chance of that positive result being purely down to chance becomes really really high.

It is frustrating and time consuming to have to work like this, and it's a pain because human perception gets tried so easily with repetition - but anything else is just tweaking in the dark.


equipment faults

As in the thread about 16-bit vs 24-bit, the only fault I can think of is that the DAC behaves strangely / specially when LSBs are zeroed. One answer is to fill the bits that would be zeroed with completely random values - i.e. random noise. This is not much use for lossyFLAC encoding, but it is useful for tracking down the source of the audible difference. It's not a completely fair test as the character of the random numbers may not perfectly match the character or power of the truncation noise - but it's a lot closer than the difference between -S and -I, so worth a try.


hearing

Errr, it's about ten years to late to get this into my PhD. Someone else will have to look at this! Maybe it doesn't have to be understood, so much as predicted "well enough" by lossyWAV to avoid audible problems. But it makes sense to rule out the other two factors first I think.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-02-10 14:13:51
With the dualstream and wvlossy, The added noise gets louder with lowered bitrate. Its really easy to learn as hiss or noise is the only artifact. Why should Lossywav be different ? The difference should get louder as the quality setting is lowered. That -I is abxed and not -S ?? not even -P ?

The effect is background hiss or (noise - instruments sound 'dusty').. There is nothing else, no thin mids etc etc, no traditional mp3,aac,mpc artifacts. It can be easily confirmed by lowering the quality level. This Oleo track should be tested @ quality 1..2
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-10 15:02:07
Well, yes, so the explanation is...?

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-02-10 15:15:21
Well I dunno. Even the track oleo is too 'busy' to have real problems for these codecs. LossyW and co like busy action like many instruments = more masking.. right ?

I am not saying the abx is wrong. I think the others have to confirm the problem and that can be difficult. The OP description doesn't match the hiss / noise thing - what is he hearing ? He needs to find out by setting less trials like 8 or 16 at lower quality setting and get a perfect score without fatigue. Then it will be very clear what the difference is. If there are 30 trials + fatigue @ quality <= 3 then I think something is sus.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-10 19:36:42
Nick,

I don't suppose there's a hidden option in lossyWAV to dump the bits_to_remove list into a text file is there?

I'm not asking you to add it in a new version since I'd need it in older versions too to compare, and that would be a mad amount of work for little benefit. I'm just wondering if it's been there all along?

If not, I'll just keep looking at the correction files in Cool Edit. That shows differences easily enough.

Cheers,
David.
Yes, there is - but it's not hidden: -d or --detail will display a per channel per block bits-to-remove table. Combined with -w or --writetolog and you get a "permanent" record in the form of a lossywav.log file.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-10 20:44:55
equipment faults

As in the thread about 16-bit vs 24-bit, the only fault I can think of is that the DAC behaves strangely / specially when LSBs are zeroed. One answer is to fill the bits that would be zeroed with completely random values - i.e. random noise. This is not much use for lossyFLAC encoding, but it is useful for tracking down the source of the audible difference. It's not a completely fair test as the character of the random numbers may not perfectly match the character or power of the truncation noise - but it's a lot closer than the difference between -S and -I, so worth a try.
I'll use the random number generator, previously used for dither, to provide the necessary random bits - this will be enabled by -r or --randombits.

[edit] lossyWAV beta 1.1.2g attached to post #1 in this thread. [/edit]

[edit2] rounding noise, surely, not truncation noise? [/edit2]
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-11 00:52:24
2b Decided - Thaks for the effort & detailed reply.
I have no conclusions or answers to some of the questions raised,
I can provide more info on what I noticed as requested.
Hopefully this thread will generate enough interest to bring in many pros & people with good ears to help LossyWav evolve even further.



Re: Results
-----------
I brought in this DAC so I can test if gear will bring anything different for the testing results here.

This last result with the Oleo Track was done with this borrowed DAC,
with this DAC it was just easier for me to hear the difference in the Midbass thickness, as opposed to the one used before.

(This does not tell me a WHOLE lot as I again tested after a working day,
but that's the time I have now, so that's how it is,
maybe focusing on the lower range was just easier for me because of it ,or maybe this DAC's plain better, not sure.)

I tried to lock on to the noise aspect, but could not really do it,
could be the busy music,being tired.this DAC , no idea.

Thing is ,The real file just sounded fuller in the Mid/Bass region.

What I did not report is that for most of the run, I didnt even bother using the AB buttons.
I thought I locked in on a lack of oomph in the midbass , played X & Y,
decided what's the real file & clicked the answer ( the log times can hint at it).

Re: Runs/Length
--------
The 20 minimum to 40 maximum run choice I use was discussed earlier in this thread,if you recall , when it was suggested that the longer sessions be dropped.
I did drop them, mostly I saw the 20 run usually just tells it like it is.

(Further runs will either weaken or strengthen your score, I get that,
but maybe do have some merit if consecutive positive or negative results are obtained as fatigue does have an effect here , or can prove/disprove a user being able to lock on to it , as sort of additional indication ? you tell me.

I will adjust to anything offered as needed, you decide.


Re: LSBs
--------
The theory of DAC's behaving strangely when LSBs are zeroed,
is interesting , sounds like something that needs to be checked !
but must be first verified in a more scientific way, if you can tell me how to check that myself ?

I mean this is something (at least my) ABX cannot probably be be trusted with,
when all of us I guess, tilted towards one or the other bias polarity.. do not trust these machine-gun type based ABX tests like this 100%).

Is there some way I can test for it ? please advise.


Hearing
--------
The Noise was reported in previous tests if I am not mistaken,
but obviously I only have my current ears & experience to test with.

the first thing Ill do when I can, is test the noise with a non professional user,
that has his hearing intact - with verified good sensitivity up to 20khz,
& see if that at all makes any difference on his subjective uninformed listening impressions, and detailed tests after pointing out the noise differences to him,
& test if he/she can lock onto it.

will report when I can, hope this helps.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-11 11:55:45
BORK,

Regarding run length, just pick a single number and stick to it. Even saying "it will be between 20 and 40" breaks the statistics. You've got to say "from now on, I will do 25", and always do 25.

Hard, I know, but far easier than calculating the stats for the "I'll stop when I decide to on-the-fly" case. The stats ABX shows as you go along are based on the "pre-chosen fixed number of trials" model. The stats for "I'll stop when I decide to on-the-fly" are much stricter and require many many more trials.


Getting opinions from other people would be very valuable. They'd ultimately have to pass the ABX test too to be considered reliable. Otherwise they're just agreeing with you! Single blind tests (you know what track you're playing, but they don't) do not count as "reliable". It's probably something to try first, but it's not proof.


Thanks for taking the time to do this. I know that it is hugely time consuming!

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-11 19:44:10
BORK,

Regarding run length, just pick a single number and stick to it. Even saying "it will be between 20 and 40" breaks the statistics. You've got to say "from now on, I will do 25", and always do 25.

Hard, I know, but far easier than calculating the stats for the "I'll stop when I decide to on-the-fly" case. The stats ABX shows as you go along are based on the "pre-chosen fixed number of trials" model. The stats for "I'll stop when I decide to on-the-fly" are much stricter and require many many more trials.


Getting opinions from other people would be very valuable. They'd ultimately have to pass the ABX test too to be considered reliable. Otherwise they're just agreeing with you! Single blind tests (you know what track you're playing, but they don't) do not count as "reliable". It's probably something to try first, but it's not proof.


Thanks for taking the time to do this. I know that it is hugely time consuming!

Cheers,
David.


Thanks , will stick to a single value for runs, I agree that stats gathering would be way more effective when the same run times are used for all sessions , & even better for all testers, if there will be such an agreed upon number I will adjust to it.

Nick C.
I am going to do all the test files to the new version & parameters,
please tell me what command line to use for it.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-11 20:16:45
I'm not sure what quality level / shaping level you want to test....

"--extreme --shaping 0" would be a sensible start to see if there is anything definitive.

--extreme equates to --quality 7.5, so if you easily ABX --extreme then you can gradually increase the quality level without jumping straight to --insane (--quality 10.0).

At this time, I would suggest that you may also want the processing to be influenced more by higher frequencies. With this in mind, you could add --limit 17500 to the command line for a second run if the first is a simple ABX.

My guess is that you can hear sounds quite some way up the frequency scale....
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-11 20:58:58
I'm not sure what quality level / shaping level you want to test....

"--extreme --shaping 0" would be a sensible start to see if there is anything definitive.

--extreme equates to --quality 7.5, so if you easily ABX --extreme then you can gradually increase the quality level without jumping straight to --insane (--quality 10.0).

At this time, I would suggest that you may also want the processing to be influenced more by higher frequencies. With this in mind, you could add --limit 17500 to the command line for a second run if the first is a simple ABX.

My guess is that you can hear sounds quite some way up the frequency scale....


Nick, the reason Im asking is Id like to work on a Test File Set for all testers, (I will upload it), & using a single testing method , we can gather data you can analyze the stats for all submissions as testers join ,so I am assuming this will only have some value if we all use a single mode & method.

Only you can pick the one that should be the most urgent to gather stats for.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-11 21:22:24
Hmmm.... Since no-one previously had successfully ABX'd --standard then I think that that would be the starting point - with no modification to any other settings.

If we get some feedback indicating that --standard can indeed be ABX'd then maybe --extreme would be the next step. I would think of this as a sort of binary search (0 to 10, pick 5; if 5 is ABX'd then 5 to 10, pick 7.5; etc.) to determine the transparency point (if any).

There has been some talk regarding the noise-shaping implementation. I believe that this is pretty much boxed off in that the blips seem to have largely been removed in 1.1.2f onwards.

Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-12 11:39:17
If you want lots of people to test, I think you'd better have examples starting from portable, or even lower!

You've got to be able to learn what you're listening for in an obvious way before making it less obvious.

(I suspect what BORK is hearing is quite different, but we'll have to start this in a conventional way at least).

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-12 11:58:54
I understand - it wouldn't help me though as I have not yet found a problem with --portable in all of the hours of play on my DAP.

So, it now seems to be the time for a call for samples (say 6 at the most) to be tested.

Following David's suggestion, these could be processed at --portable and --standard in the first instance, to which I would add --quality 0. Using --quality 0 results in the most likely case where artifacts will be identified and could be use as the initial learning tool for the higher quality presets.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-12 14:47:00
I can't keep up with all the different releases - which one do you want us to test Nick?

EDIT: another "problem" is that even -q 0 isn't obviously bad to me. In fact, let me be bluntly honest here: I can't hear a difference! This is 24dB more noise than BORK has ABXed with -I.

When I was first playing with lossyFLAC (using quite a different algorithm for judging the amount of noise to add I suppose) the difference between "transparent" and "awful" was about 12dB. Your tweaks make it much harder to simply add too much noise (also, there's no threshold shift to brute force it higher), so it's not a fair comparison back to those days.

Even so, with almost any audio codec in existence, you wouldn't expect 24dB difference between what two people think is "transparent". If you increase the coding noise in a "good" mp3 by 24dB, it'll make your ears bleed!

Just rambling/sharing - not really trying to make any particular point (other than the fact that this might be quite difficult!).

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-12 14:56:39
Beta v1.1.2f and beta v1.1.2g are essentially the same other than the inclusion of the --randombits parameter, so I would recommend that beta v1.1.2g be used for testing.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-12 15:03:43
OK.

Does anyone have any samples that are problematic at -q 2.5? Or -q 0.0?

I fear these might be quite different from the type of samples that BORK has been ABXing, but we have to start somewhere.

Nick, can -q go negative? I know it can't now, but what I mean is, is it possible to do worse than -q 0? Just to hear how the thing goes wrong?

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-12 15:09:13
I can certainly create a negative extension to the scale for the purposes of evaluation. I would expect this to go down to -q -2.0 or --nasty (). I will extrapolate the parameters a bit more steeply than the delta from -q 2.0 to -q 0.0.

From memory Sauvage78 can ABX "01 - Ginnungagap - The Black Hole" at -q 1.5.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-12 15:38:10
lossyWAV beta 1.1.2h attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-12 16:38:50
Even -N is quite subtle - I'm not complaining though.

Where's "01 - Ginnungagap - The Black Hole"?

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-12 16:45:04
ygem.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-12 20:19:46
There's a bug in beta v1.1.2h which causes the quality selection mechanism not to work properly. I would suggest that any testing is performed using beta v1.1.2g in the interim until I can find and fix the problem which should be rectified in beta v1.1.2i.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-12 20:32:37
Nick C.

I am redoing the test files NOW with the G version as requested,
and starting testing with testers today.

I gather using ONLY --standard as an option is what you want, if other parameters needed please tell me now.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-12 20:50:55
I would recommend only using the available quality presets at present. As David has indicated to train the listeners with respect to types of artifacts it may be necessary to use --portable or even --zero in the first instance.

As indicated in the post immediately above your, I am currently bug-hunting and would request that you use beta v1.1.2g in the interim. (Now again available at post #1 in this thread)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-12 22:21:55
lossyWAV beta v1.1.2i attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-12 23:46:07
lossyWAV beta v1.1.2i attached to post #1 in this thread.


Thanks for all the hard work Nick !
First All test File Links:

new 2496 File : http://rapidshare.com/files/197392375/3_vi...ANDARD.rar.html (http://rapidshare.com/files/197392375/3_views_2496_-_1.1.2g_--STANDARD.rar.html)

not sure if these can still be downloaded , if anyone that downloaded them can put these somewhere for other testers thatll be great.

Samba e Amor http://rapidshare.com/files/194915589/112e...ping_0.rar.html (http://rapidshare.com/files/194915589/112e_Insane_Shaping_0.rar.html)
First Three Files : http://rapidshare.com/files/191350600/TEST_FILES.rar.html (http://rapidshare.com/files/191350600/TEST_FILES.rar.html)

Here is today's test results:
Tester was 28 years old, female , non audio pro, non musician (scientist), with music listening habits that consist of 90% in car only music listening, mostly radio.

Did play an instrument at a young age , as a hobby, quit around 14.

Tests were done with a training phase beforehand , with a mandatory 10 run training or more,to get accustomed to the music.

None of the tracks playe were familiar to her.
After the first non trial , I explained what to listen for for each track tested,
& sometimes suggested start points.

These are the tester's first ever AB tests.

112e Insane Shaping 0 results

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/12 21:48:50

File A: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bebel Gilberto - Samba e Amor - SLICE.flac
File B: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bebel Gilberto - Samba e Amor - SLICE.lossy.flac

21:48:50 : Test started.
21:49:14 : 01/01  50.0%
21:49:29 : 02/02  25.0%
21:50:36 : 03/03  12.5%
21:52:14 : 04/04  6.3%
21:53:34 : 04/05  18.8%
21:54:03 : 05/06  10.9%
21:54:24 : 05/07  22.7%
21:54:42 : 06/08  14.5%
21:54:53 : 06/09  25.4%
21:55:08 : 07/10  17.2%
21:55:30 : 07/11  27.4%
21:55:48 : 07/12  38.7%
21:56:16 : 08/13  29.1%
21:57:09 : 08/14  39.5%
21:57:58 : 09/15  30.4%
21:58:20 : 09/16  40.2%
21:58:57 : 09/17  50.0%
21:59:29 : 09/18  59.3%
22:00:01 : 10/19  50.0%
22:00:23 : 11/20  41.2%
22:00:39 : Test finished.

----------
Total: 11/20 (41.2%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/12 22:01:30

File A: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bebel Gilberto - Samba e Amor - SLICE.flac
File B: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bebel Gilberto - Samba e Amor - SLICE.lossy.flac

22:01:30 : Test started.
22:02:19 : 01/01  50.0%
22:02:33 : 02/02  25.0%
22:02:47 : 02/03  50.0%
22:03:07 : 02/04  68.8%
22:03:33 : 02/05  81.3%
22:03:39 : 03/06  65.6%
22:03:58 : 04/07  50.0%
22:04:10 : 05/08  36.3%
22:04:55 : 06/09  25.4%
22:05:33 : 06/10  37.7%
22:05:52 : 06/11  50.0%
22:06:24 : 07/12  38.7%
22:06:39 : 07/13  50.0%
22:06:55 : 07/14  60.5%
22:07:15 : 08/15  50.0%
22:07:27 : 08/16  59.8%
22:07:41 : 08/17  68.5%
22:08:03 : 09/18  59.3%
22:08:26 : 09/19  67.6%
22:08:36 : 09/20  74.8%
22:09:02 : Test finished.

----------
Total: 9/20 (74.8%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/12 22:09:15

File A: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bebel Gilberto - Samba e Amor - SLICE.flac
File B: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bebel Gilberto - Samba e Amor - SLICE.lossy.flac

22:09:15 : Test started.
22:09:35 : 01/01  50.0%
22:09:41 : 01/02  75.0%
22:09:45 : 02/03  50.0%
22:09:50 : 03/04  31.3%
22:09:58 : 04/05  18.8%
22:10:03 : 05/06  10.9%
22:11:55 : Trial reset.
22:12:40 : 00/01  100.0%
22:12:52 : 01/02  75.0%
22:13:03 : 02/03  50.0%
22:13:12 : 03/04  31.3%
22:13:28 : 04/05  18.8%
22:13:38 : 05/06  10.9%
22:13:49 : 06/07  6.3%
22:14:16 : 06/08  14.5%
22:14:31 : 06/09  25.4%
22:14:41 : 06/10  37.7%
22:14:51 : 07/11  27.4%
22:15:00 : 08/12  19.4%
22:15:09 : 09/13  13.3%
22:15:55 : 09/14  21.2%
22:16:49 : 10/15  15.1%
22:17:00 : 11/16  10.5%
22:17:08 : 12/17  7.2%
22:17:18 : 12/18  11.9%
22:17:32 : 13/19  8.4%
22:17:36 : 13/20  13.2%
22:17:53 : Test finished.

----------
Total: 18/26 (3.8%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/12 22:18:29

File A: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bebel Gilberto - Samba e Amor - SLICE.flac
File B: F:\MP3\Favourite\TEST FILES\112e Insane Shaping 0\Bebel Gilberto - Samba e Amor - SLICE.lossy.flac

22:18:29 : Test started.
22:19:02 : 00/01  100.0%
22:19:16 : 00/02  100.0%
22:19:21 : Trial reset.
22:19:56 : 01/01  50.0%
22:20:03 : 02/02  25.0%
22:20:08 : 03/03  12.5%
22:20:13 : 04/04  6.3%
22:20:25 : 05/05  3.1%
22:20:34 : 06/06  1.6%
22:20:48 : 06/07  6.3%
22:20:58 : 06/08  14.5%
22:21:05 : 07/09  9.0%
22:21:15 : 08/10  5.5%
22:21:22 : 08/11  11.3%
22:21:34 : 08/12  19.4%
22:21:40 : 09/13  13.3%
22:21:45 : 09/14  21.2%
22:21:57 : 09/15  30.4%
22:22:05 : 10/16  22.7%
22:22:11 : 11/17  16.6%
22:22:18 : 11/18  24.0%
22:22:25 : 12/19  18.0%
22:22:33 : 12/20  25.2%
22:22:49 : Test finished.

----------
Total: 12/22 (41.6%)


1.1.2g --STANDARD Results, conducted AFTER the above
Note she kept hitting the wrong reset button sometimes to choose a new start point,
but the results ARE INCLUDED in the logs , & are added to the overall score,
(I was not paying enough attention monitoring the progress.)

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/12 22:40:18

File A: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bebel Gilberto - Samba e Amor - SLICE.flac
File B: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bebel Gilberto - Samba e Amor - SLICE.lossy.flac

22:40:18 : Test started.
22:40:53 : 01/01  50.0%
22:41:05 : 01/02  75.0%
22:41:14 : 01/03  87.5%
22:41:53 : 01/04  93.8%
22:42:10 : 02/05  81.3%
22:42:50 : 02/06  89.1%
22:43:19 : 03/07  77.3%
22:43:26 : 03/08  85.5%
22:43:31 : 03/09  91.0%
22:43:39 : Trial reset.
22:43:50 : 01/01  50.0%
22:43:57 : 02/02  25.0%
22:44:05 : 03/03  12.5%
22:44:10 : 03/04  31.3%
22:44:22 : 03/05  50.0%
22:44:29 : 03/06  65.6%
22:44:34 : 03/07  77.3%
22:45:05 : 04/08  63.7%
22:45:12 : 04/09  74.6%
22:45:16 : 05/10  62.3%
22:45:23 : 06/11  50.0%
22:45:30 : 07/12  38.7%
22:45:36 : 08/13  29.1%
22:45:42 : 09/14  21.2%
22:45:47 : 09/15  30.4%
22:45:54 : 10/16  22.7%
22:46:03 : 10/17  31.5%
22:46:08 : 10/18  40.7%
22:46:17 : 10/19  50.0%
22:46:23 : 10/20  58.8%
22:46:43 : Test finished.

----------
Total: 13/29 (77.1%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/12 23:31:44

File A: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Off The Wall.flac
File B: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Off The Wall.lossy.flac

23:31:44 : Test started.
23:32:11 : 00/01  100.0%
23:32:14 : Trial reset.
23:32:26 : 00/01  100.0%
23:32:27 : Trial reset.
23:33:14 : 01/01  50.0%
23:33:33 : 02/02  25.0%
23:33:39 : 03/03  12.5%
23:33:53 : 03/04  31.3%
23:34:01 : 03/05  50.0%
23:34:10 : 04/06  34.4%
23:34:18 : 04/07  50.0%
23:34:37 : 05/08  36.3%
23:35:01 : 05/09  50.0%
23:35:08 : 06/10  37.7%
23:35:18 : 06/11  50.0%
23:35:31 : 07/12  38.7%
23:35:50 : 08/13  29.1%
23:35:56 : 09/14  21.2%
23:36:01 : 10/15  15.1%
23:36:06 : 11/16  10.5%
23:36:12 : 12/17  7.2%
23:36:21 : 12/18  11.9%
23:36:27 : 12/19  18.0%
23:36:32 : 13/20  13.2%
23:36:46 : Test finished.

----------
Total: 13/22 (26.2%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/13 00:10:24

File A: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bob Berg - Oleo.flac
File B: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bob Berg - Oleo.lossy.flac

00:10:24 : Test started.
00:10:43 : 01/01  50.0%
00:11:22 : 02/02  25.0%
00:11:46 : 03/03  12.5%
00:12:06 : 04/04  6.3%
00:12:13 : 05/05  3.1%
00:12:19 : 06/06  1.6%
00:12:26 : 07/07  0.8%
00:12:32 : 07/08  3.5%
00:12:43 : 08/09  2.0%
00:12:48 : 08/10  5.5%
00:13:55 : 09/11  3.3%
00:14:03 : 10/12  1.9%
00:14:23 : 11/13  1.1%
00:15:06 : 11/14  2.9%
00:15:15 : 12/15  1.8%
00:15:35 : 13/16  1.1%
00:15:46 : 13/17  2.5%
00:15:58 : 14/18  1.5%
00:16:12 : 14/19  3.2%
00:16:21 : 14/20  5.8%
00:16:50 : Test finished.

----------
Total: 14/20 (5.8%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/13 00:17:17

File A: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\3 Views 2496.flac
File B: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\3 Views 2496.lossy.flac

00:17:17 : Test started.
00:18:19 : 01/01  50.0%
00:19:49 : 02/02  25.0%
00:19:54 : 02/03  50.0%
00:20:05 : 03/04  31.3%
00:20:16 : 03/05  50.0%
00:20:22 : 03/06  65.6%
00:20:50 : 04/07  50.0%
00:20:55 : 05/08  36.3%
00:21:08 : 06/09  25.4%
00:21:16 : 07/10  17.2%
00:21:22 : 07/11  27.4%
00:21:29 : 08/12  19.4%
00:22:00 : 09/13  13.3%
00:22:15 : 10/14  9.0%
00:22:27 : 11/15  5.9%
00:23:05 : 12/16  3.8%
00:23:12 : 12/17  7.2%
00:23:32 : 12/18  11.9%
00:23:39 : 12/19  18.0%
00:23:44 : 12/20  25.2%
00:23:59 : Test finished.

----------
Total: 12/20 (25.2%)


END
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-02-13 08:47:21
I'd like to help with ABXing. I've not had much success in the past with identifying shortcomings in Lossywav. I think that may be partly due to my PC audio setup not being ideal - the environment is a bit too noisy and my headphones are open-backed so computer fans etc are audible when I'm wearing them. I'd like to do the testing on my main hifi but can't connect my PC to my hifi (it's 20 meters away with two brick walls in between). If I were to burn the test files to CD - say 10 times - and use the random play facility of my CD player with my wife noting which track was actually playing while I was listening  (the CD player is behind me so I can't see the display myself) would that be satsifactory? It seems to be double-blind to me.

Are we going to compile a set of tesfiles or use Bork's? I could provide some "simple" recordings - solo harp, acoustic guitar unaccompanied voices (renaissance choral). Alternatevly I've got the usual mix of pop, rock, folk, classical etc if anybody wants to suggest something
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-13 11:25:54
...Here is today's test results:
Tester was 28 years old, female ...

To me the 1.1.2g --standard result on Oleo is pretty impressive.
I'd really like to see a comparison between version 1.1.0 (or 1.1.0b) and 1.1.2g, both using --shaping 0, because IMO it would be good to see proven wrong or confirmed what so far is my personal suspicion due to the significantly more audible noise of 1.1.2 with Amor e Samba compared to 1.1.0.
Can you please suggest to your tester to repeat the test using both 1.1.0 and 1.1.2, both with --standard --shaping 0?
LossyWAV's signal analysis did change from 1.1.0 to 1.1.2, and though it seems not very likely that this change was essential we don't really know until a meaningful comparison was made (my own comparison wasn't meaningful).

Thank you for all your efforts.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-13 15:52:53
To me the 1.1.2g --standard result on Oleo is pretty impressive.
It is, but it needs repeating (by the same listener, or another). Why? Because there's 8 tests. The chance of at least one of them hitting 5% is about 1/3rd due to pure chance (I think - someone who did stats less than 15 years ago can correct me on this!).


FWIW I've tried to ABX Oleo at -q 0 -s 0 with both 1.1.0 and 1.1.2i, and I can't hear any difference. Both are transparent to me.

EDIT:

Whereas 01 - Ginnungagap - The Black Hole.wav has audible hiss added at -q 0 -s 0 in both 1.1.0 and 1.1.2i. It's in the drum hits just before the guitar chord (e.g. 6.3 seconds). There's already a little burst of hiss there in the original - lossyWAV just makes it louder.

Of course -q 0 isn't supposed to be transparent - it's the lowest possible setting (until Nick added --nasty!)

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2
2009/02/13 15:56:15

File A: D:\audio\audio\codec testers\lossyWAV\01 - Ginnungagap - The Black Hole.wav
File B: C:\Program Files\lw112i\01 - Ginnungagap - The Black Hole.lossy 112i -q 0 -s 0.wav

15:56:15 : Test started.
15:56:54 : 01/01  50.0%
15:57:01 : 02/02  25.0%
15:57:06 : 03/03  12.5%
15:57:16 : 04/04  6.3%
15:57:32 : 05/05  3.1%
15:57:38 : 06/06  1.6%
15:57:47 : 07/07  0.8%
15:58:12 : 08/08  0.4%
15:58:13 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2
2009/02/13 15:59:09

File A: D:\audio\audio\codec testers\lossyWAV\01 - Ginnungagap - The Black Hole.wav
File B: C:\Program Files\lw110\01 - Ginnungagap - The Black Hole.lossy 110 -q 0 -s 0.wav

15:59:09 : Test started.
15:59:41 : 01/01  50.0%
15:59:50 : 02/02  25.0%
15:59:59 : 02/03  50.0%
16:00:09 : 03/04  31.3%
16:00:15 : 04/05  18.8%
16:00:20 : 05/06  10.9%
16:00:33 : 06/07  6.3%
16:00:41 : 07/08  3.5%
16:00:42 : Test finished.

----------
Total: 7/8 (3.5%)


I couldn't hear any difference between 1.1.0 and 1.1.2i - both had the same amount of the same artefact.

(The different score is due to me staring out of the window with boredom and clicking the wrong part of the screen! (Play B instead of Play Y - so "Y" sounded like "B", whereas Y was actually A!))

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-13 19:35:06
2BDecided, I agree with you that in case there is something wrong it should be clearly audible using -q 0.

So I wanted to try on my own your ABX test of Oleo using 1.1.0 and 1.1.2i with -q 0 -s 0.
Well, to me too both results were very good, but they are not identical. At second 2.8...4.9 I could ABX the 1.1.2i result 7/10 with my first test and 8/10 with my second. I could not ABX the 1.1.0 result. The difference of 1.1.2i compared with the original doesn't sound like hiss or noise to me, to me it sounds like the balance of the various frequencies has changed a tiny bit, probably the same thing BORK described with other words.

EDITED:
Text about error signal removed - I must have been victim to placebo. After trying to ABX the error signals I can't say there is an audible difference to me. I reencoded OLEO at quality --insane. Now I can ABX differences between the 1.1.0 and the 1.1.2i error signal but without being able to prefer one error signal over the other. I found a spot where to me the 1.1.0 error is a little bit less pronounced, but at another spot it was the other way around. Differences are subtle anyway.
Which made me wonder whether the changed analysis of the last 1.1.2 versions is closer to that of 1.1.0. So I encoded Samba e Amor with 1.1.2i --insane --shaping 0. The error of 1.1.2i is in this case easy to ABX as being louder than that of 1.1.0, as was in my previous 1.1.2 vs. 1.1.0 test. Difference is not subtle.
Title: lossyWAV 1.2.0 Development Thread
Post by: B0RK on 2009-02-13 21:20:24
To me the 1.1.2g --standard result on Oleo is pretty impressive.
It is, but it needs repeating (by the same listener, or another). Why? Because there's 8 tests. The chance of at least one of them hitting 5% is about 1/3rd due to pure chance (I think - someone who did stats less than 15 years ago can correct me on this!).


Well, I guess we might have some difference of perception of the interpretation of the test results and even tests goals.

AB tests are notriouly known (& more widely used) as a better tool to disprove something, rather then prove it.

When used 'positively' , like promoting a competitor product for example,
no one minds them , but that's not the case usually.

Our case is different, (at least The way I see it  ..), In our case we are trying to test if casual testers can identify the compressed wavefile with any kind of consistency  AT ALL, once he is made aware of the difference in the files, & given an example of it,& a verbal description.

If what we aim here to show that a tester has higher failure rates percentage wise,
then we can stop testing right now,(as that is the most common AB test result) cause all we need to do is have 20 runs in a row, that'll take care of it...


so My baseline for the tests was to find out if a tester with good ears,
but no experience, can show ANY consistency identifying the differences in more then a single 'lucky' run.

I don't believe in a 'lucky' run of 14/20.
For me the test ended there, and had the tester failed 20 complete failures afterwards,
I would have some doubt in the validity of the score , but still regard it as a 'win' for him/her, as the odds are just too slim for that to happen to a pro,
ont to mention a casual listener.

If a few of these listeners can do it in one of the most important modes we want to perfect lossywav for , then maybe improvements are in order

The other results may help show what's the tester's average,if need be.


For the hell of it , I did 3 runs of MINDLESS CLICKING short 20 trial tests:
THESE ARE NOT REAL ABXs - I just clicked without even listening.
Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/13 22:58:09

File A: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bebel Gilberto - Samba e Amor - SLICE.flac
File B: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bebel Gilberto - Samba e Amor - SLICE.lossy.flac

22:58:09 : Test started.
22:58:16 : 00/01  100.0%
22:58:17 : 01/02  75.0%
22:58:17 : 02/03  50.0%
22:58:18 : 03/04  31.3%
22:58:19 : 03/05  50.0%
22:58:20 : 03/06  65.6%
22:58:21 : 03/07  77.3%
22:58:21 : 03/08  85.5%
22:58:23 : 03/09  91.0%
22:58:25 : 03/10  94.5%
22:58:26 : 03/11  96.7%
22:58:28 : 03/12  98.1%
22:58:29 : 03/13  98.9%
22:58:30 : 04/14  97.1%
22:58:31 : 04/15  98.2%
22:58:32 : 04/16  98.9%
22:58:33 : 04/17  99.4%
22:58:34 : 05/18  98.5%
22:58:35 : 06/19  96.8%
22:58:36 : 06/20  97.9%
22:58:40 : Test finished.

----------
Total: 6/20 (97.9%)

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/13 22:58:50

File A: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bebel Gilberto - Samba e Amor - SLICE.flac
File B: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bebel Gilberto - Samba e Amor - SLICE.lossy.flac

22:58:50 : Test started.
22:58:52 : 00/01  100.0%
22:58:53 : 01/02  75.0%
22:58:54 : 01/03  87.5%
22:58:56 : 01/04  93.8%
22:58:57 : 02/05  81.3%
22:58:58 : 03/06  65.6%
22:58:59 : 03/07  77.3%
22:59:00 : 03/08  85.5%
22:59:01 : 04/09  74.6%
22:59:02 : 05/10  62.3%
22:59:03 : 06/11  50.0%
22:59:04 : 06/12  61.3%
22:59:06 : 06/13  70.9%
22:59:07 : 07/14  60.5%
22:59:08 : 07/15  69.6%
22:59:09 : 07/16  77.3%
22:59:11 : 07/17  83.4%
22:59:12 : 07/18  88.1%
22:59:13 : 07/19  91.6%
22:59:14 : 07/20  94.2%
22:59:16 : Test finished.

----------
Total: 7/20 (94.2%)

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2 beta 2
2009/02/13 22:59:28

File A: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bebel Gilberto - Samba e Amor - SLICE.flac
File B: F:\MP3\Favourite\TEST FILES\1.1.2g --STANDARD\Bebel Gilberto - Samba e Amor - SLICE.lossy.flac

22:59:28 : Test started.
22:59:30 : 00/01  100.0%
22:59:31 : 01/02  75.0%
22:59:32 : 02/03  50.0%
22:59:33 : 02/04  68.8%
22:59:33 : 02/05  81.3%
22:59:34 : 03/06  65.6%
22:59:36 : 04/07  50.0%
22:59:37 : 04/08  63.7%
22:59:39 : 04/09  74.6%
22:59:40 : 04/10  82.8%
22:59:42 : 05/11  72.6%
22:59:43 : 06/12  61.3%
22:59:44 : 06/13  70.9%
22:59:45 : 06/14  78.8%
22:59:46 : 06/15  84.9%
22:59:47 : 07/16  77.3%
22:59:48 : 07/17  83.4%
22:59:49 : 07/18  88.1%
22:59:50 : 07/19  91.6%
22:59:51 : 07/20  94.2%
23:00:00 : Test finished.

----------
Total: 7/20 (94.2%)


As this is mindless clicking , no REAL ERRORS can be made,
real world tests where real errors can be made ,can even end up looking worse.
Looking at 3 or more consecutive results like the above using foobar , can suggest no differences are found by the tester.

In other words , when a tester cannot hear it , you probably will not get 8 results like the (Real) ones posted in my previous post.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-13 21:47:23
BORK,

With purely random clicking and a fixed number of trials, you'll get a result which yields 5% or lower (i.e. "significant") 1 time in 20 (on average). That's what 5% is - 1 in 20.

"The chance of hitting this purely by chance is 5%" - that's what it means - no more, no less.


That's my understanding anyway. There are far far far smarter statisticians around here.


I'm not discounting it though - far from it - taken together with your results, it really is significant.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-02-13 21:49:03
@halb27,

Thanks for that test report, and for giving other people an idea of what/where/when to listen.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-18 21:55:11
lossyWAV beta 1.1.2j attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-19 08:14:28
Thanks a lot, Nick.
I prefer the old spreading function (though nothing is really known whether it is better in a general view -  there's just an isolated suspicion for this),  but I'd also like to stay with current development and not remain with 1.1.0.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-19 16:02:18
Thanks a lot, Nick.
I prefer the old spreading function (though nothing is really known whether it is better in a general view -  there's just an isolated suspicion for this),  but I'd also like to stay with current development and not remain with 1.1.0.

Just tried Samba e Amor -I -s 0 -O. Error signal is now essentially the same as was when using 1.1.0 -I -s 0. Thanks again.
Title: lossyWAV 1.2.0 Development Thread
Post by: traymond on 2009-02-20 07:18:43
Excuse my lack of knowledge on this subject but:

I know this is an old language but how could I go about coding to play both 4 or 8-bit Riff Wave files in 8-bit assembly langauge?   

I would need the loading process, converting to either 4 or 8-bit process, and the playing process.

I have a hobby with an older 8-bit computer I would like to try this on, I assume your coding here must be C code. 

I have looked quite a bit with Google and came upon this forum and decided to ask.   

Laugh at me if you want to but I want to learn the skill of learning 6502 assembly and would like to try playing Wave files.   

Thank you.
traymond
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-20 08:18:32
You can find links to WAV format specification documents on en.wikipedia.org;

ps. lossyWAV is coded in Delphi (Pascal) with a liberal sprinkling of IA-32 / x87 assembly language.
Title: lossyWAV 1.2.0 Development Thread
Post by: Gregory S. Chudov on 2009-02-21 04:23:29
I noticed that http://lossywav.svn.sourceforge.net/viewvc/lossywav/ (http://lossywav.svn.sourceforge.net/viewvc/lossywav/) has not been updated since initial checkin, and the sourcecode link in the first post is also quite old.
Would be nice to see recent changes in svn, so that i could fix C# implementation accordingly, since there was at least one major change - a fix to the noise shaping algorithm.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-21 08:53:05
I'll try to release beta 1.1.3 source this weekend.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-22 21:51:01
lossyWAV beta 1.1.3 attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-24 13:08:19
Following the reinstatement of the old spreading function in beta 1.1.3, I have been looking at a hybrid variant which adds the core of the new method with the process of the old. This slightly lowers bits-to-remove for some but not all of the files in my 55 problem sample set resulting in a fraction of 1kbps small increase in FLAC resultant bitrate. There is no real speed penalty associated with the variant spreading function when compared to either the new or old versions.

I will post beta 1.1.3c this evening which will incorporate -V or --varspread parmeter to allow the variant spreading function to be tested.

Following some discussion earlier in this thread, I will also take a stab at rewording the text describing the lossyWAV method itself.

[edit]
Code: [Select]
lossyWAV beta v1.1.3c indicative bitrates.
|-----|----------|----------|----------|----------|----------|----------|----------|
| SPR |--insane  |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
|-----|----------|----------|----------|----------|----------|----------|----------|
| NEW | 651 kbps | 581 kbps | 505 kbps | 417 kbps | 313 kbps | 252 kbps | 205 kbps |
| OLD | 654 kbps | 584 kbps | 508 kbps | 425 kbps | 322 kbps | 257 kbps | 208 kbps |
| VAR | 659 kbps | 590 kbps | 514 kbps | 432 kbps | 327 kbps | 260 kbps | 210 kbps |
|-----|----------|----------|----------|----------|----------|----------|----------|
Table indicates resultant FLAC -b 512 -5 bitrates from processing of my 55 problem sample set.[/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-24 19:32:09
Thanks for your work, Nick.
Can you describe a bit the new hybrid variant? I wasn't aware you introduced a new process with the new spreading function. I thought you just exchanged the arithmetic averaging for an average which gives a greater weight to the center bin (and use only an uneven number of bins so that there is a center bin).
I'll try your new variant within the next days. Increasing bitrate with problem samples is welcome especially if the bitrate increase is more with problems than with regular music.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-24 20:16:07
The variant is based on the old method in that it averages an integer number of bins. The new method takes a proportion of the two bins immediately adjacent to the centre bin and calculates the average [i.e. result = (fraction x (left_bin + right_bin) + centre_bin) / (1 + 2 x fraction)]. This takes a proportion between 0 and the maximum for that particular fft_length as the bin-frequency increases.

The variant takes the minimum of the old bin average and the new bin average as the bin-result. The lowest of there bin-results is used, as is the average of all of the bin-results calculated.

lossyWAV beta 1.1.3c attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-24 21:53:09
Thanks, Nick, for the explanation. So the hybrid variant makes things more defensive. I like this in case bitrate of regular music doesn't increase a lot.
I couldn't resist and tried 1.1.3c on my regular tracks test set:

1.1.3c --portable: 376 kbps
1.1.3c --portable -O: 375 kbps
1.1.3c --portable -V: 383 kbps
1.1.3c --standard: 474 kbps
1.1.3c --standard -O: 466 kbps
1.1.3c --standard -V: 474 kbps

To me this is an insiginificant increase in bitrate.
I was astonished a bit that bitrate of -O is lower than that without this option.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-24 22:17:41
Thanks for the testing - I processed my 10 album test set and got 377kbps for -P and -P -O, but 385kbps for -P -V.

Now we need some critical ears to come forward and volunteer to ABX....
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-25 21:45:59
It's been quite a while that I tested my problem sample set consisting of badvilbel, bibilolo, dither_noise_test, eig, furious, keys_1644ds, S37_OTHERS_MartenotWaves_A, under_the_boardwalk.

I started with the lowest setting -q 0.0 -V which yields an average bitrate of 280 kbps on average for my regular music test set (I still use FLAC -l 6 -e -m -r 2 which is a little bit slower than -5 but can save roughly 1 to 2 kbps when using --standard). Quality of the problem samples in general is astonishingly high. Pretty bad is only eig and furious. furious has a strange error in the 2.0...2.5 second range.

Using -q 0.5-V (301 kbps for my regular music test set) improves things a lot, but the issues with eig and furious are still easily abxable.

-q 1.0 -V (322 kbps on average for my regular test set) makes the problems acceptable to me though I can still abx them.

With -q 1.5 -V (344 kbps for my regular music) I could not abx the problems any more. I guess other testers will be able to abx them, may be I can do it when using a lot more effort (which however wouldn't have anything in common with my usual listening and so would be only of a theoretical value to me).

So from this (and listening to some regular music) I think -q 1.5 -V is a good setting for portable use. Or - in other words - --portable has already a certain safety margin for its intended use.

Guess I'll switch from --standard to --portable, with my new DAP I'm short on storage space anyway.

Thanks a lot, Nick, for your great work.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-02-25 21:52:25
Many thanks, as usual, for your listening time. I am pleased that --portable has proven in this instance to have an acceptable safety margin. The bitrates you are proposing using at -q 1.5 -V are also palatable at circa 350kbps for DAP use - less than 10% more than max.MP3. With the right player and FLAC's low processing load on decoding this may give a good battery life for your player.

As an aside, I recently purchased a Sansa Fuze which I am extremely impressed with. It plays FLAC (after a firmware update) and presents itself as two USB drives when plugged into my PC - 1 x 8GB (internal) and 1 x 16GB (microSDHC). The library function is seamless across the two storage areas so finding tracks (albums in my case) is simple.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-02-25 23:10:32
... I recently purchased a Sansa Fuze which I am extremely impressed with ... 1 x 8GB (internal) and 1 x 16GB (microSDHC). ...

With respect to storage space my Meizu M6 SL certainly wasn't a good choice. On the other hand sound quality is great - and probably it's not bad to get used to lossyWAV and mp3 settings a bit lower than what I was used to so far. Quality is excellent anyway.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-05 07:33:59
lossyWAV beta 1.1.3d attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-03-06 07:38:44
Thanks, Nick, for your improvements.
Title: lossyWAV 1.2.0 Development Thread
Post by: rbrito on 2009-03-06 10:54:12
I noticed that http://lossywav.svn.sourceforge.net/viewvc/lossywav/ (http://lossywav.svn.sourceforge.net/viewvc/lossywav/) has not been updated since initial checkin, and the sourcecode link in the first post is also quite old.
Would be nice to see recent changes in svn, so that i could fix C# implementation accordingly, since there was at least one major change - a fix to the noise shaping algorithm.


Hi there.

I'm back from some quite nasty personal problems (including very serious health problems). []

I noticed that since my first proposal to rewrite the code in C, some higher level implementations were made. This is good. I think, though, that it would be a good thing to have a C version of the code, since it would be good to profile the programs with both some already known code (especially for the FFT) and with libraries such as the liboil (oil = Optimized Inner Loops).

This would mean that we would have a moderately fast binary, portable to many operating systems and platforms of an entirely free tool that seems to be just what the doctor prescribed.

Well, Nick, where is the latest source code? Do you know how to use a version control system like SVN, where I put the source? It would be nice if we tagged the versions and created diffs so that we can see how the code progresses.


Regards, Rogério Brito.

P.S.: I see many threads here involving the lossyWAV development and I must confess that I'm slightly lost with all these releases.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-06 21:15:30
Hi Rogério,

I'm sorry to hear that you have had some problems - I hope that your health is improving and that life is getting better.

Gregory S. Chudov is leading the C conversion and [JAZ] has been working on a J version.

The latest source code is contained in the ZIP file "lossyWAV_Source_20090405_0732_beta_1.1.3d.zip" (obvious typo in the name ) which is attached to post #1 in this thread. I tend to attach revised source whenever I reach a stable beta.

I have no familiarity with diff files so have not been creating them. Equally, I have never used the SVN.

This thread is the only thread where any development is occurring - the other development and release threads are dormant as far as I'm concerned (unless someone revives them inadvertently).

On the development front I am awaiting feedback on the variant spreading function - I consider that it should be the only one remaining in the next version.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-17 13:48:37
I've been working on the code (as ever) and have made a few speed improvements in the Delphi code (of the order of a 25% reduction in processing time for --zero).

At the same time as this I have been working through the variant spreading function and am satisfied that it achieves its aim.

As an aside, I processed my 10 album test set at --portable --varspread --analyses 5 and the resultant FLAC -b 512 -5 bitrate was 397kbps (v1.1.0b --portable: 376kbps; v1.1.3d --portable --varspread: 385kbps). Although the processing time increased by about 60%, I feel that using three additional FFT lengths (128, 256 and 512 samples) catches more low dips in the noise floor.

So, in the imminent run-up to v1.2.0 I propose that the variant spreading function is adopted as the standard and that the existing "new" and "old" versions are removed. Also, SebastianG's proposed dynamic noise shaping method is to be deferred until v1.3.0.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-03-17 15:27:37
...So, in the imminent run-up to v1.2.0 I propose that the variant spreading function is adopted as the standard and that the existing "new" and "old" versions are removed. Also, SebastianG's proposed dynamic noise shaping method is to be deferred until v1.3.0.

I welcome this very much.

... I processed my 10 album test set at --portable --varspread --analyses 5 and the resultant FLAC -b 512 -5 bitrate was 397kbps (v1.1.0b --portable: 376kbps; v1.1.3d --portable --varspread: 385kbps).
...

What makes -V bitrate of your version under investigation increase compared to the v1.1.3d result?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-17 16:05:49
The bitrate increase is due to the use of the additional analyses, the v1.1.3d -P -V result is for the standard two analyses.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-03-17 17:58:23
The bitrate increase is due to the use of the additional analyses, the v1.1.3d -P -V result is for the standard two analyses.

I welcome going more defensive especially as also to me -P now is the way to go.
However I feel that we should stop this some day cause otherwise we end up with a bitrate for -P which once was that of -S.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-17 19:18:10
I think it best that the user should select the number of analyses (between 2 (default) and 5 (maximum)) using the -a or --analyses parameter.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-03-17 20:15:24
I think it best that the user should select the number of analyses (between 2 (default) and 5 (maximum)) using the -a or --analyses parameter.

To me this is a good solution.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-17 20:26:47
I'll put out beta v1.1.3e tomorrow as a final beta before v1.2.0.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-18 13:07:19
lossyWAV beta v1.1.3e attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-03-18 16:59:06
I welcome going more defensive especially as also to me -P now is the way to go.
However I feel that we should stop this some day cause otherwise we end up with a bitrate for -P which once was that of -S.

What you say were also my thoughts when Nick introduced --varspread. Yes it is more defensive, but so is choosing a higher -q setting .
Everytime the bitrates changes, users should (in theory) find their sweetspot (size/quality) again.

Just a thought, but don't let it stop progress 
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-18 17:07:20
I *think* that the variant spreading function in conjunction with a slightly lower -q setting will give an overall better result in that the combination of spreading functions will spot dips in the noise floor better than either in isolation.

The resultant bitrate of the output from beta v1.1.3e at --quality 2.2 is the same as that for v1.1.0b --portable for my 55 problem sample set.
Title: lossyWAV 1.2.0 Development Thread
Post by: monoton on 2009-03-18 17:38:53
Hello... is there any possibility to force even greater quality loss for lossy wav? Actually I'm yet having a hard time to spot the difference even in the "awful" setting (when using "-q 0" for encoding) though I can spot artifacts on mp3-files. I guess its "noise" (reduction) is a bit different. Generally I'd wonder how the signal would sound like with really low quality settings.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-18 18:28:01
The lowest quality setting is actually -q -4 or -A or --awful.

The artifacts are *very* obvious.
Title: lossyWAV 1.2.0 Development Thread
Post by: monoton on 2009-03-18 22:49:47
Ok.. I used jLossyWav before and that didn't accept negative numbers for quality settings.

Now that I've got the original lossyWav version running I can see the difference. (I would have appreciated some link or hint info in the "use with foobar2000" section of the wiki for some noob like me what to install and set where though, that've saved me an hour or two...)

However, also partly to my defense, even with these settings my first test track wasn't so easy to identify as my other. I've firstly tried it on AC/DC's "Smash N Grab" (Black Ice) and now that I know from the second, orchestral track, where it was clearly audible, I'm quite astonished how good it still sounded!

I think I have two explanations for that: even with -q -4 ACDC was "only" compressed to around 25% of its original size, the orchestral track went to 14% of original size. The other thing is of the nature of these songs: ACDC at making their best to give loudness war more fuel and the orchestral, while still being heavy for an orchestral track, but slight or no compression applied.
Title: lossyWAV 1.2.0 Development Thread
Post by: monoton on 2009-03-18 23:25:02
Hmm... I tried to create the correction file, but it gives me an error:

"Conversion failed: The encoder has terminated prematurely with code 1; please re-check parameters"
... lossywav - -A -f -correction --silent --stdout ...

is what I'm using with foobar2000. Am I doing something wrong here? It neither works with --correction nor -C (I saw the caps) but works if I omit -C or --correction and not include these parameters.

Thanks for your help in advance!
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-19 06:14:21
Creation of the correction file is incompatible with piped output (as the two output WAV files require to be written simultaneously).
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-03-19 17:41:38
Thank you Nick, for version 1.1.3e.
I didn't want to do a listening test now, but just compared the results of 1.1.3d -P -V with those of 1.1.3e -P.
I was a bit astonished that the results weren't bit-identical, but looking at the blockwise bits to remove of a few short tracks I saw that there is nearly no difference and the new version was always more defensive (in case of differences).
I can also confirm that using -a n makes things blockwise more defensive with increasing n - as should be.
So everything is fine from this small test.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-19 17:46:01
I made a very small (and conservative) change to the revised spreading function (it increases the resultant FLAC bitrate by less than 1kbps) at the last minute.
Title: lossyWAV 1.2.0 Development Thread
Post by: monoton on 2009-03-19 18:08:37
Hey there... can you perhaps tell me the parameter setup to get the correction files with foobar2000? I'm quite new to this and I couldn't find it in the documentation or work it out by try and fail yet...

Thanks for your help and all your efforts on this!
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-03-19 18:36:45
Hey there... can you perhaps tell me the parameter setup to get the correction files with foobar2000? I'm quite new to this and I couldn't find it in the documentation or work it out by try and fail yet...

Thanks for your help and all your efforts on this!

You can create a batch file, say lossyFLAC.bat, in your FLAC folder with this contents:

c:\programme\flac\lossyWAV.exe %1 %3 %4 %5 %6 %7 %8 %9 --below --silent
ren "%~N1.lwcdf.wav" "%~N2.lwcdf.wav"
c:\programme\flac\flac.exe -b 512 -l 6 -e -m -r 2 -f --totally-silent "%~N1.lossy.wav" -o"%~N2.flac"
del "%~N1.lossy.wav"

Change the paths of lossyWAV.exe and flac.exe according to your system.

In foobar, Command Line Encoder Settings, you choose this lossyFLAC.bat as your encoder.
As parameters you use %s %d --portable -C (resp. your quality setting).
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-19 21:32:34
I would add compression of the .lwcdf.wav file if it exists:
Code: [Select]
c:\programme\flac\lossyWAV.exe %1 %3 %4 %5 %6 %7 %8 %9 --below --silent
if exist "%~N1.lwcdf.wav" c:\programme\flac\flac.exe -b 512 -l 6 -e -m -r 2 -f --totally-silent --keep-foreign-metadata "%~N1.lwcdf.wav" -o"%~N2.lwcdf.flac"&&del %~N1.lwcdf.wav
c:\programme\flac\flac.exe -b 512 -l 6 -e -m -r 2 -f --totally-silent  --keep-foreign-metadata "%~N1.lossy.wav" -o"%~N2.flac"
del "%~N1.lossy.wav"

Title: lossyWAV 1.2.0 Development Thread
Post by: monoton on 2009-03-20 01:48:03
Thank you both! That worked well! My knowledge of batch programming is pretty limited  But I could work out for most what was meant and added the lwcdf conversion already.
Title: lossyWAV 1.2.0 Development Thread
Post by: cBc on 2009-03-29 20:31:46
Hello there
I've been using lossyWAV exclusively to create music files for my DAP (rockboxed Sansa e280) and the reduction in filesize while retaining SQ is quite amazing.

I am using this batch file to drop my archived FLACs onto, would someone be so kind as to advise if I have it optimized for use with the latest build 1.1.3e :

@echo off
:repeat
if %1.==. goto end
if exist %1 g:\lossyWAV\flac -d %1 --stdout --silent|g:\lossyWAV\lossywav - --stdout -P --limit 17500 --shaping 0 --stdinname %1|g:\lossyWAV\flac - -b 512 -o "%~dpn1.lossy.flac" --silent && g:\lossyWAV\tag --fromfile %1 "%~dpn1.lossy.flac"
shift
goto repeat
:end


I am sadly not very knowledgeable with batchfiles  , and wish to make sure I have the setting correct. Thanks to Nick & all who are helping on this project.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-29 21:16:47
Glad that you are happy with lossyWAV.

With respect to your batch file, if you are trying to:
then, that's what your batch file will do.

Is there any particular reason why you are removing shaping?
Title: lossyWAV 1.2.0 Development Thread
Post by: cBc on 2009-03-30 02:57:39
Glad that you are happy with lossyWAV.

With respect to your batch file, if you are trying to:
  • process using the --portable preset;
  • remove shaping and
  • increase the calculation range upper limit to 17.5kHz
then, that's what your batch file will do.

Is there any particular reason why you are removing shaping?

hmm..honestly..no. I may have copied some of the parameters from one of the other posts in here. So what I have doesn't sound like what I would need
for best possible SQ using --portable . What would you recommend I change it to?

Thanks again!
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-03-30 05:55:54
I dunno if it's of any use but Abfahrt Hinwil (eig) is ABXable with lossyWAV beta 1.1.3e -q 1, the artefact is like a little synthetizer effect in the background. It doesn't really sound bad but it is obvious at -q 1.

Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/03/30 05:36:02

File A: D:\02- Artefact Only\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: D:\02- Artefact Only\01- Abfahrt Hinwil (Artefact Only).lossy.flac

05:36:02 : Test started.
05:36:30 : 01/01  50.0%
05:36:42 : 02/02  25.0%
05:36:53 : 03/03  12.5%
05:37:04 : 04/04  6.3%
05:37:20 : 05/05  3.1%
05:37:35 : 06/06  1.6%
05:37:57 : 07/07  0.8%
05:38:13 : 08/08  0.4%
05:38:15 : Test finished.

----------
Total: 8/8 (0.4%)


I tried at -q1.5 & -q2, I had 8/8 & 10/12 but the artefact was gone. I was guessing if it was here & I think I was lucky. I would say the artefact is medium at -q1, light at -q1.5, gone at -q2 & up. It seems to confirm my previous test (Old Test) (http://www.hydrogenaudio.org/forums/index.php?showtopic=63254&st=50&p=566484&#entry566484) for me the point of transparency of lossywav is near -q 2 based on this two samples. If I would spend hours on it maybe I could ABX -q 2 on this two samples, but I think it's a job for a madman as the result would be based on subliminal feelings more than on rationnal listening.

I report because it's the only killer sample that I tested in my DCT listening test that also affect lossywav. I tested them all at -q1.

Nothing to worry about -P is still safe for me ...
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-30 07:24:28
@sauvage78 - thanks for the identification of another problem sample and also your ABX'ing to determine where the transparency point is. If you could link to the sample that would be useful.

@cBc: for DAP use, I would suggest simply:

g:\lossyWAV\lossywav - --stdout -P --stdinname %1

[edit] as the lossyWAV related element of the relevant line in your batch file. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-03-30 07:34:55
The Abfahrt Hinwil sample is available here Sample (http://www.hydrogenaudio.org/forums/index.php?showtopic=70442). Grab the lossless version of the sample pack (I tested the short version).

The artefact is not the same as DCT codecs, DCT codecs have problem with the little explosions. Lossywav doesn't, it's just an added noise. Test at -q 1 or below cause at  -q1.5 the artefact is so low that you will miss it. I think I had some success with my guessing at -q1.5, only because I knew where to suspect a variation thks to -q 1. So far for me lossywav becomes transparent between -q1.5 & -q2. -q2 is a pain to ABX, it's like Nero AAC Q0.55, but even harder.

Edit:
Here is log for -q 1.5, I don't have the log for -q 2.0 as I didn't save it. It is missleading as I only have good results while I couldn't really spot an artefact. What is important to notice in it is that
a small quality increase lead me to double the ABX time for the same result as -q 1. So -q1.5 was already much harder to ABX than -q 1.

Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/03/30 05:57:37

File A: D:\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: D:\01- Abfahrt Hinwil (Artefact Only).lossy.flac

05:57:37 : Test started.
05:59:18 : 01/01  50.0%
05:59:47 : 02/02  25.0%
06:00:10 : 03/03  12.5%
06:00:27 : 04/04  6.3%
06:00:46 : 05/05  3.1%
06:01:10 : 06/06  1.6%
06:01:38 : 07/07  0.8%
06:02:01 : 08/08  0.4%
06:02:03 : Test finished.

----------
Total: 8/8 (0.4%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-30 07:46:27
Thanks, got them. One thing that you might try (if you have the time) is adding extra analyses at the same quality level (--analyses <n>; 2<=n<=5) to process using additional FFT lengths. My gut feeling is that this will find more of the dips in the noise floor that cause issues with problem samples.
Title: lossyWAV 1.2.0 Development Thread
Post by: cBc on 2009-03-31 02:47:53
@cBc: for DAP use, I would suggest simply:

g:\lossyWAV\lossywav - --stdout -P --stdinname %1

[edit] as the lossyWAV related element of the relevant line in your batch file. [/edit]


Perfect! Thank you for the help.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-03-31 07:27:07
Just tried on the Therion/Ginnungagap problem sample at -q 0, (I will call it Therion now as it comes from this band/album: Therion - 2001 - Secret of The Runes - 01- Ginnungagap (The Black Hole) (Prologue) & I have the full album). This -a 5 thing is a real improvement, I was instantly able to ABX a positive gain for only a 19ko difference !!! The artefact is still here & very ABXable compared to the original, but if you compare -q 0 with & without -a 5, the artefact is much softer because the added noise is much more stable. From what I heard, this -a 5 thing should be used as default.

Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/03/31 08:12:35

File A: D:\a00- Therion (Artefact+Context) Lossless -q 0 -a 5.lossy.flac
File B: D:\a00- Therion (Artefact+Context) Lossless -q 0.lossy.flac

08:12:35 : Test started.
08:13:06 : 01/01  50.0%
08:13:24 : 02/02  25.0%
08:13:47 : 03/03  12.5%
08:14:54 : 04/04  6.3%
08:15:14 : 05/05  3.1%
08:16:02 : 06/06  1.6%
08:16:20 : 07/07  0.8%
08:16:52 : 08/08  0.4%
08:16:53 : Test finished.

----------
Total: 8/8 (0.4%)


Same thing for Abfahrt Hinwil, with same settings. The artefact is not more stable here as it is not noisy like the Therion sample, but it's like as if the volume level of the artefact was reduced within the sample. In the end, it's the same: it's much better & easy to ABX for a 4ko difference !!! On this sample it is possible that this -a 5 thing alone lower the transparency point by 0.5 for me. I should test at -q 1 -a 5, to see if the artefact is still here.


Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/03/31 08:32:32

File A: D:\a01- Abfahrt Hinwil (Artefact Only) -q 0.lossy.flac
File B: D:\a01- Abfahrt Hinwil (Artefact Only) -q 0 -a 5.lossy.flac

08:32:32 : Test started.
08:32:48 : 01/01  50.0%
08:33:06 : 02/02  25.0%
08:33:27 : 03/03  12.5%
08:33:41 : 04/04  6.3%
08:33:56 : 05/05  3.1%
08:34:16 : 06/06  1.6%
08:34:28 : 07/07  0.8%
08:34:48 : 08/08  0.4%
08:34:50 : Test finished.

----------
Total: 8/8 (0.4%)


Edit:
Just tested at Abfahrt Hinwil -q 1 -a 5 vs. -q 1, unfortunatly even reduced the artefact was still here, so I compared -q 1 -a 5 vs. -q 1.5, -q 1.5 is better, so for me the improvement can be evaluated as a 0.25 direct gain within the quality scale. Personnaly I think it's a great gain.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-31 08:15:05
Many thanks for the additional ABX testing of the effect of --analyses 5 on these problem samples.

I have been thinking about a modification to the method which reduces the influence of the adjacent codec-blocks on the bits-to-remove for the codec-block being analysed. For 44.1kHz auido, at the moment two 1024 sample FFT analyses are carried out per 512 sample codec-block: -512..511 samples and 0..1023 samples (Samples -512..-1 are in the previous codec-block; 0..511 are the current codec-block and 512..1023 are the next codec-block). This basically takes all of the previous codec-block and the next codec-block into account when analysing the current codec-block. Changing this behaviour so that the overlap outwith the current codec-block is restricted to 50% of the codec-block length rather than 50% of the FFT sample length results in a single analysis: -256 to 768 samples which changes the overall bits-to-remove (some plusses, some minuses compared to the previous method).

This was previously evaluated and discarded but I think too many things were changing together at that time to fully appreciate the impact of the single central 1024 sample FFT analysis.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-03-31 08:17:19
... Personnaly I think it's a great gain.

Thank you very much for testing and having found -a 5 useful. Looks like I have to change my mind towards the additional analyses.

Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-31 08:28:56
It does make a difference.

However, in addition to the central FFT proposal above I have been thinking (again) about the underlap of the overlapping FFT analyses that are carried out. At present the FFT analyses overlap by 50%, i.e. underlap is also 50%. This behaviour can be modified to reduce to the underlap length to any proportion but sensibly I would limit this to 25%; 12.5%; 6.25%; etc. i.e. length/4; length/8; length/16; etc.

All this does is add extra analyses between those already calculated. This can be performed with or without the single central 1024 sample FFT analysis proposed above.

I will work up beta v1.1.3f with a --centre parameter to enable the single central 1024 sample (or 2 x codec-block length for higher than standard sample rates) FFT analysis and a --underlap <n> parameter where 1<=n<=(yet to be determined) and the underlap length will be FFT length / (2^n).

This should be posted later today.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-03-31 08:41:54
np, I will test lossywav even more in the future if needed, I was away from it as I was playing with blu-ray lately & lossywav is not very suited for AC3 transcoding  (& AC3 is crap anyway) ... I thougth for sometimes that I would use vorbis specially as I liked its recent 5.1 improvement, but my listening test disgusted me from vorbis for a long time I think ... It was not the expected affect ... but I want a codec that is free of artefact at high bitrate, open sourced & easyly splitable. Lossywav is the only codec which has all this features. So I am back using it, despite the big bitrate.

Nick:
you pm about testing a -v option (if I recall well), a month ago, I told you that I didn't had the time, if ever this thing still need testing, just explain what you want me to test. I deleted the pm & as I wasn't following the development, I don't even know what it was supposed to do.

Edit: I must say I don't understand a word of the technical discussion, just code it if you think it can help on paper, I'll listen if it works in reality
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-31 08:46:55
The -v (--varspread) parameter has been integrated with the --oldspread parameter as the revised (and slightly more conservative) spreading function.

Thanks for the offer - but it's already been finalised in v1.1.3e.

I will appreciate greatly any time that you can devote to testing of the --centre and --underlap parameters.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-31 09:46:31
lossyWAV beta v1.1.3f attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-03-31 10:45:45
I tested --centre, I immediatly suspected a regression, but I couldn't ABX it & I quickly feel tired. I add a first row of seven success but then it became very random. I tried to re-focus in order to fall below 0.4%, so I went up to 10 then 12 then 16 trials but I always failed to recover.

Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/03/31 11:00:00

File A: D:\a00- Therion (Artefact+Context) -q 0 --centre.lossy.flac
File B: D:\a00- Therion (Artefact+Context) -q 0.lossy.flac

11:00:00 : Test started.
11:00:39 : 01/01  50.0%
11:01:10 : 02/02  25.0%
11:01:56 : 03/03  12.5%
11:02:34 : 04/04  6.3%
11:02:57 : 05/05  3.1%
11:03:52 : 06/06  1.6%
11:04:17 : 07/07  0.8%
11:05:17 : 07/08  3.5%
11:06:29 : 08/09  2.0%
11:07:49 : 08/10  5.5%
11:09:49 : 08/11  11.3%
11:10:29 : 09/12  7.3%
11:11:20 : 10/13  4.6%
11:12:28 : 11/14  2.9%
11:13:52 : 11/15  5.9%
11:16:27 : 12/16  3.8%
11:16:42 : Test finished.

----------
Total: 12/16 (3.8%)


I tried Abfahrt Hinwil to see if it weren't easier, same thing, I instantly suspected a regression & add a 7/8 success in the beginning then it became random, so I gave up.

Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/03/31 11:21:13

File A: D:\a01- Abfahrt Hinwil (Artefact+Context) -q 0.lossy.flac
File B: D:\a01- Abfahrt Hinwil (Artefact+Context) -q 0 --centre.lossy.flac

11:21:13 : Test started.
11:21:48 : 01/01  50.0%
11:22:14 : 02/02  25.0%
11:22:49 : 03/03  12.5%
11:23:34 : 04/04  6.3%
11:24:56 : 05/05  3.1%
11:25:39 : 05/06  10.9%
11:27:13 : 06/07  6.3%
11:28:10 : 07/08  3.5%
11:28:57 : 08/09  2.0%
11:31:31 : 08/10  5.5%
11:31:43 : Test finished.

----------
Total: 8/10 (5.5%)


I suspect that there is a slight regression that I could catch better after a night of sleep. The only thing I can tell for sure is that in no way there is a progression with --centre.
Personnaly, I am convinced there is a regression but I cannot seriously backup my claim. Even if my logs aren't really good, I am self confident because I know which artefact I was trying to catch: a very light crackling that was making the sound unstable on both samples (even on Abfahrt Hinwil which was more stable than Therion so far)
I will try the other setting later as I am not in the best mood for it actually. I'm asleep at the wheel.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-03-31 13:04:44
Sure I'm very conservative, but after having reached such a good quality with lossyWAV, Nick, I'm always afraid that with a changing machinery quality can get worse.
There's a golden way however by doing things the way you did them with -V or -a, that is going more defensive in a strict sense with a new machinery. This allows even for automatic testing, cause for every block of every sample thrown to a new machinery bits to remove should be equal or less than was the case with the current version.

In case of your central 1024 bin FFT it would mean doing it additionally to the end point centered FFTs and choose the lowest number of bits to remove.

All these things especially when done in the secure way described above makes the quality settings more defensive. This means we have to consider bitrate go up a bit (and encoding time increase). IMO this is welcome for the low quality settings in case we get a quality increase like the one reported by sauvage78. It's more questionable for --portable and above as quality is excellent already as known so far.

In order to give the user a choice maybe you can give him a switch (named --superanalysis or similar) which incorporates all this extra stuff which makes the machinery extra-defensive. May be it should be defaulted up to -q 1.5 or --portable.
Or the other way around: use --superanalysis as default (maybe up to a certain quality level like for instance --standard) and give the user the opportunity to use --standardanalysis.
When defaulting up to a specific quality level no matter which way --standardanalysis or --superanalysis would both be welcome for the user in order to use whatever he likes for every quality level.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-03-31 13:24:49
@sauvage78: Thanks for the testing - I expect --centre to change things slightly - how it changes them in terms of better or worse has yet to be determined.

@halb27: I think that removing --centre will be the outcome of this latest phase - unless someone finds a sample where it improves the output compared to the existing method. Adding extra FFT analyses at existing FFT lengths by reducing the FFT underlap between successive analyses should always be a conservative approach.

In the same way that adding additional FFT lengths to the analysis using the --analyses parameter examines different sample windows, the reduction in underlap length increases the number of FFT analyses for a current FFT length over the same data-range.

Quick check on resultant bitrates at --portable for my 55 problem sample set:

--underlap 1: 432.4kbps; (default)
--underlap 2: 443.0kbps;
--underlap 3: 453.9kbps;
--underlap 4: 465.3kbps;
--underlap 5: 477.0kbps;
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-03-31 14:46:21
I think all these things work together.

The point of the spreading functions is partly to match how the human ear works, but also partly to help iron out some of the inevitable, but insignificant (IMO) very low results in various random FFT bins. I know you tuned these spreading functions extensively previously.

If you add more analyses, you're increasing the chance of getting an insignificant very low FFT bin. You're also increasing the chance of spotting a real and significant dip in the spectrum.

It seems to me that if you get more of these pesky meaningless isolated very low FFT bins, then you might need more spreading to hide them; also, if you are going to spot more genuine dips (or spot them more reliably than before) then the ideal spreading might be more than you had before (because, presumably, if things worked OK before, the spreading must have been slightly conservative to account for the fact that you might half-miss some genuine spectral dips).

At the very least, it seems to me that having more FFT data available gives the chance (and it makes sense) to use some subtly different spreading.


So the "right" thing to do is figure out what magnitude of frequency dip you want to catch, and design the FFTs and spreading functions to ensure that you catch it.

Or, to put it another way, if it was working properly previously, then the "right" thing to do can't be to change one without the other. You need to experiment with both FFT properties and spreading functions.


Sounds like too much hard work to me...

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-03-31 23:51:57
I just tested --underlap 5, I first tried --underlap 8 as it is written up to 8 in the --longhelp but it crashed. I used -q 0 instead of -q 1 just to make the ABXing easier, the result is that --underlap 5 give result very similar to -a 5, there is a real gain. The result is almost the same as with -a 5, Therion became more stable (less random noise) & the volume of the artefact within Abfahrt Hinwil is reduced. It was much much easier to ABX than --centre. For testing purpose I begin to think that Abfahrt Hinwil is a better sample than Therion because it is less noisy & less tiring (Edit: not true for --centre). Is --underlap 5 supposed to be compatible with -a 5 ? Is it supposed to stack ?

Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/04/01 00:36:35

File A: D:\a00- Therion (Artefact+Context) -q 0 --underlap 5.lossy.flac
File B: D:\a00- Therion (Artefact+Context) -q 0.lossy.flac

00:36:35 : Test started.
00:37:00 : 01/01  50.0%
00:37:20 : 02/02  25.0%
00:37:41 : 03/03  12.5%
00:38:10 : 04/04  6.3%
00:38:31 : 05/05  3.1%
00:38:52 : 06/06  1.6%
00:39:10 : 07/07  0.8%
00:39:52 : 08/08  0.4%
00:39:54 : Test finished.

----------
Total: 8/8 (0.4%)


Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/04/01 00:43:44

File A: D:\01- Abfahrt Hinwil (Artefact Only) -q 0 --underlap 5.lossy.flac
File B: D:\01- Abfahrt Hinwil (Artefact Only) -q 0.lossy.flac

00:43:44 : Test started.
00:44:01 : 01/01  50.0%
00:44:12 : 02/02  25.0%
00:44:22 : 03/03  12.5%
00:44:29 : 04/04  6.3%
00:44:37 : 05/05  3.1%
00:44:49 : 06/06  1.6%
00:45:06 : 07/07  0.8%
00:45:20 : 08/08  0.4%
00:45:22 : Test finished.

----------
Total: 8/8 (0.4%)


Then I retried --centre as I was in better health than yesterday:

Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/04/01 01:03:05

File A: D:\00- Therion (Artefact+Context) -q 0.lossy.flac
File B: D:\00- Therion (Artefact+Context) -q 0 --centre.lossy.flac

01:03:05 : Test started.
01:03:38 : 01/01  50.0%
01:04:01 : 02/02  25.0%
01:04:24 : 03/03  12.5%
01:04:42 : 03/04  31.3%
01:05:03 : 04/05  18.8%
01:05:25 : 05/06  10.9%
01:05:57 : 06/07  6.3%
01:06:42 : 07/08  3.5%
01:07:24 : 08/09  2.0%
01:08:03 : 09/10  1.1%
01:08:15 : Test finished.

----------
Total: 9/10 (1.1%)


Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4 beta 2
2009/04/01 01:09:06

File A: D:\01- Abfahrt Hinwil (Artefact Only) -q 0 --centre.lossy.flac
File B: D:\01- Abfahrt Hinwil (Artefact Only) -q 0.lossy.flac

01:09:06 : Test started.
01:09:28 : 01/01  50.0%
01:09:53 : 01/02  75.0%
01:10:57 : 02/03  50.0%
01:11:50 : 03/04  31.3%
01:12:06 : 03/05  50.0%
01:12:33 : 04/06  34.4%
01:13:15 : 04/07  50.0%
01:14:16 : 05/08  36.3%
01:15:12 : 06/09  25.4%
01:16:06 : 07/10  17.2%
01:16:14 : Test finished.

----------
Total: 7/10 (17.2%)


My personnal feeling is that there is a regression on the Therion sample, but that this regression affects Abfahrt Hinwil much less than the Therion sample. I tried twice & twice it happened that I highly suspected that Therion add some added cracklings. I add a raw of success yesterday on Abfahrt Hinwil but I couldn't reproduce this result today while my health was better. So personnaly I consider the first raw of success that I had yesterday on Abfahrt Hinwil as a possible raw of lucky guessing. It doesn't remove anything to the fact that there is no progression at all with --centre on any of the two killer samples & worst a possible regression (with high probability) on the Therion sample.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-01 08:23:09
@David: I think that I see what you mean with respect to significant vs insignificant dips. More FFT analyses of the same length within the same overall data range will presumably only spot more dips (if all of the previous analyses are still carried out). I also see what you mean about changes to the spreading function being "required" when more analyses are carried out.

As a clarification, more analyses in this sense is more overlapped FFT analyses within the same data range.

Adding extra FFT lengths to the overall analysis has a much more benign effect (see table below) so I think that the spreading function would require to be changed only if the underlap length is shortened.

For my 55 problem sample set (processed at --portable):
Code: [Select]
+--------------+--------------+--------------+--------------+--------------+--------------+
| beta v1.1.3f | --underlap 1 | --underlap 2 | --underlap 3 | --underlap 4 | --underlap 5 |
+--------------+--------------+--------------+--------------+--------------+--------------+
| --analyses 2 |   432.4kbps  |   443.0kbps  |   453.9kbps  |   465.3kbps  |   477.0kbps  |
| --analyses 3 |   438.2kbps  |   449.1kbps  |   460.7kbps  |   472.7kbps  |   485.0kbps  |
| --analyses 4 |   441.1kbps  |   452.2kbps  |   464.2kbps  |   476.5kbps  |   489.0kbps  |
| --analyses 5 |   443.6kbps  |   454.9kbps  |   467.1kbps  |   479.8kbps  |   492.6kbps  |
+--------------+--------------+--------------+--------------+--------------+--------------+


@sauvage78: Many thanks for the extensive testing. Taking into account David's suggestion regarding number of FFT analyses carried out on a data range requiring modifications to the spreading function, maybe the --centre parameter requires a change to be made to the spreading function for the 1024 sample FFT analysis.

The --underlap parameter description in the --longhelp is wrong, as you have already determined, the upper limit for --underlap is currently coded as 5.
I am still iterating with some ideas - will have another think and post revised code later (if I get anywhere).
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-01 09:22:57
Just a suggestion for quick checking whether new details in the machinery are worth testing:
Bitrate for a problem sample set should go up (in a significant way in an ideal world), while this shouldn't be the case for regular music.
In case bitrate increase is more or less the same for the two cases there is no reason to prefer a new mechanism over just increasing the quality setting.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-01 13:41:22
Absolutely.... .... but some "regular" music could suffer from the same artifact type as is being dealt with by the modified method.

I looked at reducing the underlap to 1 sample and all that really happened was that the resultant bitrate went to 399kbps for --zero --underlap 13 processing at  the standard two FFT lengths from 327kbps for the original method.

As an aside, and prompted by something David said, I am looking at (yet) another approach where the FFT results for the frequency bands of interest are sorted after taking their magnitude and multiplying by the skewing factor. After sorting (at the moment) the second lowest bin value is taken as the result and the minimum of that and the SNR adjusted average is taken as the overall result for that analysis. For my 10 Album Test Set, this results in:
Code: [Select]
+------------------------------------------------------------------------------------------------------+
¦   Version    ¦   FLAC   ¦ --insane ¦--extreme ¦--standard¦--portable¦  --zero  ¦ --nasty  | --awful  |
+--------------+----------+----------+----------+----------+----------+----------+----------+----------¦
¦Version 1.1.0b¦ 854kbit/s¦ 632kbit/s¦ 548kbit/s¦ 463kbit/s¦ 376kbit/s¦ 285kbit/s¦ -------- | -------- |
+--------------+----------+----------+----------+----------+----------+----------+----------+----------¦
¦ beta v1.1.3e ¦   "  "   ¦ 641kbit/s¦ 556kbit/s¦ 472kbit/s¦ 386kbit/s¦ 292kbit/s¦ 231kbit/s| 200kbit/s|
+--------------+----------+----------+----------+----------+----------+----------+----------+----------¦
| beta v1.1.3g |   "  "   ¦ 673kbit/s| 598kbit/s| 519kbit/s| 424kbit/s| 306kbit/s| 244kbit/s| 207kbit/s|
+------------------------------------------------------------------------------------------------------+
These results use the same corresponding noise-threshold-shift values used for the existing spreading function and do not necessarily reflect bitrates for a final version - I just thought that I would share these interim findings.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-04-01 14:44:04
Just a suggestion for quick checking whether new details in the machinery are worth testing:
Bitrate for a problem sample set should go up (in a significant way in an ideal world), while this shouldn't be the case for regular music.
In case bitrate increase is more or less the same for the two cases there is no reason to prefer a new mechanism over just increasing the quality setting.
Exactly - there are a million different ways of pushing the bitrate up - but the ones that appear do this fairly uniformly across the board are just complicated ways of increasing the Q value.

However, I'm inclined to like mechanisms which make all moments sound equally bad, rather than those which leave audible differences in the "badness" experienced at different moments. So from this perspective, sauvage78's testing is encouraging.

Cheers,
David.

P.S. I'm still hoping that Nick will provide us with an explanation of the algorithm one day!
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-02 08:31:46
... For my 10 Album Test Set, this results in: ...

IMO this way of going more defensive makes the low bitrate settings more useful in case a real quality improvement is experienced.
For --portable and above bitrate is too high for my taste especially as quality is excellent with what we have had so far.
Maybe it's a good thing to use these defensive variants up to -q 1.5 or similar in case of positive experience.

In any case I think it would be helpful to see bitrate increase percentage for a regular music and a problem sample set.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-02 08:44:10
halb27:
I don't understand your point in wishing to apply usefull parameters only to low/not transparent settings, -a 5 is usefull no matter the setting as it brings a quality improvement without a bitrate increase. As such, it should be used as default.

--underlap 5 is nice but it increases the bitrate, so the only thing that is needed to know is if --underlap 5 is better or not than just simply increasing the bitrate. Only a test with & without --underlap 5 at the same target bitrate can tell it. If --underlap 5 wins it should be defaulted, if it loses it should be improved or forgotten. This is not a question of good or bad setting, above or below --portable.

The only goals of lossywav should be to lower its transparency point in order to have even smaller files for DAP. It's a no-sense to artificially improve non-transparent setting by 0.25 by using special parameters on it ... it will not make them more tansparent, -q0 --underlap 5 is not better than -q0, both are rotten. If someone wants non-transparent lossy, he can use any other lossy codec at ~128Kbps, he will save much place. For me lossywav will swallow the niche that mpc had several years ago: audiophiles who don't care for bitrates, but only care for transparency on all kind of music (specially electronic music). The advantage of lossywav is not to be better or worse qualitywise than other lossy codecs, to me when transparency is reached they all sound the same. The advantage of lossywav is that it is affected by much less killer samples. Lossywav is a lossy codec with a bulletproof jacket on it.

-q 1.0 is usefull to find killer samples for lossywav, but I consider anything below -q 1.0 as not suited for any use I can think about.
Depending on your hearing skills -q 1.5, -q 2 & q 2.5 can be used for DAP,

-q 1.5 is non-transparent with light artefacts.
-q 2.0 is already really good but is likely to not be transparent on all future problem samples as the margin is low.
-q 2.5 is supposed transparent with a decent margin.

-q 5.0 is transparent with a huge margin for paranoid people. It remains to be proved that -q 5 is better for transcoding than -q 2.5.
Personnaly, I am convinced that nobody will ever be able to ABX this claim.

For me anything below -q 1.5 is non-sense as it is crap, anything above -q 2.5 is non-sense as it is huge.

I would rather use any pure lossy codec at ~128Kbps  than use lossywav at -q 0.
I would rather use any pure lossless codes at ~900Kbps than use lossywav at -q 5.

So lossywav should focus on what it does best, transparency for audiophile with DAP.
For me lossywav is not a flexible codec & it will never be.
There is no point in wishing that -q 0 would sound better if it is not transparent anyway: -q 0 is not like vorbis 64Kbps, nobody will ever stream it.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-04-02 10:25:33
I have a slightly different take on it.

The lowest setting needs to have audible problems (as I've said many times!). Those audible problems need to be related to the (hopefully inaudible) "problems" at higher settings. So you can't switch in a fundamentally different method for lower settings.

If you can change the algorithm and make a setting sound better at the same bitrate, then yes - change it - across the board though. Ditto if the change simply makes a given setting sound more consistent (at a similar bitrate to before) - because again that should be useful across the board.


I think lossyWAV with dynamic noise shaping (and maybe dynamic block size switching) could produce transparency at surprisingly low bitrates. Not mp3 bitrates, but not far off.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-02 10:28:46
halb27:
I don't understand your point in wishing to apply usefull parameters only to low/not transparent settings, -a 5 is usefull no matter the setting as it brings a quality improvement without a bitrate increase. ...
Actually I consider anything below -q1 as not suited for any use I can think about. Depending on your hearing skills -q 1.5, -q 2 & q 2.5 can be used for DAP,

-q 1.5 is non-transparent with light artefacts.
-q 2.0 is likely to not be transparent as the margin is low.
-q 2.5 is supposed transparent with a decent margin.

-q 5.0 is transparent with a huge margin for paranoid people....

I do use -a 5 with --portable thanks to your listening test.
Bitrate goes up with -a 5 (from 387 kbps to 400 kbps for my regular music test set).  I personally can live with this, but users should have some confidence that --portable bitrate remains roughly the same with new variants and there shouldn't be a steady upwards trend. Especially as -portable quality has been excellent.

My point is that I see it exactly like you: --portable so far can be expected to be transparent (without any further potential improverments of the machinery which always are also potentially risky when not done in a strictly defensive way), and the very edge of transparency is expected to be a little bit lower.
So any improvents in quality are most welcome for the lower quality settings, and it's matter of taste up to what quality level. If for instance -q 1.0 or -q 1.5 gets a more homogenous good quality (2Bdecided's idea) by a change in the machinery, and bitrate doesn't go up significantly to me this is the way to go. This is done by the -a 5 improvement you found.
But to me these things have their limit at least with a quality level of --portable. With the last thoughts about the machinery we get a bitrate of --portable which was that of --standard not a long time ago, and in this case I prefer the technical details of --standard, especially as this is the --standard machinery we do have quite a lot of experience meanwhile.
Different from you I see some usefulness also for the higher quality levels for which any bitrate increase is not so welcome because of great quality already achieved. The higher quality levels are useful for all those who don't need real lossless encodings but are content with lossy encodings of a quality that can be considered transparent with an extreme degree of confidence. Allows for a single collection.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-02 11:10:20
I know that --standard is interesting due to the history of lossywav & I think --standard is usefull as a high archor reference, but in all honesty --insane & --extreme are useless because the margin is much too wide. Using these two settings when -q 2.5 is already almost unABXable & -q 5 is unABXable at all, you will not increase the quality of your files, you will only waste space. I consider myself as a conservative user & even for me, this is obvious. The old 3 settings has always been the way to go for me. I still use lossywav this way -q 0/-q 2.5/-q 5. -q 0 is already bad enough to find killer samples, & -q5 is already good enough for paranoid audiophiles. I consider the other settings as missleading for new lossywav users, if you are an advanced user & want to test lossywav you can use the good'old quality scale. For serious testing purpose the actual presets are useless because you need much more accurate variation to tune lossywav. The -a 5 parameter is only a 0.25 gain on the quality scale. You cannot be so accurate with the presets, so telling the preset are made for tuning is a non-sense. --nasty & --awful are for deaf people. I agree that a very bad setting is needed, so I think -q -2.5 is more usefull than -q 7.5. Also having 4 setting is nice to create bitrates tables like Nick is doing regulary. 7 presets is too much IMHO. The good'old 3 presets + a very bad one are enough IMHO.

Because my test files are so short I didn't realize there could be a variation of ~15Kbps on real music that's why I didn't understood why you were so worried about non-transparent settings.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-02 11:21:53
All negative presets will be removed for v1.2.0.

Any bitrate less than the lossless FLAC is a saving - users should be permitted to select their own personal preference from FLAC - circa 20% to circa 300kbps.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-02 11:22:01
IMO too --portable and --standard are the most useful quality levels though I think --extreme is still useful for people using extreme quality lossyWAV as a substitute for lossless.
Anything else can go into the -q settings.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-02 11:24:13
IMO too --portable and --standard are the most useful quality levels though I think --extreme is still useful for people using extreme quality lossyWAV as a substitute for lossless.
Anything else can go into the -q settings.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-02 11:35:53
<feck> Mods, please delete this duplicate post.... Thankyou!
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-04-02 12:06:07
The extreme settings are (potentially) useful for transcoding and (almost certainly) useful for multi-generation use.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-02 15:06:33
The results of a bit of comparative analysis:
Code: [Select]
10 Album Test Set
+------------------------------------------------------------------------------------------------------+
| beta v1.1.3g |   FLAC   | --insane |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
|   default    | 854kbit/s| 641kbit/s| 558kbit/s| 472kbit/s| 386kbit/s| 292kbit/s| 231kbit/s| 200kbit/s|
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
| --sortspread |   "  "   | 646kbit/s| 569kbit/s| 488kbit/s| 399kbit/s| 294kbit/s| 236kbit/s| 204kbit/s|
+------------------------------------------------------------------------------------------------------+

55 Problem Sample Set
+------------------------------------------------------------------------------------------------------+
| beta v1.1.3g |   FLAC   | --insane |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
|   default    | 782kbit/s| 660kbit/s| 591kbit/s| 515kbit/s| 432kbit/s| 327kbit/s| 260kbit/s| 218kbit/s|
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
| --sortspread |   "  "   | 655kbit/s| 590kbit/s| 518kbit/s| 437kbit/s| 326kbit/s| 262kbit/s| 220kbit/s|
+------------------------------------------------------------------------------------------------------+

55 Problem Sample Set (--analyses 5)
+------------------------------------------------------------------------------------------------------+
| beta v1.1.3g |   FLAC   | --insane |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
|   default    | 782kbit/s| 667kbit/s| 601kbit/s| 527kbit/s| 444kbit/s| 334kbit/s| 265kbit/s| 221kbit/s|
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
| --sortspread |   "  "   | 663kbit/s| 599kbit/s| 529kbit/s| 447kbit/s| 332kbit/s| 266kbit/s| 222kbit/s|
+------------------------------------------------------------------------------------------------------+


I'll try to post beta v1.1.3g this evening.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-02 15:46:36
Looking at this --sortspread is not very promising. Bitrate increase is higher for regular music than for problem samples.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-02 15:49:06
I'm still tuning the changes to the noise-threshold-shift values. As there is *no* averaging involved, I feel that --sortspread will find true holes more effectively (and ignore the first one).

I think that what it is finding in the 10 Album Test Set are potential problems which have not been recognised. The increase in 10 set bitrate when compared to the increase in 55 set bitrate is counterintuitive - so maybe it is finding something - or maybe it is padding the sausage.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-02 21:14:08
lossyWAV beta v1.1.3g attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: carpman on 2009-04-02 23:51:30
For me, settings > --standard are useful, and one cannot look at lossywav in isolation in regards to --extreme etc .. Until there's a transcoding test from the various transparent lossywav settings to LAME at -v 5 and above, I'll remain a crazy in Sauvage's eyes. I've noticed for some samples there's a difference in bitrate from transcoding from TAK > MP3 as against Lossy.TAK > MP3. I'm guessing LAME is encoding some of that added noise.

And as Nick, 2Bdecided and Halb27 have pointed out, some people (me included) use LossyWAV as a substitute for lossless, so LossyWAV files become the source files for any transcodes.

C.

p.s. I liked this from Sauvage:

Quote
Lossywav is a lossy codec with a bulletproof jacket on it.
Title: lossyWAV 1.2.0 Development Thread
Post by: jido on 2009-04-03 11:42:25
I think lossyWAV with dynamic noise shaping (and maybe dynamic block size switching) could produce transparency at surprisingly low bitrates. Not mp3 bitrates, but not far off.

Cheers,
David.

Wouldn't dynamic block size switching require the lossless encoder to use the same or fraction of the same block size dynamically? Would you use metadata to pass on the information?
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-04-03 11:58:01
I've got no idea how to implement it. Metadata is a good idea - I hadn't thought of that!

Otherwise you have to integrate lossyWAV and the specific lossless encoder together.

I really like the idea of having a "blocksize" metadata file - though of course the lossless encoders would have to be modified to read it - but that would be possible if this was ever developed. Thanks for the suggestion jido!

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-04-03 16:04:14
Looking at this --sortspread .... Bitrate increase is higher for regular music than for problem samples.

I'm still tuning the changes to the noise-threshold-shift values.

By the time you have the mechanisms in place and tested, and I mean in the last phase before a new non-beta release, I think the scale should be adjusted.
Originally -q 5 should have been transparent (for known samples),  -q 0-4.9 lower bit rates than that, -q 5.1-9 gradually nearing lossless bit rates. At the moment -q 5 is almost considered overkill (yes, some exaggeration) and - 2.5 has to be transparent. I hope with all the more defensive approaches it would make sense to lower the bit rates over the whole -q scale and re-determine what bit rate is really necessary for the goal (transparency for known samples).

After that the positions for the settings like --portable have to be chosen again, this one could match for example -q 3.2, 2.7, 2.4 or whatever.

I know it is easier said than done, but how do you guys feel about the idea?

(BTW for me the biggest interest in lossyFlac are bit rates around or below 400)
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-03 19:31:18
Just tested --sortspread, don't know if it was supposed to be tested alone or in conjunction with other parameter but alone it is not good, it adds new artefacts that I never heard before to both samples.
The result is strange because it soften old artefacts & it add new artefacts in the same time. For Abfahrt Hinwil the little "synthetizer effect" (I don't know how to call that) sound is soften but it adds a clic, which is instantly ABXable (1min15 for 8 success). For Therion the "tearing in the noise" is reduced but it adds cracklings. For Abfahrt Hinwil, there is a regression for sure. For Therion the cracklings might be masked by the tearing in the -q 0 encoding, so maybe --sortspread just reveals them, it is possible but I doubt it as I think I heard cracklings even after the tearing was supposed to end & anyway I prefer the tearing compared to cracklings.

Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/03 20:12:36

File A: D:\01- Abfahrt Hinwil (Artefact Only) -q 0 --sortspread.lossy.flac
File B: D:\01- Abfahrt Hinwil (Artefact Only) -q 0.lossy.flac

20:12:36 : Test started.
20:12:53 : 01/01  50.0%
20:13:02 : 02/02  25.0%
20:13:10 : 03/03  12.5%
20:13:17 : 04/04  6.3%
20:13:24 : 05/05  3.1%
20:13:31 : 06/06  1.6%
20:13:40 : 07/07  0.8%
20:13:48 : 08/08  0.4%
20:13:49 : Test finished.

----------
Total: 8/8 (0.4%)


Quote
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/03 20:15:14

File A: D:\00- Therion (Artefact+Context) -q 0 --sortspread.lossy.flac
File B: D:\00- Therion (Artefact+Context) -q 0.lossy.flac

20:15:14 : Test started.
20:15:45 : 01/01  50.0%
20:16:47 : 02/02  25.0%
20:17:47 : 03/03  12.5%
20:18:30 : 04/04  6.3%
20:19:11 : 05/05  3.1%
20:20:07 : 06/06  1.6%
20:20:50 : 07/07  0.8%
20:21:25 : 08/08  0.4%
20:21:27 : Test finished.

----------
Total: 8/8 (0.4%)
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-03 19:45:03
@GeSomeone:

I feel your concern about bitrate going up too high for --portable with new details of the machinery under investigation.
I feel like you.

For the easier part: for --standard I'd still like it to be exactly where it is now with respect to 2Bdecided's original idea. Moreover IMO there's absolutely no need for changes in the machinery that brings --standard bitrate up because quality is excellent (I'd rather see it the other way around but IMO there's no real need for changes in this respect.)
Same goes for --extreme and --insane.

For the -q levels below --portable especially the cutting edge transparency levels around -q 1.5 I welcome any experienced quality improvement like that of -a 5 even if bitrate goes up a bit. It simply makes these levels more usable.

Hardest decision is about --portable. I personally like it to be at the noise threshold of the original idea where it is now (so I like it to be -q 2.5). But I dislike variants of details which brings bitrate up more and more. Like you I'd like to see --portable's bitrate at 400 kbps or below. --portable as is even without -a 5 shows up a quality which now can be expected great for portable purposes with even a small security amrgin (which I also like - that's why I also use -a 5).

Things may change when essentially new aspects come up in practice like with dynamic noise shaping.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-03 20:00:17
@sauvage78 - hmm..... it appears that something unexpected is happening. I will have a look at the --sortspread code and revert. It certainly isn't meant to introduce clicks. What you describe seems like a bug.

@halb27 - I do not want bitrate creep (upwards) either. I do want as good a quality as can be achieved though. It may be that further exploration of --sortspread needs to be deferred until after the release of v1.2.0.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-05 22:04:38
lossyWAV beta v1.1.3h attached to post #1 in this thread.

I "tuned" the --sortspread noise-threshold-shift table so that the resultant bitrate for my 55 problem sample set was the same for both default spreading and --sortspread.

Resultant FLAC bitrates:
Code: [Select]
10 Album Test Set
+------------------------------------------------------------------------------------------------------+
| beta v1.1.3h |   FLAC   | --insane |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
|   default    | 854kbit/s| 641kbit/s| 558kbit/s| 472kbit/s| 386kbit/s| 292kbit/s| 231kbit/s| 200kbit/s|
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
| --sortspread |   "  "   | 651kbit/s| 571kbit/s| 488kbit/s| 398kbit/s| 297kbit/s| 236kbit/s| 197kbit/s|
+------------------------------------------------------------------------------------------------------+

55 Problem Sample Set
+------------------------------------------------------------------------------------------------------+
| beta v1.1.3h |   FLAC   | --insane |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
|   default    | 782kbit/s| 660kbit/s| 591kbit/s| 515kbit/s| 432kbit/s| 327kbit/s| 260kbit/s| 218kbit/s|
|--------------+----------+----------+----------+----------+----------+----------+----------+----------|
| --sortspread |   "  "   | 660kbit/s| 591kbit/s| 515kbit/s| 432kbit/s| 327kbit/s| 260kbit/s| 218kbit/s|
+------------------------------------------------------------------------------------------------------+
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-07 13:18:49
lossyWAV beta 1.1.3i attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-07 18:31:18
Here is the ABX log for 1.1.3i (I didn't even tried 1.1.3h). --sortspread now work as it should no more clics on Abfahrt Hinwil. The improvement was easy Abfahrt Hinwil & Therion (more stable), & nice to listen to on both samples. I begin to think that Abfahrt Hinwil is a perfect sample for tuning lossywav easyly: it is always faster to ABX, gives clics when you break things & is nicer to listen to. (Therion is tiring, I am bored of it). At -Q 0 the improvement is instantly ABXable, very comparable to -a 5. Maybe better maybe worst, I dunno yet. But this time I was instantly seduced by both the same way. I should do a direct comparison to know. But again if it stacks then --sortspread -a 5 is the way to go. If it stacks then --portable can most likely be remap to -q 2. Some test is needed for this. But I won't test if in the end -portable stay at -q 2.5. I am pretty convinced that if it stacks -q 2 --sortspread -a 5 is the same or better than -portable now, specially as -q 2 was already very hard to ABX even without this parameters.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/07 18:54:04

File A: C:\01- Abfahrt Hinwil (Artefact Only) -Q 0 --sortspread.lossy.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) -Q 0.lossy.flac

18:54:04 : Test started.
18:54:17 : 01/01  50.0%
18:54:27 : 02/02  25.0%
18:54:42 : 03/03  12.5%
18:54:54 : 04/04  6.3%
18:55:04 : 05/05  3.1%
18:55:14 : 06/06  1.6%
18:55:24 : 07/07  0.8%
18:55:35 : 08/08  0.4%
18:55:37 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/07 19:04:21

File A: C:\00- Therion (Artefact+Context) -Q 0 .lossy.flac
File B: C:\00- Therion (Artefact+Context) -Q 0 --sortspread.lossy.flac

19:04:21 : Test started.
19:04:52 : 01/01  50.0%
19:05:37 : 02/02  25.0%
19:06:15 : 03/03  12.5%
19:06:44 : 04/04  6.3%
19:07:12 : 05/05  3.1%
19:07:35 : 06/06  1.6%
19:08:10 : 07/07  0.8%
19:08:41 : 08/08  0.4%
19:08:42 : Test finished.

----------
Total: 8/8 (0.4%)



If someone can upload 1.1.3c or 1.1.3d, no matter if it is usefull or not, just to see if Nick didn't break anything with --varspread while I was away  the old --sortspread shows that Nick can break things without instantly noticing it so I'd prefer to have a look by myself even if he tells me it's useless.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-07 18:49:59
--sortspread is strictly superior to -a 5 (which is a great news as -a 5 was already a nice improvement), but the different is slight, I ABXed it on Abfahrt Hinwil, I may fell on Therion, as Therion is always harder & the difference is tiny. I had one failure in the beginning (the time to get used to the reduced artefact) but instead of cancelling & restarting I decided to go up to 12 trials as it was a tiny difference but not hard to ABX. For me I ereased my first failure, it's a completely successfull ABX.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/07 19:40:37

File A: C:\01- Abfahrt Hinwil (Artefact Only) -Q 0 -a 5.lossy.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) -Q 0 --sortspread.lossy.flac

19:40:37 : Test started.
19:41:29 : 00/01  100.0%
19:42:39 : 01/02  75.0%
19:43:06 : 02/03  50.0%
19:43:43 : 03/04  31.3%
19:43:58 : 04/05  18.8%
19:44:54 : 05/06  10.9%
19:45:16 : 06/07  6.3%
19:45:32 : 07/08  3.5%
19:45:48 : 08/09  2.0%
19:46:04 : 09/10  1.1%
19:46:29 : 10/11  0.6%
19:46:56 : 11/12  0.3%
19:46:59 : Test finished.

----------
Total: 11/12 (0.3%)


So to sum up, at this point, --centre & the bugged old --sortspread are both good for the delete bin & then we have the new --sortspread, -a 5 & --underlap 5, that all three brings some improvements.
-a 5 is strictly inferior to a 0.5 increase within the scale (so it's something like a ~0.25 increase) & --sortspread is strictly superior to -a 5 (so it's superior to a ~0.25 increase).

What would be needed now is to compare -Q 0 --sortspread Vs. -Q 0.5,  & also -Q 0 --underlap 5 Vs. -Q 0.5 to evaluate the gain more precisely & then see if the gain stacks in some way.
If at last two of those stacks then -Q 2 can be considered as transparent as the old -Q 2.5.
If any of -Q 0 --sortspread or -Q 0 --underlap 5 directly beats -Q 0.5 then it's even clearer. I suspect --sortspread alone to be able to achieve this, or to be very near to achieve this.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-07 21:20:34
I just tested  -Q 0 --sortspread Vs.  -Q 0.5, things didn't ended as I expected. It was too good to be true. First -Q 0 --sortspread beats -Q 0.5 on Abfahrt Hinwil, that was a very good news & I was quite happy. Then I tested the same thing with Therion & I ABXed a click on my right hear (headphones) at 1 second just after the big tearing & just before the guitar begins. I wasn't able to ABX the tearing so if you consider only the tearing artefact there is a real improvement of ~0.5 in the quality scale like with the Abfahrt Hinwil sample, the problem is that there is a click that make me lose focus.
I think that this is a new click that wasn't there when there was a click with Abfahrt Hinwil in the previous version of --sortspread. So the result is mixed for --sortspread it's both a big improvement (same or superior to a 0.5 jump in the quality scale) but it adds clicks, first it was with Abfahrt Hinwil, now it's fixed, but the fix introduced another click with Therion. Remove all the clicks bug & you have only the big improvement of ~-9Kbps on average (~-2 mo saved by album).

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/07 21:47:16

File A: C:\01- Abfahrt Hinwil (Artefact Only)  -Q 0.5.lossy.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) -Q 0 --sortspread.lossy.flac

21:47:16 : Test started.
21:47:37 : 01/01  50.0%
21:47:48 : 02/02  25.0%
21:48:03 : 03/03  12.5%
21:48:18 : 04/04  6.3%
21:48:31 : 05/05  3.1%
21:48:55 : 06/06  1.6%
21:49:10 : 07/07  0.8%
21:49:21 : 08/08  0.4%
21:49:23 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/07 22:04:49

File A: C:\00- Therion (Artefact+Context)  -Q 0.5.lossy.flac
File B: C:\00- Therion (Artefact+Context) -Q 0 --sortspread.lossy.flac

22:04:49 : Test started.
22:05:23 : 01/01  50.0%
22:05:41 : 02/02  25.0%
22:05:58 : 03/03  12.5%
22:06:19 : 04/04  6.3%
22:06:47 : 05/05  3.1%
22:07:17 : 06/06  1.6%
22:07:33 : 07/07  0.8%
22:07:54 : 08/08  0.4%
22:07:58 : Test finished.

----------
Total: 8/8 (0.4%)
Title: lossyWAV 1.2.0 Development Thread
Post by: monoton on 2009-04-08 15:28:11
In all this discussion about bitrates and stuff, I find it a bit funny from a more or less outside point of view: What's the matter of having a text variant like "portable" as an option for new users as long as you can set everything via double vale parameters as well? As long as you can expect these double values to give you about the same bitrates in each development status I don't think that there should be a problem other than the scale being not fixed or opened ended eventually... but if the users may want to encode with -q 15.5 for whatever reason, I don't see why it shouldn't be allowed if the algorithms can take it. Even if the resulting file is perhaps bigger than the original one (how could that be though).
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-04-08 20:00:42
What's the matter of having a text variant like "portable" ...
Quote
may want to encode with -q 15.5 for whatever reason, I don't see why it shouldn't be allowed if the algorithms can take it.

The presets like --standard and --portable were requested by users/testers. Can help to get a quick starting point.
It is very sensible to have some "sanity checks" on the parameters, to be sure the algorithm can take it. 
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-08 20:26:31
@sauvage78 - beta v1.1.3c appended to the first post as requested. Thanks again for the ABX testing - I'm going to have to delve deeper into this click that you are experiencing. Could you try ABXing --sortspread --maxclips 0 and see if that makes a difference?

We may well end up remapping the presets - that will require some ABX effort to determine the transparency point and then some work on my part to come up with proposed presets.

I still think that --insane will stay somewhere near where it is - offering nearer to lossless but still about a 20% resultant bitrate reduction.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-08 21:38:01
just tried -q 0 --sortspread --maxclips 0 Vs. -q 0 --sortspread, it doesn't help, it sound exactly the same, there is a click that is not in the lossless file but might be masked in encodes where the tearing is bigger (like without --sortspread). I didn't bother saving an ABX log. Thks for beta v1.1.3c will have a look.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-09 08:27:30
Out of interest, at what quality level does the click disappear?
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-09 10:25:54
Just tested -Q 1 --sortspread Vs -Q 0 --sortspread, the click was gone so I tried -Q 0.5 --sortspread, the click was gone too, so it might be a normal artefact due to very low setting, I dunno. All I know is that it is here at -Q 0 --sortspread. Then, as I had the files already encoded at hand, I made a very quick test (4 runs) to see how transparent were both -Q 0.5 --sortspread & -Q 1 --sortspread, none is transparent for me, the click is gone but some tearing remains, so if we ever remap setting, we should focus on -Q1.5 & Q2, I suspect -Q 2 --sortspread to already be transparent & -Q1.5 --sortspread to be near transparency, if we add -a 5 & --underlap 5, & if it stacks then -Q1.5 --sortspread -a 5 --underlap 5 is likely to be very very near transparency & -Q 1 --sortspread -a 5 --underlap 5 to be near transparency. All this remains to be tested further for confirmation. In fact IMHO we can already assume that transparency is reached with -Q 2 --sortspread, what remains to be tested is how wide is the safety margin compared to -Q 1.5 with added parameters & if -Q 1.5 with added parameters is already transparent too or not. (There I am not sure, it depends on how parameters stacks)

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/09 11:02:37

File A: C:\00- Therion (Artefact+Context) -Q 0 --sortspread.lossy.flac
File B: C:\00- Therion (Artefact+Context) -Q 1 --sortspread.lossy.flac

11:02:37 : Test started.
11:02:55 : 01/01  50.0%
11:03:09 : 02/02  25.0%
11:03:23 : 03/03  12.5%
11:03:44 : 04/04  6.3%
11:04:01 : 05/05  3.1%
11:04:25 : 06/06  1.6%
11:04:37 : 07/07  0.8%
11:04:49 : 08/08  0.4%
11:04:50 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/09 11:06:37

File A: C:\00- Therion (Artefact+Context) -Q 0 --sortspread.lossy.flac
File B: C:\00- Therion (Artefact+Context) -Q 0.5 --sortspread.lossy.flac

11:06:37 : Test started.
11:07:01 : 01/01  50.0%
11:07:16 : 02/02  25.0%
11:07:35 : 03/03  12.5%
11:07:48 : 04/04  6.3%
11:08:04 : 05/05  3.1%
11:08:18 : 06/06  1.6%
11:08:30 : 07/07  0.8%
11:08:48 : 08/08  0.4%
11:08:49 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/09 11:11:28

File A: C:\00- Therion (Artefact+Context) Lossless.flac
File B: C:\00- Therion (Artefact+Context) -Q 0.5 --sortspread.lossy.flac

11:11:28 : Test started.
11:11:41 : 01/01  50.0%
11:11:54 : 02/02  25.0%
11:12:12 : 03/03  12.5%
11:12:32 : 04/04  6.3%
11:12:53 : Test finished.

----------
Total: 4/4 (6.3%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/09 11:13:09

File A: C:\00- Therion (Artefact+Context) Lossless.flac
File B: C:\00- Therion (Artefact+Context) -Q 1 --sortspread.lossy.flac

11:13:09 : Test started.
11:13:28 : 01/01  50.0%
11:13:44 : 02/02  25.0%
11:14:09 : 03/03  12.5%
11:14:30 : 04/04  6.3%
11:14:35 : Test finished.

----------
Total: 4/4 (6.3%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-09 15:42:41
Could you possibly upload your problem section in which the click is produced on -q 0 processing? I will look at it in detail and revert.

Thanks.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-09 16:33:29
Here is a screenshot of the problem in audacity:

(http://img10.imageshack.us/img10/8067/00therionartefactcontex.png)

It's the Therion (Artefact+Context) sample at V1.1.3i -Q 0 --sortspread, but in the screenshot the sample is shortened to 2 sec & the left channel is splitted & muted in order to zoom on the problem.
It's somewhere between 1.70 & 1.80, I am not sure I highlighted the exact time. It is hard to ABX at the beginning (a light click only on the right hear), but now I can catch it 100% of time.

I am not sure it matters so much now as with or without this click -Q 0 --sortspread is not transparent anyway & settings that might be transparent are unaffected (-Q 2 --sortspread, -Q 1.5 --sortspread+ other parameters), but well if you can get rid of it.

You can download the 2 sec sample here (http://www.hydrogenaudio.org/forums/index.php?showtopic=71097)
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-10 00:22:10
I have managed my attachments in order to gain some space, so you can now download the a lossless version of the 2 sec sample in order to compare, I have added a screenshot of the lossless version that highlight in green another click that shouldn't be confused with the real artefact click, if you have very good ears you might hear a very light click in the lossless version, it is not the same click at all, the artefact I am speaking about doesn't occurs at the same timing. The normal very light click happens right in the middle of the tearing artefact so it's always masked by the tearing in not transparent lossywav encodes.

If you already don't hear the artefact click in the lossywav encode, I doubt you will ear the normal click in the lossless version, it's even lighter I think. But I prefer warning you to be sure that you don't mix both.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-10 22:09:43
I have (finally) worked out the formula for the added noise due to bitdepth reduction:

reference_threshold[fft_bit_length,bits_removed]:=((0.5*fft_bit_length+bits_removed-2.9163)+log10(1+2^-(2*bits_removed-1)/(2*log10(2))*20*log10(2).

The only empirical value is the -2.9163.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-10 22:46:12
I just tried V1.1.3c -Q0 --varspread Vs. V1.1.3c -Q0, I completely failed so there was no regression (no real progression either). The small success in Abfahrt Hinwil is in favor of --varspread if it wasn't placebo, so everything is fine for me.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/10 23:28:28

File A: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3c -Q0 --varspread.lossy.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3c -Q0.lossy.flac

23:28:28 : Test started.
23:29:01 : 01/01  50.0%
23:29:25 : 01/02  75.0%
23:29:40 : 02/03  50.0%
23:30:04 : 03/04  31.3%
23:30:28 : 03/05  50.0%
23:30:50 : 04/06  34.4%
23:31:16 : 05/07  22.7%
23:31:45 : 05/08  36.3%
23:32:28 : 06/09  25.4%
23:33:25 : 07/10  17.2%
23:33:59 : 08/11  11.3%
23:34:48 : 09/12  7.3%
23:35:17 : 09/13  13.3%
23:36:10 : 09/14  21.2%
23:36:35 : Test finished.

----------
Total: 9/14 (21.2%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/10 23:38:12

File A: C:\00- Therion (Artefact+Context) V1.1.3c -Q0 --varspread.lossy.flac
File B: C:\00- Therion (Artefact+Context) V1.1.3c -Q0.lossy.flac

23:38:12 : Test started.
23:39:20 : 00/01  100.0%
23:40:03 : 01/02  75.0%
23:40:32 : 01/03  87.5%
23:41:28 : 01/04  93.8%
23:42:10 : 02/05  81.3%
23:42:55 : 03/06  65.6%
23:43:24 : 04/07  50.0%
23:43:51 : 04/08  63.7%
23:43:54 : Test finished.

----------
Total: 4/8 (63.7%)
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-11 17:37:39
Just tested V1.1.3i -Q 0 --underlap 5 Vs. V1.1.3i -Q 0.5, I did it because I wanted to evaluate the gain from --underlap 5 compared to --sortspread & -a 5. I was happyly surprised -Q 0 --underlap 5 beats -Q 0.5. This is not an evidence on the Therion sample, on this sample both are near to be tied (which is already quite good for --underlap 5), then I tried Abfahrt Hinwil & it was much easier to ABX in favor of --underlap 5.

The conclusion of all this is that:
--sortspread > 0.5 gain in the scale (with a minor click on -q 0)
--underlap 5 >= 0.5 gain in the scale
-a 5 > 0.25 gain in the scale (In fact -a 5 is the weakest of the additionnal parameter)

So, this remains to be tested, but I am pretty confident that if all this stacks together we can expect -q 1.5 --sortspread --underlap 5 -a 5 to be transparent. In all logic it should be.

This should lead lossywav to be transparent near ~360Kbps & I expect a -~18Kbps or 4Mo by album gain from it.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/11 18:07:43

File A: C:\00- Therion (Artefact+Context) V1.1.3i -Q 0 --underlap 5.lossy.flac
File B: C:\00- Therion (Artefact+Context) V1.1.3i -Q 0.5.lossy.flac

18:07:43 : Test started.
18:08:57 : 01/01  50.0%
18:09:39 : 02/02  25.0%
18:10:03 : 03/03  12.5%
18:10:36 : 04/04  6.3%
18:11:25 : 05/05  3.1%
18:11:54 : 05/06  10.9%
18:12:41 : 05/07  22.7%
18:14:25 : 06/08  14.5%
18:15:17 : 07/09  9.0%
18:17:11 : 07/10  17.2%
18:17:14 : Test finished.

----------
Total: 7/10 (17.2%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/11 18:18:28

File A: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3i -Q 0 --underlap 5.lossy.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3i -Q 0.5.lossy.flac

18:18:28 : Test started.
18:19:25 : 01/01  50.0%
18:19:33 : 02/02  25.0%
18:19:43 : 03/03  12.5%
18:20:01 : 04/04  6.3%
18:20:25 : 05/05  3.1%
18:21:19 : 06/06  1.6%
18:21:43 : 07/07  0.8%
18:21:58 : 08/08  0.4%
18:22:01 : Test finished.

----------
Total: 8/8 (0.4%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-15 20:32:28
lossyWAV beta 1.1.3j appended to post #1 in this thread.

Explanation of revised -X or --sortspread <n> parameter:

After FFT analysis and skewing of the FFT results, the portion of the FFT_result array between 20Hz and 16kHz (or as specified by --limit) is sorted in order of ascending numerical value. Once sorted, up to 8 values are used to determine the result of the spreading function. i.e.

for --sortspread 0, only the second lowest result is used to determine the result;
for --sortspread 1, the first three results are multiplied by (0.2500,0.5000,0.2500) respectively and added  to determine the result;
for --sortspread 2, the first three results are multiplied by (0.3333,0.3333,0.3333) respectively and added  to determine the result;
for --sortspread 3, the first four results are multiplied by (0.4214,0.2770,0.1820,0.1197) respectively and added  to determine the result;
for --sortspread 4, the first five results are multiplied by (0.4594,0.2608,0.1481,0.0841,0.0477) respectively and added  to determine the result;
for --sortspread 5, the first six results are multiplied by (0.4781,0.2548,0.1357,0.0723,0.0385,0.0205) respectively and added  to determine the result;
for --sortspread 6, the first seven results are multiplied by (0.4880,0.2522,0.1303,0.0674,0.0348,0.0180,0.0093) respectively and added  to determine the result;
for --sortspread 7, the first eight results are multiplied by (0.4934,0.2511,0.1278,0.0651,0.0331,0.0168,0.0086,0.0044) respectively and added to determine the result;

3 to 7 have a constant ratio between adjacent elements of the multiplier array and 0 to 7 are "centred" about the second element.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-16 01:48:11
I just tested V1.1.3j -Q 0 --sortspread 7 to see if the click was gone. It is not gone, --sortspread 7 doesn't affect it at all. Honestly if you can't get rid of it, I can sleep with it as I know it disappears at bitrate that are supposed to be transparent. This is very -q 0 specific & I will never use -q 0 as long as it is not transparent even without the click. I begin to think that this click is due to low setting, that's all.
You should focus on what you want to swallow as used by default within the new parameters IMHO. What do you want to swallow between --sortspread 2, --underlap 5 & -a 5 ?
I failed to noticed any difference between --sortspread 2 & --sortspread 7 on Abfahrt Hinwil (I tried quickly Therion, there was no obvious difference too).

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/16 02:43:03

File A: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 0 --sortspread 7.lossy.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 0 --sortspread 2.lossy.flac

02:43:03 : Test started.
02:44:15 : 00/01  100.0%
02:44:23 : 01/02  75.0%
02:44:50 : 02/03  50.0%
02:45:15 : 02/04  68.8%
02:45:42 : 03/05  50.0%
02:45:53 : 03/06  65.6%
02:46:08 : 03/07  77.3%
02:46:51 : 03/08  85.5%
02:46:59 : Test finished.

----------
Total: 3/8 (85.5%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-16 07:35:48
Using --underlap 5 (I fixed the check in the parameter processing and it will now accept up to the advertised 8) will increase the processing time dramatically - using 2 would be a compromise (for one thing, it adds a central 1024 sample FFT analysis in addition to the two existing per codec-block). I prefer using --analyses 5 to add different FFT lengths into the calculation. My preference for --sortspread is 3 as it is weighted more to the first (i.e. lowest) result in the sorted FFT_Result array.

I haven't had time to examine the cause of the click in Therion - I feel that it may be something underlying in the lossless version which is exacerbated during the bit-removal process.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-04-16 16:37:42
Unfortunately I can't help out with testing of these new settings as my hearing/listening situation don't allow me to pick up any problems. However, would it be a good idea to test the new settings with some other material? I realise that these are problem pieces but if transparency were achieved with other pieces surely that would prove their usefulness
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-16 22:48:40
For me what you're asking for is non-sense, on average music lossywav is transparent at much lower than -P, if you try to ABX this you will get useless 50% random guessing results which will not help improving the codec in any way. Yes, lossywav need more ABXable problem samples in its test database for safety. Therion & Abfahrt Hinwil are the only samples I know that can be used to tune lossywav near its transparency point. Adding popular song X by artist Y that is transparent on -q 0.5 is completely useless for testing if we know that other samples test fail at -q 1.5. Who cares if you're a Metallica fan & the complete Metallica discography is transparent at lossywav -q 0.5 for you, if lossywav fails to bring transparency on Therion & Abfahrt at -q 1.5 for me, it fails overall up to -q 1.5, that's all.
If you know new problem samples for lossywav at -q 1.5 (or -q 1) just post them, otherwise you'd better stay out of it.

For exemple Roberto's listening test Multiformat at 128kbps (http://www.rjamorim.com/test/multiformat128/results.html) shows that on average music most people are happy with aotuv -q4, but if you know about the block switching issue it is obvious that -q4 fails on many samples Multi-Codec Listening Test (http://www.hydrogenaudio.org/forums/index.php?showtopic=70442). What does that means ? For me, it means that testing codec on average music with average users leads you to average conclusion. If you let other people decide for yourself then you can happily use MP3 at 128KBps, 95% user will find that it sound very good if don't specifically point them to listen to audio artefact. The average Joe doesn't know what to listen to, so he doesn't know how to criticize audio quality. If you only read Roberto's listening test then the block switching issue doesn't exist & aotuv is near perfect. Maybe there was a guy within the testers who was able to tell where it fails but if 99% of other guy tells that there is no problem, then the problem doesn't exist. Prey you're not one of those able to ABX the artefact without knowing it, because if it happens to you & you discover it , you're good to delete your whole collection, I already deleted my lossy backup several time before switching to 100% lossless & stop trusting anyone else but me.

Like it or not the only way to test lossy audio codec is through problem samples, ... but before you can even start listening to the problem cases, you have to actually find them. So your proposal to test lossywav on average music is non-sense & a pure waste on time. You can test lossywav on average music to find possible problem samples, you cannot test lossywav on average music to judge its quality.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-17 11:08:32
I just tested --underlap 2 vs --underlap 8, --underlap 8 is always better & ABXable.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/17 11:26:04

File A: C:\00- Therion (Artefact+Context) V1.1.3j -Q 0 --underlap 2.lossy.flac
File B: C:\00- Therion (Artefact+Context) V1.1.3j -Q 0 --underlap 8.lossy.flac

11:26:04 : Test started.
11:26:27 : 01/01  50.0%
11:26:53 : 02/02  25.0%
11:27:15 : 03/03  12.5%
11:27:48 : 04/04  6.3%
11:28:22 : 05/05  3.1%
11:28:44 : 06/06  1.6%
11:29:06 : 07/07  0.8%
11:29:25 : 08/08  0.4%
11:29:27 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/17 11:29:42

File A: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 0 --underlap 2.lossy.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 0 --underlap 8.lossy.flac

11:29:42 : Test started.
11:29:55 : 01/01  50.0%
11:30:05 : 02/02  25.0%
11:30:15 : 03/03  12.5%
11:30:22 : 04/04  6.3%
11:30:31 : 05/05  3.1%
11:30:42 : 06/06  1.6%
11:30:50 : 07/07  0.8%
11:30:58 : 08/08  0.4%
11:30:59 : Test finished.

----------
Total: 8/8 (0.4%)


Then I tested -Q 0 --underlap 2 Vs. -Q 0.5 in order to see if --underlap 2 was better or worse than --underlap 5 (which I previously tested & is near to a 0.5 gain in the scale), first I add some trouble catching a difference but then I catched something & then it was a 100% success so I went up to 20 trials to erease the first failures. It was even easier on Abfahrt Hinwil which was a 8/8 from the start. The conclusion is --underlap 2 is worst than a 0.5 increase in the scale. --underlap is a time greedy parameter but it does improve quality in a very scalable way:

--underlap 2 < --underlap 5 (+0.5 increase in the scale) < --underlap 8, but --underlap 8 is very slow.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/17 11:33:05

File A: C:\11- Lossywav Test\00- Therion (Artefact+Context) V1.1.3j -Q 0 --underlap 2.lossy.flac
File B: C:\11- Lossywav Test\00- Therion (Artefact+Context) V1.1.3j -Q 0.5.lossy.flac

11:33:05 : Test started.
11:33:39 : 00/01  100.0%
11:36:06 : 01/02  75.0%
11:36:27 : 02/03  50.0%
11:37:32 : 02/04  68.8%
11:38:06 : 02/05  81.3%
11:38:56 : 03/06  65.6%
11:39:13 : 04/07  50.0%
11:39:26 : 05/08  36.3%
11:39:37 : 06/09  25.4%
11:39:49 : 07/10  17.2%
11:40:02 : 08/11  11.3%
11:40:14 : 09/12  7.3%
11:40:26 : 10/13  4.6%
11:40:33 : 11/14  2.9%
11:40:47 : 12/15  1.8%
11:41:06 : 13/16  1.1%
11:41:19 : 14/17  0.6%
11:41:34 : 15/18  0.4%
11:42:02 : 16/19  0.2%
11:42:32 : 17/20  0.1%
11:42:37 : Test finished.

----------
Total: 17/20 (0.1%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/17 11:44:12

File A: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 0 --underlap 2.lossy.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 0.5.lossy.flac

11:44:12 : Test started.
11:44:57 : 01/01  50.0%
11:45:17 : 02/02  25.0%
11:45:37 : 03/03  12.5%
11:45:56 : 04/04  6.3%
11:46:11 : 05/05  3.1%
11:46:29 : 06/06  1.6%
11:46:49 : 07/07  0.8%
11:47:07 : 08/08  0.4%
11:47:09 : Test finished.

----------
Total: 8/8 (0.4%)



I tried too ABX -q 0 --sortspread 3 Vs. -q 0 --sortspread 7, but I failed, it seems --sortspread is not as scalable as --underlap. Unlike --underlap it seems --sortspread is a flat ~0.5 quality increase in the scale that doesn't improve much if you increase the parameter.

It remains to be tested for transparency but IMHO swallowing (-q 1.5 --sortspread 3 --analyses 5) or (-q 2 --sortspread 3 --analyses 5) as the new -portable might be a good start if --underlap is too CPU greedy. (-q 2 --sortspread 3 --analyses 5) is for almost sure already transparent with no drawback ... in fact all that remains to be tested is how far (-q 1.5 --sortspread 3 --analyses 5) is from being transparent. Also what is the min/max for --analyses, I know from --help the default is 2 but is there any logic behind 5 analyses instead of more or less, do you have a preference for --analyses ? Is 5 random ?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-17 11:49:00
The 2 default FTT analyses are 64 samples and 1024 samples (1.45msec and 23.2msec for 44.1kHz). --impulse adds a 32 sample FFT for impulse detection (0.725msec). What --analyses 5 does is add 128, 256 and 512 sample FFT analyses into the calculation.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-17 12:06:54
I tested V1.1.3j -Q 1.5 --sortspread 3 for transparency, it is not transparent but it's medium/hard to ABX & artefact are very low. It doesn't sound bad at all. It might be transparent with the help of --underlap 2 or --analyses 5.
Just tell me if swallowing --underlap 2 is possible or not. --analyses 5 alone might help but alone it seems a little weak IMHO.

It's either you swallow -q 1.5 --sortspread 3 --underlap 2 --analyses 5 as the new -portable (but without --underlap 2 the safety margin will be too low IMHO)
or you only swallow -q 2 --sortspread 3 --analyses 5 as the new -portable (in fact -q 2 --sortspread 3 is most likely already transparent & --analyses 5 only adds a bigger safety margin here)

another solution is -q 1.75 --sortspread 3 --analyses 5 as the new -portable, the advantage is that you left --underlap 2 out, but you still gain some more Kbps.

--sortspread 3 is strong (~0.5 gain) but both --underlap 2 & --analyses 5 are weaker (~0.25 gain each) & --underlap 2 is slower. --underlap has a major speed drawback but it can be just as strong as --sortspread if you are ready to sacrifice encoding speed. It's lossy & you only encode once.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/17 12:50:07

File A: C:\11- Lossywav Test\00- Therion (Artefact+Context) Lossless.flac
File B: C:\11- Lossywav Test\00- Therion (Artefact+Context) V1.1.3j -Q 1.5 --sortspread 3.lossy.flac

12:50:07 : Test started.
12:50:25 : 01/01  50.0%
12:50:41 : 02/02  25.0%
12:50:52 : 03/03  12.5%
12:51:21 : 04/04  6.3%
12:52:27 : 05/05  3.1%
12:53:00 : 06/06  1.6%
12:53:53 : 07/07  0.8%
12:54:41 : 08/08  0.4%
12:54:42 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/17 12:55:12

File A: C:\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 1.5 --sortspread 3.lossy.flac

12:55:12 : Test started.
12:55:29 : 01/01  50.0%
12:55:50 : 02/02  25.0%
12:56:12 : 03/03  12.5%
12:56:30 : 04/04  6.3%
12:57:21 : 05/05  3.1%
12:57:51 : 06/06  1.6%
12:58:27 : 07/07  0.8%
12:59:16 : 07/08  3.5%
12:59:18 : Test finished.

----------
Total: 7/8 (3.5%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-17 13:30:37
I think that over the weekend I will process my 55 problem sample set using a number of different settings to determine which give the best improvement (determined by your existing ABX'ing) for the lowest increase in bitrate (determined by these tests). In this way we should be able to determine which are the best combination(s).

I think that users should be able to "slow down" the calculation speed by increasing the underlap - I would be prepared to accept --underlap 2 as a default value (with 1 available as a user selection). Similarly, maybe --analyses 5 is a desirable default, again with user options to use 2 to 4 analyses. I am reluctant to reduce the processing rate too much, however, if better quality can be achieved at the same bitrate but just a bit slower then that is a desirable result.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-17 13:41:30
I quickly tested the impact of the various new parameters on a real album just to have an idea. I didn't run the encode untill the end, I only took the estimated time after 30sec of encoding:

-q 2 4min
-q 2 --sortspread 3 4min 18 sec
-q 2 --underlap 2  5min 44 sec
-q 2 --analyses 5  6min 37 sec

It seems --analyses 5 has a bigger impact on speed than --underlap 2 for a comparable gain in the scale. (previously I thought --underlap 2 was worst)

--sortspread 3 is ok but adding both --underlap 2 & --analyses 5 at the same time will have a major impact on encoding time, it might be something like X2 slower. (4min+1min44+2min37)

I should test --underlap 2 Vs. --analyses 5 to see what is best qualitywise, but if you take speed in consideration then -q 1.75 --sortspread 3 --underlap 2 (WITHOUT --analyses 5) might very well be the best compromise between speed/quality/size. At first look that's my personnal favorite. So far I didn't realized --analyses 5 & --underlap 2 were so slow, --sortspread 3 is much much stronger as a new parameter, not only it sounds better, but it's much faster. I am wondering if a 0.25 quality gain is worth the speed cost for both --analyses 5 & --underlap 2, a new noise shaping method might be a much better option  Maybe just swallowing -q 2 --sortspread 3 as the new -portable is the best option afterall, it's a flat 0.5 gain in the scale without any real speed drawback, without the headache of ABXing minor 0.25 gain in the scale. It seems a little strange to me to swallow --underlap 2 & left out --analyses 5, when both are so close.

I dunno ... I like the quality gain, but I don't like the speed drawback personnaly.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-17 14:26:04
A quick test using one album:
Code: [Select]
|---------------------------------------------------------|
| -q 2 | -X 3 | -U 2 | -a 5 |  Time  |  Bitrate  |Increase|
|------+------+------+------+--------+-----------+--------|
|   X  |      |      |      |  51.32 | 394.8kbps | ------ |
|   X  |   X  |      |      |  66.31 | 414.1kbps | +4.89% |
|   X  |      |   X  |      |  84.32 | 406.6kbps | +3.00% |
|   X  |      |      |   X  |  99.43 | 415.2kbps | +5.17% |
|   X  |   X  |   X  |      | 112.56 | 424.2kbps | +7.44% |
|   X  |   X  |      |   X  | 136.81 | 425.0kbps | +7.64% |
|   X  |      |   X  |   X  | 179.03 | 428.4kbps | +8.51% |
|   X  |   X  |   X  |   X  | 247.19 | 435.0kbps |+10.19% |
|---------------------------------------------------------|
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-17 15:17:31
As -U is very scalable it might be more interesting to increase -U rather than using -a 5 at all.

It might be interesting to add -U 3-8 to your table, so that we know what -U (Number) alone encode as slow as -U 2 -a 5 together, then I could ABX both to see if it's not more interesting to increase -U alone than using -a 5 at all. Maybe it's not worth at all, but it remains to be tested.

I know you previously said you prefer -a 5 over - U 2, but speed test shows the exact opposite & -U 2 has the advantage of being scalable.

Personnaly I prefer - U 2 over -a 5, because it's faster for the same quality gain. So following some logic, - U 3 (or maybe  - U 4, I have no clue) should be better qualitywise for the same speed.

- U 2 -a 5 is very time greedy (179.03), I am confident that - U 3 (or - U (Number) ) alone will be faster for an almost comparable quality gain because from - U 2 (84.32) to - U 2 -a 5 (179.03) the margin is big for a -U (number) in between to swallow the quality gain of -a 5 without being as slow.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-04-17 16:12:04
I think this is getting a bit too empirical!  Well, for me to contribute anyway. So I'll add this intead...


With --underlap, you know exactly what it's doing - with the default settings, there's a moment between two adjacent analyses which is 6dB down from the centre of each - that's the hanning window function. --underlap 2 changes this to only 1.4dB down.

It's a little over simplistic, but basically that sets the maximum magnitude of reduction you could get in the added noise with --underlap 2: 4.6dB. Mostly you won't get anything like this, but you should never get any more - i.e. it can never give a greater improvement that this.

Remember that lossyWAV adds noise in 6dB increments - that's what "one bit" is. Underlap 2, at its most generous, is never as generous as keeping one extra bit overall - at most it'll keep 1 extra bit sometimes. Given the 6dB resolution of the final process, it seems a bit mad going for an underlap greater than 2 because you're getting seriously diminishing returns.

e.g. from underlap 2 to underlap 3, the improvement is less than 1dB. Given the 6dB step size, I think this is silly - unless you can prove that it's the less than 1dB "error" in this domain that's the cause of occasional audible problems.


With --analyses x, you're adding more time/frequency resolution stages. This could be a good thing. I can't think of an easy way of predicting the min/max improvement it can bring: it's dependent on the signal and spreading functions. On specific signals, it could force down the added noise dramatically at specific moments. If perceptually relevant, this is highly useful. It's unlikely to force the added noise down across the board (good, because there isn't a problem across the board!) - because most signal dips are already being correctly caught by two analyses.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-17 16:32:13
I don't understand all this theoric stuff, but from what I understund this sentence: "it seems a bit mad going for an underlap greater than 2 because you're getting seriously diminishing returns." is wrong I can ABX --underlap 2 from --underlap 5 & --underlap 2 from --underlap 8. Not only I can ABX it but it is not very hard to ABX. So despite all this theoric knowledge there is a gain in real listening test.

Also, Quote: "I think this is silly" ... again sorry but I am 100% sure of what I hear. The "diminishing returns" of --underlap are not as small as you think. --underlap may have its wall, I am not sure I can ABX --underlap 5 vs. --underlap 8, I never tried, but --underlap 2 is not the upper limit of --underlap at all, that is 100% sure. I have no explanation for it, there might be holes in your theoric knowledge, I dunno. All I know is what I hear.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-17 16:50:50
Just tried --underlap 2 vs. --underlap 4, this should remove you any doubt about --underlap 2 not being the upper limit of --underlap.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/17 17:47:02

File A: C:\00- Therion (Artefact+Context) V1.1.3j -Q 0 --underlap 2.lossy.flac
File B: C:\00- Therion (Artefact+Context) V1.1.3j -Q 0 --underlap 4.lossy.flac

17:47:02 : Test started.
17:47:17 : 01/01  50.0%
17:47:32 : 02/02  25.0%
17:47:51 : 03/03  12.5%
17:48:20 : 04/04  6.3%
17:48:39 : 05/05  3.1%
17:49:01 : 06/06  1.6%
17:49:23 : 07/07  0.8%
17:49:43 : 08/08  0.4%
17:49:45 : Test finished.

----------
Total: 8/8 (0.4%)
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-04-17 17:55:33
@sauvage78,

I'm not trying to argue against what you hear - far from it - that's far more important than what I think!


My understanding of how it probably works might not be consistent will all the tunings Nick has added, or the way it's really implemented. It's strange it makes a huge difference, but I'm sure there's an explanation.

I think it's worth investigating what's happening in this specific sample.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-04-17 17:56:29
For me what you're asking for is non-sense, on average music lossywav is transparent at much lower than -P, if you try to ABX this you will get useless 50% random guessing results which will not help improving the codec in any way. Yes, lossywav need more ABXable problem samples in its test database for safety. Therion & Abfahrt Hinwil are the only samples I know that can be used to tune lossywav near its transparency point. Adding popular song X by artist Y that is transparent on -q 0.5 is completely useless for testing if we know that other samples test fail at -q 1.5. Who cares if you're a Metallica fan & the complete Metallica discography is transparent at lossywav -q 0.5 for you, if lossywav fails to bring transparency on Therion & Abfahrt at -q 1.5 for me, it fails overall up to -q 1.5, that's all.
If you know new problem samples for lossywav at -q 1.5 (or -q 1) just post them, otherwise you'd better stay out of it.

For exemple Roberto's listening test Multiformat at 128kbps (http://www.rjamorim.com/test/multiformat128/results.html) shows that on average music most people are happy with aotuv -q4, but if you know about the block switching issue it is obvious that -q4 fails on many samples Multi-Codec Listening Test (http://www.hydrogenaudio.org/forums/index.php?showtopic=70442). What does that means ? For me, it means that testing codec on average music with average users leads you to average conclusion. If you let other people decide for yourself then you can happily use MP3 at 128KBps, 95% user will find that it sound very good if don't specifically point them to listen to audio artefact. The average Joe doesn't know what to listen to, so he doesn't know how to criticize audio quality. If you only read Roberto's listening test then the block switching issue doesn't exist & aotuv is near perfect. Maybe there was a guy within the testers who was able to tell where it fails but if 99% of other guy tells that there is no problem, then the problem doesn't exist. Prey you're not one of those able to ABX the artefact without knowing it, because if it happens to you & you discover it , you're good to delete your whole collection, I already deleted my lossy backup several time before switching to 100% lossless & stop trusting anyone else but me.

Like it or not the only way to test lossy audio codec is through problem samples, ... but before you can even start listening to the problem cases, you have to actually find them. So your proposal to test lossywav on average music is non-sense & a pure waste on time. You can test lossywav on average music to find possible problem samples, you cannot test lossywav on average music to judge its quality.

Well, I suggested "other pieces" not "average music". Sorry for trying to help. If everybody's happy to try to tune Lossywav on 2 samples that's fine with me
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-17 18:30:28
botface:
The problem is that there is no other pieces easyly ABXable at -q 1.5 that we are aware of, hopefully tuning this 2 samples will help tuning lossywav on all music. Everything is tied, it's not as if every sample would be separated from the rest of the music. So far each parameter that affects one of the 2 samples affect the second in the same way (good or bad) & so far it never happen that improving the quality on this 2 samples would decrease the quality of lossywav on the rest of the music, this is not random. I am not completely nuts listening endlessly to the same 2 samples ... well not yet I hope but I may become 
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-04-17 20:18:29
botface:
So far each parameter that affects one of the 2 samples affect the second in the same way (good or bad) & so far it never happen that improving the quality on this 2 samples would decrease the quality of lossywav on the rest of the music,....

OK. My concern was that in concentrating on these two samples adverse effects could paerhaps be introduced that may affect other samples that had been OK up to now. I didn't realise you were checking other samples for regression too.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-18 03:32:21
I just tried to encode an album 3 times in order to compare the final filesize:

204 Mo (214 186 464 octets) -q 2.5
200 Mo (210 518 460 octets) -q 2 --sortspread 3
204 Mo (214 348 111 octets) -q 2 --analyses 5

The conclusion of this is:
--analyses 5 completely not worth it, it is slow & produce file of similar size than a 0.5 increase in the scale while bringing only a quality improvement of 0.25.
--sortspread 3 is completely worth it, it is not much slower & produce smaller size than a 0.5 increase in the scale while bringing a quality improvement slightly superior to 0.5

--underlap 2 is somewhere in between, it is faster/smaller/better than --analyses 5, but not as good as --sortspread 3. It is a trade between quality & speed, with a medium size increase.
It is not absolutely worth it, but not absolutely worthless at the same time. So I think this should be left out for now but remembered as a possible option.

IMHO --analyses 5 must be forgotten, --underlap must be tested further & --sortspread 3 must be immediatly swallowed.

If this is confirmed by your encoding test, plz adopt --sortspread 3 as default & -q 2 --sortspread 3 as the new -portable & make a release.

The overall conclusion of this is that halb27 was right, a new parameter that increase the quality must be tested for speed & outputed filesize before making any conclusion, at first look --analyses 5 looked promising on problem samples, but on real life encoding it worth nothing.

Note: I think I am done with lossywav testing for some time now, real life is calling me.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-18 08:58:07
Both --analyses 5 and underlap 2 add extra FFT analyses. As both increase the bitrate / reduce the bits-removed for a given file / quality level combination it is inferred that they are both finding low points in the noise floor which are not found when they are not used.

I will get to the comparative analysis later today - it may indeed be that we end up with a revised --portable using --quality 2 --sortspread 3.
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-04-18 21:11:16
IMHO --analyses 5 must be forgotten, --underlap must be tested further & --sortspread 3 must be immediatly swallowed.

If this is confirmed by your encoding test, plz adopt --sortspread 3 as default & -q 2 --sortspread 3 as the new -portable & make a release.

First, thanks for the testing you did. Second, I like to remind that these conclusion would hold for qualities around -q 2 (I guess +/- 1).  Should --standard become -q 4.5 --underlap 3, or is it better (and faster) at -q 5 (without -U)?
Perhaps we can never tell?

Both --analyses 5 and underlap 2 add extra FFT analyses. As both increase the bitrate / reduce the bits-removed for a given file / quality level combination it is inferred that they are both finding low points in the noise floor which are not found when they are not used.

yes, that is confirmed by your test table. Wouldn't that mean the thresholds should be adjusted too? (I'm asking just in general)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-18 21:28:57
I think that the main problem we have is that there are a large number of parameters to be balanced - some of which are not user selectable, i.e. skew, noise-threshold-shift and signal-to-noise-ratio.

My gut feeling is that --underlap 2 is a worthwhile addition to the default settings, as is --sortspread 3 (from the much appreciated testing by sauvage78!).

Bitrate creep (upward) has set in - but, if as has been suggested, we can always tweak the presets downward (i.e. maybe --portable maps to -q 2.1 not -q 2.5; etc.).

I think that I should remap the noise-threshold-shifts so that --sortspread 3 output is bitrate matched to default spreading for each of the 10 permanent quality levels (as I did for the (now old) --sortspread parameter).

I will carry out this and revert --sortspread to a switch rather than taking a parameter - this with a view to final round of testing before (if) --sortspread becomes the default spreading and the existing default spreading function is removed.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-19 06:56:56
I fear that -q 2 --sortspread 3 is not fully transparent, this is very dispointing, below is logs from -q 2 --sortspread 3 Vs. lossless & -q 2.5 Vs. lossless. -q 2 --sortspread 3 Vs. lossless is not a full success but it is not like -q 2.5 Vs. lossless a full failure, worse it's closer to be a success than to be a failure.  -q 2 --sortspread 3 is hard to ABX & artefact are very light, it sounds good, but I would lie if I would tell that I think it is transparent. After all this tests this is quite discouraging. We are very near to transparency but not there yet & if we increase the bitrate then --sortspread 3 might become useless    I dunno what to think ... I think that the explanation to this is diminishing returns, near -q 0 --sortspread 3 there is a big boost of ~0.5 in the scale, but the closer you get to -q 2.5, the slimest is the gain. All logic would have been that -q 2 --sortspread 3 would have been transparent but it is not. Sadly it is slightly below.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/19 07:23:34

File A: C:\11- Lossywav Test\00- Therion (Artefact+Context) Lossless.flac
File B: C:\11- Lossywav Test\00- Therion (Artefact+Context) V1.1.3j -Q 2 --sortspread 3.lossy.flac

07:23:34 : Test started.
07:23:57 : 01/01  50.0%
07:24:21 : 02/02  25.0%
07:24:48 : 02/03  50.0%
07:25:01 : 03/04  31.3%
07:25:17 : 04/05  18.8%
07:25:33 : 05/06  10.9%
07:25:57 : 06/07  6.3%
07:26:28 : 06/08  14.5%
07:27:10 : 07/09  9.0%
07:27:55 : 08/10  5.5%
07:28:47 : 08/11  11.3%
07:29:48 : 09/12  7.3%
07:30:06 : 10/13  4.6%
07:31:28 : 11/14  2.9%
07:32:22 : 11/15  5.9%
07:33:37 : 12/16  3.8%
07:33:47 : Test finished.

----------
Total: 12/16 (3.8%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/19 07:34:08

File A: C:\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 2 --sortspread 3.lossy.flac

07:34:08 : Test started.
07:34:35 : 01/01  50.0%
07:35:09 : 02/02  25.0%
07:35:43 : 03/03  12.5%
07:36:11 : 03/04  31.3%
07:36:39 : 04/05  18.8%
07:37:03 : 05/06  10.9%
07:37:28 : 06/07  6.3%
07:37:46 : 07/08  3.5%
07:38:14 : 08/09  2.0%
07:38:50 : 08/10  5.5%
07:38:55 : Test finished.

----------
Total: 8/10 (5.5%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/19 07:42:02

File A: C:\00- Therion (Artefact+Context) Lossless.flac
File B: C:\00- Therion (Artefact+Context) V1.1.3j -Q 2.5.lossy.flac

07:42:02 : Test started.
07:42:42 : 00/01  100.0%
07:43:01 : 01/02  75.0%
07:43:31 : 01/03  87.5%
07:44:05 : 01/04  93.8%
07:44:24 : 02/05  81.3%
07:44:42 : 02/06  89.1%
07:45:12 : 03/07  77.3%
07:45:40 : 04/08  63.7%
07:45:51 : Test finished.

----------
Total: 4/8 (63.7%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/19 07:46:03

File A: C:\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 2.5.lossy.flac

07:46:03 : Test started.
07:46:22 : 01/01  50.0%
07:46:36 : 01/02  75.0%
07:46:50 : 02/03  50.0%
07:47:12 : 03/04  31.3%
07:47:27 : 04/05  18.8%
07:47:56 : 04/06  34.4%
07:48:17 : 04/07  50.0%
07:48:36 : 04/08  63.7%
07:48:38 : Test finished.

----------
Total: 4/8 (63.7%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-19 08:19:48
This is actually very encouraging - although transparency is the goal, it is *not* the goal for --portable. lossyWAV *requires* to be transparent (as far as we know due to testing / samples / etc) at --standard. It's good to see how low down the quality scale the transparency point seems to be.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-19 10:53:50
It's true that officially transparency was not the target for option '--portable'.
In practice however I see things have changed in users' minds. Despite the fact that in theory we would allow for small problems with --portable what most people really did is expect transparency for --portable, more or less. It took me a long time to do so (I sticked with standard) but facing the fact that --standard is overkill I jumped up the --portable expected transparency train as well.
And it's fine IMO. 2Bedecided's basic idea was great and worked well in the 500 kbps region from the very start. Your tuning details were great as well, Nick, and it is your merit that you brought transparency to lossyWAV in the 350...400 kbps region.
Maybe -q 2.5 is still overkill for 'practical transparency' (whatever this may mean exactly - something like allowing for a priori not exactly determined very small deviations from transparency), but the current proposals for additional switches on a smaller -q basis are only slight variations of the noise analysis as is. My gut feeling is that this won't really bring us forward, but it's also a bit dangerous in the face of the fact that what we've reached already is excellent. It's a bit dangerous as there is already a tendency that bitrate of our excellent quality settings have a strong tendency to go up, and it's also dangerous because changes in the machinery can also bring quality regression, be it simply because of implementation errors or by quality regressions of the new techniques which can remain unnoticed for a long time. IMO such a danger must be accepted when real new and promising ideas come up, but this is not the case IMO for the current variants of decision making based on the FFT analysis and also not for more elaborate variants of the FFT analyses itself. For these noise analysis variants my suggestion is as stated before: in order to be a promising variant bitrate for the --portable quality should go up more for a problem sample set than for a regular music sample set. If this is not the case I would not go into listening tests - all with respect to the fact that it's excellent already what we've got.

To back up what I mean in addition to what sauvage78 has found I just tested v1.1.3j on my typical sample set with bitrate in mind:

--portable      385 kbps
-q 2              371 kbps      (not so much lower than --portable so not too attractive)
-q 2 -X 3       384 kbps      (pretty useless compared to --portable especially as quality decreases)
-q 2 -s 0.3     376 kbps      (as an alternative approach focusing on noise shaping instead of noise analysis variants)

In the end I think we can be very happy with plain --portable, and I stop using -a 5.
I think using a slightly stronger noise shaping like -s 0.3 or -s 0.35 for -q 2 and/or --portable can bring a slight advantage (in terms of a slight bitrate decrease and/or a slightly increased safety margin) against using plain --portable as it is now.

@sauvage78:
Thanks a lot for testing. Do you mind testing -q 2 -s 0.3 and/or -q 2 -s 0.35 ?
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-19 12:50:01
Because ABXing -q2 takes a time, I tried  -s 0.35 at -q 0, for me  -s 0.35 is a regression, it make sound slightly crackles.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/19 13:38:33

File A: C:\00- Therion (Artefact+Context) V1.1.3j -Q 0 -s 0.35.lossy.flac
File B: C:\00- Therion (Artefact+Context) V1.1.3j -Q 0.lossy.flac

13:38:33 : Test started.
13:38:52 : 01/01  50.0%
13:39:01 : 02/02  25.0%
13:39:38 : 03/03  12.5%
13:40:18 : 04/04  6.3%
13:40:48 : 05/05  3.1%
13:41:08 : 06/06  1.6%
13:41:34 : 07/07  0.8%
13:41:51 : 08/08  0.4%
13:41:53 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/19 13:43:06

File A: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 0 -s 0.35.lossy.flac
File B: C:\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -Q 0.lossy.flac

13:43:06 : Test started.
13:43:26 : 01/01  50.0%
13:43:38 : 02/02  25.0%
13:43:55 : 02/03  50.0%
13:44:08 : 02/04  68.8%
13:44:46 : 03/05  50.0%
13:45:04 : 04/06  34.4%
13:45:24 : 05/07  22.7%
13:45:38 : 06/08  14.5%
13:45:57 : 07/09  9.0%
13:46:20 : 07/10  17.2%
13:46:37 : 08/11  11.3%
13:47:04 : 09/12  7.3%
13:47:07 : Test finished.

----------
Total: 9/12 (7.3%)


IMHO the best thing to do is to release 1.2 as it is without all the additionnal parameters which have no "must have" gain. I switched back to the good'old -P which make me happier personnaly.

It's time to release 1.2 & start the Implementation of SG's new noise shaping method.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-19 13:04:50
Thanks for testing.
A stronger noise shaping value concentrates the noise more in the 13+ kHz region.
With very low quality settings this may be counterproductive.
So if you have the time I'd welcome if you could give -s 0.3 or -s 0.35 a chance for -q 2. This would be the very first listening test to comfirm or improve on the optimum value for noise shaping strength (at -q 2). So far just a plausible value is chosen as a function of the -q level (q/10), but there has never been a real justification. It would be great if you could do that, especially as it is about the sensitive --portable setting.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-20 11:09:51
I found a new sample that I can ABX at -q 1.5, it is not as easy as Abfahrt Hinwil but it is less tiring than Therion, it produces some very light cracklings at regular interval on a repeted specific instant of the melody. At -q 1.5 it is hard to ABX when you haven't catched it, listen to -q 0 first to have an idea then increase the bitrate. Once trained it's easy/medium to ABX. The sample comes from the lame sample archive, I have shortened it. I will upload my version later.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.4
2009/04/20 08:04:29

File A: C:\02- Fool's Garden (Artefact Only) Lossless.flac
File B: C:\02- Fool's Garden (Artefact Only) V1.1.3j -q 1.5.lossy.flac

08:04:29 : Test started.
08:04:56 : 01/01  50.0%
08:05:18 : 02/02  25.0%
08:05:49 : 03/03  12.5%
08:06:01 : 03/04  31.3%
08:06:15 : 04/05  18.8%
08:06:39 : 05/06  10.9%
08:06:54 : 06/07  6.3%
08:07:16 : 07/08  3.5%
08:07:32 : 08/09  2.0%
08:07:47 : 09/10  1.1%
08:08:03 : 10/11  0.6%
08:08:28 : 11/12  0.3%
08:08:32 : Test finished.

----------
Total: 11/12 (0.3%)


Edit:
The edited sample is available here
Fool's Garden (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=70442&view=findpost&p=628502)

On the short version (artefact only) you can split the 8 movements like this  1-2/1-2/1-2/-1-2, there is always a light crackling on 2. So it's very located.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-21 21:50:48
I tried eig (aka Abfahrt Hinwil) and Fool's Garden trying to find out audible differences for -q 1.5 for the default -s 0.15 in comparison with slightly increased -s values. However I can't hear the problem even with plain -q 1.5. Looks like I'm pretty bad at ABXing, at least right now.

However I found something strange. When encoding with foobar using the usual piping for foobar | lossyWAV | FLAC bitrate a lot higher for eig than when I used lossyWAV and FLAC directly from the command line.
So I compared these two procedures on my regular music test set for --portable, and bitrate came down from 385 kbps (piped procedure) to 381 kbps (commandline procedure).
Looking at the files bitrate decrease was higher for smaller files, which explains the very remarkable difference on problem sample snippets.

Well 381 kbps isn't a lot lower than 385 kbps, but it's not negligible to me. So I'm gonna use a good old bat file for the lossyFLAC encoding from within foobar which is the automatic equivalent of doing things on the command line.

I wonder what extra information is piped which makes it's way into the output .lossy.flac file.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-21 22:29:08
It's FLAC in combination with foobar2000. For an unlimited file length (the way that Foobar pipes), passed through lossyWAV, into FLAC, there is a 64kB padding block added to the resulting FLAC file.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-22 18:11:40
Thank you, Nick.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-27 19:07:26
Now that I'm using a .bat file again for encoding from within foobar I'd like to tag the output as being preprocessed by lossyWAV, something we've done before using --keep-foreign-metadata. But no matter wheather I do it this way or having FLAC tag the file by -T Comment="LossyWAV 1.1.3e --portable" I don't get the tag in the output file. Guess it's foobar's final tagging which deletes the comment tag.  I also tried with a specific preprocessor tag but it's the same.

Any ideas?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-27 19:29:24
In foobar's output window (if you have selected it to popup on completion of processing) Ctrl-A to select all of the files and Alt-Enter to open the properties window - there, add a tag named LOSSYWAV with a value "1.1.3e --portable".

Not elegant, but if you are doing a large transcode quite quick per file.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-27 20:28:47
I've been working on the FFT part of the program and have implemented a Radix-4 FFT algorithm. This improves processing time for the pure Delphi version but not at all for the version with IA-32 / x87 speedups (about a 10% reduction in throughput - I reckon the assembler Radix-2 FFT algorithm is quite tight and the Radix-4 version will take quite some optimisation....).

Has there been any clear decision regarding the --sortspread parameter? My interpretation thus far is that it has met with little enthusiasm. Not a problem, I would just like to know if it should be culled from v1.2.0.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-04-27 20:54:33
My personal opinion on --sortspread/--underlap/--analyses is that all this parameters does increase the quality within the same setting but all have drawback in speed/bitrate that make them useless because if you compare at same bitrate the old scale is always better. If you can increase speed/decrease bitrate while keeping the same quality improvement some of these parameter might become usefull (specially --sortspread). If you cannot, in there actual state, none of the parameters I tested are usefull. I was optimistic at the beginning but now that all have been tested, I was asking myself why you weren't releasing lossywav 1.2 ? (... and also why you were still displaying -a 5 in your signature  )
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-27 21:09:07
Using --analyses 5 does indeed slow down processing rate (significantly) and increase bitrate (marginally) but that is the setting that I used for the last full transcode I did.

I still think that --underlap / --analyses and --sortspread have merit, I just don't know exactly which "level" of each is appropriate.

On speed - if the desired result is a reasonable bitrate / quality combination then processing rate (within reason) doesn't matter - just leave the computer merrily transcoding to its hearts content and wait until it is finished.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-04-27 22:44:23
In foobar's output window ...

Thank you, Nick, for the hint. Really not very elegant, as you say. What a pity. I'll keep doing it in the Explorer by means of a context menu entry for folders.

As for underlap etc. my personal feeling is like what sauvage78 said. --portable is fine as is, and there's not a big chance for a lower -q setting and an added feature out of these to get at a remarkably lower bitrate while maintaining quality. But there's no need to put away with these features either. I know you would like them to prove useful so why not keep them? LossyWAV has a lot of advanced options anyway, but thanks to the main quality options --portable, --standard, etc. everybody who doesn't like them can ignore them.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-04-28 09:07:34
Has there been any clear decision regarding the --sortspread parameter? My interpretation thus far is that it has met with little enthusiasm. Not a problem, I would just like to know if it should be culled from v1.2.0.

My view is that they're all worth keeping, just because they may be useful in some circumstances - Sauvage78 found some improvements on his problem samples. Perhaps they may help on others too (if any are ever found). There's always the option of simply using a higher q level if you want to keep things simple. I wouldn't worry about extra encode times. You only encode once. Sorry I can't be of any help suggesting suitable "levels"
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-04-28 10:50:24
@Nick,

Can you see "inside" the processing to see what was happening here...

http://www.hydrogenaudio.org/forums/index....st&p=628046 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=65499&view=findpost&p=628046)

...i.e. why that caused an easy to ABX difference?

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: PowerPaul86 on 2009-04-29 19:58:34
Hi there,

i wanted to ask if there is still any special wine parameter that you have to add when trying to use lossywav with an higher version than 1.0?

Or did the wine support stopped after that version?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-29 21:01:54
The issues with WINE are undetermined. WINE support up to v1.0.0b was a fortuitous happening, not a conscious decision to support it. I can only hazard a guess that it may have something to do with the piped input / output which was introduced at v1.1.0. Source is available should anyone wish to track down the cause of the recent incompatibility.

[edit] However, thinking about it, it may be possible to remove the piping altogether and revert to simple screen output rather than sending all text output to the console. I will endeavour to produce a version which does this, although it may not be the cause of the issue and I don't have any platform upon which to test WINE compatibility. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-30 08:15:50
lossyWAV beta 1.1.3k attached to post #1 in this thread. This is attempt #1 to regain WINE compatibility.
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-04-30 14:28:17
lossyWAV beta 1.1.3k attached to post #1 in this thread. This is attempt #1 to regain WINE compatibility.



Thanks. It works again. Wine stable 1.01
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-30 14:30:04
Fantastic, thanks very much for the quick confirmation - now, is there any appetite for a "fixed" v1.1.0b to be released prior to the forthcoming (but as yet date not fixed) release of v1.2.0?

[edit] Should have said earlier - piping is still very much integrated, I found a likely culprit problem which didn't require me to remove piped I/O. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-04-30 14:32:32
Fantastic, thanks very much for the quick confirmation - now, is there any appetite for a "fixed" v1.1.0b to be released prior to the forthcoming (but as yet date not fixed) release of v1.2.0?



Yes please.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-30 14:56:28
lossyWAV 1.1.0c attached to post #1 of the lossyWAV 1.1.0 release thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: PowerPaul86 on 2009-04-30 15:58:43
wow thank you very much for that fast response and solution!
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-04-30 17:59:59
No problem - to be honest, it's something that has been annoying me for a while now. I realised today that it had to be to do with the piped output in some way - it turned out to be the way that I was ensuring that text output went to the console. I used an old method whereby a file with the name "con" is opened and all text destined for the screen is written to that "text file". I googled a reference today that informed me that stderr also goes to the console and is not redirected when stdout is. This is available in Delphi as the "ErrOutput" file handle - so I used that and deleted the previous method.

I'm very glad that it proved to be the issue as I did not want to remove piped I/O due to the significant speed gains achievable when using it.
Title: lossyWAV 1.2.0 Development Thread
Post by: PowerPaul86 on 2009-05-02 19:14:30
I still got some problems converting flac files to lossyWAV

Maybe you got a clue what i am doing wrong :/

I can convert to flac files without any problems and i can execute lossywav.exe with wine as you can see in the following screenshot:

[a href="http://www.abload.de/image.php?img=screenshot-paulpp-.winyt5s.png" target="_blank"]


Thanks in advance!
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-05-02 20:34:34
Try changing C:\"Program Files"\bin\lossywav.exe to "C:\Program Files\bin\lossywav.exe" - that may help (similar with the FLAC part of the command line).
Title: lossyWAV 1.2.0 Development Thread
Post by: PowerPaul86 on 2009-05-03 14:15:39
Unfortunately i get the same error messages as before. 

Could i test the lossywav conversion without the foobar player so i can exclude foobar as the source of the I/O error?

Something like that:

[a href="http://www.abload.de/image.php?img=screenshot-paulpp-.winu927.png" target="_blank"]
Title: lossyWAV 1.2.0 Development Thread
Post by: Hancoque on 2009-05-03 14:53:57
Isn't there another syntax in Linux to deal with spaces? If I remember correctly you put a backslash before each space, like Program\ Files.
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2009-05-03 17:34:28
Unfortunately i get the same error messages as before. 


The message should be self explanatory:  lossyWav does not support lossless (or lossy) formats. Just plain PCM ( inside RIFFWave ). 

The syntax you have put in foobar2000 does the lossywav processing of the wave (that foobar2000 has previously decoded from the original flac), and pipes that PCM data to flac for encoding.

But i guess that being a linux user, you may already know that
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-05-03 18:43:31
Inserting
Code: [Select]
FLAC -d <filename> --stdout --silent
(followed by a pipe symbol) before the rest of the command line would allow you to test outside of foobar2000.
Title: lossyWAV 1.2.0 Development Thread
Post by: PowerPaul86 on 2009-05-03 21:37:41
@Hancoque
Yes, i know that you need to escape the space character in a linux command, but using it under wine?
You would end up with something like that:

Quote
/d /c C:\"Program\ Files"\bin\lossywav - --standard --silent --stdout|C:\"Program\ Files"\bin\flac - -b 512 -5 -f -o%d


Which doesn't work at all. Sure you could try different methods to get that path right, but to me moving the exe files to another easier to address path seems to be the simplest/best option.

@[JAZ]

I realized, after looking at the RIFF article on wikipedia, that i needed a WAV file instead of a FLAC file.
But i only wanted to illustrate with that picture what i had in mind - a simple conversion.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-05-05 08:20:05
lossyWAV beta 1.1.4a attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-05-05 18:01:29
Thank you, Nick.
Title: lossyWAV 1.2.0 Development Thread
Post by: jido on 2009-05-06 13:20:48
Has there been any clear decision regarding the --sortspread parameter? My interpretation thus far is that it has met with little enthusiasm. Not a problem, I would just like to know if it should be culled from v1.2.0.

My view is that they're all worth keeping, just because they may be useful in some circumstances - Sauvage78 found some improvements on his problem samples. Perhaps they may help on others too (if any are ever found). There's always the option of simply using a higher q level if you want to keep things simple. I wouldn't worry about extra encode times. You only encode once. Sorry I can't be of any help suggesting suitable "levels"

Are these settings useful to achieve lower bitrates with acceptable quality?

Since the nature of lossyWAV is that it improves compression, it would make sense to spend more encoding time on lower quality files than on higher quality files. Since they achieve the lowest bitrate. The settings could even be part of the low-quality presets
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-05-06 14:57:42
I'm not the best person to answer this but, yes, the --sortspread parameter does improve quality a bit (note that now it's been implemented it is a simple on/off switch not a 0-7 range anymore). However, using it results in an increase in the bitrate compared with not using it (at the same -q level). The debate over its usefulness is that simply using a higher q level (like -q 2.8 instead of -q 2.5) achieves the same thing : IE better quality at a higher bitrate. Having said that you may find that using say, -2.5 --sortspread gives a lower bitrate and the same perceptual quality as -q 2.8 with the type of music you listen to. I think it's basically a 'try it and see if it works for you' situation
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-05-14 08:23:17
lossyWAV beta 1.1.4b attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-05-30 11:46:05
Why not use FFTW (http://www.fftw.org/) for FFT analysis?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-05-30 13:12:23
If I could find a pre-compiled Delphi compatible library and an interface then I would. However, I have enjoyed optimising the FFT routines in Delphi / IA-32/x87.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-01 13:00:52
If I could find a pre-compiled Delphi compatible library and an interface then I would. However, I have enjoyed optimising the FFT routines in Delphi / IA-32/x87.

Might this (http://www.fftw.org/install/fftw_usage_from_delphi.txt) be useful?

I found that here (http://www.fftw.org/install/windows.html) (scroll down to "FFTW 3.x and Borland Delphi", there is also a link to a Pascal interface file there for calling FFTW from Borland Delphi, and also a link to the compiled library), which was the first result of this Google search (http://www.google.com/search?hl=en&q=fftw+delphi&aq=0&oq=fftw+delp&aqi=g1).

I hope that helps.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-01 20:53:37
Thanks for that - I have to admit that I didn't look too hard for it. I have downloaded the DLL and an interface unit and have begun the integration process. It is currently working, however not very fast, so I think that I have some learning to do regarding using FFTW.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-01 21:21:32
Thanks for that - I have to admit that I didn't look too hard for it. I have downloaded the DLL and an interface unit and have begun the integration process. It is currently working, however not very fast, so I think that I have some learning to do regarding using FFTW.

No problem, I'm really glad I managed to help! I think lossyWAV is awesome and I'm happy to help its development in any way I can, even if it's just by providing a link.

And I believe using FFTW should help increase lossyWAV's sound quality, because it's one of the most precise FFT libraries, if not the most precise (http://www.fftw.org/accuracy/Pentium4-3.60GHz-icc/) (in those graphs, lower is better because they measured the calculation error. Other accuracy benchmark results can be found here (http://www.fftw.org/accuracy/)).
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-02 21:25:32
Right - thanks to the gentle nudge from doccolinni - I have implemented a -F or --fftw parameter in lossyWAV. This enables the exclusive use of libfftw3-3.dll for FFT calculations. The slow running I was experiencing initially was as a result of my testing (initially) on my 55 problem sample set (12m28s in total). Processing short files vastly overemphasises the effect of the time used in the fftw planning phase compared to faster FFT calculations. Using a single album (39m41s duration) as a test piece and --zero as the only other parameter, the assembler version processes in 27.7 seconds and in 24.7 seconds with --fftw enabled, a saving of approximately 11%. At --standard the difference is more pronounced: 45.0 seconds vs 36.3 seconds or a 19% saving.

This is certainly worthwhile (and the integration is complete) so I will be releasing beta 1.1.4c as soon as I can determine exactly what text to add where to credit fftw.org and the creators of the Delphi fftw interface unit. The libfftw3-3.dll file would also require to be distributed (or downloaded) to allow the --fftw parameter to be used.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-02 21:26:18
Oh, and the output of my assembler version and the fftw output are identical....

[edit] Problem with short files fixed (at least on my Intel C2D) by changing the flags used when setting up the plan for each FFT length. 55 problem sample set now processes faster using --fftw than default. I am well pleased and will endeavour to release ASAP. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-03 01:20:06
Oh, and the output of my assembler version and the fftw output are identical....

Well that's interesting, I thought it might have different results... But anyway,...

[edit] Problem with short files fixed (at least on my Intel C2D) by changing the flags used when setting up the plan for each FFT length. 55 problem sample set now processes faster using --fftw than default. I am well pleased and will endeavour to release ASAP. [/edit]

...yay!  I'm glad that the initial slow running was not a serious problem and that it works faster now! Can't wait till you release the next version.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-03 19:48:16
Well, since I'm here I might as well ask a question and make another proposition I've had in mind:

Question(s): Do FLAC's quality commands, like -8 for example, set a specific value for the block size? If they do, will -b 512 override them?

Proposition: This one is regarding the dithering. As the Wikipedia article plainly puts it:
Quote
Dither is an intentionally applied form of noise, used to randomize quantization error, thereby preventing large-scale patterns such as "banding" (stepwise rendering of smooth gradations in brightness or hue) in images, or noise at discrete frequencies in an audio recording, that are more objectionable than uncorrelated noise.

Of course, I'm not presuming that you don't know what "dither" means, I was just making sure that you're aware that I know what it means and that what I'm about to propose is actually based on understanding the concept of dithering. Since the noise added by lossyWAV is already transparent at higher quality settings, perhaps dithering should take place only with --zero, --nasty and --awful to reduce the "unpleasantness" of the added noise? It's probably best to just forget about dithering for the moment and implement it in future development, or maybe implement a testing command --dither or something like that.

Just to talk a bit more about dithering itself, to make sure that I've got the right idea in my mind:

So let's say that we've got an audio file with bit-depth of, for the sake of simplicity, 4 (yes, that's ridiculous, but I just want to keep it simple, to check whether my idea of dithering is correct). So the possible values for audio peaks in that file range from 0000 to 1111. Now let's say that one peak has a value of 1001. And then BAM, lossyWAV comes in and decides that two bits are enough for the chunk wherein lies that very audio peak (that's even more ridiculous, but bear with me). So now the only possible values in that chunk are 0000, 0100, 1000 and 1100. Without dithering, the value of our peak would simply be rounded to 1000. But with dithering, lossyWAV would first calculate how far the value of that peak is from the two of these new possible values in-between which it lies, which are 1000 and 1100. The (absolute) difference between 1001 and 1000 is 1, and between 1001 and 1100 it's 3 (11 in binary). And now, lossyWAV would randomly assign one of these two values to the peak in question, so that the possibility of it being 1000 or 1100 depends on the ratio of these two (absolute) differences. So since the initial value of the peak is three times closer to 1000 than it is to 1100, the possibility of the new value being 1000 has to be three times greater than the possibility of it being 1100. So it would be 75% chance that the new value will be 1000 and 25% chance that the new value will be 1100.

Is that the correct understanding of dithering?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-03 21:12:34
Dither has previously been included but was removed after consultation with David.

Noise shaping is included using SebastianG's 44.1kHz and 48kHz coefficients - increasing the proportion of noise shaping may solve what you're hearing. The default shaping is quality/10, i.e. 25% at --portable, 50% at --standard, etc. The proportion shifts the noise shaping from pure white noise (--shaping 0) to pure noise shaped noise (--shaping 1). The downside of noise shaping is that its output is difficult for FLAC to predict, so the resultant FLAC bitrate increases quite dramatically.

One final point, FLAC has no quality commands - it has compression settings. FLAC is lossless after all....
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-03 22:18:30
One final point, FLAC has no quality commands - it has compression settings. FLAC is lossless after all....

Oh, yes, that's, of course, what I actually meant.  So will -b 512 override any block size settings that might be defined by them?

I hope my understanding of dithering is correct (can you verify that it is, I've explained my idea in my previous post?), but I'm not really familiar with the process of noise shaping... I'll read through the Wikipedia article on it though and try to figure it out.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-03 22:53:10
Yes, the -b 512 parameter will set the FLAC block size to 512 samples, over-riding any internal presets.

The bits that are being removed have a 50% chance of rounding up, 50% down - exactly the same as dither in the range -0.5 to +0.5.

For 4 bits (signed) we would have a range 1000 (-8) to 0111 (+7). lossyWAV does its rounding in floating point so there is no chance of rollover from +7 to -8. +7 would indeed round to +8 in the floating point domain if two bits were to be removed but lossyWAV has a check in place which limits the maximum rounded value to (2^(bits_per_sample-1) shr (bits_to_remove)) shl (bits_to_remove), i.e. 4 bits per sample, 2 bits to remove would yield a maximum value of 0100. I agree 4 bits per sample with 2 bits to remove is an extreme example. lossyWAV has a defined static_minimum_bits_to_keep which is equal to 5 and calculates a dynamic minimum_bits_to_keep which relates to the RMS value of the input samples (for each FFT analysis).
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-03 23:00:05
The bits that are being removed have a 50% chance of rounding up, 50% down - exactly the same as dither in the range -0.5 to +0.5.

Oh. I thought it depends on how far from each end the particular peak is. :-/

Thanks for the explanation(s).

So how is the development of the next beta coming along?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-03 23:06:37
I would expect to sort out the required text for the next beta by this weekend.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-03 23:26:21
I would expect to sort out the required text for the next beta by this weekend.

Cool.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-05 08:21:42
lossyWAV beta 1.1.4c attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-06-05 14:39:50
How about a link somewhere to http://www.fftw.org/download.html (http://www.fftw.org/download.html) to make the lib easy to find (and avoid a lot of asking)?
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-05 16:05:31
How about a link somewhere to http://www.fftw.org/download.html (http://www.fftw.org/download.html) to make the lib easy to find (and avoid a lot of asking)?

Yeah, actually, I can't find where to download libfftw3-3.dll either...

I've found it here (http://www.forex-tsd.com/indicators-metatrader-4/15911-fast-fourier-transform-cycle-extraction-3.html), though, but I don't feel like registering on a forum just to download a dll library... :-/
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-06-05 17:23:12
actually, I can't find where to download libfftw3-3.dll either...

Use the link I gave, on this page you find links for Windows (http://www.fftw.org/install/windows.html) (there's 64bits too), OS-X (http://www.fftw.org/install/mac.html), Linux (http://www.fftw.org/install/linux.html) and so on.

The current version is FFTW 3.2.1 in the zip file you find libfftw3-3.dll

BTW. At the beginning of this thread Nick says "Use of FFTW requires the presence of "libfftw3-3.dll" on the host", I tell you it needs that lib always with version 1.1.4c whether used or not.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-05 17:51:03
Use the link I gave, on this page you find links for Windows (http://www.fftw.org/install/windows.html) (there's 64bits too), OS-X (http://www.fftw.org/install/mac.html), Linux (http://www.fftw.org/install/linux.html) and so on.

The current version is FFTW 3.2.1 in the zip file you find libfftw3-3.dll

Oh. Thanks.
But I'm on Ubuntu and I've got libfftw3-3 installed, but I doubt Wine will care about that.  So I still need the dll.

BTW. At the beginning of this thread Nick says "Use of FFTW requires the presence of "libfftw3-3.dll" on the host", I tell you it needs that lib always with version 1.1.4c whether used or not.

Probably just a bug.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-05 17:55:15
Meh - I think that I'll have to compile a separate version without FFTW support.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-05 17:58:18
Meh - I think that I'll have to compile a separate version without FFTW support.

Is there a reason why you can't redistribute the libfftw3-3.dll?
Title: lossyWAV 1.2.0 Development Thread
Post by: lvqcl on 2009-06-05 18:11:51
Quote
I tell you it needs that lib always with version 1.1.4c whether used or not.

I created DLL stub (only 1kb in 7z archive)  But yes, static linking isn't reasonable here.

Quote
I think that I'll have to compile a separate version without FFTW support.

Maybe it's better to use LoadLibrary() when user chooses to use fftw library?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-06 10:49:59
I'm unwilling to add circa 500kiB to the download as it will eat into HA's bandwidth, so I do not intend to add the DLL to the distribution.

I am searching for error trapping principles to allow the program to continue in the event that the DLL is missing.
Title: lossyWAV 1.2.0 Development Thread
Post by: lvqcl on 2009-06-06 11:45:07
Don't know how to make this in Pascal, but C code looks like this:
Code: [Select]
//init
if(fftw_switch)
{
    hinstLib = LoadLibrary("libfftw3-3.dll");
    if(hinstLib == NULL)
    {
        print_error("...");
        exit(-1);
    }
    proc1 = GetProcAddress(hinstLib, "_fftw_execute");
    if(proc1 == NULL)
    {
        print_error("...");
        exit(-1);
    }

    //here we can call proc1
}


//cleanup
if(fftw_switch)
{
    FreeLibrary(hinstLib);
}
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-07 18:47:04
I'm unwilling to add circa 500kiB to the download as it will eat into HA's bandwidth, so I do not intend to add the DLL to the distribution.

I am searching for error trapping principles to allow the program to continue in the event that the DLL is missing.

Well I could host it for you if you don't find a way to make it run without the DLL being present.

Although, granted, it would be better to fix it so it can run even if there is no libfftw3-3.dll.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-07 20:23:33
Thanks for the offer of hosting. I think that I have cracked the running without 'libfftw3-3.dll' problem (thanks for the pointer lvqcl!!). Still testing - will try to post beta 1.1.4d and source later.

[edit] Should lossyWAV terminate with an error if --fftw is included in the command line and 'libfftw3-3.dll' cannot be found? The alternative would be to continue using the existing FFT routines. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-07 21:38:03
Thanks for the offer of hosting. I think that I have cracked the running without 'libfftw3-3.dll' problem (thanks for the pointer lvqcl!!). Still testing - will try to post beta 1.1.4d and source later.

No problem.  I'm glad you managed to fix it.

[edit] Should lossyWAV terminate with an error if --fftw is included in the command line and 'libfftw3-3.dll' cannot be found? The alternative would be to continue using the existing FFT routines. [/edit]

If I understand correctly, the only difference is the speed and not the quality, right? In that case, I think it would be the best if it just printed out a warning that it can't use FFTW because the libfftw3-3.dll is not present and continue "encoding" using the existing FFT routines.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-07 21:51:58
If I understand correctly, the only difference is the speed and not the quality, right? In that case, I think it would be the best if it just printed out a warning that it can't use FFTW because the libfftw3-3.dll is not present and continue "encoding" using the existing FFT routines.
We're certainly thinking on the same wavelength....

lossyWAV beta 1.1.4d attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-07 22:23:01
If I understand correctly, the only difference is the speed and not the quality, right? In that case, I think it would be the best if it just printed out a warning that it can't use FFTW because the libfftw3-3.dll is not present and continue "encoding" using the existing FFT routines.
We're certainly thinking on the same wavelength....

lossyWAV beta 1.1.4d attached to post #1 in this thread.

Just for your information, I've tested it on No You Girls from Franz Ferdinand's latest album, here are the results:

Size of the original wav: 32.3 MB

Size of the flac compressed using -8 compression setting without lossyWAV preprocessing: 27.1 MB

After preprocessing with lossyWAV using -S setting (and with --fftw, but that's not important here) and then compressing with flac using settings -8 -b 512 --keep-foreign-metadata: 12.0 MB

 

I haven't measured the time it takes with and without --fftw, but I don't really care because these results are purely awesome! And damn you're a quick developer. (No sarcasm intended.)
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-07 22:38:09
Okay, I tested the difference between running it with and without --fftw now (on the same song as above). Without --fftw it takes 6.31 seconds, with --fftw it takes 5.21 seconds - approx. 17.43% faster.

The resulting files are the same, of course.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-07 23:38:35
Oh, it doesn't support wild-cards in file-names?

I can only run it through Wine so it may also be a Wine issue, but it gives me an error when I put *.wav for input...
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-08 06:23:14
lossyWAV doesn't support wild cards in the input filename. If foobar2000 works under WINE then I would suggest that as a very good option for processing.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-08 14:36:51
lossyWAV doesn't support wild cards in the input filename.

Perhaps it would be a good feature to implement sometime in the future?

I prefer doing things directly from command-line, although I would be willing to use Foobar2000 until such a feature was implemented (if you ever decide implement it, that is), but it seems the latest versions of it don't work quite well under latest versions of Wine (http://appdb.winehq.org/objectManager.php?sClass=application&iId=1749).

So for the moment I'll just settle with encoding file-by-file. 

Luckily, command-line interface of Linux is rather advanced so I think there might be a trick to make the command-line itself call "wine lossyWAV.exe *.wav -S --fftw", replacing '*' with any filename which ends with ".wav". I'll look into that, I might be wrong, though.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-09 06:16:08
I don't know if batch files are of any use to you using WINE, but there was a drag-n-drop batch file somewhere in one of these threads which also copied the tags from the source file.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-09 08:59:10
I don't know if batch files are of any use to you using WINE, but there was a drag-n-drop batch file somewhere in one of these threads which also copied the tags from the source file.

Hm. Thanks, maybe I'll dig through sometime, but right now I have no problem with doing it file-by-file. 

What can I say - laziness shows itself in weird ways... :-/
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-06-09 19:47:42
Having finished encoding for my DAP (using 1.1.3e)  I didn't do any encodings recently.
But now with these news on lossyWAV I wanted to compare 1.1.4d -P --fftw against 1.1.3.e -P.
I noticed bitrate was always a bit lower (up to 4 kbps with my test set) when using 1.1.4d --fftw.

Can you tell about what has changed, Nick?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-09 20:18:38
At beta 1.1.4a I separated the spreading function average calculations so that one is calculated for the results of the old spreading method and one is calculated for the results of the new spreading method. This does not affect the minima at all - the lowest value is still the lowest but the average is now taken as the lower of old and new rather than the average of the minimum overall.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-06-09 21:52:17
Thank you, Nick.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-12 00:31:34
Er, so, what features are you trying to implement in the next beta?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-12 06:16:40
I'm looking again at window functions - specifically, implementing the Blackman window which takes a parameter alpha in the range 0..1. Zero is equivalent to the existing Hanning window function used, 0.57 is approximately equal to the Flat-Top window. I have been re-calculating noise threshold values for values in the permissible range (although I am thinking of limiting the user selectable alpha range to 0.0 to 0.6) and have spotted a relationship between some constants specific to each alpha value.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-06-12 07:46:55
I'm looking again at window functions - specifically, implementing the Blackman window which takes a parameter alpha in the range 0..1. Zero is equivalent to the existing Hanning window function used, 0.57 is approximately equal to the Flat-Top window. I have been re-calculating noise threshold values for values in the permissible range (although I am thinking of limiting the user selectable alpha range to 0.0 to 0.6) and have spotted a relationship between some constants specific to each alpha value.

Ah, cool. So do you aim for high-resolution (http://en.wikipedia.org/wiki/Window_function#High-_and_moderate-resolution_windows) or for high-dynamic-range (http://en.wikipedia.org/wiki/Window_function#Low-resolution_.28high-dynamic-range.29_windows)? The article claims that this graph might also be useful (http://en.wikipedia.org/wiki/Window_function#Comparison_of_windows).
Title: lossyWAV 1.2.0 Development Thread
Post by: [JAZ] on 2009-06-29 20:49:04
Nick.

This weekend i've been working on jLossyWav in order to upgrade it to 1.1.4.

I've implemented buffered I/O (speed has increased considerably due to this), and copied partially the console interface (most of the switches, help, etc..)

On the algorithm itself, i haven't got that further though. There are a couple of things i'd like to get your opinion on.

First, about the Hanning window:
I've implemented it so that it is equal on both sides, while your implementation is off by one (starts at zero, and then it's equal). I remember i had a problem and i modified the window to be like it is now, but I can't remember what the problem exactly was.

Next, about the procedures : Post_Process_FFT_Results , Sort_FFT_Results. I see they are for the new sortspread.
About the first, it seems it gets the Real value out of the Complex one and stores it, skewed, in fft_result, similarly of what the begining of Spreading_Complex does. is that it?
About the second, is that a bubble sort algorithm, so that fft_result is ordered from smallest to biggest? Meaning, the goal being to get the smallest ones?

Spread_Complex also changed (I remember you tried different versions, so this is to be expected) If i get it right, it gets the minimum of the x-1 to x+1 vs the minimum of the x to x+somewidth, plus the acummulators of the averages of these values plus.. well.. it's quite complicated to explain :S

I still have some work to readapt the things that have changed, as well as to add the missing things. I hope for a release during july.



Btw.. have you considered refactoring the source a bit? That lossywav.dpr file is getting too complex. Especially the massive amount of variables. (naming conventions, constants placed as vars, local variables put as global variables...). This would greatly help other developers.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-06-29 21:45:09
[JAZ],

Glad that you're still interested in the project - it's straining my brain a bit at the moment (I'm trying to clearly understand how the reference_threshold values are made up when the bit-removal noise is calculated in an unpublished routine).

Hanning window: I changed it to the zero start to allow index shift access (store all arrays in one) - however I realise that that is not such a good idea and have reverted (in the unreleased 1.1.4e) to fully symmetrical values;

Post_Process_FFT_Results: Yes - you got it in one;

Sort_FFT_Results: Again correct.

Spread_Complex combines the old_spread and new_spread algorithms and calculates min_old; min_new; average_old and average_new concurrently. Then the lower minimum has the noise_threshold_shift applied and the lower average has the snr_value applied. Then the lower of the modified minimum and modified average is used as the single result. Each discrete old_value averages the range result
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-07-01 21:56:31
In looking at the reference threshold values again I have been playing (yet again) with dither.

Previously the order was:

take each sample, divide by 2^bits-to-remove, add optional dither, round, multiply by 2^bits-to-remove.

This increased the bitrate quite dramatically.

However, each bit that is removed from the sample is random, so why not use:

take each sample, add triangular dither, divide by 2^bits-to-remove, round, multiply by 2^bits-to-remove

instead?

This does not change the reference threshold values beyond 4 bits-to-remove (higher than undithered for 1, 2 & 3).

Thoughts?
Title: lossyWAV 1.2.0 Development Thread
Post by: jido on 2009-07-02 11:51:52
Quote
take each sample, divide by 2^bits-to-remove, add optional dither, round, multiply by 2^bits-to-remove.

Doesn't that effectively multiplies your dither by 2^bits-to-remove? Seems wrong.

As for your question, what is the distribution of a bit removed from the sample before calling it "random"?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-07-02 13:08:01
Doesn't that effectively multiplies your dither by 2^bits-to-remove? Seems wrong.

As for your question, what is the distribution of a bit removed from the sample before calling it "random"?
Music will *probably* follow a fairly random distribution, at least in the lowest few bits.

In code, no dither:

processed_sample:=Round(raw_sample/powersoftwo(bits_to_remove))*powersoftwo(bits_to_remove)

Old dithering:

processed_sample:=Round(raw_sample/powersoftwo(bits_to_remove)+dither)*powersoftwo(bits_to_remove)

Proposed dithering:

processed_sample:=Round((raw_sample+dither)/powersoftwo(bits_to_remove))*powersoftwo(bits_to_remove)

Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-07-02 13:36:54
To me this sounds healthier than old dithering.
As from my understanding dithering was amplified with the old scheme according to bits-to-remove though still remained at supposed noise level.
Anyway I thought dithering was expected not to be useful. Is bitrate lower with this kind of dithering than when not doing dithering at all?
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-07-02 14:36:12
Anyway I thought dithering was expected not to be useful.

Wasn't it even working against the idea of lossyWav?
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-07-02 14:37:16
Old dithering:

processed_sample:=Round(raw_sample/powersoftwo(bits_to_remove)+dither)*powersoftwo(bits_to_remove)

Proposed dithering:

processed_sample:=Round((raw_sample+dither)/powersoftwo(bits_to_remove))*powersoftwo(bits_to_remove)

...but if the "dither" is at the LSB level in both cases, then in the proposed case, the dither isn't doing much once you're removing more than a couple of bits.

What is point, "dither"?

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-07-02 15:57:44
Anyway I thought dithering was expected not to be useful.

Wasn't it even working against the idea of lossyWav?

Exactly. The point of lossyWAV is to remove trailing n bits which are estimated to be just noise. The point of dithering is to transfer the noise from the removed bits to the last remaining bit (in order to mask the artefacts produced by removing the said bits). So basically, dithering makes the last bit which was estimated not to be noise noisy.
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-07-13 09:44:58
Just got into some more lossy codec testing recently. I was interested to see how more tonal music fares and there seem to be some issues with quality/bitrate. Note I only tested latest 'stable' release.

1 - On the attached sample: http://www.hydrogenaudio.org/forums/index....showtopic=73344 (http://www.hydrogenaudio.org/forums/index.php?showtopic=73344)

Heavy noise @ Q0 (360 k) , still noisy @ Q1 (402 k ). In contrast wavpack lossy v4.5 ABR 230k performs better - nearly transparent.


2 - On an album i have 'black tape for a blue girl'. The lossless bitrate is 647 k. Lossywav -S managed to produce bigger files than the original (2 tracks) while lossywav -P didn't achive any saving for those 2 tracks. The rest where smaller though.


I cannot draw a solid conclusion but a few thoughts: VBR rate may be too unstable for more silent music (very high bitrate and if quality lowered - noise and still high bitrate). -P has to produce saving in all tracks (at least not make the bigger) while being fully or very close to transparent. I believe -P is doing okay in that regards.

Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-07-15 20:48:54
Thank you for your sample, shadowking.
I'll listen to it when I'll be back from holidays.
We have known that -q 0 doesn't provide good quality. Real good quality starts in the -q 1.5 ... 2.0 zone.
It has also been known that simple tonal tracks are handled inefficiently compared to lossless codecs because lossless codecs work very efficiently in this case when using a large blocksize, but lossyWAV blocksize is only 512 samples (using a multiple of 512 samples helps only on rare occasion as it makes the lossyWAV procedure less efficient). Moreover in case FLAC is used FLAC often works inefficiently for this kind of music, With simple tonal music TAK/wavPack lossless/Monkey are often as efficient or even more efficient than lossyFLAC -P.
Your sample seems to combine these worst case scenarios for lossyWAV (bitrate is unusually high). As such it looks like it's a good example for testing potential improvements of lossyWav.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-07-16 10:21:33
I'm sure we've been round this loop at some point before.

Solutions are

1. Simple: don't use the lossy version when the lossless is smaller
2. Complex: try different blocksizes
3. Very complex: use dynamic blocksizes

3 is a complete redesign of lossyWAV and FLAC, so isn't going to happen.
2 and 1 require file functionality that's outside of lossyWAV itself - maybe in the .bat file, front-end, whatever. lossyWAV as stands can't do it because it never sees the final FLAC file.


I don't think there are significant improvements needed to the "model" for tonal signals - it already knows that it shouldn't touch them - hence the high bitrate. Wavpack lossy does better because it can use appropriate block lengths.

Cheers,
David.


Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-07-22 22:06:08
lossyWAV beta 1.1.4e attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-07-23 10:15:57
Thank you very much for the new version, Nick.
Can you tell a bit about the new parameters?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-07-23 12:59:44
The new --maxsnr parameter works in the same way as the old -nts parameter did, except it is a threshold below the maximum spread FFT result for a particular analysis rather than above or below the minimum spread FFT result. Another name for it could be upper_noise_threshold_shift. SNR is still in there as a threshold below the average spread FFT result.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-07-24 20:33:45
Bugfix release. lossyWAV beta 1.1.4f attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: carpman on 2009-07-25 00:22:29
There's been so many releases recently, and to be honest, I don't understand much of what is being done in LossyWAV these days.
I use LossyWAV all the time, but I'm currently using version 1.0.1.0. Is there a reason to upgrade or are all the updates experimental fine tuning?

I'm confused.

C.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-07-25 09:14:26
... Is there a reason to upgrade or are all the updates experimental fine tuning? ...

Not really I think.
The simple fact is that lossyWAV according to 2Bdecided's principle worked from the very start. It is Nick.C's merit that he added some internal procedures of precaution that enabled lossyWAV to work excellently with the 'portable' quality level at ~380 kbps. Since the establishment of the 'portable' quality there had been variants of certain technical aspects none of which really led to an improvement. Probably with just variants of the current principles no improvement can be expected. This is my personal opinion, but if I understand some of 2Bdecided's remarks correctly this is the way he thinks too. There is also no need for improvement IMO, the results are great (maybe with the exception of the encoding of simple tonal music where lossyWAV + a lossless codec is pretty inefficient compared to pure lossless which is very efficient here - but again, this is immanent to the lossyWAV principles and when using 'portable' the situation is usually acceptabe even in this case).
Title: lossyWAV 1.2.0 Development Thread
Post by: carpman on 2009-07-25 10:41:51
Thanks halb27, that's very clear.
I agree with what you said: "There is also no need for improvement IMO, the results are great".

C.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-04 21:23:02
Just got into some more lossy codec testing recently. I was interested to see how more tonal music fares and there seem to be some issues with quality/bitrate. Note I only tested latest 'stable' release.

1 - On the attached sample: http://www.hydrogenaudio.org/forums/index....showtopic=73344 (http://www.hydrogenaudio.org/forums/index.php?showtopic=73344)

Heavy noise @ Q0 (360 k) , still noisy @ Q1 (402 k ). In contrast wavpack lossy v4.5 ABR 230k performs better - nearly transparent. ...
-P has to produce saving in all tracks (at least not make the bigger) while being fully or very close to transparent. I believe -P is doing okay in that regards.

Finally I managed to listen to your sample. Not hard to ABX at -q 0 (I used v 1.1.0c as I thought this is what you call the latest stable release). But I couldn't ABX -q 1.
My -q 0 bitrate is 332 kbps when using FLAC -b 512 -8. I guess you used piping which produces some overhead significant with short track snippet. Ignoring this difference bitrate is unusually high anyway.

wavPack lossy is an alternative to lossyWAV of course, if player support isn't a restriction. Especially in the 300 kbps area and below it is expected to be the more appropriate choice IMO, while in the 380 kbps range I'd prefer lossyWAV qualitywise though wavPack lossy is great too. I admire David Bryant's work, but unfortunately there is no DAP support except for those players that can make use of Rockbox.
Title: lossyWAV 1.2.0 Development Thread
Post by: gmwiz05 on 2009-08-19 05:08:27
Dumb question, but what type of noise does lossywav use for reducing the file size. Could another variant of lossywav use the different types of sound "color variants". Would an implementation for specific genre types be created, producing better compression for different frequencies? Could it allow lower bitrates?
Just seems interesting...
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-19 07:54:09
The added noise is "created" by the bit-reduction of codec blocks. At its simplest it is white noise but, with noise shaping linked to the quality scale, is more noise shaped as quality setting increases.

Since lossyWAV carries out all of its modification to the audio on raw samples (simply rounding the sample to a number of bits) there is no possibility of treating different frequencies differently.

Some time ago the --awful (-A) and --nasty (-N) presets (equivalent to --quality -4 and -2 respectively) were introduced to allow user to *very* easily create artifacts in the audio. This was to demonstrate the reasoning behind the present --quality 0 (--zero or -Z) settings. Forcing the bitrate down really hurts the quality and introduces glaring artifacts at --awful, less to at --nasty and less still at --zero.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-20 20:12:27
lossyWAV beta 1.1.4g attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: Trondis on 2009-08-21 18:41:03
I sniffed at LossyWAV yesterday, and it seems like a good idea. But I have a question: suppose my need is to have the music files compressed for a portable device. I want the files to be as small as possible without quality loss. LossyWAV consider the setting --standard to be transparent. LAME MP3 files are considered to be transparent at the setting -V2. And the LAME files are much smaller than a LossyWav file at these settings. If "transparent" means the same here, what advantages would LossyWAV give over LAME?
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-08-21 20:21:08
People with higher knowledge than you, told you Lame -V2 is transparent. You were naive enough to trust them. The truth is no lossy codec is transparent on all samples. Lossywav is no exception, but due to the technique used, the probability that a given sample is not transparent can be (much) lower with lossywav than with Lame (depending on the input).

The above is only true if you give lossywav enough bitrate, if you want to go below ~400Kbps then stay with classic lossy codec.

Advantage:
1: Splitable & Joinable (usable as CDImage+cue)
2: Not affected by the usual DCT killer samples
3: The above means Lossywav is much better than ANY other lossy codec (Including musepack) on electronic & live music.
4: Perfect with Sansa Fuse DAP
5: Not tied to a norm. (MPEG)

Flaw:
1: Twice the bitrate of other lossy codecs for the same quality on 95% samples. (non-electronic & non-live music)
2: Not tied to a norm. (MPEG) (for me this is an advantage, but some people like the safety of big norms)

Code: [Select]
I want the files to be as small as possible without quality loss.

for this nero ACC 256Kbps VBR is (much) better than lame, but you need a DAP that support AAC.

If you don't already have a DAP then the best choice is to get a Sansa Fuse & use lossywav -portable, IMHO. This is where lossywav really shines.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-21 22:57:24
lossyWAV beta 1.1.4g attached to post #1 in this thread.

Thank you, Nick.
I encoded several tracks and carefully listened to them. Everything was fine.

Biitrate is a little bit lower than v1.1.3e's which I used so far for productive purposes (379 kbps vs. 381 kbps for portable with my standard test set, 338 kbps vs. 341 kbps for -q 1.5).
Parameter -p is targeting at something new which is quite interesting IMO for the lower quality settings.
Using -p bitrate goes up 1 kbps for portable and -q 1.5. For -q 0 bitrate increase is 3 kbps.
Looking at this I can imagine we can afford having -p take care of an even better S/N ratio.
I'll try to ABX -p vs. non -p during the next days.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-22 18:42:32
Hallo Nick,

If  you can find the time, do you mind allowing the --limit parameter to go below 16000 Hz?

I found a 128 kbps mp3 track in my collection originating from web radio with a drop out in it. I converted it to wave, edited the wave file, and wanted to convert the result to lossyFLAC -P. Bitrate was an adequate 580 kbps which is certainly due to the fact that the mp3 track was lowpassed at roughly 15 kHz. So I'd like to use --limit 15000 (or maybe a bit lower) in order to get at an adequate bitrate.

Apart from this it would be interesting IMO to see by how much bitrate will drop when using --limit 15000 or similar, and of course what impact this will have on quality. Maybe interesting for the very low quality settings and quite in line with the effect of noise shaping (allowing for a higher S/N ration for the very high frequency range) which is used to a minor extend with the very low quality settings.
Title: lossyWAV 1.2.0 Development Thread
Post by: zorzescu on 2009-08-22 19:17:05
Hi everybody 
I also record from the radio (in fact FM broadcast) and I found, that I get best results (i.e. smallest file size of the resulting FLAC file for a given quality level) when I cut frequencies at 16300 Hz. To apply lowpass filter on  WAV I use the free Stereotool (mentioned already here on Hydrogenaudio). In such a case I get usually lossyFLAC bitrate between 430 and 470 kbps (lossyWAV 1.1.4g -p -q 2).

I use lossyWAV/FLAC for several months now for archiving purposes of FM brodcast recordings.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-22 20:11:04
If  you can find the time, do you mind allowing the --limit parameter to go below 16000 Hz?
Wishes, commands, that sort of stuff....  lossyWAV beta 1.1.4h attached to post #1 in this thread.

.... To apply lowpass filter on  WAV I use the free Stereotool....
I don't know that you really need to lowpass before processing with lossyWAV.
Title: lossyWAV 1.2.0 Development Thread
Post by: zorzescu on 2009-08-22 21:25:18
I don't know that you really need to lowpass before processing with lossyWAV.


I did some test:
Code: [Select]
                       lossyWAV/FLAC  bitrate       file size             FLAC           size
------------------------------------------------------------------------------------------------
without lowpass                    578 kbps         26.6 MB            1081 kbps      49.8 MB
after lowpass at 16300 Hz          494 kbps         22.7 MB             967 kbps      44.6 MB
 
After lowpassing WAV file, the resulting bitrate lowers. Cutting frequencies above 16 kHz does not cut sound data in FM radio broadcast. So I decided I need lowpass before further WAV file manipulations 

Edit:
lossyWAV q=3, FLAC -5
Title: lossyWAV 1.2.0 Development Thread
Post by: zorzescu on 2009-08-22 21:49:19
I compressed the same sample as above (6 mins WAV, FM broadcast) with lossyWAV 1.1.4h and --limit 15000
Here are the results:

Code: [Select]
                       lossyWAV/FLAC  bitrate       file size
--------------------------------------------------------------------------------------
without lowpass                    441 kbps         20.3 MB
after lowpass at 16300 Hz          390 kbps         17.9 MB
 

Quick look shows that I can hear no difference  In fact I am not very young 
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-23 00:22:01
... lossyWAV beta 1.1.4h attached to post #1 in this thread. ....

Incredible, you must be a wizard! Thank you very much.

Bitrate for my mp3 track at portable quality comes down from an exact 594 kbps (580 kbps was from memory so not exact) to 485 kbps when using --limit 14500. So it helps a lot.

I also tried zorzescu's trick of lowpassing before doing lossywav but this didn't improve things more as was expected because the mp3 track is lowpassed already.

I was curious about bitrate reduction and tried --limit 14500 on my usual test set. Average bitrate for --portable comes down to 361 kbps.
As -q 2.0 can be considered so far to yield extremely good quality too I also tried -q 2.0 -p --limit 14500. Result is 348 kbps and quality is fine from just careful listening.
I must say though that I had a suspicion that at the beginning of Simon & Garfunkel's 'I am a Rock' which is pretty noisy in the original the very high frequency part of the noise was a tiny bit louder. Got at 4/5 but lost afterwards (arrived at 4/8). I will go into this more in the next days.

Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-23 00:30:18
I compressed the same sample as above (6 mins WAV, FM broadcast) with lossyWAV 1.1.4h and --limit 15000
Here are the results:

Code: [Select]
                       lossyWAV/FLAC  bitrate       file size
--------------------------------------------------------------------------------------
without lowpass                    441 kbps         20.3 MB
after lowpass at 16300 Hz          390 kbps         17.9 MB
 

Quick look shows that I can hear no difference  In fact I am not very young 

FM radio bandwidth usually is around 15.5 kHz (though I'm not sure about the exact situation in your country). As you do have audio content above 16.3 kHz I guess that's noise from the transmission or your radio.
390 kbps is what can be expected on average from quality -q 3.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-23 10:10:13
Hallo Nick,

I think the idea behind going --limit 14500 in order to not analyze frequency areas without audio contents can be put on a much better basis.

2Bdecided's principle is based on looking at the audio frequency with lowest audio energy and choosing number-of-bits-removed in such a way that this energy is not suppressed by the method's added noise.
If I see it correctly in real world computation so far 'frequency with lowest audio energy' is replaced by 'frequency with lowest energy - no matter whether there is audio contents or not'. 2Bdecided was aware of this leading to a poor amount of bits-to-remove, and weakened things by use of a spreading function to get at usable results.
This approach has a basic flaw which tends to yield results that are too pessimistic. Things are very clear with lowpassed music when lowpass is lower than --limit. But I think this is also the reason why music originating from one or few instruments is not encoded efficiently. There will be frequency areas without musical contents driving the current machinery to uselessly hold bits-to-remove low.
IMO a simple soulution would be to ignore the result of the spreading function (or let the spreading function return an artificially big value) whenever the audio content for the bins involved is zero. As a criterion for this situation due to rounding errors in the FFT calculation and/or the spreading calculation we probably cannot compare the spreading result to an exact zero but have to allow it to be below a (very low) threshold to consider this as zero audio contents for the bins involved.

I can imagine this will help a lot for such situations like mine when reencoding modified mp3 tracks, or when encoding FM radio like zorzescu does. No special values for --limit necessary, it's all automatic.
I guess it will help a lot for the currently unlucky situation when bitrate is inadequately high with 'simple' music (simple with respect to the audio spectrum).
For other kind of music it mght help locally on occasion, and thus may bring bitrate down also a little bit.
No drawback seen.

Thinking of this and of the low quality settings IMO it can be tolerated to ignore the ends of the audio spectrum for the noise analysis to a higher degree the lower we go down with quality demand. The idea behind is that within real world music and ignoring very artificial music it is the not too extreme frequency range which dominates the musical contents. So audio analysis can concentrate on this. So I'd welcome if you could provide an experimental --lolimit and --hilimit parameter which has the result of the spreading function be ignored (or lets the spreading function return an artificially high value) in case one of the bin's frequency involved is below --lolimit or above --hilimit.
If this proves to be helpful we may consider using it internally depending on -q value (only for low -q value).

Once on a wishing trip: do you mind giving the -p option a parameter demanding the error to be not only at the signal level or below, but to be a certain amount below the signal level? Ideally the parameter has the meaning of 'S/N ratio' (so a value of 2 means noise energy is half the signal energy in worst case).
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-23 13:50:04
The mechanism already disregards bins which correspond to frequencies lower than 20Hz or higher than 16kHz (or --limit).

You've given me a few ideas - I'll have a think then post some proposals....

--lolimit and --hilimit would be fairly easy to implement - --hilimit exists already (as --limit).

If we are talking of ignoring very low bin values then I would probably scrap the existing spreading mechanism(s) - an average would be taken then every bin below (as yet unknown)% of the average would be disregarded.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-23 14:08:32
... an average would be taken then every bin below (as yet unknown)% of the average would be disregarded.

I'm not sure whether this is what I have in mind. This disregards bins with low energy relative to something. But whatever 'something' is: in case there is nearly no audio contents in 'something', bins compared to it aren't ignored. What I have i mind is ignorance of the spreading outcome based on an absolute criterion (being below a very low threshold which only accounts for rounding errors that make the spreading result non-zero even with zero valued bins).
OK, maybe if 'something' is related to the RMS of the block or such, this may be useful. I guess however that some kind of spreading will still be useful for the higher frequencies.

As --limit already works as I described for --hilimit: can you allow for a lower value than 14500?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-23 14:30:37
Easily - it's a matter of changing the help text and one value in the parameter input checking routine.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-23 17:02:02
I wrote:
'This approach has a basic flaw which tends to yield results that are too pessimistic.'
and
'No drawback seen.'.

Not exactly true I'm afraid. The method's added noise still has to be low also when there is no audio content in a certain frequency area.
So it's not a basic flaw, rather a basic disadvantage.
I don't think this makes these new ideas worthless because the fact remains that with no audio contents in a certain range there is no good decision basis for choosing bits-to-remove.
My feeling is that we should apply such bin-ignoring-tactics for zero-audio-containing bins at the outer edges of the frequency range (not necessarily only at the extreme edges), something like for instance above 8 kHz and below 150 Hz, and only, if RMS of the block's total signal is above a certain value (for a very basic idea of masking).
It's all heuristics of course, but this applies already for the spreading function.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-23 17:15:18
Thanks for the added input - still thinking about how best to modify the --postanalyse parameter and spreading function.

lossyWAV beta 1.1.4j attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-23 21:26:42
Thank you, Nick.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-23 22:45:39
I couldn't resist and tried --portable --limit 10000 to get an idea of at what a bitrate we may get when following the current ideas. For my usual test set of various pop music bitrate came down to 317 kbps!

I tried to ABX 'I am a rock' again but didn't succeed (3/3 turned to 5/7 turned to 7/10). Of course we don't know whether I'm just bad at ABXing at least right now. The track wasn't suspicious to me today though, and I just tried because of yesterday's experience.
My suspicion today was more with another track (Friedemann's Sentimental Elegance), but a starting success of 4/4 turned to 6/10. Suspicion remains though.
I went over to my problem sample set, and the first thing to be remarked is that despite --limit 10000 the sensitivity towards difficult tracks isn't seriously reduced. For instance the sample 'Mandylion' shadowking gave recently has a bitrate of 443 kbps which is a lot higher than the average of 317 kbps, and it's only a bit lower than when using 1.1.3e --portable which takes 465 kbps.
I tried to ABX these samples, but again without success. Best result was with eig (7/10). With eig I had the impression most of the time that pitch is shifted up a subtle bit towards higher frequencies - an impression I remember I also had with some former lossyWAV listening tests.

To be clear I don't think that --portable --limit 10000 is a transparent setting - I'm just bad at ABXing.
But I think the ignoring-bins-when-there-is-no-audio-tactics is a promising way to go.
Even with what we've got right now I personally prefer --portable --limit 10000 over a low -q setting with same resulting average bitrate.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-24 08:07:55
Found a few things I wanted to change in the --postanalyse function - it now uses the existing spreading function among others.

lossyWAV beta 1.1.4k attache to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-08-24 10:09:27
If I see it correctly in real world computation so far 'frequency with lowest audio energy' is replaced by 'frequency with lowest energy - no matter whether there is audio contents or not'. 2Bdecided was aware of this leading to a poor amount of bits-to-remove, and weakened things by use of a spreading function to get at usable results.
This approach has a basic flaw which tends to yield results that are too pessimistic. Things are very clear with lowpassed music when lowpass is lower than --limit. But I think this is also the reason why music originating from one or few instruments is not encoded efficiently. There will be frequency areas without musical contents driving the current machinery to uselessly hold bits-to-remove low.
Well, it's "useless" if it's masked, but not "useless" if it's not masked.

...and to determine whether it's masked or not, you need a proper psychoacoustic model.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: 2Bdecided on 2009-08-24 10:14:33
I don't know that you really need to lowpass before processing with lossyWAV.


I did some test:
Code: [Select]
                       lossyWAV/FLAC  bitrate       file size             FLAC           size
------------------------------------------------------------------------------------------------
without lowpass                    578 kbps         26.6 MB            1081 kbps      49.8 MB
after lowpass at 16300 Hz          494 kbps         22.7 MB             967 kbps      44.6 MB
 
After lowpassing WAV file, the resulting bitrate lowers. Cutting frequencies above 16 kHz does not cut sound data in FM radio broadcast. So I decided I need lowpass before further WAV file manipulations 
Yes, you should - almost all FM receivers allow the 19kHz pilot tone through, and various noise either side of that frequency. It's not loud enough for (almost) anyone to hear it, but it's certainly there, visible on a spectrogram, and wastes bits when encoding. This is a fundamental FLAC (and also mp3!) issue - more high frequency data = more bits needed. The bitrate increase/decrease is incidental to the use of lossyWAV in this case - lossyWAV doesn't care.

It is worth lowering the frequency at which lossyWAV stops checking for spectral low points in this case - but for efficiency, it's better still to resample to 32kHz.

Cheers,
David.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-24 12:44:44
....There will be frequency areas without musical contents driving the current machinery to uselessly hold bits-to-remove low.
Well, it's "useless" if it's masked, but not "useless" if it's not masked. ...

Yes, and the current approach certainly is sure-proof, but also very pessimistic. Maybe we should leave it like that for the --standard quality or better.

And yes, my suggestions given are over-simplicistic.
But I guess it's worth while striving at a rude psychoacoustical model which may still be defensive for quality levels below but close to --standard but not as super-pessimistic as it's done right now. With --portable we allow for a very small chance of subtly audible errors. Below --portable we allow for more.

A simple psychoacoustical model may be just ATH-related, of the kind:
if the outcome of the computational spreading function is below a roughly ATH related frequency dependent threshold let the spreading function return the threshold value instead. The threshold can have the property that at say -q 4.0 or above it is exactly related to the ATH such that added noise is below the ATH.
This is still pessimistic as usually there is some masking.
We can take care of this in a rude form by rising the threshold values with decreasing -q value. Which brings an increasing chance of audible errors, but that's what we want from lower -q values (sure the chance for errors should be very low at --portable, and audible issues should be subtle).

Maybe some day a more elaborate method can take account of masking in a preciser way by measuring the energy in masking effect-related frequency bands as is done with mp3 etc.

For the moment a rough ATH related machinery can already lead to improvement IMO, especially with respect to the HF area.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-25 07:40:29
Resultant bitrate tables updated for beta 1.1.4k with / without --postanalyse.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-25 08:04:32
Bitrate increase for -p is higher now for the --portable and --standard than for the lower quality settings other than with the previous menchanism. I thought --postanalyze takes care of something similar to the S/N ratio as a whole not related to the usual lossyWAV mechanism (execpt for implementation details). What exactly does --postanalyze do?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-25 12:20:43
The --postanalyse parameter simply carries out a single windowed FFT analysis per channel of the original codec-block and calculates the spreading function average and minimum. These two values are modified by noise_threshold_shift_average and noise_threshold_shift_minimum (were snr_value and noise_threshold_shift respectively). The minimum value of these two is stored.

Once the first pass remove_bits procedure has produced a clipping limit compliant set of correction data, that data is analysed using the same windowed FFT analysis. The spreading function results are calculated and only the average is used, unmodified.

If the correction data spreading average exceeds the previously stored minimum value for the original data then the bits_to_remove value is reduced by 1, the remove_bits procedure is repeated and the correction data FFT process is repeated until the added noise is below the acceptable level.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-25 12:55:15
I wonder what part of this process is depending on -q. There must be something because otherwise bitrate increase for the very low quality settings is expected to be much higher compared to the increase of say --standard. Is it the noise_threshold_shifts? If so wouldn't it be better to constantly use those noise_threshold_shift values here that belong to say --portable in order to maintain a certain minimum quality this way? I guess the main advantage of this recent addition is for the low quality settings. Quality from --portable on is excellent already.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-25 18:01:56
The noise_threshold_shift_average and noise_threshold_shift_minimum values are the same quality level dependent values as are used in the rest of the (previous to bit-removal) processing.

Actually, I've had a thought.... (dangerous, I know).

This approach could be taken with *every* single analysis to determine the resultant noise allowable bits to remove rather than just using the correction data analysis for the codec-block-length single analysis per channel as used in beta 1.1.4k.

I'll work out how best to implement and see what happens....
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-25 19:23:04
I see. So you have something different in mind than I assumed. --postanalyze is an extra procedure not directly related to the classical lossyWAV approach targeting at a quality adequate for the quality level chosen in the very same sense the usual procedure does.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-25 19:30:12
Nothing has changed except for the possible modification of the calculated bits-to-remove value for a particular codec-block channel when the noise level of the analysed correction data is compared to the spreading function result (amended by quality dependent shifts) of the source audio.

The post-analysis is (optionally) carried out after the existing bit-removal process - after which the processed result is written to the output file.

[edit] I tried the immediate check to see whether the noise associated with the correction data for the calculated bits-to-remove value for each and every analysis FFT is consistent with the calculated bits-to-remove value and it is (well, almost - my 55 sample set at --zero is 134 bytes bigger when modifications are made based on noise modified bits-to-remove). [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-26 08:09:23
Hmmm.... It appears that the post-analyse parameter in its present incarnation simply duplicates (nearly) the inclusion of analyses using an FFT the same length as the codec-block.

Will ponder again, but I think that at the moment simply reversing the inclusion order for additional analyses will allow the inclusion of codec-block length FFTs more easily. (currently 2=64,1024; 3=64,128,1024; 4=64,128,256,1024; 5=64,128,256,512,1024 - would be changed to: 2=64,1024; 3=64,512,1024; 4=64,256,512,1024; 5=64,128,256,512,1024).
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-29 09:55:53
I've come to an end for thinking about those ideas originating from encoding more efficiently 15 kHz lowpassed music.
For --portable I'm using --limit 15000 -s 0.1 now which yields 362 kbps for my test set.
I like the idea that this is a (for me) universally good setting which also deals well with lowpassed music.
From listeneing tests I'm totally happy with the quality.

I don't care about the lost noise control in the 15...16 kHz region, and if it were only for this I'd even go a bit lower with the 15 kHz limit.
I care more about the side effect this has because controlling noise in the x ... 16 kHz region has the defensive side effect of less bits-to-remove as a general tendency compared to not doing so, which makes the entire process more defensive (noise is lowered in the entire frequency range) even if it would not really be necessary in the x ... 16 kHz range.
With --portable we have a little but not a huge safety margin from the transparency border IMO, so to me using --limit 15000 is appropriate.

The procedure is more interesting with higher quality levels.
Using --standard --limit 14500 -s 0.2 I arrive at 435 kbps which is significantly lower than pure --standard, and lowering the -s value has its part in reducing bitrate - to a larger extent than when not using --limit.
That's why I also lowered -s with the --portable quality.
--extreme --limit 14000 -s 0.3 takes 501 kbps.
--insane --limit 13500 -s 0.4 takes 574 kbps.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-29 12:26:21
Thanks for these thoughts - decreasing the --shaping below q/10 will always save resultant bitrate to some extent.

I am intrigued by your use of the --limit parameter but remain convinced (at the moment) that the default should remain at 16kHz per David's method as "published".

Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-29 20:03:04
The idea behind --limit <=15000 is like this:
Let's first assume that we are at --standard quality.
--limit 14500 makes sure that there is no audible noise in the range up to 14.5 kHz according to 2Bdecided's basic principle.
A high value of bits-to-remove means that there's pretty much energy all up to 14.5 kHz. With pretty high energy near 14.5 kHz I consider noise in the 14.5 ... 16 kHz range is masked enough. Considering further our low sensitivity in this very high frequency range.  On the other hand a low value of bits-to-remove means a good S/N ratio at whatever frequency.
To me this is sufficiently plausible to do it like this especially as at the --standard quality level we can assume that we have a pretty large safety margin towards the transparency border.
At the --portable quality level things aren't that clear as we are in the range of heuristics below 2Bdecided's basic principle. But we aren't at the very start of lossyWAV experience. --portable seems to have some small safety margin. That's why I decided for me to use a little bit of a more aggressive setting in line with these ideas.
My motivation for these things comes also from the fact that this way I don't have to care about reencoding rather strongly lowpassed mp3 tracks after having modified them.

So things just come together, and I like the results I listened to.

May be you can look at it like this: Noise analysis has always been restricted to 16 kHz for good reason (at least as a default). What I'm doing right now is put the question on the 16 kHz. Why 16 kHz? In the end it's an arbitrary choice (though it's clear that it should be at very high frequency). I just prefer 15 kHz at the --portable quality level, and even less for the higher ones.

The -s setting isn't necessarily touched by these considerations. I just found that the -s bitrate penalty is higher with a lower --limit. And allowing for not directly controlled noise in the 14.5 .... 16 kHz area is in line with reducing the overall noise shift towards higher frequencies. But it's not a necessary conclusion.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-29 20:12:14
Sounds like a well thought out approach. I'll try some transcoding and see what the results sound like....

btw, I have made more progress with [JAZ]'s request to split the code up into functional chunks - the main program only contains the main loop itself, all functions and procedures have been farmed out into supporting units.

[edit] Tried some transcoding - My Mike Oldfield collection: 383 tracks, 34h18m: --portable --postanalyse --limit 13750 --shaping 0.15 results in 373kbps. [/edit]

[edit2] In foobar2000, add "-S- -P=32768" to the FLAC element of the lossyWAV command line in the converter to remove the seek-table and reduce padding to 32kiB. [/edit2]

[edit3] Same 383 tracks, --portable --limit 14000 --shaping 0.10 results in 365kbps. Sounds fine at relatively quiet nocturnal listening levels.[/edit3]

[edit4] Changed size of padding - 20kiB was too small for adding tags and Replay Gain tags. [/edit4]
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-29 21:41:44
What's the bitrate when not using --limit?

You have encoded a lot of Mike Oldfield's tracks. I would prefer a bit if you could use a certain mix of various kind of pop music (while staying within the same genre which I also do with my test set). In return I am convinced that it needn't be that many tracks to encode which makes things a lot easier. I have always used an identical test set of just 12 tracks of very varying pop music, starting with The Beatles and ending with tracks which were actual at the time (2 or 3 years ago) I set up the test set. My bitrate results were always pretty close to what came out from your many album tests as well as the results of other members, though just 12 tracks is certainly a rather low number which can be improved (I'm just lazy).
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-29 21:54:26
My MO collection is always the one I will turn to for checking if I like the quality of the output. I am working on a vanilla --portable transcode of it at the moment.

I will post some details of my 10 album test set tomorrow.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-29 21:59:37
As it's about quality: preferred and well-known tracks are among the best candidates for a test of course.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-29 22:16:05
Vanilla --portable on the 383 tracks comes in at 392kbps. 365kbps for --portable --limit 14000 --shaping 0.10 seems quite a good reduction by comparison. I'll probably do a full transcode tomorrow and see what the results are for my 760hours, 10826 tracks, 880kbps collection which comes in at 378kbps at --portable.

So, if 392kbps goes to 365kbps for my MO collection then 378kbps may go to around 351 to 353kbps for my whole collection with these revised settings.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-29 23:40:25
--portable --limit 14000 -s 0.1 takes 347 kbps for my test set.

I am glad we're on the same train.
Of course it's disputable where exactly to put the limit.
Maybe it's more appropriate to have the same --limit value with all the quality levels, and maybe we can allow for a lower --limit value than 15000 for --portable, just as you are trying right now.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-30 14:06:28
I did some ABXing with --portable --limit 14000 --shaping 0.1.
I couldn't ABX a problem, but with Rickie Lee Jones' 'Under The Boardwalk', range 2:25.2 ... 2:28.1, I got at 6/7 which turned into 7/10 (I really hate results like this). So there's a bit of a supicion left.
When using --portable --limiit 14500 --shaping 0.1 I was lost with ABXing this spot from the very start.
--portable --limiit 14500 --shaping 0.1 takes 254 kbps for my test set.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-30 14:11:17
I would probably use 21/64*44100 = 14470(.3125) as the cutoff - it's a nice round number of bins.

[edit] sp. [/edit]

[edit2] Cancelled the --portable --limit 14000 --shaping 0.10 transcode in favour of --portable --limit 14470 --shaping 0.10. Should be ready in about 4h30m. [/edit2]

[edit3] Currently 287h38m, 355kbps resultant (FLAC -5 -S- -P=40960) bitrate; 1 file per album. [/edit3]
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-30 19:21:36
I would probably use 21/64*44100 = 14470(.3125) as the cutoff - it's a nice round number of bins. ...

Wonderful.
From experience so far this seems to be a reasonable value.
--portable --limit 14470 --shaping 0.1 takes 354 kbps for my test set as was the case with --limit 14500 (sorry for writing 254 kbps in my last post).

It would be great if we could get some feedback from listening tests of interested members helping to bring lossyWAV forward.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-30 20:11:30
My 760hr collection results in 357kbps at --portable --limit 14470 --shaping 0.10. Quite a nice reduction from the 378kbps for vanilla --portable. I am encouraged by your ABX testing and similarly would appreciate it very much if other user with gifted hearing could devote some of their time to testing this particular setting.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-08-31 16:15:59
My 760hr collection results in 357kbps at --portable --limit 14470 --shaping 0.10. Quite a nice reduction from the 378kbps for vanilla --portable. I am encouraged by your ABX testing and similarly would appreciate it very much if other user with gifted hearing could devote some of their time to testing this particular setting.

I've had a go using the suggested settings.
I have been using LossyWav 1.1.4a up to now and often use it with -q 2.5 -s 0.5 -a 4 --sortspread. By way of explanation I actually find -q 2.5 transparent but I add the "extras" to give me peace of mind in case whatever I'm encoding happens to contain a difficult case.


I happened to have  Bob Marley album to convert so I used my usual settings in v1.1.4a and got an average bitrate of 353. Using 1.1.4k with --limit 11470 -s 0.1 and FLAC -b 512-5 -S- -P=40960 I got an average of 334. I realise that doesn't tell you anything useful about the relative bitrates. However on listening I can't hear any difference at all. I should qualify that by pointing out that I do not have "gifted hearing" - I can't hear as high as 14470 and I'm not good at ABXing. But assuming any noticeable difference would be in the background noise level, I certainly wasn't able to spot a change.

Hope that helps
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-08-31 18:42:04
Thanks for your efforts - the result is encouraging.

On another topic, I am interested to gather opinion on the --sortspread parameter - is anyone using it?
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-08-31 19:23:42
I haven't used --sortspread. I didn't understand the background and was happy without it. So I had no motivation to try it.
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-09-01 11:53:19
Thanks for your efforts - the result is encouraging.

On another topic, I am interested to gather opinion on the --sortspread parameter - is anyone using it?

I use it but only because it's there and the rationale behind it made sense to me. I wouldn't miss it if you decide it's superfluous
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-09-01 17:01:34
Has anyone found that -s 0.1 is better than -s 0 with these settings (--limit 14470) ?

edit: corrected limit
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-09-01 17:34:03
Has anyone found that -s 0.1 is better than -s 0 with these settings (--limit 11470) ?

To avoid any confusion, that was a typo on my earlier entry. It should read --limit 14470. I'll edit my original post if the system will allow it

edit - unfortunately I can't change my original post - \edit

Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-09-02 12:49:41
Just wanted to see how much below -P i can go and still get decent quality. My general impression is Q1.5 (320k) is the limit and quality is not bad even on more difficult tracks. For non-fussy people maybe Q1. Quality <=1 can be a disaster as the noise can fluctuate like static.

The 'serioustrouble' sample is a good one. abxable until Q1.5 - but quality is still good at that level.

Another interesting sample:

http://www.hydrogenaudio.org/forums/index....showtopic=74486 (http://www.hydrogenaudio.org/forums/index.php?showtopic=74486)


I could abx upto Q1.5 easily (Q0~1 is a disaster) . At Q2 i needed more concentration but i believe there is a hiss in the right channel but quality is very good (abx 7/8). Maybe i'll try -P later when things are more quite.

In contrast i tried the wv lossy mode:

250k -hhx  - hiss . Quality is not bad
300k -hhx  -hiss more subtle 7/8
320k - hhx - even more subtle i failed at 5/8

Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-09-02 12:59:48
Thanks for your efforts, encouraging results. (If you have the time / inclination) how is the sample processed with the latest beta using the --sortspread parameter?
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-02 13:28:15
Just wanted to see how much below -P i can go and still get decent quality.  ...

Thanks a lot for testing. Confirms so far that with --portable quality there seems to be a small safety margin.
Did you use plain -q x? Or did you give --limit 14470 a try?
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-09-02 13:33:51
Just wanted to see how much below -P i can go and still get decent quality.  ...

Thanks a lot for testing. Confirms so far that with --portable quality there seems to be a small safety margin.
Did you use plain -q x? Or did you give --limit 14470 a try?


Plain -q x with -s0 using v1.1 encoder.
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-09-02 15:42:04
Plain -q x with -s0 using v1.1 encoder.

You mean before 1.1.1 ? That was over a year ago. Not that there was anything obviously wrong with that version but Nick has been been busy trying to improve.
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-09-02 16:48:48
Ok. Updated to the latest encoder.

My sample is improved. Q0 sounds less static and Q1.5 is about transparent.

Serioustrouble is still serious trouble with Q0~1, quality is not bad at Q1.5 but noise is still fluctuating like static. Q2 was trasparent 5/8. Bitrate is amazing >600 k vs 780k original even for Q0. Q5 is 739 , Q2 645

In contrast i also compared wavpack quality / bitrate (ABR):

200k -hhx5: heavy hiss. poor quality but better than Q0~1 LW (no static) - 201k avg

256k -hhx5: moderate hiss 7/8 - 265k

300k -hhx5: failed to abx 3/8 - 309k


When going for the more 'common' bitrate , less than perfect but still very good quality say 256 ~ 320 k , ABR / CBR seems more efficient and safer..
Doing the same with VBR as in lossywav may at times give inferior quality with still very high close to lossless bitrate. Bitrate saving from Q2.5 >> Q2 isn't much . Q1.5 is a candidate but bitrate with 1.1.4 is higher around 350k. So perhaps going lower than -P isn't so attractive.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-02 19:12:34
Yes, I think we have to accept that it takes -P or at least -q 2.0 to get at what we expect from lossyWAV: perfect quality in a practical sense.
Bitrate of critical spots in critical tracks is high usually - it's a feature. Average bitrate of an entire collection should be pretty attractive for a non-transform codec (though you may get a better one with wavPack lossy according to personal demands).
The current approach of lowering bitrate is to use --limit 14470.
Do you mind trying your samples with this option? Most interesting is the quality of -P, but the results of -q 2.0 or -q 1.5 are very welcome too.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-02 20:21:02
The current approach of lowering bitrate is to use --limit 14470.

was to be (can't edit any more):
The current approach of lowering bitrate is to use --limit 14470 -s 0.1.
Title: lossyWAV 1.2.0 Development Thread
Post by: gmwiz05 on 2009-09-02 21:28:55
Seems like a grand idea to reduce size.
I tested [--portable --limit 14470 -s 0.1] and it sounds transparent to that of [--portable] on a few albums that I tried.
One album had a 15kbps difference and dropped the size down by about 5mb. The others are still being worked on. : /

I will continue doing some testing with different [-s] values.

Has anyone tried -s 0.2 or -s 0.3?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-09-02 21:32:13
--portable has a default --shaping value of 0.25 - reducing this will reduce resultant bitrate.
Title: lossyWAV 1.2.0 Development Thread
Post by: gmwiz05 on 2009-09-02 22:35:28
oh ok then I will test from 0-0.1 -haha
But sure enough it does reduce a good amount for that quality.
If this sounds transparent for the majority will this replace --portable? Or Would be an approximate?
Title: lossyWAV 1.2.0 Development Thread
Post by: gmwiz05 on 2009-09-02 23:54:09
couldn't edit my previous post but I tried the same setting with -s 0.08, faint hisses every now and then.
The second time I re-sampled to 32khz and which reduced the file size even further and the hisses aren't as apparent. (someone may want to confirm it, may just be placebo.)
354kbps=299kbps and the quality was an improvement compared to trying to lower the shaping to 0.080.

"Re-sampling appears to reduce the hissing..." -hope this helps out.

A Yoko Ishida album I had was 140mb before resample, after became 118mb.
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-09-21 14:13:40
Based on the latest tests, that indicate that -s 0.1 is an acceptable value at -q 2.5 (--portable), I came up with this suggestion. Lower the the automatic value of -s a bit quicker accross the board.

for example like this: -s=Q*0.12-0.2 . It should only be decided what the minimum should be, 0.0 or maybe 0.1 .
This gives -q 2.5 -s 0.1 ; -q 5 -s 0.4 ; -q 10 -s 1

And along the --limit way of thinking, maybe some (bin) steps would make sense:
-q 3    --limit 14470 (would be better to derive from sample rate)
-q 4    --limit 15159
-q 5    --limit 15848

Not sure if it would make sense to go above or below these.

The idea would be to incorporate these mechanisms into the defaults.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-22 21:17:04
I second that, using modified --limit / -s values as defaults for the various -q levels.

I am happy with -P --limit 14470 -s 0.1, so this is my favorite -P combination.
Using a -s value of 0.12*q - 0.2 would be fine to me though I personally would prefer a mechanism targeting at somewhat smaller -s values for the higher quality levels, something like s value = 0.008 * q^2 +0.02 * q (q=5 -> s=0.3, q=7.5 -> s=0.6, q=10 -> s=1).
As for --limit I'd prefer to have a constant --limit of 14470 no matter what quality level.
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-09-23 12:02:39
As for --limit I'd prefer to have a constant --limit of 14470 no matter what quality level.

We should make sure however that lossyWav is usable for other sample rates than 44100. This value might not be optimal for 48000Hz (or higher). BTW --limit 15000 could be exactly right for 48000Hz files.
Those considerations might make it too complex maybe?
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-23 12:27:44
OK, I had only 44.1 kHz sample frequency in mind.
From 'clean' bin considerations maybe another limit is more attractive for other sample frequencies. Can't judge on this.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-09-23 19:12:09
I will work on a --altpreset parameter to change the automatic shaping and limit values in line with the proposal.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-23 21:18:17
Thanks a lot, Nick.
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-09-24 12:41:11
I will work on a --altpreset parameter to change the automatic shaping and limit values in line with the proposal.

Thanks for considering this. Do you think it is too soon to amend the normal -q? For testing the manual --limit is OK.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-09-26 21:42:29
lossyWAV beta 1.1.4m attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: jetpower on 2009-09-26 21:57:16
lossyWAV beta 1.1.4m attached to post #1 in this thread.

Why --postanalyse was removed?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-09-26 22:00:41
The postanalyse parameter effectively duplicated the effect of adding an analysis to the process of the same FFT length as the codec-block. I have reversed the order that the additional analyses are added so that the codec-block length FFT analysis is added first - so in effect -a 3 replaces --postanalyse.
Title: lossyWAV 1.2.0 Development Thread
Post by: jetpower on 2009-09-26 22:30:29
ok, good to know it isn't really gone as it seemed to be a useful feature.  I admit i don't understand much of lossywav technicalities but the more analyses the better quality, right?
Title: lossyWAV 1.2.0 Development Thread
Post by: gmwiz05 on 2009-09-27 06:23:54
nice... so does the --altpreset has the new shaping and limit adjustments... 
will try out things lower than --portable... it does go after -portable right?
iLike 
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-09-27 12:55:01
I just tried 1.1.4m on a CD that I originally ripped yesterday. Using --altpreset brought the overall size down from 144MB to 136MB with no discernable difference in noise levels or any other undesirables. Nice work Nick, thanks
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-09-27 21:13:02
Good to hear that recent modifications are as yet well received.

Bugfix: lossyWAV beta 1.1.4n attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-28 10:13:51
Thank you, Nick.
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-09-28 11:31:39
@halb27:

is --limit a lowpass as in MP3 --lowpass ?
Is noiseshaping used to just lower bitrate, improve quality or both ?  - i understand its non-adaptive ATM
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-28 12:34:07
--limit isn't a lowpass. lossyWAV doesn't do any lowpassing.
--limit is a limit to the noise analysis. Noise analysis is looking for low energy in the spectrum, and the lossyWAV principle is targeting at keeping added noise (by removed bits) below this energy.
At very high frequency there's the problem that energy can be very low at a certain frequency range which makes lossyWAV keep nearly all the bits for no good reason. That's why there was always a --limit of 16 kHz to the analysis. --limit is a bit lower now when using --altpreset.

Noise shaping is meant to improve quality. Bitrate goes up a bit when using it, especially at the low quality levels. (Part of the) Noise is shifted this way into the roughly 13+ kHz region IIRC.
Nick can certainly tell more about the details.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-28 21:58:33
Nick, can you please tell about --altpreset's limit default for the portable and standard quality when encoding 44.1 kHz sampled music? (I didn't really understand your formula).
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-09-28 22:09:22
The following table shows the target frequencies and calculated resultant for each quality level. Shaping is appended in the last column.
Code: [Select]
     +--------------------------------+
     | --hilimit target and outcome(s)|
+----+--------+-------+-------+-------+-------+
| -q | Target | 32000 | 44100 | 48000 |Shaping|
+----+--------+-------+-------+-------+-------+
| -4 | 14000  | 14000 | 13781 | 14250 | 0.000 |
| -3 | 14000  | 14000 | 13781 | 14250 | 0.000 |
| -2 | 14000  | 14000 | 13781 | 14250 | 0.000 |
| -1 | 14000  | 14000 | 13781 | 14250 | 0.000 |
|  0 | 14000  | 14000 | 13781 | 14250 | 0.000 |
|  1 | 14200  | 14000 | 14470 | 14250 | 0.051 |
|  2 | 14400  | 14500 | 14470 | 14250 | 0.108 |
|  3 | 14600  | 14500 | 14470 | 14250 | 0.172 |
|  4 | 14800  | 15000 | 14470 | 15000 | 0.247 |
|  5 | 15000  | 15000 | 15159 | 15000 | 0.333 |
|  6 | 15200  | 15000 | 15159 | 15000 | 0.434 |
|  7 | 15400  | 15500 | 15159 | 15750 | 0.549 |
|  8 | 15600  | 15500 | 15848 | 15750 | 0.681 |
|  9 | 15800  | 16000 | 15848 | 15750 | 0.831 |
| 10 | 16000  | 16000 | 15848 | 15750 | 1.000 |
+----+--------+-------+-------+-------+-------+
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-09-29 15:50:56
Thanks a lot Nick.  I think the --altpreset settings are a clever step forward in the development/tuning of lossyWAV/lossyFLAC. At last (with the new settings) the bit rates are well below that of 1.1.0 again and, so far, still without audible problems, even as low as --portable --altpreset.

I hope we can trust the mechanisms enough to let these replace the standard setting after a while (maybe in 1.2?  )

Now, can we think of a use for --lolimit ? I was under the impression that noise is not masked as good by low frequenties, but that may not be the point at all. Be aware that most playback systems hardly output anything below 40Hz, but some do.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-09-30 08:34:45
At the low end of the spectrum the noise analysis is different from the high end.
At the high end the frequency resolution of noise measurement is very fine as there are very many FFT bins the energy of which is evaluated. Chance is pretty high this way that low or no energy is found in a small frequency range driving lossyWAV to remove no or few bits only.
At the low end however the energy of a pretty large frequency range is concentrated in just 1 bin. IMO there is no need for a low frequency limit to the noise analysis.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-10-01 21:05:09
I'm glad that --altpreset is working - it's thanks to your proposal (GeSomeone & halb27) that it happened.

I agree that --lolimit is redundant - it will be removed for v1.2.0.

Is anyone using --alpha? If not then I will remove the parameter which varies it (thus simplifying the code).
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-10-01 23:03:00
I just did a short listening test using -P --altpreset with Atem-lied, badvilbel, bibilolo, bruhns, herding_calls, under the boardwalk, keys_1644ds, triangle-2_1644ds, and everything was fine.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-10-04 18:58:44
I transcoded my whole collection (4wk 3d 15:56, 10826 tracks) in just over 4 hours on my AMD 940 / 4GB DDR2-800 / 1TB Samsung HDD from my server : 358kbps using --portable --altpreset.
Title: lossyWAV 1.2.0 Development Thread
Post by: doccolinni on 2009-10-13 03:47:01
I've been off the forums for a while (relatively), but I just have to say, the development is really astonishing...

Nice work, Nick.C!
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-10-21 21:44:29
With recent threads here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=75098&view=findpost&p=659962) and here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=75539&view=findpost&p=663267) regarding transcoding from lossyWAV output and multiple lossyWAV processing, I am seeking feedback regarding shaping.

Should shaping be on automatically (at whatever level, either q/10 or --altpreset)?

Should shaping be off unless enabled?
Title: lossyWAV 1.2.0 Development Thread
Post by: lvqcl on 2009-10-21 22:14:07
Is it possible to have more noise at lower frequencies than at higher? (WavPack lossy often does noise shaping this way)
Title: lossyWAV 1.2.0 Development Thread
Post by: botface on 2009-10-22 09:18:35
I must admit I've always been a bit confused about the noise shaping in LossyWav. As I understand it the higher the quality setting the more noise shaping is applied. Since the lower quality settings produce the most added noise it always seemed to me that shaping should be applied the other way round IE shaping being applied in an inverse relationship to quality setting. Having said I've never found noise to be a problem without noise shaping. Or, to put it another way. If I ever found noise to be noticeable I simply increased the quality setting to get rid of it. I probably should have done some comparisons of various quality settings with and without shaping but it's never really mattered that much to me as I'm happy to use higher bitrates if necessary.

In answer your question however, I would say leave it as it is. That's only because I have command lines set up in Foobar and EAC that rely on the current approach and I'd rather not have to change them. I don't feel particularly strongly about either way
Title: lossyWAV 1.2.0 Development Thread
Post by: shadowking on 2009-10-22 09:49:18
With recent threads here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=75098&view=findpost&p=659962) and here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=75539&view=findpost&p=663267) regarding transcoding from lossyWAV output and multiple lossyWAV processing, I am seeking feedback regarding shaping.

Should shaping be on automatically (at whatever level, either q/10 or --altpreset)?

Should shaping be off unless enabled?



I have better (flawless) results transcoding WV lossy, Lossywav -P to mp3 *without* NS. I took me a very long time to figure it out and i suspected something along those lines. A few weeks ago i tested again several samples . transcoding from WV and lossywav with default NS produced abxable results. Using flat white noise (S 0) fixed every sample with each encoder. This time around i am more convinced. Florin Ghido told me a few years back that for best transcoding results he prefers flat noise regulated by some quality model (dualstream VBR , lossywav). Also bitrate with mp3 inflates because the bad SFb21 issue - more so with lossywav -P + NS (flat noise is close to native bitrate). This makes some sense as NS is pushing noise more into HF and mp3 is already HF handicapped.

For encoders with a quality model the default should be flat noise. For WV lossy current ABR mode i think the default NS is the way to go. Around 400k it can be turned off as long as one is using slow compression (-hhx4 or better) as the encoder reaches very high quality.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-10-22 21:03:21
Firstly, I think that I've finally got to the bottom of deriving a formula for the amount of noise added during bit-removal - the results of the formula match even more closely with actual bit-removal noise from random data. I've modified the code to adopt this revised method and as a bonus the resultant FLAC bitrate is reduced a bit.

Secondly, taking on board what Shadowking said, I've run some tests on my 55 problem sample set and 10 album test set and arrived at the following resultant FLAC -b 512 -5 bitrates:
Code: [Select]
55 Problem Sample Set
+------------------+----------+----------+----------+----------+----------+----------+----------+----------+
|Version / Settings| FLAC -8  | --insane |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
+------------------+----------+----------+----------+----------+----------+----------+----------+----------+
| beta 1.1.4n      | 780kbit/s| 658kbit/s| 589kbit/s| 515kbit/s| 431kbit/s| 325kbit/s| 259kbit/s| 218kbit/s|
| --altpreset      | 780kbit/s| 652kbit/s| 578kbit/s| 504kbit/s| 419kbit/s| 316kbit/s| 254kbit/s| 216kbit/s|
+------------------+----------+----------+----------+----------+----------+----------+----------+----------+
| beta 1.1.4p      | 780kbit/s| 653kbit/s| 584kbit/s| 509kbit/s| 426kbit/s| 320kbit/s|----------|----------|
| --shaping 0      | 780kbit/s| 649kbit/s| 579kbit/s| 504kbit/s| 422kbit/s| 320kbit/s|----------|----------|
| --altpreset      | 780kbit/s| 647kbit/s| 572kbit/s| 498kbit/s| 414kbit/s| 311kbit/s|----------|----------|
| --altpreset -s 0 | 780kbit/s| 646kbit/s| 570kbit/s| 495kbit/s| 412kbit/s| 311kbit/s|----------|----------|
+------------------+----------+----------+----------+----------+----------+----------+----------+----------+

10 Album Test Set
+------------------+----------+----------+----------+----------+----------+----------+----------+----------+
|Version / Settings| FLAC -8  | --insane |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
+------------------+----------+----------+----------+----------+----------+----------+----------+----------+
| beta 1.1.4n      | 854kbit/s| 639kbit/s| 556kbit/s| 471kbit/s| 383kbit/s| 288kbit/s| 230kbit/s| 200kbit/s|
| --altpreset      | 854kbit/s| 624kbit/s| 534kbit/s| 451kbit/s| 363kbit/s| 275kbit/s| 224kbit/s| 198kbit/s|
+------------------+----------+----------+----------+----------+----------+----------+----------+----------+
| beta 1.1.4p      | 854kbit/s| 633kbit/s| 550kbit/s| 465kbit/s| 378kbit/s| 284kbit/s|----------|----------|
| --shaping 0      | 854kbit/s| 621kbit/s| 537kbit/s| 453kbit/s| 371kbit/s| 284kbit/s|----------|----------|
| --altpreset      | 854kbit/s| 617kbit/s| 527kbit/s| 445kbit/s| 359kbit/s| 272kbit/s|----------|----------|
| --altpreset -s 0 | 854kbit/s| 615kbit/s| 520kbit/s| 438kbit/s| 355kbit/s| 272kbit/s|----------|----------|
+------------------+----------+----------+----------+----------+----------+----------+----------+----------+


lossyWAV beta 1.1.4p attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: gmwiz05 on 2009-10-23 03:36:55
How would this be different from the previous one? I would like to know if the changes affected the --altpreset from before in any way.
Oh! 
Does this help with the re-coding of the same file through lossywav... 
--> i just read the post before the results 
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-10-23 22:32:38
Thanks, shadowking, for your findings.
Thanks, Nick, for your new version.

To me shadowkings findings are sufficient not to do noise shaping as a default. As a bonus bitrate decreases a tiny bit.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-10-24 15:28:21
Completed full collection transcode at --portable --altpreset --shaping 0 | flac -5 which results in 349kbps.

I am now more keen to get some critical feedback regarding the transparency (or not) of this setting.

I will change settings such that -s or --shaping (by themselves) will enable automatic shaping proportion and -s n or --shaping n will set the shaping proportion to n.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-10-27 09:15:04
Lately I didn't follow lossywav development closely but give me a week & I will test every new settings introduced since V1.1.3j (which is obviously the last version I tested as I still have the encoded files on my HDD) on my usual suspects (Abfahrt Hinwil-Fool's Garden-Therion).

This time I will take care of bitrate much more to avoid the misstake I did in the past. (Find plenty of setting intesting via ABXing just to realize in the end that increasing the bitrate without the new setting result in a comparable quality gain ... stupid me ...)

I must do some sorting on my HDD today, read the topic backward & encode the files, then I will probably run the test tomorrow after a good night of sleep.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-10-27 12:37:57
I'd welcome if you could test especially Nick's latest version using -P --altpreset --shaping 0.
This is a very attractive seetting IMO as for -P bitrate is lower compared to what we had before. Hopefully quality is still fine.

I will also do such a test within the next days.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-10-27 19:15:37
Testing of the latest agreed setting (per shadowking's experience) will be greatly appreciated gentlemen!

I am working up to 1.1.4q and expect that it will be the last beta before 1.2.0 - only tidying up of parameters and help text, no mechanism changes.

Acceptable (i.e. transparent) quality at --portable --altpreset --shaping 0 would be an excellent outcome.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-10-30 16:41:06
I didn't have the time yet to test -P -t -s 0 but it's still in the work.

First I tested V1.1.4p vs. lossless as a warm up round because I needed to re-train myself, then I tested V1.1.3j vs. V1.1.4p because I wanted to to know if there was any progression/regression with the basic presets (without  the -t -s 0 switches). It is a necessary step IMHO to be sure that I can test -t -s 0 separatly.

So far it is not very interesting, so here is a summary: all the settings that were not transparent with V1.1.3j are still not transparent with V1.1.4p & I cannot tell any difference neither good or bad between V1.1.3j /V1.1.4p.

Here are the logs:

Abfahrt Hinwil - Lossless Vs. V1.1.4p -q 1.0: Complete Success

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/10/29 16:26:03

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) V1.1.4p -q 1.0.lossy.flac

16:26:03 : Test started.
16:27:45 : 01/01  50.0%
16:28:05 : 02/02  25.0%
16:28:20 : 03/03  12.5%
16:28:34 : 04/04  6.3%
16:28:47 : 05/05  3.1%
16:29:01 : 06/06  1.6%
16:29:12 : 07/07  0.8%
16:29:32 : 08/08  0.4%
16:29:36 : Test finished.

----------
Total: 8/8 (0.4%)


Abfahrt Hinwil - V1.1.3j  -q 1.0 Vs. V1.1.4p -q 1.0: Complete Failure

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/10/29 16:30:29

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) V1.1.3j -q 1.0.lossy.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) V1.1.4p -q 1.0.lossy.flac

16:30:29 : Test started.
16:31:07 : 01/01  50.0%
16:31:41 : 01/02  75.0%
16:32:59 : 01/03  87.5%
16:34:01 : 01/04  93.8%
16:34:37 : 02/05  81.3%
16:35:38 : 02/06  89.1%
16:36:31 : 03/07  77.3%
16:36:54 : 04/08  63.7%
16:37:09 : Test finished.

----------
Total: 4/8 (63.7%)


Fool's Garden - Lossless Vs. V1.1.4p -q 1.5: Complete Success

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/10/29 16:41:22

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) V1.1.4p -q 1.5.lossy.flac

16:41:22 : Test started.
16:41:54 : 01/01  50.0%
16:42:11 : 02/02  25.0%
16:42:22 : 03/03  12.5%
16:42:40 : 04/04  6.3%
16:42:54 : 05/05  3.1%
16:43:04 : 06/06  1.6%
16:43:19 : 07/07  0.8%
16:43:37 : 08/08  0.4%
16:43:43 : Test finished.

----------
Total: 8/8 (0.4%)


Fool's Garden - V1.1.3j  -q 1.5 Vs. V1.1.4p -q 1.5: Complete Failure

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/10/29 16:48:32

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) V1.1.3j -q 1.5.lossy.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) V1.1.4p -q 1.5.lossy.flac

16:48:32 : Test started.
16:49:04 : 00/01  100.0%
16:50:17 : 01/02  75.0%
16:50:37 : 01/03  87.5%
16:51:49 : 02/04  68.8%
16:52:40 : 03/05  50.0%
16:52:56 : 03/06  65.6%
16:53:21 : 03/07  77.3%
16:53:35 : 04/08  63.7%
16:53:38 : Test finished.

----------
Total: 4/8 (63.7%)


Therion - Lossless Vs. V1.1.4p -q 1.5: Complete Success

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/10/30 17:06:06

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) V1.1.4p -q 1.5.lossy.flac

17:06:06 : Test started.
17:06:21 : 01/01  50.0%
17:06:34 : 02/02  25.0%
17:07:38 : 03/03  12.5%
17:08:04 : 04/04  6.3%
17:08:34 : 05/05  3.1%
17:09:02 : 06/06  1.6%
17:09:20 : 07/07  0.8%
17:10:06 : 08/08  0.4%
17:10:10 : Test finished.

----------
Total: 8/8 (0.4%)


Therion - V1.1.3j  -q 1.5 Vs. V1.1.4p -q 1.5: Complete Failure

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/10/30 17:11:44

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) V1.1.3j -q 1.5.lossy.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) V1.1.4p -q 1.5.lossy.flac

17:11:44 : Test started.
17:12:04 : 00/01  100.0%
17:12:23 : 00/02  100.0%
17:12:48 : 00/03  100.0%
17:13:47 : 01/04  93.8%
17:14:21 : 01/05  96.9%
17:14:50 : 01/06  98.4%
17:15:34 : 02/07  93.8%
17:16:33 : 02/08  96.5%
17:16:37 : Test finished.

----------
Total: 2/8 (96.5%)


According to me V1.1.4p seems safe to use but without the new settings it doesn't seem to bring any improvement over V1.1.3j. (At least on these tree samples)

I will test -t -s 0 as soon as I get some time.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-10-30 18:51:07
Thank you for taking the time to test, however I am a bit confused as to why you are ABXing lossy vs lossy in three of the tests and also why you are using an old version of lossyWAV at all.

A more effective use of your time might have been:

Lossless vs lossyWAV v1.1.4p -q 1.0;
Lossless vs lossyWAV v1.1.4p -q 1.5;
Lossless vs lossyWAV v1.1.4p -q 2.0;
Lossless vs lossyWAV v1.1.4p -q 2.5;

That would have gone some way to assessing the transparency point of the revised settings.

If extra time was available then

Lossless vs lossyWAV v1.1.4p -t -s 0 -q 1.0;
Lossless vs lossyWAV v1.1.4p -t -s 0 -q 1.5;
Lossless vs lossyWAV v1.1.4p -t -s 0 -q 2.0;
Lossless vs lossyWAV v1.1.4p -t -s 0 -q 2.5;

would be greatly appreciated.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-10-30 19:22:28
I tested against this old version, because it was the last version which I previously tested so I knew how its artefacts sounded like, I only needed to refresh my memory.

This test wasn't made to test transparency but to test for improvement/regression, between v1.1.3j & v1.1.4p, before I can even start testing -t -s 0 itself.

Maybe it is not obvious for you but I needed to know what v1.1.4p had in the stomach before starting testing new settings. If I would have directly tested a setting with plenty of new parameters I wouldn't have known what improvement regression is due to what parameter.

This first test may seem useless but now I know that v1.1.4p itself is neutral, which is fundamental.

Testing
Lossless vs lossyWAV v1.1.4p -q 1.0;
Lossless vs lossyWAV v1.1.4p -q 1.5;
Lossless vs lossyWAV v1.1.4p -q 2.0;
Lossless vs lossyWAV v1.1.4p -q 2.5;
is not very usefull IMHO, because my short test shows that there is no major improvement/regression, so the transparency point is (very) likely to be the same.

Furthermore I don't really understand why you're asking me to test this again because that's what I partially did: three tests are Lossless vs lossyWAV v1.1.4p, it shows that v1.1.4p -q 1.5 is clearly not transparent. With all logic, even if I didn't test them, -q 2.0 is very likely to be abxable but very hard & -q2.5 transparent with a reasonnable safety margin. With the basic scale, nothing seems to have changed since v1.1.3j. I'm sorry if you expected an improvement... I couldn't hear any & I know these samples by heart.

Testing
Lossless vs lossyWAV v1.1.4p -t -s 0 -q 1.0;
Lossless vs lossyWAV v1.1.4p -t -s 0 -q 1.5;
Lossless vs lossyWAV v1.1.4p -t -s 0 -q 2.0;
Lossless vs lossyWAV v1.1.4p -t -s 0 -q 2.5;
is more interesting as it will help see if  -t -s 0 lower the transparency point.

I will test -t -s 0, I plan to test it soon but I couldn't directly test -t -s 0 because if I would have directly tested it I wouldn't have known if the improvement/regression were due to v1.1.4p or due to -t -s 0

As I said these first results are a boring but necessary step to test  -t -s 0 the right way.

-t -s 0 transpancy results will come later just gimme some more time.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-10-30 20:55:59
Okay - thanks very much for your continued contributions.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-10-30 22:09:17
Thank you very much for your tests, sauvage78.
I welcome lossyWAV development very much, but I am also afraid that something may have gone wrong with a new version, be it only due to a subtle implementation bug. I am a developer myself, and I know this can happen.
So I'm glad your test shows that with current version this seems not to be the case (as usually is).

Of course top interest is in the new changes of --altpreset -s 0 which bring bitrate down a bit based on considerations and experience which make it plausible to do so.
Transparency border so far was around -q 2.0, so the essential question is: does --altpreset -s 0 change this to a major extent, such that -P --altpreset -s 0 isn't transparent?
I'm looking forward to your further tests.

Sorry I didn't do any tests so far on my own. In the last evenings I wasn't alone in the room where I can do ABXing. Hopefully I can do it this weekend.

Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-10-31 19:26:45
Finally I managed to try to ABX deviations from the original for the current version using setting -P --altpreset -s 0 for the problem snippets Atem-lied, badvilbel, bibilolo, bruhns, dither_noise_test, furious, herding_calls, keys_1644ds, S37_Others_MartenotWaves_A_Essential, triange-2_1644ds, Under The Boardwalk (critical snippet of it).

Everything was fine with the exception of S37_Others_MartenotWaves_A_Essential where I arrived at 8/9 which turned into a 8/10. Not a totally fine result but sufficient IMO to rise a serious suspicion.
At ~ second 1.2 a certain tone starts and there is a subtle inaccuracy IMO with the start of it.

Sample is in the upload section. Experience of other members is highly welcome.

I will try and find out whether this is something we had before and was gone unnoticed, or whether it is due to the slightly reduced accuracy requirements of current version and --altpreset -s 0.
I just tried 1.1.4n --altpreset on the sample, and I can't hear an issue. Unfortunately --altpreset -s 0 isn't possible with 1.1.4n.
For an --altpreset -s 0 equivalent of an earlier version I tried 1.1.4j -P --limit 14470 -s 0. I couldn't hear an issue.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-10-31 21:40:45
There has been a change to the formula used to re-create the results of the noise calculations which are performed to determine the added noise on bit removal.

I will have another look at this to see if there are any issues. Thanks for the investigation using previous versions.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-11-01 13:29:48
Here is the results for -q 1.5 -t -s 0: not transparent.

I skipped -q 1 -t -s 0, because I was guessing -q 1.5 -t -s 0 would already be likely to not be transparent & it happened that I was right.

Now I will soon test -q 2 -t -s 0 which is likely to be much harder.

Then maybe I will test a low setting at same bitrate with & without -t -s 0, to see if there is any regression/progression at same bitrate.

Abfahrt Hinwil was the easier to ABX, Fool's Garden & Therion were a little harder but still not really hard.
Here is the 3 logs:

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/01 14:02:55

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) V1.1.4p -q 1.5 -t -s 0.lossy.flac

14:02:55 : Test started.
14:03:04 : 01/01  50.0%
14:03:20 : 02/02  25.0%
14:03:41 : 03/03  12.5%
14:04:09 : 04/04  6.3%
14:04:18 : 05/05  3.1%
14:04:29 : 06/06  1.6%
14:04:50 : 07/07  0.8%
14:05:00 : 08/08  0.4%
14:05:02 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/01 14:05:24

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) V1.1.4p -q 1.5 -t -s 0.lossy.flac

14:05:24 : Test started.
14:06:02 : 01/01  50.0%
14:06:50 : 02/02  25.0%
14:07:21 : 03/03  12.5%
14:08:06 : 04/04  6.3%
14:08:51 : 05/05  3.1%
14:09:42 : 06/06  1.6%
14:10:11 : 07/07  0.8%
14:10:53 : 08/08  0.4%
14:10:55 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/01 14:16:43

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) V1.1.4p -q 1.5 -t -s 0.lossy.flac

14:16:43 : Test started.
14:18:11 : 01/01  50.0%
14:18:52 : 02/02  25.0%
14:19:31 : 03/03  12.5%
14:20:15 : 04/04  6.3%
14:21:50 : 05/05  3.1%
14:22:37 : 06/06  1.6%
14:23:21 : 07/07  0.8%
14:24:12 : 08/08  0.4%
14:24:14 : Test finished.

----------
Total: 8/8 (0.4%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-02 21:13:18
lossyWAV beta 1.1.4q attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-02 21:57:11
Thank you, Nick.
Title: lossyWAV 1.2.0 Development Thread
Post by: carpman on 2009-11-02 22:45:55
Typo from 1st page:

Change log 1.1.4p: 02/11/09

I think this should be:

Change log 1.1.4q: 02/11/09

C.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-03 07:44:25
lossyWAV beta 1.1.4r attached to post #1 in this thread. [Bugfix]
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-11-04 13:31:11
Here is my result for V1.1.4p -q 2 -t -s 0: not transparent.

To my big surprise it was much easier to ABX than I expected, I am beginning to have doubt that -t -s 0 does anything at all, because in my memory -q 2 was stronger than that.
Maybe I am getting too trained on these samples, I dunno.
Sure -t -s 0 decreases bitrate but in no way it seems to improve quality.
Only a test at same bitrate with & without -t -s 0 will tell if this switch is really worth it IMHO.

Here is the logs:

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/04 14:01:53

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) V1.1.4p -q 2 -t -s 0.lossy.flac

14:01:53 : Test started.
14:02:23 : 01/01  50.0%
14:02:42 : 02/02  25.0%
14:03:02 : 03/03  12.5%
14:03:26 : 04/04  6.3%
14:03:52 : 05/05  3.1%
14:04:27 : 06/06  1.6%
14:05:04 : 07/07  0.8%
14:05:33 : 08/08  0.4%
14:05:36 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/04 14:16:26

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) V1.1.4p -q 2 -t -s 0.lossy.flac

14:16:26 : Test started.
14:16:50 : 01/01  50.0%
14:17:02 : 02/02  25.0%
14:17:27 : 03/03  12.5%
14:17:51 : 04/04  6.3%
14:18:13 : 05/05  3.1%
14:18:40 : 06/06  1.6%
14:19:00 : 07/07  0.8%
14:19:23 : 08/08  0.4%
14:19:25 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/04 14:20:02

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) V1.1.4p -q 2 -t -s 0.lossy.flac

14:20:02 : Test started.
14:21:01 : 01/01  50.0%
14:21:34 : 02/02  25.0%
14:22:22 : 03/03  12.5%
14:23:07 : 04/04  6.3%
14:23:49 : 05/05  3.1%
14:24:22 : 06/06  1.6%
14:24:53 : 07/07  0.8%
14:25:32 : 08/08  0.4%
14:25:34 : Test finished.

----------
Total: 8/8 (0.4%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-04 14:23:10
Which version of lossyWAV did you use for these samples?

Could you please try this with beta v1.1.4q?

It may be that --altpreset is not for everyone, i.e. younger ears may not find it to be as transparent as older ears due to high frequency age related attenuation.[/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-11-04 14:48:07
According to a test file posted on HA by someone else I can hear up to 17Khz (likely somewhere between 17 & 18Khz) so I am not particulary sensitive to high frequency & I am not particulary young either ... I am 31yo.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-04 15:54:55
Thanks for your test, sauvage78.
As I found with another sample (see above) version 1.1.4p seems to be a regression and - as Nick said already - it would be very welcome if you would use 1.1.4q or 1.1.4r. So your ABX results may have nothing to do with -t -s 0, but just with the general regression of version 1.1.4p. We don't know.

Of course -t -s 0 does not increase quality. It decreases bitrate which does not necessarily mean that quality decreases, at least not in a significant way that makes -P --altpreset -s 0 not transparent.

Though -q 2 is expected to yield great quality the results may not necessarily be transparent.
Top interest is in the transparency or non-transparency of -P of newest version (versions 1.1.4q and 1.1.4r are the same in our context).

So it would be great if you could test 1.1.4r -P --altpreset (which implies -s 0 as a default) on your samples. -q 2 results are also welcome of course.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-11-04 16:31:12
Ok, I will re-test -q 2 -t -s 0 with V1.1.4r soon, then I will test at same bitrate with & without -t -s 0.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-11-06 22:01:42
Here is the result for V1.1.4r -q 2 -t -s 0: not transparent.

Abfahrt Hinwil was easy & Fool's Garden was very easy, I didn't notice any difference between V1.1.4p & V1.1.4r.

Then I made a misstake when ABXing Therion, instead of ABXing V1.1.4r -q 2 -t -s 0 vs. Lossless, I abxed V1.1.4r -q 2 -t -s 0 vs. V1.1.4r -q 2.5 -t -s 0 & I failed ... I didn't realize my misstake untill I read the log the end of ABXing.

So I retried V1.1.4r -q 2 -t -s 0 vs. Lossless & I succeded 7/8 & as I was angry about my misstake I went up to 9/10.

I must say that after the first uncessfull ABX, which take more time than when it's easy, I was bored ... bored & angry at me, so I put the 1 misstake on the loss of focus.

I had the "feeling" that Therion was harded with V1.1.4r than with  V1.1.4p but it is likely due to my mixing of files which created a lack of focus.

So for me there is no major difference between V1.1.4p & V1.1.4r, at -q 2 -t -s 0 both are not transparent.

Here are the 3 valids logs & the 4th is my misstake ...

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/06 22:24:35

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) V1.1.4r -q 2 -t -s 0.lossy.flac

22:24:35 : Test started.
22:24:51 : 01/01  50.0%
22:25:04 : 02/02  25.0%
22:25:22 : 03/03  12.5%
22:25:42 : 04/04  6.3%
22:26:21 : 05/05  3.1%
22:27:02 : 06/06  1.6%
22:28:15 : 07/07  0.8%
22:28:53 : 08/08  0.4%
22:28:55 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/06 22:32:28

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) V1.1.4r -q 2 -t -s 0.lossy.flac

22:32:28 : Test started.
22:32:44 : 01/01  50.0%
22:32:55 : 02/02  25.0%
22:33:10 : 03/03  12.5%
22:33:26 : 04/04  6.3%
22:33:38 : 05/05  3.1%
22:33:54 : 06/06  1.6%
22:34:07 : 07/07  0.8%
22:34:23 : 08/08  0.4%
22:34:24 : Test finished.

----------
Total: 8/8 (0.4%)


Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/06 22:48:12

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) V1.1.4r -q 2 -t -s 0.lossy.flac

22:48:12 : Test started.
22:49:41 : 01/01  50.0%
22:50:02 : 02/02  25.0%
22:50:52 : 02/03  50.0%
22:51:46 : 03/04  31.3%
22:52:04 : 04/05  18.8%
22:52:21 : 05/06  10.9%
22:52:47 : 06/07  6.3%
22:53:18 : 07/08  3.5%
22:53:48 : 08/09  2.0%
22:54:19 : 09/10  1.1%
22:54:22 : Test finished.

----------
Total: 9/10 (1.1%)


Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/06 22:35:38

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) V1.1.4r -q 2 -t -s 0.lossy.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) V1.1.4r -q 2.5 -t -s 0.lossy.flac

22:35:38 : Test started.
22:38:30 : 01/01  50.0%
22:41:25 : 01/02  75.0%
22:42:29 : 01/03  87.5%
22:42:59 : 01/04  93.8%
22:43:33 : 01/05  96.9%
22:44:14 : 01/06  98.4%
22:45:43 : 02/07  93.8%
22:46:03 : 02/08  96.5%
22:46:06 : Test finished.

----------
Total: 2/8 (96.5%)
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-06 23:21:52
So you can ABX rather easily V1.1.4r -q 2 -t -s 0 on your samples.
And you couldn't ABX a difference between V1.1.4r -q 2 -t -s 0 and V1.1.4r -q 2.5 -t -s 0 on Therion.
So we do have to fear you can ABX V1.1.4r -q 2.5 -t -s 0 vs. lossless.
I am curious for your results.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-07 13:29:49
lossyWAV beta 1.1.4s attached to post #1 in this thread.

I'm wondering if 14.470kHz is a bit low to stop looking for the low bins. The next bin up (for a 64 sample FFT) is 15.159kHz. This is still below the original 16kHz target and will probably result in a lossyFLAC bitrate somewhere between that achieved using 14.47kHz and 16kHz.

I would be interested to know the result of ABX testing on beta 1.1.4s -q 2.0 --limit 15159.

[edit] For my full collection --portable --limit 15159 results in a lossyFLAC average bitrate of 361kbit/s. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-11-09 11:48:14
I'm wondering if 14.470kHz is a bit low to stop looking for the low bins.

I would be interested to know the result of ABX testing on beta 1.1.4s -q 2.0 --limit 15159.

I'd like to remind that when --limit was first tested the commands were like -P --limit 14470 -s 0.1 .
-s 0.1 was dropped in favor of -s 0 after shadowking remarked that the result for transcoding to mp3 were better without noiseshaping.

So another questing would be is -q 2.5 --limit 14470 -s 0.1 transparent while the same with -s 0 is not?

It is totally possible that a higher --limit could counter this difference, but then again, at what -q must we claim transparency? At least for -q 2.0 and lower --limit 14470 seems suitable, you results for -P --limit 15159 however seems still reasonable, if it brings an improvement compared to --limit 14470.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-18 20:09:30
lossyWAV beta 1.1.5a attached to post #1 in this thread.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-18 20:22:15
Thank you, Nick.

As for the decision about how to proceed with -P I think we should wait a bit yet until hopefully sauvage78 comes up with his important listening test for -P.
At the moment - as long as no one else comes up - he is the most important person to prove that -P (together with the --limit 14470 -s 0 equivalent setting) isn't transparent.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-11-18 20:32:46
I still have some listening tests to do for lossywav ... but lately I started to play Dragon Age: Origins (http://dragonage.bioware.com/) so I will be away from audio testing soon. I will test 2 (or 3) more settings tomorow before I completely get swallowed by the game.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-18 20:35:48
Many thanks for the offer of (yet more!) ABXing - if the current --altpreset is not good then please try --limit 15159.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-11-18 20:41:29
I intend to test:
1: -q 2.5 -t -s 0
2: -t -s 0 vs. normal preset at same bitrate

Then if I am in the good mood
3: --limit 15159 at an unknown yet but non transparent setting (Edit: vs. normal preset at same bitrate)

After this I will stop testing lossywav for a looong time, because as I said I will play DAO in december & in january I will buy a Core I3 530 & upgrade to Win 7 64Bits so I will change all my setup including my actual audio setup. After the upgrade, I will have to re-test my setup before I can do ABXing again.
Title: lossyWAV 1.2.0 Development Thread
Post by: sauvage78 on 2009-11-19 18:20:53
Here is the result for V1.1.4r -q 2.5 -t -s 0: Not always transparent.

I failed to ABX Abfahrt Hinwil & Therion but I successfully ABXed Fool's Garden.
I don't know how this affect my result but I must say I was more sleepy than usually (had a short night 7h vs. 9h usually when I ABX) & not really in the mood for ABXing, but I did it as I promised to do more test yesterday.
Anyway if I would have to wait to be in perfect health & in enthousiastic mood to do ABXing, I would never get the job done, so take it as a random part of ABXing.
One of the reason why I wasn't enthousiatic about this test in particular is that I expected that Abfahrt Hinwil & Therion would be painfull to ABX, & both were really painfull, I did my best for Abfahrt Hinwil but after 4 trials on Therion I didn't focus as much as it was obvious to me that I would fail.

Now that I have tested -t -s 0 at various settings, I begin to have an opinion on it ... my opinion is that it doesn't do much anything, if it does anything at all.
It must be confirmed by a test at same bitrate with & without -t -s 0, but it seems obvious to me that it doesn't drastically affect the transparency point.

Now about myself being able to ABX -q 2.5 -t -s 0 on Fool's Garden, well I think it has nothing to do with -t -s 0, but with Fool's Garden being ABXable at -q 2.5 with or without -t -s 0 & maybe even at higher settings.

I haven't browsed the thread back to see if I already tried to ABX Fool's Garden at -q 2.5 & failed with an old version of lossywav ... but it is very likely that I failed as the sample was new to me, now that I am highly trained on Fool's Garden, I suspect that I can ABX it at -q 2.5. Even if I would fail, for me it means that the margin of safety of -q 2.5 is lower than I expected in the past. It is likely that -q 2.5 will fail to be transparent one day, providing someone with good earing find the good killer sample.

PM for Nick.C:
Thks for the credit offer, I don't really care about it, just use my HA nick if you want to add me. I started ABXing lossywav as I was wondering if I would use it for my own backup use. To be honest it is unlikely that I will use it because I always buy bigger & bigger HDD. I am a lossless user at heart. I don't plan to do much more testing in the future. So adding me now would be more of a swansong thks than of an encouragement. All in all I have only spend something like a dozen of hours ABXing lossywav.

Here is the logs:

Abfahrt Hinwil: Failed

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/19 17:40:57

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\01- Abfahrt Hinwil (Artefact Only) V1.1.4r -q 2.5 -t -s 0.lossy.flac

17:40:57 : Test started.
17:43:07 : 01/01  50.0%
17:44:32 : 02/02  25.0%
17:45:25 : 02/03  50.0%
17:45:46 : 03/04  31.3%
17:46:56 : 04/05  18.8%
17:47:25 : 05/06  10.9%
17:47:54 : 05/07  22.7%
17:48:38 : 05/08  36.3%
17:48:41 : Test finished.

----------
Total: 5/8 (36.3%)


Fool's Garden: Success

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/19 17:49:19

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\02- Fool's Garden (Artefact Only) V1.1.4r -q 2.5 -t -s 0.lossy.flac

17:49:19 : Test started.
17:49:35 : 01/01  50.0%
17:50:16 : 02/02  25.0%
17:51:35 : 03/03  12.5%
17:52:50 : 04/04  6.3%
17:53:50 : 05/05  3.1%
17:54:30 : 06/06  1.6%
17:55:18 : 07/07  0.8%
17:56:26 : 08/08  0.4%
17:56:31 : Test finished.

----------
Total: 8/8 (0.4%)


Therion: Failed

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/11/19 18:24:35

File A: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) Lossless.flac
File B: C:\01- Disque C\03- Save\08- Audio Temp\01- Samples For Encoding Test\01- Listening Test\07- Lossywav Test\03- Test\03- Therion (Artefact+Context) V1.1.4r -q 2.5 -t -s 0.lossy.flac

18:24:35 : Test started.
18:25:49 : 00/01  100.0%
18:26:31 : 00/02  100.0%
18:28:21 : 01/03  87.5%
18:29:22 : 02/04  68.8%
18:30:26 : 03/05  50.0%
18:31:58 : 03/06  65.6%
18:32:16 : 04/07  50.0%
18:33:23 : 04/08  63.7%
18:33:27 : Test finished.

----------
Total: 4/8 (63.7%)
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-19 18:39:15
Many thanks for the ABX testing - not sure exactly how to proceed now....
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-19 19:57:25
Thanks for testing, sauvage78.

To me it looks like this:
-P isn't transparent in any case.
Probably this isn't due to neither the lower --limit nor -s 0.
As for the latter IMO we should default to a lower limit like 14470 or 15119 and to -s 0.
shadowking's arguments are relevant IMO because good transcoding properties are an important feature, and the sfb21 bloat issue is a bigger one with noise shifted up, and because of the lower transcoding quality when doing noise shift.

Should we continue using -P as a substitute for -q 2.5?
With the --portable quality we did never really demand for transparency in any case. At least in theory. In practice however we did struggle for it.
I wouldn't care giving away the q-equidistance for the portable, standard, extreme and --insane quality levels. This equidistance just looks good but has no real significance.
I just tried 1.1.4r -q 3 --altpreset for finding average bitrate for my standard test set and arrived at 368 kbps.
1.1.4r -q 3.5 --altpreset yields 381 kbps.
As these are not significantly higher bitrates compared to -q 2.5 I suggest we use -q 3 or -q 3.5 internally for --portable.
A higher -q value is the most substantial contribution for improving quality.

Putting it all together and to make a precise suggestion: I suggest to use -P as an equivalent for -q 3.5 --limit 14470 -s 0.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-19 20:06:31
My gut reaction to this is that we have changed two things and are unsure which has caused --portable not to be transparent (using these particularly difficult samples) - if it ever was (have we definitively proven that the existing --portable was transparent?).

I am not initially inclined to move the -q relationship for --portable however I will probably change the --limit to 15159.

I am happy to keep --shaping defaulting to off.

The last ABX test that would be of very useful is --portable - if that was transparent then that would be great.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-20 08:03:58
--limit 15159 is fine for me too.
As for the -q correspondance of -P:
I think there are reasons to have -P correspond to -q 2.5 (-q 2.5 is very fine though not perfect).
I just hope that it's not the equidistant -q distribution among --portable to --insane which has a major impact on your decision. Equidistance is fine for --standard to --insane because it doesn't matter at all what exact -q value to use for --extreme or --insane. It's different IMO for --standard (which should correspond to 2Bdecided's original method) and --portable (which should be transparent or next-to-transparent). It's been pure chance that the equidistant oriented -q correspondance of --portable so far was considered to be transparent and still can be considered close to that.

In case it's equidistance that matters you most you could consider to remap the current -q values to new ones in a linear way that makes old -q 2 new -q 0 and keeps old -q 5 as new -q 5.
This stretches the old 2...8 scale to a new 0...10 scale, keeps the meaning of -q 5 constant, gives away the outer old values <2 and >8 which are either not worse the lossyWAV quality demands or too paranoid, and maps old -q 3.5 to new -q 2.5.
This way --portable etc. can correspond to the -q values in numerically exactly the same way it is done now, but with -q meaning the new -q values.
Together with the lower --limit and -s 0 this even keeps -P bitrate pretty much the same for the new situation (new -q values together with lower --limit and -s 0) as compared to the old situation (old -q values together with old 'original' limit value and default noise shaping).
So bitratewise we wouldn't change things seriously when considering the old and new situation for -P.
For --standard, --extreme, --insane we get smaller bitrates, with increasing savings the higher the quality demands. Fine behavior IMO.
Qualitywise at -P we get a significantly better safety margin IMO due to the improved -q value. I think -q value is the crucial thing when it comes to safety margin. We can do so without increasing bitrate seriously because of the new --limit and -s 0 strategy. I definitely think it's better to have bitrate increase be caused by a higher -q value than by using --limit 16000 and noise shaping.
For --extreme and --insane quality demands decrease with this suggestion compared to the old situation. IMO this is nothing at all to worry about.

Instead of remapping the old 2...8 to a new 0...10 scale you could also remap old 1...9 to new 0...10. The principle is the same, but it keeps the meaning of -q very close to what it is now.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-20 13:09:03
lossyWAV beta 1.1.5b attached to post #1 in this thread.

I took your idea of re-mapping the scale.... old -q 0 is now -q -4 (negative -q values have been re-enabled for the bitrate conscious); -q 5 is about the same as old -q 5.

Revised bitrate table:
Code: [Select]
+---------+----------+----------+----------+
| v1.1.5b | default  | -l 15159 |    -t    |
+---------+----------+----------+----------+
| -q 10.0 | 654kbit/s| 647kbit/s| 626kbit/s|
| -q  9.5 | 641kbit/s| 633kbit/s| 614kbit/s|
| -q  9.0 | 628kbit/s| 619kbit/s| 603kbit/s|
| -q  8.5 | 614kbit/s| 605kbit/s| 591kbit/s|
| -q  8.0 | 599kbit/s| 591kbit/s| 579kbit/s|
+---------+----------+----------+----------+
| -q  7.5 | 584kbit/s| 576kbit/s| 567kbit/s|
+---------+----------+----------+----------+
| -q  7.0 | 570kbit/s| 561kbit/s| 555kbit/s|
| -q  6.5 | 555kbit/s| 546kbit/s| 544kbit/s|
| -q  6.0 | 540kbit/s| 531kbit/s| 532kbit/s|
| -q  5.5 | 525kbit/s| 516kbit/s| 520kbit/s|
+---------+----------+----------+----------+
| -q  5.0 | 510kbit/s| 501kbit/s| 508kbit/s|
+---------+----------+----------+----------+
| -q  4.5 | 493kbit/s| 485kbit/s| 496kbit/s|
| -q  4.0 | 477kbit/s| 469kbit/s| 485kbit/s|
| -q  3.5 | 463kbit/s| 456kbit/s| 474kbit/s|
| -q  3.0 | 450kbit/s| 443kbit/s| 462kbit/s|
+---------+----------+----------+----------+
| -q  2.5 | 427kbit/s| 422kbit/s| 442kbit/s|
+---------+----------+----------+----------+
| -q  2.0 | 416kbit/s| 411kbit/s| 431kbit/s|
| -q  1.5 | 389kbit/s| 384kbit/s| 421kbit/s|
| -q  1.0 | 365kbit/s| 362kbit/s| 411kbit/s|
| -q  0.5 | 344kbit/s| 341kbit/s| 400kbit/s|
+---------+----------+----------+----------+
| -q  0.0 | 325kbit/s| 322kbit/s| 391kbit/s|
+---------+----------+----------+----------+
| -q -0.5 | 306kbit/s| 303kbit/s| 381kbit/s|
| -q -1.0 | 289kbit/s| 287kbit/s| 372kbit/s|
| -q -1.5 | 274kbit/s| 271kbit/s| 363kbit/s|
| -q -2.0 | 259kbit/s| 257kbit/s| 354kbit/s|
| -q -2.5 | 247kbit/s| 245kbit/s| 345kbit/s|
| -q -3.0 | 236kbit/s| 234kbit/s| 337kbit/s|
| -q -3.5 | 226kbit/s| 225kbit/s| 329kbit/s|
| -q -4.0 | 218kbit/s| 217kbit/s| 322kbit/s|
+---------+----------+----------+----------+
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-20 15:11:17
Thanks a lot for remapping.
Can you tell a bit about how excatly is the mapping (you say old and new -q 5 are only roughly the same)?

I hope the bitrates are for your 55 problems set. Because otherwise -q 2.5 bitrate looks very high.
I guessed -l 15159 means --limit 15159 and noise shaping with a value of q/10, and -t means --limit 14470 and no noise shaping. But bitrates confuse me. With high quality settings bitrate of -t is lower than that of -l 15159, and with mediocre and low quality settings it's the other way around. I don't really understand this.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-20 16:23:06
I think I have come to understand it:
default means: old -q system, --limit 16000, no noise shaping (noise shaping is off by default since quite a while).
-l 15159: same as default but with --limit 15159 instead of --limit 16000.
-t: new -q system, --limit 15159, no noise shaping.

Right?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-20 17:52:59
It certainly is - you are right, these results are for my 55 problem sample set (and your second set of assumptions are correct).

I am working on results for my 10 album test set and will post tonight.

Shaping is off by default. --altpreset has a new -q system and a fixed --limit of 15159.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-20 20:08:47
Thank you Nick.

1.1.5b -P --altpreset has an average bitrate of 380 kbps for my standard test set. That's fine to me.

Can you please tell a bit about how the old -q values are mapped to the new -q values?
1.1.4j -q 3.0 --limit 15159 -s 0 yields 380 kbps, as does 1.1.5b -q 3.0 --limit 15159. But the encodings are NOT bit-identical (they start to differ from sec. 43 for the track I tested, so there must be a lot in common but not everything is identical).
1.15b -P --altpreset yields a filesize for the track I tested which is a very tiny bit bigger than the result of 1.15b -q 3.0 --limit 15159, so probably new -q 2.5 doesn't map exactly to old -q 3.0. Probably it's the same for -q 5 hence your remark.
Can you please bring some light into these things?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-20 20:20:00
The following will (hopefully) clarify the changes to the --altpreset settings:
Code: [Select]
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Param |  -4   |  -3   |  -2   |  -1   |   0   |   1   |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |  10   |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| dbtk  |  1.50 |  1.75 |  2.00 |  2.25 |  2.50 |  2.75 |  3.00 |  3.25 |  3.50 |  3.75 |  4.00 |  4.25 |  4.50 |  4.75 |  5.00 |  
| ntsm  | 36.00 | 32.00 | 28.00 | 24.00 | 20.00 | 16.00 |  9.00 |  6.00 |  3.00 |  0.00 | -2.40 | -4.80 | -7.20 | -9.60 |-12.00 |
| ntsa  | -2.00 | -6.00 |-10.00 |-14.00 |-18.00 |-22.00 |-23.50 |-23.50 |-23.50 |-25.00 |-28.00 |-31.00 |-34.00 |-37.00 |-40.00 |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Param |  -4   |  -3   |  -2   |  -1   |   0   |   1   |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |  10   |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| dbtk  |  2.50 |  2.60 |  2.70 |  2.80 |  2.90 |  3.00 |  3.10 |  3.20 |  3.30 |  3.40 |  3.50 |  3.60 |  3.70 |  3.80 |  3.90 |
| ntsm  | 20.00 | 17.75 | 15.50 | 13.25 | 11.00 |  8.75 |  6.50 |  4.25 |  2.00 | -0.25 | -2.50 | -4.75 | -7.00 | -9.25 |-11.50 |
| ntsa  |-18.00 |-19.00 |-20.00 |-21.00 |-22.00 |-23.00 |-24.00 |-25.00 |-26.00 |-27.00 |-28.00 |-29.00 |-30.00 |-31.00 |-32.00 |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

nb: dbtk == Dynamic_Bits_To_Keep; ntsm == Noise_Threshold_Shift_Minimum; ntsa == Noise_Threshold_Shift_Average.


Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-20 21:06:51
I see, you didn't remap -q itself but some (all?) of the internal parameters which are controlled by the -q value.
I'm not into these details any more (in case I ever was), but I see -q 2.5 gets more defensive with respect to all of the three parameters you mentioned.
Not quite true for -q 5 however where 1 parameter is more aggressive.
I'd welcome if all the -q levels up to -q 5 would become more defensive in any internal aspect. Ideally to me -q 5 has exactly the same meaning no matter old or new -q system.
Is there any reason for giving other weights to the -q controlled internal details than you do when not using --altpreset? If not why not hang on to the usual weigths and give -q 5 the exact internal meaning as before,
give -q 2.5 the internal meaning of former -q 3 (or a similar value like -q 3.5), and so on?

Sorry for being not totally content for so many times, and you do such a hard job on lossyWAV.
I really appreciate your great work.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-20 22:14:58
One notable change regarding the --altpreset settings is that the rate of change is constant with quality setting. This cannot be said of the previous settings. with this in mind and also the fact that the existing --portable is "close" to transparent, I feel quite comfortable with the minor change to Dynamic_Minimum_Bits_To_Keep (which at -q 5 --altpreset is still above the level for the existing -q 2.5).

The rate of change of resultant bitrate for my 55 problem sample set is significantly more consistent:
Code: [Select]
+---------+----------+----------+----------+
| v1.1.5b | default  | -l 15159 |    -t    |
+---------+----------+----------+----------+
| -q 10.0 | -------- | -------- | -------- |
| -q  9.5 | -13kbit/s| -14kbit/s| -11kbit/s|
| -q  9.0 | -14kbit/s| -14kbit/s| -12kbit/s|
| -q  8.5 | -14kbit/s| -14kbit/s| -12kbit/s|
| -q  8.0 | -14kbit/s| -15kbit/s| -12kbit/s|
| -q  7.5 | -15kbit/s| -15kbit/s| -12kbit/s|
| -q  7.0 | -15kbit/s| -15kbit/s| -12kbit/s|
| -q  6.5 | -15kbit/s| -15kbit/s| -12kbit/s|
| -q  6.0 | -15kbit/s| -15kbit/s| -12kbit/s|
| -q  5.5 | -15kbit/s| -15kbit/s| -12kbit/s|
| -q  5.0 | -15kbit/s| -15kbit/s| -12kbit/s|
| -q  4.5 | -17kbit/s| -16kbit/s| -12kbit/s|
| -q  4.0 | -16kbit/s| -16kbit/s| -12kbit/s|
| -q  3.5 | -14kbit/s| -14kbit/s| -11kbit/s|
| -q  3.0 | -13kbit/s| -12kbit/s| -11kbit/s|
| -q  2.5 | -23kbit/s| -21kbit/s| -21kbit/s|
| -q  2.0 | -12kbit/s| -11kbit/s| -11kbit/s|
| -q  1.5 | -27kbit/s| -26kbit/s| -10kbit/s|
| -q  1.0 | -23kbit/s| -23kbit/s| -10kbit/s|
| -q  0.5 | -21kbit/s| -21kbit/s| -10kbit/s|
| -q  0.0 | -20kbit/s| -19kbit/s| -10kbit/s|
| -q -0.5 | -19kbit/s| -18kbit/s| -10kbit/s|
| -q -1.0 | -17kbit/s| -17kbit/s|  -9kbit/s|
| -q -1.5 | -15kbit/s| -15kbit/s|  -9kbit/s|
| -q -2.0 | -14kbit/s| -14kbit/s|  -9kbit/s|
| -q -2.5 | -13kbit/s| -12kbit/s|  -9kbit/s|
| -q -3.0 | -11kbit/s| -11kbit/s|  -8kbit/s|
| -q -3.5 | -10kbit/s|  -9kbit/s|  -8kbit/s|
| -q -4.0 |  -8kbit/s|  -8kbit/s|  -8kbit/s|
+---------+----------+----------+----------+


The only blip for revised --altpreset is between -q 3 and -q 2.5 which is  caused by --impulse (on between -q 3 and -q 10).

[edit] My full collection just finished transcoding using --portable --altpreset: 378kbit/s. [/edit]
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-20 22:27:10
I see.
I just don't understand the dbtk settings for -q 5. Doesn't the lower table correspond to the --altpreset mapping and the upper table to the usual one?
Then what do you mean by saying the new -q 5 dbtk setting is still more defensive than it was (at least I read it like this)?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-20 22:32:22
I meant that the revised --altpreset setting at --standard is more onerous than the existing setting at --portable. (revised above).
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-21 17:15:47
lossyWAV beta 1.1.5c attached to post #1 in this thread.

Minor revision to --altpreset settings.
Code: [Select]
v1.1.4s internal settings
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Param |  -4   |  -3   |  -2   |  -1   |   0   |   1   |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |  10   |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| dbtk  |  1.50 |  1.75 |  2.00 |  2.25 |  2.50 |  2.75 |  3.00 |  3.25 |  3.50 |  3.75 |  4.00 |  4.25 |  4.50 |  4.75 |  5.00 |  
| ntsm  | 36.00 | 32.00 | 28.00 | 24.00 | 20.00 | 16.00 |  9.00 |  6.00 |  3.00 |  0.00 | -2.40 | -4.80 | -7.20 | -9.60 |-12.00 |
| ntsa  | -2.00 | -6.00 |-10.00 |-14.00 |-18.00 |-22.00 |-23.50 |-23.50 |-23.50 |-25.00 |-28.00 |-31.00 |-34.00 |-37.00 |-40.00 |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

v1.1.5b internal settings
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Param |  -4   |  -3   |  -2   |  -1   |   0   |   1   |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |  10   |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| dbtk  |  2.50 |  2.60 |  2.70 |  2.80 |  2.90 |  3.00 |  3.10 |  3.20 |  3.30 |  3.40 |  3.50 |  3.60 |  3.70 |  3.80 |  3.90 |
| ntsm  | 20.00 | 17.75 | 15.50 | 13.25 | 11.00 |  8.75 |  6.50 |  4.25 |  2.00 | -0.25 | -2.50 | -4.75 | -7.00 | -9.25 |-11.50 |
| ntsa  |-18.00 |-19.00 |-20.00 |-21.00 |-22.00 |-23.00 |-24.00 |-25.00 |-26.00 |-27.00 |-28.00 |-29.00 |-30.00 |-31.00 |-32.00 |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

v1.1.5c internal settings
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Param |  -4   |  -3   |  -2   |  -1   |   0   |   1   |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |  10   |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| dbtk  |  2.50 |  2.64 |  2.78 |  2.92 |  3.06 |  3.19 |  3.33 |  3.47 |  3.61 |  3.75 |  4.00 |  4.25 |  4.50 |  4.75 |  5.00 |
| ntsm  | 20.00 | 17.78 | 15.56 | 13.33 | 11.11 |  8.89 |  6.67 |  4.44 |  2.22 |  0.00 | -2.22 | -4.44 | -6.67 | -8.89 |-11.11 |
| ntsa  |-18.00 |-19.00 |-20.00 |-21.00 |-22.00 |-23.00 |-24.00 |-25.00 |-26.00 |-27.00 |-28.00 |-29.00 |-30.00 |-31.00 |-32.00 |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

nb: dbtk == Dynamic_Bits_To_Keep; ntsm == Noise_Threshold_Shift_Minimum; ntsa == Noise_Threshold_Shift_Average.


Revised 55 problem sample bitrate table:
Code: [Select]
+---------+----------+----------+----------+
| v1.1.5c | default  | -l 15159 |    -t    |
+---------+----------+----------+----------+
| -q 10.0 | 654kbit/s| 647kbit/s| 623kbit/s|
+---------+----------+----------+----------+
| -q  9.5 | 641kbit/s| 633kbit/s| 612kbit/s|
| -q  9.0 | 628kbit/s| 619kbit/s| 600kbit/s|
| -q  8.5 | 614kbit/s| 605kbit/s| 588kbit/s|
| -q  8.0 | 599kbit/s| 591kbit/s| 577kbit/s|
+---------+----------+----------+----------+
| -q  7.5 | 584kbit/s| 576kbit/s| 565kbit/s|
+---------+----------+----------+----------+
| -q  7.0 | 570kbit/s| 561kbit/s| 553kbit/s|
| -q  6.5 | 555kbit/s| 546kbit/s| 542kbit/s|
| -q  6.0 | 540kbit/s| 531kbit/s| 530kbit/s|
| -q  5.5 | 525kbit/s| 516kbit/s| 518kbit/s|
+---------+----------+----------+----------+
| -q  5.0 | 510kbit/s| 501kbit/s| 506kbit/s|
+---------+----------+----------+----------+
| -q  4.5 | 493kbit/s| 485kbit/s| 495kbit/s|
| -q  4.0 | 477kbit/s| 469kbit/s| 483kbit/s|
| -q  3.5 | 463kbit/s| 456kbit/s| 472kbit/s|
| -q  3.0 | 450kbit/s| 443kbit/s| 461kbit/s|
+---------+----------+----------+----------+
| -q  2.5 | 427kbit/s| 422kbit/s| 441kbit/s|
+---------+----------+----------+----------+
| -q  2.0 | 416kbit/s| 411kbit/s| 431kbit/s|
| -q  1.5 | 389kbit/s| 384kbit/s| 421kbit/s|
| -q  1.0 | 365kbit/s| 362kbit/s| 411kbit/s|
| -q  0.5 | 344kbit/s| 341kbit/s| 401kbit/s|
+---------+----------+----------+----------+
| -q  0.0 | 325kbit/s| 322kbit/s| 391kbit/s|
+---------+----------+----------+----------+
| -q -0.5 | 306kbit/s| 303kbit/s| 381kbit/s|
| -q -1.0 | 289kbit/s| 287kbit/s| 372kbit/s|
| -q -1.5 | 274kbit/s| 271kbit/s| 363kbit/s|
| -q -2.0 | 259kbit/s| 257kbit/s| 354kbit/s|
| -q -2.5 | 247kbit/s| 245kbit/s| 346kbit/s|
| -q -3.0 | 236kbit/s| 234kbit/s| 337kbit/s|
| -q -3.5 | 226kbit/s| 225kbit/s| 329kbit/s|
| -q -4.0 | 218kbit/s| 217kbit/s| 322kbit/s|
+---------+----------+----------+----------+


Code: [Select]
55 Problem Sample Set
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|Version| Settings | FLAC -5  |--insane  |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.0.0b| default  | 780kbit/s| 655kbit/s| 582kbit/s| 503kbit/s| 417kbit/s| 330kbit/s|----------|----------|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.1.0c| default  | 780kbit/s| 654kbit/s| 583kbit/s| 508kbit/s| 425kbit/s| 321kbit/s|----------|----------|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.1.5c| default  | 780kbit/s| 654kbit/s| 585kbit/s| 510kbit/s| 427kbit/s| 325kbit/s| 259kbit/s| 218kbit/s|
|v1.1.5c| -l 15159 | 780kbit/s| 647kbit/s| 576kbit/s| 501kbit/s| 422kbit/s| 322kbit/s| 257kbit/s| 217kbit/s|
|v1.1.5c|    -t    | 780kbit/s| 623kbit/s| 565kbit/s| 506kbit/s| 441kbit/s| 391kbit/s| 354kbit/s| 322kbit/s|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+

10 Album Test Set
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|Version| Settings | FLAC -5  |--insane  |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.0.0b| default  | 854kbit/s| 626kbit/s| 539kbit/s| 452kbit/s| 365kbit/s| 295kbit/s|----------|----------|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.1.0c| default  | 854kbit/s| 632kbit/s| 548kbit/s| 463kbit/s| 376kbit/s| 285kbit/s|----------|----------|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.1.5c| default  | 854kbit/s| 627kbit/s| 544kbit/s| 460kbit/s| 376kbit/s| 288kbit/s| 230kbit/s| 200kbit/s|
|v1.1.5c| -l 15159 | 854kbit/s| 611kbit/s| 527kbit/s| 444kbit/s| 367kbit/s| 283kbit/s| 228kbit/s| 199kbit/s|
|v1.1.5c|    -t    | 854kbit/s| 582kbit/s| 514kbit/s| 450kbit/s| 385kbit/s| 341kbit/s| 310kbit/s| 283kbit/s|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-22 20:01:44
Thank you, Nick. I like it this way.

Average bitrate for my standard test set is 379 kbps now using -P --altpreset.
It was 380 kbps with version 1.1.5b, and I expected bitrate to go a tiny bit up not down.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-22 20:19:33
The noise_threshold_shift_minimum has gone up from 5.375 to 5.555 (more noise) while the dynamic_minimum_bits_to_keep has gone up from 3.15 to 3.40 (more bits to keep). Noise_threshold_shift_average remains the same.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-11-22 21:45:31
I see, and I like it all.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-11-22 21:57:39
Now, I think that there will be a pause of a number of days for bug-reports followed by the release of v1.2.0.
Title: lossyWAV 1.2.0 Development Thread
Post by: Skymmer on 2009-12-03 02:22:57
I don't know if situation which I'm going to describe have happened before but, honestly speaking, I'm lazy to read the whole (and no doubt) great topic 
The deal is that I found a piece of audio data which being proccessed with LossyWAV and then packed with FLAC gives larger files then without LossyWAV. In details. I have the audio image of Elve - Infinite Garden album. Here is the result table:

Code: [Select]
Original.wav                        702 775 292
FLAC -8                             265 453 324
LossyWAV -q 10\FLAC -8 -b 512       272 760 641
LossyWAV -q 10 -t\FLAC -8 -b 512    267 741 220

Versions used: LossyWav 1.1.5c, FLAC 1.2.1
Well, I suppose if I will publish this audio here it will be violation so, Nick, if you need it for investigations I can upload it and PM you the link.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-12-03 06:15:03
This does happen occasionally with some types of audio - lossyWAV doesn't remove many (if any) bits from the audio (especially at -q 10) and encoding the result with FLAC -b 512 is less efficient than default the blocksize (4096).
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-12-03 07:36:34
...
Code: [Select]
Original.wav                        702 775 292
FLAC -8                             265 453 324
LossyWAV -q 10\FLAC -8 -b 512       272 760 641
LossyWAV -q 10 -t\FLAC -8 -b 512    267 741 220

I guess it's the results from 3 different snippets of the track.
The last snippet is interesting showing that it can be quite vital to have the analysis' HF limit a little bit lower than 16 kHz.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-12-03 07:57:37
I think that they are all FLAC (or lossyFLAC) files created from the same WAV file.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-12-03 08:19:55
But there are three columns. What should they mean if not three different tracks or track snippets?
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-12-03 08:43:41
I think that those are 9-digit integers with the commas missing.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-12-03 08:51:49
I see. I think you are right.
And it would make sense because FLAC is very efficient in this case which is the typical situation where lossyWAV doesn't improve things.
Title: lossyWAV 1.2.0 Development Thread
Post by: Skymmer on 2009-12-03 09:53:12
This does happen occasionally with some types of audio - lossyWAV doesn't remove many (if any) bits from the audio (especially at -q 10) and encoding the result with FLAC -b 512 is less efficient than default the blocksize (4096).

Thanks. I wonder what's the theoretical probability of appearing of such file(s)?

But there are three columns. What should they mean if not three different tracks or track snippets?

Nick is right. There is only one column. Its the regional standard in my country to split the digit groups with 0xA0 which is looks like space. Comma used for integer and fractioanal part splitting. I'll take it into account to avoid confusion next time 
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-12-03 10:19:13
... I wonder what's the theoretical probability of appearing of such file(s)? ...

It can happen with higher lossyWAV quality levels when music contains only relatively 'simple' tones, that is, music originating from one or few instruments (including voice) only, or when the music is pretty quiet on average.
With music of that kind it may be worth trying whether FLAC alone isn't sufficiently efficient or even more efficient than lossyWAV preprocessed FLAC because of lossyWAV's blocksize restriction. Just in case this situation bothers you. But as efficiency difference in this case usually is small it's not necessary to do so, and bits removed by lossyWAV are next to nothing in this situation (otherwise it would be the lossyWAV version which is more efficient).

BTW with the --portable quality level such a situation happens next to never.
Title: lossyWAV 1.2.0 Development Thread
Post by: GeSomeone on 2009-12-03 15:24:08
Thanks. I wonder what's the theoretical probability of appearing of such file(s)?

I'd say, as a rule of thumb: if FLAC -8 gives you already close to the bit rate you'd expect of lossyFlac, then don't apply lossyWav at all. Those are the potential cases for size increase.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-12-03 20:58:51
I just tested v1.1.5c -P --altpreset with my usual problem samples Atem-lied, badvilbel, bibilolo, bruhns, dither_noise_test, furious, herding_calls, keys_1644ds, S37_Others_MartenotWaves_A_Essential, triange-2_1644ds, Under The Boardwalk (critical snippet of it), and everything was fine.
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-12-03 21:03:57
Thanks for that.

Title: lossyWAV 1.2.0 Development Thread
Post by: markanini on 2009-12-04 08:37:41
Sorry for asking a noobish question but how is lossywav different from wavpack hybrid lossy w/o correction file. And is lossywav more sofisticated/higher quality for in between lossy and lossless bitrates?
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-12-04 12:42:02
lossyWAV pros:

a) lossyWAV is a preprocessor before using an appropriate lossless encoder. This way you have a choice for the final codec, with FLAC, TAK and wavPack lossless as the most prominent ones.
a1) As you can use FLAC you get good hardware support so that you can play your music even on a series of mobile DAPs.
a2) If some day in the future you may want to use another favorite lossless .xxx-codec instead of say FLAC, you can losslessly reencode your .lossy.flac files to .lossy.xxx files.
b) lossyWAV has an explicit error control. It controls the number of least significant bits removed (that is the amount of added noise). The energy of the music is checked in a frequency dependent way, and the number of bits to remove are chosen in such a way that the added noise stays more or less (depending on quality level) below the signal in all of the frequency ranges controlled (which are pretty narrow) up to roughly 16 kHz.
This way for the --portable quality setting (yielding ~ 380 kbps on average) or better we can have a high degree of confidence that the results are transparent.
wavPack lossy/hybrid in contrary doesn't have a real error control. wavPack lossy essentially controls the precision with which the predictor values are encoded and choses this precision according to quality setting. Predictor error and total error are not necessarily close to each other, and audibility of these errors is another story.

wavPack lossy/hybrid pros:

a) wavPack lossy has the better quality when using relatively low quality settings (yielding ~ 300kbps on average or less). A dynamic noise shaping process has its part in this property.
b) wavPack lossy can also have the better quality with higher quality settings. The fact that there is no control of audible error gets less significant the higher the quality setting. At a quality setting yielding ~500 kbps or more on average wavPack lossy can be considered transparent even with problematic samples. With quality settings yielding say between 300 and 500 kbps quality usually is still great especially when above 300 kbps for quite an amount, but there is some uncertainty the higher the lower the bitrate.
Title: lossyWAV 1.2.0 Development Thread
Post by: pdq on 2009-12-04 13:03:20
Has there been any thought of adding correction file capability to lossy wav? I realize there are no lossless decoders that would directly make use of this file, but perhaps a post-processor to do the correction when you wanted to generate the original wav file?

Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-12-04 13:10:35
The capability to create correction files / merge lossy and correction files has been incorporated in lossyWAV since the initial release.
Title: lossyWAV 1.2.0 Development Thread
Post by: markanini on 2009-12-04 13:30:43
Thanks for the thorough explanation halb27. So I guess once noise shaping is implemented into lossyWAV it will have all quality benefits plus compatibility with many lossless codecs? Can't wait!

EDIT:I reread your post. Didnt really understand how the explicit error control worked. Do I understand correctly that there will be more noise in loud frequency ranges than soft frequency ranges? For example a section with a solo bass notes will have most noise added in the bass frequencies, and solo cymbals will have most noise added in treble frequencies? Assuming I've understood correctly this seems better in a scenario where eq and other processing would be applied post decode compared to ATH type noise shaping.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-12-04 14:03:46
Nick will be able to tell better, but I'm afraid dynamic noise shaping is something that doesn't go too well with lossyWAV.
I guess it would require the implementation of a rather sophisticated psy model (taking account of masking etc. - things like done by mp3 encoders) in order to have noise still under control.
With such a machinery more refined considerations about frequency dependent noise allowed can make lossyWAV more efficient in its own right.

Even with such a more sophisticated machinery we can't expect to get close to the bitrates of high quality mp3 settings (~250 kbps).

I think everybody who likes the lossyWAV way of a very-close-to-lossless procedure can be content with the bitrate of ~380 kbps on average we have with the extremely high quality --portable setting of lossyWAV.

Let's see what the future will bring, but I wouldn't expect too much.
Title: lossyWAV 1.2.0 Development Thread
Post by: halb27 on 2009-12-04 14:20:12
I reread your post. Didnt really understand how the explicit error control worked. Do I understand correctly that there will be more noise in loud frequency ranges than soft frequency ranges? ...

Roughly speaking error control is looking for the frequency range with lowest signal energy and takes care that added noise (by zeroing least significant bits) remains more or less below the signal energy in this range.
It works well with loud complex music (like most of popular music) having high energy in the entire spectrum (as added noise can be pretty high).
The procedure is (necessarily) inefficient for a sine wave, that is just one frequency in the whole spectrum. In this case there are a lot of frequency ranges without signal energy driving lossyWAV to keep added noise as zero or nearly zero, that is no or very few bits can be removed.
With simple tonal music originating from one or few instruments it's similar as probabilty for having a frequency range without signal energy is high.
Apart from energy spectrum of the music the absolute energy is relevant as well. Even when having a full spectrum for low energy music added noise must be low driving lossyWAV to keep all or nearly all of the bits.
Luckily these situations where lossyWAV can remove (nearly) no bits are more or less those situations where the final lossless codec works very efficiently. So in an overall sense the procedure lossyWAV + lossless codec remains more or less in the usual bitrate range.

The added noise comes directly from zeroing least significant bits. Without noise shaping there is no preference for specific frequency ranges of the noise.
With lossyWAV's noise shaping you can shift (part of) the noise to the 13+ kHz range. This has nothing to do with the frequency analysis of the signal. It just adresses the fact that we are less sensitive to noise in the very high frequency range.
With adaptive noise shaping at least in theory noise can be shifted to the frequency areas with high energy so that it is masked.
Apart from implementation difficulties there may be legal issues with this approach (it's not quite clear if such a noise shaping would touch a patent).
Title: lossyWAV 1.2.0 Development Thread
Post by: Nick.C on 2009-12-16 21:52:23
lossyWAV 1.2.0 attached to post #1 in this thread.