HydrogenAudio

Lossy Audio Compression => Other Lossy Codecs => Topic started by: Nick.C on 2008-07-14 12:34:34

Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-14 12:34:34
lossyWAV 1.1.0c is now available for download.
Code: [Select]
lossyWAV 1.1.0c, Copyright (C) 2007,2008 Nick Currie. Copyleft.

This program is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful,but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program.  If not, see <http://www.gnu.org/licenses/>.

Process Description:

lossyWAV adds white noise to the processed output. The amount of added noise is
based on analysis of the signal levels in the frequency range 20Hz to 16kHz.

If signals above the upper limiting frequency are at an even lower level, they
can be swamped by the added noise. This is usually inaudible, but the behaviour
can be changed by specifying a higher --limit (in the range 16kHz to 20kHz).

For many audio signals, there is little content at very high frequencies, and
forcing lossyWAV to keep the added noise level lower than the content at these
frequencies can increase the bitrate dramatically for no perceptible benefit.

Usage   : lossyWAV <input wav file> <options>

Example : lossyWAV musicfile.wav

Quality Options:

-I, --insane        highest quality output, suitable for transcoding;
-E, --extreme       high quality output, also suitable for transcoding;
-S, --standard      default quality output, considered to be transparent;
-P, --portable      good quality output for DAP use. Not considered to be fully
                    transparent, but considered fit for its intended purpose.

Standard Options:

-c, --check         check if WAV file has already been processed; default=off.
                    errorlevel=16 if already processed, 0 if not.
-C, --correction    write correction file for processed WAV file; default=off.
-f, --force         forcibly over-write output file if it exists; default=off.
-h, --help          display help.
-L, --longhelp      display extended help.
-M, --merge         merge existing lossy.wav and lwcdf.wav files.
-o, --outdir <t>    destination directory for the output file(s).
-v, --version       display the lossyWAV version number.

Advanced Options:

-                   if filename="-" then WAV input is taken from STDIN.
    --blockdist     show distribution of lowest significant bit of input
                    codec-blocks and bit-removed codec-blocks.
-D, --dither <n>    enable variable PDF dither of output; default=off;
                    0 = rectangular; 1 = triangular; 0.5 = half way between.
-l, --limit <n>     set upper frequency limit to be used in analyses to n Hz;
                    (16000<=n<=20000), default = 16000.
    --linkchannels  Revert to original single bits-to-remove value for all
                    channels rather than channel dependent bits-to-remove.
-q, --quality <n>   quality preset (10=highest quality, 0=lowest bitrate;
                    default = --standard = 5; --insane = 10; --extreme = 7.5;
                    --portable = 2.5)
    --sampledist    show distribution of lowest significant bit of input
                    samples and bit-removed samples.
    --scale <n>     scaling factor from WaveGain, etc; (0.0<n<=8.0),default=1.0
-s, --shaping <n>   enable fixed noise shaping; (0.00<=n<=1.00); default=q/10;
                    0.00 = off, 1.00 = 100% effectiveness, 0.50 = 50%, etc.
    --stdinname <t> pseudo filename to use when input from STDIN.
    --stdout        write processed WAV output to STDOUT.
-w, --writetolog    create (or append to) lossyWAV.log in the output directory.

System Options:

-B, --below         set process priority to below normal.
-d, --detail        enable detailed bits-to-remove information output mode
    --low           set process priority to low.
-n, --nowarnings    suppress lossyWAV warnings.
-Q, --quiet         significantly reduce screen output.
    --silent        no screen output.

Special thanks:

David Robinson      for the publication of his lossyFLAC method, guidance, and
                    the motivation to implement the method as lossyWAV.
Horst Albrecht      for ABX testing, valuable support in tuning the internal
                    presets, constructive criticism and all the feedback.
Sebastian Gesemann  for the noise shaping coefficients and help in using them
                    in the lossyWAV noise shaping implementation.
Don Cross           for the Complex-FFT algorithm used.
[/size]Link to the hydrogenaudio wiki article (http://wiki.hydrogenaudio.org/index.php?title=LossyWAV)

Suggested foobar2000 converter setup:

lossyFLAC:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.flac
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - -b 512 -5 -f -o%d
Format is: lossless or hybrid
Highest BPS mode supported: 24
lossyTAK:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24
lossyWV:
Code: [Select]
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.wv
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

Enclose the element of the path containing spaces within double quotation marks ("), e.g. C:\"Program Files"\directory_where_executable_is\executable_name. This is a Windows limitation.
N.B.: lossyWAV 1.2.0 development thread here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=65499&view=findpost&p=584559)
Title: lossyWAV 1.1.0 released.
Post by: botface on 2008-07-14 13:47:22
lossyWAV 1.1.0 is now available for download.

Well done, Nick. Another hassle-free release
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-14 19:32:30
Well done, Nick. Another hassle-free release
Many thanks - glad to be of service!
Title: lossyWAV 1.1.0 released.
Post by: Axon on 2008-07-14 19:55:34
Yayayay! Thanks a lot Nick.

Has there been a comparison of lossless encoder performance when combined with lossyWAV?
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-14 20:23:08
Has there been a comparison of lossless encoder performance when combined with lossyWAV?
Erm, no - but it would not be too difficult to set one up.

I would imagine Lame, ogg vorbis and neroAacEnc would be the targets at varying quality levels spanning the range for each, from both lossless FLAC and lossyFLAC (--portable, --standard, --extreme and --insane presets).

So, maybe 3 codecs x 6 levels per codec x (4 quality presets of lossyFLAC + lossless FLAC) = 90 average bitrates for an agreed test set of tracks / samples.

Comments / suggestions please! [edit] Can't count.... [/edit]
Title: lossyWAV 1.1.0 released.
Post by: carpman on 2008-07-14 20:38:48
Has there been a comparison of lossless encoder performance when combined with lossyWAV?

Though lossy would also be interesting, perhaps.

C.
Title: lossyWAV 1.1.0 released.
Post by: Axon on 2008-07-14 20:50:38
Yeah, I'm mainly curious as to if comparing lossyTAK/lossyFLAC would yield an increase in, um, performance increase relative to a comparison of TAK/FLAC. Certainly some encoders seem to handle white noise better than others, so by reducing the bit depth in lossyWAV's fashion, that impact could be reduced.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-14 20:54:24
Has there been a comparison of lossless encoder performance when combined with lossyWAV?
Though lossy would also be interesting, perhaps.

C.
Apparently I can't read and can't count.... 

I'll get to work on the lossyFLAC / lossyTAK / lossyWV comparison using recommended FLAC / TAK / Wavpack settings above at -q 0, --portable, --standard, --extreme and --insane.

[edit] After a quick batch file or two and making sure that my source FLAC set was encoded at -5. Lossless FLAC / TAK / Wavpack all encoded without using the 512 sample blocksize parameters. So, for my 55 problem sample set:
Code: [Select]
             +------------+------------+------------+
             |    TAK     |    FLAC    |  Wavpack   |
+------------+------------+------------+------------+
| lossless   | 749 kbit/s | 787 kbit/s | 772 kbit/s |
+------------+------------+------------+------------+
| --insane   | 634 kbit/s | 654 kbit/s | 661 kbit/s |
| --extreme  | 564 kbit/s | 583 kbit/s | 599 kbit/s |
| --standard | 488 kbit/s | 508 kbit/s | 529 kbit/s |
| --portable | 405 kbit/s | 425 kbit/s | 444 kbit/s |
| -q 0       | 299 kbit/s | 321 kbit/s | 342 kbit/s |
+------------+------------+------------+------------+
Title: lossyWAV 1.1.0 released.
Post by: sauvage78 on 2008-07-14 21:21:38
thks Nick, now that noise shaping was done in 1.1.0, is there still room for improvement (lower bitrate) in 1.2.0, or are you reaching the end of what you wanted to do with lossyWAV ? I mean now the todo list for 1.2.0 seems rather small from my non-technical point of view ... shouldn't 1.2.0 be a 1.1.1 now ? will there be just cosmetic changes & bugfix from now on ???
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-14 21:38:07
thks Nick, now that noise shaping was done in 1.1.0, is there still room for improvement (lower bitrate) in 1.2.0, or are you reaching the end of what you wanted to do with lossyWAV ? I mean now the todo list for 1.2.0 seems rather small from my non-technical point of view ... shouldn't 1.2.0 be a 1.1.1 now ? will there be just cosmetic changes & bugfix from now on ???
My intention is to understand and implement SebastianG's new noise shaping method, but for that I will also have to introduce / find a PSY model of some kind.

I would hope that by using the new noise shaping method some additional bits can be removed for the same apparent quality level of output, thereby further reducing the bitrate. However, this could be some time in happening as my basic understanding of the topic has to improve somewhat before I start implementation.

Unlike last time, I will not be starting the development of 1.2.0 a day or two after the release of the new version.

In a lot of ways, I've achieved what I set out to do (about a year ago) in accordance with David's original method (albeit with a few tweaks here and there).

I have an idea for a cosmetic change for the --detail parameter and may make modifications so that its output can be written to the logfile, although as more than one instance of lossyWAV could be running I want to keep the time that the logfile is open for output to an absolute minimum.

Cosmetically, I am happier with 1.1.0 than I was with 1.0.0b, but that has only come about by changing the way that the screen / log output is handled.
Title: lossyWAV 1.1.0 released.
Post by: Agent69 on 2008-07-16 13:44:47
Out of curiousity, and with all due respect to its developer, is LossyWAV just for use with DAPs? Is there any reason to use it when disk space is not an issue?
Title: lossyWAV 1.1.0 released.
Post by: carpman on 2008-07-16 14:25:33
Your question assumes that disk space is never an issue on HDDs.
I use lossyWAV but don't have a DAP.
Also it could be used to reduce bandwidth issues in terms of selling high quality transcodable files online (I have a big problem with buying audio that's trapped in a particular format, i.e MP3).
If you can save space without any loss in audible quality it seems sensible to do so. Space = money, so why not save it where you can, even if money is not an issue.

C.
Title: lossyWAV 1.1.0 released.
Post by: Synthetic Soul on 2008-07-16 14:42:11
Yeah, I'm mainly curious as to if comparing lossyTAK/lossyFLAC would yield an increase in, um, performance increase relative to a comparison of TAK/FLAC. Certainly some encoders seem to handle white noise better than others, so by reducing the bit depth in lossyWAV's fashion, that impact could be reduced.

Using Nick's figures above, it seems it is much of a muchness, although WavPack seems to benefit a little less (as can be seen by FLAC actually producing smaller bitrates with most settings:

Code: [Select]
             +------------------+------------------+------------------+
             |    TAK           |    FLAC          |  Wavpack         |
+------------+------------------+------------------+------------------+
| lossless   | 749 kbit/s       | 787 kbit/s        | 772 kbit/s      |
+------------+------------------+------------------+------------------+
| --insane   | 634 kbit/s (85%) | 654 kbit/s (83%) | 661 kbit/s (86%) |
| --extreme  | 564 kbit/s (75%) | 583 kbit/s (74%) | 599 kbit/s (78%) |
| --standard | 488 kbit/s (65%) | 508 kbit/s (65%) | 529 kbit/s (69%) |
| --portable | 405 kbit/s (54%) | 425 kbit/s (54%) | 444 kbit/s (58%) |
| -q 0       | 299 kbit/s (40%) | 321 kbit/s (41%) | 342 kbit/s (44%) |
+------------+------------------+------------------+------------------+


IIRC Thomas mentioned something about a specific LossyWAV setting coming for TAK.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-16 16:30:08
IIRC Thomas mentioned something about a specific LossyWAV setting coming for TAK.
He did indeed !
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-07-16 17:10:14
Out of curiousity, and with all due respect to its developer, is LossyWAV just for use with DAPs? Is there any reason to use it when disk space is not an issue?
Yes, most modern CDs don't deserve lossless (they're such bad quality) while requiring very high lossless bitrates. LossyWAV tames them with little to no side effects.

I can always find something better to do with the HDD space. A few TBs don't go far when you have video and photos, plus a lot of audio to back up.

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: Axon on 2008-07-16 19:02:31
To say nothing about vinyl. Gawd! My needledrop FLACs are shrinking by 45% at --standard!

If there ever was a demonstration of the reduced dynamic range of vinyl, lossyWAV has to be it. But clearly, there's a silver lining to it.

Thanks for the new comparisons Synthetic. It's very interesting to see that TAK actually pulls ahead of FLAC by an additional 2% at -q0, although the gain is eliminated at --standard.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-16 19:11:35
Out of curiousity, and with all due respect to its developer, is LossyWAV just for use with DAPs? Is there any reason to use it when disk space is not an issue?
Yes, most modern CDs don't deserve lossless (they're such bad quality) while requiring very high lossless bitrates. LossyWAV tames them with little to no side effects.

I can always find something better to do with the HDD space. A few TBs don't go far when you have video and photos, plus a lot of audio to back up.

Cheers,
David.
As David said (and lossyWAV is an implementation of his lossyFLAC proposal (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=55522&view=findpost&p=498179)) there are little or no side effects - even lossyWAV --insane|FLAC yields a 15% (or more) saving over lossless FLAC.

Also, I am also noticing that newer albums seem to compress less well in FLAC - but compress as well as older material after both have been processed with lossyWAV.

[edit] I use lossyFLAC solely with CorePlayer on my iPAQ h2210 with 32GB CF card and 4GB SD card. My FLAC archive is safe on my server. As an aside, I processed 262 CD images (252h32m38s) from FLAC on my server to lossyWAV -q 1.25|FLAC -5 in 3h05m last night - so transcoding for immediate use is really quite feasible (Intel C2D @ 3.0GHz, 4GB RAM). My iPAQ would play lossless FLAC of course, but my hearing is not good enough to determine the difference and circa 335 kbit/s for lossyWAV -q 1.25|FLAC -5 is palatable from a size viewpoint. I believe that lossyFLAC requires less processor effort when decoding than, say, MP3 - so I should get longer battery life (although I installed a 3600mAh battery just to be sure.... ). [/edit]
Title: lossyWAV 1.1.0 released.
Post by: Brent on 2008-07-16 20:36:21
I searched the entire 1.1.0 development thread, but I couldn't get the answers I'm looking for: has anyone ever succesfully ABX'ed --portable? Or, if there just is no direct answer to that question, what's the lowest quality setting anyone has ever succesfully ABX'ed? I'm curious how paranoid I should be to use a higher setting (I myself am quite happy with --portable, but I obviously havn't had time to check my entire library for errors, let alone ABX'ing).
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-16 20:45:45
That's a good question - I believe that Mardel or Sauvauge78 managed to ABX at -q 2 with specific samples using an older beta of lossyWAV, which prompted a bit of tightening of internal preset parameters. However, as --portable == -q 2.5, I think it includes a suitable safety margin over -q 1 - but it's all down to your ears / equipment / listening environment really....

As my sig says, I am happy with -q 1.25 for my purposes (listening to it right now on my PC) - however my lossyFLAC collection is my lossy copy - I keep the lossless FLAC safely tucked away.

halb27 is, I believe, best placed to advise on archive quality lossyFLAC - his setting is --extreme for archiving.
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-07-16 21:05:52
... has anyone ever succesfully ABX'ed --portable? ...

I just finished another listening test as we have a new final release and there were some changes since my last test. Moreover I have some CDs to encode which I didn't want to encode before this final release.

I decided to use the --portable quality as I don't remember having ever tested it since noise shaping is involved with it as a default. I didn't read your post before my test.

I used my usual set of potential problems Atem-lied, harp40_1, bruhns, badvilbel, trumpet, furious, keys_1644ds, triangle-2_1644ds, bibilolo, herding_calls, S37_OTHERS_MartenotWaves_A, dither_noise_test, Fiocco, Under The Boardwalk, utb, 00000_00595ms, Livin_In_The_Future, eig_essence, castanets.

I was not able to abx a problem. If there was a suspicion against transparency for me furious is a candidate for this, but it's beyond my ability to abx it (and I'm not sure if my weak suspicion is unconscious placebo cause using foobar I've seen furious' bitrate is relatively low compared to that of other problem samples).

I also tested a series of full length tracks and everything was fine to me.

To finish it up I encoded my regular music test set using --extreme and listened to the error files. In many - if not most situations - the error is inaudible even with no music to mask the noise. When noise is audible it's very low volume. I also looked at the added noise's spectrum and everything works as expected: noise is in the very high frequency region most of the time and seldom goes down to 10 kHz (judging from Nero wave editor display).

So everything's fine, and even --portable provides excellent results for my ears.

Thank you, Nick, for your great work.


As for your question, Brent: lossyWAV has had a long life as far as the various versions are concerned. With regard to the versions since the introduction of the quality naming scheme like --portable I'm pretty sure nobody abxed --portable. Moreover we can't but talk about nowadays quality of --portable since the default inclusion of noise shaping. That's not a long time ago.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-16 21:25:24
Thank you, Nick, for your great work.
A pleasure, Sir! Said before but worth repeating - lossyWAV would not have happened without David's proposal publication, Matlab script and guidance (& restraint ), our early collaborative efforts and all of your ABX testing (if the output was good enough to require to be ABX'ed ).
Title: lossyWAV 1.1.0 released.
Post by: carpman on 2008-07-17 01:45:42
IIRC Thomas mentioned something about a specific LossyWAV setting coming for TAK.
He did indeed !

Just out of interest, did he mention an approx. timescale for this?

C.
Title: lossyWAV 1.1.0 released.
Post by: Axon on 2008-07-17 02:59:02
Now this is cute. Just for grins I tried lossyWAV --standard on a 24/44 needledrop I had. The bitrate dropped from 1504kbps compressed to 506kbps - a 65% reduction in file size? -q1.25 was 403kbps.

That makes a lot of sense though, most notably because my needledrops are not RIAA equalized, so the amplifier noise is largely flat.
Title: lossyWAV 1.1.0 released.
Post by: Brent on 2008-07-17 10:17:17

... has anyone ever succesfully ABX'ed --portable? ...

As for your question, Brent: lossyWAV has had a long life as far as the various versions are concerned. With regard to the versions since the introduction of the quality naming scheme like --portable I'm pretty sure nobody abxed --portable. Moreover we can't but talk about nowadays quality of --portable since the default inclusion of noise shaping. That's not a long time ago.

As a lowly enduser not dedicated enough to evaluate every -q setting, the reason I enquired about --portable was that I supposed there was some rationale behind it being -q2.5. --standard is q5 and the commandline utility informs me that it is considered to be transparent, but from the looks of it --portable is transparent too. Because I read about a lot of testing in the q0-q2 range, I wondered wat exactly the reason could be to go any higher than --portable, because most reported no succes in ABX-ing in the topend of that range.

So, thank you very much for your results. And, if I may, what exactly is your reason to go with --extreme for your noise test? Are you considering moving your lib to that setting, and you've picked it over lower q settings just to be on the safe side?
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-07-17 10:27:21
.. what exactly is your reason to go with --extreme for your noise test?

For finding out whether --portable is transparent to me I did the listening test.
For testing the machinery not to have a bug I decided to listen to the error noise and look at it's spectrum.
I did this with --extreme on one hand because this is my usual encoding setting (I don't keep a losslesss archive any more, so I want extremely high quality with my lossyWAV archive). On the other hand --extreme is necessary to see most of the noise go to the extreme end of the spectrum. You can also expect only from a very high quality setting to hear only low volume noise or no noise at all when there's no masking by the music. This is totally irrelevant for judging about transparency (that's what ABX listening tests are for), I just wanted to make sure noise gets low volume in an absolute sense with an extremely high quality setting. Just a more formal, technical test whether everything's fine.
Title: lossyWAV 1.1.0 released.
Post by: shadowking on 2008-07-17 10:36:26
i think -standard quality losswav is aiming for transparency for every sample and every listener . Lower setting like Q3 or 2.5 or lower might very well be transparent for the vast majority of cases. If keeping the bitrate down is a concern,  one could use a lower quality like 2.5 and still get very high quality and enough for that transcode to layer 3 if needed.

The other advantage of quality 6 / 7 is that you could make your lossywav archive smaller if needed like transcode it to a lower quality setting with great safety. Example: Say I have a lossywav Q6 archive at home and I need high quality transcodable on-demand files for my internet radio station which is in another location. I could transcode Q6 archive to quality 3 and transcode that to a streaming format like layer 3 or he-aac. Its likely to work well.

Another scenario is sound processing and editing. I could edit Q6 archive dozens of times of my music projects. Another is a high quality backup - transcode Q6 to Q3 as my everyday archive keeping Q6 as the master on some other offline harddrive. the space used will be nearly the same as normal lossless in such a situation but there is the advantage of splitting archives across drives. These overkill setting are lossless replacements. In theory an online music store should be able to use Q6 / 7 to transcode to lossy with great confidence.
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-07-17 12:21:41
Another scenario is sound processing and editing. I could edit Q6 archive dozens of times of my music projects. Another is a high quality backup - transcode Q6 to Q3 as my everyday archive keeping Q6 as the master on some other offline harddrive. the space used will be nearly the same as normal lossless in such a situation but there is the advantage of splitting archives across drives. These overkill setting are lossless replacements. In theory an online music store should be able to use Q6 / 7 to transcode to lossy with great confidence.
"Should" is the important word in all those statements. This needs some testing and ABXing before we can claim it comfortably. I did make a start, but got waylaid.

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: botface on 2008-07-17 14:22:03
To say nothing about vinyl. Gawd! My needledrop FLACs are shrinking by 45% at --standard!

If there ever was a demonstration of the reduced dynamic range of vinyl, lossyWAV has to be it.

I'm not sure that's a valid conclusion. 45% is about right for FLAC:LossyFLAC irrespective of the source. I did some tests a few months ago when I first came accross LossyWav. I didn't keep most of the results but I do still have one set of test data left. This was the first side of Dire Straits 1st LP (or first 5 tracks if you're looking at the CD).

For CD the figures were:
The uncompressed WAVs were 200 Mb
After FLAC processing (can't remember the level I used) it became 111Mb
Processing Via LossyWav --Standard/FLAC 5 it became 63.2Mb.
That's a reduction of around 43% compared with the FLAC'd data and 68.4%(ish) against the WAVs

For Vinyl the figures were:
The uncompressed WAVs (@16/44.1) were 194.4 Mb (presumably slightly less than CD due to speed inaccuracy of the turntable)
After FLAC processing (again I can't remember the level I used) it became 109Mb
Processing Via LossyWav --Standard/FLAC 5 it became 62.6Mb.
That's a reduction of around 43% compared with the FLAC'd data and 67.8%(ish) against the WAVs

I'd say they're pretty similar.

I also checked the same tracks recorded from vinyl at 24/48 to see what happened. The resultant LossyWav --Standard/FLAC 5 files came out at 64.7Mb. That's about 79% smaller than the original WAVs (which were 315.5Mb) but still bigger than the same files at 16/44.1 albeit by only a couple of Mb. However, I interpreted it to mean that recording from vinyl at 24/48 was worthwhile as LossWav was obviously finding some of the extra data to be worth keeping, although it may only be the potential extra data in the 22 - 24 kHz range, while the resulting files were only marginally bigger than the 16/44.1 ones.

Obviously one can't draw any hard conclusions from a few tracks but at the time I did get similar figures  with several other albums in different genres.
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-07-17 14:59:24
If you take a 44.1kHz file and simply resample it to 48kHz, I think you'll see the bitrate go up - losslessFLAC or lossyFLAC. There's no mechanism in either to directly recover the redundancy.

That has some relationship to your experiment, though not an exact one.


As you say though, it's interesting the difference was so small. It's not just the vinyl noise floor that lossyWAV is quantising during the music itself.

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: botface on 2008-07-17 16:10:39
If you take a 44.1kHz file and simply resample it to 48kHz, I think you'll see the bitrate go up - losslessFLAC or lossyFLAC. There's no mechanism in either to directly recover the redundancy.

That has some relationship to your experiment, though not an exact one.


As you say though, it's interesting the difference was so small. It's not just the vinyl noise floor that lossyWAV is quantising during the music itself.

Cheers,
David.

Sorry, I don't understand your final paragraph - presumably my lack of understanding on the subject.

Using 24/48 will result in over 60% more data being produced than 16/44.1. I ended up with about 3% extra data after Lossywav and FLAC had processed it. What surprised me was that there was any extra data left at all and that LossyWav didn't "throw away" all of the extra as I was assuming that it would all be below the noise floor. As I say, I took it to mean that there was some useful data in that extra 60%
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-07-17 17:15:19
Sorry - I didn't write that very well.

What I meant to say is that 24 or 16 probably makes little difference in this case, whereas 44.1 vs 48 will.

You could convert the 24/48 to 16/48 and then throw it at lossyWAV+FLAC and see if there's any difference at all. You might prove my guess to be completely wrong.

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: raygrote on 2008-07-17 19:50:43
Lossy Wav really sounds very interesting. I've got a few questions about it.
1. How do I use Lossy Wav with Foober, or if there isn't a way, is their some kind of frontend for it?
2. If I compress a wav to flac, then run lossy flac, will the resulting file be much smaller than just running lossy wav on the wav file? I assume it wouldn't make too much difference, since to my understanding all lossy codecs use lossless compression in some way. Any info please?
Thanks.
Title: lossyWAV 1.1.0 released.
Post by: botface on 2008-07-17 20:12:59
What I meant to say is that 24 or 16 probably makes little difference in this case, whereas 44.1 vs 48 will.

You could convert the 24/48 to 16/48 and then throw it at lossyWAV+FLAC and see if there's any difference at all. You might prove my guess to be completely wrong.

Cheers,
David.

Good idea. Interesting result.

I converted to 16/48 using Cooledit. I did it twice. 1st time I used triangular dither and no noise shaping (I didn't want to affect the high frequencies too much). The 2nd time I used no dither or noise shaping. In both cases I ran the resulting file through Lossywav --standard/FLAC -5. In both cases the file ended up at 64.36 Mb.

So, it would seem that most, but not all, of the increase in file size is due to hf content above 22050kHz - assuming my choice of dither/noise shaping was appropriate.

I guess that a much wider choice of material and larger numbers of files would be needed before any real conclusions could be drawn but as I said; interesting
Title: lossyWAV 1.1.0 released.
Post by: carpman on 2008-07-17 20:23:44
1. How do I use Lossy Wav with Foober, or if there isn't a way, is their some kind of frontend for it?
2. If I compress a wav to flac, then run lossy flac, will the resulting file be much smaller than just running lossy wav on the wav file?

For 1 and 2 see: http://wiki.hydrogenaudio.org/index.php?title=LossyWAV (http://wiki.hydrogenaudio.org/index.php?title=LossyWAV)
LossyWAV won't make the WAV file smaller, it's a pre-processor for lossless encoders, it processes the WAV file to make it easier for FLAC et al to compress it. The WAV pre-processing is lossy; the subsequent lossless encoding (i.e. to FLAC or TAK) is lossless.

C.

[EDIT: Typos ; EDIT2: Clarity ; EDIT3: Clarification -- OMG!!!]
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-17 20:32:23
^^ What he said....
Title: lossyWAV 1.1.0 released.
Post by: carpman on 2008-07-17 20:37:00
Don't listen to me, just read the wiki; it's clearer and better written. 

C.
Title: lossyWAV 1.1.0 released.
Post by: raygrote on 2008-07-17 21:38:06
Thanks for info.
It still doesn't work with Foober, though. I'm using Foober 0.9.5 and Lossy Wav 1.1.0. I just copied and pasted the commandlines posted on the wiki for flac files, and adjusted everything the way they said, all I did was change the paths to the lossywav and flac files. I set encoder to c:\windows\system32\cmd.exe, extention to lossy.flac, format is lossless )or hybrid), and max bitrate supported 24.
The commandline I enter is:
/d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o%d
And the error I get is
"Source: "E:\data files\audio\abx1.wav"
  An error occured while writing to file (The encoder has terminated prematurely with code 1; please re-check parameters) : "E:\data files\audio\abx1.lossy.flac"
Additional information:
Encoder stream format: 44100Hz / 2ch / 16bps
Command line: "C:\WINDOWS\SYSTEM32\cmd.exe" /d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o"abx1.lossy.flac"
Working folder: E:\data files\audio\

  Conversion failed: The encoder has terminated prematurely with code 1; please re-check parameters"


I just know I did something wrong.
Title: lossyWAV 1.1.0 released.
Post by: carpman on 2008-07-17 21:47:04
I set encoder to c:\windows\system32\cmd.exe, extention to lossy.flac, format is lossless )or hybrid), and max bitrate supported 24.

Well that's all correct.

The commandline I enter is:
/d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o%d

Above I've highlighted in red what may be a problem. Try it with a space between -o%d, so it's:
-o %d

Or, paste this into your parameters:
Code: [Select]
/d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o %d

Try that first.

C.

[EDIT1] By the way Nick, if that is the problem the wiki needs changing as that's also in the wiki setting for FLAC.
[EDIT2] Just ran mine with -o%d (i.e. without the space) on foobar 9.4.3 and also 9.5.4 and it worked okay. I don't have any spaces (thus e:\"data files"\etc\etc ) in my encoders' addresses. So I'm out of ideas.
[EDIT3] Just ran it with identical parameters including no space with -o%d and with an address with spaces enclosed in " " and it worked fine.
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-07-17 23:02:14
... /d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o%d ...

Guess it's the "data files" which causes the problems, or the space between the | and the flac path.
To find out I'd (temporarily?) use a lossywav/flac path like: e:\codecs\lossywav.
Title: lossyWAV 1.1.0 released.
Post by: sauvage78 on 2008-07-17 23:59:29
-o%d is ok, --stdout| e: maybe is not ... try --stdout|e: without space
Title: lossyWAV 1.1.0 released.
Post by: Dynamic on 2008-07-18 00:32:38
So, it would seem that most, but not all, of the increase in file size is due to hf content above 22050kHz - assuming my choice of dither/noise shaping was appropriate.


Thanks to the way lossless encoders work, it's almost certainly down to the increased number of samples, because lossyWAV should ignore extra content above 16 kHz (and in any case, good sample rate conversion shouldn't add significant HF content not in the original). The fact there's no difference in bitrate between the two dither types when processed by lossyWAV is confirmation that your dither was below the original noise floor.

There are almost 9% more samples per second at 48kHz than at 44.1kHz, but with limited frequency response, it probably results in lower prediction error per sample (the error is what has to be stored by a lossless encoder), though this is not compensating fully for the 9% increase in number of samples per second.
Title: lossyWAV 1.1.0 released.
Post by: raygrote on 2008-07-18 00:49:54
I tried all suggestions, nothing is working. I'm starting to think this may have something to do with cmd.exe? Later I'll try moving the codecs to the directory used in the wiki for the heck of it and see what happens.
Perhaps CMD.exe has problems with external hard drives?
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-18 08:23:53
I tried all suggestions, nothing is working. I'm starting to think this may have something to do with cmd.exe? Later I'll try moving the codecs to the directory used in the wiki for the heck of it and see what happens.
Perhaps CMD.exe has problems with external hard drives?
I have noticed that if I switch my server on and try to convert files with foobar2000 immediately I hear it finish its boot then the foobar2000 conversion will fail. However, if I use windows explorer to "look at" the root directory of the external drive where the audio files are first then the foobar2000 conversion will work perfectly.
Title: lossyWAV 1.1.0 released.
Post by: botface on 2008-07-18 09:06:45
I tried all suggestions, nothing is working. I'm starting to think this may have something to do with cmd.exe? Later I'll try moving the codecs to the directory used in the wiki for the heck of it and see what happens.
Perhaps CMD.exe has problems with external hard drives?

I had a few problems when I had spaces in the directory, quotes or no quotes. So, I moved LossyWav and FLAC to a directory with no saces in the name (C:\auydio\lossywav) and it's worked perfectly ever since. In case it's of any interest here's my command line :
Parameters: /d /c c:\audio\lossywav\lossywav - -q 5 --silent --low --stdout|c:\audio\lossywav\flac - -b 512 -5 -o%d

(Yes, I'm still using the old "q" paramter instead of --standard).

I guess this shows how useful it would be if someone could knock up a GUI front end
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-07-18 10:49:32
There are almost 9% more samples per second at 48kHz than at 44.1kHz, but with limited frequency response, it probably results in lower prediction error per sample (the error is what has to be stored by a lossless encoder), though this is not compensating fully for the 9% increase in number of samples per second.
That's right. Even if you resample 44.1kHz to 48kHz (so there's no increase in the actual information - the extra frequency range is completely empty) the lossless bitrate will still go up. Most lossless codecs simply aren't designed to make use of this kind of redundancy (I think TAK might though).

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: Synthetic Soul on 2008-07-18 11:45:18
Using Nick's figures above, it seems it is much of a muchness, although WavPack seems to benefit a little less (as can be seen by FLAC actually producing smaller bitrates with most settings):

Code: [Select]
             +------------------+------------------+------------------+
             |    TAK           |    FLAC          |  Wavpack         |
+------------+------------------+------------------+------------------+
| lossless   | 749 kbit/s       | 787 kbit/s        | 772 kbit/s      |
+------------+------------------+------------------+------------------+
| --insane   | 634 kbit/s (85%) | 654 kbit/s (83%) | 661 kbit/s (86%) |
| --extreme  | 564 kbit/s (75%) | 583 kbit/s (74%) | 599 kbit/s (78%) |
| --standard | 488 kbit/s (65%) | 508 kbit/s (65%) | 529 kbit/s (69%) |
| --portable | 405 kbit/s (54%) | 425 kbit/s (54%) | 444 kbit/s (58%) |
| -q 0       | 299 kbit/s (40%) | 321 kbit/s (41%) | 342 kbit/s (44%) |
+------------+------------------+------------------+------------------+

In addition, here's some figures for the corpus (http://www.synthetic-soul.co.uk/comparison/lossless/corpus.asp) that I use for my lossless comparison:

Code: [Select]
              TAK               FLAC              Wavpack
================================================================
lossless      917 kbps          941 kbps          938 kbps
================================================================
--insane      629 kbps (69%)    639 kbps (68%)    661 kbps (70%)
--extreme     538 kbps (59%)    548 kbps (58%)    574 kbps (61%)
--standard    448 kbps (49%)    459 kbps (49%)    485 kbps (52%)
--portable    362 kbps (39%)    375 kbps (40%)    394 kbps (42%)

As per Nick's results, the difference between codecs is minimal; however my corpus (http://www.synthetic-soul.co.uk/comparison/lossless/corpus.asp) benefits a lot more from LossyWAV preprocessing.

Once again, thank you all for your hard work.
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-07-18 12:33:03
... however my corpus (http://www.synthetic-soul.co.uk/comparison/lossless/corpus.asp) benefits a lot more from LossyWAV preprocessing. ...

The results for your corpus are pretty similar to the results of my small 12 full length regular track test set, and these are very similar to those of Nick's many-albums test set.
If there's one thing I don't like with Nick's great work it's his preferred usage of his test set consisting of problem snippets for the biggest part of it which I'm afraid leads to a wrong impression to most people. 
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-18 18:43:41
If there's one thing I don't like with Nick's great work it's his preferred usage of his test set consisting of problem snippets for the biggest part of it which I'm afraid leads to a wrong impression to most people. 
As we're past the second full release, I'll produce a similar table to that above using my 10 album test set. The reason I stick to the 55 problem sample set is that it is (relatively, as I have added a few samples to it over the last six months) constant and relative changes in bitrates at quality presets can be compared. No need to stick to that now - I'll publish some bitrates tonight.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-19 20:30:47
If there's one thing I don't like with Nick's great work it's his preferred usage of his test set consisting of problem snippets for the biggest part of it which I'm afraid leads to a wrong impression to most people. 
As requested - results for my 10 album test set:
Code: [Select]
             +------------------+------------------+------------------+
             |       TAK        |       FLAC       |     Wavpack      |
+------------+------------------+------------------+------------------+
| lossless   | 820 kbit/s       | 854 kbit/s       | 852 kbit/s       |
+------------+------------------+------------------+------------------+
| --insane   | 615 kbit/s (75%) | 632 kbit/s (74%) | 641 kbit/s (75%) |
| --extreme  | 532 kbit/s (65%) | 548 kbit/s (64%) | 563 kbit/s (66%) |
| --standard | 447 kbit/s (55%) | 463 kbit/s (54%) | 481 kbit/s (56%) |
| --portable | 359 kbit/s (44%) | 376 kbit/s (44%) | 390 kbit/s (46%) |
| -q 0       | 266 kbit/s (32%) | 285 kbit/s (33%) | 296 kbit/s (35%) |
+------------+------------------+------------------+------------------+
Wiki article indicative bitrate table amended to reflect these results.
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-07-19 21:57:45
If there's one thing I don't like with Nick's great work it's his preferred usage of his test set consisting of problem snippets for the biggest part of it which I'm afraid leads to a wrong impression to most people. 
As requested - results for my 10 album test set:
Code: [Select]
             +------------------+------------------+------------------+
             |       TAK        |       FLAC       |     Wavpack      |
+------------+------------------+------------------+------------------+
| lossless   | 820 kbit/s       | 854 kbit/s       | 852 kbit/s       |
+------------+------------------+------------------+------------------+
| --insane   | 615 kbit/s (75%) | 632 kbit/s (74%) | 641 kbit/s (75%) |
| --extreme  | 532 kbit/s (65%) | 548 kbit/s (64%) | 563 kbit/s (66%) |
| --standard | 447 kbit/s (55%) | 463 kbit/s (54%) | 481 kbit/s (56%) |
| --portable | 359 kbit/s (44%) | 376 kbit/s (44%) | 390 kbit/s (46%) |
| -q 0       | 266 kbit/s (32%) | 285 kbit/s (33%) | 296 kbit/s (35%) |
+------------+------------------+------------------+------------------+
Wiki article indicative bitrate table amended to reflect these results.

Thank you, Nick. A near-perfect match with Synthetic Soul's results as well as mine (371/462/542 kbps for --portable/--standard/--extreme when using FLAC).
Title: lossyWAV 1.1.0 released.
Post by: smok3 on 2008-07-19 22:40:19
how would a piped cmd look for win? (flac to lossywav and back to flac)
p.s. and i can survive without metadata
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-19 22:45:57
how would a piped cmd look for win? (flac to lossywav and back to flac)
p.s. and i can survive without metadata
On the cmd.exe command line:

flac -d thisflac.flac --stdout --silent|lossywav - --stdout|flac - -b 512 -othisflac.lossy.flac --silent

will show the lossyWAV processing with the FLAC decode / encode silenced.

It's really easier with foobar2000 though....
Title: lossyWAV 1.1.0 released.
Post by: smok3 on 2008-07-20 21:05:41
cool, tnx,

what makes you think it will be easier with foobar?
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-20 21:22:54
cool, tnx,

what makes you think it will be easier with foobar?
It automatically copies the tags across - although you did say that you weren't too bothered about them. Also, it constructs the command line for you for multiple tracks / images.
Title: lossyWAV 1.1.0 released.
Post by: smok3 on 2008-07-20 21:55:35
true, somehow i have this weird obsession to construct batches for total commander, so basically one would just select files and then the batch and it goes on.
Title: lossyWAV 1.1.0 released.
Post by: sauvage78 on 2008-07-22 09:24:11
hi,
today I tried to ABX lossywav 1.1.0 with the Ginnungagap problem sample in order to see if there was any improvement from my previous small personal test (Old Test (http://www.hydrogenaudio.org/forums/index.php?showtopic=63254&st=50) see post 68 & 69), I was still able to ABX 100% Q1.0 easily, but I was quickly surprised that Q1.5 was, at the beginning, much harder than previously (I failed once then I was forced to concentrate) but after some retraining I was still successfully able to ABX it 100% ...

then I tried Q2.0 & I failed 100%  ... I failed so badly that I decided to edit the wav to focus on the 3 easiest seconds to ABX ... I retried & put the sound level up & ... I failed 100% ...

so well done Nick I cannot ABX Ginnungagap at Q2.0 anymore & the flaw is slightly reduced at lower settings.
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-07-25 03:36:14
I have three questions:

1. Is noise shaping only used with the portable preset (and below) or also with the standard preset (and maybe higher presets)?
Seems that it's always used with respect to the chosen quality value (q/10).

2. A comparison of ReplayGain values indicates that peak sample values increase with lossyWAV, reaching 1.0 where the originals were below 1.0. May that indicate possibly clipped samples?

3. Can lossyWAV safely be used with 96 kHz material without any disadvantages? Does it depend on whether noise shaping is used or not?
Title: lossyWAV 1.1.0 released.
Post by: Neasden on 2008-07-25 04:00:33
lossyWAV is very interesting!

Is it like...

Album in WAV - 230 mb
Album in lw    - 196 mb
Album in flac  - 99 mb

...?
Title: lossyWAV 1.1.0 released.
Post by: Synthetic Soul on 2008-07-25 09:09:48
lossyWAV is very interesting!

Is it like...

Album in WAV - 230 mb
Album in lw    - 196 mb
Album in flac  - 99 mb

...?
No.  As the LossyWAV file is still a WAVE it has the same size as the source.

Take a look at these stats posted on page 2 of this thread to see the sort of bitrate savings:
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-25 17:47:11
...then I tried Q2.0 & I failed 100%  ... I failed so badly that I decided to edit the wav to focus on the 3 easiest seconds to ABX ... I retried & put the sound level up & ... I failed 100% ...

so well done Nick I cannot ABX Ginnungagap at Q2.0 anymore & the flaw is slightly reduced at lower settings.
Many thanks for the additional ABX testing sauvage78! I'm very pleased that you couldn't ABX the most recent problem sample at -q 2.0. This goes some way to reassure me that the --portable preset is a good one (-q 2.5).

[edit] sp. [/edit]
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-25 19:09:39
I have three questions:

1. Is noise shaping only used with the portable preset (and below) or also with the standard preset (and maybe higher presets)?
Seems that it's always used with respect to the chosen quality value (q/10).

2. A comparison of ReplayGain values indicates that peak sample values increase with lossyWAV, reaching 1.0 where the originals were below 1.0. May that indicate possibly clipped samples?

3. Can lossyWAV safely be used with 96 kHz material without any disadvantages? Does it depend on whether noise shaping is used or not?
1) You can disable shaping with --shaping 0;
2) Samples will sometimes increase due to rounding off lsb's, sometimes decrease - increasing to 32768 will cause some clipping for 16-bit and will be changed to (32767 shr bits-to-remove) shl bits-to-remove. Clips are counted per channel per codec block. If the number of clips exceeds a preset value (see --longhelp) then the bits-to-remove for that channel is reduced by one and the bit removal process is repeated until the number of clips for that channel does not exceed the permitted number or the number of bits removed is zero;
3) Yes - noise shaping is optimised for 44.1kHz and 48kHz - thanks to SebastianG's coefficients - if worried about this noise shaping, use --shaping 0 as above.

[edit] Explanation at #2 [/edit]
Title: lossyWAV 1.1.0 released.
Post by: Dynamic on 2008-07-25 22:28:23
2) Peak samples will sometimes increase due to rounding off lsb's - this will cause some clipping at +32768 for 16-bit and will be changed to (32767 shr bits-to-remove) shl bits-to-remove;


NickC, I was also interested in this explanation.

Am I right to presume that 'shr' is 'bit-shift to the right' by the number of places following the 'shr' command (discarding the LSBs that shift off the end), and 'shl' is 'bit-shift left', shifting to the left while setting the new LSBs to zero.

I.e. 32767 is expressed in 16-bit signed binary as:

0111 1111 1111 1111

If bits to remove is 5, this clips to what I can also express as:
32768 - (2 ^ bits-to-remove) = 32768 - 2^5 = 32768 - 32 = 32736 =
0111 1111 1110 0000

Rather than do my 2 to the power of 5 calculation, this was done
by shifting the maximum value of 32767 by 5 places to the right,
going to 0000 0011 1111 1111
then back to 5 places to the left, filling LSBs with zeroes:
going to 0111 1111 1110 0000

So (2 ^ bits-to-remove) = 32 is the clipping error, which is 1/1024th of the target signal amplitude in this case, and represents a smaller error than 32 from the original signal (presumably between 16 and 31), where the target bits-to-remove would have generated a rounding error of 15 if it had 17 bits available to round upwards instead of having to round downwards.

In this 5-bit case, this clipping error is equivalent to clipping caused by increasing gain by 0.0085 dB above full scale, which is very low-level, and might reassure other users (e.g. try amplifying a full-scale signal by 0.0085 dB and ABX the clipping distortion, which will exceed the distortion in lossyWAV [edit]when 5 bits are removed, that is[/edit]). The sample error adds energy at about -60dB relative to a full-scale sample in this case, which is only mildly indicative of what scale of event may happen in the frequency domain to which the ear responds.

Things get more complicated with noise shaping in use, though presumably there's a feed-forward of accumulated error (like with error-diffusion dither in imaging) which enables the always-negative clipping adjustment to be offset by greater likelihood of positive shifts in following samples, and presumably the instantaneous clipping is likely to be incorporated into the high frequency end of the shaped noise unless there happen to be numerous successive clipped samples, which naturally means lower frequencies.
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-07-26 12:17:34
If I disable noise shaping, I should enable dithering, right? And is dithering or noise shaping really necessary for 24-bit material?

What would be best practice in terms of settings (dithering, noise shaping) for these types of audio:
- 16-bit, 44.1/48 kHz
- 24-bit, 44.1/48 kHz
- 16-bit, 88.2/96/176.4/192 kHz
- 24-bit, 88.2/96/176.4/192 kHz
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-26 12:46:39
If I disable noise shaping, I should enable dithering, right?
You can although by default, lossyWAV does not enable dithering of any kind at any quality preset. Dither will add more noise than simple rounding. Also, the simple rounding is in some ways analogous to a random dither, albeit with an indeterminate probability density function.

In terms of best practice, the default setting (--standard == -q 5 --shaping 0.5) should be more than adequate for most samples / music. There is little difference between 16 and 24 bit sample depth as you may end up with, say, 8 bits left from the same codec-block for each depth if the method determines that.

Most testing has been carried out at 44.1kHz / 16bit - but higher sample rates should be fine.

[edit] --standard == -q 5 --shaping 0.5 NOT -q 0.5 --shaping 0.5 [/edit]
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-07-26 13:02:48
So you think that noise shaping should also be used with higher sample rates instead of normal dithering or no dithering at all?
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-26 14:48:02
So you think that noise shaping should also be used with higher sample rates instead of normal dithering or no dithering at all?
I have heard no problems which would seem to require dither to be enabled.

My only reservation is that the noise shaping method in place is for 44.1kHz and 48kHz - it may or may not work for much higher sample rates.

My suggestion would be to try it and see how the output sounds - you are in effect the first person to my knowledge who will be testing lossyWAV at sample rates significantly greater than 48kHz.

Best of luck!
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-26 19:50:24
What would be best practice in terms of settings (dithering, noise shaping) for these types of audio:
- 16-bit, 44.1/48 kHz
- 24-bit, 44.1/48 kHz
- 16-bit, 88.2/96/176.4/192 kHz
- 24-bit, 88.2/96/176.4/192 kHz
It may be that I should put in place a mechanism whereby the lengths of the FFT analyses used in the processing change from 64/1024 samples at 44.1/48kHz to 128/2048 samples at 88.2/96kHz and 256/4096 samples at 176.4/192kHz. This would require that the codec-block-length would change from 512 to 1024 and 2048 samples respectively.... I'll think on it.
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-07-26 19:56:30
I created four test samples containing white noise: 16-bit/48 kHz, 16-bit/96 kHz, 24-bit/48 kHz and 24-bit/96 kHz. Then I did a frequency analysis of the difference between the original and lossy conversions (default, shaping 0, shaping 0 + dither 1). I conclude three things:

Click here (http://www.devir.de/temp/lossywav.png) for a graphical comparison between 48 kHz and 96 kHz (portable preset).
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-26 20:19:36
I created four test samples containing white noise: 16-bit/48 kHz, 16-bit/96 kHz, 24-bit/48 kHz and 24-bit/96 kHz. Then I did a frequency analysis of the difference between the original and lossy conversions (default, shaping 0, shaping 0 + dither 1). I conclude three things:
  • Dithering seems to have no benefit. It just adds noise on top of the existing quantization noise, thus increasing the noise floor.
  • The bit-depth doesn't seem to matter.
  • Noise shaping benefits from higher sample rates because the noise is moved even further into inaudible frequency ranges.
Click here (http://www.devir.de/temp/lossywav.png) for a graphical comparison between 48 kHz and 96 kHz (portable preset).
Thank you kind sir - it looks pretty good.

So, are you more comfortable now having seen the results - I certainly am, thanks for the effort.


So (2 ^ bits-to-remove) = 32 is the clipping error, which is 1/1024th of the target signal amplitude in this case, and represents a smaller error than 32 from the original signal (presumably between 16 and 31), where the target bits-to-remove would have generated a rounding error of 15 if it had 17 bits available to round upwards instead of having to round downwards.

In this 5-bit case, this clipping error is equivalent to clipping caused by increasing gain by 0.0085 dB above full scale, which is very low-level, and might reassure other users (e.g. try amplifying a full-scale signal by 0.0085 dB and ABX the clipping distortion, which will exceed the distortion in lossyWAV [edit]when 5 bits are removed, that is[/edit]). The sample error adds energy at about -60dB relative to a full-scale sample in this case, which is only mildly indicative of what scale of event may happen in the frequency domain to which the ear responds.

Things get more complicated with noise shaping in use, though presumably there's a feed-forward of accumulated error (like with error-diffusion dither in imaging) which enables the always-negative clipping adjustment to be offset by greater likelihood of positive shifts in following samples, and presumably the instantaneous clipping is likely to be incorporated into the high frequency end of the shaped noise unless there happen to be numerous successive clipped samples, which naturally means lower frequencies.
As bits-to-remove increases, then the difference between what the sample should have been and the clipped sample increases accordingly. However if the bits-to-remove value is high then it is very likely that the audio in that codec-block channel is loud and the clipped sample may be obscured.

It is worth saying again that at preset --standard, only one clip is allowed per channel per codec-block (22.7 micro-seconds.) and from testing, this does not seem to be audible.
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-07-28 16:28:19
Nick,

1. The FFT size should vary with sample rate, though good luck making that happen easily with all the optimisation you've done!

2. IMO and IIRC, enabling dither shouldn't raise the noise floor much on average - the noise floor should stay the same, but more bits will have to be kept the achieve this.

I don't enable dither either.

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-07-28 20:14:20
Nick,

1. The FFT size should vary with sample rate, though good luck making that happen easily with all the optimisation you've done!

2. IMO and IIRC, enabling dither shouldn't raise the noise floor much on average - the noise floor should stay the same, but more bits will have to be kept the achieve this.

I don't enable dither either.

Cheers,
David.
I will modify 1.1.0 to increase the FFT lengths at 69.08kHz, 138.15kHz and 276.3kHz, i.e. 64 to 128 to 256 and 512 samples respectively and correspondingly for the other lengths with a similar increase in codec-block length, 512 to 1024 to 2048 to 4096 samples. This means that the maximum length of FFT will be 8192 samples. [edit] arithmetic failure [/edit]

The spreading functions will not require to be changes as they will be working over approximately the same number of FFT bins after each change, taking into account the increase in sample rate.

I feel that an upper frequency limit of 384kHz is high enough, although I am open to suggestions.

Basically this means that at present I am not confident in the high (i.e. >48kHz) sample rate performance of lossyWAV 1.1.0 and would caution anyone using it at these sample rates against using it for anything other than testing purposes.

On dither, of course you're right - the increase in noise due to dithering will mean that fewer bits can be removed to keep the added noise to the same level that it would have been had to no dither been used used. I will look at creating additional reference threshold constants for the range of dither between --dither 0 (rectangular) to --dither 1 (triangular) to allow dither to be "safely" used.
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-08-03 02:33:33
Basically this means that at present I am not confident in the high (i.e. >48kHz) sample rate performance of lossyWAV 1.1.0 and would caution anyone using it at these sample rates against using it for anything other than testing purposes.
Do you refer to lossyWAV in general or to how noise shaping is applied?

My intention is to understand and implement SebastianG's new noise shaping method, but for that I will also have to introduce / find a PSY model of some kind.

I would hope that by using the new noise shaping method some additional bits can be removed for the same apparent quality level of output, thereby further reducing the bitrate.

What would happen if noise shaping is disabled via --shaping 0? Would that be taken into account by *not* removing those "additional bits" then? Otherwise the non-shaped results might be pretty bad in comparison if used with lower quality settings.

I'm also wondering if trading further removal of bits for better noise shaping really yields useful results as both methods seem to cancel each other out:

Removing more bits:
- lower filesize
- more noise

Stronger noise shaping:
- higher filesize
- less (perceived) noise
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-03 08:45:33
Do you refer to lossyWAV in general or to how noise shaping is applied?

What would happen if noise shaping is disabled via --shaping 0? Would that be taken into account by *not* removing those "additional bits" then? Otherwise the non-shaped results might be pretty bad in comparison if used with lower quality settings.

I'm also wondering if trading further removal of bits for better noise shaping really yields useful results as both methods seem to cancel each other out:

Removing more bits:
- lower filesize
- more noise

Stronger noise shaping:
- higher filesize
- less (perceived) noise
I was referring to lossyWAV in general as the 64/1024 sample fft lengths are fixed at present.
Disabling noise shaping will reduce filesize and increase perceived noise. However the added noise (especially at higher quality presets) should be at or below the existing noise floor.

The noise shaping implementation results in a trade-off between bits removed and filesize. That is why the option remains for the user to disable noise shaping. At --insane the addition of noise shaping only adds a few kbit/s to the FLAC encoded lossyWAV output. The increase in bitrate is substantially more at lower
quality presets.

Modifications for 1.1.0b:I expect to post lossyWAV 1.1.0b tonight.
Title: lossyWAV 1.1.0 released.
Post by: botface on 2008-08-03 08:51:29
Quote
  • reference threshold constants for rectangular dither and triangular dither have been calculated so added noise should be the same for dither off and any dither level between 0 and 1 - the number of bits-to-remove will however reduce with "increasing" dither.
    I expect to post lossyWAV 1.1.0b tonight.


I'm not clear under what circumstances it is appropriate to use dither. Can anybody explain?
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-08-03 11:59:21
I was referring to lossyWAV in general as the 64/1024 sample fft lengths are fixed at present.

Is that the reason why material with 96 kHz is currently reduced to the same filesize as material with 48 kHz? Are higher sample rates currently treated like if they are played back at a lower speed/rate, thus lowering the frequency spectrum accordingly, which results in a wrong calculation of how many bits can be removed?

The noise shaping implementation results in a trade-off between bits removed and filesize. That is why the option remains for the user to disable noise shaping.

Does that mean that if noise shaping is disabled less bits are removed? That would reassure me.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-03 22:13:54
Is that the reason why material with 96 kHz is currently reduced to the same filesize as material with 48 kHz? Are higher sample rates currently treated like if they are played back at a lower speed/rate, thus lowering the frequency spectrum accordingly, which results in a wrong calculation of how many bits can be removed?
Wrong might be a little strong - the sample rate is used correctly where it needs to be but the codec-block-size equates to less time and the granularity of the spreading function will be higher and possibly a bit too coarse.

Does that mean that if noise shaping is disabled less bits are removed? That would reassure me.
Exactly the same (except if noise-shaping causes clips) bits-to-remove values will be achieved for noise shaping 1 and 0.

lossyWAV 1.1.0b attached to post #1 in this thread:
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-08-04 03:07:52
Is it intended that the dithering noise is also shaped? A frequency analysis reveals that it's also mildly shaped. I always thought that rectangular or triangular dither corresponds to white noise.

I also wonder if dithering is really that useful. Look at the graph, it shows AC/DC - Fire Your Guns:

(http://www.devir.de/temp/lossywav2.png)

Wouldn't you rather use -q 3 without dithering than -q 2.5 with dithering when the bitrate is (nearly) identical?
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-04 07:24:11
Is it intended that the dithering noise is also shaped? A frequency analysis reveals that it's also mildly shaped. I always thought that rectangular or triangular dither corresponds to white noise.
Which version of lossyWAV are you using? I ask as the fix to ensure that added noise remains the same when dither is applied has only been introduced at 1.1.0b.

Also, with respect to the triangular dither, I have used a method (discussed on these forums previously) which uses 1 new random number per sample and recycles the previous random number, i.e. tpdf_random = new_random - old_random; (random values in the range +/-0.5). In my implementation and to achieve intermediate dither types I have used: tpdf_random = new_random * dither_type_multiplier - old_random; (dither_type_multiplier in the range 0..1). This may have had an effect on the shape of the dither.

[edit] One other thing: --dither 0 <> no dither. If the dither parameter is used then dither will be applied, somewhere between rectangular and triangular, depending on the parameter value after --dither, i.e. --dither 0 = rectangular dither; --dither 1 = triangular dither; --dither 0.5 = something in between rectangular and triangular dither. [/edit]
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-08-04 11:58:43
I'm using the current version 1.1.0b.

I don't know if it's good to reuse old random values. Why not just use two independent random values? Is the performance impact too high? As it's not used by default I don't think that would be much of an issue.

I have redone the above graph and now differentiated correctly between dither off and dither 0. I also included ReplayGain values to illustrate the theoretically perceived loudness of the noise.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-04 12:36:51
I'm using the current version 1.1.0b.

I don't know if it's good to reuse old random values. Why not just use two independent random values? Is the performance impact too high? As it's not used by default I don't think that would be much of an issue.

I have redone the above graph and now differentiated correctly between dither off and dither 0. I also included ReplayGain values to illustrate the theoretically perceived loudness of the noise.
I'll find my pest control gear and go bug-hunting - the -s 0 -D 1 curve looks suspiciously like it has been noise-shaped.

There is no real performance hit using two independent random values (especially as I have changed from a fibonacci based RNG to a multiply-with-carry RNG) - I'll have a look at that too.
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-08-04 13:11:08
Of course the dither should/will be noise shaped. It should/will sit next to the quantisation stage within the noise shaping loop.

Current random number minus previous random number is a well accepted way of getting slightly high passed dither.

Hancoque, no one is suggesting that you should use dither. It's there for people who incorrectly think that they need it - not for those who know that they don't (nor for those that don't know or care!).

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-04 13:33:13
Of course the dither should/will be noise shaped. It should/will sit next to the quantisation stage within the noise shaping loop.

Current random number minus previous random number is a well accepted way of getting slightly high passed dither.

Hancoque, no one is suggesting that you should use dither. It's there for people who incorrectly think that they need it - not for those who know that they don't (nor for those that don't know or care!).

Cheers,
David.
1) Dither is added pre noise shaping;
2) Thanks - it will remain so;
3) I'll leave the option to dither up to the user.

Thanks David.
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-08-05 00:29:52
I don't plan to use dithering either. I was just curious.

Now that higher sample rates are officially supported, do I understand it right that they require a higher codec block size?

I will modify 1.1.0 to increase the FFT lengths at 69.08kHz, 138.15kHz and 276.3kHz, i.e. 64 to 128 to 256 and 512 samples respectively and correspondingly for the other lengths with a similar increase in codec-block length, 512 to 1024 to 2048 to 4096 samples.
So, does this equate to the following sample rate to codec block size analogy?

44.1/48 kHz ->  512
88.2/96 kHz -> 1024
176.4/192 kHz -> 2048
Title: lossyWAV 1.1.0 released.
Post by: Gow on 2008-08-05 03:23:59
I started doing some personal listening and file size tests with 1.1.0b and I must say I am very impressed.  The idea behind it all was quite something and now we have it implemented with TAK, FLAC and Wavpack.  I use TAK on my computers because that is meant for foobar2000 playback and I use mp3 for my DAP because it is compatible and can be mp3gain'd.

So far, I have been using --extreme and I am in love with the results and it looks like I might switch the all lossless library to the lossyTAK library and if my calculation is correct I should expect the lossyTAK library to be 61% of the size of the current library at -p5.  So roughly 305gb down to 186gb and have yet to notice anything that could be a problem sample.  Of course, I still have the original image files of the CDs ripped using EAC safely tucked away on DVDs so if I do replace the entire library and a problem is discovered it is quite easy to recover from with the DVD, all done to increase disc longevity and to maintain perfect rips.

Good job, keep it up.

On a theoretical note,

This might move the lossless formats even more into the mainstream although the end file is not technically lossless but it so high quality of an output that the problem samples that exist in the lossy formats do not exist here.

I am all for increasing the overall usage of lossless over lossy and with the lossyWAV a new field of battle has been opened up that might sway those who think lossless is TOO big for use to the lossless side.
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-08-05 10:30:17
1) Dither is added pre noise shaping;
But in-loop or out-of-loop? Traditionally it should be in-loop, like this...

http://www.hydrogenaudio.org/forums/index....st&p=580394 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=65038&view=findpost&p=580394)

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-05 10:57:03
But in-loop or out-of-loop? Traditionally it should be in-loop, like this...

http://www.hydrogenaudio.org/forums/index....st&p=580394 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=65038&view=findpost&p=580394)

Cheers,
David.
Sorry, not clear enough - dither is added in the middle of the noise shaping process, just before rounding.
Title: lossyWAV 1.1.0 released.
Post by: smok3 on 2008-08-07 09:18:34
continued, about bat files and tagging, this oneliner seems to be working;

lossyflac.bat
Code: [Select]
flac -d %1 --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o %~p1%~n1.lossy.flac --silent && tag --fromfile %1 %~p1%~n1.lossy.flac


requirements: flac, lossywav and tag somewhere on your %path%

example usage: lossyflac file.flac will return file.lossy.flac (with all the tags from file.flac)

p.s. the usual warning about spaces and batch files on win.

edit1: slight fix with paths, added %~p1
edit2: became oneliner
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-07 10:12:29
continued, about bat files and tagging, this two liner seems to be working;

lossyflac.bat
Code: [Select]
flac -d %1 --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o %~n1.lossy.flac --silent
tag --fromfile %1 %~n1.lossy.flac
...
Very, very elegant - now no need to use foobar2000 for the processing. You could put both lines on one line with a && in between .
Title: lossyWAV 1.1.0 released.
Post by: smok3 on 2008-08-07 13:41:11
Quote
You could put both lines on one line with a && in between

any benefits from that? or do we want oneliner? 
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-07 13:53:37
Quote
You could put both lines on one line with a && in between
any benefits from that? or do we want oneliner? 
Minimalist, I am a fan of one-line code where it can be done.... The final batch file would be:
Code: [Select]
@flac -d %1 --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o %~p1%~n1.lossy.flac --silent && tag --fromfile %1 %~p1%~n1.lossy.flac
Title: lossyWAV 1.1.0 released.
Post by: Synthetic Soul on 2008-08-07 14:19:23
One minor benefit:  As you are using && (as opposed to &) the call to Tag will only be made if the previous command executed successfully.
Title: lossyWAV 1.1.0 released.
Post by: smok3 on 2008-08-07 15:04:55
fixed the post - 'downgraded' twoliner to oneliner  http://www.hydrogenaudio.org/forums/index....st&p=581437 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=64666&view=findpost&p=581437)
Title: lossyWAV 1.1.0 released.
Post by: Hanky on 2008-08-07 15:16:43
Nice batch file. But if the specified input file does not exist, this results in a crash of lossywav.exe
Windows XP SP2, lossywav 1.1.0b
(http://dzzn76.googlepages.com/lossywav.png)
Code: [Select]
F:\lossywavtest\>flac -d doesnotexist.flac --stdout --silent  | lossy
wav - --stdout -P  | flac - -b 512 -o doesnotexist.lossy.flac --silent

doesnotexist.flac: ERROR initializing decoder
                   init status = FLAC__STREAM_DECODER_INIT_STATUS_ERROR_OPENING_
FlossyWAV 1.1.0b, Copyright (C) 2007,2008 Nick Currie. Copyleft.
This is free software under the GNU GPLv3+ license; There is NO WARRANTY, to
the extent permitted by law. <http://www.gnu.org/licenses/> for details.
ILE

An error occurred opening the input file; it is likely that it does not exist
or is not readable.
ERROR: for encoding a raw file you must specify a value for --endian, --sign, --
channels, --bps, and --sample-rate
Type "flac" for a usage summary or "flac --help" for all options

F:\lossywavtest\>tag --fromfile doesnotexist.flac doesnotexist.lossy.
flac
Tag - Automatic Tag from filename      Copyright (c) 2002-2003 Case
Version 2.0.39b, Compiled 2003-04-11

File not found: 'doesnotexist.lossy.flac'
fromfile: Can't open file 'doesnotexist.flac'.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-07 15:48:27
Nice batch file. But if the specified input file does not exist, this results in a crash of lossywav.exe
This could probably be dealt with
Code: [Select]
@if exist "%1" flac -d "%1" --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o "%~n1.lossy.flac" --silent && tag --fromfile "%1" "%~n1.lossy.flac"
Title: lossyWAV 1.1.0 released.
Post by: Hanky on 2008-08-07 15:53:06
Thanks Nick.C, that fixed the error 

One more small issue (not lossywav related) is the fact that TAG does not copy embedded cover art.

edit:
I found out I was using an old version of TAG (2.0.39b) and updated to 2.0.52, but that version did not copy embedded cover art either.

But this issue should not be discussed here, rather in Changes Made to TAG.EXE, Tags from text file & OptimFrog support (http://www.hydrogenaudio.org/forums/index.php?showtopic=26497)
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-07 20:03:37
Thanks Nick.C, that fixed the error 
This will process multiple files and place the result in the same directory as the original.
Code: [Select]
@echo off
:repeat
if %1.==. goto end
if exist %1 flac -d %1 --stdout --silent|lossywav - --stdout -P --stdinname %1|flac - -b 512 -o "%~dpn1.lossy.flac" --silent && tag --fromfile %1 "%~dpn1.lossy.flac"
shift
goto repeat
:end
Title: lossyWAV 1.1.0 released.
Post by: Hanky on 2008-08-07 20:33:22
That one works great when you pass  one or more filenames, but when I tried with *.flac  I got a crash of LossyWav.exe again.
It seems that we need a construction with a FOR (%%I) IN %1 DO loop here to handle wild cards.

This command line does the job:
Code: [Select]
FOR %X in (*.flac) DO lossyflac.bat "%X"


edit: quotes added, thanks again Nick.C
Title: lossyWAV 1.1.0 released.
Post by: smok3 on 2008-08-07 20:43:10
my usual approach is to do one batch that handles one file (action)

and 2nd batch that loops over the first one, (loopy)

one way or the other, depending on the usage. That way you can keep your action bat short and dumb and leave all the file management logic to the loopy one.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-07 20:45:34
That one works great when you pass  one or more filenames, but when I tried with *.flac  I got a crash of LossyWav.exe again.
It seems that we need a construction with a FOR (%%I) IN %1 DO loop here to handle wild cards.

This command line does the job:
Code: [Select]
FOR %X in (*.flac) DO lossyflac.bat %X

But this one skips filenames that include spaces.
If you enclose the second %X in double quotes it will handle the files properly.
Title: lossyWAV 1.1.0 released.
Post by: sauvage78 on 2008-08-17 06:49:15
Quote: Nick.C
"My intention is to understand and implement SebastianG's new noise shaping method, but for that I will also have to introduce / find a PSY model of some kind."

Do you have any clue of how long this will take ? 3 months, 6 months or a year ?
Is SebastianG giving you any accelerated private lessons ?
It's not that I want to hurry you  or being rude in any way  , but I care a lot for lossywav ... it's already my favorite lossy codec ... & I plan to convert tera of lossless to Lossy|Tak -P|-p2e ... (without lossless backup) so I care a lot for this new noise shaping method if it can make me save some kbps (& also for the new special Tak setting for lossywav  )

without a 1.2.0 development thread I am asking myself everyday:
1: is it a TODO thing that is already actively worked on in the shadows. 
2: is it a TODO thing that is just an idea. 
3: are you in vacation with wife & kids. 
so I'd rather simply ask ...

as I have been disapointed by vaporware feature from Christopher 'Monty' Montgomery in the past ... I am very suspicious about open source developers claims ... so excuse me if I sound rude I just want to test your determination to get lossywav to the max of its possible efficiency. (I don't want the "new noise shaping method" to become the "bitrate peeling" of lossywav)

I know you just released V1.1.0 a month ago & that I shouldn't already be longing for more ... but now that you are a developer you'll have to learn that end-users are relentless vampires !!! LOL
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-08-17 07:20:56
Nick.C will tell better, but as for my understanding the different kind of noise shaping can't be expected soon.
Moreover it's not even clear at the moment whether or not it will be the better alternative to the current kind of noise shaping at least when targeting at 'lossless quality in a practical sense' (using '--standard').
The most promising usage of this hopefully advanced noise shaping will be at the lower bitrate quality end. Hopefully we will get the very good quality of '--portable' with a significantly lower bitrate than 370 kbps which is needed on average at the moment.

So waiting for the new noise shaping IMO is necessary only when targeting at low bitrate settings.

BTW I have to update my signature. Though I don't keep a lossless version of my music any more I do not use '--extreme' any more (with the exception of very precious tracks). '--standard' is fine even for this purpose.
Moreover I don't care any more about 'bitrate bloat' on rare occasion (solo instruments) compared to lossless wavPack. Relative to my total collection it's negligible, and I prefer having all my music in FLAC and mp3 format (thinking a bit of a future DAP).
Title: lossyWAV 1.1.0 released.
Post by: BlAcKnOiSe on 2008-08-17 10:23:52
With all the respect, what's the sense of lossyWAV+WavPack encoding?... Why don't use WavPack in lossy mode?... As I understand it, it uses a similar compression... and maybe the encoding is a little less hassle... Or lossyWAV has some compression/size benefit over lossy WavPack file?...
Title: lossyWAV 1.1.0 released.
Post by: lvqcl on 2008-08-17 10:56:22
With all the respect, what's the sense of lossyWAV+WavPack encoding?... Why don't use WavPack in lossy mode?... As I understand it, it uses a similar compression... and maybe the encoding is a little less hassle... Or lossyWAV has some compression/size benefit over lossy WavPack file?...

You can transcode lossyWAV+WavPack files to FLAC/WMALossless/TAK/etc and get exactly the same quality and nearly the same bitrate as original WV files. WavPack lossy mode doesn't have this feature: you cannot transcode it to any other format without loss in quality or compression.
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-08-17 13:19:13
With all the respect, what's the sense of lossyWAV+WavPack encoding?...

I don't know whether this addresses me. In case it does (you wrote it after my post):
I used lossless wavPack (without lossyWAV preprocessing) in all those cases were this yielded smaller files than lossyWAV | FLAC.
It is extremely rare that this happens, but with solo instrument music it can be like that. The main reason is that FLAC doesn't work well with this kind of music (considering just lossless encoding FLAC yields larger files than wavPack or TAK up to 10% or more for solo instrument tracks and similar music).
At the moment I can use wavPack on my DAP, but I don't want to rely on that for the future. wavPack is a great piece of software, but unfortunately not well supported on DAPs.
That's why I concentrate on FLAC and mp3, and don't care any more about the very few files where the lossyWAV | FLAC procedure yields larger files than a lossless wavPack encoding.

Other than that lossyWAV | wavPack is really not very attractive. With best compatibility for players in mind lossyWAV | FLAC is more attractive. With best overall efficiency in mind lossyWAV | TAK is more attractive. lossyWAV | wavPack is the least efficient combination, even after David Bryant's improvements on this situation. And wavPack enthusiasts have no reason to switch from wavPack lossy to another very high qualty lossy procedure. wavPack lossy is great as well (and nearly for sure is the better solution when targeting at a bitrate below 300 kbps).

lossyWAV | lossless codec on the other hand has the great feature that the lossless codec can be replaced whenever another codec will become more attractive. This is a future-proof feature and a lossless process. And of course quality also is superb when targeting at ~350+ kbps.
Title: lossyWAV 1.1.0 released.
Post by: BlAcKnOiSe on 2008-08-17 14:01:56
Quote
I don't know whether this addresses me.

No...  It was a coincidence that you talked about wavpack. But you and lvqcl pointed out that the future lossless transcoding is the reason to consider the lossyWAV+WavPack combo...
So other than that it's better to stay with wavpacks native lossy format, or use lossyWAV + "more popular lossless codec"...
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-17 21:13:46
Quote: Nick.C
"My intention is to understand and implement SebastianG's new noise shaping method, but for that I will also have to introduce / find a PSY model of some kind."

Do you have any clue of how long this will take ? 3 months, 6 months or a year ?
I don't really know - I have been thinking about it for a while and I believe that one of the nice things about lossyWAV is that it's psy model (if it can be called that) is extremely simplistic - basically take more account of fft results at the lower end of the frequency spectrum and disregard any results above 16kHz.
Is SebastianG giving you any accelerated private lessons ?
Nope.
It's not that I want to hurry you  or being rude in any way  , but I care a lot for lossywav ... it's already my favorite lossy codec ... & I plan to convert tera of lossless to Lossy|Tak -P|-p2e ... (without lossless backup) so I care a lot for this new noise shaping method if it can make me save some kbps (& also for the new special Tak setting for lossywav  )
The new noise shaping method may not actually reduce bitrate (although I would hope that it would). However, until a psy model is identified as being compatible with the rest of the NS mechanics then the development cannot progress past the implementation of SebastianG's method sans psy model.
without a 1.2.0 development thread I am asking myself everyday:
1: is it a TODO thing that is already actively worked on in the shadows. 
2: is it a TODO thing that is just an idea. 
3: are you in vacation with wife & kids. 
so I'd rather simply ask ...
1: Not yet;
2: Not really;
3: I was for a week, and we all had a nice time.
as I have been disapointed by vaporware feature from Christopher 'Monty' Montgomery in the past ... I am very suspicious about open source developers claims ... so excuse me if I sound rude I just want to test your determination to get lossywav to the max of its possible efficiency. (I don't want the "new noise shaping method" to become the "bitrate peeling" of lossywav)

I know you just released V1.1.0 a month ago & that I shouldn't already be longing for more ... but now that you are a developer you'll have to learn that end-users are relentless vampires !!! LOL
lossyWAV has absorbed most of my "me" time over the last 13 months or so. It is still being worked on, but only speedups and code optimisations at present. I would expect to start the first attempt at implementing SG's new method "soon" but I am not prepared (as this is a hobby project) to set a definitive date for completion.

I am very pleased that you think enough of the project to be seeking "more" from it, and also appreciative of the ABX testing that you have carried out during the tuning phase of 1.1.0.
Title: lossyWAV 1.1.0 released.
Post by: sauvage78 on 2008-08-18 05:34:55
halb27 & Nick.C
Thanks for the good answers, it helped me see clearer in the future of lossywav.

Nick.C,

1: Did you ever think of having a separate site for lossywav ? other than the wiki I mean. Noobs like to have a homesite for their bookmarks. It's nicer to point a noob to an "official" site than to a wiki or a forum, for sharing links. A forum can disapear & a wiki is too anonymous IMHO. Maybe a simple one like wavpack, hosted by rarewares ?

2: Did you ever think of asking for a logo like other codec did on HA in the past, sure it will not improve the audio quality but it will help identify the codec & I could put it on my avatar as a free ad for publicity
Some guys are pretty skilled on the forum, the tak logo happened to be very nice !

3: Did you ever think putting lossywav under Xiph umbrella ? (I am not sure it is a good idea at all LOL but Xiph doesn't have any hybrid codec) & I mean, from the start you seemed more attached to flac more than any other lossless codec ...    Xiph is maybe a communist place to be    but maybe it would attract some Linux users to lossywav & maybe would incite Josh to dig his brain to find a dedicaded flac setting for lossywav like Tom is doing for Tak

quote: Nick C
"and also appreciative of the ABX testing that you have carried out during the tuning phase of 1.1.0"
well i must have spended less than 4 hours in total for that sample, & only -q2 was very time greedy if I recall well, all credit goes to Mardel for finding it  ... but I would redo it if a sample would threaten my -q2.5 transparency
... to be fully honest I tested for myself first  , this sample come out just when I needed to see what lossywav had in the stomach. Nowaday from time to time when I suspect a part a my favorite songs would be problematic for lossywav I encode it at -q1 & see what happens ... I didn't found any killer sample yet

Edit:
Also for me -p2e should be recommended over -p2m for lossytak, the m switch is too time greedy for the very small spacegain, it's not a bad switch as it brings you the best compression for a target decoding speed, but it shouldn't be recommended as default IMHO even if Tom said so. (In fact he didn't said so if I recall well, he said there was almost no spacegain for lossytak past -p2m which is true if you consider only spacegain & decoding speed, but false if you add encoding speed as a consideration & you concluded that -p2m was recommended ... which is a missinterpretation IMHO ... there is a nice speedgain between -p2e & -p2m even if there is almost no spacegain) The m switch is nice only in two case IMHO: -1 best size for a target decoding speed for DAP, there -p2m may be usefull as it brings you the absolute max compression inside the area of the -p2 decoding speed hardware requirement setting & the other case is -2 the absolute max compression for Tak aka -p5m. The average lossytak user doesn't fall in these cases.
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-08-18 11:00:20
lossyWAV with a good psymodel and arbitrary noise shaping could be as "good" as a typical lossy codec - it would avoid some of the restrictions of some lossy codecs, but would have plenty of its own. Whether it can ever compete in efficiency terms depends in part on the design of the lossless codec it's partnered with.

It would be a fascinating project, but the quality would depend completely on the psychoacoustic model. If you borrowed the psychoacoustic model from Musepack, and used arbitrary noise shaping to push the noise to match that added by musepack, this would be a good start. It's not "optimal" (you're imposing some musepack restrictions on lossyWAV), but it's easier than building a psychoacoustic model from scratch.

I'd join in, if I had the time. However, you have to ask the question: who would use it, and why? If you want a good psychoacoustic codec, there are plenty to choose from. lossyWAV + psymodel is unlikely to beat the best of them in quality, or any of them in bitrate.

I guess it could be higher quality and more compatible in some situations?

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-08-18 11:31:23
I think that the advantage of lossyWAV is that there is *no* psymodel involved, so that the compression stays robust and can be used for post-processing without revealing hidden artefacts. I also think that bitrates around 400 kbps are small enough for this approach and that other codecs should be used if (much) lower bitrates are needed. It would be re-inventing the wheel if lossyWAV incorporated psychoacoustic methods to achieve (near-)transparent results below 300 kbps. We already have MP3, Vorbis and many more like that. In my opinion one should focus on the original goal: Introduce a negligible loss while maintaining the benefits of lossless compression.
Title: lossyWAV 1.1.0 released.
Post by: sauvage78 on 2008-08-18 11:47:24
well I am not technical guy but as far as I understund the psymodel will be used to tell how the added noise must be shaped in order to be unlistenable ... it will not be a psymodel used to affect the music, but the added noise only so it is not "re-inventing the wheel" ... I may be completly wrong in how I understand things but I think Nick is not stupid enough to transform lossywav in ... yet another DCT codec ...

psymodel+DCT is evil, psymodel+noise shaping is good ... I wouldn't even have asked for this enhancement if I knew it would be a regression ... I would be damn stupid too ... hum well maybe I am afterall
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-08-18 12:55:23
...but that's exactly what psymodels do in all codecs - judge where noise can be added, such that it will (or should!) be inaudible.

DCT (e.g. mp3, AAC), or not (e.g. mp2, Musepack) - that only affects how accurately you can place the noise in the time and/or frequency domains.

The lossyWAV difference signal (i.e. the mathematical difference between the original and the coded version, calculated by simple waveform subtraction) sounds so different because the noise is currently white, or with a fixed shape. If you aggresively shape the noise based on a psy model, it'll sound fairly similar to mp3, AAC or whatever.

It's true you could avoid DCT artefacts specifically, but AAC does this pretty well with temporal noise shaping, and Musepack does this entirely since it's not a transform codec.

I still think it's interesting to try this - for one thing, it creates a new type of audio codec - but potential problems will be similar to some of those already faced by other psychoacoustic based codecs. It won't be a magic faultless codec, like lossyWAV without a psy model might be.

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-08-18 13:59:15
Sounds like planning to implement a rather sophisticated psy model.
I had thought that it's only about a very elementary psy model something that's necessary for placing say in one block noise in the HF region, in another in the low frequency region, in third keeping noise flat, or something like that. I guess wavPack lossy's dynamic noise shaping is of that kind and I thought that it's about something similar.
I personally don't see much sense in struggling for < 250 kbps with the lossyWAV approach. We have a lot of good codecs here - as was said - which probably do a better job than wavPack lossy does with this approach.

Improving quality in the ~300 kbps region is the more promising approach IMO.
Title: lossyWAV 1.1.0 released.
Post by: SebastianG on 2008-08-18 15:37:49
I think that the advantage of lossyWAV is that there is *no* psymodel involved,

First of all, how would that be an advantage? Sencond, what's your definition of "psymodel" ? Doesn't lossyWav's analysis stage qualify as "psymodel"?

so that the compression stays robust and can be used for post-processing without revealing hidden artefacts.

I'm pretty sure that this is simply a matter of headroom. Wouldn't it be even more "robust" if the noise floor's shape matched the hearing threshold's shape -- assuming the the same NMR (noise-to-mask ratio) on average?

...but that's exactly what psymodels do in all codecs - judge where noise can be added, such that it will (or should!) be inaudible.

Right, and lossyWav is trying the same thing by selecting "wasted_bits" on a per-block basis ("temporal noise shaping").

Obviously temporal noise shaping is a good idea. It hides noise in "places" (in time) where it's not easily recognisable. If it weren't a good idea, there wouldn't be a good reason to favour lossyWAV over plain static word length reduction. The same logic applies to spectral noise shaping. It'll hide noise in "places" (in frequency) where it's not easily recognisable. So, the lack of spectral noise shaping isn't really a feature, is it?

I still think it's interesting to try this - for one thing, it creates a new type of audio codec but potential problems will be similar to some of those already faced by other psychoacoustic based codecs. It won't be a magic faultless codec, like lossyWAV without a psy model might be.

In what way is the current lossyWAV faultless? Are you sure this isn't entirely due to the "high bitrate headroom"?

Cheers,
Sebastian
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-08-18 16:34:01

I still think it's interesting to try this - for one thing, it creates a new type of audio codec but potential problems will be similar to some of those already faced by other psychoacoustic based codecs. It won't be a magic faultless codec, like lossyWAV without a psy model might be.
In what way is the current lossyWAV faultless?
I didn't claim it was. I said "might be", in reference to what others may believe or hope (from experience so far).

I was discussing the hope that some people have, that a version with a psy model would be as "safe" as the current version. If it relied on the psy model completely, then I don't believe it would be any more "safe" than mp3/musepack/vorbis/etc. That was the point I was making.


Quote
Are you sure this isn't entirely due to the "high bitrate headroom"?
I'm fairly sure it's because it keeps the added noise below the signal (measured using a variety of temporal and spectral windows), and because the point where the added noise comes closest to the signal is usually at a high frequency where it would be less audible/objectionable anyway. The result is usually over coding at most frequencies, and a comparatively high bitrate.

There's isn't much bitrate "headroom" though - maybe there is more in the ways Nick has intelligently developed it, but in the simple early versions a 6dB increase in noise made that noise a little audible, a 12dB increase made it very audible. In that sense, there was little headroom.


Let me turn the question around: what would be involved in making the error signal from, say, vorbis, Musepack or AAC (I won't include mp3, because you'd need freeformat to have sufficient bitrate available), comparable to that from lossyWAV?

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: SebastianG on 2008-08-18 18:09:39
I didn't claim it was. I said "might be", in reference to what others may believe or hope (from experience so far).

Right. I missed that part.

[...] The result is usually over coding at most frequencies [...] There's isn't much bitrate "headroom" though [...]

That's precisly what I mean. Quality varies. Why would you want to do that?

It should be clear that spectral noise shaping can prevent such large discrepancies (some parts are over-coded and some are just good enough to not sound bad). This issue came up a number of times by now and I still fail to understand the obsession of some people with a spectrally flat noise floor.

Regarding presence/lack of a psychoacoustic model: Excuse me but where do the 'wasted_bits' values come from? Certainly there's a unit in lossyWAV that does some analysis on how much spectrally flat noise is tolerable. It already does some spectral analysis ("different temporal and spectral windows"). It might be simplistic but it certainly qualifies as psyachoacoustic model.

Let me turn the question around: what would be involved in making the error signal from, say, vorbis, Musepack or AAC [...] comparable to that from lossyWAV?

You make it sound like if that's a good thing to try. I don't see where you're going with this. But in theory it should be possible -- at least in case of Vorbis. You probably have to design some new code books (i.e. for those parts that are heavily over-coded) ;-) Encoding the floor curve is a piece of cake since it's a boring straight line with an offset depending on 'wasted_bits'.

Cheers,
Sebastian
Title: lossyWAV 1.1.0 released.
Post by: Hancoque on 2008-08-18 22:23:32
First of all, how would that be an advantage?

Whether it is an advantage or not depends on what you want to achieve. Personally, I want to use lossyWAV because it allows for heavy post-processing. So, for me it isn't only important that a resulting file sounds good "as is" but also that it doesn't show any issues after equalizing, pitch-shifting or HRTF processing. For me it's a compromise between "pure lossy" and lossless. And I thought that this is the whole purpose of lossyWAV: being lossy to lower the bitrate while staying close to lossless and maintaining all the advantages of lossless compression like post-processability or the assurance that your dog or invisible aliens won't hear nasty artefacts while you enjoy a seemingly good sound.

Sencond, what's your definition of "psymodel" ? Doesn't lossyWav's analysis stage qualify as "psymodel"?

Well, basically it is already psychoacoustic in a way as it utilizes the masking effect (keeping added noise below the noise floor). But I don't think that some added white noise is a big problem. White noise is neutral and some tape hiss was never an issue during the days of cassette recorders.

Some time ago I read about the different dithering techniques and it has been stated that (flat) triangular dither is the preferred choice if further processing is planned while coloured dither should only be used in the final step. I'm pretty sure that for many people encoding to lossyWAV will be the final processing stage but for me it isn't. It's just the last stage where something is saved as a file.

I'm writing all this because I'm concerned that lossyWAV might be optimized for noise shaping in a way that non-shaped quality suffers as it is no longer of importance to the developer and those that are involved in the optimization process. It would reassure me to know that the quality of --portable --shaping 0 will only increase (not so important) but never decrease (very important) in the future.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-19 07:24:13
I'm writing all this because I'm concerned that lossyWAV might be optimized for noise shaping in a way that non-shaped quality suffers as it is no longer of importance to the developer and those that are involved in the optimization process. It would reassure me to know that the quality of --portable --shaping 0 will only increase (not so important) but never decrease (very important) in the future.
I have no intention at all of forcing the user to use noise shaping, neither do I want to reduce the perceived quality of any of the quality presets. I hope that that in some way reassures you....

[edit] Taking this opportunity to seek advice from interested users, I am wondering as to the real need for the correction file? Reversion to lossless, although possible, is not a painless procedure (although when creating a correction file the user cannot use STDOUT processing) and as the file "addition" process works outside of a lossless codec the re-integration cannot be performed at the player (as WavPack so elegantly can, I believe). [/edit]
Title: lossyWAV 1.1.0 released.
Post by: smok3 on 2008-08-19 08:24:46
correction files would be a waste of time imho, there are other who can do that (wavpack & optimfrog for example), so why bother.
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-08-19 10:34:49
Oh, I liked them  OK, I admit I haven't used one yet(!), but I liked having the option.
Title: lossyWAV 1.1.0 released.
Post by: SebastianG on 2008-08-19 10:42:31
Hancoque, there's nothing in between "pure lossy" and lossless. You're aming for a large headroom in quality. That's all. You didn't mention a single reason against spectral noise shaping.

Quote
Well, basically it is already psychoacoustic in a way as it utilizes the masking effect (keeping added noise below the noise floor).

I know what you mean by "noise floor" and this is probably part of a misunderstanding. How would you know the "nosie floor" in order to keep additional quantization noise below it? You don't. And it doesn't really matter at all. Hearing thresholds matter. You want to be on the safe side w.r.t. further processing etc? Use a large headrooms.

...like for example quantization noise that's right under min(S - 12dB, M - 25dB) within every cricital band where S=signal, M=estimated masking threshold. The above formular grauantees a minimal SNR of 12 dB and a headroom of at least 15 dB considering a medium quality psychoacoustic model that's within +/- 10 dB of accuracy.

Cheers,
SG
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-08-19 11:03:32
That's precisly what I mean. Quality varies. Why would you want to do that?
The same is true of LPCM, isn't it? The "overcoding" for loud signals is much greater than the "overcoding" for quiet signals.

Yet some people insist on using lossless coding of LPCM - not everyone looks at LPCM and thinks "why would you want to do that?" - not every one feels the need to add noise to LPCM to make it "constant quality".

lossyWAV is a bit like LPCM, and a bit like psychoacoustic coding. It's a "half way house", and maybe not many people will want to use it. However, those who have a reason for avoiding full psychoacoustic coding (e.g. a desire to post-process), and also a reason for avoiding lossless (e.g. a desire not to throw 1000kbps+ at a lo-fi 2008 CD issue which would be largely transparent at 8-bit resolution!) may want to try it.


Remember (as if I need to tell you!  ) that conventional codecs often put "noise" above the "signal", since basic spectral masking theory says this is often OK, because it's inaudible. In contrast, lossyWAV tries to keep the added noise below the signal, always.

It should be obvious why this is useful for post-processing - if the noise is below the signal on every "useful" time and frequency scale, then it's near impossible to drag it above the signal, never mind to make it audible. Whereas the "inaudible" noise from conventional codecs, which could be above the signal, can "easily" be EQ'd (for example) to the level where it's no-longer masked.

I admit that concepts such as "noise below the signal" and "on every useful time and frequency scale" are not as simple and clear-cut as I imply when describing lossyWAV, but Nick really does seem to have hammered out most of the wrinkles in practice.

Cheers,
David.


...like for example quantization noise that's right under min(S - 12dB, M - 25dB) within every cricital band where S=signal, M=estimated masking threshold. The above formular grauantees a minimal SNR of 12 dB and a headroom of at least 15 dB considering a medium quality psychoacoustic model that's within +/- 10 dB of accuracy.
You could do that. You could introduce noise shaping and "real" psychoacoustics like that - or alternatively you could introduce noise shaping based on the minimum energy in each critical band (or fraction thereof). That's what I envisaged. Keep the noise well below the signal, even if the resulting noise level is lower than the masking threshold estimate.

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: SebastianG on 2008-08-19 11:58:34
The same is true of LPCM, isn't it? The "overcoding" for loud signals is much greater than the "overcoding" for quiet signals.

Yeah, and that's a particularly bad thing to do, isn't it?  That's the whole point of lossy coding, isn't it? 

btw: AFAIR Bryant suggested using WavPack lossy instead of going from 24bit to 16bit LPCM to save space. This sounds like a sensible thing to do, doesn't it? It'll allow you to do some postprocessing (like dynamic range compression etc) whereas at 16 bit you're stuck with a certain noise floor that may get audible after some further manipulations...

Remember (as if I need to tell you!  ) that conventional codecs often put "noise" above the "signal", since basic spectral masking theory says this is often OK, because it's inaudible. In contrast, lossyWAV tries to keep the added noise below the signal, always.

This is without doubt desirable. I'm not suggesting anything else, ("min(S - 12dB, M - 25dB)", numbers are chosen more or less arbitrarily).

There are a couple of reasons why this is a good idea in this context: you don't need to dither at all. It won't add much entropy which is good for encoders like FLAC. It won't increase energy anywhere by more than 0.26 dB in this case (SNR>=12 dB, measured over critical bands and not single frequency lines).

Your other suggestion (the level of minimal energy in a critical band) would be a conservative version of masking with a narrow spreading function. I'd still prefer to just add headroom instead of messing with the spreading function. At least this is a tweaking question and a use case question. The typical use case is that we don't do a lot of post processing. For those who do, there can be enough headroom by reducing overall noise levels. You could make a switch for "conservative spreading function" with a comment like "allows heavy equalization but is not recommended for normal use".

Cheers,
SG
Title: lossyWAV 1.1.0 released.
Post by: carpman on 2008-08-19 14:26:58
Taking this opportunity to seek advice from interested users, I am wondering as to the real need for the correction file?

Don't use them with LossyWAV. If I really did want them I'd use WavPack Hybrid.

Suggestion for later down the line, when LossyWAV has been more exposed and more thoroughly tested, but how about:

Changing --extreme to -q 6
Changing --insane to -q 7.5

I'd only recommend this on the proviso that --standard (-q 5) hasn't a single problem sample and is thus still considered transparent 100% of the time.

C.
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-08-19 15:40:24
SebastianG,

The nice thing about lossyWAV is that you can "easily" try all these approaches. There's no fixed filterbank, bitrate cap, DCT legnth(s) to accommodate or limit exactly where noise can be added vs not added.

You (currently) have a fixed block size, and you're stuck with that time resolution in terms of changing how many bits you remove. However, frequency domain-wise you can do anything you want - I guess the only restriction is having a stable realisable noise shaping filter which won't cause clipping. Time-domain wise and time/frequency wise you're quite free to shift noise around as you wish (but the total noise power can only jump by a minimum of 6dB with transitions every 512 samples).

One caveat is that some of the things you could do would be quite pointless because they would dramatically reduce the efficiency of the subsequent lossless coding.

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: sauvage78 on 2008-08-19 16:00:28
I don't use correction files I have never used it & don't plan to ever use it.

carpman:
we already had a lot of discussions on lossywav presets, everybody has different opinions on it ...

Q2.5  --portable is a preset because problem samples were ABXable up to Q2 (but very hard at Q2)
                        , so Q2.5 has a small but nice 0.5 margin.

Q5    --standard is a preset has it is the original historic setting, where the added noise is always masked in theory

Q7.5 --extreme is a miror of Q2.5 with a paranoid margin

Q10 --insane is just pushing the incrementation logic of a 2.5 by 2.5 step by preset to the limit of the quality scale.

linking --extreme & --insane to Q6 & Q7.5 just because you consider it 100% transparent is pure random ... personnaly I consider Q2.5 100% transparent until someone prove the contrary by providing an ABXable problem sample. So far Q2.5 is the "real life" transparency point of lossywav & Q5 is the theoric optimal setting for transparency of lossywav ... anything above Q5 is overkill, only maybe usefull for people willing to use a transform codec later.

Q10 --insane is a useless preset IMHO, but if Nick wants it, amen ... I will not ask him to remove it (Edit: well I already did, but he didn't listen  )

you could randomly link --insane & --extreme to any setting above Q5 ... because it is overkill anyway ... using a 2.5 step is just a "little" less random ...

IMHO the rational choice for a lossywav setting is between either Q2.5 or Q5 ... using Q2.5 you take the risk that a problem sample arise & prove that Q2.5 is not 100% transparent ... using Q5 the margin is already VERY big & you have the theory on your side so you may have the felling of doing things the right way ... anything above q5 is pure paranoia or a transcoding setting.

presets were made as a guide for noobs because lossywav is very different from any other lossy codec ... if you like q6 then use the quality scale, as long as you know why you use q6, it's ok ... obviously having 2 overkill presets is missleading for noobs as people used to the vorbis/nero quality scale think "the bigger the better" ... it is quickly only true in theory & not in real life (ABX) for lossywav ...
Title: lossyWAV 1.1.0 released.
Post by: carpman on 2008-08-19 16:12:15
linking --extreme & --insane to Q6 & Q7.5 just because you consider it 100% transparent is pure random ... personnaly I consider Q2.5 100% transparent until someone prove the contrary by providing an ABXable problem sample. So far Q2.5 is the "real life" transparency point of lossywav & Q5 is the theoric optimal setting for transparency of lossywav ... anything above Q5 is overkill, only maybe usefull for people willing to use a transform codec later.

Q10 --insane is a useless preset IMHO, but if Nick wants it, amen ... I will not ask him to remove it

you could randomly link --insane & --extreme to any setting above Q5 ... because it is overkill anyway ... using a 2.5 step is just a "little" less random ...

As I said: "Suggestion for later down the line, when LossyWAV has been more exposed and more thoroughly tested".

Everything you said is correct NOW, we'll have to wait and see if it's still correct a year from now. Until thousands of people are using it and testing it with all their myriad forms of music, we cannot be as certain as your post suggests you are.

C.

[EDIT: typo]
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-19 22:04:07
All this talk of noise shaping, psy=modelling and spreading functions got me to thinking about something....

From the earliest stages of variable spreading lengths per frequency band, the "spread" result has been calculated as follows for different spreading lengths (spl) [fft_result = array of skewed magnitudes of the fft analysis]:

spl=1; result:=fft_result[a];
spl=2; result:=(fft_result[a]+fft_result[a+1])/2;
spl=3; result:=(fft_result[a]+fft_result[a+1]+fft_result[a+2])/3;
etc.

However for the even cases, maybe the following should apply:

spl=1; result:=fft_result[a];
spl=2; result:=(fft_result[a-1]/2+fft_result[a]+fft_result[a+1]/2)/2;
spl=3; result:=(fft_result[a-1]+fft_result[a]+fft_result[a+1])/3;
spl=4; result:=(fft_result[a-2]/2+fft_result[a-1]+fft_result[a]+fft_result[a+1]+fft_result[a+2]/2)/4;
etc.

This seems to have the advantage of taking the adjacent results into account for both odd and even spreading lengths rather than just for [edit] odd even [/edit] spreading lengths.
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-08-19 22:07:20
... anything above Q5 is overkill, only maybe usefull for people willing to use a transform codec later ...

I think very high quality archiving as an alternative for a lossless archive makes the availability of --extreme (-q 7.5) and --insane (-q 10) desirable though I personally don't use --insane, and I use --extreme only for very precious tracks.
As carpman said we can't be absolutely sure about --standard's transparency so for archving quality the quality headroom of presumed overkill settings is welcome.
Everybody can use the quality settings according to his needs, and IMO for everybody there is a quality setting which makes him happy.

As for the correction file: I personally don't use it as a correction file and don't plan to use it this way. But I like the ability to be able to listen to the error signal. Once I did when using noise shaping improved my confidence in the usefulness of current noise shaping a lot.
Title: lossyWAV 1.1.0 released.
Post by: carpman on 2008-08-20 00:00:56
Precisely.

I'm using -q 6 because my lossyTAK files are a replacement for my lossless archive. I felt it sensible to have a little headroom over --standard as they'll be used for transcoding to MP3, and presently it's too early to tell whether lossyWAV is transparent from 2.5 upwards. Though it looks pretty good right now.

It just struck me that you probably would have to actually be insane to use --insane. 

C.
Title: lossyWAV 1.1.0 released.
Post by: GeSomeone on 2008-08-20 12:00:37
The opinions about the quality presets will be endless, but not important.
The scale to use is -q (and then there are some presets to give the general idea). As I understand it, the -q scale is meant cover all the bit rates lossyWav can be used for, without limiting to practical use only. So a gradual scale from almost lossless bit rates to the point where lossyWav becomes audibly non-transparent.
There is for everybody a choice that's right. You want minimum or maximum overkill/headroom? very safe or on the edge? It's there.

I think changing the quality scale causes more confusion than it's worth.
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2008-08-21 20:30:34
I think changing the quality scale causes more confusion than it's worth.
I agree - there are users with pretty much diametrically opposed views, e.g. some wanting lowest bitrate with acceptable quality and some wanting to see extremely high quality / bitrate (i.e. even closer to lossless).

As an aside, I am looking further into the spreading-function. Up to 1.1.0 it has always averaged an integer number of FFT_result bins to produce its output. Thinking through the modifications necessary to take partial results at the edges for even spreading lengths (i.e. still centred on a particular bin rather than centred between two bins) it has occurred to me that this could be extended to take any proportion of the outlying bins as one possible method. I am still working on this, but it will probably be included in the first beta 1.12.1.
Title: lossyWAV 1.1.0 released.
Post by: sauvage78 on 2008-08-24 13:58:24
just for my own fun, & (to be honest) to convince myself that lossywav was the right way to go for my own use, I have made a .txt throwing ideas of why I should or shouldn't use lossywav as my main codec ... I just copy past it here so that anyone can improve/disagree with my opinion

Pro:
- Transparent at ~370Kbps (so far)
- ~50% space gain on default setting compared to wav (More if you push the codec to its limits)
- Editable (Split/Join Losslessly, most other lossy codecs split losslessly but cannot be rejoined losslessly) ==> Usefull for Lossy CDImage+cue
- 100% Native Gapless (No tagging tricks like mpeg codecs, no matter if the audio player can read the tag info, it IS gapless by nature)
- No Agressive Lowpass (Don't know if generous flat lowpass ... say 20Khz should really be considered evil as it would maybe save some Kbps, test (& new soundcard) needed)
- No Transform (No DCT, No Subband) ==> free of artefacts linked to transform codecs (Pre-echo ...)
- Future-safe: Usable with several of the best lossless codecs (Flac/Tak) (Lossywav improves as all supported lossless codecs improve so if any of these codec die, lossywav doesn't die)
- Best solution at overkill bitrate ~320Kbps
- Can (most likely) be trancoded with a transform codec once at --standard & above without audible loss.
  (Lossywav artefacts being of a different nature than transform codec artefacts both (if not audible) can (most likely) be added while still achieving transparency)

Anti:
- Added noise compared to lossless (Immanent to any lossy codec)
- Young Codec: Transparency point (actually -- portable IMHO) is likely to change as more problem samples arise
  [One could argue that this is a problem for all lossy codec Transparency point move at each new release, hopefully it will move downward but it could move upward until it becomes mature]
- Not optimal with Wavpack
- Can't compete & will (most likely) never compete at highbitrate ~256Kbps
- Can't compete & will never compete at midbitrate  ~128Kbps
- Can't compete & will never compete at lowbitrate  ~064Kbps

Note: Many of the advantages of lossywav (Editable/Gapless ...) come from the fact that it is in fact not a codec but a pre-filter for wav ==> NO decoder=more freedom

Edit1: Includes some halb27 comments
Title: lossyWAV 1.1.0 released.
Post by: halb27 on 2008-08-24 16:29:21
Pro: ...
Anti ...

That's pretty perfect IMO.

I'd just add to the pro side that lossyWAV is future-safe in the sense that the lossless codec can be exchanged at any time whenever this is appropriate to the user. Exchanging the lossless codec is a lossless process (decode the lossless codec and reencode with the new lossless codec).

It's correct that it's a disadvantage that lossyWAV adds noise. But we should keep in mind that this is relevant only when comparing with lossless codecs. Adding noise is immanent to any lossy codec, and the essential question there is about the character of the added noise ('can it be kept kept inaudibel in a robust way?' on one end of the scale, or: 'can it be very annoying?' on the other end).
I personally don't think that it's likely that the transparency point will change in the future, though of course this can happen. In this problem area I'd rather focus on the fact that experience on transparency is restricted at the moment. I wouldn't care much about whether a problem will come up which will push the current transparency 'border' from -q 2.5 to -q 3 or -q 3.5. I'd rather see it this way: I expect from --portable great quality and wouldn't care about very subtle problems if they would come up. But I expect from --standard transparency. It would be a major issue if --standard should prove not to be transparent. And the restricted experience so far is a negative point. Though I don't consider it likely that --standard will be proven not to be transparent.
Title: lossyWAV 1.1.0 released.
Post by: smok3 on 2008-08-25 08:16:51
the only questions (for me) is how it will behave in editing/mixing enviroment together with bunch of eq's and other DSPs compared to say high-bitrate mp3?
Title: lossyWAV 1.1.0 released.
Post by: jido on 2008-08-26 11:29:57
One caveat is that some of the things you could do would be quite pointless because they would dramatically reduce the efficiency of the subsequent lossless coding.

Cheers,
David.

And that is what we expect from LossyWAV, efficient lossless compression for a minimal sacrifice in quality. If you introduce new algorithms they should not reduce subsequent lossless compression. In fact I believe new algorithms should specifically aim at producing sound waves that are known to compress well, so that you can keep constant quality for a better compression.

Keep up the good work!
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2008-08-26 12:00:50
the only questions (for me) is how it will behave in editing/mixing enviroment together with bunch of eq's and other DSPs compared to say high-bitrate mp3?
I would guess the answer is "far better".

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: Dynamic on 2008-08-26 21:07:49

the only questions (for me) is how it will behave in editing/mixing enviroment together with bunch of eq's and other DSPs compared to say high-bitrate mp3?
I would guess the answer is "far better".


I'd have to agree with David.

Traditional lossy encoders with a full psychoacoustic model try a number of tricks which can be problematic with various EQ, DSP or effects.

Temporal masking. A sudden loud sound can mask quieter sounds for a time period afterwards, with a masking level that decays for a time (post-masking) and masking sound for a much shorter period prior to the loud sound (pre-masking). This can be exploited by allowing added noise (e.g. coarser quantization) below the temporal masking threshold. Tempo adjustment to slow the tempo could unmask sounds and result in pre-echo or post-echo noise artifacts. LossyWAV doesn't exploit temporal masking, so this shouldn't be a problem. Neither should any temporal smearing occur with LossyWAV. Even high bitrate MP3 (e.g. LAME -V0 generally exploits these same effects, but uses the extra bits (compared to say -V2) to increase the margin of safety between the added noise and the masking threshold, and uses some extra bits to encode higher frequencies (increased lowpass filter) that are unlikely to be audible.

Tonal masking. A loud tone at one frequency can mask quieter simultaneous tones at other frequencies. This can be exploited by allowing added noise (e.g. coarser quantization) at adjacent frequencies according to the frequency-dependent masking threshold. While encoders usually make some allowances for frequency response of the playback equipment and room acoustics to vary from the ideal, the use of an extreme EQ setting, filter or tuning/pitch-shifting DSPs (or indeed a tweeter separated by some distance from the midrange speaker) could cause unmasking of certain distortions, which may sound very artificial and unmusical. LossyWAV does not exploit tonal masking at all, nor does it apply a lowpass filter to the audio. Again, high bitrate MP3 like LAME -V0 exploits the masking but sets a greater margin of safety between the calculated masking threshold and the permitted added noise.

The only frequency-dependency in lossyWAV is that the analysis by default ignores frequencies above 16 kHz when determining the noise floor (though this can be over-ridden). These frequencies are typically inaudible in music. For some high-fidelity music, content above 16 kHz is likely to have a low noise floor (e.g. if filtered or beyond the frequency response of equipment used) thereby reducing the bit-depth reduction that lossyWAV could achieve if it didn't ignore the >16 kHz region. There is potential for frequencies >16kHz to be brought into the audible range by pitch-shifting DSPs or to be enhanced dramatically by extreme EQ. The chances are that the noise floor determined with the original 16kHz analysis limit will be sufficient to achieve transparency, and if it's a high noise floor, the masking threshold into the >16kHz band is likely to be pretty high too so even if the noise floor is raised, it's still likely to remain transparent in most situations and to be far better than high bitrate conventional lossy encoding.

Lossy stereo. For some lower-bitrate approaches that aren't aiming for transparency, the stereo image is deliberately degraded to save bits in a less annoying way that other approaches. This generally doesn't apply to high-bitrate lossy, where lossless stereo is usually employed. LossyWAV makes no adjustment to the stereo image. However, lossyWAV now measures the noise floor in each channel separately, so it can independently adjust the bit-depth of the two channels rather than locking them together.
Title: lossyWAV 1.1.0 released.
Post by: Dynamic on 2008-08-26 21:21:34
Another thing that occurs to me, that I don't recall any mention of is that if one uses lossyWAV as one's "lossless archive" from which to listen and transcode, any HDCDs will lose the subcarrier in the LSB that can be used for control of headroom extension (which I believe is audible) and one or two other things (which are possibly inaudible).

If archiving blindly (which is "safe" with pure lossless), it would be good if some software would detect HDCD and possibly pre-emphasis (info in CUEsheet) and optionally pause and issue a warning so that the user could process the PCM appropriately before running lossyWAV on the resulting WAV. E.g. Use SoX for de-emphasis into 24-bit WAV, for example, or using an HDCD decoder (again, ideally to 24-bit WAV) to achieve the "correct" sound.

I don't know where such functions would be best placed, though it may be possible to script something with REACT to use appropriate helper programs (I think there was an hdcd commandline prog in a thread some time ago).
Title: lossyWAV 1.1.0 released.
Post by: cBc on 2009-01-15 00:06:30
hello!

I recently came across a mention of  lossyWAV , read thru the wiki page & here, and was able to set up EAC & ecode the files using it (50% smaller than vanilla FLACS - cool!)
However, I am now trying to use the batchfile mentioned on this post

http://www.hydrogenaudio.org/forums/index....st&p=581571 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=64666&view=findpost&p=581571)

to drop flac files onto (I have latest TAG.exe, lossywav.exe, and flac.exe , along w/ the batchfile.bat in one folder, but dropping files onto the batchfile results in a quick blip of a dos window viewing then nothing.
I'm pretty DOS ignorant, can anyone advise what I need to do to get it working.
To create the batchfile I copied the text into a notepad page & renamed it as a .bat extension

Many thanks!
Title: lossyWAV 1.1.0 released.
Post by: Big_Berny on 2009-01-15 05:22:08
There is a known problem within foobar2000 (although more likely to do with cmd.exe itself) when running an executable within the cmd.exe command line from a path which includes spaces. The suggested fix for this is to enclose the element of the path which contains spaces within double quotation marks ("), e.g. c:\"program files"\directory_where_executable_is\executable_name


IMHO it's simpler to enclose the whole path in quotation marks and it should work too. This way you can just use it always - also if there's no space in the path:
"<Path>"
"c:\program files\directory_where_executable_is\executable_name"

Just my two cents.
Title: lossyWAV 1.1.0 released.
Post by: cBc on 2009-01-16 16:03:21
hello!

I recently came across a mention of  lossyWAV , read thru the wiki page & here, and was able to set up EAC & ecode the files using it (50% smaller than vanilla FLACS - cool!)
However, I am now trying to use the batchfile mentioned on this post

http://www.hydrogenaudio.org/forums/index....st&p=581571 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=64666&view=findpost&p=581571)

to drop flac files onto (I have latest TAG.exe, lossywav.exe, and flac.exe , along w/ the batchfile.bat in one folder, but dropping files onto the batchfile results in a quick blip of a dos window viewing then nothing.
I'm pretty DOS ignorant, can anyone advise what I need to do to get it working.
To create the batchfile I copied the text into a notepad page & renamed it as a .bat extension

Many thanks!



anyone?
Title: lossyWAV 1.1.0 released.
Post by: 2Bdecided on 2009-01-16 16:26:17
cBc,

Maybe no one knows the answer, or maybe it's in the more recent thread (this one is very old)...

http://www.hydrogenaudio.org/forums/index....5499&st=100 (http://www.hydrogenaudio.org/forums/index.php?showtopic=65499&st=100)

Sorry I can't help.

Cheers,
David.
Title: lossyWAV 1.1.0 released.
Post by: Dynamic on 2009-01-17 02:07:13
cBc, you might be able to debug the Batch File by editing it (e.g. in Notepad) and inserting a PAUSE command near the end, which will ask you to press a key to continue, preventing Windows from clearing the window on completion.
Title: lossyWAV 1.1.0 released.
Post by: cBc on 2009-01-18 16:50:30
cBc, you might be able to debug the Batch File by editing it (e.g. in Notepad) and inserting a PAUSE command near the end, which will ask you to press a key to continue, preventing Windows from clearing the window on completion.


Could it be that I don't have the required .exe files in the proper place? Currently they are all in one folder, along with the batchfile.

One of the posts earlier mention having them "in the path", and again my lack of DOS knowledge is hobbling me here. 

Thanks
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2009-01-18 17:00:50
The "path" is a list of directories that DOS will look into, as well as the current directory, when searching for a COM/EXE/BAT file to execute before giving a "not found" error.

The batch file assumes that the executables are in a directory pointed to by the path variable list.

If your flac.exe, tag.exe and lossyWAV.exe and batch file were, for example, in C:\BIN then:
Code: [Select]
@echo off
:repeat
if %1.==. goto end
if exist %1 c:\bin\flac -d %1 --stdout --silent|c:\bin\lossywav - --stdout -P --stdinname %1|c:\bin\flac - -b 512 -o "%~dpn1.lossy.flac" --silent && c:\bin\tag --fromfile %1 "%~dpn1.lossy.flac"
shift
goto repeat
:end
should work....
Title: lossyWAV 1.1.0 released.
Post by: cBc on 2009-01-18 18:28:07
The "path" is a list of directories that DOS will look into, as well as the current directory, when searching for a COM/EXE/BAT file to execute before giving a "not found" error.

The batch file assumes that the executables are in a directory pointed to by the path variable list.

If your flac.exe, tag.exe and lossyWAV.exe and batch file were, for example, in C:\BIN then:
Code: [Select]
@echo off
:repeat
if %1.==. goto end
if exist %1 c:\bin\flac -d %1 --stdout --silent|c:\bin\lossywav - --stdout -P --stdinname %1|c:\bin\flac - -b 512 -o "%~dpn1.lossy.flac" --silent && c:\bin\tag --fromfile %1 "%~dpn1.lossy.flac"
shift
goto repeat
:end
should work....


Nick,

many thanks...worked a treat!! Completely missed the directory callouts in the string. 
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2009-01-18 18:58:16
. Should anyone wish to comment / ask questions / etc. please do it at the end of this thread (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=65499&view=findpost&p=584559).
Title: lossyWAV 1.1.0 released.
Post by: Nick.C on 2009-04-30 14:50:25
lossyWAV 1.1.0c attached to post #1 in this thread. This is a maintenance release as the cause of the WINE incompatibility has been determined.