Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: lossyWAV 1.1.0 released. (Read 142902 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

lossyWAV 1.1.0 released.

Reply #75
Is that the reason why material with 96 kHz is currently reduced to the same filesize as material with 48 kHz? Are higher sample rates currently treated like if they are played back at a lower speed/rate, thus lowering the frequency spectrum accordingly, which results in a wrong calculation of how many bits can be removed?
Wrong might be a little strong - the sample rate is used correctly where it needs to be but the codec-block-size equates to less time and the granularity of the spreading function will be higher and possibly a bit too coarse.

Does that mean that if noise shaping is disabled less bits are removed? That would reassure me.
Exactly the same (except if noise-shaping causes clips) bits-to-remove values will be achieved for noise shaping 1 and 0.

lossyWAV 1.1.0b attached to post #1 in this thread:
  • implementation of increasing fft length for increasing sample rate;
  • improved logfile output and --detail output;
  • reference threshold constants for rectangular dither and triangular dither have been calculated so added noise should be the same for dither off and any dither level between 0 and 1 - the number of bits-to-remove will however reduce with "increasing" dither.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #76
Is it intended that the dithering noise is also shaped? A frequency analysis reveals that it's also mildly shaped. I always thought that rectangular or triangular dither corresponds to white noise.

I also wonder if dithering is really that useful. Look at the graph, it shows AC/DC - Fire Your Guns:



Wouldn't you rather use -q 3 without dithering than -q 2.5 with dithering when the bitrate is (nearly) identical?

lossyWAV 1.1.0 released.

Reply #77
Is it intended that the dithering noise is also shaped? A frequency analysis reveals that it's also mildly shaped. I always thought that rectangular or triangular dither corresponds to white noise.
Which version of lossyWAV are you using? I ask as the fix to ensure that added noise remains the same when dither is applied has only been introduced at 1.1.0b.

Also, with respect to the triangular dither, I have used a method (discussed on these forums previously) which uses 1 new random number per sample and recycles the previous random number, i.e. tpdf_random = new_random - old_random; (random values in the range +/-0.5). In my implementation and to achieve intermediate dither types I have used: tpdf_random = new_random * dither_type_multiplier - old_random; (dither_type_multiplier in the range 0..1). This may have had an effect on the shape of the dither.

[edit] One other thing: --dither 0 <> no dither. If the dither parameter is used then dither will be applied, somewhere between rectangular and triangular, depending on the parameter value after --dither, i.e. --dither 0 = rectangular dither; --dither 1 = triangular dither; --dither 0.5 = something in between rectangular and triangular dither. [/edit]
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #78
I'm using the current version 1.1.0b.

I don't know if it's good to reuse old random values. Why not just use two independent random values? Is the performance impact too high? As it's not used by default I don't think that would be much of an issue.

I have redone the above graph and now differentiated correctly between dither off and dither 0. I also included ReplayGain values to illustrate the theoretically perceived loudness of the noise.

lossyWAV 1.1.0 released.

Reply #79
I'm using the current version 1.1.0b.

I don't know if it's good to reuse old random values. Why not just use two independent random values? Is the performance impact too high? As it's not used by default I don't think that would be much of an issue.

I have redone the above graph and now differentiated correctly between dither off and dither 0. I also included ReplayGain values to illustrate the theoretically perceived loudness of the noise.
I'll find my pest control gear and go bug-hunting - the -s 0 -D 1 curve looks suspiciously like it has been noise-shaped.

There is no real performance hit using two independent random values (especially as I have changed from a fibonacci based RNG to a multiply-with-carry RNG) - I'll have a look at that too.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #80
Of course the dither should/will be noise shaped. It should/will sit next to the quantisation stage within the noise shaping loop.

Current random number minus previous random number is a well accepted way of getting slightly high passed dither.

Hancoque, no one is suggesting that you should use dither. It's there for people who incorrectly think that they need it - not for those who know that they don't (nor for those that don't know or care!).

Cheers,
David.

lossyWAV 1.1.0 released.

Reply #81
Of course the dither should/will be noise shaped. It should/will sit next to the quantisation stage within the noise shaping loop.

Current random number minus previous random number is a well accepted way of getting slightly high passed dither.

Hancoque, no one is suggesting that you should use dither. It's there for people who incorrectly think that they need it - not for those who know that they don't (nor for those that don't know or care!).

Cheers,
David.
1) Dither is added pre noise shaping;
2) Thanks - it will remain so;
3) I'll leave the option to dither up to the user.

Thanks David.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #82
I don't plan to use dithering either. I was just curious.

Now that higher sample rates are officially supported, do I understand it right that they require a higher codec block size?

I will modify 1.1.0 to increase the FFT lengths at 69.08kHz, 138.15kHz and 276.3kHz, i.e. 64 to 128 to 256 and 512 samples respectively and correspondingly for the other lengths with a similar increase in codec-block length, 512 to 1024 to 2048 to 4096 samples.
So, does this equate to the following sample rate to codec block size analogy?

44.1/48 kHz ->  512
88.2/96 kHz -> 1024
176.4/192 kHz -> 2048

lossyWAV 1.1.0 released.

Reply #83
I started doing some personal listening and file size tests with 1.1.0b and I must say I am very impressed.  The idea behind it all was quite something and now we have it implemented with TAK, FLAC and Wavpack.  I use TAK on my computers because that is meant for foobar2000 playback and I use mp3 for my DAP because it is compatible and can be mp3gain'd.

So far, I have been using --extreme and I am in love with the results and it looks like I might switch the all lossless library to the lossyTAK library and if my calculation is correct I should expect the lossyTAK library to be 61% of the size of the current library at -p5.  So roughly 305gb down to 186gb and have yet to notice anything that could be a problem sample.  Of course, I still have the original image files of the CDs ripped using EAC safely tucked away on DVDs so if I do replace the entire library and a problem is discovered it is quite easy to recover from with the DVD, all done to increase disc longevity and to maintain perfect rips.

Good job, keep it up.

On a theoretical note,

This might move the lossless formats even more into the mainstream although the end file is not technically lossless but it so high quality of an output that the problem samples that exist in the lossy formats do not exist here.

I am all for increasing the overall usage of lossless over lossy and with the lossyWAV a new field of battle has been opened up that might sway those who think lossless is TOO big for use to the lossless side.
Zune 80, Tak -p4 audio library, Lossless=Choice


lossyWAV 1.1.0 released.

Reply #85
But in-loop or out-of-loop? Traditionally it should be in-loop, like this...

http://www.hydrogenaudio.org/forums/index....st&p=580394

Cheers,
David.
Sorry, not clear enough - dither is added in the middle of the noise shaping process, just before rounding.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #86
continued, about bat files and tagging, this oneliner seems to be working;

lossyflac.bat
Code: [Select]
flac -d %1 --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o %~p1%~n1.lossy.flac --silent && tag --fromfile %1 %~p1%~n1.lossy.flac


requirements: flac, lossywav and tag somewhere on your %path%

example usage: lossyflac file.flac will return file.lossy.flac (with all the tags from file.flac)

p.s. the usual warning about spaces and batch files on win.

edit1: slight fix with paths, added %~p1
edit2: became oneliner
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

lossyWAV 1.1.0 released.

Reply #87
continued, about bat files and tagging, this two liner seems to be working;

lossyflac.bat
Code: [Select]
flac -d %1 --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o %~n1.lossy.flac --silent
tag --fromfile %1 %~n1.lossy.flac
...
Very, very elegant - now no need to use foobar2000 for the processing. You could put both lines on one line with a && in between .
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #88
Quote
You could put both lines on one line with a && in between

any benefits from that? or do we want oneliner? 
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

lossyWAV 1.1.0 released.

Reply #89
Quote
You could put both lines on one line with a && in between
any benefits from that? or do we want oneliner? 
Minimalist, I am a fan of one-line code where it can be done.... The final batch file would be:
Code: [Select]
@flac -d %1 --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o %~p1%~n1.lossy.flac --silent && tag --fromfile %1 %~p1%~n1.lossy.flac
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #90
One minor benefit:  As you are using && (as opposed to &) the call to Tag will only be made if the previous command executed successfully.
I'm on a horse.

lossyWAV 1.1.0 released.

Reply #91
fixed the post - 'downgraded' twoliner to oneliner  http://www.hydrogenaudio.org/forums/index....st&p=581437
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

lossyWAV 1.1.0 released.

Reply #92
Nice batch file. But if the specified input file does not exist, this results in a crash of lossywav.exe
Windows XP SP2, lossywav 1.1.0b

Code: [Select]
F:\lossywavtest\>flac -d doesnotexist.flac --stdout --silent  | lossy
wav - --stdout -P  | flac - -b 512 -o doesnotexist.lossy.flac --silent

doesnotexist.flac: ERROR initializing decoder
                   init status = FLAC__STREAM_DECODER_INIT_STATUS_ERROR_OPENING_
FlossyWAV 1.1.0b, Copyright (C) 2007,2008 Nick Currie. Copyleft.
This is free software under the GNU GPLv3+ license; There is NO WARRANTY, to
the extent permitted by law. <http://www.gnu.org/licenses/> for details.
ILE

An error occurred opening the input file; it is likely that it does not exist
or is not readable.
ERROR: for encoding a raw file you must specify a value for --endian, --sign, --
channels, --bps, and --sample-rate
Type "flac" for a usage summary or "flac --help" for all options

F:\lossywavtest\>tag --fromfile doesnotexist.flac doesnotexist.lossy.
flac
Tag - Automatic Tag from filename      Copyright (c) 2002-2003 Case
Version 2.0.39b, Compiled 2003-04-11

File not found: 'doesnotexist.lossy.flac'
fromfile: Can't open file 'doesnotexist.flac'.

lossyWAV 1.1.0 released.

Reply #93
Nice batch file. But if the specified input file does not exist, this results in a crash of lossywav.exe
This could probably be dealt with
Code: [Select]
@if exist "%1" flac -d "%1" --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o "%~n1.lossy.flac" --silent && tag --fromfile "%1" "%~n1.lossy.flac"
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #94
Thanks Nick.C, that fixed the error 

One more small issue (not lossywav related) is the fact that TAG does not copy embedded cover art.

edit:
I found out I was using an old version of TAG (2.0.39b) and updated to 2.0.52, but that version did not copy embedded cover art either.

But this issue should not be discussed here, rather in Changes Made to TAG.EXE, Tags from text file & OptimFrog support

lossyWAV 1.1.0 released.

Reply #95
Thanks Nick.C, that fixed the error 
This will process multiple files and place the result in the same directory as the original.
Code: [Select]
@echo off
:repeat
if %1.==. goto end
if exist %1 flac -d %1 --stdout --silent|lossywav - --stdout -P --stdinname %1|flac - -b 512 -o "%~dpn1.lossy.flac" --silent && tag --fromfile %1 "%~dpn1.lossy.flac"
shift
goto repeat
:end
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #96
That one works great when you pass  one or more filenames, but when I tried with *.flac  I got a crash of LossyWav.exe again.
It seems that we need a construction with a FOR (%%I) IN %1 DO loop here to handle wild cards.

This command line does the job:
Code: [Select]
FOR %X in (*.flac) DO lossyflac.bat "%X"


edit: quotes added, thanks again Nick.C

lossyWAV 1.1.0 released.

Reply #97
my usual approach is to do one batch that handles one file (action)

and 2nd batch that loops over the first one, (loopy)

one way or the other, depending on the usage. That way you can keep your action bat short and dumb and leave all the file management logic to the loopy one.
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

lossyWAV 1.1.0 released.

Reply #98
That one works great when you pass  one or more filenames, but when I tried with *.flac  I got a crash of LossyWav.exe again.
It seems that we need a construction with a FOR (%%I) IN %1 DO loop here to handle wild cards.

This command line does the job:
Code: [Select]
FOR %X in (*.flac) DO lossyflac.bat %X

But this one skips filenames that include spaces.
If you enclose the second %X in double quotes it will handle the files properly.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 released.

Reply #99
Quote: Nick.C
"My intention is to understand and implement SebastianG's new noise shaping method, but for that I will also have to introduce / find a PSY model of some kind."

Do you have any clue of how long this will take ? 3 months, 6 months or a year ?
Is SebastianG giving you any accelerated private lessons ?
It's not that I want to hurry you  or being rude in any way  , but I care a lot for lossywav ... it's already my favorite lossy codec ... & I plan to convert tera of lossless to Lossy|Tak -P|-p2e ... (without lossless backup) so I care a lot for this new noise shaping method if it can make me save some kbps (& also for the new special Tak setting for lossywav  )

without a 1.2.0 development thread I am asking myself everyday:
1: is it a TODO thing that is already actively worked on in the shadows. 
2: is it a TODO thing that is just an idea. 
3: are you in vacation with wife & kids. 
so I'd rather simply ask ...

as I have been disapointed by vaporware feature from Christopher 'Monty' Montgomery in the past ... I am very suspicious about open source developers claims ... so excuse me if I sound rude I just want to test your determination to get lossywav to the max of its possible efficiency. (I don't want the "new noise shaping method" to become the "bitrate peeling" of lossywav)

I know you just released V1.1.0 a month ago & that I shouldn't already be longing for more ... but now that you are a developer you'll have to learn that end-users are relentless vampires !!! LOL