Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: lossyWAV 1.2.0 Development Thread (Read 305561 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

lossyWAV 1.2.0 Development Thread

Reply #75
Thanks.

Christopher's decoder outputs 24-bit WAV files (for decoded HDCD output).

In fact, it outputs 24-bit WAV files with 4 lower bits set to zero.

I still don't understand, can (or should) i somehow instruct lossyWAV that 4 lower bits are zero, and will it make any difference? In other words, does lossyWAV support wierd bitsPerSample values, or only fixed set of 16/24?
CUETools 2.1.6

lossyWAV 1.2.0 Development Thread

Reply #76
Any low bits that are already zero will remain zero. What lossy wav does is, when appropriate, zero out additional lower bits to make lossless compression more effective.

lossyWAV 1.2.0 Development Thread

Reply #77
Any low bits that are already zero will remain zero. What lossy wav does is, when appropriate, zero out additional lower bits to make lossless compression more effective.

Thanks a lot, i really appreciate the answers but those are the things that i already know. What i wanted to know, is does lossyWAV support 20-bit input natively.

Let me explain:
If i take 16 bit wav, and turn in into 24 bit wav by adding 8 zero bits, then process with lossyWAV, then turn it in 16 bit wav again by removing 8 lower bits, i'm not sure i will get the same result as if i processed the 16 bit input directly.

16 bit WAV -> add 8 bits -> lossyWAV -> remove 8 bits == 16 bit WAV -> lossyWAV?

If yes, than it makes no difference if the input is 20 bit or 24 bit with 4 wasted bits.
But if no, than it makes a difference if 20 bit input is supported natively.
CUETools 2.1.6

lossyWAV 1.2.0 Development Thread

Reply #78
A 20-bit sample in a 24-bit container will already be "left justified" as required by the WAV specification, i.e. effectively multiplied by 16, therefore lowest 4 bits are going to be zero.

16>24>lossyWAV>16 will probably not equal 16>lossyWAV, additionally, if the lossyWAV>16 step includes dither then the carefully zeroed lsb's will disappear and the wasted-bits compression advantage will be lost.

[edit] lossyWAV supports 9-bit to 32-bit integer samples in 16, 24 and 32-bit signed integer "containers", exactly the same as the WAV specification. [/edit]
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

 

lossyWAV 1.2.0 Development Thread

Reply #79
A 20-bit sample in a 24-bit container will already be "left justified" as required by the WAV specification, i.e. effectively multiplied by 16, therefore lowest 4 bits are going to be zero.

16>24>lossyWAV>16 will probably not equal 16>lossyWAV, additionally, if the lossyWAV>16 step includes dither then the carefully zeroed lsb's will disappear and the wasted-bits compression advantage will be lost.

[edit] lossyWAV supports 9-bit to 32-bit integer samples in 16, 24 and 32-bit signed integer "containers", exactly the same as the WAV specification. [/edit]


Thanks.
Ok, i ported lossyWAV to C#, and will try to fool around with 20-bit input.
I can also send my code when i tidy it up a little, if you are interested.
CUETools 2.1.6

lossyWAV 1.2.0 Development Thread

Reply #80
Yes, thanks.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)


lossyWAV 1.2.0 Development Thread

Reply #82
Processing 1-hour wav (16/44):
LossyWav.exe: 73 sec;
LossyWavSlow.exe: 239 sec;
LossyWavSharp.exe: 144 sec;

Cute. And yes, LossyWavSharp and LossyWavSlow produced identical output.

lossyWAV 1.2.0 Development Thread

Reply #83
Here it is:

http://www.hydrogenaudio.org/forums/index....showtopic=67418

Tested to be bit identical with 1.1.1e on at least one file, at least with default quality setting
Gregory,

Many thanks for taking the time to port the core functionality of lossyWAV from Delphi to C#. The speed of this first version is impressive!

Looking at the code for the FFT functions - are you using a trick to only have to create one array of A(Re,Im) values by shifting the index by the bitlength difference of the FFT length in question and the maximum FFT bitlength?

Rógerio Brito (rbrito on these forums) is interested in a C port - how different is C# to C (I *really* don't know anything about C)?

Well done again!

Nick.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.2.0 Development Thread

Reply #84
Looking at the code for the FFT functions - are you using a trick to only have to create one array of A(Re,Im) values by shifting the index by the bitlength difference of the FFT length in question and the maximum FFT bitlength?

Yep. I'm also using a similar trick for reversedbits table.

Rógerio Brito (rbrito on these forums) is interested in a C port - how different is C# to C (I *really* don't know anything about C)?

It's still a lot of work, but about twice easier now - the basic operators are the same in C and C#, as are the basic concepts, such as arrays indexing. Most of the problems i had at first were caused by pascal-style arrays indexing (often starting at 1 instead of 0). With these obstacles removed, it's now only a matter of time to create a C port. Anyone who is familiar with C can read C# code. I hope someone will find a time to do this

UPD: found a small bug in my WriteChunk function from AudioCodecsDotNet.cs, the chunks were not aligned on the word boundary. Corrected version is in CUETools SVN.
CUETools 2.1.6

lossyWAV 1.2.0 Development Thread

Reply #85
I added lossyWAV support to CUETools.

New CUETools also include command line lossyWAV utility.

Binaries can be found here, and source code is here.

LossyWAVDotNet is a C# class library which implements the algorithm,
AudioCodecsDotNet is a C# WAV reader/writer library
LossyWAVSharp is a command line utility which uses them.

Those 3 projects can be built independently of other projects in CUETools SVN.

Changes to the first version: class library interface improved, fft very slightly optimized and command line utility supports some basic command line options.
CUETools 2.1.6

lossyWAV 1.2.0 Development Thread

Reply #86
It looks like I should "hang up" Delphi and get my head around C....

Well done again!
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.2.0 Development Thread

Reply #87
Did people give up on ABX'ing LossyWAV? I didn't see usual ABX tests anymore. I hope it's that good!

lossyWAV 1.2.0 Development Thread

Reply #88
--standard (-S or -q 5) appears, from all of the individual testers' responses appears to be transparent.

--portable (-P or -q 2.5) also appears to be transparent from the limited testing done on it so far.

My HTPC local copy of my full collection is --portable lossyFLAC, and comes in at about 372kbps, down from 880kbps for lossless FLAC. I have not yet heard an artefact in that collection - however I am not an ABX tester....
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.2.0 Development Thread

Reply #89
Did people give up on ABX'ing LossyWAV? I didn't see usual ABX tests anymore. I hope it's that good!

New ears are always welcome!
I did a lot of ABX testing before, but now that lossyWAV has been in a mature state since a long time there is no sense to continue testing. To me and those samples I tested lossyWAV is perfect. But this doesn't mean that in the universe of music there may be problems showing up. That's why it would be nice if we had new testers.

In case you like to distribute I suggest you do it with the --portable quality. In theory at least there should be samples with --portable being not transparent. If such samples should show up it is important to check whether the --standard quality is fine with them or not.
lame3995o -Q1.7 --lowpass 17

lossyWAV 1.2.0 Development Thread

Reply #90
I gave up ABX testing it. It was really difficult on --portable, even though I came very close on a few samples in my transcoding test.

lossyWAV 1.2.0 Development Thread

Reply #91
LossyWAVDotNet is a C# class library which implements the algorithm,
AudioCodecsDotNet is a C# WAV reader/writer library
LossyWAVSharp is a command line utility which uses them.


I want to add that i've started a Java port from those C# sources.
The performance will be lower, for sure, and i'll have to recheck limits here and there ( for java, everything is signed, unsigned values do not exist. multi-level arrays are not matrixes, but arrays with references to other arrays. Casting can not be freely done like in C/C++/C#.)

I am trying to write it in the most correct way, as being more a reference than a best-of. (I've extracted the FFT parts into its own class, for example. I'm making constants in its own file, I'm removing redundant information and try to simplify the workflow).

I guess i will need still two or three days to complete the port. I'll post it here when done.

lossyWAV 1.2.0 Development Thread

Reply #92
Thanks for your effort [JAZ], it's appreciated.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.2.0 Development Thread

Reply #93
Some updates on the port to Java:

I have a working code already, based mostly on the C# sources and to a lesser extent to the delphi sources or the web when something didn't look quite right/clear enough.

I don't have yet a .jar that could be tested, but i'll look into making one soon.

Currently:

It is a lot slower than the C# sources. ( by two orders of magnitude). This is because i've removed some of the lookup tables and implemented the FFT methods as they would be by the book.
Of course, i am in the process to add those tables back, in a way that could reflect what they really are and why they are there.

Another thing that i've discovered so far is that while the output seems to produce similar results, it does not produce the same ones than lossyWavSharp. 
I've processed a .wav with both programs, and they play fine in foobar, but the one generated by jLossyWav, if encoded with .flac, is smaller.
I have no idea yet why and if this is good or bad. I hope to get a clearer idea after finishing the source separation.

Code: [Select]
jCD1_02.wav.lossy.wav: wrote 995611 bytes, ratio=0,342

CD1_02.wav.lossy.wav: wrote 1026802 bytes, ratio=0,353

lossyWAV 1.2.0 Development Thread

Reply #94
Sounds good [JAZ] - keep up the good work!

The lookup tables, especially in the FFT save a lot of time.

I have used Gregory's "trick" of shifting the lookup index by the appropriate number of bits to reduce (by about half) the memory requirements of some of the lookup tables in what will be lossyWAV 1.1.1f (Delphi).
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.2.0 Development Thread

Reply #95
Some updates on the port to Java:

I have a working code already, based mostly on the C# sources and to a lesser extent to the delphi sources or the web when something didn't look quite right/clear enough.

I don't have yet a .jar that could be tested, but i'll look into making one soon.

Currently:

It is a lot slower than the C# sources. ( by two orders of magnitude). This is because i've removed some of the lookup tables and implemented the FFT methods as they would be by the book.
Of course, i am in the process to add those tables back, in a way that could reflect what they really are and why they are there.

Another thing that i've discovered so far is that while the output seems to produce similar results, it does not produce the same ones than lossyWavSharp. 
I've processed a .wav with both programs, and they play fine in foobar, but the one generated by jLossyWav, if encoded with .flac, is smaller.
I have no idea yet why and if this is good or bad. I hope to get a clearer idea after finishing the source separation.

Code: [Select]
jCD1_02.wav.lossy.wav: wrote 995611 bytes, ratio=0,342

CD1_02.wav.lossy.wav: wrote 1026802 bytes, ratio=0,353

To my experience, FFT is the most CPU consuming part of the algorithm, practicly the only one that should be really optimized.

At the early stages i also had quite different results from the delphi project. I ended up debugging them simultaniously and comparing internal values at each step. Tedious work, but had to be done. I recently found out that even to this day there are audio files, on which C# version removes one bit more or one bit less on a block or two. I blame the rounding errors, due to the different floating point precision. Delphi used float, double and extended values. C# port only uses float and double. I tried to preserve the float values, even though the temptation was strong to do everything in double. Changing the precision changes the algorithm output. Not that there's something bad about it, the higher the precision - the more accurate the results are, obviously, but in that case i would not be able to compare result to the reference delphi implementation.

I don't really think there can be 100% match, each language does it's arithmetic slightly different when it comes to rounding and such, but the difference in ratio that you quoted (0,342 vs 0,353) indicates that it's not the only issue here right now.
CUETools 2.1.6

lossyWAV 1.2.0 Development Thread

Reply #96
The pure Delphi and Delphi / IA-32 / x87 versions of 1.1.1e produce *slightly* different results with my 54 test sample set (less than 512 bytes in 30MBytes of resultant FLAC).
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)


lossyWAV 1.2.0 Development Thread

Reply #98
Thanks again [JAZ] - more potential platforms for lossyWAV processing is certainly a good thing.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.2.0 Development Thread

Reply #99
I too would like 32-bit float support, if that's even possible. My reasons are weird: I'm planning on releasing an EP at some point in the not-so-distant future. I want to amplify the release at +15dB past its ReplayGain value to protest the Loudness War. ReplayGain users won't run into the issue, naturally, just everyone else. However, bitrates get kinda crazy with 32-bit float. So I want to use LossyWAV to chop the bitrate down to something sane for release in FLAC.