Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: lossyWAV 1.3.0 Delphi to C++ Translation Thread (Read 158552 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #50
Tested against lossyWAV 1.3.0 delphi version for bit identical output with internal FFT and FFTW.

- standard quality setting: OK (bit identical).

It would be good to split the port into a library, and an app (if this is not being done already). Then other apps can use the library.  Bear in mind that FFTW, being GPL, is not good for libraries; Ooura's fft4g is almost as fast, double-precision, and unrestricted.

I really look forward to try this out on Linux one day.

From the progress reports above, I think that day could be pretty soon!

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #51
- portable quality setting: KO (not bit identical).
Having difficulty replicating this (when the 'fact' chunk is not present, i.e. piped output).

If the 'fact' chunk is present then the file will be two bytes longer than a file processed using the reference Delphi version (the "d" in "lossyWAV 1.3.0d" and a padding byte).
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #52
I have received samples that should allow me to determine the cause of the error. I can confirm that I have duplicated the processing that shows up the discrepancies. The debugging can therefore continue....
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #53
lossyWAV beta 1.3.0e attached to post #1 in this thread.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #54
lossyWAV beta 1.3.0e attached to post #1 in this thread.

All quality profiles tested on one single file resulted bit identical (OK).

The only remaining rouding issues that I´m seeing now are some time differences about hundredth of a second.
This difference shows off in both the wave souce´s time lenght and in several time points for "Detailed bits-to-remove data per channel per codec-block" (-d switch).
As the resulting files are identical this is just a minor display issue.

Example of different time lenght reports from my source test file:
lossyWAV 1.3.0 reports 0:32.26, lossyWAV 1.3.0e reports 0:32.27 and foobar2000 reports 0:32.267.

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #55
Thanks for the corroborative testing. I have been trying to get the times to match but have not expended too much effort there so far.

I am now at a point where I am content that the output is identical to the output of the Delphi reference version.

As previously indicated, I will be implementing a new parameter to ignore-chunk-sizes (in a manner similar to FLAC et al). This will hopefully allow use with RipBot264 for transcode audio tracks.

I am now using Code::Blocks as my IDE with MinGW 4.7.0. There are some very nice compiler switches available to me using C++ that were completely absent in Turbo Delphi, i.e. CPU targets for the compile. When I compile with -march=pentium4 the optimised executable runs approximately 13% faster than a non-optimised version (on my FX-8150) - and faster than the Delphi / IA-32 / x87 version.

Delphi / IA-32/ x87: 50.7 seconds
C++ Vanilla: 52.9 seconds
C++ Pentium4: 46.8 seconds.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #56
why delphi version is faster than c version?

on my Q6600@3GHz c version was noticable faster.

btw. huge thanks for upcoming ignore chunk size switch.

that switch plus basic multithreading and this tool is complete! 

for short stereo track single threading is not a problem but for looong 6ch tracks conversion may take really looong time.

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #57
The Delphi version (i.e. the Delphi version with significant quantities of IA-32 / x87 hand written assembly language included) is faster because of the assembly language content. A plain Delphi compile is *much* slower.

Delphi Vanilla: 4 minutes 3.2 seconds.

I should explain that my standard test (used here and in the post above) is to process my 55 sample test set (12:43.93) using FLAC | lossyWAV | FLAC with --quality extraportable and default adaptive noise shaping.

On the other hand, it may be that the executable simply performs better in an Intel CPU rather than my AMD.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #58
single integer unit in amd fx cpus is really slow in my opinion. i won't be surprised if your fx8150 will not be faster in this task than my old overclocked Q6600

this is another reason why we need some kind of multithreading.

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #59
any news about current progress?

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #60
I've been working on speeding up the code mostly.

Recently I have managed to start the multi-threading re-write, although I have not got very far with this.

The --ignore-chunk-sizes is still on my list. I would propose to handle this the same way as handling foobar2000 output. When piping, foobar2000 sends a RIFF chunk size of -1 (0xFFFFFFFF) and a 'data' chunk size of -1 (0xFFFFFFFF). If RipBot264 were to follow this convention I think it would be best all round.

Code is getting cleaner. FFT code is re-written using operators for complex calculations. Overall, my C++ experience (thus far) has been very satisfying.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #61
unfortunatelly ripbot264 is just a gui for other tools. something like megui. i hope that new version will work with this chain

ffmpeg -> lossywav -> ffmpeg/flac

i can not use flac.exe for decoding because it can't send large wavs via pipe.

how multithreading code will work? proccesing all channels at once or something else?

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #62
I'll work on ffmpeg compatibility.

Multi threading will probably be at the fft calculation level as input and output of processed audio is linear due to sample interleaving.

I'll see how it goes.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #63
lossyWAV beta 1.3.0f attached to post #1 in this thread.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

 

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #64
have you figured out how to accept pipe output from eac3to?

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #65
I will try it out and see where I get to.

Sorry for the lack of communication - I've been busy making the FFT routines thread safe and I've started on the rest of the processing steps. So far, speed has improved a bit.

I've been a bit distracted by the UK Kickstarter for Elite: Dangerous....
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #66
great! don't forget also about broken progress indicator with 4gb+ wavs

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #67
Hello everybody,

I am very new at this forum but I'm very interested in lossywav.
I'd like to mention from a general scientific view that audio compression (lossy/lossless) is a very mind expanding subject. Just the fact that people with the right tools, insight and with help of math/calculus can accomplish such things.

I would like to thank the author Nick.c and all other people contributing in this thread for making this possible.
Certainly from my point of view it is a great accomplishment to be able to shrink to smaller audio files while trying to stay as close as possible to the original.
Thank you very much for creating such a piece of software and please, if you have time and resources, keep up the good work.
I understand that Tycho has been a great contributor or fellow author with the C / C++ version.

I'm still using the Delphi 1.3.0 version, but as I understand the C++ version will have some speed improvements ?

I do have another question about the block size of FLAC. Is the block size of 512 the most optimized size ? Is it possible to use a blocksize of 256 with the extraportable setting in Lossywav or would this be too restricting to let flac store the audio in a correct manner ?

Anyway, thanks again and I will follow the development and this thread of course.

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #68
If you look back at the 1.0 thread (find it via the end of the LossyWAV Wiki entry) I beileve that's where the tests were done to compare different block sizes using FLAC, I believe, as the reference codec.

At that time, testing indicated that 512 seemed to be the sweet spot, giving the flexibility to reduce bits further in short parts where the noise floor allowed. Some things have changed such as adaptive noise shaping, so it's plausible that the sweet spot might shift to 256 or 1024, though probably not very likely.

FLAC and most lossless codecs default to about 4096 sample blocks to get better compression. That's to do with the predictor working well once the first few samples of the block are encoded, but not being allowed to refer back to the last samples of the previous codec block at the start of the next block for reasons of robustness to corruption. This means the first few samples of a block require more bits to encode than the later ones.

Often quietly mastered music (some classical solo stuff) doesn't benefit much from bit-depth reduction in lossyWAV and can actually end up worse off due to the short block length, so the bitrate is increase slightly by lossyWAV.

I think TAK has a lossyWAV optimised mode that works around some of the limitations.
Dynamic – the artist formerly known as DickD

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #69
Hi Jan7887,

You've firstly got to thank 2Bdecided for publishing his method here. Eventually I got started on a Delphi version with lots of help from halb27. Noise shaping was next with a fixed IIR method from SebastianG. Adaptive noise shaping followed with a FIR method from SebastianG.

It's been a great hobby / project and a *huge* thief of time for about five years now - however, I am still enjoying working on the code, albeit in C++ now rather than Delphi / IA32 / x87.

When FFTW is not used, the C++ code is now faster than the assembler optimised Delphi version. When FFTW is used, the Delphi version sneaks ahead. However, with Pentium 4 C++ compiler options selected, the C++ is faster yet again.

I've been working on making the code thread safe to speed up single file processing, but been a bit sidetracked.

I'll publish a new beta soon.

On the block size, you can experiment with the FLAC encoding side of things by changing the blocksize yourself (-b [256/512/1024/etc.]) to see which is optimal for your particular audio.

Nick.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #70
Thank you Dynamic and Nick.C for the fast response.

Also, I shamefully admit, I wasn't fully aware of a wiki page with further tests. Thanks Dynamic for the link with the wiki page.
As you might have noticed I'm rather new here, but good research and googling around can save the day.
Nevertheless I will make sure I have a valid question to ask, I mean, I will do a bit of research on my own before asking questions that I could have searched and find myself.

Thank you 2Bdecided and Halb27 and still Nick.C for making this happen and Tycho for the C++ version, very much appreciated.

Indeed the blocksize is rather a difficult beast to handle. I think that Dynamic was about right that 512 is the sweet spot for block size. I have tried various sizes, but the result is inconclusive to me. sometimes 256 seems to be allright, but anything bigger than 1024 is in my opinion a bit too much, I mean in the resulting flac size.
512 seems to be about the right size for any kind of audio.

Thanks again and cheers and I hope such innovations as Lossywav and also FLOAT16 or PFM as like to call it (Pulse Float Modulation) will continue to exist.

2nd Edit:
(The previous sentence wasn't exactly right. I mean the implementation of Float16 by Nick.C in his other project. Float16 in itself already existed of course. And I called that PFM (Pulse Float Modulation) just for fun. It probably isn't a very correct way naming it thus. Anyway, thanks and I hope I make some sense. English isn't my native language, but I think I manage quite allright.)

Cheers!,
Jan.

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #71
Tycho has kindly started the translation of lossyWAV from Delphi to C++.

Wow! Didn't see that coming...

I'll need to learn it to be able to contribute meaningfully....

If you're interested in book recommendations, try "Accelerated C++" and/or check out http://stackoverflow.com/questions/388242

Kohlrabi, I haven't thought about using github yet.  My initial thought was just to get most of the code to compile, and then upload the code here. Useful help would be that people browsed through it and verified that the .pas and the .h/.cpp files does the same thing, but I wouldn't mind if people worked on it too.

Please put it on github or google code with git or mercurial versioning.
It'll make distributed development so much easier

I have downloaded the Dev-CPP IDE with MinGW 4.6.0.2 - primarily because I wanted a portable mode. Quite liking it and have started "playing" with it to get my head round C++.

If you don't want to use Microsoft tools, try a recent nightly build of the Code::Blocks IDE. But I have to say that debugging via "Visual C++ 2010 Express" is pretty sweet on a windows box.

The C++ executable is *much* bigger than the Delphi one (1716kiB vs. 168kiB).

Yeah. That's what you get for using MinGW on Windows. You might also want to check compiler options (not to include things like "-g" for the release build, for example). Finally, try "strip -s yourbinary.exe".

I guess that's also a good chance to learn something about CMake. With CMake as "build file generator" it does not matter what Toolchain/IDE developers use since they can use CMake to convert the cross-platform description "CMakeLists.txt" to a GNU Makefile or a VS2010 project file, etc etc etc. So, the nice thing about it is that one just has to maintain this cross-platform meta build script. CMake is still on my list of things to get deeper into. Unfortunately, my experience with it is rather limited at the moment.

I am in the process of implementing a radix-4 IFFT while optimising the other FFT routines for speed.

Did you profile the whole process? I would not expect the FFT to be a bottleneck here. The GNU toolchain offers some instrumentation tools you could try -- like "gprof" (GNU profiler). In order to make this work, the code needs to be compiled with the "-pg" switch if I remember correctly.

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #72
Hi Sebastian,

I was kick-started into learning C++ when Tycho so kindly started the conversion. Eventually, I started to use Code::Blocks - it's really good and I'm now using MinGW 4.7.2.

Some hair tearing and a significant amount of bug-hunting later, we have bit-exact output and some code improvements. I took the opportunity to introduce some of the optimisations that had already been included in TransPCM (same codebase as lossyWAV).

I'm working on making the processing steps thread safe at the moment.

On the FFT front, I've now implemented up to Radix-16 forward and reverse FFT code. The larger radices do make quite a significant difference in improving the processing speed. I have been using the code profiler - so simple from within Code::Blocks.

Once I have finished hacking about the core variable setup to allow threaded operation, I fully intend to use github or Google Code to allow other interested parties to contribute.

I got a bit side-tracked (again) with Float16 audio - saratoga kindly included Float16 compatibility within Rockbox. foobar2000 has been compatible for over a year now. ! I found some code that codes and decodes Float16 to float without any if/then statements - quite quick really.

Nick.



lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #73
@SebastianG

Thank you very much for all your work on Adaptive Noiseshaping as implemented in Lossywav (delphi version). I saw I completely forgot to thank you 

@Nick.C

Good luck with the programming on both Lossywav and the Float16 project, it is very much appreciated.
I sometimes get a comment about why I further shrink the flac files. "Why don't you just use the original flacs?" I usually get.
Well, on a desktop computer it is all fine and dandy but I'd like to listen to flac or other lossless formats on portable devices.
And as cheap as storage is, it is still not very practical to have original flacs on those devices, at least in my opinion.

Besides the subject but I'm also very interested in the Opus codec from xiph. Call me crazy, but on portable devices like phones, the opus files are quite astonishing in sound, considering the low bitrate (usually between 64 and 96 kbps for music).
Anyway, I'm going offtopic here.

lossyWAV 1.3.0 Delphi to C++ Translation Thread

Reply #74
Opus is very good indeed. If I was looking for very low bitrate audio for my Rockboxed Sansa Fuze I would certainly use that. Even more impressive is that it is now a standard!
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)