Skip to main content

Topic: Have a working 'expander' based on DolbyA (not same design) -- works well. (Read 1908 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • jsdyson
  • [*][*]
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #25
As promised -- this is the source code for the 4 band expander, with built binaries for Linux.  I think that it is all there.  The only bugaboo might be that I used zip to put the files together instead of my normal tar Jcvf...   I don't normally use zip...   I do get real noise reduction, and if you use this -- the --info switch can really help understand what is going on (and you can see the gains move all over the place.)  Yea, the display is slow, but good enough for me to be able to debug the code.

At least, I try to follow through with my promises :-).

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #26
Many thanks for your persistence and intellectual curiosity, John! While I do not have a Linux machine at the moment, your efforts are no less appreciated.

I take it that with level matching, this can be used to properly decode raw Dolby A audio -- useful for anyone transferring such material to the digital domain without a hardware decoder?

  • jsdyson
  • [*][*]
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #27
Regarding not having a Linux machine....   I think that I can build a Windows64 binary pretty quickly.  Giveme a day or so, but maybe as quick as an hour or so -- I can PROBABLY produce a Windows64 binary -- I built a Windows64 version of this program a few weeks ago -- so it should work.   So, if I don't have it done in a day, there was a problem.   Most likely, I'll be able to post a Windows64 command line binary (along with some needed .dlls) -- probably in a .zip file in a few hours.

Regarding being able to decode DolbyA -- I cannot make a promise, but all of the parameters match DolbyA fairly exactly -- including careful interpretation of whatever hard copy materials (e.g. maybe even the schematic) so as to emulate the smallest level of transient behavior.   The actual gain curve might be off a little -- again, I am an EE and can interpret very subtle discrete designs to a pretty good precision -- but there are so darned many variables -- including the selected components.   One thing that I can promise -- the 'decoder' does do a very good, gentle 'uncompression' of a lot of available material.

Again -- this decoder is VERY GENTLE -- and in some cases might disappoint because it is so very gentle (10dB gain change over a 40dB gain range on input) -- but DolbyA is like that.   Also, the decoder does have a slight possibility of increasing low frequency distortion -- the detector is a full wave linear (not squared) detector, has a fairly quick attack time and a release time that just barely handles low audio frequencies.  So, if the signal had already been distorted by a fast compressor, then this decoder can benefit the signal by undoing some of the distortion.   If the signal has not already been compressed, it can (and sometimes does) sound like hell -- as expected.

I have a few bugs to deal with right now -- my demo scripts had an error, where the temporary files were prematurely converted to 44.1k, so wasn't getting the most benefit from the quality of the various components.   I am correcting the scripts so that everything stays at 96k (or whatever frequency is selected) until the final, mp3 conversion stage.

After this bug fix above, then I'll see about producing the Win64 binary.

John dyson

  • jsdyson
  • [*][*]
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #28
Good news (I hope).  I think that I now have a working Windows64 binary -- command line only.  I don't do GUI code, so if someone wants to do a simple GUI and repackage for sale -- just give me credit and be nice (of course, if someone distributes a commecial copy, they should probably compile it on their own so that they can maintain the codebase.)

I tried to leave a simple README.txt to help someone to use the program.  Included in the zip are some cygwin dlls necessary to help the program to work.  I built all of the fancy support stuff into the binary -- statically linked it in from the cygwin distribution so that you don't have to deal with a myriad of shared libs (actually, not that many -- perhaps only a few others like the standard library stuff.)

Good luck...  I do promise that I didn't place anything evil into the .zip, and I created the .zip under Linux, so it is probably clean...

John Dyson

  • jsdyson
  • [*][*]
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #29
For anyone doing listening tests -- I MADE A MISTAKE IN THE pseudo-DolbyA decoder...  If you look at the source, you'll notice two filters that are commented out in the main processing/filtering class.   Those filters should NOT have been commented out, therefor there is a bit of edginess that is fixed by reinstalling the filters.   The demos will be fixed by EOD today (16nov2017.)  I'll also post a description of the fix so that people with source code cfan fix the problem.  (IT IS EASY TO FIX, JUST remove the // comment out, and recompile.)   I'll recompile for the Windows people also.

  • john33
  • [*][*][*][*][*]
  • Developer
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #30
Thanks, John, looking forward to having a bit of a play with this. ;)
John
----------------------------------------------------------------
My compiles and utilities are at http://www.rarewares.org/

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #31
I have not tried it myself but I just want to add that this is really interesting and also a very big thank you for right away supporting Linux.

  • jsdyson
  • [*][*]
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #32
Good news -- I have the corrected source ready -- and the binaries for Linux reside in the zip file -- they are the latest.  The actual patch is this:

In the file 'audioproc.cpp', at approx line 1410 (plus or minus) , there should be a code sequence like this:
        //
        // Slow down the gain changes a little (help to avoid aliasing)
        //
        pgainl = dfiltlA(pgainl);
        pgainr = dfiltrA(pgainr);

Unfortunately, during debugging/optimizing I had commented out those two lines.  It was wrong to have commented out those lines, additionally I just made a change to the filter parameters to remove just a bit more of the aliasing.  These filters are not critical for normal situations, but material with lots of MF/HF content (like ABBA) will cause the expander to produce a harsh edge on the sound.  It is exactly this kind of thing (not filtering out potential aliasing) that can make computer processing using conventional compressor/limiter design sound harsh.  (I finally noticed this harshness when running the music through my multiband finalizer -- and it is VERY VERY GOOD at bringing out faults in digital material.)  When I started chasing down the reason for the terrible sound (coming out of my finalizer -- and had realized that the signal was being disrupted in the simple 4 band expander.)  The more complex 'recovery' software doesn't have this problem - it has a much stronger solution to the alaising problem, along with this simple one.

So, throw away the old attachment, and only use this one.  The abba files on the demo repository have been corrected so far up to abba-119-waterloo.  I am going to put the rest of the abba demos aside, and work on the Sergio Mendez examples next (they are almost a religious experience in sound!!!)

I have attached the corrected code, and the zip file contains the built versions of the programs for Linux.  The windows64 version will come out soon.  The previous windows64 version DOES WORK, but it also has that harshness that I'll be fixing in a few hours.

SORRY -- REALLY, REALLY for the mistake.  I am mistakenly not using my professional discipline when working on this stuff. but I hesitate to make to much of a bureaucracy -- maybe it is time to do so.

John Dyson

  • mycroft
  • [*]
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #33
I do not get it.

Why are you comparing mp3 with mp3?

In source code I see script makemp3 which converts wav to mp3.

Comparing lossy transcoding with another lossy transcoding is futile.

This software could be more scientific if you actually compare lossless with lossless.

  • jsdyson
  • [*][*]
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #34
The processor code (the da-avx or da-*** program)  (the 4 band one) basically (in the actual program) does 32bit floating or 16bit signed integer wav file I/O -- 44.1k, 48k, 96k or 192k.   The only reason for the flac/mp3 file conversions is for convienience -- there is also a script that allows playing the music directly in realtime.  MY USE OF MP3 ON SIMPLE DEMOS IS DUE TO LIMITED SPACE.   I do use the best reasonable conversion params for the MP3.   Flac 96k/24bit available by arrangements if you really need.  24bit/96k or floating-point/96k are my native formats for doing work.

So, nothing severely distorting is going on.  MP3 on external demo stuff is out of necessity  -- but the conversions are usually well above 200k.  For convenience, I supplied flac file and mp3 file conversion scripts.  If the makemp3 script is missing, then that is my fault -- I have a whole series of scripts that do conversion IN/OUT of the pseudo DolbyA (4 band) processor.   I'll check to make sure that the makemp3 script exists -- it is up to you to make sure that you have sox (a very common audio tool) and lame (one of the better mp3 converters.)  THE ONLY REASON FOR MP3 over OTHER LOSSY SCHEMES IS IT BEING COMMON.   I prefer opus, and usually use it at 258k or 384k.  Flac is essentially perfect, and is what I use for high quality archives (usually 96k/24bit.)

Using the program in its most basic mode -- think of this:
1)  You have a piece of over-compressed music -- youtube stuff is often a very good example.  However, a lot of commercial stuff is also over compressed (refer to the originals on my repository -- directly from the source that I got them modulo a moderate/high bitrate mp3 conversion.   Only reason for using mp3 on demos is a lack of space.   I can provide people .flac 24bit 96k to anyone who asks.
2) You want something a little less 'obnoxious' sounding, but mostly just want to simple tool without lots of tweaks (the tool has one tweak that isn't terribly sensitive.

HERE IS WHAT YOU CAN DO on LINUX (or if you have a good WINDOWS BINARY):

LInux or some windows configs:
da-avx --thresh=-3.0 <infile.wav >outfile.wav

Windows confs that the above doesnt work:
da-avx --thresh=-3.0 --inf=infile.wav --outf=outfile.wav

The infile.wav would be the ugly inputfile.   The outfile.wav would be the new output file.

The output file will have a little less excess ambiance and often have significantly less tape/room noise (on older stuff.)

Notes about the da-*** programs:
* The expander is very gentle -- not much information is really lost unless it is truncated away by using 16bit I/O and ineffective dither.
* Theoretically, the 4 band expander can be undone, but I haven't had to do that, so I didn't bother writing an undo program -- more useful things to do.
* Any of my external use of mp3 is because of limited external storage.   I have almost unlimited storage locally, so mostly use 24bit flac at 96k.

MP3s at 48k produced from pristine sources are good enough to see the improvement based upon the recovery/correction software or the simple 4band expander.   I understand that mp3 isn't perfect -- I only use it for external compatibility purposes, and normally use the 'extreme' mode preset of lame when producing mp3s.   Otherwise, for internal archives where I don't want to use flac, I use opus at the highest reasonable rate.   I don't use .wav for archive because it essentially wastes O(half) of the space.

On the archive site I have only provided several unchanged source files -- there are also 'resurrected' versions of those files to be able to hear the difference.   If you listen to those without a comparison -- you'll be able to tell that something VERY GOOD is happening in the processing.  Almost all decisions are based upon space limitiations.   If you want 'my best' copies of something that you are interested in so as to 'better' evaluate -- just let me know
I have some copies that are so good -- they are dolbyA master tape copies.  I will not/should not release because I happened into them many years ago when I was working on another thing all together and forget the history.

So -- let me know what will help you, and I'll try to do so.

John Dyson
  • Last Edit: 17 November, 2017, 08:07:21 AM by jsdyson

  • jsdyson
  • [*][*]
Promised updated expander binary for win64.
Reply #35
Attached is the corrected binary version of the 'mild 4 band expander' project (esp good at undoing mild multi-band side-chain compressors. (as I had promised).   I have talked about the sources elsewhere (forgot what I have said here -- been chatting at a lot of places.)   One place for the source code is da.zip on: https://spaces.hightail.com/space/bOPBXTkeeT

Please contact me here (or email -- if preferred):  johndyson10@yahoo.com

Be good, safe, and enjoy.

John

  • jsdyson
  • [*][*]
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #36
I have posted some tangible results plus some hints about listening for improvements when using the da expander.  (It is NOT dolbyA, but it is da for its name!!!)   Remember:  the expander is fairly gentle, and when it is helpful -- it helps.   If it isn't applicable, it tends to be very obvious -- so it does undo moderate amounts of unneeded compression -- but it DOES NOT "REMIX" the sound!!!!  If this program was generally destructive, I would NOT be making it available.  Also, the same location below that has the two .flac examples, it also has the source code, Windows64 binary and Linux binary.   The processor is likely more reliable at 96k instead of 44.1k, but I did do the run at 44.1k -- most testing has been done at 96k and 48k, however.

I have posted the music output of two runs on the following site: https://spaces.hightail.com/space/bOPBXTkeeT
The file names are:  03.with-orig.flac  (the original version), and 03.with-da.flac (the processed version).

If you listen very carefully to the two even though the levels SEEM to be similar, the beginning noise is significantly less on the processed 'da' version.  Have to listen quickly for the noise.  Also, the voices peak significantly higher on the processed 'da' version.  There is less of the 'tight' compressed sound throughout the example also.

For tangible results below -- you'll notice that on the 'da' version the peak/RMS is significantly higher than for the 'orig' version.
Also, the crest factor is higher on the 'da' version.


jdyson@JDworkstation s0]$ play 03.with-da.flac stats

03.with-da.flac:

 File Size: 3.00M     Bit Rate: 800k
  Encoding: FLAC         
  Channels: 2 @ 16-bit  
Samplerate: 44100Hz     
Replaygain: off        
  Duration: 00:00:30.00 

In:100%  00:00:30.00 [00:00:00.00] Out:1.32M [      |      ] Hd:1.3 Clip:0   
             Overall     Left      Right
DC offset  -0.000711 -0.000711 -0.000104
Min level  -0.944031 -0.944031 -0.726776
Max level   0.885742  0.885742  0.852661
Pk lev dB      -0.50     -0.50     -1.38
RMS lev dB    -20.54    -20.54    -20.54
RMS Pk dB      -9.65    -14.02     -9.65
RMS Tr dB     -95.38    -95.38    -94.94
Crest factor       -     10.05      9.08
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      16/16     16/16     16/16
Num samples    1.32M
Length s      30.000
Scale max   1.000000
Window s       0.050
Done.
[jdyson@JDworkstation s0]$ play 03.with-orig.flac stats

03.with-orig.flac:

 File Size: 3.11M     Bit Rate: 828k
  Encoding: FLAC         
  Channels: 2 @ 16-bit   Track: 3
Samplerate: 44100Hz      Album: The Very Best
Replaygain: off         Artist: Sérgio Mendes & Brasil '66
  Duration: 00:00:30.00  Title: With A Little Help From My Friends

In:100%  00:00:30.00 [00:00:00.00] Out:1.32M [      |      ] Hd:4.4 Clip:0   
             Overall     Left      Right
DC offset  -0.000790 -0.000790 -0.000140
Min level  -0.794220 -0.794220 -0.499115
Max level   0.732880  0.732880  0.599609
Pk lev dB      -2.00     -2.00     -4.44
RMS lev dB    -22.02    -21.39    -22.76
RMS Pk dB     -12.90    -15.25    -12.90
RMS Tr dB     -94.08    -94.03    -94.08
Crest factor       -      9.32      8.24
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      16/16     16/16     15/16
Num samples    1.32M
Length s      30.000
Scale max   1.000000
Window s       0.050
Done.


  • jsdyson
  • [*][*]
DolbyA concept put aside -- better design in the works and is functioning.
Reply #37
I have put the 4 band aside for now -- pretty much latest sources reside on the repository.   I found out (by some experiments) that the 'large scale' processor can usually do 90% of what the 4 band can do -- but also MUCH MUCH MORE.   Frankly, in some of my experiments, the noise reduction capability (due to some uncommon algorithms) SEEMS TO GO far beyond a normal expander.  In the same way the 'large scale' APPEARS to do a pretty good (if not amazing) job of dynamic range expansion for transients.   I did learn some limitations of the transient expansion mechanism -- should have realized from the math -- so I had to only allow enable/disable rather than tuning it.  (The transient expansion tends to fit hand-in-glove with the original compression that damaged the music and the normal expansion mechanisms in the expander section of the processor.)   I still plan to give out binaries and a little later source code -- but there is A LOT more work than the simple 4 band.   Because this new processor is so very aggressive -- it even has the ability to recompress and relimit if too much expansion is requested.   The tunables are too numerous right now, and I need to do some work on that.   Here are some short examples (before/after.)  The original version is converted directly from the source copy used to produce the 'after' version.  There was no special tuning in this case, but the expander was set to a more aggressive mode.   The most recent work on the expander has been to further refine the NR capabilites, which I estimate at approx 10-12dB (with an 18dB maximum NR.)  This is NOT your average audio processor!!!   Attached are the shortened examples...

  • jsdyson
  • [*][*]
Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Reply #38
One more example -- slightly less processing.   There might be too much processing for some ears on the original 'restored' version, so I backed off about 30% of the processing in this example.   Note that I still don't have 100% of the surging fixed again -- I have retuned for NR a bit better, and so getting the higher level dynamics tuned to the 'ear' is important.  Need to decide if I need to make the depth of fast (and linear) expansion faster, make the slow (dB/dB) expansion slower or faster or what other combination.  It sounds to me like the hold time is a little too long -- but the tradeoff is LF distortion.  There are so darned many tradeoffs that it can be very UNFUN to work on this stuff!!!  So, below is a version of the demo with less expansion.  (I dropped the expansion from 1:1.60 to 1:1.40 to give an idea as to the difference.)   I didn't change any timeconstants (which are pretty much fixed anyway -- but appear to need to be retuned because of the vast amount of NR enabled.)

  • jsdyson
  • [*][*]
One more 'full function' processor example -- NR and dynamics improvement
Reply #39
Got another example -- I have partially retuned the restoration processor -- most got rid of the surging.   it was mostly due to the depth of linear expansion being allowed to be too deep in the time/dB axis.   I made the dB expander sections more solid (longer averaging times).   The dB expander sections use an atypical attack/release structure -- because it isn't one.  It basically doesn't understand attack/release but rather chases the signal around for reasonable dB bounds.   If I had used a traditional attack/release structure, the surging and expansion artifacts would be MUCH worse.  Unfortunately, there was a small loss in noise reduction -- because the gain control for low levels is less quick.   This whole thing is a tradeoff.  Even though it is not scientific, I have a sample of another song off the same album.  The same song gets boring over and over again -- also the original example '14.pais*' wasn't really the 'best' quality recording, but REALLY SHOWED the general capability of the restoration processor.  In that previous example, there was even some pre-limiting and limiting and super-fast dynamic limiting in the result -- but I'll bet that you cannot tell when it was happening!!! (very good limiter design -- doesn't distort or intermod unless desperate.)   So, attached is the example with the retuned result.  (One reason why I sometimes have troubles with the examples is that my ears are so darned variable -- so at night I'll think that the sound is okay -- then in the morning, I'll hear a defect.)  The only defect that I can tell this morning in this example would normally be inaudible (might be able to see the noise floor, however with careful digital analysis.)