Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Have a working 'expander' based on DolbyA (not same design) -- works well. (Read 41159 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #25
As promised -- this is the source code for the 4 band expander, with built binaries for Linux.  I think that it is all there.  The only bugaboo might be that I used zip to put the files together instead of my normal tar Jcvf...   I don't normally use zip...   I do get real noise reduction, and if you use this -- the --info switch can really help understand what is going on (and you can see the gains move all over the place.)  Yea, the display is slow, but good enough for me to be able to debug the code.

At least, I try to follow through with my promises :-).

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #26
Many thanks for your persistence and intellectual curiosity, John! While I do not have a Linux machine at the moment, your efforts are no less appreciated.

I take it that with level matching, this can be used to properly decode raw Dolby A audio -- useful for anyone transferring such material to the digital domain without a hardware decoder?

 

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #27
Regarding not having a Linux machine....   I think that I can build a Windows64 binary pretty quickly.  Giveme a day or so, but maybe as quick as an hour or so -- I can PROBABLY produce a Windows64 binary -- I built a Windows64 version of this program a few weeks ago -- so it should work.   So, if I don't have it done in a day, there was a problem.   Most likely, I'll be able to post a Windows64 command line binary (along with some needed .dlls) -- probably in a .zip file in a few hours.

Regarding being able to decode DolbyA -- I cannot make a promise, but all of the parameters match DolbyA fairly exactly -- including careful interpretation of whatever hard copy materials (e.g. maybe even the schematic) so as to emulate the smallest level of transient behavior.   The actual gain curve might be off a little -- again, I am an EE and can interpret very subtle discrete designs to a pretty good precision -- but there are so darned many variables -- including the selected components.   One thing that I can promise -- the 'decoder' does do a very good, gentle 'uncompression' of a lot of available material.

Again -- this decoder is VERY GENTLE -- and in some cases might disappoint because it is so very gentle (10dB gain change over a 40dB gain range on input) -- but DolbyA is like that.   Also, the decoder does have a slight possibility of increasing low frequency distortion -- the detector is a full wave linear (not squared) detector, has a fairly quick attack time and a release time that just barely handles low audio frequencies.  So, if the signal had already been distorted by a fast compressor, then this decoder can benefit the signal by undoing some of the distortion.   If the signal has not already been compressed, it can (and sometimes does) sound like hell -- as expected.

I have a few bugs to deal with right now -- my demo scripts had an error, where the temporary files were prematurely converted to 44.1k, so wasn't getting the most benefit from the quality of the various components.   I am correcting the scripts so that everything stays at 96k (or whatever frequency is selected) until the final, mp3 conversion stage.

After this bug fix above, then I'll see about producing the Win64 binary.

John dyson

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #28
Good news (I hope).  I think that I now have a working Windows64 binary -- command line only.  I don't do GUI code, so if someone wants to do a simple GUI and repackage for sale -- just give me credit and be nice (of course, if someone distributes a commecial copy, they should probably compile it on their own so that they can maintain the codebase.)

I tried to leave a simple README.txt to help someone to use the program.  Included in the zip are some cygwin dlls necessary to help the program to work.  I built all of the fancy support stuff into the binary -- statically linked it in from the cygwin distribution so that you don't have to deal with a myriad of shared libs (actually, not that many -- perhaps only a few others like the standard library stuff.)

Good luck...  I do promise that I didn't place anything evil into the .zip, and I created the .zip under Linux, so it is probably clean...

John Dyson

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #29
For anyone doing listening tests -- I MADE A MISTAKE IN THE pseudo-DolbyA decoder...  If you look at the source, you'll notice two filters that are commented out in the main processing/filtering class.   Those filters should NOT have been commented out, therefor there is a bit of edginess that is fixed by reinstalling the filters.   The demos will be fixed by EOD today (16nov2017.)  I'll also post a description of the fix so that people with source code cfan fix the problem.  (IT IS EASY TO FIX, JUST remove the // comment out, and recompile.)   I'll recompile for the Windows people also.

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #30
Thanks, John, looking forward to having a bit of a play with this. ;)

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #31
I have not tried it myself but I just want to add that this is really interesting and also a very big thank you for right away supporting Linux.

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #32
Good news -- I have the corrected source ready -- and the binaries for Linux reside in the zip file -- they are the latest.  The actual patch is this:

In the file 'audioproc.cpp', at approx line 1410 (plus or minus) , there should be a code sequence like this:
        //
        // Slow down the gain changes a little (help to avoid aliasing)
        //
        pgainl = dfiltlA(pgainl);
        pgainr = dfiltrA(pgainr);

Unfortunately, during debugging/optimizing I had commented out those two lines.  It was wrong to have commented out those lines, additionally I just made a change to the filter parameters to remove just a bit more of the aliasing.  These filters are not critical for normal situations, but material with lots of MF/HF content (like ABBA) will cause the expander to produce a harsh edge on the sound.  It is exactly this kind of thing (not filtering out potential aliasing) that can make computer processing using conventional compressor/limiter design sound harsh.  (I finally noticed this harshness when running the music through my multiband finalizer -- and it is VERY VERY GOOD at bringing out faults in digital material.)  When I started chasing down the reason for the terrible sound (coming out of my finalizer -- and had realized that the signal was being disrupted in the simple 4 band expander.)  The more complex 'recovery' software doesn't have this problem - it has a much stronger solution to the alaising problem, along with this simple one.

So, throw away the old attachment, and only use this one.  The abba files on the demo repository have been corrected so far up to abba-119-waterloo.  I am going to put the rest of the abba demos aside, and work on the Sergio Mendez examples next (they are almost a religious experience in sound!!!)

I have attached the corrected code, and the zip file contains the built versions of the programs for Linux.  The windows64 version will come out soon.  The previous windows64 version DOES WORK, but it also has that harshness that I'll be fixing in a few hours.

SORRY -- REALLY, REALLY for the mistake.  I am mistakenly not using my professional discipline when working on this stuff. but I hesitate to make to much of a bureaucracy -- maybe it is time to do so.

John Dyson

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #33
I do not get it.

Why are you comparing mp3 with mp3?

In source code I see script makemp3 which converts wav to mp3.

Comparing lossy transcoding with another lossy transcoding is futile.

This software could be more scientific if you actually compare lossless with lossless.

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #34
The processor code (the da-avx or da-*** program)  (the 4 band one) basically (in the actual program) does 32bit floating or 16bit signed integer wav file I/O -- 44.1k, 48k, 96k or 192k.   The only reason for the flac/mp3 file conversions is for convienience -- there is also a script that allows playing the music directly in realtime.  MY USE OF MP3 ON SIMPLE DEMOS IS DUE TO LIMITED SPACE.   I do use the best reasonable conversion params for the MP3.   Flac 96k/24bit available by arrangements if you really need.  24bit/96k or floating-point/96k are my native formats for doing work.

So, nothing severely distorting is going on.  MP3 on external demo stuff is out of necessity  -- but the conversions are usually well above 200k.  For convenience, I supplied flac file and mp3 file conversion scripts.  If the makemp3 script is missing, then that is my fault -- I have a whole series of scripts that do conversion IN/OUT of the pseudo DolbyA (4 band) processor.   I'll check to make sure that the makemp3 script exists -- it is up to you to make sure that you have sox (a very common audio tool) and lame (one of the better mp3 converters.)  THE ONLY REASON FOR MP3 over OTHER LOSSY SCHEMES IS IT BEING COMMON.   I prefer opus, and usually use it at 258k or 384k.  Flac is essentially perfect, and is what I use for high quality archives (usually 96k/24bit.)

Using the program in its most basic mode -- think of this:
1)  You have a piece of over-compressed music -- youtube stuff is often a very good example.  However, a lot of commercial stuff is also over compressed (refer to the originals on my repository -- directly from the source that I got them modulo a moderate/high bitrate mp3 conversion.   Only reason for using mp3 on demos is a lack of space.   I can provide people .flac 24bit 96k to anyone who asks.
2) You want something a little less 'obnoxious' sounding, but mostly just want to simple tool without lots of tweaks (the tool has one tweak that isn't terribly sensitive.

HERE IS WHAT YOU CAN DO on LINUX (or if you have a good WINDOWS BINARY):

LInux or some windows configs:
da-avx --thresh=-3.0 <infile.wav >outfile.wav

Windows confs that the above doesnt work:
da-avx --thresh=-3.0 --inf=infile.wav --outf=outfile.wav

The infile.wav would be the ugly inputfile.   The outfile.wav would be the new output file.

The output file will have a little less excess ambiance and often have significantly less tape/room noise (on older stuff.)

Notes about the da-*** programs:
* The expander is very gentle -- not much information is really lost unless it is truncated away by using 16bit I/O and ineffective dither.
* Theoretically, the 4 band expander can be undone, but I haven't had to do that, so I didn't bother writing an undo program -- more useful things to do.
* Any of my external use of mp3 is because of limited external storage.   I have almost unlimited storage locally, so mostly use 24bit flac at 96k.

MP3s at 48k produced from pristine sources are good enough to see the improvement based upon the recovery/correction software or the simple 4band expander.   I understand that mp3 isn't perfect -- I only use it for external compatibility purposes, and normally use the 'extreme' mode preset of lame when producing mp3s.   Otherwise, for internal archives where I don't want to use flac, I use opus at the highest reasonable rate.   I don't use .wav for archive because it essentially wastes O(half) of the space.

On the archive site I have only provided several unchanged source files -- there are also 'resurrected' versions of those files to be able to hear the difference.   If you listen to those without a comparison -- you'll be able to tell that something VERY GOOD is happening in the processing.  Almost all decisions are based upon space limitiations.   If you want 'my best' copies of something that you are interested in so as to 'better' evaluate -- just let me know
I have some copies that are so good -- they are dolbyA master tape copies.  I will not/should not release because I happened into them many years ago when I was working on another thing all together and forget the history.

So -- let me know what will help you, and I'll try to do so.

John Dyson

Promised updated expander binary for win64.

Reply #35
Attached is the corrected binary version of the 'mild 4 band expander' project (esp good at undoing mild multi-band side-chain compressors. (as I had promised).   I have talked about the sources elsewhere (forgot what I have said here -- been chatting at a lot of places.)   One place for the source code is da.zip on: https://spaces.hightail.com/space/bOPBXTkeeT

Please contact me here (or email -- if preferred):  johndyson10@yahoo.com

Be good, safe, and enjoy.

John

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #36
I have posted some tangible results plus some hints about listening for improvements when using the da expander.  (It is NOT dolbyA, but it is da for its name!!!)   Remember:  the expander is fairly gentle, and when it is helpful -- it helps.   If it isn't applicable, it tends to be very obvious -- so it does undo moderate amounts of unneeded compression -- but it DOES NOT "REMIX" the sound!!!!  If this program was generally destructive, I would NOT be making it available.  Also, the same location below that has the two .flac examples, it also has the source code, Windows64 binary and Linux binary.   The processor is likely more reliable at 96k instead of 44.1k, but I did do the run at 44.1k -- most testing has been done at 96k and 48k, however.

I have posted the music output of two runs on the following site: https://spaces.hightail.com/space/bOPBXTkeeT
The file names are:  03.with-orig.flac  (the original version), and 03.with-da.flac (the processed version).

If you listen very carefully to the two even though the levels SEEM to be similar, the beginning noise is significantly less on the processed 'da' version.  Have to listen quickly for the noise.  Also, the voices peak significantly higher on the processed 'da' version.  There is less of the 'tight' compressed sound throughout the example also.

For tangible results below -- you'll notice that on the 'da' version the peak/RMS is significantly higher than for the 'orig' version.
Also, the crest factor is higher on the 'da' version.


jdyson@JDworkstation s0]$ play 03.with-da.flac stats

03.with-da.flac:

 File Size: 3.00M     Bit Rate: 800k
  Encoding: FLAC         
  Channels: 2 @ 16-bit  
Samplerate: 44100Hz     
Replaygain: off        
  Duration: 00:00:30.00 

In:100%  00:00:30.00 [00:00:00.00] Out:1.32M [      |      ] Hd:1.3 Clip:0   
             Overall     Left      Right
DC offset  -0.000711 -0.000711 -0.000104
Min level  -0.944031 -0.944031 -0.726776
Max level   0.885742  0.885742  0.852661
Pk lev dB      -0.50     -0.50     -1.38
RMS lev dB    -20.54    -20.54    -20.54
RMS Pk dB      -9.65    -14.02     -9.65
RMS Tr dB     -95.38    -95.38    -94.94
Crest factor       -     10.05      9.08
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      16/16     16/16     16/16
Num samples    1.32M
Length s      30.000
Scale max   1.000000
Window s       0.050
Done.
[jdyson@JDworkstation s0]$ play 03.with-orig.flac stats

03.with-orig.flac:

 File Size: 3.11M     Bit Rate: 828k
  Encoding: FLAC         
  Channels: 2 @ 16-bit   Track: 3
Samplerate: 44100Hz      Album: The Very Best
Replaygain: off         Artist: Sérgio Mendes & Brasil '66
  Duration: 00:00:30.00  Title: With A Little Help From My Friends

In:100%  00:00:30.00 [00:00:00.00] Out:1.32M [      |      ] Hd:4.4 Clip:0   
             Overall     Left      Right
DC offset  -0.000790 -0.000790 -0.000140
Min level  -0.794220 -0.794220 -0.499115
Max level   0.732880  0.732880  0.599609
Pk lev dB      -2.00     -2.00     -4.44
RMS lev dB    -22.02    -21.39    -22.76
RMS Pk dB     -12.90    -15.25    -12.90
RMS Tr dB     -94.08    -94.03    -94.08
Crest factor       -      9.32      8.24
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      16/16     16/16     15/16
Num samples    1.32M
Length s      30.000
Scale max   1.000000
Window s       0.050
Done.


DolbyA concept put aside -- better design in the works and is functioning.

Reply #37
I have put the 4 band aside for now -- pretty much latest sources reside on the repository.   I found out (by some experiments) that the 'large scale' processor can usually do 90% of what the 4 band can do -- but also MUCH MUCH MORE.   Frankly, in some of my experiments, the noise reduction capability (due to some uncommon algorithms) SEEMS TO GO far beyond a normal expander.  In the same way the 'large scale' APPEARS to do a pretty good (if not amazing) job of dynamic range expansion for transients.   I did learn some limitations of the transient expansion mechanism -- should have realized from the math -- so I had to only allow enable/disable rather than tuning it.  (The transient expansion tends to fit hand-in-glove with the original compression that damaged the music and the normal expansion mechanisms in the expander section of the processor.)   I still plan to give out binaries and a little later source code -- but there is A LOT more work than the simple 4 band.   Because this new processor is so very aggressive -- it even has the ability to recompress and relimit if too much expansion is requested.   The tunables are too numerous right now, and I need to do some work on that.   Here are some short examples (before/after.)  The original version is converted directly from the source copy used to produce the 'after' version.  There was no special tuning in this case, but the expander was set to a more aggressive mode.   The most recent work on the expander has been to further refine the NR capabilites, which I estimate at approx 10-12dB (with an 18dB maximum NR.)  This is NOT your average audio processor!!!   Attached are the shortened examples...

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #38
One more example -- slightly less processing.   There might be too much processing for some ears on the original 'restored' version, so I backed off about 30% of the processing in this example.   Note that I still don't have 100% of the surging fixed again -- I have retuned for NR a bit better, and so getting the higher level dynamics tuned to the 'ear' is important.  Need to decide if I need to make the depth of fast (and linear) expansion faster, make the slow (dB/dB) expansion slower or faster or what other combination.  It sounds to me like the hold time is a little too long -- but the tradeoff is LF distortion.  There are so darned many tradeoffs that it can be very UNFUN to work on this stuff!!!  So, below is a version of the demo with less expansion.  (I dropped the expansion from 1:1.60 to 1:1.40 to give an idea as to the difference.)   I didn't change any timeconstants (which are pretty much fixed anyway -- but appear to need to be retuned because of the vast amount of NR enabled.)

One more 'full function' processor example -- NR and dynamics improvement

Reply #39
Got another example -- I have partially retuned the restoration processor -- most got rid of the surging.   it was mostly due to the depth of linear expansion being allowed to be too deep in the time/dB axis.   I made the dB expander sections more solid (longer averaging times).   The dB expander sections use an atypical attack/release structure -- because it isn't one.  It basically doesn't understand attack/release but rather chases the signal around for reasonable dB bounds.   If I had used a traditional attack/release structure, the surging and expansion artifacts would be MUCH worse.  Unfortunately, there was a small loss in noise reduction -- because the gain control for low levels is less quick.   This whole thing is a tradeoff.  Even though it is not scientific, I have a sample of another song off the same album.  The same song gets boring over and over again -- also the original example '14.pais*' wasn't really the 'best' quality recording, but REALLY SHOWED the general capability of the restoration processor.  In that previous example, there was even some pre-limiting and limiting and super-fast dynamic limiting in the result -- but I'll bet that you cannot tell when it was happening!!! (very good limiter design -- doesn't distort or intermod unless desperate.)   So, attached is the example with the retuned result.  (One reason why I sometimes have troubles with the examples is that my ears are so darned variable -- so at night I'll think that the sound is okay -- then in the morning, I'll hear a defect.)  The only defect that I can tell this morning in this example would normally be inaudible (might be able to see the noise floor, however with careful digital analysis.)

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #40
Learned something new about the 4 band expander (pseudo-DolbyA) expander design.  I had noticed a certain harshness in the decoded output -- heard that problem before, but didn't add two and two together.   Whenever one does an expand or AGC compress, there are interesting (and often unpleasant) modulation products produced.    First note, I don't have an entire DolbyA scehmatic, and simply assumed that the outputs of each band were just summed together) which they might still be.  However, whenever doing a compression (especially something close to a 2:1 type compression or more with moderatly fast attack/release times), there are significant intermod products produced.   So, if you have two signals -- perhaps at 1000Hz and 1030Hz -- youll get 30Hz in the output.   Normally, in my compressors and expanders, I tend to remove as much of those products as possible.  I didn't do so in my 4 band expander, because I had assumed that there might be no 'high pass filter' after each band IN THE ENCODER/COMPRESSOR to remove those products.  I suspect that there is indeed SOME high pass filtering that might help to remove some of the products.  Alas -- in my expander I had NO removal of the products except for a rather rough 12Hz type HPF (to basically remove most stuff below 20Hz.)  I was using only a singular HPF in the previous version of the code for the total/full/wideband output.

So, I kept on hearing this 'rough' sound (esp in groups like ABBA -- where the voices do tend to intermodulate) and using that group as source material has been one of my personal audio tests.  This 'rough' sound has been worse than what I desire -- then finally eureka -- I suspect that it is possible that DolbyA (or other compressors used in the source material) is definitely doing a least a minor amount of HPF on each channel.

So, as a gross experiment, I added a 2 stage each of single pole HPF of approx 12Hz, 60Hz, 1kHz, 3kHz for each of the 0-80Hz, 80-3k, 3k-20k (or 3k-9k -- depending on where it is in the circuit) and 9k-20k directly on the outputs of each of the bands individually.  This will help to keep most of the intermod that is created below each of the above bands attenuated to some extent.  The worst intermod appears in the 0-100Hz range, and that is obliterated in the two higher bands, and MUCH of the intermod produced by expanding the 80-3k range is also removed.   I am doing nothing about the intra band intermod for this case (that is a more difficult problem, and less of an audible issue, but still can be important), but that intermod would definitely exist also in the compressed (encoded) version and some amount of cancellation can be beneficial (alas -- tape might have too many phase prolbems for the cancellation to be complete.)
Anyway -- some of the additional intermodulation artifacts have indeed been removed, and there seems to be a noticeable improvement.  In the past, I have definitely measured the artifacts, but would be of minimal benefit in this case since I don't have a DolbyA to experiment with -- but that is not important right now since I am not selling anything.   All I can definitely say is that there is now an improvement, and the 4 band expander DOES improve some (most) commercial material that I have processed.  (I do NOT attribute that to DolbyA encoding, but rather a lot of compression being left in commercial music.)  I have a much more flexible expander that shows the DolbyA parameters for expansion to be helpful, but do NOT do all possible to improve most commercial materials.
So -- the good & useful news -- especially for those in the future who are developing (esp mutliband) compressors/expanders -- make sure that you have mitigated the LF side of the intermod produced by any gain control mixing with the audio signal.  Even I have just got caught by that issue -- and I have been aware of that problem for years!!!

John Dyson

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #41
I just learned something new -- this is about the 'full scale' restoration processor -- not just the 4 band expander.   You might remember after reading previous posts that the full scale processor has some transient restoration math in it -- it actually does a good job of modeling the transient behavior of the music, and help to re-establish the intensity of transients that were previously blunted off.
When implementing the transient processing code, I added 'diapers' around the code to keep it from doing too much.  The theory was that I was worried about adding distortion to portions of the waveforms that werent transients.  So, I slowed down the transient handling to about 250usec (about 4kHz) at the highest frequencies, and equally relatively lower on lower bands (approx 2-4X lower than the lowest frequency in the band.)  In order to mitigate the ripple, I did a peak filter a couple of times to make sure that intermod would be minimized.
Well -- I might have been wrong.  I removed many of the 'diapers' (and added carefully crafted replacements that do things differently) in the code while playing back a really good original of Brasil'66...  You know -- the male vocals having a 'clipped' or 'heavily limited' sound to them...  Instead of getting 'distortion', I ended up with better reproduction of the transient portions of the voice.  It isn't perfect (yet), but much more perfect than the original and much better sounding so far.   Removing the 'diapers' required changing some of the math so that the levels were more carefully measured and matched to the gain control algorithms (akin to using RMS or peak limiting -- that kind of modification, but not the same.)  So, I am continuing the experiments with even better reproduction of some old stuff.  The results of this experiment help to demonstrate the effectiveness of my transient recovery algorithms -- however slow they are.

Another goal -- not yet 100% achieved -- is to help to restore the sound of the lower order of the guitar in some Simon and Garfunkel songs (in the older recordings.)   There is some definite over-compression/intermod/room effects that could benefit with just a little less of those distortions.   The current version of code (before this last change -- no examples released yet of either) does help with some of that old-time '60s intermodulated sound, but not quite 100% (or -- maybe 50-75% qualitatively is my goal.)

John Dyson  

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #42
 John,

When you started this whole thing you mentioned the "original sound" of The Carpenters, so of course I thought you were talking about the original vinyl.     And as you may have heard, they've just re-released a re-mastered vinyl box set and .  I just thought that was an amazing coincidence!  

I have no idea if you'd like the remastering or if you have any interest in vinyl but  I thought about your post when I heard about the re-release.  As of now, it looks like the re-mastered version is only on vinyl but if it sells I assume they'll release it digitally.

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #43
Yea -- when I speak of the 'original sound' -- I mean the original sound that isn't all scruched up with compression.   Since I have been working on this project,  I have begun to notice 'compression everywhere', and 'too much compression' too often.   When listening to every normal source that I have of the Carpenters --including a friend supplied quad tape -- I notice FAR FAR too much compression for what is necessary.  They did it somewhat artfully, but totally unnecessary.   In the quad tape, I notice less false sibilance (sp)  in Karen's voice.  EVERY other piece of the the specific songs that I have heard, they put an edge on her voice (almost a raspy sound) that is totally unnecessary.  My processor does seem to help that a little, but I haven't done tests on the Carpenters music recently, because I am working aggressively on percussion type sounds.   I'll probably rotate back through and look at the general smooth compression like the Carpenters and the aggressive matrixed compression like ABBA again soon.   Also, I have made inroads into the old-style effects like the intermod/distortion in the original Simon&Garfunkel guitar stuff, and also have made pretty good improvement in the clipped/over-compressed male voices in Brasil'66.   This has been an iterative thing -- optimizing the structure of the gain control filters (the kind of linearity issues, transient recovery, etc.)  Amazingly, my latest breakthrough has been in transient recovery, where have made significant (but not major) improvement in removing some of the over-compressed/almost clipped sounds in the voices on Brasil'66 & Sergio Medez stuff.  The transient recovery algorithms are KEY to the general dynamic range correction capability.  I really hope to have a USABLE version of the processor ready soon -- I just added the simpified command line structure for the processor - so that one doesn't have to jigger around with 20-30 parameters -- even though most of those parameters don't need to change.  I have it down to about 5-6, which is reasonable to deal with.
I am currently not demoing the results much (outside of small group of friends) right now, due to a lack of feedback -- which I really need to get this thing working without depending upon the eccentricities of my hearing and my limited abilitity to measure some of the behaviors.   Actually, the processor is being slightly simplified, but the algorithms are VERY SIGNIFICANTLY refined even relative to 1wk ago.   The reconstruction of previously lost transient and peaks in voices and from instruments appears pretty natural sounding, even though if you listen VERY CAREFULLY, you can hear that it is not quite perfect.  Oh well -- I am making progress, and actually demoing the results using the Carpenters music is one of the first times that I got a WOW.
John

A WORST CASE example of the full-out restoration processor

Reply #44
I kind of feel bad about not putting out any demos -- so here is one.  This is a segment of 'Pais Tropical' from Brasil'66.  That song is recorded in a very 1960s' style, and has sound quality that we'd probably call today -- distorted.   This is a rather extreme case, and the processor can actually sound MUCH better on sources that are somewhere between pristine and this...  On pristine sources, you don't want to use the processor.  However, on sources that are just over compressed or have had too much peak limiting, or needs significant noise reduction, then the processor is 'your friend.'  Again, this is an extreme case, and IF YOU CAREFULLY COMPARE, you'll find that on the processed version, the male voices come out slightly clearer and you can actually hear the voices rather than just being a distorted, over compressed mess.   THIS RESULT IS NOT PERFECT, but does show some benefit even on music with a rather intense 'personality.'  To give you an idea of the kind of processing -- it has a total of approx  1:1.33 4band expansion, 1:1.86 8 band expansion, with further transient expansion.  It doesn't always do the full expansion ratio, but also doesn't always add 6dB of transient peaks either (that wouldn't be right.)  However, the processor is practically 'turned up all of the way.'   The ONLY reason why the processor can do so much dynamics processing is that it is being done carefully and NOT with a simple brute force expander (NOTHING like a traditional RMS/VCA HW expander algorithm.)   I don't claim perfection on this (on some other stuff, it can knock your socks off, but this is intellectually interesting to me.)

A typical case result when using the restoration processor

Reply #45
Here is one more -- probably more typical song -- Mais que nada?   You can hear the NR there, but of course the transient processing is pretty good also.

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #46
Update on pseudo-DolbyA...  Because of the difference in dynamics between feedback/feedforward, there has been an error (I believe) in the gain tracking of my decoder.   After some minor mods, been running some tests, and the results appear to be improved when running tests on some very-likely DolbyA material (2tk master from popular group.)  Sorry that I don't have stronger objective results yet, but even at this stage the investment would be significant.   With the mods, the gain control appears to track more deeply into the signal -- which means that it is now significantly faster without forcing a change in timeconstant/etc.  (Some mathematical changes were made to emulate the difference between feedback/feedforward without changing the basic timeconstants as painstakingly gathered from schematics and paper research.)   Of course, the TCs from the schematics are feedback timeconstants and some kind of correction is needed -- but simple change to the timeconstant doesn't seem to be the best choice.   The goal is to maintain some traceability without 'stealing' another design.

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #47
Finally got the newest version ready to try. It is better than any version before, and I have added some of the secret sauce that I had previously used on other projects. It just works better with the new code. There are three files:

1) Windows binaries "da-win-30dec2017.zip"
2) Windows sources and binaries "da-30dec2017src.zip"
3) Linux sources and binaries. "da-Linux-30dec2017.txz"

The only difference between the Windows builds and the Linux builds is that the Windows builds tunables are decreased a little so as to run more quickly on slow machines. The actual quality difference is probably nil.   After some more tests by me and any feedback that I get -- I plan to do one more release within a week with all bugfixes that we might find.   The code is essentially complete, but some tunables might be optimized.   After this iteration, the larger scale 'cleanup' or 'restoration' processor will be my only main project.

Make sure that the windows dlls (as in the Windows binary packages) reside in the same place as the da-win.exe program.

To decode a file, do it like this:
da-xxx --thresh=-1.50 --info=99 <infile.wav >outfile.wav

The threshold is dependent on the dolby level used on the tape. it makes an important and sometimes not-subtle difference. Values between +1.5 and -9.0 are usually best, but I have had the most luck between -0.75 and -6.0.

Program names:
da-avx -- latest Intel CPUs on 64 bit Linux
da-sse3 -- Recent ATOM CPUs on 64 bit Linux
da32-p4 -- older CPUs on 32 or 64 bit Linux
da-win.exe -- ATOM CPUs or newer on Windows64

Use the latest program version that you can -- newer CPUS witih the right programs can sometimes run realtime. The program is NOT multi-threaded, but it is beneficial to have two cores available if you are on the edge while running realtime.

I don't have time to describe all of the options -- but other postings might give some hints, and this decoder can do much more than just a straightforward pseudo-DolbyA decode.   If I get some feedback (and needs arise), I'll document some of the more esoteric features in the code.

The expansion logic is NOT DolbyA compliant -- but should be superior in some ways. The processor can sometimes run close enough to DolbyA to undo some of the intermod that naturally occurs when encoding the material in the first place.

The files reside on this repository:

https://spaces.hightail.com/space/bOPBXTkeeT



John Dyson

Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #48
Here I am posting some of the smaller processor files here.  Hopefully, someone will get some benefit.   The programs are free to use -- there is some GPL based libraries which are part of the cygwin suite and can be downloaded at their site.  The processor source is also free, but if it is used, just tell me and if it is for sale, give me a bit of credit.  There are actually some good ideas in there far beyond just the specific application...


Re: Have a working 'expander' based on DolbyA (not same design) -- works well.

Reply #49
I have something perhaps useful -- a screenshot of the spectrum of the following:
Top -- full processor only (not including the psuedo-dolbyA)
Middle -- pseudo-dolbyA
Bottom -- original.
Song:  Rainy Days & Mondays from the Carpenters.

It is possible (and desirable) to use both the pseudo-DolbyA and the 'full processor' (not yet named), and the benefit of the two is greatly increased, but the artifacts don't get much (any?) worse.  The pseudo-DolbyA being tested is only a few minor source code changes from the version that I have distributed.

How to analyze the output:  take a look at the density of the areas without significant music.  The areas where there is no material will show some 'snow' or constant low level blue coloration.  The two processors remove/decrease a significant amount of the 'blue areas'.  The effect is subtle -- so you do have to look closely to see the difference.   When running the software, the pseudo-DolbyA is swinging a range of approx 10dB gain between 9-20K, and perhaps 6dB at 6-9K.   The lower ranges show only 2-3dB gain swing except in certain cases (where sometimes the LF shows a wider 6dB change.)

For the 'full processor', it shows approx 9-16dB gain swing for the freq ranges above 5K, and perhaps 6dB from 100Hz to 5kHz.
(The gain swing measurements come directly from the software itself -- impossible to measure from the spectrograms, which are qualitative only.)

The spectrograms are attached -- where the 'top' is the full processor, 'middle' is the psuedo-dolbyA that I am distributing today, and the 'bottom' is the original.  The differences are VERY SUBTLE!!!