HydrogenAudio

Hydrogenaudio Forum => Scientific Discussion => Topic started by: jsdyson on 2017-10-18 05:49:42

Title: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-10-18 05:49:42
Here is a pointer to some examples, plus Linux X86/X86-64 executables.   Will eventually distribute source when I get a few more responses.   Just removed all vestiges of my GP finalizer/expander to avoid distributing more than just the DA-similar acting design.

You'll notice that the quality seems pretty darned good, and have gotten pretty good responses from audio 'listeners' (not just techies.)  This program is NOT meant as a general purpose expander (my other programs do that better), but this seems to do better for a certain subset of recent purchases/downloads.   (The other expander goes far beyond a normal dB/dB thingie, and this one is more of a nonlinear/very limited expansion range of approx 10dB or 15dB depending on audio band.)

https://spaces.hightail.com/space/bOPBXTkeeT

I plan to destroy the examples in a week or so, but will plan to provide Win64 (maybe Win32) command line binaries and a copy of AT LEAST the mainline expansion code.  I might provide the whole thing -- most of the unneeded stuff from the other project has now been removed from all of the codebase.

GOOD LUCK -- and even if you are not interested in my project, the music is generally very pretty (in some cases, better than I have ever heard some of the songs.)

John Dyson

Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: mycroft on 2017-11-07 11:22:29
I'm interested in this expander. But you nowhere provided linux executables.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: mmrkaic on 2017-11-07 18:04:18
I'm interested in your work. Could you please describe the algorithm? I'm sure many would like to know.

Thanks in advance.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-07 20:24:02
Sorry that I did remove the executables a few weeks ago...  I'll put them onto my demo site again within a day or so -- I had made so many updates to the various expander/compressor projects that unless I get feedback, I wasn't going to keep the binaries up to date.   I'll put the pseudo-DolbyA decoder binaries for X86-64 Linux back on my site hopefully tonight (been a little ill, so it might be delayed by a day or so.)   So -- I can immediately provide x86-64 linux binaries for processors with sse3 or newer, and x86-32 binaries for P4 or newer.   This will be for the simple 3.5band expander that happens to help alot of music recently distributed.    Been having troubles sometimes with my builds based upon GCC -- most of my code is better tuned to CLANG/LLVM -- so the 32 bit version might be slgihtly delayed -- not sure yet.   It won't include the larger expander complex that pretty much fully recovers music -- it is really a big program, lots of threads, and lots of options that I haven't really decided on the best tweaks yet.   The simple 3.5 band FIXED expander is only single threaded and can run 10X realtime on an Intel Haswell processor.   Give me until 9 Nov at the latest to redeposit the files, and they'll reside on:

https://spaces.hightail.com/space/bOPBXTkeeT

*NOTE THAT THE SITE ABOVE ALREADY CONTAINS SOME RESULTS OF PROCESSING AND NOISE REDUCTION EXAMPLES.   The README is out of date, but the comments on noise reduction are still accurate.

John Dyson
Title: Re: Have a working 'expander' based on DbyA (binary available again)
Post by: jsdyson on 2017-11-07 21:04:13
Okay -- lucky day!!!   I just checked, and the status for the simple 'almost-DolbyA-like' decoder is good.   So, double checked the binaries and they appear to work fine and they are stable from the standpoint of no recent changes.   I uploaded my most recent binaries, and they do work on Fedora Linux.  A windows64 version can probably be cobbled together in a few hours if desired at all.   Please don't look for amazing improvement, because the maximum noise reduction is only about 10dB (a little more at 9K+), and the real noise reduction per my measurements is only about 5-7dB.   This expander improves the dynamics a little also -- removes a good part of that nasty compressed sound that is all too common nowadays...   This is NOT the full scale expander, but will improve A LOT of recently released music.   Invoke the expander on the comand line like:

da-avx --threshold=-3.0 <infile.wav >outfile.wav

Read the file damanual.txt on the repository site below for more info:

https://spaces.hightail.com/space/bOPBXTkeeT

Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-08 05:24:23
Here is a quick description (sorry if it is difficult to read), about how the expander/recovery code works.


Expander algorithm (rough):


Firstly -- there is a psuedo-DolbyA phase.   It is a 4 band expander with a well specified expansion curve as below
(the input/output are in dB):

  //
  // THIS IS THE BASIC GAIN CURVE IN dB/dB:
  //
  template <typename DT>
   DT fullgf(DT lvl)
   {
      return (DT(-0.0) - (DT(0.046986) * lvl)) - (DT(0.006598) * lvl * lvl);
   }

   //
   // the code below applies limits to fullgf, and expands the nonlinear function beyond its numerical range
   //
   template <typename DT>
   DT fgfe(DT llvl)
   {
      DT gfval = fullgf(llvl);
      DT extgf = Vselect(Vgt(llvl, DT(-7.0)), DT(-0.50000) + (llvl - DT(-7.0)) * DT(0.100), gfval);
      extgf = Vselect(Vlt(llvl, DT(-40.0)), (DT(-9.17736) + (llvl + DT(40.0))* DT(1.0)), extgf);
      extgf = Vmin(extgf, DT(0.0));
      return extgf;
   }

Each channel has this exact same curve -- and is expanded in four bands:  0-80,80-3k,3k-9k,9k-20k, while the actual gains
are calculated from the bands 0-80,80-3k,3k-20k and 9k-20k.   The gain control for 9k-20k is a combination of the measurement
of 3k-20k and 9k-20k measurement ranges.  The resulting gain control is 10dB each for 0-80, 80-3k and 3k-9k, while there is
15dB of gain control available for 9k-20k.

-------------------------------------------------------------------------

Next -- more complicated -- is an 8 band expander with the following ranges:
0-100,100-240,240-600,600-1.2k,1.2k-2.4k,2.4k-5k,5k-10k and 10k-20k.

The expander has a dB linear expansion component that is tunable from 1:1 all the way beyond 1:1.20 or beyond.   The
dB linear attack/decay is fairly slow -- in the range of 40dB/second attack and <1dB/second decay -- but is fully dynamic,
so can be faster or slower.  The decay time is based upon the signal dwell time, and the attack time is based upon the waveshape.

Additionally, there is a true linear expansion component that has faster attack (about 300usec - 1msec, depending on
waveshape and/or frequency band), and a faster decay than the dBlinear measurement (the decay is
between 10msec and >1second -- standard exponential gain decay).  The two linear/dBlinear signal measurement/gain
signals are put together in a 'maximum' operation, where the largest value wins.

Next -- also as calculated from a modified version of the linear level measurement, there is a 'transient' recovery
calculation.  The transient recovery is NOT just a simple high pass filter on the gain control signal, but rather is
a Volterra-like series expansion of the signal level measurement.   This additional measurement is ALSO merged with
the already merged dBlinear and true linear gain control signals. IT IS IMPORTANT TO NOTE THAT ATTEMPTING TO
USE A SIMPLE HIGH PASS FILTER ON THE GAIN MEASUREMENT TO IMPROVE TRANSIENTS
DOES NOT WORK WELL.  The series expansion is critical so that the transients are handled well.


So -- the 'big expander' unit (and three are usually used as described below) provides a fast short term response,
a good longer term response, and a quick transient response to the input (lets say, obnoxiously compressed) signal.
After the signal is modified by the calculated gain control, then 'it sounds better' :-).   Of course, I am careful to anti-alias
and make sure that transitions seldom extend outside of the audio band, and there is some extra filtering going on
to keep things clean.

------------------------------------------------------------------------


We have the psuedo-DolbyA-like processing, and the second -- more complex expander right?  Well, not
quite...  The expanders are put together so that the psuedo-DolbyA-simple expander is first in the chain, then there
are THREE of the complex expanders used.   The complex expanders operate on the following signals:  
LEFT+RIGHT, then BETWEEN-LEFT/RIGHT&MID/SIDE, finally MID/SIDE.  So there are three phases of the big expander used, and the
expansion ratios are set identical.  The only important thing that might be set differently between each of the three
expanders, is that any of the 8 audio bands can be excluded from the gain calculations and SOMETIMES it is beneficial to
exclude 0-100Hz and maybe 100-240Hz.  Usually, if needed, excluding 0-100 is adequate.   The expander also has some
protections in the calculations to keep the gains from being too askew -- thereby maintaining frequency balance.

-----------------------------------------------------------------------

With all of these expander phases (plus there are more math going on in the code -- just too many details right now), some music can
handle perhaps 1:1.30 (dB/dB) expansion, significantly less hiss/noise in older material,  and sound significantly better.  Of course, the dB oriented
expansion ratio doesn't really describe all of the expansion going on -- which can be effectively the dB/dB
ratio + 3-6dB more expansion for transients.   As it is designed, the expander can be incredibly fast (faster than one might expect
an expander should be without distortion) or very smooth and slow.   It follows the signal in a way so that it deals super well with
typical popular music (and seems to be okay for some compressed orchestral stuff also.)

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-08 12:29:50
Been re-reviewing some of the expander/recovery code -- and just realized that I left 'training wheels' on the transisent expander code.   I need to do a lot of testing with the improved parameters (that is, disabled training-wheels.)   There is a demo song on the repostory site:   "carptransient-improved.mp3".   It isn't major in the case of this song, but you'll hear an 'openness' that didn't previously exist.  The transient recovery code is now more assertive about gain increases/decreases on its behalf.  (Changes like this are one reason why it is difficult for me to release anything yet...  If I can find collaborators/testers, then I'll have a better chance of releasing the working binaries and probably source code sooner!!!   So -- on the same repository, there is a readme with a notice about the upcoming improvement, but also there is an .mp3 example that shows some progress.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-11 10:19:51
Regarding the full-scale expander...  I am in the midst of a significant rework to simplify the code for release.   There is A LOT of stuff in there than isn't necessary for music recovery -- e.g. finalizing for music production.  I have decided to split out all of the 'compression' code except for the final stage limiters.  (Yes -- the limiters go far beyond a simple signal limiter), the limiters are almost unnoticeable at even 6dB of limiting in many cases.   I am going to try for a minor architectural change which I believe to be slightly more valid than the current architecture, and also will be faster/more able to support the more advanced transient recovery calculations in realtime.   It will be a week or so to do all of the modifications, but if anyone is REALLY interested in trying the full-scale music recovery software program, I am willing (within a day or two from now) to do a better controlled build for Linux or Windows-64 command line -- and release an end-user viable version of the full-scale expander (the compression/finalizing code will probably be suppressed, however.)   Even the full scale expander requires very few 'tweaks' because it really does follow the music dynamics and nowadays has very little 'ad-hoc' hackery in it.   I hope to be able to at least release the more complete expander in free-binary form and possibly even in source form if I can clean it up.  (I don't want to give away all of the very advanced proprietary code that I have written, but the expander/recovery software itself uses a limited selection of it.)

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-11 14:04:46
As an experiment -- proving viability of the full expander WITH NO psuedo-DolbyA processing enabled -- I have deposited some before/after recovery examples.  Currently only one example, but plan to deposit a few more.   These demos show significant improvement (when/if needed.)  I am not claiming perfection -- I am not a mastering engineer, but this might show the usefulness.   I DO INTEND ON RELEASING SOURCE CODE TO THE RECOVERY SOFTWARE.   Just made that decision.   I hope to have a version of the full recovery software available in a week or so -- right now, it is fully functional, but there are TOO MANY OPTIONS (mostly not used), and the command line interface needs to be simplified.  As is, there is little or no need to tune or control the expander/recovery software other than expansion ratio and possibly the transient recovery amount.   This example includes significant transient recovery -- and you can probably tell that it is very functional...   The site below is very safe, and all it contains are raw text or .mp3 files (I am myself very sensitive to security also.)

https://spaces.hightail.com/space/KzA53ZKeFc

Let me know what you think -- any feedback will be used to improve.
John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-12 11:59:34
There are some more impressive examples on my repository.  Will probably be deleting this one in a few days -- but there is both a really big improvement in dynamics and some noise improvements.   The music is pretty also...  Give me feedback if you see any issues.  Thanks...
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: Compact Dick on 2017-11-12 20:32:42
Just wanted to say that I am truly fascinated and impressed by what you're doing — it deserves more attention and acclaim. May I suggest that you also share this with tapeheads.net? They comprise a bunch of dedicated tape lovers, and would be most thrilled to know that someone is finally creating a freely accessible software Dolby implementation.

Many thanks, and keep up the great work.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-12 21:32:02
Thanks for the feedback.   The only thing holding me back from distributing the errsatz-DolbyA source RIGHT NOW (and I don't claim that it is really DolbyA compliant at all -- but it seems to match the DolbyA parameters and seems to MILDLY and CORRECTLY undo a lot of various over-compressed music out there) is that I am still not sure that it is 100% final.   I don't have a github account (yet), and thinking about how to distribute the source responsibly.   I do promise that if someone asks for it, they can have a tar.gz (tar.bz or zip or whatever) image of the tree right now.   It is ugly because I haven't done a housecleaning in a few weeks but I don't want that to get in the way of anyone with immediate need.   Binaries for Linux are online right now in the "musicdata" repository below.  If asked, I can make a Windows64 command line version available in a few days (well under 1wk) -- I have had windows builds running in recent times.  Also, the repository that I have recently talked about in this thread does have a .webm animation (in the "Rawexpandexpamples" repository) of the pseudo-DolbyA-like decoder running with the gains of each of the 4 expansion bands being displayed.   Watching the .webm animation should make one feel like it is damned close if not a reasonably perfect match for decoding some DolbyA material (the gains make a lot of sense, including the high band having 15dB gain change range as would be expected.)

The full-scale recovery software is a little farther away than the errsatz-DolbyA-like decoder because I am still not happy with the CPU usage.  * the math needed to reconstruct the transients is not simple nor is it quick to do...  It is NOT just a peak enhance, but rather a transient reconstruction.   If someone has an immediate need, I can probably put together a USABLE version of the full scale recovery software, but currently will only be usable on a Linux system -- and most usable on a 4 core (or larger) Haswell (or newer) processor.   (Nothing keeps the full scale software from running on Windows -- it is just a lot of work right now to do the re-porting effort.)  It takes approx 3/4 of my 4 core CPU right now to run realtime (that is, the full recovery software.)   The simple errsatz-DolbyA-like expander runs 10X faster than realtime -- I even removed the multi-threading from it because it runs quickly enough.

https://spaces.hightail.com/space/bOPBXTkeeT

https://spaces.hightail.com/space/KzA53ZKeFc

Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: john33 on 2017-11-12 21:43:04
I am equally fascinated by what you're doing here, although I lack the technical expertise to contribute. When, and if, you reach the point where you would like binaries to be hosted for general consumption, I am happy to offer Rarewares for that purpose. :)
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-13 06:04:21
Wow -- I just made a big mistake in running of some demos...  Happens to prove that the lower speed/higher quality recovery setting is desirable.    Normally, for real-time I use a faster set of transient recovery parameters than for a runoff using the best quality that the recovery software can do.   I forgot to use the maximum quality settings in the current set of demos (on the repo-site),  I FOUND OUT about the incorrect setting BECAUSE my recent listening test was disappointing (that is, of the music that is sitting on the 'Rawexpandexpamples' repository as of approx 13nov at 00:00 USA EST.)   It isn't a disaster, because the music is still improved, but I noticed a SLIGHTLY larger amount of 'surging' than what my high quality results normally show.   So, I am rerunning the demos and will reload the repository by about 10:00AM on 13nov (USA EST.)  The good news is that I ended up (being unaware) recognizing the detectable difference between the high quality and the 'usable' quality settings for the recovery software.   The only reason for using the lower quality setting is that the best quality settings run slower than realtime on my computer.    I TRULY WISH THAT I COULD FIGURE OUT A MATH SHORTCUT SO THAT I COULD MAKE THE RECOVERY SOFTWARE RUN FASTER!!!   The code generated by the compiler is as fast as possible given the instruction set -- but I am going to research the possibility of a faster algorithm (but I doubt it exists.)
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-13 10:25:38
Okay -- i have uploaded the new versions of the files with the correct recovery parameters.   The recovery software can be more aggressive (it is somewhat tunable), but the transient recovery is carefully calculated.   I only used middle level amounts of expansion and a slight boost to the transient recovery.  The only current tunable in the transient recovery is a gain setting parameter.  Of course, the basic expander array also has an expansion ratio.   The demo expansion ratio is a total of 1:1.36 -- which is quite respectably high without seriously bad artifacting.   Also, the transient recovery adds on another 3-6dB (both up and down) of gain change.   These demos are probably a little 'dead' sounding -- normally, I runoff with a set of low pass filters with slight peaking which gives a brighter sound (without over-enhancing the high end.)   I use two sets of filters, plus a brick wall -- the first filter is 2nd order at 9kHz, with a q of 1, and the next filter is a 2nd order at 15kHz with a q of 1.4, and a brickwall at 16kHz.   The actual rolloff doesn't strongly affect the audible region, but tends to sharpen the HF transients -- THESE FILTERS ARE NOT BEING USED FOR DEMO PURPOSES.  It might be desirable for casual listening to add the filters, which will give tigher and some shine to the sound.  (If you choose to use the low pass filters, don't try to get by with a q of 0.707 or somesuch -- that will really soften the sound.)
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-13 20:45:39
Super good news -- the recovery processor and settings is doing a darned good job of beating ABBA into submission.  It still sounds like ABBA, but the severe intensity and loss of individual voices are improved a little.   Like the rest of the demos, I am only leaving them on the repo site for a few days.   Hopefully, within that time, I'll have the recovery processor available in binary form for everyone to try.   Right now, there are still a few tweaks, and literally 50+ command line options which are being removed pretty quickly.   Most of the evil ad-hoc algorithms are going away, and the more 'scientific' algorithms are proving themselves.

If I have an ongoing bugaboo left to resolve, it is the matter of maintaining frequency response balance when running at high expansion ratios.   I do have a current solution that works pretty well, but can lend itself to some mathematical instability.  The current demos are using all of the 'right' settings, and very-little ad-hoc nonsense (other than the frequency response rebalancer.) 

The current recovery processor can actually produce usable results at 1:1.6dB/dB expansion ratio with a full-out transient enhancement of 6dB!!!   Of course, most music does not benefit from so much processing.   I am finding that 1:1.3 with 3-6dB of transient enhancement (all demos are using approx that amount) works pretty well.

Just look at the same sites as before -- with 'abba' in the name, like abba-106-supertrouper.mp3.   I have been pretty generous with the bits on the MP3 compression -- so there should be no quality loss from the flac files that I actually work with.

These demos REALLY DO sound different (and I think -- really tames the  ABBA sound.)

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: krabapple on 2017-11-13 21:18:04
can you describe the characteristics of the *original* (prior to processing) ABBA track?  Is it from CD, if so ,  is it the first CD release, or a remaster, it is from an album or from a greatest hits collection, can you post some objective data about it (DR measurement [crest factor], frequency profile..?)

 
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-13 21:53:35
The originals of ABBA are the best copies that I could find & have collected over these many years.  For example, the original that I used for 'SuperTrouper' sounds a lot like the Polar release, a bit clearer but too restricted/compressed sounding.   I copied that version to digital MANY years ago, and don't even know the manufacturer anymore.   I definitely started with GOOD sounding ABBA copies, but the resulting processed copies are much better (more listenable.)  The American releases of supertrouper in particular often have HORRID intermod at the beginning -- that is what (many years ago) motivated me to find every copy of ABBA that I could find, and try to pull together a nice sounding album for my OWN listening.    Then, eventually (being an Electrical Engineer, Software Developer and DSP developer) decided to do something serious to improve the sound quality of alot of the old music available out there (Including my fairly good --but still not quite right -- ABBA stuff that I have laying around for 10-20yrs.)   I'll plan to deposit 30seconds of one of the unmodified songs corresponding to some of the demo versions to give you an idea of what I started with    Again, the ABBA copies were the VERY BEST that I could find -- and I have been keeping my little 'ABBA hoard' for many years.   Give me a few hours, and I'll deposit a little bit of the original copies that I started with.
I must confess -- the ABBA demos are not the most challenging of the recovery examples - however even though their music is truly beautiful, the recordings sometimes seem overly intense/harsh -- and that is what the recovery software has mitigated.   I have found that there is definitely a 'secret sauce' of some kind that is helping to freshen up my copies of ABBA music -- still not sure what part of the processor is doing the good.    On the other hand, some of the Sergio and/or HerbAlpert music has enough dead time that the noise reduction and other attributes of the recovery software become obvious.  ABBA music GENERALLY does benefit from the operation of the processor, however.  *When I do figure out what aspect of the processor benefits what kind of music, I'll definitely document it.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-13 22:10:22
You asked about some stats of one of the examples -- here I am providing stats from the sox program...  The first one is from the polar disk, the second one is from my old archival copy from many, many years ago (the best one that I have), and the last is the processed copy from the mp3 - the .flac master version of the mp3 isn't very different.   The stats don't really have enough resolution to describe very much.   WRT the frequency range -- I might be able to put together a spectogram of each one -- I'll see if it is very helpful.   One thing that you would notice is that the processed copy chops the audio off at 18kHz - I have a habit of doing so to avoid  the effects of a signal outside the range of normal hearing  (if I don't know where the file will be used.)

[jdyson@JDworkstation a0]$ sox /music/abbaall/polar/ABBA/1980*roup*/*Troup* -t raw /dev/null stats
             Overall     Left      Right
DC offset   0.000126  0.000126  0.000122
Min level  -0.947998 -0.802673 -0.947998
Max level   0.845001  0.845001  0.837646
Pk lev dB      -0.46     -1.46     -0.46
RMS lev dB    -20.29    -20.27    -20.30
RMS Pk dB     -13.53    -13.53    -13.54
RMS Tr dB     -75.36    -75.36    -74.35
Crest factor       -      8.72      9.81
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      16/16     16/16     16/16
Num samples    11.1M
Length s     251.907
Scale max   1.000000
Window s       0.050
[jdyson@JDworkstation a0]$ sox /music/agflac/raw/106.flac -t wav /dev/null stats
             Overall     Left      Right
DC offset  -0.001000  0.000336 -0.001000
Min level  -0.966064 -0.861938 -0.966064
Max level   0.999969  0.999969  0.999969
Pk lev dB      -0.00     -0.00     -0.00
RMS lev dB    -20.91    -21.34    -20.53
RMS Pk dB     -13.01    -13.83    -13.01
RMS Tr dB     -92.56    -92.56    -92.21
Crest factor       -     11.67     10.62
Flat factor     0.00      0.00      0.00
Pk count           3         2         4
Bit-depth      16/16     16/16     16/16
Num samples    11.2M
Length s     254.198
Scale max   1.000000
Window s       0.050
[jdyson@JDworkstation a0]$ sox abba-106-super* -t wav /dev/null stats
             Overall     Left      Right
DC offset   0.000001  0.000001 -0.000000
Min level  -0.920204 -0.920204 -0.885318
Max level   0.952650  0.952650  0.932306
Pk lev dB      -0.42     -0.42     -0.61
RMS lev dB    -20.75    -20.75    -20.75
RMS Pk dB     -11.64    -11.64    -11.81
RMS Tr dB    -127.93   -121.57   -127.93
Crest factor       -     10.39     10.17
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      29/29     29/29     29/29
Num samples    12.1M
Length s     252.960
Scale max   1.000000
Window s       0.050
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: krabapple on 2017-11-13 23:27:53
We all lose high frequency hearing as we age -- I certainly have -- but the limit commonly accepted is 20 kHz.  Are you worried about aliasing of >20kHz frequencies into that lower (18-20kHz) range?

This whole project seems...idiosyncratic? 
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-13 23:44:51
My worry about HF is mostly that I am doing nonlinear processing, and I try to minimize intermod/beats/aliasing.   It is mostly superstition that I typically chop off 18kHz -- but it is an attribute of  my scripts and NOT the programs.  It seems like there is very little energy up there.   I'd definitely hesitate chopping off at anything below that -- but truly, if you think that it is a bad idea -- I'll DEFINITELY CONSIDER REVISING THAT DECISION.   (My mind is open!!!)  At 61, I have found that my hearing is gone at about 15kHz, but I can hear just below that...  Perhaps I underestimate what the 'youth' will be missing :-).  (In the olden days, I could do 18-19kHz pretty reliably, and the old 15kHz horizontal sweep on NTSC TV sets often drove me nuts...  No trouble now for many reasons :-)).

Here is the history -- playing with gain control, and recognizing that any time that the gain changes at a rate whose spectrum approaches the audio range, then there'll definitely be some sidebands appear.   I have learned that avoiding (usually minimizing) any kind of sidebands/intermod/aliasing/etc is a very good thing.   However, the real benefit of my habit of chopping at 18kHz in my scripts has little basis in need or fixing any specific problem.   I have run test with my multiband compressor, and during my early design phases, I had recognized the 'hash' that is caused by doing quick gain control. (Some people notice it as a hardened sound.)  Now, I have a method to remove the most obnoxious sidebands, but some of those evil things still happen.  (SO THAT IS MY HISTORY/FEAR OF INTERMOD/ALIASING/BEATS/ETC)  ;)

John  
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-14 00:20:07
We all lose high frequency hearing as we age -- I certainly have -- but the limit commonly accepted is 20 kHz.  Are you worried about aliasing of >20kHz frequencies into that lower (18-20kHz) range?

This whole project seems...idiosyncratic? 
Well -- regarding the 'whole project' being idiosyncratic -- I don't think so.  A lot of people (including non-techies) have mentioned the bad sound on many older (and newer) recordings, and it appears that a lot of 'bad sound' comes from too much dynamic range restriction.  Even if the 'compression'/'limiting' is done perfectly -- and I do have software that can do it without significant distortion products, it still makes the music sound 'tight' or perhaps claustrophobic (sp?).   It has taken me a while to figure out how to undo some of the overly compressed sound (and do it with maximum quality.)   One thing I did learn - much of what makes a compressor work really well is NOT what makes an expander work well.   There are some things that do help (avoidance of intermod products/etc), but other things like tending to use a certain kind of attack/decay algorithm would be wrong for expanders vs. compressors.   I have been getting A LOT of positive feedback from non-techies (less-so from techies who might like that wonderful compressed sound.)   I do NOT claim perfection so far, and in fact just realized that my ABBA demo had too much transient restoration enabled.   I am going to back that off to the theoretically correct value (instead of a boosted value) later on tonight.   Too much compression is 'bad', even though it is very common -- geesh -- I have code that can almost undo the compression in 'Shake it off', even thought the purpose of that would be idiosyncratic :-).
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-14 02:52:54
I have a simple demonstration of the benefit of using a good piece of audio clean-up/recovery software on older (and probably some newer recordings.)   There are two examples on the repository site:  "13.pretty-raw.mp3" and "13.pretty-recoveredB.mp3".   You'll hear a significant decrease in background noise, and more natural (less compressed) dynamics in the recovered version.  Another good comparison would be "03.with-unproc.mp3" and "03.with-recoveredB.mp3".  The difference in realism is significant -- not quite the same as listening to a group in person -- but certainly an improvement!!!   The '03.with-*" version even has defects in the recording, but the improvement is still very worthwhile.   I am not arguing that the correction/improvement/expander software is perfect, but it is eyeopening to hear the comparison.   My little diversion into ABBA has been a pet project for years, and one reason why I got started on this.   I am hoping to donate this technology (not really all that much of an invention -- just a good application of technology) so that we can all benefit, and only need some help with choices on these matters.   REALLY -- do take a listen!!!  Due to security reasons, just don't trust any files any newer than 9:50EST (USA EST) time until I do some probable updates later on.

https://spaces.hightail.com/space/KzA53ZKeFc
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-14 11:41:11
Another enhancement is forthcoming.  I didn't like how the 'latin beat' was sometimes being handled.  This error had to do with the long term attack/release which gives the internal longest-term limit as to how much gain change is allowed.   The gain was allowed to hang too long, which then causes a 'jerking' or 'surging' sound.   The attack/decays are a very careful tradeoff that has to match the ear, and NOT be tuned to each song.  I decided that the conventional fast attack/slow decay in the dB domain is the wrong algorithm for the longer term time constant.   So, I resurrected a dynamic&symmetrical attack/decay which can actually provide long hold times and long attack times (or short attack/hold times) depending on the difference between the current power level and the recent average power level.  So, the filter is both more agile and also filters more (capabiility because the attack/decay is nonlinear.)   I have used that technique for a long time before and it worked well...  However a few weeks ago I removed it because my ears temporarily lost the ability to hear certain defects...  I am currently testing the new code -- but downloaded the new one for comparison:  the old one is 13-pretty-recoveredB-OLDOLD.mp3 and the new one is 13.pretty-recoveredC.mp3.   If you listen carefully, you'll notice the latin beat to have less of a 'surge'.  This result is due to my two-sided attack/decay processing as I described above.  I haven't finalized the tuning, but it is already pretty close to correct -- the way that we hear pretty much defines the parameters, and I had already produced some reasonable nonlinear attack/decay values (not in the units of dB/sec or RC timeconstant, but kind of a dB^2/sec type thing.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-14 23:28:32
I have been updating the files quite often (the names on previous posts.)  You might have to choose a later filename (e.g. instad of recoveredC for the filename end, it might be recoveredD or recoveredF.)  Always look for the latest file by looking at the end of the filename.  I am making amazing progress on the full scale recovery code -- it has been a lot of work.  I sure hope that this starts a movement where the music industry quits damaging the music with too much processing.  At least, in the meantime, this program and others can help to undo some of the damage (I hope.)

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-15 00:17:09
As promised -- this is the source code for the 4 band expander, with built binaries for Linux.  I think that it is all there.  The only bugaboo might be that I used zip to put the files together instead of my normal tar Jcvf...   I don't normally use zip...   I do get real noise reduction, and if you use this -- the --info switch can really help understand what is going on (and you can see the gains move all over the place.)  Yea, the display is slow, but good enough for me to be able to debug the code.

At least, I try to follow through with my promises :-).
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: Compact Dick on 2017-11-15 10:23:24
Many thanks for your persistence and intellectual curiosity, John! While I do not have a Linux machine at the moment, your efforts are no less appreciated.

I take it that with level matching, this can be used to properly decode raw Dolby A audio -- useful for anyone transferring such material to the digital domain without a hardware decoder?
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-15 13:09:10
Regarding not having a Linux machine....   I think that I can build a Windows64 binary pretty quickly.  Giveme a day or so, but maybe as quick as an hour or so -- I can PROBABLY produce a Windows64 binary -- I built a Windows64 version of this program a few weeks ago -- so it should work.   So, if I don't have it done in a day, there was a problem.   Most likely, I'll be able to post a Windows64 command line binary (along with some needed .dlls) -- probably in a .zip file in a few hours.

Regarding being able to decode DolbyA -- I cannot make a promise, but all of the parameters match DolbyA fairly exactly -- including careful interpretation of whatever hard copy materials (e.g. maybe even the schematic) so as to emulate the smallest level of transient behavior.   The actual gain curve might be off a little -- again, I am an EE and can interpret very subtle discrete designs to a pretty good precision -- but there are so darned many variables -- including the selected components.   One thing that I can promise -- the 'decoder' does do a very good, gentle 'uncompression' of a lot of available material.

Again -- this decoder is VERY GENTLE -- and in some cases might disappoint because it is so very gentle (10dB gain change over a 40dB gain range on input) -- but DolbyA is like that.   Also, the decoder does have a slight possibility of increasing low frequency distortion -- the detector is a full wave linear (not squared) detector, has a fairly quick attack time and a release time that just barely handles low audio frequencies.  So, if the signal had already been distorted by a fast compressor, then this decoder can benefit the signal by undoing some of the distortion.   If the signal has not already been compressed, it can (and sometimes does) sound like hell -- as expected.

I have a few bugs to deal with right now -- my demo scripts had an error, where the temporary files were prematurely converted to 44.1k, so wasn't getting the most benefit from the quality of the various components.   I am correcting the scripts so that everything stays at 96k (or whatever frequency is selected) until the final, mp3 conversion stage.

After this bug fix above, then I'll see about producing the Win64 binary.

John dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-15 14:59:59
Good news (I hope).  I think that I now have a working Windows64 binary -- command line only.  I don't do GUI code, so if someone wants to do a simple GUI and repackage for sale -- just give me credit and be nice (of course, if someone distributes a commecial copy, they should probably compile it on their own so that they can maintain the codebase.)

I tried to leave a simple README.txt to help someone to use the program.  Included in the zip are some cygwin dlls necessary to help the program to work.  I built all of the fancy support stuff into the binary -- statically linked it in from the cygwin distribution so that you don't have to deal with a myriad of shared libs (actually, not that many -- perhaps only a few others like the standard library stuff.)

Good luck...  I do promise that I didn't place anything evil into the .zip, and I created the .zip under Linux, so it is probably clean...

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-16 16:34:42
For anyone doing listening tests -- I MADE A MISTAKE IN THE pseudo-DolbyA decoder...  If you look at the source, you'll notice two filters that are commented out in the main processing/filtering class.   Those filters should NOT have been commented out, therefor there is a bit of edginess that is fixed by reinstalling the filters.   The demos will be fixed by EOD today (16nov2017.)  I'll also post a description of the fix so that people with source code cfan fix the problem.  (IT IS EASY TO FIX, JUST remove the // comment out, and recompile.)   I'll recompile for the Windows people also.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: john33 on 2017-11-16 17:03:51
Thanks, John, looking forward to having a bit of a play with this. ;)
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: punkrockdude on 2017-11-16 19:01:51
I have not tried it myself but I just want to add that this is really interesting and also a very big thank you for right away supporting Linux.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-16 19:11:37
Good news -- I have the corrected source ready -- and the binaries for Linux reside in the zip file -- they are the latest.  The actual patch is this:

In the file 'audioproc.cpp', at approx line 1410 (plus or minus) , there should be a code sequence like this:
        //
        // Slow down the gain changes a little (help to avoid aliasing)
        //
        pgainl = dfiltlA(pgainl);
        pgainr = dfiltrA(pgainr);

Unfortunately, during debugging/optimizing I had commented out those two lines.  It was wrong to have commented out those lines, additionally I just made a change to the filter parameters to remove just a bit more of the aliasing.  These filters are not critical for normal situations, but material with lots of MF/HF content (like ABBA) will cause the expander to produce a harsh edge on the sound.  It is exactly this kind of thing (not filtering out potential aliasing) that can make computer processing using conventional compressor/limiter design sound harsh.  (I finally noticed this harshness when running the music through my multiband finalizer -- and it is VERY VERY GOOD at bringing out faults in digital material.)  When I started chasing down the reason for the terrible sound (coming out of my finalizer -- and had realized that the signal was being disrupted in the simple 4 band expander.)  The more complex 'recovery' software doesn't have this problem - it has a much stronger solution to the alaising problem, along with this simple one.

So, throw away the old attachment, and only use this one.  The abba files on the demo repository have been corrected so far up to abba-119-waterloo.  I am going to put the rest of the abba demos aside, and work on the Sergio Mendez examples next (they are almost a religious experience in sound!!!)

I have attached the corrected code, and the zip file contains the built versions of the programs for Linux.  The windows64 version will come out soon.  The previous windows64 version DOES WORK, but it also has that harshness that I'll be fixing in a few hours.

SORRY -- REALLY, REALLY for the mistake.  I am mistakenly not using my professional discipline when working on this stuff. but I hesitate to make to much of a bureaucracy -- maybe it is time to do so.

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: mycroft on 2017-11-17 09:28:34
I do not get it.

Why are you comparing mp3 with mp3?

In source code I see script makemp3 which converts wav to mp3.

Comparing lossy transcoding with another lossy transcoding is futile.

This software could be more scientific if you actually compare lossless with lossless.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-17 13:00:20
The processor code (the da-avx or da-*** program)  (the 4 band one) basically (in the actual program) does 32bit floating or 16bit signed integer wav file I/O -- 44.1k, 48k, 96k or 192k.   The only reason for the flac/mp3 file conversions is for convienience -- there is also a script that allows playing the music directly in realtime.  MY USE OF MP3 ON SIMPLE DEMOS IS DUE TO LIMITED SPACE.   I do use the best reasonable conversion params for the MP3.   Flac 96k/24bit available by arrangements if you really need.  24bit/96k or floating-point/96k are my native formats for doing work.

So, nothing severely distorting is going on.  MP3 on external demo stuff is out of necessity  -- but the conversions are usually well above 200k.  For convenience, I supplied flac file and mp3 file conversion scripts.  If the makemp3 script is missing, then that is my fault -- I have a whole series of scripts that do conversion IN/OUT of the pseudo DolbyA (4 band) processor.   I'll check to make sure that the makemp3 script exists -- it is up to you to make sure that you have sox (a very common audio tool) and lame (one of the better mp3 converters.)  THE ONLY REASON FOR MP3 over OTHER LOSSY SCHEMES IS IT BEING COMMON.   I prefer opus, and usually use it at 258k or 384k.  Flac is essentially perfect, and is what I use for high quality archives (usually 96k/24bit.)

Using the program in its most basic mode -- think of this:
1)  You have a piece of over-compressed music -- youtube stuff is often a very good example.  However, a lot of commercial stuff is also over compressed (refer to the originals on my repository -- directly from the source that I got them modulo a moderate/high bitrate mp3 conversion.   Only reason for using mp3 on demos is a lack of space.   I can provide people .flac 24bit 96k to anyone who asks.
2) You want something a little less 'obnoxious' sounding, but mostly just want to simple tool without lots of tweaks (the tool has one tweak that isn't terribly sensitive.

HERE IS WHAT YOU CAN DO on LINUX (or if you have a good WINDOWS BINARY):

LInux or some windows configs:
da-avx --thresh=-3.0 <infile.wav >outfile.wav

Windows confs that the above doesnt work:
da-avx --thresh=-3.0 --inf=infile.wav --outf=outfile.wav

The infile.wav would be the ugly inputfile.   The outfile.wav would be the new output file.

The output file will have a little less excess ambiance and often have significantly less tape/room noise (on older stuff.)

Notes about the da-*** programs:
* The expander is very gentle -- not much information is really lost unless it is truncated away by using 16bit I/O and ineffective dither.
* Theoretically, the 4 band expander can be undone, but I haven't had to do that, so I didn't bother writing an undo program -- more useful things to do.
* Any of my external use of mp3 is because of limited external storage.   I have almost unlimited storage locally, so mostly use 24bit flac at 96k.

MP3s at 48k produced from pristine sources are good enough to see the improvement based upon the recovery/correction software or the simple 4band expander.   I understand that mp3 isn't perfect -- I only use it for external compatibility purposes, and normally use the 'extreme' mode preset of lame when producing mp3s.   Otherwise, for internal archives where I don't want to use flac, I use opus at the highest reasonable rate.   I don't use .wav for archive because it essentially wastes O(half) of the space.

On the archive site I have only provided several unchanged source files -- there are also 'resurrected' versions of those files to be able to hear the difference.   If you listen to those without a comparison -- you'll be able to tell that something VERY GOOD is happening in the processing.  Almost all decisions are based upon space limitiations.   If you want 'my best' copies of something that you are interested in so as to 'better' evaluate -- just let me know
I have some copies that are so good -- they are dolbyA master tape copies.  I will not/should not release because I happened into them many years ago when I was working on another thing all together and forget the history.

So -- let me know what will help you, and I'll try to do so.

John Dyson
Title: Promised updated expander binary for win64.
Post by: jsdyson on 2017-11-18 01:20:00
Attached is the corrected binary version of the 'mild 4 band expander' project (esp good at undoing mild multi-band side-chain compressors. (as I had promised).   I have talked about the sources elsewhere (forgot what I have said here -- been chatting at a lot of places.)   One place for the source code is da.zip on: https://spaces.hightail.com/space/bOPBXTkeeT

Please contact me here (or email -- if preferred):  johndyson10@yahoo.com

Be good, safe, and enjoy.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-19 13:52:10
I have posted some tangible results plus some hints about listening for improvements when using the da expander.  (It is NOT dolbyA, but it is da for its name!!!)   Remember:  the expander is fairly gentle, and when it is helpful -- it helps.   If it isn't applicable, it tends to be very obvious -- so it does undo moderate amounts of unneeded compression -- but it DOES NOT "REMIX" the sound!!!!  If this program was generally destructive, I would NOT be making it available.  Also, the same location below that has the two .flac examples, it also has the source code, Windows64 binary and Linux binary.   The processor is likely more reliable at 96k instead of 44.1k, but I did do the run at 44.1k -- most testing has been done at 96k and 48k, however.

I have posted the music output of two runs on the following site: https://spaces.hightail.com/space/bOPBXTkeeT
The file names are:  03.with-orig.flac  (the original version), and 03.with-da.flac (the processed version).

If you listen very carefully to the two even though the levels SEEM to be similar, the beginning noise is significantly less on the processed 'da' version.  Have to listen quickly for the noise.  Also, the voices peak significantly higher on the processed 'da' version.  There is less of the 'tight' compressed sound throughout the example also.

For tangible results below -- you'll notice that on the 'da' version the peak/RMS is significantly higher than for the 'orig' version.
Also, the crest factor is higher on the 'da' version.


jdyson@JDworkstation s0]$ play 03.with-da.flac stats

03.with-da.flac:

 File Size: 3.00M     Bit Rate: 800k
  Encoding: FLAC         
  Channels: 2 @ 16-bit  
Samplerate: 44100Hz     
Replaygain: off        
  Duration: 00:00:30.00 

In:100%  00:00:30.00 [00:00:00.00] Out:1.32M [      |      ] Hd:1.3 Clip:0   
             Overall     Left      Right
DC offset  -0.000711 -0.000711 -0.000104
Min level  -0.944031 -0.944031 -0.726776
Max level   0.885742  0.885742  0.852661
Pk lev dB      -0.50     -0.50     -1.38
RMS lev dB    -20.54    -20.54    -20.54
RMS Pk dB      -9.65    -14.02     -9.65
RMS Tr dB     -95.38    -95.38    -94.94
Crest factor       -     10.05      9.08
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      16/16     16/16     16/16
Num samples    1.32M
Length s      30.000
Scale max   1.000000
Window s       0.050
Done.
[jdyson@JDworkstation s0]$ play 03.with-orig.flac stats

03.with-orig.flac:

 File Size: 3.11M     Bit Rate: 828k
  Encoding: FLAC         
  Channels: 2 @ 16-bit   Track: 3
Samplerate: 44100Hz      Album: The Very Best
Replaygain: off         Artist: Sérgio Mendes & Brasil '66
  Duration: 00:00:30.00  Title: With A Little Help From My Friends

In:100%  00:00:30.00 [00:00:00.00] Out:1.32M [      |      ] Hd:4.4 Clip:0   
             Overall     Left      Right
DC offset  -0.000790 -0.000790 -0.000140
Min level  -0.794220 -0.794220 -0.499115
Max level   0.732880  0.732880  0.599609
Pk lev dB      -2.00     -2.00     -4.44
RMS lev dB    -22.02    -21.39    -22.76
RMS Pk dB     -12.90    -15.25    -12.90
RMS Tr dB     -94.08    -94.03    -94.08
Crest factor       -      9.32      8.24
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      16/16     16/16     15/16
Num samples    1.32M
Length s      30.000
Scale max   1.000000
Window s       0.050
Done.

Title: DolbyA concept put aside -- better design in the works and is functioning.
Post by: jsdyson on 2017-11-22 09:42:29
I have put the 4 band aside for now -- pretty much latest sources reside on the repository.   I found out (by some experiments) that the 'large scale' processor can usually do 90% of what the 4 band can do -- but also MUCH MUCH MORE.   Frankly, in some of my experiments, the noise reduction capability (due to some uncommon algorithms) SEEMS TO GO far beyond a normal expander.  In the same way the 'large scale' APPEARS to do a pretty good (if not amazing) job of dynamic range expansion for transients.   I did learn some limitations of the transient expansion mechanism -- should have realized from the math -- so I had to only allow enable/disable rather than tuning it.  (The transient expansion tends to fit hand-in-glove with the original compression that damaged the music and the normal expansion mechanisms in the expander section of the processor.)   I still plan to give out binaries and a little later source code -- but there is A LOT more work than the simple 4 band.   Because this new processor is so very aggressive -- it even has the ability to recompress and relimit if too much expansion is requested.   The tunables are too numerous right now, and I need to do some work on that.   Here are some short examples (before/after.)  The original version is converted directly from the source copy used to produce the 'after' version.  There was no special tuning in this case, but the expander was set to a more aggressive mode.   The most recent work on the expander has been to further refine the NR capabilites, which I estimate at approx 10-12dB (with an 18dB maximum NR.)  This is NOT your average audio processor!!!   Attached are the shortened examples...
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-22 10:01:24
One more example -- slightly less processing.   There might be too much processing for some ears on the original 'restored' version, so I backed off about 30% of the processing in this example.   Note that I still don't have 100% of the surging fixed again -- I have retuned for NR a bit better, and so getting the higher level dynamics tuned to the 'ear' is important.  Need to decide if I need to make the depth of fast (and linear) expansion faster, make the slow (dB/dB) expansion slower or faster or what other combination.  It sounds to me like the hold time is a little too long -- but the tradeoff is LF distortion.  There are so darned many tradeoffs that it can be very UNFUN to work on this stuff!!!  So, below is a version of the demo with less expansion.  (I dropped the expansion from 1:1.60 to 1:1.40 to give an idea as to the difference.)   I didn't change any timeconstants (which are pretty much fixed anyway -- but appear to need to be retuned because of the vast amount of NR enabled.)
Title: One more 'full function' processor example -- NR and dynamics improvement
Post by: jsdyson on 2017-11-22 11:24:44
Got another example -- I have partially retuned the restoration processor -- most got rid of the surging.   it was mostly due to the depth of linear expansion being allowed to be too deep in the time/dB axis.   I made the dB expander sections more solid (longer averaging times).   The dB expander sections use an atypical attack/release structure -- because it isn't one.  It basically doesn't understand attack/release but rather chases the signal around for reasonable dB bounds.   If I had used a traditional attack/release structure, the surging and expansion artifacts would be MUCH worse.  Unfortunately, there was a small loss in noise reduction -- because the gain control for low levels is less quick.   This whole thing is a tradeoff.  Even though it is not scientific, I have a sample of another song off the same album.  The same song gets boring over and over again -- also the original example '14.pais*' wasn't really the 'best' quality recording, but REALLY SHOWED the general capability of the restoration processor.  In that previous example, there was even some pre-limiting and limiting and super-fast dynamic limiting in the result -- but I'll bet that you cannot tell when it was happening!!! (very good limiter design -- doesn't distort or intermod unless desperate.)   So, attached is the example with the retuned result.  (One reason why I sometimes have troubles with the examples is that my ears are so darned variable -- so at night I'll think that the sound is okay -- then in the morning, I'll hear a defect.)  The only defect that I can tell this morning in this example would normally be inaudible (might be able to see the noise floor, however with careful digital analysis.)
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-25 06:20:38
Learned something new about the 4 band expander (pseudo-DolbyA) expander design.  I had noticed a certain harshness in the decoded output -- heard that problem before, but didn't add two and two together.   Whenever one does an expand or AGC compress, there are interesting (and often unpleasant) modulation products produced.    First note, I don't have an entire DolbyA scehmatic, and simply assumed that the outputs of each band were just summed together) which they might still be.  However, whenever doing a compression (especially something close to a 2:1 type compression or more with moderatly fast attack/release times), there are significant intermod products produced.   So, if you have two signals -- perhaps at 1000Hz and 1030Hz -- youll get 30Hz in the output.   Normally, in my compressors and expanders, I tend to remove as much of those products as possible.  I didn't do so in my 4 band expander, because I had assumed that there might be no 'high pass filter' after each band IN THE ENCODER/COMPRESSOR to remove those products.  I suspect that there is indeed SOME high pass filtering that might help to remove some of the products.  Alas -- in my expander I had NO removal of the products except for a rather rough 12Hz type HPF (to basically remove most stuff below 20Hz.)  I was using only a singular HPF in the previous version of the code for the total/full/wideband output.

So, I kept on hearing this 'rough' sound (esp in groups like ABBA -- where the voices do tend to intermodulate) and using that group as source material has been one of my personal audio tests.  This 'rough' sound has been worse than what I desire -- then finally eureka -- I suspect that it is possible that DolbyA (or other compressors used in the source material) is definitely doing a least a minor amount of HPF on each channel.

So, as a gross experiment, I added a 2 stage each of single pole HPF of approx 12Hz, 60Hz, 1kHz, 3kHz for each of the 0-80Hz, 80-3k, 3k-20k (or 3k-9k -- depending on where it is in the circuit) and 9k-20k directly on the outputs of each of the bands individually.  This will help to keep most of the intermod that is created below each of the above bands attenuated to some extent.  The worst intermod appears in the 0-100Hz range, and that is obliterated in the two higher bands, and MUCH of the intermod produced by expanding the 80-3k range is also removed.   I am doing nothing about the intra band intermod for this case (that is a more difficult problem, and less of an audible issue, but still can be important), but that intermod would definitely exist also in the compressed (encoded) version and some amount of cancellation can be beneficial (alas -- tape might have too many phase prolbems for the cancellation to be complete.)
Anyway -- some of the additional intermodulation artifacts have indeed been removed, and there seems to be a noticeable improvement.  In the past, I have definitely measured the artifacts, but would be of minimal benefit in this case since I don't have a DolbyA to experiment with -- but that is not important right now since I am not selling anything.   All I can definitely say is that there is now an improvement, and the 4 band expander DOES improve some (most) commercial material that I have processed.  (I do NOT attribute that to DolbyA encoding, but rather a lot of compression being left in commercial music.)  I have a much more flexible expander that shows the DolbyA parameters for expansion to be helpful, but do NOT do all possible to improve most commercial materials.
So -- the good & useful news -- especially for those in the future who are developing (esp mutliband) compressors/expanders -- make sure that you have mitigated the LF side of the intermod produced by any gain control mixing with the audio signal.  Even I have just got caught by that issue -- and I have been aware of that problem for years!!!

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-27 13:55:53
I just learned something new -- this is about the 'full scale' restoration processor -- not just the 4 band expander.   You might remember after reading previous posts that the full scale processor has some transient restoration math in it -- it actually does a good job of modeling the transient behavior of the music, and help to re-establish the intensity of transients that were previously blunted off.
When implementing the transient processing code, I added 'diapers' around the code to keep it from doing too much.  The theory was that I was worried about adding distortion to portions of the waveforms that werent transients.  So, I slowed down the transient handling to about 250usec (about 4kHz) at the highest frequencies, and equally relatively lower on lower bands (approx 2-4X lower than the lowest frequency in the band.)  In order to mitigate the ripple, I did a peak filter a couple of times to make sure that intermod would be minimized.
Well -- I might have been wrong.  I removed many of the 'diapers' (and added carefully crafted replacements that do things differently) in the code while playing back a really good original of Brasil'66...  You know -- the male vocals having a 'clipped' or 'heavily limited' sound to them...  Instead of getting 'distortion', I ended up with better reproduction of the transient portions of the voice.  It isn't perfect (yet), but much more perfect than the original and much better sounding so far.   Removing the 'diapers' required changing some of the math so that the levels were more carefully measured and matched to the gain control algorithms (akin to using RMS or peak limiting -- that kind of modification, but not the same.)  So, I am continuing the experiments with even better reproduction of some old stuff.  The results of this experiment help to demonstrate the effectiveness of my transient recovery algorithms -- however slow they are.

Another goal -- not yet 100% achieved -- is to help to restore the sound of the lower order of the guitar in some Simon and Garfunkel songs (in the older recordings.)   There is some definite over-compression/intermod/room effects that could benefit with just a little less of those distortions.   The current version of code (before this last change -- no examples released yet of either) does help with some of that old-time '60s intermodulated sound, but not quite 100% (or -- maybe 50-75% qualitatively is my goal.)

John Dyson  
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: DVDdoug on 2017-11-28 16:59:38
 John,

When you started this whole thing you mentioned the "original sound" of The Carpenters, so of course I thought you were talking about the original vinyl.     And as you may have heard, they've just re-released a re-mastered vinyl box set and .  I just thought that was an amazing coincidence!  

I have no idea if you'd like the remastering or if you have any interest in vinyl but  I thought about your post when I heard about the re-release.  As of now, it looks like the re-mastered version is only on vinyl but if it sells I assume they'll release it digitally.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-11-28 20:39:46
Yea -- when I speak of the 'original sound' -- I mean the original sound that isn't all scruched up with compression.   Since I have been working on this project,  I have begun to notice 'compression everywhere', and 'too much compression' too often.   When listening to every normal source that I have of the Carpenters --including a friend supplied quad tape -- I notice FAR FAR too much compression for what is necessary.  They did it somewhat artfully, but totally unnecessary.   In the quad tape, I notice less false sibilance (sp)  in Karen's voice.  EVERY other piece of the the specific songs that I have heard, they put an edge on her voice (almost a raspy sound) that is totally unnecessary.  My processor does seem to help that a little, but I haven't done tests on the Carpenters music recently, because I am working aggressively on percussion type sounds.   I'll probably rotate back through and look at the general smooth compression like the Carpenters and the aggressive matrixed compression like ABBA again soon.   Also, I have made inroads into the old-style effects like the intermod/distortion in the original Simon&Garfunkel guitar stuff, and also have made pretty good improvement in the clipped/over-compressed male voices in Brasil'66.   This has been an iterative thing -- optimizing the structure of the gain control filters (the kind of linearity issues, transient recovery, etc.)  Amazingly, my latest breakthrough has been in transient recovery, where have made significant (but not major) improvement in removing some of the over-compressed/almost clipped sounds in the voices on Brasil'66 & Sergio Medez stuff.  The transient recovery algorithms are KEY to the general dynamic range correction capability.  I really hope to have a USABLE version of the processor ready soon -- I just added the simpified command line structure for the processor - so that one doesn't have to jigger around with 20-30 parameters -- even though most of those parameters don't need to change.  I have it down to about 5-6, which is reasonable to deal with.
I am currently not demoing the results much (outside of small group of friends) right now, due to a lack of feedback -- which I really need to get this thing working without depending upon the eccentricities of my hearing and my limited abilitity to measure some of the behaviors.   Actually, the processor is being slightly simplified, but the algorithms are VERY SIGNIFICANTLY refined even relative to 1wk ago.   The reconstruction of previously lost transient and peaks in voices and from instruments appears pretty natural sounding, even though if you listen VERY CAREFULLY, you can hear that it is not quite perfect.  Oh well -- I am making progress, and actually demoing the results using the Carpenters music is one of the first times that I got a WOW.
John
Title: A WORST CASE example of the full-out restoration processor
Post by: jsdyson on 2017-11-28 21:19:10
I kind of feel bad about not putting out any demos -- so here is one.  This is a segment of 'Pais Tropical' from Brasil'66.  That song is recorded in a very 1960s' style, and has sound quality that we'd probably call today -- distorted.   This is a rather extreme case, and the processor can actually sound MUCH better on sources that are somewhere between pristine and this...  On pristine sources, you don't want to use the processor.  However, on sources that are just over compressed or have had too much peak limiting, or needs significant noise reduction, then the processor is 'your friend.'  Again, this is an extreme case, and IF YOU CAREFULLY COMPARE, you'll find that on the processed version, the male voices come out slightly clearer and you can actually hear the voices rather than just being a distorted, over compressed mess.   THIS RESULT IS NOT PERFECT, but does show some benefit even on music with a rather intense 'personality.'  To give you an idea of the kind of processing -- it has a total of approx  1:1.33 4band expansion, 1:1.86 8 band expansion, with further transient expansion.  It doesn't always do the full expansion ratio, but also doesn't always add 6dB of transient peaks either (that wouldn't be right.)  However, the processor is practically 'turned up all of the way.'   The ONLY reason why the processor can do so much dynamics processing is that it is being done carefully and NOT with a simple brute force expander (NOTHING like a traditional RMS/VCA HW expander algorithm.)   I don't claim perfection on this (on some other stuff, it can knock your socks off, but this is intellectually interesting to me.)
Title: A typical case result when using the restoration processor
Post by: jsdyson on 2017-11-28 21:31:22
Here is one more -- probably more typical song -- Mais que nada?   You can hear the NR there, but of course the transient processing is pretty good also.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-12-19 13:19:00
Update on pseudo-DolbyA...  Because of the difference in dynamics between feedback/feedforward, there has been an error (I believe) in the gain tracking of my decoder.   After some minor mods, been running some tests, and the results appear to be improved when running tests on some very-likely DolbyA material (2tk master from popular group.)  Sorry that I don't have stronger objective results yet, but even at this stage the investment would be significant.   With the mods, the gain control appears to track more deeply into the signal -- which means that it is now significantly faster without forcing a change in timeconstant/etc.  (Some mathematical changes were made to emulate the difference between feedback/feedforward without changing the basic timeconstants as painstakingly gathered from schematics and paper research.)   Of course, the TCs from the schematics are feedback timeconstants and some kind of correction is needed -- but simple change to the timeconstant doesn't seem to be the best choice.   The goal is to maintain some traceability without 'stealing' another design.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-12-30 09:14:18
Finally got the newest version ready to try. It is better than any version before, and I have added some of the secret sauce that I had previously used on other projects. It just works better with the new code. There are three files:

1) Windows binaries "da-win-30dec2017.zip"
2) Windows sources and binaries "da-30dec2017src.zip"
3) Linux sources and binaries. "da-Linux-30dec2017.txz"

The only difference between the Windows builds and the Linux builds is that the Windows builds tunables are decreased a little so as to run more quickly on slow machines. The actual quality difference is probably nil.   After some more tests by me and any feedback that I get -- I plan to do one more release within a week with all bugfixes that we might find.   The code is essentially complete, but some tunables might be optimized.   After this iteration, the larger scale 'cleanup' or 'restoration' processor will be my only main project.

Make sure that the windows dlls (as in the Windows binary packages) reside in the same place as the da-win.exe program.

To decode a file, do it like this:
da-xxx --thresh=-1.50 --info=99 <infile.wav >outfile.wav

The threshold is dependent on the dolby level used on the tape. it makes an important and sometimes not-subtle difference. Values between +1.5 and -9.0 are usually best, but I have had the most luck between -0.75 and -6.0.

Program names:
da-avx -- latest Intel CPUs on 64 bit Linux
da-sse3 -- Recent ATOM CPUs on 64 bit Linux
da32-p4 -- older CPUs on 32 or 64 bit Linux
da-win.exe -- ATOM CPUs or newer on Windows64

Use the latest program version that you can -- newer CPUS witih the right programs can sometimes run realtime. The program is NOT multi-threaded, but it is beneficial to have two cores available if you are on the edge while running realtime.

I don't have time to describe all of the options -- but other postings might give some hints, and this decoder can do much more than just a straightforward pseudo-DolbyA decode.   If I get some feedback (and needs arise), I'll document some of the more esoteric features in the code.

The expansion logic is NOT DolbyA compliant -- but should be superior in some ways. The processor can sometimes run close enough to DolbyA to undo some of the intermod that naturally occurs when encoding the material in the first place.

The files reside on this repository:

https://spaces.hightail.com/space/bOPBXTkeeT



John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-12-31 05:26:26
Here I am posting some of the smaller processor files here.  Hopefully, someone will get some benefit.   The programs are free to use -- there is some GPL based libraries which are part of the cygwin suite and can be downloaded at their site.  The processor source is also free, but if it is used, just tell me and if it is for sale, give me a bit of credit.  There are actually some good ideas in there far beyond just the specific application...

Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2017-12-31 11:31:06
I have something perhaps useful -- a screenshot of the spectrum of the following:
Top -- full processor only (not including the psuedo-dolbyA)
Middle -- pseudo-dolbyA
Bottom -- original.
Song:  Rainy Days & Mondays from the Carpenters.

It is possible (and desirable) to use both the pseudo-DolbyA and the 'full processor' (not yet named), and the benefit of the two is greatly increased, but the artifacts don't get much (any?) worse.  The pseudo-DolbyA being tested is only a few minor source code changes from the version that I have distributed.

How to analyze the output:  take a look at the density of the areas without significant music.  The areas where there is no material will show some 'snow' or constant low level blue coloration.  The two processors remove/decrease a significant amount of the 'blue areas'.  The effect is subtle -- so you do have to look closely to see the difference.   When running the software, the pseudo-DolbyA is swinging a range of approx 10dB gain between 9-20K, and perhaps 6dB at 6-9K.   The lower ranges show only 2-3dB gain swing except in certain cases (where sometimes the LF shows a wider 6dB change.)

For the 'full processor', it shows approx 9-16dB gain swing for the freq ranges above 5K, and perhaps 6dB from 100Hz to 5kHz.
(The gain swing measurements come directly from the software itself -- impossible to measure from the spectrograms, which are qualitative only.)

The spectrograms are attached -- where the 'top' is the full processor, 'middle' is the psuedo-dolbyA that I am distributing today, and the 'bottom' is the original.  The differences are VERY SUBTLE!!!



Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-01-02 19:22:29
Claification about the 'pseudo-DolbyA' decoder.  I want to make it clear (because this question came up in another forum), that this decoder often has positive effect on NON-DOLBYA-ENCODED material. In fact, the Rainy-days-and-mondays example that I gave the spectrogram for was NOT advertised to be DolbyA encoded.  The 'decoder' is an expander that undoes some analog sidechain compression.  Because of that architecture, and (helpfully) the use of a non-traditional attack/decay calculation -- it can be effective in other cases where the material was compressed with a sidechain (or even non-sidechain) compressor.   There'll still be SNR and some limited transient improvement, even if not DolbyA encoded (but is compressed.)

So, when I suggest that people give it a try -- it is NOT necessary to have 'DolbyA' encoded material, even though results from such experiments would be interesting.   I do have some material that is DolbyA encoded, but I cannot give it out as examples (2Tk mixed down master of pop music.)  Because of the kind of intentionally distorted dynamics in the original material, it is impossible for me to accurately determine the quality of the DolbyA decoding.  (every CD that I have heard based upon the material that I have sounds so very different, because the mastering engineers have really played with the 2tk master.)

Regarding the spectrograms:  The reduction of 'blue fog' (mostly hiss) occurred on compressed material that benefited from mild expansion (but not so much expansion as to badly distort the dynamics.)

The 'larger scale' expander can make more significant impact on dynamics, and it is a bit more tricky to do that correctly without causing unpleasant expansion artifacts.  However, the design of the 'large scale' expander (dynamics restoration software) is such that it is easy to set the parameters so that the results can sound better than the original.

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-01-03 08:06:49
I have posted a much more substantial demo of the available noise reduction using the 'four band expander' or psuedo-DolbyA decoder. This demo should work for most people who still have at least some hearing (remember -- I do have age related hearing loss, so sometimes HF NR is hard for me to discern.)
On my repo site, I have a few files which should pretty clearly show the expected NR associated with a Dolby-like NR scheme. This demo shows both HF and LF NR (which DolbyB only has HF NR.) The resulting NR is modest, but useful. The NR doesn't generally have a negative impact on the sound when it is needed.

I simulated the noise from a typical cassette deck -- so it is pretty obvious. Also, for a 'bonus', I simulated a little bit of turntable rumble that had been cut off at 20Hz. II have included the noise generation/addition script for additional review.

There is a definitely noticeable NR -- both files have been roughly equalized in volume. Due to the effect of of the expander, certain portions of the music show a difference in HF, but the difference (IMO) seems to make the music more accurate. The original piece seemed to either have a little too much HF emphasis or might have been DolbyA encoded.

Files at the repo site -- available for download and comparison:
carp-noise.mp3 -- noisy version of the music
carp-noisered.mp3 -- less noisy version of the music
carp-noise-stats.txt -- sox stats showing approx signal levels for the noisy file
carp-noisered-stats.txt -- sox stats showing approx signal levels for the noise reduced file
noisever.sh -- shell script with sox commands used to add noise

The repo site: https://spaces.hightail.com/space/bOPBXTkeeT

This NR does work with files that are NOT explicitly DolbyA encoded. I just reran a test with a song from this 'shawn' person (youtube download), and this 4 band expander did improve the dynamics (tastefully sharpened the sound -- slghtly backed off some of the over compression.)

It order to avoid confusion, I am not explicitly talking about the more aggressive expander project at this time. I am really trying to help the community by trying to help improve the sound quality of (mostly) legacy recordings. It is possible to do more processing, but I believe that the processing should make the music/sound 'better/cleaned-up', not 'more' or 'overdone'.  I am trying to keep the results at 'tasteful' levels.

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-01-03 09:14:28
following up on the previous post -- I am attaching a spectograph output resulting from the NR associated with the various expanders.  I am concentrating on the 4 band right now, but this posting shows the results of the following:
carp-noise-8band-4band -- the spectrum for using both the 'pseudo-DolbyA' and full-out expander.
carp-noise-8band -- the spectrum for using just the full-out expander
carp-noisered -- the spectrum for using just the 4 band pseudo-DolbyA expander.
carp-noise -- the spectrum of the noise added signal alone with no processing.

If you look at the spectrograms carefully, you can see that the carp-noisered does help with the HF noise (it is not easy to see the LF NR also, but it is noticeable when listening.

carp-noise-8band (which I am not concentrating on right now, but the results are here) shows that it is a little better at finding the 'blank spots' to do noise reduction, but isn't quite as good at the ultimate NR at the highest frequencies.  You can see relatively blank sections in the upper-middle range where the 8 band found some of the noise to remove (it is both faster and has narrow bands to work with than the 4 band.)

When using both NR/expander systems, the advantages of both can be seen.   There are much larger 'lighter' areas where noise has been removed.  However, the darker areas with real signal are still there.

The 'carp-noise' and 'carp-noisered' examples reside on the repository that I have referred to before.  I'll send the results including the 8 band results right away (you should probably be able to see them on the repo by the time you get done reading this.)

John
Title: Very major improvement to 4band expander (pseudo-DolbyA decoder.)
Post by: jsdyson on 2018-01-03 21:47:58
Got a major improvement -- source code and Linux binaries only at this point.  All along, there was a problem that sounded somewhat like a garbling that wasn't quite right (under certain very dense music conditions.)   This manifested on a 2trk master tape that I use for some tests, but has gotten better and better over time as I had improved the dynamics of the gain control (gain change vs. time.)

After I did a more careful code review, I noticed that there were some gain control operations that were happening against an overly wide band representation of certain parts of the audio signal.   This basically created excess intermod products.   So, I did a very careful review of the cases where the gain change operation was being done at slightly the wrong time and corrected that problem.  Also, there was a minor 1msec delay problem between the signal level sense and the actual signal itself.   This also caused a slightly rough sound.   These problems were all forms of intermod -- the bane of any kind of audio gain control.

All in all, these improvements are worthy of an early re-release of the source code.   On simple kinds of music (e.g. Brasil'66), the improvement is not really noticeable.  But some kinds of very dense music (e.g. ABBA stuff), the improvement is quite important.

This early release has some test code in it (commented out), and is a little more 'unclean' than usual.  However, the fixes are installed.   I'll release a windows version when I am done with the full release (windows is a bit of a pain in the b*tt for me to deal with -- I usually just use windows for circuit analysis.)

Attached is the new code.   It also resides on the repository.

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-01-06 21:31:46
After the above improvement to the psuedo-DolbyA, and some additional gain-control/signal synchronization on both the psuedo-DolbyA and the restoration processor -- I have gotten some amazing results.  It has been critical to shape the attack times very carefully to avoid intermodulation type effects (a kind of garbling.)   For ANYONE who likes ABBA (and a Carpenters example), give the below a listen.  I plan to release a semi-final version of the psuedo-DolbyA within days, and start considering/pruning down a release of the restoration processor.

Listen to these results -- they sound like ABBA -- but not quite, maybe like ABBA was recorded in the 2000's instead of the 1970s'.

https://spaces.hightail.com/space/pG4t4ZFnyB

Imagine VERY SMOOTH/CLEAR ABBA with a LOT less compression.

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-01-06 21:45:38
Attached is the latest pseud-DolbyA with very significant bugs fixed again.  These aren't the kind of bug where there is an 'off by one', but rather the attack times are more carefully calculated.  The resulting improvement is seldom heard except on HF intensive material like ABBA.   The included fixes (and the associated fixes in the 'restoration' processor) were critical for the amazing results that I recently got on 'cleaning up' some old ABBA stuff.   Up until now, ABBA always had a garbling effect when multiple voices were in play -- and the result was kind of ugly.   The problem seldom existed anywhere else, and most of my previous 'fixes' did  help, but were either work-arounds or fixes in minor portions of the code path.  These new improvements change how the attack (gain attack) occurs, rather than just being an instantaneous response or a RC delayed response -- there is a hybrid technique that I saw R Dolby use, and I started applying it in other cases -- with rather amazing results (the guy was a genius.)  My own implementation is software based version of the hardware concept.   Attached is the latest source code and build for the psuedo-DolbyA on Linux.  Like I mentioned before, the Windows version will be forthcoming when I have a chance to do development on my Windows64 box.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-01-22 11:10:03
Real nasty gain control bugs in my projects (partially due to hearing troubles -- fixed.):

I have been working on some hellish bugfixes -- I had lost part of my hearing recently, and luckily it had come back.  I made some serious mistakes in both the pseudo-DolbyA and restoration processor.  During the process of fixing the bugs, I have learned some semi-interesting facts about these kinds of high speed expander/compressors in the digital realm.  Note that  I have definitely fixed a lot of potential design issues over the years (incl mitigation of some pretty evil intermodulation distortions, etc) on compressor designs.  Well, my hearing lost approx 6dB at 10kHz, and I missed a huge number of artifacts in this new expander stuff.   The technical explanation of the bugs is basically -- Be VERY careful about attack characteristics.   After listening to some of my processors with my new hearing, I noticed very severe HF harshness.  Almost 100% of the harshness came from the idea that a gentle attack (I mean, gentle in a fast way -- e.g. 1msec attack time) is a good thing in the digital realm -- not necessarily true.   I had been softening the attack to make the middle frequencies work better -- but unknowingly at the cost of some HF hash.

SO -- when doing a review of the simpler software of the two projects -- the psuedo-DolbyA, the key 'notch' in the attack characteristics that come from the original DolbyA HW design does work pretty well also to help with the 'hash.'  A raw attack is not a good thing for a compander because tape machines do not maintain great phase character at high frequencies, so a bit of attack time roll-off is important.
However, even though this Dolby-notch does help the hash, it really needs more help in the digital domain where the signal frequency might only be sampled at a couple of times.   The indeterminate  timing seems to cause a bit of 'hash' when mixed with the gain control (THIS DOES NOT apply to many compressor designs that do the simple approach of SLOW attack/release times, like 20msec or slower.)   However, there is a critical design rule #1 that I didn't follow 100%, but should have (and this is most critical for fast attack times):  put all of your attack time constant on the first measurement of the signal level.  If one doesn't do that (or use another bit of technology that I had temporarily sidelined until now), then there will be an obnoxious raspyness if the attack/release times are quick (that is, when you might have a 1msec measurement, then an additional 1msec timeconstant somewhere, then a 100msec release time.)  That is a formula for hellish sound (esp on groups with lots of potential HF intermod like ABBA.)
So, when I do need to further filter something like:  1msec attack time constant, then have an eventual set of release times with a fast component (like a 100msec component for the release time), then either I make sure that there are no other time constants in the chain, or use a special dynamic filter which basically dynamically follows the signal -- and I'll release the code in the future which DOES significantly mitigate the problem of subsequent attack times -- that is multiple in series is troublesome without great care.)

The other problem that I ran into (which I knew about, but couldn't hear -- so didn't believe it to be severe.):  when one uses a medium attack time like 1msec, with a combined absolute value function (basically a linear rectifier), then there is still a matter of intermodulation (which is VERY bad with groups like ABBA -- multiple female voices with high pitch, intermodulating quite a bit.)  This nonlinear activity along with the 1msec filter is not slow enough to filter out the intermod, and is not fast enough to confine the instantaneous change in gain to a very narrow timeframe.  This narrow time frame is minimally inaudible.  I have previously had experience MOSTLY with square law detection, which seems to be a little less sensitive to the intermod than the hard absolute value function.  SO, it is critical to have the ability to quickly respond to the input with the gain change.  However, I have found some filter functions that spread the quick transient intermod further and become totally inaudible (this is a nonlinear filter function vs. time and helps quite a bit.)
Working in the domain of the linear (absolute value) detection instead of the square law works/sounds different from everything that I had worked with before.  By directly using the absolute value, the responsiveness of the filters is much quicker (with perhaps a bit more distortion -- the intermod that I spoke of above included), and very helpful for volume expanders.  For compressors, square law does work better -- even though the responsiveness is slower.
So, after very careful lining up of all of the attack times, utilization of the 'notch' trick, and also using the dynamic nonlinear filters (basically they create a symmetrical decay time based upon signal dwell -- giving fast response but don't propagate the glitches as badly into audible noise.)
I'll be release a very, very improved version of the 4 band psuedo DolbyA with 99% of the raspyness fixed.   Sometimes, a nice, soft timeconstant is destructive, and other approaches might need to be taken!!!

Results of the VERY improved code reside in the repository: https://spaces.hightail.com/space/pG4t4ZFnyB

The ABBA voices don't intermodulate as nasty now.

I'll be posting the new pseudo-DolbyA within a day or so -- I have had to include some of my 'no-desire to release' technology to filter out the hash from the gain control -- it really does help!!!


John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-01 01:18:47
New, vastly superior pseudo-DolbyA decoders now available (source, Linux and Windows).  On Linux, there is sse3 and avx optimized code (don't have a Pentium 4 version ready yet), and for Windows there is an Atom optimized version that should work on recent Atom and i3,i5,i7,i9 CPUS.   The Linux versions should work on recent CPUs -- for the Atom, one must use the SSE versions.

New improvements -- vastly superior level detector, much better than then two level detector used in the original DolbyA design, but rather very high resolution for the attack times.   This minimizes intermod on the attacks, and *might* perform better than even the original hardware.
The speed is much faster now -- I removed the attack/decay enhancer -- which is more useful for normal expanders, and not when emulating an existing design.

Source code now provides some hints as to some of the intermod reduction & dynamic attack/decay technology that I had created.
The source code for both Linux & Windows resides in da-winsrc-31jan2018.zip.   The only difference is the Makefile.
The binaries for Linux reside in da-linuxbins.txz, and the binaries for windows reside in da-win-31jan2018.zip.   Note that the dlls
must be in the same directory or in the search path for the windows da-win.exe to work.
The usage instructions are identical to previous versions.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-01 11:26:38
Good news and bad news -- the good news first:  the previous version worked OK.   The bad news (or  good, however you see it): these new versions are better yet.   Here is what happened -- I was trying to speed up the operation by removing my transient enhancement code.   Actually, it isn't really 'enhancement', but appears to properly manipulate the attack/decay so that it better matches the old HW implementation.   Even though I continue to use the transient improvement on the 'restoration' software, I thought that the psuedo-Dolby would work okay with the simpler/faster code.  Again, it does work both ways -- but there are some end cases where certain kinds of distortions created by a DolbyA encoder aren't undone UNLESS using the more advanced transient 'enhancement' code.   If you listen to a DolbyA encoded version of Rainy Days & Sundays from the Carpenters (and there are more of them out there than you might think), you'll hear the removal of the crackles in Karen's voice when you use the better code (which I have reluctantly provided here.)  I made a few mods to speed up the transient recovery mechanism (it resides in recursexp.h), and it seems to do okay -- obliterating the crackles in Karen C's voice.   So, here are the versions that have more complete capability:
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-01 12:52:48
Here, I have produced two snippets...  The first is the obviously DolbyA encoded version of 30sec of Rainy Days & Mondays, the second is a fully decoded (with the da software also using the restoration processor...)   (please don't expect to get this sound with just the psuedo-DolbyA decoder -- but it does a critical and necessary part of the job.)   If you listen CAREFULLY (and it is difficult to distinguish), but the crackles that I have talked about are removed...   If you listen to the 'rainy' at near the end of the example (about 22 seconds in), the crackle is turned into a fairly correct crescendo.  (It is difficult to distinguish unless you set the level very carefully.)  Again, without this most recent update, I couldn't remove the crackle. 
damondays-noproc.mp3 is the raw DolbyA encoded version (and yes, I got it that way)
mondays-proc.mp3 is the cleaned-up version.

The frequency balance of the 'cleaned-up' version is quite different from the DolbyA encoded version -- note the HF enhancement in the DolbyA version.

Regarding the availability of DolbyA encoded material -- listen VERY carefully to what I have provided as a DolbyA example.  You'll notice a relatively high level of background hiss and a little bit of high frequency emphasis (similar to, but not the same as DolbyB.)  The big difference (in listening) between DolbyA and DolbyB is that DolbyA compresses a wider freq range.

Now -- another odd thing is that (in this example) I didn't use a normal mode for the psuedo-DolbyA, but I used a modified version (as had been suggested to me by people from history.)

Firstly, you CAN decode the music with the raw, straightforward pseudo-DolbyA decode, but there is something else that they have apparently done (after some reverse engineering by me) that they are doing both a L+R and M+S DolbyA compression on high frequencies.   Apparently, they are doing a standard L+R DolbyA encode with an extra M+S DolbyA for the top two bands only to enhance Karen C's voice.

So, to decode this song -- I am using a M+S, then L+R HF only decode (using DBAMODE=1 on the psuedo DolbyA decoder), then use the restoration processor that easily takes care of the two low bands of DolbyA decode plus some more expansion at HF.  (Even though the restoration processor has a VERY different architecture, the restoration processor does a reasonable approximation of DolbyA decoding with some major caveats -- but it is better to use a more correct DolbyA approximation when possible.)

Note that the restoration processor (unlike most expanders) DOES NOT SKEW THE FREQ RESP!!!   This is one reason why I don't call the restoration processor 'an expander.'  The processor itself is still a little in flux, but it is slowly, but surely getting cleaned up and producing better and better sound.   I plan to release it very soon, when it wont be an embarrassment and I wont be making mistaken releases like I just did with the pseudo-DolbyA.

Full song is at: https://spaces.hightail.com/space/d4fpzoa0zD

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-02 03:19:28
Quick note -- the files in the repository just happened to be the wrong versions...  They weren't too far off, but there was an error in the production of the mp3.   I UPLOADED NEW VERSIONS OF SOME DEMOs to the repository.   The versions that I uploaded to this site were correct and I believe accurately show the improvement. You might want to check again if you already listened.   Another case where the harshness is removed is in 'Reason to believe', where some of the voice crescendos(sp?) have no distortion and are reasonably reproduced.  Most versions that I have recently heard have a kind of ugly distortion of some kind (probably compressor or DolbyA encoder artifacts.)  The processors do get rid of most of that kind of stuff.

I am also JUST THIS MINUTE working on the restoration processor -- cleaning up some of the overdesign & hacks.   I think that I am going to separate the compressor and 'restoration' functionality -- and MIGHT even distribute the compressor sources first since that code is a little more mature.   I haven't written much about the compressor recently just because the expander/restoration part of the software is usually more useful to audiophile types.

When utilized in cases where a compressor is really needed -- it is generally not very obnoxious sounding -- but does sound like a compressor/limiter.  SOMETIMES I actually have to check to see if it is operational.  The compressor has various features like a totally automatic attack/release scheme, where the release is 100% mirrored from the dwell time of the signal.  Also, there are several forms of intermodulation reduction (some are partially redundant.)  I don't know if I am going to make the release time adjustable (it does a good job automatically determining the ideal release time), and the attack time is also carefully crafted to avoid intermod -- yet has apparently instantaneous attack.   I am just thinking that sometimes it is expected to be able to tune the release time.  Most often, if the release time is forced to be a certain value -- it won't sound as good as the compressor/limiter can do for itself.  Of course, the threshold, input gain, and compression ratios will be adjustable.   The eventual configuration will probably be stripped down to a 3band on input, an 8 band, a prelimiter (which is basically a moderate compression ratio single band), and then a limiter.   When used in combination (where the 8 band is optional), the results are pretty good -- and it is simple to use.  Each stage sounds good even at over 10-20dB of gain reduction -- but usually that kind of gain reduction is more trouble than helpful.

However, the highest priority is making the restoration processor code look better, and fix nits whenever I find them.  It is NOT complex code, but utilizes lots of vector operations (SIMD instructions), which often require a different kind of programming technique -- and sometimes the code is a bit odd looking.   Just trying to make it more presentable.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-03 15:11:17
Just for fun -- I produced two dynamic range compressed files derived from 'rainy days and mondays' and 'reason to believe' on the repo site.  These files are dynamic range compressed with a fairly close-to-release version of my compressor.   It is a true RMS/DB-linear compressor & limiter system.   It has a 3band, an 8 band, a prelimiter (which is just a very safe single band), and then a true limiter (plus there is a clipper, but very seldom -- if ever -- becomes invoked.)

The demos have had NO tuning, and the source material were the exact same processed (restoration processed) mp3 files on the site -- so they had quite a bit of dynamic range.   Also, these files resulted from several generations of mp3 conversion (I originally have pristine flac DolbyA files to start with, but would have been more work to regenerate a flac version to start the test.)

The compressor should sound pretty good to most people (as compressor/limiters go.)   This has my latest anti-intermod technology, and has fully dynamic decay time (with carefully time attack times for minimal intermod.)   Frankly, the attack time is not a typical RC generated timing, so it is hard to claim what the attack time is - but it could be described as anywhere from instantaneous to 2.5msec, with probably an average of 500usec to 1.5msec as the actual amount most of the time.  The total effective attack time (for the entire chain) IS instantaneous however.  The decay times use a combination of fixed decay, input dwell time mirrored decay time and dB linear decay (depending upon the phase of the decay slope.)   There is always a minimum decay which is directly related to the period of the audio waveform (actually ends up being an approximation to a multiple of the period.)

The audio detection is a combination of a RMS and linear calculation, where the detection itself is a dynamic attack as described above.  Because of the attack shape, the intermod transient is minimized (not totally zero, but as good as one can get with a sampled system.)  Most of the rest of the gross levels of intermod are removed by specialized filtering.

Much of my testing was done with a combination of music styles (and test signals) which excited some 'troublesome' results without careful design consideration.   This is one reason why I have used ABBA alot -- they have two female voices with a similar pitch (a little different, but similar), and their voices can easily produce some ugly stuff if processed with a nonlinear system (like a fast compressor or limiter.)   If the ABBA voices pass through fairly cleanly, then that is one of my test successes.  (note that it is easy to pass such material without noticeable distortion, but the solution is to use long attack/release times -- my processing is for all practical purposes instantaneous, so there is a lot of opportunity for intermodulation at high audio frequencies -- thereby POTENTIALLY producing nasty, ugly sound.  I chose NOT to use the easy way out.)   A good example of such intermod (a gross amount) is the beginning of the ABBA song SuperTrouper on most releases (precious few releases of that song have clean rendition of the beginning.)   My 'restoration processor' does produce near-perfect results on that material.  *Below I posted 5 second start of SuperTrouper to show the intermod (recent US CD release) and no-intermod (the restoration processor) versions.

So, this is just on a lark, also since I mentioned the compressor on a previous post.   The example files are now on the same repository as the 'restored' versions of 'Rainy days and mondays' and 'Reason to Believe'.   Note the word 'compressed' in the filenames for the re-compressed versions.  These compressed versions were NOT renormalized, so you can also see how well the limiter works -- the output of the prelimiter would hit the limiter a little bit once in a while, but the limiter is essentially noticeable distortion free at least at 6dB of gain reduction or beyond. (Controls are placed on the release times so that a deep release doesn't accelerate the decay.)

Site with the Carpenters examples mentioned here below again:

https://spaces.hightail.com/space/d4fpzoa0zD
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-05 03:03:14
Made some improvements on restoration software.  My high frequency timeconstants were a little too short, and caused a fairly much harsher sound than what is really desirable.  The timeconstants cause a tradeoff between noise reduction, distortion and noise modulation.   I was biasing too much to minimize noise reduction/noise modulation, rather than minimizing HF distortion.   The restoration processor (and entire complex) already has so much NR that it really doesn't need as much as it is capable of -- so I pulled back the HF timeconstants (the detector release time constant used to be approx 2.5msec, and now is closer to 10msec in the linear domain.)   The actual release times can be significantly quicker due to the transient recovery processing, but the basic average release time for the detector is a very important number.   As a result, I re-generated the demos on the repository site and also re-compressed some examples (look at the filenames for the word compressed for additional compressed versions.)  Most of the compression examples would be fairly heavy record distribution or almost high-quality FM broadcast.  I also produced one compressed version which is closer to how I'd produce a disk/CD (RECORDING-compressed is in the name.)

Anyway -- the sound is quite a bit sweeter now.  This is one reason why I haven't released the restoration processor yet -- even though the architecture is correct, and most of the parameters are very close to correct -- there is still some tuning to be done.  The compressor/limiter just went through some vast simplifications & improvements and is coming much closer to releaseable.  There is one architectural matter on the compressor/limiter WRT the freq response balance on the multiband compressor.  I have achieved automatic freq response balance on the restoration processor, but still dont have perfection on the compressor/limiter yet.  Otherwise, the compressor/limiter is amazing from the standpoint of fitting the contour of the music without needing attack/release time tuning.  For the repo location again:
https://spaces.hightail.com/space/d4fpzoa0zD
It truly is easier to use the repo now -- I made sure that the format of my mp3s match what they want, and you might be able to directly play the examples now!!!   My previous mp3s were too high quality and they didn't accept them fully, so I am using lame -preset extreme -V2, which gives generally just under 200k/sec (which they appear to accept better.)

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-05 17:00:31
Just uploaded a new version of the examples (again -- sorry.)  It has been very difficult to finally tune out the HF distortion while avoiding the modulation-type (or pumping like) noise.   I have extended the post-transient-improvement release times (for the 2.5k, 5k and 10k freq ranges) as far as I dare.   There is a user accessible tunable setting which can back off some of that extension, but as long as there is no noise pumping, then it is a good idea to keep the extension fully enabled.

Also, there is a rather 'different' or 'odd' attack/release dB domain scheme -- where the filter* actually does a 'leveling' rather than seeking a target.  So, the gain tends to stabilize if the signal level (at each band) is relatively stable.  So, if the signal has quick level changes, then those are followed at the peak available attack/release rates (rather they be in the dB or linear domains), but if the signal stabilizes, then the effective (most of the time) release time increases (or the rate in the dB/sec sense decreases.)  There is little need to play with the gain at full speed all of the time, and letting the gain settle (in a relative sense) is only a good thing.  Right now, this effect is not very strong, but I'll be working on something that might be more helpful -- if it is helpful at all.
    * I might use the term 'filter' in some cases where it might be confusing to an Electronics Engineer -- the 'filters' are a group of C++ classes which support various attack/release schemes.   These 'filters' are pretty much plug interchangeable -- for example, there is a dB rate filter for N freq bands, a dB rate filter for 1 freq band, a linear rate filter for N freq bands and a linear rate filter for 1 freq band.   Along with those relatively standard filters, there are dynamic attack/release filters which have numerous different kinds of characters.   All of these filters support directly or indirectly an adaptation to differing sample rates.  So, this allows the various audio processors to more easily adapt without doing sample rate conversions on the input and outputs.  This 'filter' scheme supports easy experimentation in a singular framework.

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-06 03:56:23
In my numerous (literally 100's) of tests of the pseudo-DolbyA expander, I have been getting a consistently odd result -- it seems like the 9k-20k output is over enhanced by a variable (changing gain-like) amount.  After reviewing some of my math that converts the feedback to a feedforward design -- specifically for the oddly configured high two bands in the raw DolbyA design -- it appears that my math has an error.   It isn't in the 3k-9k range, but the error appears to be in the 9k-20k range.  The reason for the error is my laziness of thinking, but the complexity in the design is related to the combination of the 3k-20k and 9k-20k measurements which then control the 3k-9k and 9k-20k ranges (note the overlap in the measurement, but the separation of the actual control.)  This wouldn't be so weird, except this is also a feedback gain control, but I had to unfold that feedback into feedforward.   After careful review (and after consistent irritation in my listening), it appears that I was indeed miscalculating the gain for the 9k-20k range.   I'll be producing a new version of the psuedo-DolbyA software  soon (I actually have it working now, but requires significant testing and rebuilding on Windows platform.)   It sounds prettier and has less of a harsh HF (9k-20k) sound.  The improvement seems to be better than a decrease in fixed gain, but rather the gain control sounds smoother.
I'll post a new version ASAP, but for those with source code and willing to rebuild:  at about line 2057 in audioproc.cpp, there are two lines that look something like this:
      VFP4 pgainl3 = (pgainl[3] + pgainl[2]) * VFP4(0.562);
      VFP4 pgainr3 = (pgainr[3] + pgainr[2]) * VFP4(0.562);
these need to be changed to:
         VFP4 pgainl3 = (pgainl[3] * pgainl[2] * VFP4(0.562) + pgainl[2]) * VFP4(0.562);
         VFP4 pgainr3 = (pgainr[3] * pgainr[2] * VFP4(0.562) + pgainr[2]) * VFP4(0.562);
I have not tested carefully enough yet, but I have a good 'feeling' about it.
Again, I'll post fixed versions ASAP.

John

Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-06 05:26:20
Good news (after mildly bad news.)  I have produced a corrected version of the psuedo-DolbyA.  The very high audio frequencies sound significantly clearer.  There was always a bit of a harshness that propagated through everything after using the 4 band pseudo-DolbyA decoder.  I was always attributing it to the primitive nature of the design, but other music that was correctly decoded with commercial equipment didn't have that 'harsh' sound -- so it must have been something wrong with my design.  I think that this code that I am posting here is an improvement, and didn't require a lot of complex changes -- but only correct a calculation error for the band 3 (9k-20k) range.  The HF(9k-20k) gain is different than my original code, and it will also change differently.   This difference in change seems to have linearized some of the apparent HF intermod or distortion that kept on frustrating/irritating me.

Along with the postings of the corrected programs (and sources), I have also re-uploaded some of the music examples on the repo sites.  The only examples changed so far are the Carpenters samples.   They really do sound better.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-06 13:33:38
I have a few quick suggestions on using the psuedo-DolbyA.  I have previously only given very rough guide, but here are some real world examples:

If you want to do decode a DolbyA recording (many which are compressed and have a slightly intense high end might be DolbyA encoded):
da-win --cmd="(l,-1.5,0.0)" <infile.wav >outfile.wav
the '-1.5' above is the DolbyA threshold.  I have found the useful range (revised from before) to be between -6.0 and +3.0, where typically it is between -1.5 and 3.0.  If the music seems like it has hard sounding HF, it often means that the threshold is set too low, so you might try increasing my 0.75 or 1.5dB.  If there is odd pumping, it is usually too low, so you also might try setting it higher.  If you feel like the music is too dull, then try decreasing the threshold.
-----------------------------------------
If you have music that still sounds 'hard' sounding with too much HF, then it might have been compressed again with dolbyA or an old-style FET compressor to boost HF.  Here is a trick to get rid of that (after using the basic decoding operation above -- pass the results from above):
da-win --cmd="(l,-1.5,0.0)(m,-1.5,0.0)" <outfile.wav >outfile2.wav
This will add a layer of anti-esss sounding to the voices.  You might just need the 'l' command in the cmd switch, or you might need both the 'm' command and the 'l' command.  (l means L+R, m means M+S.)
In some cases, just the anti-ess can result in beneficial quality improvement, with the LF/MF compression will persist, but the HF will be brought better under control.
-------------------------------------
The da-win or da-avx programs can run in real time in a pipeline using sox or whatever if your computer is fast enough, and if you have a minimal number of subcommands for the expander to do.  (Each item in parens is a subcommand.)

If you want running information display as to levels (it can be interesting, and also help tune the threshold), use the --info=1005 switch.  Here is a complete decode command while using the --info=1005 switch -- this time with a zero threshold, and 1dB gain before decoding:

da-win --info=1005 --cmd="(l,0.0,1.0)" <infile.wav >outfile.wav

-------------------------------------------

Here is the general scheme for the 'cmd' switch:
--cmd="(<subcmd0>, <threshold0>, <inputgain0>)(<subcmd1>,<threshold1>,<inputgain1)..."

where subcmdn is l, m, or b .   l means L+R expansion, m means M+S expansion, and b means in-between L+R&M+S.

thresholdn is the DolbyA threshold...  Without a tone (which the program cannot use anyway, because it isn't really DolbyA),
you have to guess about the threshold (usually between -6 and +3.0) units in dB.

inputgain is the gain before that stage.  Sometimes one might need to make-up lost gain before previous stage.  Usually, when done in cascade, there was no gain between DolbyA units, so you might as well try '0.0'.

------------------------------------------------

The 'm' or M+S trick is something that appears to have been used from time to time in the old days to get better NR and/or to get special audio effects without appearing to be very compressed.
I found that the 'b' command also to be useful from time to time, however it is quite experimental (as is the entire program.)

-----------------------------------------------
Another example -- if you are using the sox program (and associated) on windows or on Linux/FreeBSD or whatever, then you can
do this if your CPU is fast enough:

sox infile.flac --type=wav - rate -v 96k | da-win --info=1005 --cmd="(l,1.0,0.0)" | play -q -

The above will take a flac music file, psuedo-DolbyA decode it, and play it on the standard output device.   The sample rate is forced to be 96k.  You can use 48k if your computer is slow, and you want to be able to try to run in realtime.  44.1k doesn't give much benefit over 48k, but might work (and the quality will probably suffer -- absolutely no testing at 44.1k other than quick verification.)

----------------------------------------------
If the input data type is not supported by da-win (must be 16bit signed or 32bit float, 44.1k, 48k, 96k or 192k), then you can specify one of the supported formats to force the mode thus (btw, 96k or secondarily 48k are preferred by da-win or da-avx):

sox infile.mp3 --type=wav --encoding=floating-point --bits=32 - rate -v 96k | da-win --cmd="(l,1.0,0.0)" | play -

In the example above, the input file type is mp3, converted into a 32 bit floating point at 96k, the decode display is disabled (no --info switch), but the default display for the play command (subset of sox) is enabled by default.
--------------------------------------------------------
So -- the above are some examples that might make the program easier to use.   I had waited to write this until I am satisfied as to the sound quality.   I am now working on finishing the cleanup for both the restoration processor and the 'super compressor/limiter.' :-).  hope to make those available within days/few weeks.

Please contact me if you have an application and wish to adapt/apply this program and want a little help.  I am more than willing to help if you want to use this program, and also willing to make reasonable changes/improvements if desired.

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-11 04:12:17
I came back to the 4 band (pseudo-DolbyA) expander, and learned another fact -- after 20+yrs working on compressors and more recently expanders.   I am offering this info to other developers (and of course, any  users) on this site.  Basically, another major improvement related to linear domain filtering of both the control signal and audio signal.   Please read further...

I kept on hearing a bit of graininess which could be partially mitigated by various tweaks like changing some of the attack/decay parameters and/or changing the delay between the AGC control signal and the audio signal.  In the last few wks, I have most of the settings set for ideal parameters, and been tweaking the high frequency (9-20k) channel, yet still getting some graininess.  The older (pre today) results sound better than the A/B comparisons that I happen to have (DolbyA encoded ABBA and various distributions), but still my intuition is telling me that something has been very wrong -- but not severe impact.   This matter has languished for a long time -- after continual improvements in adjustments -- it seems like I hit a 'graininess-minimum', but IMO, not good enough.

I had a BREAKTHROUGH again -- something that I had been thinking about off/on for years, but finally implemented it correctly:  The mixing (multiplication) of the audio and control signals produced aliasing components.  I already knew this -- but thought that I was doing almost as much as possible (adding some pre/post filtering, and controlling the attack/decay times), so the aliasing was the minimum those approaches could take -- but still some edginess was getting through.  *when multiplying two signals at say, 10kHz audio and 5-20kHz gain control and the multiplication is nonlinear (which it is), then you get all kinds of mixing products just like RF stuff -- 5kHz, 10kHz, 15kHz, 20kHz, 25kHz, 30kHz (just for the 5kHz control signal with 10kHz audio!!!)   This means that there will be ALIASING injected into the audio stream (because the sample rates cannot support all of those various by-products.)   Even filtering the aliasing after it has been generated is never 100% effective.

The mitigation (still not 100% fix) is to limit the spectrum of both the audio & control signals, but while doing the filtering -- make sure that the two signals stay synchronized.   Make sure that the spectrum of the ACTUAL control signal is limited appropriately -- must be done directly on the multiplication gain modification operation.

SO, the new version of all of my AGC/expander/pseudo-DolbyA will have a tight 20.5kHz linear phase LPF audio filter before doing the gain control, and also a 1kHz to 20kHz linear phase LPF filter for the control signal.  I have been doing various bits of testing, and I have centered on 1kHz or 3kHz LPF control signal filter for the psuedo DolbyA.   The sound is smoother, and there is much less of the low level 'grind' in some music.  Esp ABBA is much clearer (with the intense high female voices mixing together -- really tough in some cases.)  Some other examples on other kinds of music are also immediate improvements (again.)

There is significant improvement when using just a 20kHz LPF on the control signal, and I might used that on the limiter stages on the AGC/limiter processor for fast response  (and use the slower 1-3kHz filters on the AGC phases of it.)   For the expanders -- it is looking like 3kHz might be the best tradeoff -- where I hear only a very slight improvement when using the 1kHz over the 3kHz, but the attack times worry me for the 1kHz (300usec is okay in most cases -- even in some limiters, but 1msec is too long in some cases.)

I was pretty much amazed when doing a test with 20kHz LPF on the control signal -- and there was more improvement to be had with more of an LPF for the psuedo-DolbyA.   Frankly, might be able to get by with a higher frequency cutoff than 3kHz, but that is fast enough for the pseudo-Dolby (the fastest attack time needed is apprx 1.25msec, and 3kHz can provide 333usec.)

I'll be posting a new psuedo-DolbyA in a few days after much more testing.   There are numerous other little cleanups and improvements in the psuedo-DolbyA code also..   I have eliminated the need for using environment variables for specialty applications, for example.  I am also retrofitting every piece of gain control audio software that I am currently working on.

John Dyson
Title: Truly beautiful/improved version of the psuedo-DolbyA 4 band expander.
Post by: jsdyson on 2018-02-11 23:27:57
Per my previous post, I have finally run about 75% of my tests with no unexpected problems -- so I am posting the results here.  Also, I am providing a pointer to some very-difficult-to-cleanly-process music, and I intend to add some more through tonight.  Currently, the demos are just ABBA -- but they are damned hard to process (two MF/HF voices that mix very evilly together -- very vulnerable to intermodulation in hardware or software.)   All of my previous versions of ALL of my expanders and compressors had no significant true bandwidth limit on the gain control, so the gain control would have both significant (excess/unneeded) HF components and also beyond-nyquist components.   The nonlinear multiplication between the gain (some in dB plus other nonlinear math) and the signal also causes some aliasing.   By tightly restricting both the audio signal bandwidth & the gain control signal -- AND CAREFULLY KEEPING TIME-SYNC & PHASE-SYNC BETWEEN THE TWO, the results are clearer and have significantly less harsh sound.  Currently, the control signal is limited to 3kHz BW, which is approx 0.3msec, while the fastest attack time needs to be approx 1msec, so the BW is sufficent and shouldn't significantly interfere with the dynamics.   Initial tests showed that even a gain control limit to 20kHz was an improvement, and I did try 1kHz with perhaps slightly smoother results.  However, with 1kHz, it will almost definitely interfere with the required attack time.   With the control limit set to 20kHz, there is excess control signal bandwidth with NO advantage, but the disadvantage is that there will be more aliasing (probably not a lot, but enough to probably notice from time to time.)
There is no real CPU difference between my 1kHz, 3kHz or 20kHz filter implementations, so the choice is purely based on sound quality.  Also, my filters are currently 'too long', but I used the length of 1544 because I have it in my library for quick access.  I know that I should calculate the filters on-the-fly, and will probably add that capability in the near future...  Currently my full expander/compressor project has approx 1000 filter files in it's library -- would be nice to just calculate.  I intend to decrease the filters to 256 or 384 taps, which will speed up the filter by over a factor of 4 (cache+raw CPU improvement.)

I am posting the latest (hopefully final for a while unless new bugs found) version of the expander (linux src/bin, win bin, win src), and a new 'readme' that might be helpful.   I am still too lazy to write a proper man page or manual at all.

The repository with vastly improved prettier sounding music is as follows:

https://spaces.hightail.com/space/pG4t4ZFnyB
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-11 23:30:19
Whoops -- forgot the pseudo DolbyA decoder src/bins...  here they are!!!
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-12 19:24:56
I just re-processed (again) the ABBA examples as of the time of of this posting.  I was a little premature in talking about the results of the latest processing software -- but the current uploads are pretty good.   SOME of the ABBA example nditions are better than I have ever heard anywhere (e.g. UnderAttack & Dreamworld.)   I found most of the magic needed to get Dreamworld cleaned up (the version that I started with sounded like it had been run through DolbyA compression three times in sequence -- really, really bad.   These new results are pretty good in comparison.)  I didn't need to go to such extreme measures, however to clean-up Dreamworld as much as I had.    There are some other (non-ABBA) pretty good examples on the site also.  It is just that most other music isn't quite as challenging to process.
I have to be honest, however -- I made a few small improvements to the DolbyA which I doubt are very audible -- however, I did them as experiments.  If/when I make the next version of psuedo-DolbyA available, the modest improvements will be in the code.

Even though most of you'all are probably not ABBA lovers -- some of the short examples sound better than I have ever heard before.

Always -- be careful about dynamic range -- some of the songs start off very low volume and the hiss as a hint is minimized.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-13 03:33:50
Have perhaps some useful information in using the psuedo-DolbyA 4 band expander.
Firstly -- it is incredibly likely that there is significant old material out there in distribution that is either DolbyA encoded or similarly compressed.
Secondly - I have some more evidence for that likely fact, and also an example script that I am using.

Demo on the repo site -- the Carpenters songs (currently the songs with names with two digits leading) have only been decoded with the psuedo-DolbyA (no games other than a little sweetening.)   Absolutely no use of the 'restoration processor' and its adaptive features -- purely the fixed time constants.  If these songs weren't compatibly encoded, the sound would be jumping around really bad -- but the results are fairly stable/normal sounding.  (The restoration processor can 'sort of' adapt to non-formally encoded material, but the pseudo-DolbyA decoder is very picky.)

I have an archive of a lot of Carpenters music that I purchased as an anthology many years ago.  I had always thought (similar to, but not as intense as my ABBA stuff) that there was 'something wrong' with the recordings.   They were too compressed and a little harsh on the high end (please note that I have no hearing above about 12.5kHz -- even though very solid to 12kHz, so what I might believe to be a 'little harsh' might be 'terribly harsh!!).

ONE NOTE:  I removed the 'sweetening' to make my demo more valid.

My initial results are showing that using the 4 band sidechain compatible expander (what I sometimes call the psuedo-DolbyA decoder) that practically all (I believe all) of the Carpenters music in the anthology are likely DolbyA encoded.  I needed to adjust the 'threshold' to -2.0 dB using the decoder command line to get proper results.  If the setting is off by more than about 0.5 dB, there are surging issues and frequency response anomalies that aren't terrible most of the time, but can work poorly on some kinds of music.
* I added a song that is not apparently compatibly encoded:  NotDolbyAEncoded.mp3 on the repo.  It might not be 3way encoded and just one way, but still you can hear some surging.

So -- my test is something very aggressive, and if the settings are wrong as I mentioned above, the sound is far from optimal.   I am running the 'decoder' through three phases (one might not be needed), where they are L+R, between L+R/M+S and M+S.   I think that only the two L+R, M+S are correct, but using all three does more NR and does sound good.

On some example songs, music like solo pianos are highly problematical, and if the settings are wrong, then it sounds bad.  Often, the recordings are hissy, and using the decoder helps alot.   The HF sounds a bit retarded until there is true HF in the material, and it appears very cleanly (especially now.)  I don't have a good piano example ready right now, but hope to have something ready soon (and will note it on the repo site.)

Here is the script that I am using.  Note that it is a BASH command line normally used on Linux/FreeBSD or whatever, but equivalent BASH command line can be used on Windows if you have cygwin64 or other packages installed.   Below is EXACTLY what I am using to do the decoding for the Carpenters songs on my repo (as I have noted above.)  For the Carpenters ONLY, I am ONLY using three passes of the 4 band expander (psuedo-DolbyA), the sox command and the lame mp3 encoder.  I did a little EQ sweetening -- but that is how the music is usually presented anyway on CD or LP.   The mp3 encoding isn't very good -- I looked at the spectrum, and not really happy with it.  The 24bit/96kHz flac that I am actually working with are a little better.

Here is the command line that does ONLY the raw decoding, without any EQ or mp3 encoding:

 dl=2.0 bash -c 'for i in /music/carp/*/*.flac; do echo DOLBY LEVEL: $dl; sox "${i}" --type=wav --encoding=floating-point --bits=32 - rate -v 96k | /usr/local/DolbyA/bin/da-avx --info=1005 --cmd="(l,$dl,0.0)" | /usr/local/DolbyA/bin/da-avx --info=1005 --cmd="(hb,$dl,0.0)" | /usr/local/DolbyA/bin/da-avx --info=1005 --cmd="(m,$dl,0.0)" | sox - --encoding=signed-integer --bits=24 --type=wav - gain -1.5 | flac --silent --force --best -e - --output-name="carpd/$(basename "${i}" .flac).flac"; done'

I have also included an attachment that is essentially the same, but is in shell script form (haven't tested that yet -- I have been using the actual command above!!!  The attachment is easier to read than the straight command line above!!!


John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-13 11:06:50
More info -- I just ran some Carpenters tests again.  (results on the repo site, filenames end in "-2" for two passes.)   Instead of using three passes of pseudo-DolbyA, I more realistically backed off to two passes (instead of L+R, 1/2(L+R,M+S), M+S -- I am using just the more conventional L+R, M+S).
The sound is more alive when using just two passes (simply remove the 'hb' expansion da-avx command from the script), and set the threshold to approx -3.25dB instead of +2.0dB.
So, you'll notice that the examples with '-2' might sound a little better.   It is probably more realistic to just use LR & MS simply because of the slightly more detailed set-up required to do the 1/2 way matrix in real hardware.   This is probably what they used to actually encode the music (that is, the two passes instead of three.)   The 'three pass' might be best used to decode already encoded material when there is a lot of hiss that needs to be removed, but the 'two pass' does seem more alive.  It might be that the earlier material WAS encoded with the full three passes out of desperation to get rid of tape hiss -- because I did find that the later Carpenters' stuff (late 1970's and early 1980's) did not decode well with the full three passes either.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-13 14:08:48
Sorry to keep pestering -- but info is coming fast and furious:

firstly, it appears that the ideal processing (after many experiments) is 'l+r, HF(l+r), HF(m+s)', or using the 'cmd' args, it would be: l,hl,hm.  The each of the three runs of da-avx (or da-win) would have --cmd args like:
--cmd="(l,3.0,0.0)"
--cmd="(hl,0.0,0.0)"
--cmd="(hm,0.0,0.0)"

Also, after doing the runs of the various songs with the above cmd args (instead of the ones specified above or in the shell script that I had appended above), I added another processing phase -- a linear one, absolutely no use of the 'restoration processor', but reordered some of the time after looking at some information about the self-correlation (that is patterns) in the music.
So, if you listen to the files with the -exp in the filename and compare with the others, it might be interesting.   Also, I am removing the files with -2 in the filenames, and when the files with -3 appear, those will be the music with the above commands (that is, again, full band l+r, HF l+r and HF m+s.)  The effects of the DolbyA compression just with the HF is to slightly sharpen the HF of the music.

Probably about 10 minutes after this post, the new examples with the -3 in the filenames will appear.  The -exp versions are already there.   I am also considering removal of those processed with the first version -- the new ones are vastly improved and sound much more musical.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-14 00:25:31
New version of the decoder.  Additional filtering on the 80Hz and 80-3k range.  I realized that I can filter the two low bands much more to remove some more intermod products (and mitigate even more aliasing) without impacting the attack/decay times.  Demos much more complete on the repo.   -exp files have a new experimental time domain decode.  All examples except for the Sergio Mendes examples just use the 4 band (pseudo-DolbyA) decoder plus some SOX tricks...   I added the restoration processor to Sergio Medes to resurrect some of the percussion.   I am basically demoing the simple 4 band decoder to show how much a relatively simple piece of software can undo a lot of encoding.   I do have one example on the repo which was obviously not DolbyA encoded in the same way as the others -- you can hear the surging that decoding an unencoded recording can create.   (Actually, the 'non-encoded' copy was 'less' encoded than the others...)  If you decode without 'surging', 'loss of HF components' or other artifacts -- it is quite possible that you are decoding a recording which is either DolbyA or side-chain compressed.   Undoing side-chain compression is just as helpful as decoding old DolbyA recordings.  (A full expander instead of a side-chain expander can easily do TOO MUCH expansion.   The 4 band expander is designed to undo sidechain compressed material.)

BTW, I fixed the default Makefile on windows so that if you have cygwin64 installed, the program should be easier to build.  (Before, the correct Makefile was an obscure file in the source directory -- sorry about that previous mistake.)
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-15 12:48:01
I have been on a distraction -- been playing with reorganizing some old stereo recordings (mostly the same that I had been working on before.)   I re-oriented the Sergio Mendes examples a bit, and have produced much more interesting stereo images (stuff is still real -- no weird inversions that I can hear -- clearer locality/etc.)  The reoriented examples are on the repository:
01.Mais Que Nada,
03.With A little Help From My Friends
05.Look Of Love
09.Going Out Of My Head
12.Daytripper
13.Pretty World
15.Pais Tropical

I'll publish the technique when I totally get it tuned up.   Pais Tropical is special -- it has some kind of 'on-site' miking -- so it has a harsh character.   The others are really fantastic music, and I wouldn't have wasted my time if it wasn't such great stuff.
I have equivalent results with ABBA (actually even more of an improvement), * but not ready to disclose.

* CORRECTION:  I was in a hurry last night and uploaded my ABBA experiments onto the repository without thinking.  Take a listen - they are interesting.  Not 100% tuned yet -- but very promising.

John
Title: New expander release, example scripts, example results
Post by: jsdyson on 2018-02-19 11:36:50
Semi-great news!!!
Here there is a new post of the latest 4band psuedo-DolbyA expander.   I have added appropriate low pass filters and retuned some of them because they were overly attenuating the LF around 20-30Hz.   Now, there shouldn't be any more than about 2dB of total static attenuation at 20Hz (of course, the dynamic gain loss will be more due to the operation of the expander.)  It is technically necessary to at least do some DC blocking, or input offsets can cause trouble and there will definitely be more disturbances in the output.

I have cleaned up the source code so that a lot of cruft has been removed.  It will be easier to see how simple the code is from a high level viewpoint.  (There is a lot of low level stuff going on, however due to the use of C++ classes and putting a lot of the work into the supporting code.)

The source distribution was previously often not complete.  I forgot about the vector math library, and re-added that back into the source.  Hopefully, it will build now.

The Windows source distribution and the singular Linux distribution  now has some example scripts that I have actually used to decode some anthology album releases -- which were almost definitely DolbyA encoded.   It is important to follow what is going on in the scripts, or you will not get ideal results.  I carefully set the thresholds in the scripts - even though they are not perfect.  I had to set the thresholds by listening for decoding artifacts (I am well trained to hear them, but still there are some interactions that I sometimes miss.)  Specifically, the thesholds on the carpenters might not be 100% correct.

I have added a simple example of actual decoding before/after results with the psuedo-DolbyA only.  The results show that the psuedo-DolbyA really works (note the significant decrease in hiss, even some inter-syllable NR.)  Note that the result has a little less treble, which is to be expected (similar to the action of DolbyB.)  Even if you retune the treble back up (adding 3dB here or there at the HF range), there is less hiss than the original.

The resulting NR isn't quite as substantial as the 'restoration processor', but still worthwhile, and the results produced by this decoder are technically more accurate as to the original source material.

SORRY ABOUT THE PREVIOUS INCOMPLETE SOURCE RELEASES!!!
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-19 16:26:18
Regarding the threshold numbers for the Carpenters on the recent release of the expander...
These numbers are variables used in the shell scripts that I sent along with the Windows source release and the singular Linux distribution.

For Carpenters before 1981 (maybe not even accurate for the later part of 1970s, and definitely not for any Christmas album), the
level parameters appear to best be set to:  dt0=0.50 dt1=-3.5 dt2=-3.5 .   The dt2 isn't quite as critical because it applies to only the top two bands for HF enhancement removal.
For Carpenters on/after 1981, the parameters are quite a bit lower:  dt0=-6.0 dt1=-9.0 dt2=0.0.   The last value of dt2 is questionable, but does imply a lot of HF emphasis had been applied.  The much lower parameters imply recording at significantly higher level above DolbyA reference level.  The 1981 album appears to have either significant tape saturation or they are using a compression/limiter with poorly designed HF attack/decay behavior (which might be another attempt at dessing, after too much enhancement with 2HF channels of DolbyA.)   I forget what extreme tape saturation at HF @ 15ips sounds like -- but it just might be overload at HF because of recording hot to begin with, then applying an approx additional 6dB of dynamic gain between 3K and 20K with DolbyA.  (The sses in Karen's voice are really corrupted on especially the 1981 album.)

The source material are the so-called studio anthology album -- which apparently is just a runoff of tapes sent to disk/CD mastering without careful handling of decoding needs.   They were obviously not just using a single DolbyA, but using them in a configuration that enhance the capabilities (or using similar FET sidechain compressor for compression.)  My shell scripts imply the configuration.

John
Title: Early, early release of the 'restoration processor'
Post by: jsdyson on 2018-02-21 00:40:16
Here, I am distributing (giving you'all) a copy of the so-called 'restoration processor.'  It is a rather nice (IMO) expander that is good at reasonably good NR and transient recovery.  It is NOT a tick/pop remover, and in fact will likely make them much worse.

I can only distribute an Intel/AMD 64 Linux version right now.  Hope to distribute a Windows64 version before the end of this week, and likely will have it working/distributed tomorrow or the day after.   RENAME THE FILE TO ex-avx from ex-avx.out (or use that as the command name.) I promise that I am really gonna work on completing the Win64 version -- I did have earlier version working -- but it needs updating.   I wanted to give away something NOW/ASAP because I have been promising/talking about it for a long time.  The biggest last issue was the best default parameters, and to resolve the commands into something simple for a command line (it used to take a fairly big script to set-up.)   The usage is similar to the psuedo-dolbyA, except it only works at 44.1,48 and 96 (no 192.)  It is complex, and quite multi-threaded.  It really likes 4 core machines, and needs an i7-3000 series or newer.   I'll be releasing versions that don't require nearly the latest CPUs soon (about the time that the windows version is done -- which will be an ATOM/silvermont optimized ver-sion that will work on almost any i3/i5/i7 or recent ATOM.)

The readme talks more about the program, but generally it is used in a command line (pipeline or stdin/stdout.)  The command switches are simple:
--info                   provides a nice 'pacifier' that shows the amount of expansion on each band, in/out levels, and limiter status.
--exratio=0.xxx    expansion ratio, between 0.00 and 0.99.   Best/most useful values are 0.10 to 0.50.
--exthresh=-x      dB for threshold.  Usually between -20 and -3, with -6 being a good first guess.
--exmingain=-x    dB for minimum gain.  Values between -30 and -3 are good choices, I often use -12
--exmaxgain=x     dB for maximum gain.  Values between 1 and 6 are good choices.  I often use 4
* PLEASE NOTE THAT THESE COMMAND LINE PARAMS MIGHT NOT BE RANGE CHECKED -- insane commands might give insane results (and sore ears.)
* Acutually, there is a rather nice built-in limiter that keeps things from going too nuts.

Example usage:
ex-avx <infile.wav >outfile.wav
Uses default exratio of about 0.21, default exthresh of -6.0dB, exmingain of -15dB, exmaxgain of 6.0dB

Another usage:
ex-avx --exratio=0.50 --exmingain=-20 --exmaxgain=3 --info
Do more aggressive expansion with a lower minimum gain (more noise reduction) and limit the maximum gain
avoiding overload.  Also, provide a running log of what kind of gain control is being done.

I am also including some examples (directly using the exact program being distributed, with default parameters and for the 'lots' version with an expansion parameter of 0.60 (which is A LOT of expansion.)   Note the relative lack of artifacts even with huge amounts of expansion!!!

**** WITH zero dB expansion (default), there is a several dB loss through the processor.  I added in that loss to avoid clipping or exciting the limiter.   I normally use 32bit floating point, so the loss of precision isn't really all that important.  Using 16bits should really be reserved for quick checks, and 32bit floating point is MUCH BETTER considering the fact that this is an expander and I HAVEN"T ADDED DITHERING!!!

Read the README,txt for a little info about this nice toy.  The Linux binary is ex-avx.out (rename it to ex-avx to be compatible with the examples.)  The example is a 30sec clip from an old song from the 1960's called 'Downtown', source was youtube.  The -noproc version is the raw copy.   The -defaultproc version is using the 'restoration processor' without any args.  The -lotsproc is using the 'restoration processor' with much more expansion.

Again -- I promise a windows version soon (hours or days -- not weeks.)   Source code will probably be 2wks away because of maintenance/source management issues.  The source code is very big -- 120MB (yes, really), some of it is automatically generated, and it is JUST NOT READY for other people to look at it (it is A LOT better than it used to be.)

PLEASE HAVE FUN
John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-21 13:16:18
Just got the first build of this version of the FULL SCALE processor working on Windows64.   Bad news is that it runs way too slow on the Silvermont architecture (the ATOM), so I have some optimization work to do.   Will probably take a few more days -- but it is definitely in progress.   Currently, it does run nicely on the Haswell platform on Linux.

The FULL SCALE/capability 'restoration processor' works really nice with the psuedo-Dolby.  So, when doing a DolbyA decode, then doing some additional expansion/transient recovery really does help some old music (e.g. Brasil'66 type stuff.)   The restoration processor is sometimes used like this:

sox infile --> pseudo-DolbyA -> restoration processor -> outfile

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: rlhess on 2018-02-21 21:40:15
This is great news! From my perspective, the restoration processor is icing on the cake. The pseudo-DolbyA is the key. We have some other options for restoration processing (or mastering), such as iZotope Ozone (or iZotope RX Audio Editor), but the Dolby decode is a challenge. I personally like the U-He Satin on some of my tapes, it gives them more life than the hardware decode without being overly bright and do pumping.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-21 23:37:53
Thanks for the feedback!!!  I am very focused working on the Windows version -- which is now actually functioning.

When the 'restoration processor' is fully available, I REALLY HOPE that will help people with their older (and some new) recordings.   When I use the 'restoration processor' (STILL NEED A NAME -- I suck at naming things -- it might just as well be 'processor 1B' or somesuch :-)), I often use it with the pseudo-DolbyA first in a pipeline.  I am currently optimizing the code to work a little more quickly on non-AVX CPUS -- my Windows box is a ATOM 2core silvermont, so is quite slow when compared with my 4core Haswell.   The non-AVX and smaller SSE registers on the ATOM make the code longer and less efficient.  I am focused working on streamlining the code to work much better on slower computers.   I will also probably blindly build an AVX version for Windows (that will need at least a i3/i5/i7 of 3000 series or better, or recent AMD), and offer both the blindly built AVX version and a tested SSE version on Windows.  For Linux, I'll offer both a verified SSE & AVX version...

A hint to doing DolbyA decoding is that sometimes it SEEMS like they did a DolbyA encode on the normal stereo signal, but additionally on the M+S variant.  I have sometimes been successful doing a L+R decode first, and then an M+S with an equal threshold, 1 dB down threshold or 3dB down thresholds.   For the M+S (the second decode), I use the third arg of the parenthesis cmd argument to specify a gain of 0.5dB -- the default loss of the first decoder (again the first is usually L+R.)   The noise reduction and transient behavior is sometimes better when two phases of DolbyA are used (esp when not using the 'restoration processor').  So, essentially, the full pseudo-DolbyA command pipeline (for both L+R & M+S decode) looks something like this:

sox infile.flac --type=wav - rate=96k | da-avx --cmd="(l,-1.0,0.00)" | da-avx --cmd="(m,-4.0,0.50)"| sox - outfile.flac

(which means, L+R decode, threshold -1dB, no input gain and M+S decode, threshold -4.0dB, 0.50dB input gain).

Also, it seems like some vocals might have been processed by some kind of sidechain HF enhancement, where the pseudo-DolbyA can help undo that also, but instead of using the 'l' command, but instead use the 'hl' command which only enables the top two bands of the decoder in the standard L+R mode.

For example (using the pseudo-DolbyA & restoration processor both), the Brasil'66 stuff really needs expanding (really badly -- it is INCREDIBLY beautiful when properly decoded), and the DolbyA decode helps a lot -- but the results are still missing the intensity of the percussion.   The 'restoration processor' rebuilds a significant amount of percussion.  IN SOME WAYS, even though the 'restoration processor' is good at NR, it is the 'transient recovery' that one really benefit s from much of the time.

The 'restoration' software internals are a bit of a tricky beast, because of trying to find a good balance between the release times and the 'modulation noise/surging' issues.   I think that there will eventually be at least three (3) levels of dynamics management along with the adjustable expansion ratio.  (IN the large scale, the expansion ratio is dB linear, but in the short term it is simple linear -- like a FET compressor approximately is -- or DolbyA is approximately simple linear expansion - not really dBlinear.)*
  * When I meantion dBlinear or simple linear, I am speaking more of the domain of the release/attack time characteristics, and
  less the actual compression ratio type dBlinear.   A 'linear' release time has a kind of exponential characteristic, where it slows
  quickly as it approaches the target, but is much faster if it is far away from the target.   dB linear moves at a pure dB rate until
  it hits the goal exactly.  So, the dB linear scheme REALLY hits the target, but the 'linear' scheme keeps approaching the target
  closer and closer.   That shape of the dB oriented attack/release appears to match what the ears like to hear instead of the
  more common simple linear scheme as used on most direct FET compressor/expanders.
  * An example of the 'linear' sound for a compressor is that as the compression amount increases (the depth of compression
     increases), then the effective release speed also increases -- because the decrease in gain has further to go, so the gain change
     is quicker in dB/sec if the amount of gain release is large.   With dB linear release, no matter how many dB of compression are
     being done, the release speed is the same in dB/sec, and the ear hears it (the density) as pretty  much the same as lighter
    compression (other than the  larger eventual gain change.)   A linear compressor is better to use when you want to increase
    the 'compressed sound'  or 'density' by increasing the depth of compression.   A dB linear compressor is better when
    you want the sound density to stay similar whether or not you are using 6dB of compression or 12dB of compression.

So, usually, when being used in the default expansion ratio of approx 0.20 (it is approx 1:1.20), there is mostly very little need to change the dynamics characteristics on most music material.   However, at much higher expansion ratios (0.30 -> 0.70), then the dynamics become super important.  Right now, the dynamics are kind of middle of the road -- but I am going to try to put together a slower and faster mode.   The dynamics tuning is not just a dB/sec release rate type thing, but is the idea that the first part of the attack/release are fast-linear up to several dB.   After that, then it defaults to a dB linear scheme.   It is the mix of 'how much' straight linear and 'how much' dB linear that is a big part of the release scheme.   There appears to be a 'fixed' set of dB/second rates that ears like to hear -- so that rate is current fixed to a set range of values (it is also dynamically calculated.)

Also, the attack scheme is important but not seeming to need as much adjustment -- but I think that I have found a good tradeoff where transient recovery is very effective, but it is not overly sensitive to transients (causing a gain-hang effect or surging.)   Also, I am using an interesting attack scheme which really minimizes the intermodulation on the attack edge.  I haven't exhaustively tuned it yet, but hope that I can actually  measure the intermodulation effects at higher frequencies.  (I do have a scheme/method to measure the effects of attack time on LF ranges -- and it can be very obvious once the intermodulation effects are recovered from the total signal (and one can actually hear the grinding/almost static-like noise that is buried in the signal -- otherwise hidden because of masking.))

John
Title: New version of pseudo-DolbyA -- fixed some thresholds that were a little off
Post by: jsdyson on 2018-02-23 05:50:12
I think that the 3k-9k and the 9k-20k thresholds were a little off.   I changed them -- and made them a little more precise in the code also...  The HF sound in intense situations (e.g. ABBA type stuff, my 'acid test' for severe HF) is significantly cleaner and the HF sound had some irritating problems until now.  Not usually noticeable, but the error might have been measurable.

New versions of the Linux, Windows and source.

Sorry for the error.
Title: Strong suggestion to try to use the latest psuedo-DolbyA decoder - 23Feb or aft
Post by: jsdyson on 2018-02-23 15:26:28
I just did a comparison between the previous 19Feb version of the psuedo-Dolby and the 23Feb version (that is, the most recent.)  When checking on my most stringent tests USING REAL DOLBYA SOURCE, it becomes very clear that the various attack/decays of the two highest frequency ranges match MUCH better on the 23Feb version.  So, whether or not the newer version is perfect (I make no such guarantee -- cannot test against a REAL, guaranteed DolbyA -- but there is someone who might be volunteering to help), the new version has a MUCH MUCH smoother sound quality to it.   That is, it has a smoother sound even though there is less HF attenuation.

Because of the way that DolbyA did the two high frequency ranges, their attack/decays can interact, and it appears to be important to get the match between the two fairly close.  (That match is more the design of the DolbyA attack/release, not so much the external device.)  The best way to describe the improvement between the two recent pseudo-DolbyA versions is that it SEEMS like the 3-9k and the 9-20k attack/decays match more 'hand in glove', so that there aren't as many undesired undulations between the two gain ranges.

So, I am trying to make clear -- the new version of the psuedo-DolbyA does sound much better.  If you are using/playing with the pseudo-DolbyA at all -- please try to use the newest version.

This does prove one thing -- the design requirements of a DolbyA decoder has lots of little pitfalls.   It isn't super-difficult, but it is quite tedious to try to match the behavior of the old HW design.

John Dyson
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-02-23 22:10:45
With some help from another individual (don't have permission yet to divulge the name), I/we found a bug in my specification of the threshold.   I had previously been suggesting a threshold of 0dB (the middle arg to --cmd).   The better value would be 10.0 .   I am in the midst of a re-review of the software based upon these corrected threshold values (which do appear to be producing better NR, and less need to use repeated decoding!!!)

It is so good to get help when needed.  I do learn, and happy to fix mistakes when they happen.   A new version with a few minor corrections will be released within a day.

The good news is that ALL previously useful results are still valid.  The new results will be better yet!!!

John
Title: The PSEUDO-DolbyA is now working in commercial setting...
Post by: jsdyson on 2018-04-20 02:25:09
The pseudo-DolbyA has been getting LOTS of work since I last posted any status.  I got help from a very pleasant and encouraging  recording professional with real DolbyA units (many of them, in fact), and have been able to both correct the gain curves and to make the pseudo-DolbyA really work well.  In some cases, been getting feedback by pro-golden-ears that the pseudo-DolbyA is working better than a real one.  (I still need to get official agreement from him to allow me to give him official credit/use his name in the credit for his assistance.  He has been INCREDIBLY helpful with the know-how and information.)

Since I have dangled the carrot, and shown many defective previous versions (mostly threshold problems and gain curves that were slightly off -- also the attack/decay was slightly imperfect), now it works truly perfectly.  I am offering access to a binary version of the decoder for Windows10 on Silvermont X86 or better platforms.  The version that I am offering is slower than the full version that takes advantage of more advanced platforms, but it works the same.   The code has stabilized (every 'improvement' has essentially been destructive -- this thing is on the bloody edge of perfection.)

The resulting performance is very good -- it has less distortion (both intermod and nonlinear distortions) than a real unit -- probably below the theoretically possible with normal FET gain control hardware.

Post here or send a private message with an email address that can accept a 2.5MB file (it will actually be smaller), and I intend to send copies within days (the windows software is working right now, just need the time for turnaround.)  I'll essentially forget the email address upon sending the response other than to check to make sure that it is working for you.  There will be NO commercial solicitation, but I require that there be no further distribution beyond the user/company who receives each copy.  I fully expect feedback as to problems/bugs that are seen.  If no bugs, no feedback is necessary.   This offer is good until about the first of June 2018.

This is truly a fairly simple piece of software but the design details have been very subtle.

If you are just interested in the theory of operation, the previous source versions are fairly accurate -- but with several serious unintended flaws.  Perhaps the worst, which makes the sound quite incorrect for true DolbyA transparency is that the bandpass filters are not correct -- even though it seems like they should be.  There is a conversion problem between the active filter bands used in a real DolbyA unit and the digital filters (using bilinear transform) that I had originally designed.   Also, the attack decay structure was generally correct in the previous source versions -- but there are things missing that really improve the intermodulation sidebands (even visible on a spectogram.)  All of my filters also previously used an incomplete structure that caused mostly phase errors (but also some amplitude errors) when the filter frequency was close to Nyquist.  All of these bugs and others (including a much better gain vs. level fit for each band) have been corrected in the up-to-date source.   Also, the resulting binary code in the demo/test version that I am making available has these bugs fixed.

So -- this is meant as a peace offering and an apology for not distributing full source at this time.   Soon, I will find the correct terms for sale/distribution, and will make that information available to professional recording engineers.  Frankly, it JUST MIGHT still be free, but I am not sure!!!

Examples using the psuedo-DolbyA for decoding reside on: https://spaces.hightail.com/space/nXCmV47em2

Specifically the Petula Clark examples are the closest in the repository to a pure decode.  The ABBA show the great clarity because of the decoder design.


Sincerely,
John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: mycroft on 2018-04-20 13:46:30
So all this was to fetch money?

Well played, BRAVO!
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-04-20 14:24:57
My original effort was NOT to fetch money.  What happened is that the guy who REALLY HELPED with some of the really difficult details -- that is, to make it work EXACTLY like a real DolbyA (I mean, within 0.50dB or better) resulted from details that he provided. Also, some of the most important review work which forced me to reject the old 4 band filter scheme and develop a very different set of filters came from him.

It is his opinion that money can be made -- even though my own immediate (or far term) interest is not to make money, his involvement as a professional recording engineer who has really helped gives him some interest in the project.  So, he and I MIGHT decide that the project best be distributed for free (with simple attribution) or perhaps for a fair price.  He must participate for this to be an ethical project.

The DolbyA decoder really works, and most likely will be used commercially as often (or more than) as for hobby listening purposes.  So, for someone to make money using it (rather than burning up their old DolbyA units -- well, not really...), this gives them a truly superior backup for the old DolbyA units.  It is probably best that there be some financial consideration for true professional/money making use.
No matter what, I hope that the software will be available for free for noncommercial use -- frankly, a HUGE amount of material is definitely DolbyA encoded (e.g. the Carpenters album from HDtracks.)   Another very powerful example is some material that I have for Simon & Garfunkel -- almost unlistenable without decoding.   The really bad thing about the Simon & Garfunkel stuff is that there is true bass and acoustic recording which causes really bad intermod effects (from the original DolbyA encoding) which can only be removed by an accurate DolbyA decoder.   My DolbyA decoder does a credible job of removing those artifacts.
So, my attempt at putting off the decision, while also giving access for non-commercial use (really improving a lot of old, DolbyA encoded recordings) is to offer to provide a slightly crippled (but full quality) copy of the decoder for a month or so for free.  This version will also work as long as it maintains compatibility with the underlying OS and CPU technology.

When thinking about it -- if this is as good as the professional feedback that I have gotten has implied -- consideration needs to be given (from a 'being right' standpoint) that some contribution be given for the development and hard work for the final tuning.  There are other projects likely forthcoming, including a good SR decoder.   The SR decoder is at least 4-5X more trouble than the DolbyA due to the numerous kinds of sliding freq response and various TCs.  The final tuning/testing effort (after the raw development based upon all of the various materials available in public) will not be trivial.   The DolbyA was non-trivial enough trying to get the curves properly fitted (requiring 4th-5th degree equations to be accurate enough), and the very careful balance between mitigation of unnecessary intermod vs. accurate following of the DolbyA dynamics.

As a simple exercise to prove that it can be done -- it was a relatively simple effort (reflected by the available source code.)  The final version -- whose source code doesn't really look all that different, but acts differently -- was NOT a simple effort and required the involvement of a real recording engineer and at least one other 'golden ears.'

I did ask for more free help online (feedback), and there was definite interest, but I was stumped without more substantial help.  Finally, someone came forward and REALLY HELPED and participated.

However, the offer stands that I will HAPPILY provide a version that works well on most recent Intel CPUS 64bit mode, and can provide an equivalent Linux 64 bit mode version if desired!!!

John
Title: Re: The PSEUDO-DolbyA is now working in commercial setting...
Post by: krabapple on 2018-04-20 15:37:02

Examples using the psuedo-DolbyA for decoding reside on: https://spaces.hightail.com/space/nXCmV47em2



Specifically the Petula Clark examples are the closest in the repository to a pure decode.  The ABBA show the great clarity because of the decoder design.


are there 'before' and 'after' samples , or am I missing something?
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-04-20 17:35:01

Examples using the psuedo-DolbyA for decoding reside on: https://spaces.hightail.com/space/nXCmV47em2



Specifically the Petula Clark examples are the closest in the repository to a pure decode.  The ABBA show the great clarity because of the decoder design.


are there 'before' and 'after' samples , or am I missing something?


I didn't provide before samples...  They are quite interesting (esp ABBA.)  I'll provide a few of them later on today -- and will follow up with a post when ready. (Again, later on today -- 20Apr, prob before 20:00 EST USA time.)

John
Title: The pseudo DolbyA expander comparisons are ready!!!
Post by: jsdyson on 2018-04-20 20:47:12
I have placed several songs that I got (probably mistakenly) in DolbyA encoded form onto my repository.   There are two or three forms of the material on the repo:
-raw.mp3   --- an mp3 compressed version of the original DolbyA encoded material.
-DAdecodeOnly.mp3 -- an mp3 compressed version of the pseudo-DolbyA decoded material.
-full.mp3 -- fully finalized version of the material above.

The biggest difference that you might notice is that the decoded versions have less hiss and less ambience (as one would expect from an expanded version.)
On selected material, the 'full' finalizer processing was also done to demonstrate what the 'decodedonly' version might sound like on a real disk/CD or whatever.

If this material was not DolbyA encoded, there would be severe loss of HF, surging (inconsistent levels -- esp for transients) and general ugly sound.

My guess is that at least 1/2 of the material that I have recently purchased/acquired (and even some YouTube stuff) can be improved by doing a DolbyA decode.  One example that needs DolbyA decoding is the Carpenters album being sold on HDTracks, but the 'uncompressed' version of a paul mccartney album is NOT DolbyA encoded and sounds bad after trying to do a DolbyA decode.

The 'full' processing uses some very high quality expansion, compression and EQ -- and is only meant to show what the music CAN sound like.  I didn't do the full processing to every example because it can take a few hours to set-up the proper parameters for the full processing, and I don't think that is the primary reason for this discussion anyway.

I am using a new repository, and the location follows:

https://spaces.hightail.com/space/ko2yTjF5YY

Let me know what you think!!!
John
Title: More examples of DolbyA decodes
Post by: jsdyson on 2018-04-21 00:09:14
I have provided a few more demos in a different site -- these are pure pseudo-DolbyA decodes.  Some are more obvious than others, but all are improved to an extent.

Trying to show that this DolbyA problem is existent from MANY sources, not just a few select places.  Just using a 'tone control' wont help -- there'll always be the distortions in dynamic range without decoding properly.

I have also found that the threshold value is not extremely critical except in a few cases...  ABBAs "Under Attack" is VERY critical, and needs to be set within approx 0.25dB.   Most are happy with -13.50 through -15.25dB, where -14.0 works well most of the time.

https://spaces.hightail.com/space/z3H68lAgmJ

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-04-21 01:32:03
PS:  I just added a recent popular tune -- but really cranked up the expander (after the psuedo-DolbyA.)  It reveals some of the sound effects...  'taylorfun'.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: krabapple on 2018-04-22 22:19:42
So , your presumption is that  as much as half of what your are purchasing is Dolby A encoded material  that was NOT decoded during mastering... and that's why you've developed a software decoder?

Your best demonstration would be to record something using Dolby A NR, then decode it with both a Dolby A decoder and your own software, and compare the results.


Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-04-22 22:33:38
Regarding my disgust as to the quality of some of the available recordings, and finding that they were like left with DolbyA encoding intact -- yes.  That is 99% of the reason why I decided to write my own decoder.  I like challenges.  Plan to start working on SR (much more of a challenge) soon.

Regarding proof of operation/accuracy:  Yes -- an audio professional along with a golden-ears expert (for proof) has worked with me on perfecting the DolbyA decoder to be very compatible (but not bug compatible.)  So, SOME of the results of my SW decoder are better (cleaner/smoother sound -- minimal intermod -- no harmonic distortion from FET or diode gain control like the newer DolbyAs (FET) or the original (diode).)  The SW decoder has NONE of the typical defects of a SW compressor/expander regarding a harsh kind of sound.

I have been working on this for months, barely got it working credibly (kind of) about 2mos ago.  Then, a recording expert started participating (he wanted backup for his DolbyA units and probably computer environment software.)  He and I worked on making the gain curves nearly perfect (generally less than 0.5dB static error on all 4 channels, including the screwy 9k-20k channel.)
The SW doesn't use a simple attack/release scheme like a DolbyA schematic (and various patents) might imply, but uses something that strongly mitigates most mathematically unnecessary intermod, and produces a generally smoother/more accurate sound than either another commonly avaialble DolbyA SW decoder and real DolbyA units.  However, the audible attack/decay fits hand in glove with the real DolbyA.  (interestingly, even the distortion from the true LF on Simon&Garfunkel Scarborough Fair moduating with the gain control is removed because of the good matching for the decode operation.)

So, this has been acid tested with real (proven) DolbyA material and compared with real DolbyA units.  Also, very often, if material is NOT DolbyA encoded, but DolbyA is used to decode -- the sound becomes quite defective (including expander effects, terribly decreased HF, etc.)

So, if you listen to the examples on the repository, and find that the sound is reasonably good - then the original material was either DolbyA encoding somewhere in the processing chain or there was a similar sidechain compressor being used.

Please enjoy -- and the SW is available if you want.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: punkrockdude on 2018-04-23 00:35:11
I haven't used it myself but I contacted a person that uses Dolby A and I Think he has contacted you. I really appreciate what you are doing even though I have not used your software since I do not have any Dolby A encoded material. I do look forward to try out your future compressor you talked about earlier. I also love that you software is available to Linux and hopefully your compressor can be a LV2 or/and VST plugin.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-04-23 02:34:36
I haven't used it myself but I contacted a person that uses Dolby A and I Think he has contacted you. I really appreciate what you are doing even though I have not used your software since I do not have any Dolby A encoded material. I do look forward to try out your future compressor you talked about earlier. I also love that you software is available to Linux and hopefully your compressor can be a LV2 or/and VST plugin.

FIRSTLY:  if there is more interest, I can make the compressor available soon -- just needs to be updated with better code from DolbyA and expander.  If you want to hear the results of a relatively mild run of the compressor, listen to SOS-DAfull-finalized.mp3 on the repository: https://spaces.hightail.com/space/z3H68lAgmJ
0
The file SOS-DAfull-finalized.mp3 has been DolbyA decoded, expanded (to remove some of the old compression artifacts), and re-compressed/limited with the modern compressor at a fairly controlled level.  If you are really, really interested, I can make a limited version of the compressor/limiter available for Windows64 in about a week -- but it would be command line oriented (lots of options/tuning to get that special sound), and as a side-bonus the expander is built-in also.  The bad news is that heavy baggage runs a little slowly, and the compressor & expander cannot run realtime.  However, on a relatively fast machine, the compressor running alone usually has NO trouble running in realtime at 96k.  This MIGHT change as I move some of the filtering from the expander, so AS YOU CAN SEE, the code is still in flux.

REGARDING THE DOLBYA ENCODED STUFF:
One thing -- I truly do not know what you know or even about what you know, so PLEASE forgive me if this might seem rude...  Before 1yr ago, I would have also said that I had no DolbyA encoded material, yet a large amount of my music collection WERE DEFINITELY DolbyA encoded...  For example, I had my ABBA collection for over 10yrs (really, really, long time), yet knowing what I know now, and listening to it, I kind of 'whack' the side of my own head now realizing that my ABBA collection IS DolbyA encoded.  Here is an example of my UNINTENTIONAL DolbyA material collection (likely well over 1/2 of everything I own including):  Petula Clark collection of 4 CDs, massive Carpenters collection, a copy of ABBA Gold, 3 Simon&Garfunkel albums, 2 Linda Ronstadt albums, Fleetwood Mac Rumors, Christopher Cross Album, massive Olivia Newton John collection, Herb Alpert Album, Herb Alpert & Brasil66 Album, Burt Bacharach set of two albums, Dionne Warwick Album, Chicago CD, Queen CD, Carly Simon CD, 2 Anne Murray albums, Bangles CD, Bananarama CD, Moody Blues CD, The Cars, Suzanne Vega, Paul McCartney & wings from HDtracks, Carpenters from HDtracks...

My point in listing these -- you (or someone) just might (more likely than not) REALLY have a DolbyA encoded album (not vinyl, but CD or electronic.)  This has been a very common thing, because I have almost had to check my sanity when I realized this.  I have gone back directly to the CD source and lo&behold a CD that I might have purchased over 5yrs ago or something I downloaded deep in the past has been left DolbyA encoded.

I did NOT have to carefully select the material that I put on my repository, because I have found that more likely than not that a given piece of material that I have TRULY, REALLY benefits from DolbyA decoding.  I honestly cannot claim that every item that seems to be DolbyA encoded really is -- but I have some traceably DolbyA encoded material and it really doesn't sound much different in 'compressed' sound and HF emphasis from the material that I have given as examples.

The only reason why the material can be left DolbyA encoded is that the result isn't really all that fatal (just a bit too much HF empahsis, too much of a compressed sound, and more hiss than there should be on older recordings from older tape formulations and tape machines..)

Some of my collection initially sound quite nasty, but then turns into audiophlie grade upon decoding -- REALLY!!!

If you don't have anything that is DolbyA encoded, I understand...  But if you have a large electronic or CD sourced collection (or a moderate sized one like mine), then there is at least 1/2 of the selections that benefit from DolbyA decoding.  I'd estimate that at least 50% -75% of my collection is almost provably DolbyA encoded, and perhaps 10% is questionable either way (might be sidechain compressed because of excess HF loss but no expansion dynamics problems.)  The rest of the 15% is NOT DolbyA encoded.

Wrt my compressor (and possibly expander project) -- my compressor project is essentially on hold for two reasons 1) percieved lack of interest, 2) I have learned a hell of a lot on the expander and psuedo-DolbyA project -- need to feed back info into the compressor project.  Note my comment about the compressor project above.

Wrt my general purpose expander project -- it isn't really 'on hold', but has been in development on an ongoing basis during the psuedo-DolbyA project.  Initially, the psuedo-DolbyA was part of the expander project, but I found that the pseudo-DolbyA design requirements to be so very specific, that I seperated out the high level portions of the psuedo-DolbyA.  However, the vastly improved anti-intermodulation dynamics techniques that I have used to make the psuedo-DolbyA work so beautifully (clean, clear, no roughness) are being fed back primarily into the expander project and also secondarily the compressor project.

I already had pretty darned good dynamics classes (attack/decay/gain-riding/etc) in C++ for numerous purposes when starting the expander & DolbyA projects -- the compressor was the basis for all of the work.  There was already A LOT of intermodulation mitigation and well controlled attack/release characteristics,.  However, given my own sense of perfection, I have been striving for perfection in the DolbyA project -- FURTHER MORE STRONGLY controlling the mathematically unnecessary intermodulation effects -- thereby producing a DolbyA decoder that changes the character of the sound back to before the DolbyA encoding. The pseudo-DolbyA decoder has few of the typical artifacts of either SW compression/expansion or real HW compression/expansion.  I think that the ONLY  intermodulation distortions left over on the psuedo-DolbyA is greatly attenuated and the sidebands are limited to below about 50Hz wide on the LF bands, and 450Hz wide in the midrange bands and 1000Hz wide in the HF bands.   Even that those very significant limits (probably 10dB-20dB down at those bandwidths), the technically unneeded modulation products are attenuated even for the nearby sidebands.  I have successfully controlled the intermodulation without fatally affecting the COMPATIBLE attack/delay times (was quite a feat to do correctly.)

Of course, one cannot do gain control without creating sidebands, so I had to leave the NECESSARY sidebands intact so that gain control would happen (yea, I know that there are tricks to move the sideband power over to one side and to make the distortion less noticeable, but that is too much work.)  For example, the compressor is built from a 3band compressor, an 8 band compressor, a 1 band last chance compressor, and a carefully crafted set of limiters (lots of stuff.)   The expander has 48 degrees of freedom in its gain control, and the vastly simpler pseudo DolbyA needs to meet really specific design criteria.
(BTW -- I am not exaggerating about the 48 degrees of freedom in the expander -- that is why the expander can provide 1:1.6 dB or higher expansion ratio without pumping.)

Oh well, I am really just trying to help, and anyone is willing to email me for access to the psuedo_DolbyA.  The SW will work as long as the OS/HW supports the code.  I am limiting the time for the offer for my own sanity, however :-).

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: krabapple on 2018-04-23 18:35:46
Regarding my disgust as to the quality of some of the available recordings, and finding that they were like left with DolbyA encoding intact -- yes. 


So,  even though Dolby A was commonly used professionally from the mid 1960s through the rest of the analog era, either it was not noted down on many tape boxes, or the CD transfer/mastering engineers failed to notice , or simply ignored, the Dolby A indication on the tape box?

I have heard of cases of the former (Dolby used, but not indicated on the tape box) but I'd be surprised to find such cases were as widespread as you imply. 



Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-04-23 20:52:19
Regarding my disgust as to the quality of some of the available recordings, and finding that they were like left with DolbyA encoding intact -- yes. 


So,  even though Dolby A was commonly used professionally from the mid 1960s through the rest of the analog era, either it was not noted down on many tape boxes, or the CD transfer/mastering engineers failed to notice , or simply ignored, the Dolby A indication on the tape box?

I have heard of cases of the former (Dolby used, but not indicated on the tape box) but I'd be surprised to find such cases were as widespread as you imply. 





I am truly NOT meaning to misinform, and also I know that I am not crazy (at least, probably not), but it is really interesting that I am running into a lot of DolbyA encoded stuff.

Firstly, DolbyA encoding isn't 'fatal', and some people might like the hard, over compressed HF sound.
Secondly, I was SUPER surprised, and started doubting my sanity until I got help from a recording engineer -- and have PROVABLY DolbyA encoded material.  The effect of decoding that provable DolbyA material is similar to the decoding of some popular materials.

My online demos are NOT fakes, and they are NOT contrived in any way.  Most of my stuff is now on digital media, but I still have some CDs that I can grab, decode and demonstrate the proper effects of the decoding.  I have one example on my repository that I mentioned before (and here it is again), the song is 'Bicycle', and the repo is: https://spaces.hightail.com/space/z3H68lAgmJ
(The nasty sound that I hear on Bicycle -- before cleaning up with DolbyA decoding is similar to what I hear on a lot of disks (not all of them.)  Believe it or not, often the better quality disks tend to leave the DolbyA undecoded -- it is the horridly misprocessed material that isn't usefully DolbyA encoded.

I just found my Queen CD -- it is 99.9999% determined to be DolbyA encoded: 
Queen Greatest hits:  Copyright (C) 1992 Hollywood Records, ID 61265-2,

On the other hand, I have an ABBA CD that is all compressed to hell -- I know that because I also have a 2TRK copy, unprocessed and is a DolbyA master of ABBA Gold and MORE ABBA Gold.   The all compressed to hell CD (not plausibly DolbyA encoded):
ABBA Gold, 40th Anniversary Edition, Copyright (C) 2014 Polar Music, the ID code is long and unimportant since I don't claim that it is DolbyA encoded.

YES -- there are both DolbyA and non-DolbyA recordings available on CDs and online -- but, I suspect that some percentage of the 'too much treble' or 'harsh sound' complaints from people MIGHT be because of undecoded DolbyA.

I have only made passive mention about this revelation until now -- it seems like people tend not to be interested, because THEY JUST DON"T SEEM TO BELIEVE IT.

LISTEN to the examples...  If YOU have some material that sounds DolbyA encoded (too much HF, too compressed, but not DolbyB), then send me a snippet, and I can decode and send it back.   It might be easier if you just run my DolbyA decoder on your (relatively recent) PC -- IT REALLY WORKS!!!

However, there are indeed cases where the DolbyA decoder appears to work (almost), but the material is NOT DolbyA encoded.  Likewise, very often if you try to decode non-DolbyA encoded material, it sounds like cr*p.

SO -- there are two things that should happen (for the sake of audiophile and/or simple recording quality):  people should have access to DolbyA compatible decoders, or the record distributors should spend the several hours per release to just decode the material.

One idea came to mind -- leaving DolbyA encoding on a recording is kind of a very nasty form of mastering -- it brightens up the recording and does a bit of dynamic range restriction (on the low level side.)   In this case, that is just being cheap.

Too much of my music material has been changed from harsh sounding cr*p into near audiophile quality.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: krabapple on 2018-04-23 22:15:21
Your terminology is slightly confusing.  It's not surprising for a studio tape since 1965 have been Dolby A encoded; it is surprising for it not to have been decoded for commercial release .  So your concern is really 'Dolby A encoded stuff released without decoding'.

AFAIK, Dolby A decoding was not available on any consumer playback gear (e.g reel to reel decks).  So no Dolby A 2TRK tapes would have been released commercially, unless it was by mistake.  Yet you say you have  "a 2TRK copy, unprocessed and is a DolbyA master of ABBA Gold and MORE ABBA Gold."    What gives you absolute certainty that these tapes are what you say? 

Given all the adjustments to EQ and dynamic range available to CD mastering engineers, I just don't see how you can be certain that commercial releases are suffering from Dolby A nondecoding, versus normal mastering choices.   Tipped up treble is hardly rare in CD masterings, and neither is compression, and both are easily achieved without Dolby NR playing a role.

Yet you go on to claim:
Quote
Believe it or not, often the better quality disks tend to leave the DolbyA undecoded -- it is the horridly misprocessed material that isn't usefully DolbyA encoded.

I don't see how you can *know* this.

However, you also say you have 'PROVABLY' Dolby A encoded material . If that is so, then you can do the comparison I mentioned previously  (decode with a Dolby A hardware decoder vs your software).

But there's also the whole issue of the Dolby calibration tone, which would only be printed to the master tapes;  without that, setting levels for decoded playback, is guesswork....



Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-04-23 22:39:05
Your terminology is slightly confusing.  It's not surprising for a studio tape since 1965 have been Dolby A encoded; it is surprising for it not to have been decoded for commercial release .  So your concern is really 'Dolby A encoded stuff released without decoding'.

AFAIK, Dolby A decoding was not available on any consumer playback gear (e.g reel to reel decks).  So no Dolby A 2TRK tapes would have been released commercially, unless it was by mistake.  Yet you say you have  "a 2TRK copy, unprocessed and is a DolbyA master of ABBA Gold and MORE ABBA Gold."    What gives you absolute certainty that these tapes are what you say? 

Given all the adjustments to EQ and dynamic range available to CD mastering engineers, I just don't see how you can be certain that commercial releases are suffering from Dolby A nondecoding, versus normal mastering choices.   Tipped up treble is hardly rare in CD masterings, and neither is compression, and both are easily achieved without Dolby NR playing a role.

Yet you go on to claim:
Quote
Believe it or not, often the better quality disks tend to leave the DolbyA undecoded -- it is the horridly misprocessed material that isn't usefully DolbyA encoded.

I don't see how you can *know* this.

However, you also say you have 'PROVABLY' Dolby A encoded material . If that is so, then you can do the comparison I mentioned previously  (decode with a Dolby A hardware decoder vs your software).

But there's also the whole issue of the Dolby calibration tone, which would only be printed to the master tapes;  without that, setting levels for decoded playback, is guesswork....





As I wrote above -- we have done careful comparisons between a real DolbyA unit (cat 22 based/360-361), and my decoder.  My decoder works generally better than the original DolbyA unit and also the other SW decoder (I am not competing with other -- dont need to do so.)  The biggest difference betwen my decoder and a real DolbyA is a smoother/cleaner MF/HF, but close to the same response.  The 'other' SW decoder doesn't sound like a DolbyA AT ALL, but is better than nothing.*  The gain curves between a real DolbyA and my SW decoder are almost IDENTICAL (within about 0.25dB most of the time, and 0.5dB error once in a while.)  The curve shapes are indeed the same (it took high order polynomials to get it that accurate.)  ALSO, we didn't just compare against one unit, but multiple units -- and the person doing the comparison uses DolbyA and DolbySR almost daily.  In fact, we are probably going to start on an SR project, even though less SR material appears to be leaking out.

  *In my work, I found that emulating the filters on the cat22 was futile, and probably what the 'other' SW decoder did -- I designed the filters based on another concept (keeping it secret), and it works wonderfully.  The cat22 filters (or an adhoc filter bank) tend to sound like the other SW decoder and not a real DolbyA.

The lack of tone is a definite issue, but I have found that approx 3/4 or more of the material has a required DolbyA threshold within 1 or 2 dB.  It is easy to seek out the best level (usually takes me 2-3 tries to get within 0.25dB -- which is generally close enough.)  Also, if there is a tone, it is just an offset from a displayed level to calculate the necessary threshold (problem solved.)
Most of the online/CD material seems to require a threshold (defined by my software) of -14dB +- 1.5dB or so.  There are a few out-liers which require very different thresholds.  One recording that I have confounds me by requiring a -7.5dB threshold, but almost all others are in the -14dB range.  Usually, within the same album, the thresholds are the same.

Almost always, 0.25dB error is close enough, and most material works well with 0.50dB error.  Music that seems to require very close setting would be things that sound like the master copy of UnderAttack from ABBA.  Also, Band on the Run is also a little bit picky in the pauses at the beginning of the song.   Gotta get those tricky time constants correct (and a simple attack/release time constant WILL NOT WORK CORRECTLY!!!)

Of course, for professional purposes, you really do want to get within 0.25dB, but for causal listening practically all of the advantage can be had with an error of 0.5dB.

The way that I have found to use the decoder is to grab my on-computer archives and decode them (producing CD or memory stick for later use).  In some cases, it is very desirable to run some finalizing on the material (after decoding, ABBA needs a little compression/limiting to sound correct.)  However, my DolbyA copies of ABBA do sound NICE when properly decoded.  In the specific case of my ABBA recordings, almost all of them are not usefully DolbyA encoded, but I do have one which was obviously not finalized -- and fortunately (for me) left the DolbyA on as the normal dynamic range control. 

As I have claimed -- just because I have a certain tool, it doesn't mean that I believe that the tool is useful for everything.  It is just that it is useful (whichever DolbyA decoder you use -- a real DolbyA or my SW) in more cases than most people might realize!!!

Title: Got rid of a lot of 'earwax' from old DolbyA recordings.
Post by: jsdyson on 2018-04-30 13:57:13
As you probably know by now -- DolbyA recordings without decoding tend to be a little on he harsh side, and also have more old tape hiss than really should be presented to the listener.
Proper decoding can get rid of the harshness and definitely remove a fair amount of tape hiss (mostly forgotten about nowadays).  However, the DolbyA encoding process has a mechanism to keep the compressor from excessive excursions and also an intrinsic delay which on average tends to remove some of the natural brightness in the music.

So, without decoding, we have a somewhat clipped high frequency range -- boosted between 6-15dB, with lots of hiss.  (The boost is less at high signal levels, however.)  This does make the sound harsh.

I have improved the decoder further -- still in the testing stage, but initial results are superb -- the 'clipping' is still there because of the encoding, but the natural analog delay is now mildly compensated for, along with some mitgation of the more nasty effects of the clipping.  Cannot replace the removed information because of the 1msec clipping, but still there are cascade effects that are now gone (a true DolbyA decoder 'fixes' the problem by further damage to the signal and even more delay to hide the clipping.)  Some things in the DolbyA design in the encode/decode process naturally compensate for each other, but other things just get worse for each pass through an encoder/decoder.

Thanks to the digital DSP gods, if the signal defects are known about, they can more easily be compensated for.   The compensation in a real DolbyA might have required some kind of delay mechanism that doesn't exist in the design.  Back when DolbyA was designed, a loss of transients and/or some nonlinearity in the top octaves of audio wasn't such a big deal.  Nowadays, loss of ANYTHING audible is a big deal.

One thing that the listening community needs to be educated about -- a lot of the stealth DolbyA encoded listening material out there has TOO MUCH HF intensity and is actually distorted to some degree.  We (the listening public) have become used to a compressed/distorted high end on digital materials (esp on the old stuff.)

The DolbyA decoder both removes the excess intensity (while maintaining whatever true HF there is left in the recording), and now even helps to compensate for some of the encoding short cuts.  The decoder DOES NOT replace the missing transients, and unlike my 'restoration processor' or 'uncompressor' as I call it today, doesn't try to recover some of the transients.  Basically, this new version of the decoder uncovers much of the transient that is left in the recording without faking anything.

The decoder is most likely a better decoding device than has ever been publically developed, and is able to recover more of the music HONESTLY than ever before.   The result is a pure rendition of the before-encoding.

Historically, the phase of encoding that I am decoding is the tape that goes to the last phase mastering or to the pressing facility.  They, of course, tried to protect the material as long as it could be protect with NR systems like DolbyA.  In a way, it is very fortunate to the enlightened that there is now access to a really good digital DolbyA decoder.
Title: Newest/latest greatest decoder (DolbyA compatible).
Post by: jsdyson on 2018-04-30 19:05:50
Here is a copy of the decoder and a nice description of what it can do, various music that it works on, and how to use it.  To install, just put the binaries and DLLs into the current directory and use it!!!  Works on recent Intel boxes.

It is lightyears beyond anything else that I have done, and works better than any decoder out there.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-05-01 13:25:24
Found one minor tuning problem (only appears in very specific cases) and also improved the anti intermod after the processing, the intermodulation products of each bad are better filtered -- if you use --hpf, it filters the intermod even better.  The only real cases where the intermod at HF is really irritating is on certain materials with continuous natural HF material like ABBA.  The results are a little smoother yet now... (The results were already pretty good.)

The tuning problem manifested most (out of all of my recordings) on "Band on the Run', where he stuttered timing at the beginning made an error in the attack characteristics more obvious.  The actual error wasn't in the basic attack code, but rather the intermodulation mitigation. This error has been fixed (at the expense of a slightly greater amount of intermod -- probably NEVER noticeable.)  If you want to disable most of the intermodulation mitgiation, you can use the --raw=257 switch.  That disables both the early and late intermodulation mitigation.  Most of the difference can be heard when there are multiple continuous voices or tones, and there'll be a sharper/rougher edge when the mitigation is disabled.  There are other negative effects of turning off the mitigation, and the goal is that there never be a real problem when it is (by default) enabled.

Sorry for this quick follow-up release, but i was running one of my critical review passes, and found the potentials for improvement. Everything else is the same.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-05-02 01:44:22
Just a quick post -- have a minor bug on the highest freq channel.  If it is a little hard, that is because I made a mistake in moving some code.  I am doing more tests, and intend a new version within a day or so.  I found another method for intermodulation mitigation -- running final tests, and will include the 9-20k fix also.  You shouldn't notice a problem very often -- when I saw the code (I couldn't hear the difference), I was quite irritated with myself.  I did such a good job of figuring out how to get 9-20k working perfectly, and then I made a mistake in the final calculation!!!  (IT has been correct in the past -- I had an #ifdef that I didn't move a line of code around.)
Title: VASTLY superior decoder (decodes DolbyA) -- much intermodulation is now removed.
Post by: jsdyson on 2018-05-03 02:07:01
This decoder is not as good as possible, but probably better than normally available.  Beautiful results.
Previous version got really good feedback -- people starting to notice that the DolbyA encoding is real!!!

Filename containing the new -- really good -- decoder:  da-win-02may2018.zip
Short manual:  DecoderA.pdf

Location: https://spaces.hightail.com/space/tjUm4ywtDR

For convenience -- decoder posted here...  An example of a VERY DIFFICULT portion of a song -- lots of HF intensity, but intermod really suppressed!!!  (Not everyone likes ABBA -- but listen to the voices that are still seperate -- most copies of ABBA get smushed together.)  There are probably better examples -- but ABBA is about the most difficult to reproduce from my collection.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-05-03 10:48:38
Found another step of intermod mitigation that was easy to apply.  Also, the previous 02may release was a little aggressive in removing excess LF when using --hpf. (excess LF causes some troubles in some cases.)

Also -- USAGE NOTE -- intermod can be further suppressed by using the --hpf switch.  This FURTHER helps to remove the troublesome out-of-band lower freq intermod in each of the 4 bands.  Previously, the --hpf switch was way too aggressive, but the new modification above plus the slightly modified --hpf switch makes another quantum jump (in the sense of BIG quantum, not SMALL quanta :-)).  The demo repository reflects the improved LF performance & behavior.  There are already some very non-intuitive approaches for improved quality.  Unfortunately, the next performance enhancement will most likely need to be optional, because it will require approx 2-3X more CPU.  The current version is approaching a quality vs. CPU perf limit with the current architecture -- I have some performance ideas, but I don't think that the cost/benefit (cost of my dev time) would be worth it.

Sorry for this continual iteration, but as the decoder is used more, there will be more improvements.  I suspect that without the next major (2+wk ahead) improvement that this should be stabilizing.  (I do another major step planned -- upcoming version should blow away ANY dynamics processing experts WRT quality -- -this should already be pretty good in the wider scheme of things.)

The updated version also resides on the decoder code repository.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-05-03 13:50:31
Quick update -- decoder working great, but I have a usage suggestion:  alot of material has excess errant material in the upper frequency range.  I suspect that it results from previous encode/decode cycles and/or perhaps additional intermod being produced.  I have found that the intermod cascades very rapidly.  So, a minor suggestion -- if you are using SOX, apply a 'sinc -t5k -20k' to the output of SOX into the decoder/expander.  I am going to look into further shaping the attack/decay curves which might help, but it really isn't a bug -- if the careful attack/decay shaping hadnt been happening, even the sinc command above wouldn't help.
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-05-03 18:34:12
More interesting news -- I have been thinking that there was too much intermodulation between the gain control and the signal.  It was 'GOOD', and in fact 'GREAT' considering that it is on a computer using DSP code instead of analog.  But -- I had a minor EUREKA -- I redesigned the attack/decay code (significantly) to better stitch together the two attack curves.  On a real DolbyA -- there are kind of two attack curves, but they are just basically one or another (this is the 'softening' or whatever he discused in literature.)  I reworked again -- already having two very smooth curves.  However, I decided to stitch those curves together so that there is NO discontinuity.  The difference in sound is significant (in fact, I no longer need to do the fancy approach that I expected to do in about 2wks!!!)  I am in the midst of very aggressive testing.

All I can say is that the excess sizzle is 99% gone now!!!   The suggestion about the sinc filter in the previous message is absolutely no longer needed on the next release.
The good news for me is that I don't have to do a big, long design of a complex filter for the 80-3000Hz band -- I was going to have to split it in two, but no longer needed.

John
Title: Could not hold this back -- intermod really improved farther.
Post by: jsdyson on 2018-05-03 19:02:07
Up until now, I had been filtering out the out-of-band intermod products, but not trying to get rid of it at the source.  (I am speaking of the excess intermod, not the theoretically necessary intermod.)
I had thought that I had made the detection code as good as it could get, but I did another review.  It was indeed VERY GOOD -- pretty darned near perfect, but I noticed some kinks in the detection characteristic.  Ahhh.... NOWAY that would make a difference, right?   Well -- I stitched together the curves as well as I could do in a few minutes  -- no heroics, but just to make sure that the attack was smooth at the junction between the fast attack and the smooth gain control (DolbyA has two curves -- sort of) on each channel.
I joined the curves together perfectly, but also biased the joining towards the slower attack.  If you do the math, I think that one would find that the time domain joining and the level intersect when doing the bias between the speeds in the way that I did.  I did some listening, and it was close to perfect -- but then I did some more adjustments (basically compensating some other things for what I had done to the actual attack code), and magically 99% of the edginess is gone.  I ACTUALLY THOUGHT THAT THE EDGINESS WAS IN THE RECORDINGS!!!   But the sound is much smoother now, but everything is there.  Since I moved some of the timing around, I also had to better synchronize the audio with the control signal -- minor difference -- about 200usec movement in the audio signal.  That minor difference makes a big difference on how clear the transients and 'tinkle' things are.

I would NOT have done this release without it being a major improvement.  I was actually finished posting anything new for two weeks until now.  This is too good to wait.

Nothing has changed, except continue to use --hpf if you want to get the absolute minimum intermod -- some of that intermod results from the theoretical necessities of gain control, but doesn't really hurt the gain control that much when you remove them.  So -- it is good to use --hpf if you don't need 16Hz audio :-).

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-05-03 22:10:31
Been doing some tests on the decoder (very detailed), and made a few mods -- don't worry -- not worth an update yet.  However, there is more quality to be mined believe it or not.  I was SOOO amazed about the improvement based upon slightly changing the attack structure (decay has to stay the same.)  I am in the midst of writing some information on the DSPrelated forum, and hope to write the next installment soon (days.)  I am covering all of the various aspects of audio gain control and the things like different frequency bands, I'll be talking about attack/decay, intermod, dB or linear attack/decay.  Point being -- I have never been so suprised with this attack structure improvement.  It must be that so many of the details are 'perfect' that this small improvement is making a big difference.   I have a very fancy general purpose expander and GP compressor project also.  I think that I have new information to apply to the expander!!!
There are things that I am learning that are JUST NOT common knowledge -- all too often a compressor is designed with a very slow attack (slower than the audio frequency cycle time), and that limits flexibility.  I think that once I get it all together, I might be able to create a short paper on doing VERY good quality audio gain control.
This decoder (especially the current experimental version that I am working on) is doing some amazing things.  The experimental version has been more carefully tuned to fit the needed decoding characteristics -- and it is AMAZING how tricky it has been to unfold the 'inverse compressor' design concept of DolbyA into a feedforward and still match the attack/decay.  The attack is crazy-tricky because in the feedback structure, it has the initial 'surge' or 'boost' that the feedforward has to emulate.  Tricky stuff for sure!!!



John
Title: New version again -- less aliasing yet, and also found more aliasing!!
Post by: jsdyson on 2018-05-04 12:42:36
I have been focusing ONLY on distortions of any kind in the last several days.  The general attack/decay matches every recording that I have in my position (that is DolbyA recording.)  Also, theoretically I can derive all of the general attack/decay parameters proving the reasons for each one.  The attack/decay is 'perfected' -- not playing with that at all.   Trying to get that 'sweet' sound.

More good news -- firstly, I found a minor bug that the postfilter wasn't being invoked sometimes unless one specifies that it isn't disabled -- stupid, huh?  Also, there has been an ongoing calculation bug for the 9k-20k range where it was boosted by a slight amount.  You'll find that 'Sess' sound is more pure now.  Also, aliasing is boosted less.  If you want that slight 9k-20k boost back, then just use the --raw=4096 switch, but I don't see any reasons for using it.  (I am keeping it only for regression testing reasons.)

I changed the attack filters (and the post filter) to produce fewer harmonics also.  The small nonlinear postfilter has been very slightly retuned to consider the non-ideal nature of low pass filters when the rolloff is close to the nyquist frequency.  This has an additional positive effect ofreducing the amount of useless HF (produced because of nonlinearity in the necessary processing) in the control signal which only produces aliasing.

Have gotten feedback and keep on telling me if you hear something odd!!!

The short document has NOT changed, so I am just posting the decoder zip file.

John
Title: Oh my, got a complaint about decode quality on another forum
Post by: jsdyson on 2018-05-05 23:45:55
On another forum -- someone talked about the problem with decoding 'Band on the Run', but I knew about it -- blew it off.  I felt like that odd sound was the way that I remembered it when it came out (it had a bit of a duck at the beginning -- but it was 40yrs ago.)  Anyway -- I first blew-off the comments and simply 'fixed' it by disabling the MF decoding -- which is a feature that is available for when in the olden days that the MF compressor was disabled.  It was either due to failure or a quality issue -- but no matter what, the decoder has that feature.

However, back to the problem with the example...  I finally felt like I could no longer blow-off the problem.  I did a lot of thinking about the attack/decay characteristics and finally realized that I needed to compensate for the feedback-to-feedforward unfolding for the decay as well as for the attack.  I analyzed the situation, and found a reasonable solution (of course different than the attack solution), and definitely found out that there can be a decoding problem unless the compensation is done.

Anyway -- I am testing the decay code fix.  The good news is that the decay code is slower and doesn't quite have the intermod issues that the attack code does.  So -- it is NOT a big problem to fix.  However, the testing is not quick.

There will be a new version in a few days with this correction.  Several files which I though were NOT DolbyA encoded now are decodable.  So that excuse 'not DolbyA' is less likely correct now..

I am a little embarassed, but happy that I understand this better.  This DolbyA decoding thing is NOT trivial.

John
Title: decoder: Bugfix for failed handling of uncommon dynamics
Post by: jsdyson on 2018-05-07 02:52:44
This bugfix is important even though it might only help a few songs that it previous worked with.  The good news is that it works on more examples -- the code was missing an important part of the math as explained below.

The bugfix only helps on very few songs that it could previous decode at all, but when it helps -- it REALLY helps.  The fix implements handling of better feedback to feedforward conversion.   I did an EXCELLENT job on handling the attack times -- I was focused because proper handling of such makes the sound purer.  The thing that really bothers me about a lot of the recordings nowadays is the terribly harsh sound, and I was focused on that matter.  I ignored the handling of the decay time, because I didn't think that it needed correction.  When I revisited the matter with seriousness -- I realized that I sometimes suck!!! :-).

This required a lot of testing as you can imagine, because changing such a basic behavior can totally break things, or break them in difficult-to-find ways.

The REALLY good news is that several of the recordings that I claimed might have been recorded by a broken or disabled DolbyA can now be decoded by this corrected version!!!

Finding bugs like this after such assertive statements by me has been humbling.  However, the good news is that the new version is available.  Also,  I have some examples from the group 'Bread' that clearly show how much of an EXTREME improvment can be had.
On the repository below -- listen to the two songs:

BabyImAWantYou and MakeItWithYou.

The examples are BEAUTIFUL examples of traditional recording techniques and REALLY REALLY show off the decoder!!!  The before and after are like night and day!!!

Repo:
https://spaces.hightail.com/space/z3H68lAgmJ
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-05-08 14:04:55
Minor update: possible fix/improvement coming.  Ran into some troubles with low level transients (that is stuff at lest than -6dB gain.)  FIgured out that I was mistakenly basing the long-term, feedback-to-feedforward calculations on the raw signal level and not the input-filtered version of the signal level.  It is a simple matter of changing a variable name as input to the calculation.  I looked at this all night, and realized the screwup.  The version that I am testing is what I had meant to do -- and this was in the section of code that was added to properly support the decay calculation.  I plan to make this available in a day or so -- running tests again.  It is becoming very boring to listen to music!!!

The basic decay constant is the same as it always was -- approx 30ms for HF, and 60ms for LF/MF.  However, consideration needed to be given for previous history because of the feedback configuration.  That support for previous history is what I added several days ago.   When I did the history support code, I used the raw/unfiltered (well, 1 or 2msec filtered) signal level and not the dynamic input anti-intermod filter results.

The primary result of this fix is that low level material (when running less than about -6dB of gain) will have a more correct decay time.  Before the fix, the decay time would have been a little too fast and cause some gating or L/R image problems at low levels.  The version in testing seems to mitigate the problem and results from a minor code correction and not a major redo.

Giving you the idea of how I made this mistake -- the delay calculation improvement that handles the FB->FF conversion required several hundred lines of code and is stuff that needed to be written FULLY from scratch (no textbooks, appnotes or research papers to guide me.)  So, even though I often write over 600lines of new, practically bug-free code per day  - that kind of programming rate usually only happens when doing something that had previously been well researched.

This current decoder programming IS the research!!!   :D .  I have NO idea as to the way to document the design, but planning to continue my recent DSPrelated posting to talk about compressors/expanders and how to make them sound good.  Might be able to talk a liltle bit about this decoder vs. a normal expander design.

I apologize for the error!!!

John
Title: Promised decoder update
Post by: jsdyson on 2018-05-09 02:24:57
This one, if it works for you finally has gotten some serious tuning.  The decay code has now been corrected, but you might find that the threshold MIGHT have to be slightly increased, say from -15.75 to -15.50 or -15.65.    The biggest difference isn't really the corrected decay (better handling the FB->FF conversion), but also the intermod tuning is now really good.

The intermod calculations are not trivial, and there is a bit of tuning involved.  For example, I found that DolbyA sources tend to have a lot of 9kHz distortion built-in.  At the normal and higher settings, that distortion is carefully removed.  So, the decoder not only mitigates its own intermod creation, but helps with the encoder problems also.

There is a new switch, as documented on the DecoderA.pdf file.  It is --ai=<mode>, where <mode> can be none,off, min, med, high, max.

The default setting is --ai=med, which gives a good balance of negative and positive effects.  If you find some problems with the --ai=med setting, then I suggest -ai=high if you want to remove all but the worst intermod -- it might seem a little less crisp, but that crispness is 90% intermod.  --ai=off would be a setting that one might use if the crispness/fake HF might be beneficial.  I don't know a real reason for using --ai=none unless testing.

I have been getting hung-up listening to the music (at the --ai=high setting, which  I prefer) -- I am hearing things that I never heard before (like details in the ABBA chorus.)  I set --ai=med because I try to be conservative -- might end up setting 'high' as the default in the future.

Have fun!!!
John
Title: Just posted decoder has a bug -- don't try to use it -- fix soon
Post by: jsdyson on 2018-05-09 05:02:17
Please don't waste your time on the recent version.  New one coming soon.  Bad HF freq resp problem, and my hearing has troubles sometimes.

Fix coming tomorrow.
Title: Reissue of new version -- plus apologetic technical explanation
Post by: jsdyson on 2018-05-09 14:24:53
Firstly -- I apologize for wasting bandwidth and anyones time for my mistake.

okay -- here is the version with the HF restored.. Here is the technical explanation of the bug, and why it happened...  First, my hearing sometimes winks out on HF -- and I didn' tknow that the problem was happening.   The technical reason is as follows:  I had been using FIR filters that were like 1800Hz to 15000Hz bandpass, but the length was 511 at 96kHz.  I needed that 15kHz to really be 15kHz with no skirt, however, when also making a bandpass with the 1800Hz low end, it causes the skirts to widen.  The filter ended up encroaching on the HF region to below 9kHz, and any loss in that range is bad ( causing a characteristic limited HF sound.)  I didn't notice the problem till too late (hearing came back), and I panicked after distributing the broken version.

The correction was to seperate the bandpass into two separate filters -- I used a series of simple recursive high pass filters for the 1800Hz, even though the skirts are loose when doing simple filters like that.  For the HF side, I simply changed the BPF into a LF, and that tightened up the skirt at 15000 and restored the HF for the 3k-9kHz range.  (When talking about 3k-9kHz, the skirt on that filter is very wide, and so the passband for that filter extends well into the 9k-20kHz range.)  The way that the DolbyA compatible filters work are actually counter-intuitive -- where if the 80-3k band is active, it actually opens up the band well above 7kHz, and partially overwhelms the 3k-9kHz and even the 9kHz-20kHz range.  The VERY IMPORTANT band pass filter used to remove intermod on the 3kHz - 9kHz (and also the 80 - 3kHz filter also had troubles because of some special stuff being filtered there also) was badly messing up the HF most of the time.  (Sometimes the gain control would open up the HF band -- but most of the time the BW was limited.)

As i tried to describe above, the filters were re-architected -- with a compromise solution so as to avoid using LOTS more cpu to implement the LF range filters with an FIR filter.  I really need to use some kind of multirate scheme to decrease CPU usage, but frankly I worry about possible quality impact -- so for right now I am using simple textbook filter design.

The sound of THIS result is much more transparent, and even the --ai=max mode sounds fairly transparent most of the time.  I still recommend using --ai=med for pro applications, and --ai=high or --ai=max for listening (or producing tapes for listening.)  All modes using this verison of the code work MUCH better than the previous borked release from a few hours ago.

Sorry again -- everything that I am doing is IMMEDIATE and I am not holding anything back.

John
Title: Last minute updated release. Wanna give you the best possible.
Post by: jsdyson on 2018-05-09 16:07:45
Update complete.  I keep on finding things that can/need to be improved.  This is going to have to be the best so far!!!

Each time, it keeps getting better and better.  Sometimes I make mistakes when making bigger changes -- this should be pretty good.

John
Title: Bad day yesterday -- today, finally the results that I wanted
Post by: jsdyson on 2018-05-10 03:54:50
Have some patience.  I have been having troubles with one of my libraries, my hearing (fixed) and trying to get this thing sounding correctly.  I have also JUST NOW corrected my filter generating library -- amazed that it was working as well as it has been.  I caught one bug by running  a spectral analysis, and noticed a rolloff before I had expected.  Been going nuts trying to find the problem and then found that my FIR generation library (that I wrote) was borked (by me.)

This FINALLY sounds correct -- the way that I wanted.  I went for almost a full day breaking things yesterday, and I didn't realize it.  I go through day where my ears are messed up and finally they do clear up.  I woke up to a nightmare and been trying to fix problems.

ALL of my work is mathematically and engineering based, but the work has to be tested by me.  Unfornately, I ended up with a cascade of errors.  I found the culprit, and am even posting the source code for the corrected routines (that calculate the FIR filters.)

This version has been tested by GOOD ears (more than just me), and also I have some technology examples.  It is difficult to hear the difference depending on the material (some material is very obvious.)  Here are some instructions (only on this version and on) that you can disable all quality improvements and just depend on the carefully designed detector (more than just the typical RC time constant thing), but I didn't bother implementing the typical broken detector design -- so you cannot hear the worst possible from this software.

1)  The default settings are the best now.  No complicated options -- the default settings sound just as good as the suboptimal/unprocessed settings (just a little different.)
2)  To disable everything special, use the following:  "--ai=none --raw=8192"  (the sound will be crunchier on difficult material -- that is mostly the intermod.)

I PROMISE this is at least a pretty good version.  I was upset to see 14 downloads of the previous version.  I pray that people haven't wasted too much time.

I am back functioning very well now :-).  So is the software.  The culprit (but fixed) FIR filter coefficient generation library is also posted -- I didn't comment the code very well -- but to DSP people it is fairly obvious what it does.  To non DSP people - -the filiters require a long list of numbers that do things like low pass, high pass or other things like that.  These numbers are used by the general purpose filter code (kind of a filter program.)  This file 'makefir.h' generates those numbers based upon what kind of filter, how many coefficients and what kind of 'window' you want.  This primitive, but works for some cases of  what I need to write the code.

If you look at the simple DecoderA.pdf file, you might notice that I removed some of the fancy --ai options -- not needed anymore, because it always sounds good now.

John
Title: Not replacing broken code -- more sound-sound improvements.
Post by: jsdyson on 2018-05-10 15:34:12
Finally -- more good news -- hearing now 100% and my results are back there also.

Only one general minor bugfix -- should appear to be more open sounding -- but not critical for update.

This version of the decoder is exactly everything that i was attempting 2 days ago, but working 100% correctly.  I had run tests (research) earlier, so I knew what to expect -- but this is more a result of more thorough measured results.   The new capabilities are as follows:
1) More complete anti-intermod code for the 30-3kHz range.  This processing goes far beyond anything that I have heard of.  The only audible difference is a small reduction in middle frequency graininess.  The improvement is very real, and this will eventually be the default way that the decoder is running.  Currently, this mode only functions at --ai=high or --ai=max.  The normal anti-intermod code works fine at --ai=med (the default mode.)  The anti-intermod code can be partially disabled at --ai=off, or fully disabled with --ai=none.   There is almost never a reason to disable the anti-intermod code except that the more advanced modes do take more CPU (high or max modes.)

2) Totally new anti-hash facility and a control for it.  I have found that many HF intense recordings (even mellow stuff like the Carpenters -- due to vocal sibilance) have a certain hash in the sound.   From what I can tell and looking at the spectrum, I believe that it is a 'beat' between the gain control on the various frequency ranges -- most likely between the slower 80-3k range (which actually extends beyond 10kHz to at least 13kHz) and the faster 3-9k/9k-20k ranges.  Since there is a feedback design in the encoder, it appears that the compressors are interacting in an ugly way.  ALL DolbyA recordings seem to have this problem depending on the density of the HF sound.  My processor has some code/logic to avoid the appearance of the grainy HF sound.
The anti-hash mode can be enabled/disabled by --ah=on or --ah=off, and is always disabled when the anti-intermod is fully disabled.  There should never (very, very seldom) be a need to disable the anti-hash code.

The new stuff is also explained in the DecoderA.pdf file.  The decoder has been run through lots of tests and spectral review -- no unexpected notches or dips anywhere :-).  There is a dynamic dip at 9kHz (1kHz wide, which corresponds to the attack rate of the DolbyA) which fills in when really needed -- and that is part of the anti-hash processing -- should be inaudible except for slightly cleaner sibliance.

Sorry again for the previous problems -- these items had been created a while ago -- but I had to previously revert a lot of code so that I could regain control after all of the previously broken changes.

This should make people happier than even before.
John
Title: Peeling the onion -- the last improvements opened up more!.
Post by: jsdyson on 2018-05-10 22:26:18
Improvement:  detector code had some aliasing (almost the last sole source of obvious distortion) -- causing a hard edge, and also the attack was a little bit too slow (a counter-intuitive situation.)  Result after the readjustment -- simple/clearer voices without the hard edge.  None of the attack/decay general architecture was changed, but some minor improvements were made for better SIMPLE clarity while removing edge-buzz (hard edges.)  The signal attack is also followed more closely.  Without the edge-buzz the sound is cleaner -- but the sizzle might be missed.  You really don't want the sizzle -- it is full of all kinds of distortion uglies :-).  Probably could not hear this improvement before the last set of improvements decreased the other distortion sources

Background: After the last set of improvements, it opened up other possibilities of improving the audio.  The innate distortion of the internal operation of the code is practically NIL (massive amount of subtle math/filtering/etc).  This has made it possible for me to detect more (more and more subtle) problems elsewhere.  This one is a really good one -- one of my favorite subjects -- the 'level detector' and basic attack code became the biggest source of distortion again (after everything else has gotten better.)  This one was a real learning exercise for me -- and the side benefit is that the code much more closely follows the attack of the audio signal -- yet distorts impossibly little.  This has gotten to the point now where I cannot hear the 'edge' distortion from the detector at all (maybe my ears messed up again?  I don't think so -- this is really an improvement.)   You'll notice that the edge of voices (sometimes manifested) is now gone.  Karen' Carpenter's voice often has no real hard edge anymore!!!  This is getting DAMNED close to success.  ABBA still has chaos in the sound -- but the voices in the chaotic chorus are clearer.  Material like Dionne Warwick, Bread or other naturally mellow (but detailed) sound is more natural yet.  FEEL FREE TO BOOST THE TREBLE WHEN YOU WANT - the distortion is much diminished, and increasing treble isn't so painful anymore.

The features are the same -- I am not willy-nilly adding tunables anymore unless there is something that REALLY makes a difference.  The complexity of tunables and supporting so many code paths causes too many bugs -- so I am trying to keep things simpler -- maintaing quality as the singular goal (well, also consider CPU usage, which has gotten pretty bad lately.)

So, semi bad news regarding CPU usage -- the code barely runs in real-time on my Haswell (4 core, but only one is used by the program) 4770 processor.  Laptop ATOM/Silvermont processors run several times slower than realtime -- but I am sure that newer processors can run up to maybe 2-3times realtime.   I am NOT going to multi-thread the program -- I do have a really good multi-threading infrastructure, but I believe that there is some overdesign in the code, and I think that the next step will be to speed up the code without impacting quality.  Multi-threading only makes multi-platform support & debugging more trouble if simply speeding up the code can resolve the speed issue.

I am having a blast listening to my library with this processor -- simply clean sounding for the first time -- better than I remember from vinyl in many cases.

John
Title: Usage hint for new users of the decoder.
Post by: jsdyson on 2018-05-11 15:54:20
Wanna give a quick usage hint -- if I was a new user, I might also have gotten tripped up by this...  That is, even though it will become obvious to follow this rule!!!

Never change the level of the input material unless you compensate for it.  For example -- if you need to decrease the input signal by 3dB (from the original file containing the recording), then you need to specify a gain (or provide gain) before or at the decoder.

So, if you have an input file, and then did a 'gain -3' in sox, then to compensate, an '--ingain=3.0' needs to be supplied on the decoder command line.  The gain between the music file and the decoder must be zero dB, or the threshold will never match.  (Of course, you could change the threshold to make the decoder work instead -- but then your signal levels will be wrong anyway.)  So -- it is best to keep everything gain 'neutral' and/or compensate by using the '--ingain=xx' switch on the decoder.

Just trying to help to avoid confusion!!
John
Title: New update -- didn't expect this one!!!
Post by: jsdyson on 2018-05-12 01:18:15
Did a careful code inspection, and found a really stupid mistake.  The long term decay for the right channel was dependent upon the left channel!!!   Really silly, and I am surprised that I didnt' catch it by now because I do a careful listening test.  I guess that the longer term levels are generally fairly consistent, but now the code is more correct.

After finding this bug, I did a VERY careful code review to make sure that there were no other bugs like this.  The problem is that I named a lot of variables like this:  LINrmsinl and LINrmsinr, and some of the math results in fairly long lines of code.  There were two mistakes, one was simply in a normal place, and the other was at the end of a line.

This version should GENERALLY sound similar -- I think that there are a few minor improvements, but additionally this long term decay error/miscalculation is fixed!!!
Title: Found one song that required a decoder patch
Post by: jsdyson on 2018-05-12 05:13:37
One of the decoder calculations didn't go deep enough, so I added some depth so that it would handle certain kinds of long duration beats (yep -- that has to be one of the calculations to emulate the feedback to feedforward calculations for the decay time!!!)  I cannot implement a forever set of calculations that have very diminishing returns -- I very slightly underestimated what was needed for the very worst case.

The difference between this one and the previous is pretty small -- both are good.  This one is SLIGHTLY better.  Whenever I find a problem, I want to distribute it ASAP -- since there were no downloads on the previous, maybe this is in time to avoid wasting anyones time with the previous (again -- it was good also)!!!

The only other difference is that I inlined more of the functions so as to get a percent or two faster runtime.

John
Title: Commentary about doing an encoder and/or future work
Post by: jsdyson on 2018-05-12 12:28:53
I have gotten questions about the DolbyA decoder, and why I didn't do a DolbyA encoder also?   Well -- there are at least two reasons, but the first and most important is that most of the problem is to copy music from old archives and be able to use it (process/mix/finalize/produce) with current technology.  The second reason is that DolbyA is so much weaker NR than more recent technologies, that I don't suggest using it for encoding.  Perhaps I could do an encoder that produces much less distortion than anything else (like the decoder has less distortion than anything else), but why encode into DolbyA?   If/when I do a DolbySR decoder, then an encoder with that technique might be more useful.  The big problem with SR is that it is incredibly more complex than DolbyA.

Please note that this DolbyA compatible decoder truly sounds similar to a real DolbyA (sans fuzz, distortion, lack of clarity) with a very similar freq response balance.  (Even if someone uses the cat22/360/361 as a design basis in digital form, the result will not likely sound similar because the filters don't emulate well.  I found that my decoder would sound similar (but cleaner) to another known DolbyA decoder if I used the DolbyA design as a reference.)  I rejected those filters -- and I was worried that I could come up with something compatible that sounded more accurate, but I was lucky to find a better solution.

Doing a REALLY compatible/similar sounding DolbyA encoder would be a project similar in scope to the decoder project, but with even fewer users and less usefulness.   Frankly, even if I had a reel-to-reel deck, I would NOT bother encoding anything into DolbyA form.  DolbyA MIGHT be more useful than SR for very long term archival purposes (because of the simpler DolbyA decoder design), but still -- I'd try to find something BETTER QUALITY than DolbyA.   Knowing what I know now -- I am somewhat suspect of the quality of any of the dynamic gain schemes which can-not mathematically be reversed, and DolbyA (however being close to being reversable) is not reversable enough.

My criteria for a long-term analog compatible NR (and possibly dynamic range extension) system would be a constant compression ratio, multi-band system with mathematically designed analog&digital compatible characteristics for the filters.  At least, if properly executed and working on a deck with a fairly flat response, it will be totally reversable (distortion products even better cancelling.)

So, what I suggest if someone wants to use a nearly analog compaitble NR system, it would be closer to the HI-COM (afair -- not sure) type design.   The multi-band approach is good, but the DolbyA filters are kind of finicky, and I'd rather see the system designed from specification rather than HW design.   If there is an interest, and it would be really used if it works -- I could do a rough specification, and an implementation of both a HW and SW compaible design that has the best features of DolbyA and DBX.   One good thing about a compatible SW design is that it can be prototyped using software modules that act very similarly to real hardware.  For example, I'd base the design on dB linear technology (like THATCORP) stuff, and use standard filter design techniques that can be implemented in HW & SW, e.g. well constrained IIR filters that emulate well in HW.  FIR filters can be more ideal, but are also not easy to emulate in HW.   After doing a rough design and a SW implementation (probably 2X easier than my DolbyA effort), then a real HW design could be started.  Before that, I'd do as much of a spice simulation as possible.

The end result of such an effort would be at least 25dB NR, almost no level sensitivity, almost no modulation-type noise, very good transient response, much better distortion than almost any other system.  Also, encoding/decoding could be done on computer or in hardware, and the result of the encoding could be designed to be listenable.   So, it would have all of the advantages of DBX, DolbyA and DolbySR, and almost none of the disadvantages of any.

But doing new DolbyA encoding operations are only useful in museums where there are demos of ancient technologies :-).

John
Title: Usage hint for the DolbyA compatible decoder
Post by: jsdyson on 2018-05-13 03:48:54
I have a usage hint -- and let you know how I normally use the decoder.  Except for producing my own listening archives or demos, I use the decoder in realtime most of the time.  When I use it -- I don't use the default quality settings, but usually use the highest quality setting available.  There is also a 'heroic effort' setting when there might be a lot of high frequency intermod (e.g. lots of kids voices will do it.)

The normal setting runs a bit more quickly than the higher quality settings, and an upcoming version runs about 20% faster in general (at least that much better -- I just started on optimizations.)  However -- we are talking about listening quality here...

There is a very aggressive intermod removal mode, which doesn't have much of a change in frequency response or anything like that, but helps to keep voices and instruments clearly separate (intermod makes things mush together.)  The magical setting is: "--ai=high", which means set the anti-intermod on high.   There is another super-aggressive setting, which uses the aggressive improvements in the --ai=high mode, but also narrows the frequency bands that lend themselves to distortion production.  It really doesn't decrease the ultimate frequency response, but can make the music have a little less clarity.  That aggressive setting is "--ai=max".

The default mode is "--ai=med", but you should never need to specify it because it is currently the default.  The biggest disadvantage of the --ai=high mode is that it runs about 20% slower than the --ai=med mode.  Once I speed up the code to run perhaps 2-3X faster (yes, I have some ideas), I might move the default to be --ai=high at that point.

In casual listening, I always use "--ai=high", because my computer is fast enough.  The perhaps overly aggressive "--ai=max" doesn't run any slower than "--ai=high", it is just that some of the parameters are slightly different.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: dreamliner77 on 2018-05-14 03:27:40
One use of an a Dolby A encoder could be the Dolby "Air" trick:  https://www.gearslutz.com/board/high-end/128740-dolby-exciter.html
Title: Answer to possibly encountering 1/2 encoded DolbyA
Post by: jsdyson on 2018-05-14 07:29:25
Thanks for that pointer!!!  I have been hoping to find more concrete evidence that the 1/2 encoder technique was used, and below is my comment on my possible encounters with the use of that technique.   DolbyA just about maxes out the fastest compression that can be done in HW without splatting intermod all over the place -- so I can understand the use for compression only.

I do agree that some examples of my DolbyA demos just might be the playback of the 'half encoded' DolbyA HF compressed music (but unlikely in the present cases.)  My earlier decoder releases had a terrible bug that made me think that there was more use of the two HF channels only than there really was.  I had even advertised that some of my Carpenters examples failed because the specific Carpenters music was enhanced by partially disabled DolbyA.  Practically every major step of development of my DolbyA decoder has been done with public scrutiny, and versions up until about 3-4wks ago had a bug that some kinds of music wouldn't work right -- almost sounding as if it was trying to decode 1/2 encoded music -- but that was wrong.  The bug was in the decoder.  * Some of my decoding examples DO SHOW that there might still be some latent HF DolbyA encoding, but in those cases, I do believe that there just might have been HF-only use of the DolbyA encoder in those cases -- esp on vocal channels only.

However, my guess that I was trying to decode 1/2 encoded DolbyA was WRONG IN THE SPECIFIC CASES.   I had made a rather frustrating error in the conversion of feedback to feedfoward conversion.  (My expander is a compatible feedforward design -- works MUCH better than feedback and much better control of intermodulation.)  So -- after some embarrassment, I had to admit that the first versions of my DolbyA decoder were broken for some kinds of long duration changes in music patterns.  Frankly, it was just broken, and I had to rework the decay times just as the attack times had to be done differently in the conversion between feedback and feedforward.  A simple use of a 1msec/30msec and 2msec/60msec attack/decay times for the HF and LF/MF channels would have been terribly broken for a feedforward design -- even though tests showed that it ALMOST sounded like that it would work.

When doing ad-hoc comparisons in the results for decoding, a simple use of the straightforward 1/30 and 2/60 attack/decay pairs in feedforward ALMOST sound correct -- and are correct enough in perhaps 1/2 of the decoding attempts to sound okay.  But, that is simply wrong.  I have an almost 100% capable design where even the most evil and syncopated timing will cause the decoder to respond correctly.  A few weeks ago, I got a rather frustrating complaint that the decoder didn't work correctly on 'Band On The Run'.   After finally believing that complaint to be true (getting past my ego), I re-visited the basics of my design, and realized that the bug was certainly my own, and corrected the decay code so that EVERYTHING so far that seems to be completely DolbyA encoded is properly decoded.

I am not going to disclose the technique - because I have gotten some feedback  that this decoder (which my design is decoder only) is much more compatible than ANY OTHER software technique available, and also has much less distortion in difficult cases.  I have a proprietary copy of music, provably DolbyA encoded where a true DolbyA decode produces a chorus of childrens's voices that sounds like a blob of voices,  the Satin sounds like it has a messed up HF balance, and mine sounds nearly like a real DolbyA except the childrens voices are clearly distinct -- but closer to real DolbyA HF/MF balance.  This is partially due to the vastly superior intermod handling, where the amount of intermod is much closer to that which is theoretically necessary (especially when in --ai=high mode.)  Also a direct conversion between the commonly available DolbyA schematics and SW will definitely result in sound that is more similar to the Satin.  Use of that technique will result in something that kind of works, but won't sound like a real DolbyA.

I am still thinking about possible improvements, and there are even some stealth capabilities in the decoder which I am not disclosing -- some errors in the DolbyA encoding process which are compensated for.  I think that I have found the reason for that quality-loss syndrome, and somecapability for the correction can be disabled by a switch.  There is almost NO reason to disable this feature, as I have heard no music which is damaged by the default setting.  There is a difference in audible character, but the disabled version only discloses more distortion from the ENCODING process.  The default setting uses a rather tricky technique to hide the distortion from the decoder.  I think that I even know the theoretical basis for that distortion.

Truly, if you want to hear music that is closer to the pre-DolbyA encoded version than has been previously possible (including that which was decoded by a real DolbyA or Satin SW) -- my decoder will enable that ability.  Actually, '--ai=max' MIGHT be more accurate yet, but I believe that '--ai=high' is the best/safest bet.

I admit that my decoder is a decoder-only, and it will be staying that way.  i don't think that DolbyA encoding is of high enough quality in this perfect digital world to be of any long term future use.   There are too many flaws in the assumptions made, but it was a genius design for the middle 1960s.   Ray Dolby was a REAL genius.

I do have a better (easier to HW & SW replicate) design concept that is closer to a Hi-Com (AFAIR) technique, where it is almost a mix between the DBX & DolbyA techniques -- but with almost none of the disadvantages.   The amazing thing is that in the deep deep future, it would even be easier to replicate in HW only if needed.  (There are some aspects of semiconductor physics which are more consistent than the newer DolbyA design depends upon, and the reason for so damned many tweaks.)  For long term (Library Of Congress) type applications, I think that I have an idea for a better compander system which is very stable and dependent mostly just on physics.  It only requires matching -- which is easy to do on chips, but tweaking FETS is NOT so easy -- FETS do not replicate ideally so very well on chips.  The original DolbyA design, which does have some advantages over the 360, also has some disadvantages, but also comes closer to what is needed for long term archive recovery.

But, again, thanks for that pointer!!!

John
Title: Possible (and likely crazy) attempt to remove the extra add-on DolbyA sheen
Post by: jsdyson on 2018-05-14 18:32:35
Apparently, in the olden days, some record companies used a partially disabled DolbyA unit to add an additional high frequency sheen (or exciter-type) sound to their recordings.  (Apparently, CBS records had that habit.)   My decoder can provide a removal service for that sheen, but only if the entire recording was processed to create the extra sheen, and was not used for just the voices.  Any settings using the decoder for that purpose would be very experimental, but I can give you some hints.

Firstly, you would use the 'sheen removal' only after the full-on DolbyA decode (either from my compatible decoder or using an unmodified true DolbyA unit.)   Then, the way to invoke my decoder would be something like this:

sox infiles.type  --type=wav --encoding=floating-point --bits=32 - rate -v 96k | da-avx --lfoff --mfoff  --ai=med --thresh=15.75 | sox - outfiles.type  <also add needed EQ because of sheen removal>

Note the full details of using sox need to be figured out using the sox documentation, and I am just showing the general jist of doing the 'sheen removal'.  Note the specific switches for the decoder program...  -lfoff and --mfoff turn off the dynamic gain on the 20-80Hz and 80-3kHz channels.  By turning off the MF channel, some special decoding features are also disabled -- so that the decoder acts more focused towards sheen removal.  Also, the threshold (--thresh=-15.75) will likely have to be very different from the norrmal -15.00 to -16.00 values.

The additional EQ will probably look like this:  "treble 3 3k 0.707q treble 3 9k 0.707q", but this suggestion just might be a little too strong or weak -- not sure.

Another note -- do you see where I convert the input to floating point and 32bits -- that is the optimum format for the decoder, and also the decoder has maximum performance between 88.2kHz and 124kHz.  The perf falls off a little at 192kHz, and I suggest that after the rework, there is never a real benefit to running at 192kHz unless there is a specific need to support 192kHz on the output.

Nowadays, the processing at 44.1kHz is pretty good, but it is almost always beneficial to bump it up to 48k before the decoder -- this is because the internal DSP operations work at exactly the sample rate frequency, and there is little room for distortion removal and filtering when running at 44.1kHz.  44.1kHz is more of a distribution rate than an ideal production rate anyway.   The biggest benefit to running at 48k instead of 96k is that the decoder runs much faster, but the quality should be barely noticeably better in the 88.2kHz through 124kHz ranges.

There will be an upcoming new release (probably end of day today -- 14May2018, or early tomorrow), and it runs about 10-20% faster and have a few very minor quality improvements.

John
Title: Slightly faster (10-20%), slightly better decoder.
Post by: jsdyson on 2018-05-15 15:27:07
The new version is available here and on the repository site below.  I haven't updated all of the demos yet, but there are not major changes in sound quality.  This new version does some calculations now in single precision float rather than double precision.  (The reason for previously using double precision was because of an arcane behavior of the compiler that I am using.  I reworked the calculation, which both allowed using single precision -- which is more than enough, and also twice as many operations can be done because the attack/decay can be calculated for both (L+R) channels at once rather than one at a time.

The slight improvement in quality results because of quicker calculation, so better time precision can be provided by doing the longer time decay calculations more time-accurately.  Also, the longer term decay can calculate further into diminishing returns providing a better decay match in possible situations that I have not yet encountered.   Also, the --ai=high vs. --ai=max are more different than before.   --ai=max removes more of the encoder 'hash' at the slight expense of certain dynamics at HF.  --ai=high continues to be the best, safest aggressive quality, while --ai=med allows slightly faster encoding, and has a very slightly simpler decoding calculation (marginally closer to a normal decoder behavior.)  Using both  --ai=none and --raw=8192 gives a barebones decode which will seem brighter (the extra brightness is believe it or not,  distortion), but would be essentially the results of doing a traditional decoding in software.  I only use those modes for development comparisons, but thought that it might be interesting to show what all of the fast gain control can sound like, and shows one reason why SW based compressor/expanders sometimes sound different than a HW equivalent.   Naturally, a bare bones software design will produce much more distortion than HW because of the effects of sampling.  This is one case where higher-than-44.1k sample rates really DO make a difference, but it is purely because of the nonlinearity (producing beyond 20kHz signal components), and sampling.  The decoder is designed to fully mitigate those uglies.

More detail:  because of how the compiler generated code for the newer and older CPUS, I did the math for attack/decay in a math vector size 4 with a data type of double.  This fit perfectly the capability of the CPU and the number of audio bands on each of the L+R channels.  This also matched well when compiling for less capable CPUs, where they supported a vector size of 4, but only in single precision.  SO, the code was designed to use the different data type for the older CPUs -- but with absolutely no difference in quality.
I decided to do both the L+R calculations at the same time because newer CPUs can do 8 single precision operations at the same time, so now both L and R attack/decay are calculated at the same time.   The effect on the older CPUs should also have etiher a slight improvement or break even because the locality of reference is a bit better.

Anyway -- -probably more details than what you really want to know.

repo: https://spaces.hightail.com/space/tjUm4ywtDR

John
Title: Found an attack time error -- fixed
Post by: jsdyson on 2018-05-16 20:33:24
Someone on another forum turned me on to 'Howard Jones'.  He was curious if the specific recording was DolbyA encoded.  Well, I found that it was, but there was something wrong with the decoding.  After listening to a few more pieces, I found the problem, and it was related to the anti-intermod parts of the code (but not the anti-intermod code per-se.)  So, I pulled back and reverted the code to an earlier version, and the attack time problem (however small) is now gone.   The newer code was just a little too aggressive in trying to avoid unnecessary intermod, but actually the code now enabled is better all around.

There are so many variables in the code -- especially since this decode goes FAR BEYOND a basic decoder in trying to extract every bit of quality out of the music.  Anyway -- here is a new one -- and this release IS justified.
Title: New version -- not worth downloading if you got the previous version
Post by: jsdyson on 2018-05-18 00:33:07
This version has some clean-ups -- per some commentrary from another forum's leader.  Made some documentation a little more accurate/clear.  Also, the code rejects mono files now (before, it just went nuts.)  Two new switches added (for windows convienience), where you can specify --input=filename or --output=filename instead of using the '<' and/or '>'.   This is meant for windows sensibilities, not really any underlying change.

For the first time we ran a harmonic distortion analysis -- and it is unmeasurable on the normal settings of the Audacity spectrum analyzer.  A real DolbyA does show some distortion. 

The major new thing learned is that the program drops about 3k samples during the conversion.   This is true for every version, and hasn't been fixed yet.  Just letting you know, and it will be fixed soon!!!

Also, MAKE SURE YOU USE THE --outgain=-3 switch on EVERY version.  Because of precise emulation of a real DolbyA in certain regards, it will clip if you supply 0dB input, and the --outgain=-3 keeps that from happening.  Actually, it has 0.33dB excess gain, so you can fix it in the way that you want.

Again, don't bother downloading if you have the 16may version (unless you like to always keep up with the latest-latest :-).)

John
Title: Really 'cool' demo song.
Post by: jsdyson on 2018-05-18 01:52:05
Minor off topic, not being actually technical -- but is related to the current subject...

It sometimes isn't easy to find a really good demo of how the decoder maintains the stereo (and unflattens the depth), and actually sounds REALLY natural.  This demo has no 'love' put on it -- that is no, EQ.  This is raw decode from my original DolbyA encoded copy.  REALLY NEAT!!!  It doesn't have much artificiality, so you can really hear of something bogus is happening.   The only thing that I did was to normalize it to -0.25dB.   I don't have space for the original, but will provide if really asked for.

The file:  Nat-LOVE-DAdecode.mp3

Repository:  https://spaces.hightail.com/space/tjUm4ywtDR

John
Title: Decoder update -- moderately helpful
Post by: jsdyson on 2018-05-18 10:42:34
This update gives one audio quality improvement, one audio quality fix, and a usage feature.

The audio quality improvement is for --ai=max mode only, where the intermod is better controlled yet (smoother sound without loss of detail.)  There is an additional slowdown, but the extra calculation is worth it, if needed or desired.

The audio quality fix is for a calculation error on the 3k-9k and 9k-20k ranges.  You might notice slightly crisper, more detailed HF sound without an increase in intermod.  In fact, the intermod might be slightly decreased with better clarity.  I made a math error on one of the 'careful' attack time calculations (the attack/decay time calculations last for a total of about seven hundred lines in two seperate sections -- not simple -- VERY careful shaping.)  The error was the truncation of a variable name and a mistaken use of a zero initialized variable instead of a specially filtered signal level.  Because it was utilized in a 'max' calculation and was part of shaping the attack time, it wasn't fatal, just not good.  It was an obvious transcription error from a previous version of the calculation.

The usage feature is the ability to display the gains for the individual channels instead of the average gains.  So, there is one extra line per 1second display if you specify '--info=10'.  It isn't fully documented yet, but this is a heads-up that the feature exists and is intended to be permanent.
(I don't intend on working on this today -- so this will be the last version released today -- unless I get a complaint about a major failing.)
Title: Helpful usage note...
Post by: jsdyson on 2018-05-18 17:33:43
I have been chasing down some of the last of the edginess like distortion, and I found some filter issues which I am addressing.  However, a quick RIGHT NOW workaround for those who have copies of the decoder that they are using is fairly easy.

Through studying the filters and the skirts of the filters, I found that the ideal sample rate for quality is between 56k and 64k samples per second.  Those rates give the advantage of being high enough above the minimum necessary sample rate to help to avoid any noticeable aliasing given practical filters (if someting leaks beyond 20kHz, there is enough room to filter it out.)  Additionally, as the sample rate goes up, then fixed size filters become less effective at lower frequencies.  So -- running at 96k does work, but produces lower quality than running at, say 64k.

The filters in the processor currently only partially support 'wide' mode where the 21.5kHz limitation is not enforced, so running at a very high sample rate has no benefit at all given the structure of the decoder (the filters being used/etc.)  The fact that out of band filtering is really necessary for best audio quality (out of band filtering internally -- lots of nonlinear stuff going on), that there is actually a real disadvantage to allowing more than a flat 21.5kHz bandwidth.

From now on -- to infinity, I am going to be running/testing predominently at 56k-64k (still haven't decided), but that does NOT limit the sample rates being used for the data source and data sink (I ALWAYS use sox for the conversions.)  The decoder is planned to always work at 44.1k through 192k, it is just that it might not be as good as when running at 64k, for example.  Actually, there is a mathematical reason for choosing around 64k as the ideal rate  -- it happens to be approx ideal given a lot of arcane DSP factors.

John
Title: New version -- just quality improvements
Post by: jsdyson on 2018-05-24 03:20:07
New version of the DolbyA compatible decoder.  The best yet.

da-win-23may2018B.zip
location: https://spaces.hightail.com/space/tjUm4ywtDR

You can check the location above -- sometimes I forget to post about updates.  More and more people are realizing that they have DolbyA encoded material and didnt know it!!
John
Title: Here is an example justificaition of the usefulness of the DolbyA compatible dcd
Post by: jsdyson on 2018-05-26 04:48:31
I put up a demo of two (maybe eventually 3) songs that I must take down very soon.  The demo is a result of the decoder processing and afew other steps.  I plan to document everything some time soon.

If you listen and are a fan of the group, it WILL be a religioius experience.  If you are not a fan, then you JUST MIGHT become one.
This sounds different than I have EVER heard from the group, and the material is a commonly available street copy that was processed by the software that I have been talking about.

Just listen -- it is worth 10 minutes:
https://spaces.hightail.com/space/UntM4LCdcm

Give me feedback -- I'd really like to see the face of each person who listens -- and yes -- it is very safe :-).
Title: Major freq response fix (it lost too much HF before -- antialias too aggressive)
Post by: jsdyson on 2018-05-26 10:05:15
This is probably the last release for a while unless there will need to be a major bugfix.  Also, it is probably okay now to use --ai=max all of the time.  The anti-alias fix also improved the freq response for --ai=max.   There was a comedy of  freq response errors resulting from some FIR filters being too short -- silly mistake.

No change to basic design, just re-jiggering some filters.

Basic design is now solid, but if I find a new anti-alias or quality improvement I will certainly persue it.

Refer to previous posting (probably have 24 more hours) for incredibly good examples about the quality of this decoder.  Esp this version.

This is MANDATORY if you want/need the best quality.

The unexciter is also now included -- it is very slow, but is part of what made the ABBA voices sound so human instead of that horrid Aphex Exciter sound.   There are no real docs, but it can be used once or multiple times.  The switch is --dr=0.156, and that ist the best default value.  If you need the very best quality and almost total removal of that cringing sound - use the unexciter sequentially with --dr=0.120, --dr=0.078 and then --dr=0.0135.  (Values determined experimentally.)  Just pipe multiple instances of the unexciter together.  The sound (esp voices) is incredibly improved.  Unexciter name prefix is 'unex', and just choose the best for your computer.
Title: Is there any interest in a DBX I only decoder only?
Post by: jsdyson on 2018-05-28 20:44:23
I just had to write a DBX I decoder (some material was DolbyA encoded, then it was DBX I encoded after that -- fun time!!!)  By the way, the material is ABBA studio collection TCHS or something like that.  Anyway -- the DBX I encoder works pretty well and has almost all aliasing suppressed, so should sound as good as a real DBX I.  To me, it sounds perfect and after the intermod fixes, everything is smooth sounding.

If there is interest, I'll put something together -- maybe even add it to the DolbyA decoder.  If there is little/no interest, then so be it!!!  I have placed it in my 'uncompressor' package, but that is a behemoth -- so putting into the DolbyA decoder would be very simple -- but it is something that I don't want to do unless someone might want it.

John
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: punkrockdude on 2018-05-28 22:47:12
Have you posted about this decoder over at for example Gearslutz and TapeOp where a lot of studio engineers hang out?
Title: Re: Have a working 'expander' based on DolbyA (not same design) -- works well.
Post by: jsdyson on 2018-05-28 23:02:55
Have you posted about this decoder over at for example Gearslutz and TapeOp where a lot of studio engineers hang out?
Oddly enough I just started trying to set up an account with Gearslutz -- just this last second when I noticed the notification about your message.  I have been on the Hoffman forums a lot also.

Thanks for the heads up -- I'll do it.

Mostly, my interest is to help the listening people who just happen to be technical enough to use the software.  My recording engineer friend is focusing on the pro side.  I don' t know if he knows about the DBX I decoder yet.

John
Title: Have the DBX Type I decoder working almost perfectly.
Post by: jsdyson on 2018-05-30 02:52:26
This is the first time that I have a 'perfect' copy of ABBA (really.)   This is the complete studio edition that happens to be both DolbyA and DBX I encoded.  AFter some analysis - it is actually a good idea to use both.  This is really a demo of BOTH decoders...  Before the results were amazing, this is even more amazing because I know the internals of the DBX 1 compatible decoder are much more solid.

Listen -- if you hear an flaws attributable to the decoders -- let me know.

The best way to describe is silky smooth.  I'll be distributing the DBX I decoder sometime soon -- might merge with the DolbyA compatible decoder!!!

Attached is typical of the examples on the repo.  This is NOT necessarily the best example, but is a reasonably good one.   This is a case where the mp3 is noticeably different from the original -- the original is THAT GOOD.  I made the attachment at a very high quality level mp3.

REMEMBER THIS IS ABBA -- and 90% of the songs sound this good!!!  I'll be disclosing the method and source materials in as much detail as possible (I bought mine years ago, but I'll try to point your in the right direction for the music source material):

PLEASE ENJOY -- repository: https://spaces.hightail.com/space/UntM4LCdcm
Title: My decoding results are 'ALL that', but I was partially in error about a comment
Post by: jsdyson on 2018-06-04 03:27:22
My representation about DolbyA was/is 100% correct -- it is nearly perfect if not better than normal unit for decoding purposes.  However, the DBXI is not up to snuff.  The decoding results are great, but not due to the material being DBX1 compliant.

They were doing some kind of DBX style compression on the material, and in general from the ABBAstudio (very compressed copy) I am getting in the range of 12-14.5dB peak-to-RMS, with the expanding technology (which will be released soon), I am getting between 14-18dB peak-to-RMS, and with the other ABBA discography release I am getting approx the same peak-to-rms as my decoding.

The resulting peak-to-rms that I am getting is similar to, but not quite as good as the real analog abba disks, but the general sound quality appears to be much better than anything else -- that is, a good peak-to-rms balance, good sounding natural dynamics and none of the extreme strident sound that I have gotten from my ABBA studio copy without it being cleaned up.

I hope to have demos available soon, and software should be available quite soon.  The order of complexity is less than the DolbyA for the DBX expander, and I'll provide some cookbook usage information.  It actually takes me a little time how to use the software also -- just because I write the software, doesn't mean that I know or fully understand how to use it (kind of like a musical instrument builder -- except mine MUST NOT add too much of its own character.)

My goal is to revert the sound back to the original!!!

John
Title: Since the DolbyA is nearly perfect -- working on another expander
Post by: jsdyson on 2018-06-05 14:47:54
The DolbyA project is pretty much finished.  The final version has not been released publically, but individuals do have the final pre-release working copy.  Still doing the final cleanup, but the project WRT innovation is complete.

I went on this DBX Type I distraction -- but a medical problem distracting me helped to cause a mistake -- and the expander was really only a 1:1.4 instead of 1:2.  This caused some confusion, but something new was learned (by me) -- that is that the RMS detector structure implied by the THATcorp/DBX design is fantastic -- better than any of my other schemes so far.   SO -- instead of producing just an analog to a supercharged DBX style 1:N expander, I decided to consider bolting on the DBX detector onto my multiband-GP expander (the uncompressor or the restoration processor.)  THe GP expander has attributes which make it NOT create typical expander artifacts and most of those advantages do not result from my older detector design.  So, adding the DBX style detector onto the GP expander infrastructure is potentially useful.

The results of bolting the two kinds of expander together have been breathtaking - with a very good increase in crest factor, but still a sane peak-RMS level.  So -- the short term dynamics are very good without causing excessive dynamic range.  One bugaboo with making an expander that helps the dynamics is that the longer term dynamics might become excessive.  This new scheme seems to give the best of both worlds.

I am considering reorganizing the GP expander so that it is not part of the compressor/expander complex, but will be a seperate program.  This will make maintenance much easier.

The biggest change in the RMS detector scheme has been a change from the fixed rate nature associated with typical DBX schemes -- they tend not to be very adaptive, where most of the 'adaptive' nature might be a dynamic/nonlinear capacitor scheme -- not really doing very much in comparison to the very adaptive infrastructure of the GP expander.  So -- I have adjusted the RMS detector design to by highly dynamic so that the RMS averaging interval is based upon the material.  So -- both the attack characteristics and the driver for the decay characteristics are dynamic based upon the THATcorp style measurement scheme.  The older method was the a dynamic extension of the more obvious long term average of the signal squared.   The THATcorp scheme is similar to that, but is based upon a different measurement domain of the signal -- and seems to be significantly better.

As I wrote before, the crest factor on the output seems to better than the DBX expander scheme alone with a much smaller expansion ratio -- thereby giving the natural sound with much less likelihood of expansion artifacts (which are already suppressed by other methods.)

I do intend on releasing the first production worthy of the DolbyA compatible decoder in the next few days.  RIght now, a big part of my expander test suite is the day-to-day utilization of the DA decoder.

John
Title: New DolbyA compatible decoder release (much improved)
Post by: jsdyson on 2018-06-13 12:25:20
This new release has a significant improvement -- the front end has been redesigned to be very compliant behavior.  Some of the feedback architecture has been implemented so that the input filtering can now dynamically cancel some of the 'misbehaviors' of the original HW design.  This has allowed the removal of the anti-hash code, which the removal then provides a nicer spectral output -- less dynamic filtering of the HF region.  (Actually, there was no explicit dynamic filtering, but the effect was that of dynamic filtering.)  Now, the spectrum remains stable and complete (in the way that decoding requires) at all times.  Some of the feedback behavior is impossible (really -- I mean, would drive a developer crazy to do properly), but the parts that can significantly impact the sound quality are fully implemented.

You'll notice a more intense natural sound output and more complete low-end also (the low end was slightly attenuated by the previously needed filter structure.)  This new version mimicks more (probably close to all) of the behaviors associated with the feedback design.  The new general structure has not changed, but the amount of feedback possible has been implemented to emulate some of the 2nd order behaviors of the original design (previous workarounds had been very frustrating to me.)

The caveat is that I do NOT suggest running the decoder outside of the range of 48k samples and 124k samples per second, with the ideal still approx 64k samples/second (perhaps slightly higher now.)   Longer filters do kick in to better support 176-192k samples per second, but the decoder does become quite slow at those high sample rates.

The general audio quality is vastly improved.   It was already pretty amazing for a computer implementation.

John
Title: Warning -- decoder only works at 64k or less
Post by: jsdyson on 2018-06-14 07:23:17
Somehow I messed up the filters -- works great at lower rates.  Should have been perfect at 96k, but not so right now.  Fix forthcoming.  The attached version is better at higher rates -- I screwed up on a last modification.  Will be fixed better soon. (darned timing/delay issues -- not trivial sometimes.)   Have to get all filters all synchronized, and I violated one of my protocols.

The best is still at 64k, but faster should at least work without losing all of the highs.  The gain numbers are correct -- it is just that the rebuilt signal is all messed up because of my timing foobar.

John
Title: Complex timing issues got me, and I was a bit sloppy.
Post by: jsdyson on 2018-06-15 02:32:12
Made a last minute change, and I didn't think about how sensitive the timing is.  It is kind of complicated, but the standard DolbyA style filters must have EXACTLY the characteristics of analog filters, but are digital emulations.  Digital emulations add several sample delays typically, and I didn't cover for them correctly (I broke the algorithm.)  So, now the filters used to produce the bands are the standard analog to IIR conversion, but with a sample skew built in and also a necessary all pass filter analog to the 80Hz fitlter was added to keep the phases all matched up -- so with both the sample skew and the all pass filters, the timing of the filters matches just as well as a normal analog filter (for between 44.1k and 192kHz.)  I didn't bother compensating beyond that.

Before, I didn't notice a lack of proper removal of the 80Hz from the other bands -- it wasnt 100% cancelled, but now it is!!!  Much less apparent distortion.  The older front end filters did have full cancellation, but also didn't as perfectly track a real DolbyA.

I truly suspect that I might have the correct structure to implement a feedback DolbyA compatible.  I am only interested in that because I am going to try to do an SR soon.  These last strange changes to the DolbyA compatible decoder were only meant to evnetually improve the decoder.  My first attempts were in error, and I apologize for any wasted time!!!

John
Title: Finally -- got something that sounds very similar to a DolbyA 360.
Post by: jsdyson on 2018-06-30 20:43:22
Before -- the previous versions could decode pretty well, but needed some EQ from time to time.  That had always been that real freq balance problem that I had to work on - and now have been successful fixing the 'sound'.  There were a few mistakes in my filter conversions, more needed tuning due to the sampled IIR filters vs. real analog filters, and a lot of improvements have been made -- including true intermod distortion cancellation along with simplified/better distortion removing/avoiding filters.  Also, --wide mode does support out to 80kHz at 192k sample rate for people who'd perfer wider bandwidth rather than maximum removal of distortion products.
It sounds close to the consistently same between 48k and 96k, and pretty good at 192k.  44.1k is a little metallic sounding because the filters cannot remove the distortion products at HF before aliasing happens.  If you are willing to use an odd sample rate, between 64k and 72k are theoretically the best (wide enough available bandwidth to capture all of the relevant distortion products and remove them -- and the decoding speed is faster at lower sample rates.)  96k is close to ideal, but at that speed the distortion filter skirts widen a bit. They start widening enough that above 97kHz, there is a different set of distortion filters enabled so that the best quality (at expense of a bigger slowdown) at 176.4 and 192.

I also found that the distortion products are more pronounced at low frequencies than outside of each band on the high side -- so I changed the filters to only remove the low side of the distortion products, and actually the distortion and noise cancellation works better when the filters are wide-open up to 21.5kHz.

The sound is seems 90% similar to a 360/cat22, and the threshold setting is pretty easy (the threshold value is approx 1dB less than the Dolby tone.)
The best way to describe the sound is:  'sweet' while earlier versions were a little 'harder'.   Amazingly, the spectrum of the output is incredibly similar to a DolbyA given the same input.  I was really surprised as to how similar the spectrum measurements had been on test material.  The spectographs are also similar, except the distortion products are more pronounced on the real DolbyA (constant tones produce a more intense and larger number of distortion spectral components.)
I am also going to post this on my repo site, but won't do so until after tomorrow, so this is the only place to get the latest version of the decoder for now.
Title: Here is the difference in spectrum of my DA decoder vs a real DolbyA
Post by: jsdyson on 2018-06-30 21:22:56
Here I am providing both a spectrum output of my DolbyA compatible vs. a real DolbyA 360.

Mine is the top, and the real DolbyA is the bottom.  Refer to the attacked PNG screenshot for the pictures being talked about.

When comparing the spectrums, please not that the signal normalization on the real DolbyA had some troubles because of the errant peaks in the output of the real DolbyA.  If you look at the spectrum of the peaks, that spectrum of those glitches (probably intermod in the real DolbyA) are at the same 741Hz that the massive peak is on the spectrum of the real DolbyA.  The compatible decoder does have some peaks, but not nearly as often or not mostly as strong as the real DolbyA.  So, the measured energy of the real DolbyA 741Hz peaks is measurably greater -- there might even be peaks at different freqs that we cannot see adding even more to the 741Hz energy level and the energy level of some of the other slightly larger peaks.
Arguing against the idea that compatible Dolbya is just slower at dealing with peaks -- some of the energy levels at HF are the same or just slightly higher in one or two other places at HF -- again this is splitting hairs but trying to be as critical as possible.

So, if you ignore the ampltude of the LF peak, plus the fact that my decoder only filters the LF starting at 10Hz (0.00022 IIR feedback at 44100Hz -- all attack/decay stuff is normalized to 44.1kHz in my code.)  It might be more compatible for the gain filters or the input filters to remove more LF -- but I need to study the matter.  So there is the liklihood that there is more LF material in the signal than the real DolbyA can process.  Other material with less LF doesn't show the LF peak.  (This was pipe organ stuff.)  I as asked by my collaborator to provide better LF perf than the real DolbyA, and that is why I am just using several LF 1pole rolloffs at 4Hz and perhaps one pole at 10Hz also.

So, ignoring the 44Hz LF peak, and the slightly larger 741Hz glitch peak (and a few other extreme peak differences), the real DolbyA and the compatible decoder are very, very close.  You might also think that there is HF rolloff in the compatible decoder -- but all listening measurements have shown that not to be true.  The acutal listening sounds like the compatible has more HF (slightly), and might be because of numerous factors (might be because the attack/decay are slightly faster/less intermod in the compatible.)   However, I will be researching this.  There is in NO WAY a lack of HF in the compatible decoder.
Title: Minor update to the best ever SW dolbyA compatible decoder.
Post by: jsdyson on 2018-07-01 01:31:54
If you remember the slightly low response around 10kHz in the png file (ends up being about 2dB low at 15kHz.)  I corrected the filter matrix so that it is more accurate.  It is almost ruler flat now between 9k and 21kHz...  That was just a screw-up.  This fix allowed me to remove the 3dB boost at 9kHz for the Dionne Warwick demo song.   It is already near perfect, and getting better.  This is a mighty detailed and somewhat tricky piece of software.

I am glad that the previous version has not be downloaded yet.  This one really corrects a minor nit, but needs to be corrected.
Title: New version of decoder -- just verified the cause of the 44Hz peak - no rolloff
Post by: jsdyson on 2018-07-01 04:29:27
I did some experiments and did a tasteful mild rolloff at 40Hz (typical needed because of severe head-bumps on high speed tape decks).  The peak was practically obliterated and ended up being only about 1dB greater than the real DolbyA.  I am also carefully working on the slight HF issue just above 10kHz.  I think that I have a solution.

The result is a slightly brighter result -- but I still have a day of testing before making the new minor fix (only like 2 lines of code) available.
Title: Yet another update -- better/faster decay time yet accurate
Post by: jsdyson on 2018-07-01 22:02:01
I just found a problem with a 2nd order attack/decay calculation -- which means that in certain kinds of percussive music, that there would be some lingering compression.  For a single decode, the effect wasn't so bad, but for multiple passes of encode (with a real DolbyA) and decode with the earlier version of the processor, there would be an accumulating amount of very slight compression.

This version fixes that problem, and the second order calculation of the decay time is now correct.  All known bugs as of this minute are fixed.  I have some prospective improvements coming, including the input filter that keeps the lowest freqs from being over enahanced, but if that is a problem, just add a 1st order high pass filter at about 40Hz to help, maybe a 2nd order filter at 35Hz might be better -- haven't experimented very much.
Also, the gain control filters have had the nonlinear portion disabled -- so it still has the linear filters meant to limit the sidebands, but the result could be slightly better some time in the future.  ALMOST no-one would got to the lengths that I have gone to so that the quality is as good as it is.   This is very tedious programming -- I have done DSP code in the past, but this is about the most difficult (conversion of complex feedback systems to feedfoward) that I have ever done.

If after a couple of releases back to back, this didn't make a substantial improvement, I would not distribute this.  But his is a very substantial improvement for industrial applications.
Title: Warning about using SOX with the decoder -- but new version also coming soon.
Post by: jsdyson on 2018-07-05 12:55:05
I am getting into the tweaks and tuning towards perfection.  The code is stable now, expecting to make it a real 'release' in a few weeks.

I  have a warning about using SOX...  When you  have a very hot .flac file on input, and you use the technique of using a gain of -6dB before the decoder, and then use '--ingain=6' in the decoder command line -- it is actually better to use the '-v 0.5' switch on SOX instead of 'gain -6.'  I was finding that 'gain -6' doesn't avoid potential mild clipping on input for very hot .flac files.

A new prerelease version coming within a short time!!!

John
Title: New version - very important minor improvements.
Post by: jsdyson on 2018-07-05 14:47:50
his is a really, really good update. I am getting into the tweaks and tuning towards perfection. The code is stable now, expecting to make it a real 'release' in a few weeks. It really needs lots of testing and usage before blessing it. Also, if this idea of errant 'DolbyA' encoding becomes better accepted, I plan to do what is necessary to help transfer some technology to those who can better implement a GUI or plug-in application. This tool is not trivial, especially difficult to make it so very accurate in SW.

Here are the two significant changes:
1) re-instituting the attack prefilter. The attack times are the same, but now the distortion sidebands are minimized before being created, rather than just being postfiltered. This would be most interesting to the 44.1k and 192k users, but the improvement to 48k-96k should be noticeable in this  'easy' sample rate range. I find my VERY difficult material to be improved.
2) better inter-channel timing for attack/decay. Before this correction, the threshold had to be set very accurately to avoid a very minimal phasing on strong 's' sounds. It was noticeable on Scarborough Fair and the Pink Panter theme. Now, changing the threshold over a fairly wide range -- even though incorrect -- should not have as strong a phasing type sound. (It really wasn't 'phasing' but an obvious change in freq resp.)  I don't think that it is quite perfect, but might either be in the material or even in the original encoding.

As recently usual, the location of the decoder is below.  Also, the decoder is attached.  There are updated examples in the online repository also.  Esp listen to how the NR works on Scarborough Fair -- it doesn't get rid of all of the noise (but most/all of the HF hiss), but reclaims the depth and slightly undistorts the bass.  On the Bread example, it is so good as to be emtional.  There are a couple of more -- the Pink Panther actually shows that I might not have gotten the threshold perfect (the transient sounds at the beginning aren't quite 100% stable, but are darned good considering the complexity of the HW to SW conversion.)  When I am finally finished with this, it is going to be perfect -- but is now PROBABLY closer to perfect than a real HW unit.

https://spaces.hightail.com/space/tjUm4ywtDR
Title: Spectral comparison on the DolbyA compatible decoder, and minor update coming
Post by: jsdyson on 2018-07-06 20:26:27
 just ran a spectrum (one of the easier ways to compare a real DolbyA with the decoder SW), and the results are pretty darned good. The sound quality is almost identical, but with a slight amount of extra clarity(probably partially due to an absolute minimum intermodulation.) These spectrum and waveform results are representative of the next pre-release which is coming out later this afternoon will be dated 06Jul2018. The earlier pre-release which is commonly available has a bit more bass, and very slightly muted very HF -- but is still very close to accurate (this new one is more accurate yet.) Even the waveforms aren't too far off considering that this is a full reimplementation even with a totally different architecture than anything else.

Also, the spectograms look similar -- except for noticeably less obvious distortion and a little less hash. Any of the indications of actual signal are reproduced practically the same as a real unit. However, the hash associated with the real signal is significantly less obvious. That lack of hash is NOT a loss of detail in the sound (once one actually compares the sound qualities.) I didn't include the spectograms -- but listening is a better indicator. I simply like objective verification sometimes :).

I thought that this comparison might be interesting -- and the new version (as shown by the date which will be today on the repository -- maybe tomorrow if emergency) will be uploaded as soon as I fully complete testing.
The .png file is 'Screenshot from 2018-07-06 15-02-32.png' on the repository in previous postings.  It is also included here.

Fewer and fewer nits (there is also a fix for certain 24bit signed integer files that don't follow the recommended
.wav format rules.) These fixes were 'strongly' requested by my recording engineer collaborator, and he is really
doing a great service by demanding high standards for the project.

I'll post an update when the new version is available. This new version is a good thing, but not really as critical as almost any previous update.  The spectrum comparison is probably more interesting/useful than the update unless you really need closer and closer perfection :-).
Title: Last experimental release of the decoder
Post by: jsdyson on 2018-07-07 04:30:10
Future versions will be more formally controlled... Gonna try to keep it free for consumers, but this has been blessed at a first level beyond me now.  Will probably not be posting minor changes as often -- but if there are killer bugs, I'll probably fix them.  This (and many of the previous several versions) are close enough for consumers, but this is better yet.  It has more HF intensity than I like, but it matches better the real units.  The LF has been reigned in to be not just better than 1dB accuracy, but almost immeasurably different!!!  Much of the MF is also pretty close, but the HF seems a bit hot (I mean like a dB -- not huge amounts.)  The distortion (intermod) is as low as I can get it without doing some major heroics -- there are still some very aggressive techniques that aren't published anywhere.

Enjoy.  Just barely got it under the wire today -- personal matter (fell asleep :-)).

John
Title: Usage note on the free version of the DolbyA compatible decoder
Post by: jsdyson on 2018-07-07 15:22:47
There was a minor internal tuning error in the final pre-release, and there really is a 100% workaround.

The error was manifested as a continued almost 'swirl' in the sound caused by fast percussion (cymbals) or very sibilent voices.  The fix is 100% correct, and is being permantently enabled in future versions.  I had thought that I had mitigated by other means (and it did help), but not good enough.  I just noticed the problem re-appearing on some Petula Clark material, and could hear it also on some orchestral stuff (Mancini.)

In the command line, add this switch:  "--ai=med".

Don't use the quotes of course, but they will probably work on Unix style shells, just do the raw -ai=med and add it to the command line.  The position isn't critical as long as the command is associated with the decoder.

You'll notice that the dynamic high end (somewhat reminding of a phasing sound, but not nearly as severe) not occuring when using the switch.

Sorry about the trouble.  Really trying to make a good high quality tool available.

John
Title: The 06Jul2018 version of the decoder will the last for a while.
Post by: jsdyson on 2018-07-09 06:54:48
I have been getting a LOT of pressure to consider commercializing the decoder, and this is a good (temporary) cutoff point for the free version.  It has just progressed to a true release quality version with several sound quality improvements being very helpful for a real commerical or archival application.  The free version as of now (which might be updated in the future) is a few (important) notches away from the release version -- but still sounds super good.  I don't have the strength (really, health issues) to try to help people understand the benefit and use of the tool, do the development & releases, and try to add pro usability at the same time.  I have too much work to do, and I must de-emphasize the gratis offering for awhile (until I can do 'right' by my collaborator and those commercial users who are showing interest.)

The consumer give-away side has had lukewarm acceptance even though the decoder is a way for consumers to avoid repurchases in the future if/when corrected releases of pre 1990 material are made.  Some of the truly cheap labels might even be able to afford decoding, because the historical problem has been the conversion to/from digital and the amount of realtime to run an actual DolbyA unit.  The professional version will address those time/cost issues.

Up until now, there hasn't been a really good quality (yet this one is still not easy enough to use & does need usability improvements) DA decoder in the digital domain.  There is at least one other SW decoder, but it doesn't really sound like a DolbyA.  Now with the 'very interesting' interest on the commercial side (I am not handling the commercial discussions, I am just the nerd who is the author/owner, and working to make my collaborator happy and benefit from the tool also) -- I need to focus on the improvements that will move the decoder from being a 'problem solver' to a 'go to tool.'   Historical archives and pro recording engineers have been showing interest, and with the most recent improvements (the release version), there truly is nothing else out that decodes the material as well (clean, clear, accurate and really beautiful sound.)

I am leaving the examples up for a little while longer, and now they use the latest (true release) version of the decoder.  They are truly beautiful, and quite typical of (about 1/2 of) my digital copies of between 1967 and 1990 release material.  The decoder will be available indefinitely -- either on a few forums like this or the Hightail site.  If someone has problems finding the decoder somewhere, just ask me, and I'll make the most recent free version available.

Maybe the frustrated recording 'mastering' engineers who have been told 'just EQ it' will be able to follow through with their real intent of producing quality recordings again!!!   A free update wil very likely happen some time in the future, but the distraction of doing Windows (pre)releases instead of focusing on the actual development work has taken important time away.  The slower and more focused release rate will help with my time management.

If anyone really has a DA decoding quality problem and the current free decoder won't do it correctly -- then let me know, and I'll try to help.  (I don't think that the current free decoder will be inadequate for almost any consumer need -- but never know?!!?!)

PS -- I do apologize for the early attempts at making a decoder work -- it has been a long, frustrating learning curve with several kinds of new algorithms implementing it.   There is a lot of new techniques in the decoder, with a full/complete/dead-on-accurate solution for the feedback to feedforward attack/decay calculations (among other things like that.)

Thanks.
John
Title: Simple example of cleanness & function of compatible decoder.
Post by: jsdyson on 2018-07-11 13:09:54
I was just running some tests of the development copy of the decoder.   Beating on it until it is perfect!!!  (I still plan to do free decoder updates form time to time, perhaps with minor limitations WRT professional version (slower, perhaps quality deviations, processing options, etc) -- I'm just very busy right now.)

Attached are two short demo files -- one is from a CD distributed reference copy, and the other 'bestever' version is decoded by the DolbyA compatible decoder.  The CD version is about the best version of the recording that I have been able to find, except for the DolbyA version that I used for input to the decoder (which is the bestever version.)

So -- I have been doing test comparisons that are absolutely necessary to prove (to me) where/if the decoder has bugs, or needs help/fixes.  Here, I have an example of one of my favorite groups -- ABBA.  They are my favorite from two standpoints, I like their music, and their recordings are good material to test my code.  They have done things to their audio which seems to make them much more difficult to correctly process than 'normal' sources.  It is easy to make ABBA stuff sound bad - so it often requires great care and very good HW/SW to keep from producing more apparent distortion than the source material.

Here, I have two short examples of 'SuperTrouper' -- a relative sound quality basket case, especially when processed by inferior software and HW.   I have a copy that had been decoded from a DolbyA source, and a 2nd copy directly from a Polar CD (the gold standard for ABBA recordings.)   The spectrum of the copies are very similar, except for a few minor oddities, which I modified on purpose to maximize the sound quality for my listening enjoyment (avoid creation of unpleasant audible HF sound distortions.)  Some ABBA recordings have a spectograph noise band at 19.2kHz in the M channel of M+S (odd, huh?), (not sure the cause, maybe something to do with old-fashioned serial data transmission), and I have removed the noise band in the mid (M) channel and an associated distortion/noise band at 1/2 of the frequency.  You can see that noise band in the 'reference' recording, and appears to be on every copy of SuperTrouper that I can find.   If you do a spectral analysis of my decoded verison of the entire song, you might notice 10dB notches at 9.6kHz and 19.2kHz -- and those notches were done on purpose and are totally inaudible.  Many renditions of ABBA recordings have a rough HF sound in them -- and much of it is due to the 19.2kHz noise band interacting with the audio and processing equipment.  If I would have let the 19.2kHz and the associated 9.6kHz noise band through my decoder, it wouldn't have significantly affected the sound for comparison purposes, but I did this recording for myself, and so I wanted the best quality no matter what.

If you listen carefully to the two examples (one is 30 sec of the Polar reference copy, and the other is processed from my DolbyA source material), you'll hear a lot less distortion (intermod) on the decoder 'bestever' copy.  Careful comparisons of the entire song show more distinct separation of the song components (voices/etc), mostly due to the vastly superior intermod characteristics of the compatible decoder.

The Polar version does have a slightly greater amount of compression (pk-rms of 20.17-polar vs 21.61-decoder, and crest 10.19-polar, 12.05-decoder), and the slight amount of compression is noticeable, but doesn't alter the relative clean/distinct separation of the components of the music -- even audible through the 200kbps mp3 encoding.

A longer copy of the demos reside on my repository, SuperTrouper-ReferencePolar.mp3 and SuperTrouper-bestever.mp3.

I tried to make the EQ and channel content as close to the original as possible, but note the very substantial lack of intermod (that rough sound) at the beginning of the song.  Also, listening to the longer copy -- if you do careful A/B comparisons at different positions, listening to different phrases of the music, even if there might be a slightly different content, the various components of the music are more distinct on the 'decoder' version rather than the 'reference' version from the CD distribution.

Even the slightly more distorted CD distribution is very special -- many releases of that ABBA song would have a peak-rms of less than 15dB and/or crest factor less than 5.  The CD copy is especially clean and uncompressed for a commercial copy of the song SuperTrouper.  The copy of the version from the DolbyA compatible decoder (with only some EQ manipulations using SOX) is the clearest/least compressed copy of SuperTrouper that I have ever heard of.  (Kind of sad that it is so difficult to get good copies of such special music -- some songs are less distorted than this, but still there is an over-processed quality to some of their music.)

This decoder (or one of its descendants) is going to be very useful for retrieving archival material with the best quality possible -- and it also sounds pretty good!!!

Repo location with examples and the most recent free version of the decoder:
https://spaces.hightail.com/space/tjUm4ywtDR

PS -- I added some more ABBA examples.  Even though they tend to be a little harsh due to no compression, some of the sound shows more apparent original intent compared to the original CD versions.  UnderAttack is much more interesting sounding (for example.)

John
Title: Important (but not critical) quality fixes for the consumer version of decoder
Post by: jsdyson on 2018-07-12 03:00:12
This release contains 3 fixes based upon the 06jul2018 consumer pre-release.

The three fixes include:
1) mistaken choice of decay calculation feedback input.  Used the non-feedback version, which could cause deferred decay -- leaving more compression than correct.  I have heard negative effects only once, even though there are probably more instances.
2) mistaken choice in feedback calculation, where the gain was calculated based upon the non-feedback version of the signal.  This could create a deferred decay on the right channel.  I have heard this negative effect a couple of times, not realizing that it was because of something so very simple.  I actually thought it to be a characteristic of the recording.
3) error in calculating the gain for HF1 (essentially 9k-20k), the bug was minor (extraneous maximum gain limit based upon the raw HF1 gain level.)  There was no need for that gain limit, and the audible effect of the fix is to mitigate some apparent distortion-like sound and produce clearer, more accurate 9k-20k.

None of the above are very critical -- the bugs are seldom very obvious, except for #3, where the highs will generally be clearer.

The work on the professional version includes many more improvements, but these are bona-fide bugs which should be fixed in any version that people might be using.   I didn't want such simple bugs to have a negative impact, so I created this release.

On a scale of 1...10 for criticality, I'd give it a 5, but there is no real reason not to use this new version.

John
Title: Have some wonderful demos (after decoding only -- sorry.)
Post by: jsdyson on 2018-07-18 00:24:55
I uploaded some wonderful demos -- fairly hard for a real DolbyA to decode nearly as clearly.  I grabbed material which has lots of opportunity for intermodulation, possible stereo image smearing/shift, mirrored (quick echo) voices, and just vocal reproduction in general.  A lot of this older material seems to have significant vocal emphasis -- but I think that you might agree that the sound is pretty good.  In recent days the decoder has gotten more correct LF behavior (so true LF like Scarborough Fair is much better), and also the distortion sidebands as created by a changing gain slew are mostly cancelled now.

I cannot claim that the results are impressive -- but they are darned good.  The decoder really does well considering that some people say that it is impossible. 

The only frustrating thing is that the site that I use limits the mp3 to 200kbits/sec and any mp3 causes a significant loss of the subtle detail.

The full feature and quality decoder will be available sometime in the future.  Right now, the currently available 'free' version still does okay for casual use, but the very enhanced and completed version is very significantly better.  Hopefully, there will be something released as a plugin.

DEMOS:
https://spaces.hightail.com/space/z3H68lAgmJ

John

 
Title: Some embassment about a few of my demos.
Post by: jsdyson on 2018-07-23 16:37:27
After some review by some pros (and re-review by me) -- we finally determined that a few of my demos are not DolbyA encoded -- that is one of the troubles, because some kinds of material are difficult to detect either way (believe it or not.)  The recordings either had significant amounts of treble boost and/or HF compression effects.  The songs were by Dionne W and Nat King Cole.  The rest of the material (mostly during/after 1960's pop examples & Mancini) are encoded.  I was really embarassed, but got caught up in the 'fix everything with a screwdriver' syndrome.  I am still wondering a little about DionneW, but I am erring on the side of caution.

One positive item of note -- the decoder has LOTS of enhancements from the basic 'theme' -- still being able to accurately decode DolbyA -- it also avoids creation of distortions resulting from dynamic range expansion.  SO - the typical distortion artifacts from gain expansion are incredibly well suppressed.  Pure/simple expander distortions are not able to be used for indicating a lack of 'hand in glove' expansion, but one has to listen very carefully for stereo image problems, gating kinds of behavior or very muted HF sound.  These just didn't happen on either of those two albums.  Usually, like on my Mobile Fidelity Supertramp (Crime of the Century, AFAIR), there are several bad artifacts when trying to decode material that isn't in the state of being encoded.

I would NOT suggest using the DA compatible decoder as a GP expander or single ended NR (esp because I have better technology in the pipeline), but it does sometimes almost/weakly work.  Specifically the examples from those two artists had a 'veil' on the sound that kept on irritating me, and I suspect that I should have been more critical.  There was something wrong with them, but they generally sounded so good that I kept them.

The Carpenter's and other pop examples (the S&G example, however imperfect, even shows some of the LF distortion cancellation from DA decoding) still have the appropriate pre-decoding artifacts, and I carefully double checked the results.  In some cases, the examples might have been further compressed upon previous DA encoding, but the decoder is VERY GOOD at recovering the DA encoding in difficult cases.  (It isn't using any of my adaptive techniques, so will definitely sound bad when it is presented material that is far from encoded, esp totally uncompressed.)

It is just that Nat & Dionne sounded so good!!!  As I find where I have posted the examples, I have been removing them with a bit of embarassment.  The odd thing is that there was also a definite point where the so-called 'decoding' worked best...  Really kind of weirds me out, but gotta move forward, right?!!?

I don't have an infinite sized collection, and have only a few more recordings that I might try out -- hopefully I can find some more to replace the previous erroneous ones (and more carefully test next time!!!)   Buuuttt...  I so much liked the music!!!

John
Title: Almost 100% success on eliminating the intermod distortion from DolbyA decoder
Post by: jsdyson on 2018-08-05 04:41:56
I have been working on this 'tilting at windmills' project for the last several years -- really becoming useful in the last 6mos or so.  However, one major problem has dogged me for a very long time -- how to eliminate the distortion byproducts of the necessary very fast gain control of DolbyA decoding.  These distortion byproducts are the result of the way that the attack/decay works -- and is a problem on both the encode and decode side.  However, the decoding distortion seems worse -- and I have some control over the decoding distortion.
After 100s (literally) of experiments -- using numerous techniques to get around the distortion problems -- I have ended up finding several that when used together for all practical purposes -- ELIMINATES the intermod produced by the decoding process.  I cannot divulge all of the techniques -- but I have exhausted all patent resources and common sense design techniques, and finally found one that isn't documented anywhere that I know of -- and another 'obvious' approach which was the final fix.  After using the 'obvious' approach along with the tricky use of signal processing tricks (I didn't even have to do the hilbert thing so that the distortion was hidden on one side of the spectrum) - I can truly no longer detect intermod effects due to decoding. (Btw -- the 'obvious' approach really isn't 'obvious' but rather brute force, but also made simple because of some of my C++ classes already supporting the kind of operations necessary to do that calculation.)

This is a success, yet also a bit of a let down, because now my project is to teach the program to support the BW64/RF64 stuff for pro applications...  Bit-twiddling at that level is no fun!!!  The audio processing is much more interesting and more intellectually challenging instead of just tedious.

Don't expect perfection in these two examples -- one is direct from a Polar (gold standard ABBA original) CD, and the second is from my 'close to master' copy being decoded using the newest/amazing -- no intermod decoder.   ABBA isn't known for their pristine sound, but you CAN hear the difference -- and their recordings are a serious challenge to the decoding operation.   You'll notice that the voices are MUCH LESS mushed up on the decoded version relative to the gold standard CD.  One other amusing item -- there is slightly more compression on MY copy (including the DolbyA encoded source) instead of the CD -- but it SOUNDS like the CD is much more compressed.  That 'compression' sound on the CD version is the result of the huge amount of intermod -- which fuzzes things up badly.

So -- the super clean version is 'Mia-decodere.mp3', and the CD version (Polar) is 'Mia-orig.mp3' as attached.  Also, I'll have more demos online (will be updated within an hour from now -- look at the posting time) at:

 https://spaces.hightail.com/space/xghqJodgrj
Title: 100% Success -- almost no intermodulation DolbyA decoding
Post by: jsdyson on 2018-08-05 06:31:00
Just to show how good it really is -- I uploaded a small (44.1k/16 bit) flac version of a couple of the demos.  You can really 'feel' the difference in sound quality between the flac and mp3 (I can hear a difference in crispness/clarity.)  The reason for adding the higher quality example is to show that the decoder is so very good now, that mp3 causes significant quality loss even at 256k.  For listening purposes I really cannot tell the difference between the 96k/24bit copy that the decoder produces vs. the 44.1k/16bit version that is in the demo -- but I'd suspect that someone probably can somewhere :-).  The mp3 encoding is the big quality killer.

I am aggressively working on the pro version of the decoder, and we will try to produce a pro and consumer version -- and keep the difference significant enough that the pros will prefer the pro version.  I think that it might entail fewer of the quality optimizations and support for pro audio formats -- however, the results will still be superior to anything else obtainable even with the limited quality consumer version.  My guess is that some of the intermod cancellation logic might be disabled.  Doing so would leave a slight veneer of fuzz -- much less than previous DolbyA decoders, however.

The pro version has EVERY anti-distortion method that I have been able to research -- and will aways keep my eyes open for more -- but it is already SO VERY GOOD.

So, I did just add a good quality flac to the demo.  (Space is limited, and that is the reason for only one example.)

Repo site again:  https://spaces.hightail.com/space/xghqJodgrj
Title: Major update to the DolbyA compatible decoder.
Post by: jsdyson on 2018-08-20 04:01:17
This is now pro-quality but needs a little work to be complete (e.g. missing several 100 samples of audio, phase inversion, a few freq response anomalys (even though it appears that it spectrally matches a DolbyA within very few tenths of a dB. )) The distorion reduction is incredible -- anyone who has listened to ABBA on headphonrs might remember the rough sound.  That rough sound is due to DolbyA distribution onto CD and vinyl -- but vinyl releases are usually properly decoded.  The CDs (listen to SuperTrouper) are really oddly distorted -- I have heard very few American releases that aren't distorted.  Some of the Polar/Polydor releases aren't too bad.

There is distortion cancellation code in the decoder -- some I cannot describe due to intellectual property reasons.  However, there is a mode which removes more noise than a DolbyA (it takes advantage of decoding differently -- and uses some techniques at decoding time to benefit.)

The examples before weren't too bad -- mostly better that a Dolbya because of intermod.  These examples are essentially as good as theoretically possible (I might be able to eeek out a little more quality, but already the examples sound nearly perfect.)

An example is the "supertrouper" item -- there is a chorus burst which is normally mired in deep intermod -- but even though there is a bit of a distorted sound in my result, the intermod is well under control.  I use ABBA as my basket case test -- because of the voices and recording technique it drives DolbyA crazy.
I intend someday soon to distribute a consumer version of the decoder -- it will still sound really good, but might not be quite as good as the pro version, be limited to 48k, and might be limited in decoding length.  (albums tend to be longer than perhaps the 10minute timeout that I might plan -- that will probably thwart commercial use.)

Listen to the examples, and compare with your CDs...  I have to say 'wow' and I was almost crying with relief early today.

I uploaded mp3 versions of the ABBA decoding -- doesn't sound perfect and significant mp3 loss but at the beginning of snippet 1, the beginning has no intermod.  The second snippet near the end is a little rough -- but NOWHERE near the intermod of almost any release.  (It might sound slightly different than commercial releases because some aren't decoded.)  This is from ABBA Gold, 1992 which the copy that I have is undecoded (believe it or not.)  The repo has more complete examples.

https://spaces.hightail.com/space/xghqJodgrj

John