Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: HALAC (High Availability Lossless Audio Compression) (Read 46312 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #150
@Hakan Abbas

Encoding fails with HALAC_ENCODE_V.0.2.9_x64_avx.exe. Using v0.2.8 worked fine.

HALAC_ENCODE_V.0.2.8_x64_AVX.exe
HALAC Decoder (foo_input_halac): https://foobar.hyv.fi/?view=foo_input_halac
Windows 11 Home
foobar2000 v2.1.5 x64

-Encoder file: HALAC_ENCODE_V.0.2.8_x64_avx.exe
-Extension: halac
-Parameters: %s %d -mt=8
-Format is: Lossless
-Highest BPS mode supported: 16bit
-Do not convert in multiple threads
SHURE SRH1840, SENNHEISER HD660S2, SENNHEISER HD620S, SENNHEISER HD 490 Pro Plus, beyerdynamic DT 1990 PRO, HiFiMAN Edition XS, Bowers & Wilkins P7, FiiO FT5, 水月雨 (MOONDROP) 空鳴 - VOID, Nakamichi Elite FIVE ANC, SONY WH1000XM5 (made a Upgrade/Balanced Cable by myself)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #151
There is almost no difference in the compression ratio and speed in HALAC 0.2.7, 0.2.8 and 0.2.9 versions. You can already see this by trying it externally. I did not have a problem with 0.2.8 or 0.2.9 in Foobar (Windows 10, x64). They only work slower while "Do not convert in multiple threads" is not selected. You have never worked 0.2.9. Interesting situation.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #152
HALAC Decoder (foo_input_halac) v0.1.4 (2024-06-13)
https://foobar.hyv.fi/?view=foo_input_halac

FLAC 1.4.3 → HALAC 0.2.9

Windows 11 Home
foobar2000 v2.1.5 x64

@Hakan Abbas and @Case Many Thanks.


SHURE SRH1840, SENNHEISER HD660S2, SENNHEISER HD620S, SENNHEISER HD 490 Pro Plus, beyerdynamic DT 1990 PRO, HiFiMAN Edition XS, Bowers & Wilkins P7, FiiO FT5, 水月雨 (MOONDROP) 空鳴 - VOID, Nakamichi Elite FIVE ANC, SONY WH1000XM5 (made a Upgrade/Balanced Cable by myself)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #153
I updated the foobar2000 decoder to use version 0.2.9, for now just using same lazy full-file decode method as has been used before.

I'm not certain what your intentions regarding your metadata routines are, I don't see your encoder or your player having any metadata support. You could consider not inventing your own metadata format and instead use a good, existing and quite widely supported APEv2 tags. FYI the foobar2000 component has full tagging support including album art using APEv2 tags.

I'm a bit confused about the function GET_INPUT_FILE_SIZE() as there doesn't seem to be a way to retrieve the original file anymore with the DLL.

Are you certain 32-bit unsigned integer is enough for the frame count field? Or is the idea that frames will be so large that anything will be made to fit?

I really recommend you to add error and sanity checks to your decoder. Just feeding the decoder wrong data makes it crash. For example feeding the function GET_FRAME_DATAS() the entire source file instead of data starting from offset retrieved by GET_FIRST_FRAME_START_POSITION() makes the decoder crash. Or altering the file header and increasing the frame count makes your player crash.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #154
@Air KEN : Good News

@Case : Thanks again for your work, Case.

for now just using same lazy full-file decode method as has been used before
https://github.com/Hakan-Abbas/HALAC-Audio-Player/blob/8e5fbbdb4601cdd1b2aada6685f2174963549b6e/mainwindow.cpp#L621-L658

I'm not certain what your intentions regarding your metadata routines are, I don't see your encoder or your player having any metadata support.
I get the metadata data as it is from the source file. No external processing is done.

I'm a bit confused about the function GET_INPUT_FILE_SIZE() as there doesn't seem to be a way to retrieve the original file anymore with the DLL."
GET_INPUT_FILE_SIZE() returns the full file size by adding 8 bytes to the size information in the header of the source file (wav). Just like GET_INPUT_DATA_SIZE() but this is just raw data.

Are you certain 32-bit unsigned integer is enough for the frame count field? Or is the idea that frames will be so large that anything will be made to fit?
The new uncompressed frame size is 512 kbytes. The total input data size that can be supported is 512 * 1024 * 4,294,967,296 = 2,251,799,813,685,248 bytes. I thought that would be enough. It can be increased if necessary.

I really recommend you to add error and sanity checks to your decoder.
Some of the new functions have various controls. More powerful controls will be added in the future as the details become clearer.

I try to keep things as simple as possible. As I mentioned before, there are things that can be done for HALAC in terms of compression ratio. I can see the gap here, I need to focus a bit more.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #155
Code: [Select]
BUSTA RHYMES(i7 3770k, 20 tracks, Total 829,962,880 bytes)
HALAC NORMAL 1 Thread (ANS) : 4.32 sec, 574,161,601 bytes
HALAC NORMAL 1 Thread (RICE): 3.66 sec, 567,469,774 bytes

Hi, Hakan.
If those numbers are close to real life averages, then HALAC w/Rice  will be  on par with FLAC -4  :)  while still being faster enc/dec.

After -4 Flac doesn't bring groundbreaking compression gains (<1%). 
http://www.audiograaf.nl/downloads.html
http://www.audiograaf.nl/losslesstest/revision%206/Average%20of%20CDDA%20sources.pdf

Given that I see myself  (and likely other FLAC users) choosing HALAC w/Rice  if/when format will be solid.
Looking towards  HALAC release 0.3  :)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #156
Given that I see myself  (and likely other FLAC users) choosing HALAC w/Rice  if/when format will be solid.
Looking towards  HALAC release 0.3  :)
Thanks for the comment, @btc.

As I mentioned before, the compression ratio in lossless audio compression is really limited. I think the processing speed will be more important as we approach the end of silicon technology, because in recent years processors have not accelerated much on a single core basis. Multi-core solutions and parallel processing are becoming more prominent.

Looking at guruboolez's latest test, codecs at the slowest settings provide about 5% better compression on average than HALAC. However, we can't say that these are of practical use in real life. If we look at the practical use case, we can talk about a maximum gap of 3%. Some other codecs can do this at the cost of much more processing time than HALAC. I don't know how much of this gap HALAC can close at similar speeds, but Rice coding is full of surprises.

My work for HALAC v.0.3.x is not finished yet. And I don't want to share or rush it before I am fully sure of the results. The HALAC format will be much more robust in future versions (HD Audio, Multichannel...). As long as I take the necessary time and stay motivated.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #157
HALAC version 0.3.6 is both faster and has a better compression ratio. And the ‘lossyWAV’ results are also now more impressive.
https://github.com/Hakan-Abbas/HALAC-High-Availability-Lossless-Audio-Compression/releases/tag/0.3.6

Basically the entropy encoder stage has completely changed. This version uses Rice coding. It was a bit of a pain, but I finally finished my new Rice Coder. Of course, the results can be further improved both in terms of speed and compression ratio (we can see a similar effect for HALIC). That's why I'm delaying the 24/32 bit generalisation. No manual SIMD, GPU or ASM was used. Compiled as Encoder AVX, Decoder SSE2.
The results below show the single core performance of version 0.2.9 with version 0.3.6. I'll leave the API and Player update for later, I'm a bit tired.

https://www.youtube.com/watch?v=Pxq-WXja78Y
Code: [Select]
AMD RYZEN 3700X, 16 gb RAM, 512 gb fast SSD

WAV RESULTS (Encode Time, Decode Time, Compressed Size)
Busta Rhymes - 829.962.880 bytes
HALAC 0.2.9 Normal 2.985  4.563  574,192,159
HALAC 0.3.0 Normal 2.578  4.547  562,057,837
HALAC 0.2.9 Fast   2.010  4.375  594,237,502
HALAC 0.3.0 Fast   1.922  3.766  582,314,407

Sean Paul - 525.065.800 bytes
HALAC 0.2.9 Normal 1.875  2.938  382,270,791
HALAC 0.3.0 Normal 1.657  2.969  376,787,400
HALAC 0.2.9 Fast   1.266  2.813  393,541,675
HALAC 0.3.0 Fast   1.234  2.438  390,994,355

Sibel Can - 504.822.048 bytes
HALAC 0.2.9 Normal 1.735  2.766  363,330,525 
HALAC 0.3.0 Normal 1.578  2.828  359,572,087
HALAC 0.2.9 Fast   1.172  2.672  376,323,138
HALAC 0.3.0 Fast   1.188  2.360  375,079,841

Gubbology - 671.670.372 bytes
HALAC 0.2.9 Normal 2.485  3.860  384,270,613
HALAC 0.3.0 Normal 1.969  3.703  375,515,316
HALAC 0.2.9 Fast   1.594  3.547  410,038,434
HALAC 0.3.0 Fast   1.453  3.063  395,058,374
----------------
lossyWAV RESULTS
Busta Rhymes - 829.962.880 bytes
HALAC 0.2.9 Normal 3.063  2.688  350,671,533 
HALAC 0.3.0 Normal 2.891  4.453  285,344,736
HALAC 0.3.0 Fast   1.985  2.094  305,126,996

Sean Paul - 525.065.800 bytes
HALAC 0.2.9 Normal 1.969  1.766  215,403,561
HALAC 0.3.0 Normal 1.860  2.876  171,258,352
HALAC 0.3.0 Fast   1.266  1.375  184,799,107

Re: HALAC (High Availability Lossless Audio Compression)

Reply #158
I ran a comparison script with one of my sample sets, consisting of 6 audio tracks of different genres, 46 minutes in total. A few different codecs were used, all single-threaded. Results are attached.

edit: nevermind the footnotes, I just reused a script from that comparison. Also, I see now that the speed scale on the decoding graph has no numbers. The second vertical line (the one going through the TAK dots) is 0.2%, the next one 0.3%, then 0.4% etc. So, the fastest decoding codec is just shy of hitting 1000x realtime on my system.
Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #159
Thanks for your interest ktf. At the top of this test it says that HALAC 0.2.9 was used.
The “normal” decode speed of HALAC 0.2.9 is about the same as 0.3.6. It is quite close to FLAC. But "0.3.6 - fast” mode is significantly faster than "0.2.9 - fast" mode.
https://encode.su/threads/4180-HALAC(High-Availability-Lossless-Audio-Compression)/page2

Re: HALAC (High Availability Lossless Audio Compression)

Reply #160
Ah, another flaw. Sorry about that. I did really test 0.3.6, but forgot to update the graph script.
Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #161
I quickly updated "HALAC -fast" mode and now the results are much more interesting (HALAC 0.3.6 Ultra)  ;)

Code: [Select]
AMD RYZEN 3700X, 16 gb RAM, 512 gb fast SSD
WAV RESULTS (Encode Time, Decode Time, Compressed Size)
Busta Rhymes - 829.962.880 bytes
HALAC 0.2.9 Normal 2.985  4.563  574,192,159
HALAC 0.3.6 Normal 2.578  4.547  562,057,837
HALAC 0.2.9 Fast   2.010  4.375  594,237,502
HALAC 0.3.6 Fast   1.922  3.766  582,314,407
FLAC -0            2.901  3.277  636,691,981  (flac -0b3072 -r0 --no-md5 --totally-silent) // fastest -0
FLAC -5            5.015  3.803  563,256,303  (flac -5 --no-md5 --totally-silent)
HALAC 0.3.6 Ultra  1.722  2.158  585,752,400

Sean Paul - 525.065.800 bytes
HALAC 0.2.9 Normal 1.875  2.938  382,270,791
HALAC 0.3.6 Normal 1.657  2.969  376,787,400
HALAC 0.2.9 Fast   1.266  2.813  393,541,675
HALAC 0.3.6 Fast   1.234  2.438  390,994,355
FLAC -0            1.873  2.096  412,011,684  (flac -0b3072 -r0 --no-md5 --totally-silent) // fastest -0
FLAC -5            3.226  2.462  376,134,352  (flac -5 --no-md5 --totally-silent)
HALAC 0.3.6 Ultra  1.125  1.424  394,791,112

Sibel Can - 504.822.048 bytes
HALAC 0.2.9 Normal 1.735  2.766  363,330,525
HALAC 0.3.6 Normal 1.578  2.828  359,572,087
HALAC 0.2.9 Fast   1.172  2.672  376,323,138
HALAC 0.3.6 Fast   1.188  2.360  375,079,841
FLAC -0            1.781  2.027  390,509,281  (flac -0b3072 -r0 --no-md5 --totally-silent)  // fastest -0
FLAC -5            3.054  2.368  360,175,511  (flac -5 --no-md5 --totally-silent)
HALAC 0.3.6 Ultra  1.059  1.352  377,917,531

Gubbology - 671.670.372 bytes
HALAC 0.2.9 Normal 2.485  3.860  384,270,613
HALAC 0.3.6 Normal 1.969  3.703  375,515,316
HALAC 0.2.9 Fast   1.594  3.547  410,038,434
HALAC 0.3.6 Fast   1.453  3.063  395,058,374
FLAC -0            2.175  2.564  421,740,466  (flac -0b3072 -r0 --no-md5 --totally-silent)  // fastest -0
FLAC -5            3.909  2.940  368,024,298  (flac -5 --no-md5 --totally-silent)
HALAC 0.3.6 Ultra  1.304  1.765  412,729,188

Re: HALAC (High Availability Lossless Audio Compression)

Reply #162
I quickly updated "HALAC -fast" mode and now the results are much more interesting (HALAC 0.3.6 Ultra)  ;)
I thought the result were already quite interesting: nothing comes close to HALAC when it comes to encoding speed. Anyway, I'll do a more thorough test with a larger sample set soon, and with your faster -fast mode.
Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #163
Since -b1152 isn't the fastest FLAC can come up with, and HALAC competes at speed, let me suggest that the "dotted flac" is run not only with MD5 disabled, but with say,
-<N> --no-md5 -b3072 -r0

Same -b for all, higher than 1152 because it is faster, and I do think I remember that 4096 was sometimes suspiciously slower than 3072 - although I have some faint memory that a recent build has addressed it, maybe 1.4.3 maybe git ...
-r0 because the partitioning is a flac specific thing - well I don't know about HALAC of course. (Frankly I don't even know if -r0 is faster than -r3 - if it makes for fewer bits to write, it could be the other way around.)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #164
No, I think that isn't a fair comparison. The only reason I included flac with MD5 disabled, is because players generally don't do MD5 verification. In fact, MD5 checking is disabled by default in the API, it is just the flac command line tool that insists on verification when an MD5 is available. It would be even more fair to only include the "no-MD5" option on decoding, but I'll have to change my graph script to be able to do so. Maybe I'll do that anyway.

Using a blocksize of 3072 while comparing would in my opinion only be fair if that was the default, which it isn't.
Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #165
Tested.
New corpus: track 2 from each of the 38 CDs in my signature, to make it fit a RAM disk, 3h44m of CDDA.  Found out later that the Mahler symphony makes for 1/4 of the total duration.  Should just have deleted that.  Presenting ratios with and without that one, but kept the timings with all.
All FLAC decoding/encoding with 1.4.3 Win32.  With --totally-silent on top everything, it does not spend time spamming the console (which could be significant at these speeds!)
All times are median of at least 3.

6.7 is an impressive figure eh?

w/o M.all 38 tracksencodingdecoding.
60,058%56.30%6.7s 11.6s HALAC FAST
60,063%55.55%8.9s 11.6s FLAC -0b3072 -r0 --no-md5 --totally-silent
59,76%55.36%9.6s 12.5s FLAC -0  --no-md5 --totally-silent
58,10%54.03%9.8s 11.9s FLAC -1b3072 --no-md5 --totally-silent    <---- that is a "smart" joint stereo/dual mono
56,60%52.59%10.0s 13.3s HALAC
56,76%52.49%12.4s 12.7s FLAC -3 --no-md5 --totally-silent <---- that's variable-coefficient LPC but dual mono.
54,951%51.08%14.3s 17.4s TAK at p0 with no WAVE metadata (and no MD5)
54,948%51.05%30 s13.2s FLAC --no-md5 --totally-silent
HALAC isn't so happy about Mahler. Dropping it makes HALAC FAST beat the fastest FLAC at size, and makes HALAC beat FLAC -3 at size. TAK -p0 isn't so happy about that track either, but it is usually so good that it is a mild surprise to see flac -5 beat it at size - they trade about even blows here.

MD5 would add around 4 seconds to FLAC times.
https://hydrogenaud.io/index.php/topic,125248.msg1037617.html#msg1037617

@Porcus : I am not an expert on FLAC. As seen in the post above; the parameters I used seemed to be the best ones you had used before. That's why I chose them.
The block size in HALAC -fast's mode is 2048.


Re: HALAC (High Availability Lossless Audio Compression)

Reply #167
If I understand the question correctly, I think you are talking about the correlation between the channels. In its simplest form, it's like this;
Code: [Select]
L = data_left;
R = data_right;
X = L - R;
Y = (L + R) >> 1;
data_left = X;
data_right = Y;

Re: HALAC (High Availability Lossless Audio Compression)

Reply #168
Does HALAC do anything more than left+right or mid+side?
FLAC can also for each frame also select left+side or right+side. That means, if dual mono isn't the best, it can do "any + side" and cherry pick according to whether left, mid or right comes out smallest. That helps compression relatively cheap if you anyway have to calculate left, right, mid and side to compare.

You can force the reference encoder to use left+right only; presets -0 and -3 do that. The format can of course encode everything as mid+side, but you cannot force the reference encoder to do that.

(All this is stereo. For anything but stereo, the FLAC format must stick to independent channels.)


Re: HALAC (High Availability Lossless Audio Compression)

Reply #169
HALAC does not apply any transformation to data of type lossyWAV. This is because the results are slightly better this way. But in other cases there is a quick selection mechanism. However, I cannot say that it is very efficient.
In the future, I plan to develop a more sensitive and faster channel analyser for multi-channel audio data. For example, there are definitely relationships in 5+1 or 7+1 channel audio data. I have encountered such a situation before when working with multispectral image compression. It is not a good approach to process channels completely independently in multichannel (2+) audio data.

And now I see that the HALAC decoder is significantly faster in my tests with an Intel processor. In my previous tests I used an AMD processor... Anyway...

 

Re: HALAC (High Availability Lossless Audio Compression)

Reply #170
Anyway, non-rigorous testing on my 38 CD images, HALAC 0.3.6 Ultra vs flac 1.4.3 --no-md5 -0b3072 -r0 as I suggested (uh, and no padding on the FLAC), it seems to encode and decode faster (encoding is not quite twice as fast), and compresses better - 56.6 percent vs 57.3 percent. But that depends on material: FLAC compresses classical music better.

And if I give FLAC the -M which does that loose stereo decorrelation (calculating only every now and then what is the best, and then sticking to it for some frames), I get it down below 55 percent. (There is a full CD that is near-mono in there.)

Since this version of HALAC insists on file extension, I cannot output to NUL.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #171
Since this version of HALAC insists on file extension, I cannot output to NUL.
It ought to work, NUL is NUL no matter what extension you give it.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #172
Since this version of HALAC insists on file extension, I cannot output to NUL.
It ought to work, NUL is NUL no matter what extension you give it.
Nopes, it encodes to a file called NUL.halac and decodes to a file called NUL.wav .
Anyway it does so faster than flac.exe encodes/decodes to actual NUL.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #173
Since this version of HALAC insists on file extension, I cannot output to NUL.
It ought to work, NUL is NUL no matter what extension you give it.
Nopes, it encodes to a file called NUL.halac and decodes to a file called NUL.wav .
Anyway it does so faster than flac.exe encodes/decodes to actual NUL.

Code: [Select]
C:\Users\HAKAN\Documents\Audio_Converter>HALAC_ENCODE_V.0.3.6_new.exe 01 out
Total Time: 0.082 seconds

C:\Users\HAKAN\Documents\Audio_Converter>HALAC_DECODE_V.0.3.6_new.exe out org
Total Time: 0.113 seconds
HALAC does not normally deal with extensions. It acts according to the header information of the input files. If there is a mismatch, it gives information. I have already added many of these controls for the API.
I added the extension restriction after I received a request by e-mail. Now I'm removing it again, I just closed an if block and that's it. Instead of releasing a version again, I will update it on Github with the ‘ultrafast’ version. In summary, nothing has changed in terms of speed in this case. It was supposed to be like this anyway. Because the necessary checks were done under all circumstances.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #174
Does HALAC handle mono audio by the way? I ran a comparison script overnight, and it seems HALAC did strange things on two mono files.
Music: sounds arranged such that they construct feelings.