Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: HALAC (High Availability Lossless Audio Compression) (Read 54633 times) previous topic - next topic
0 Members and 3 Guests are viewing this topic.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #175
Does HALAC handle mono audio by the way? I ran a comparison script overnight, and it seems HALAC did strange things on two mono files.
Code: [Select]
HALAC 0.3.6 Multithread Encoder (11 October 2024)
Hakan Abbas (abbas.hakan@gmail.com)
Only 16 bit, 2 channels .wav types are supported for input!
HALAC currently only supports 2 channel stereo. However, if the number of channels is different from 2, you should have received a warning.
All the data I have is 16 bit stereo. No request for single channel mono has been received so far. And I don't know much about the widespread use of mono audio data. If it is important, I can add mono when preparing multichannel support.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #176
The script ignores the warning of course, but I see now that the file I saw strange behaviour from has one channel with content, and one completely silent channel. Anyway, I restarted the script, might take a while.
Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #177
So, here are some test results. I tested a set of tracks from 27 very different sources. I've added overal results and results per source. See here for details on the test setup. For each album listed in the 'per source' PDF, two random tracks were selected. CPU used is a Core i5 7200U. All codecs are run single threaded.

Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #178
Encodes twice as fast as flac -0 --no-md5 and ten times as fast as shorten? Would have been something back in the day ... although this HALAC build does require AVX, right?

Anyway, now you are doing "plain Rice" for the residual?


(ktf, I see that -3 decodes fastest again. Is that due to block size?)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #179
@ktf Thank you very much for your interest and effort.

Encodes twice as fast as flac -0 --no-md5 and ten times as fast as shorten? Would have been something back in the day ... although this HALAC build does require AVX, right?

Anyway, now you are doing "plain Rice" for the residual?
HALAC was not used in manuel SIMD. Only automatic optimizations that the compiler can do. Therefore, there is no AVX obligation. However, -mavx is a bit fast for Encoder. Maybe -mavx2 is a little more. On the contrary, the decoder is -msse2.
Currently, only a special Rice Coder I have written is used. However, I just left behind in my speed tests with ANS or HUFFMAN. So I couldn't optimize enough.
According to the tests, there are some exceptions and I think I will have to examine different types of music.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #180
According to the tests, there are some exceptions and I think I will have to examine different types of music.
The author of TAK used this set of samples to test a balanced mix of audio: https://www.rarewares.org/test_samples/

Perhaps they work well for you too. These samples were one of the sources of the comparison I posted yesterday.

(ktf, I see that -3 decodes fastest again. Is that due to block size?)
Probably? Such variations can be the result of different CPU models.
Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #181
Thank you very much for sharing @ktf . I will use these examples from now on.
I can't say that I work with a very wide range of music. When I get good results with the music I have, it looks pretty good for me. But in the real world it can be a bit different. There are probably a few other things I've overlooked.

I think it's not “FLAC -0 -0b3072 -r0 --no-md5 --totally-silent” used in the tests. Otherwise it's unlikely that HALAC would lag behind in compression ratio in -fast mode.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #182
ktf uses -0 and -0 --no-md5 because -0 is one of the presets - you see the discussion above.
(Maybe also the silent switch?)

The reasons why I chose to add the two other parameters, are that you compete at speed, and then I chose a block size that makes it faster. I should likely have used -b2048 because HALAC uses 2048 (I think?)
My -r0 has in part a different explanation. FLAC can change the Golomb-Rice exponent several times during a block, to improve compression. -0 uses -r3, which partitions it into 2^3=8 [well, provided that a partition is an integer number of samples larger than the prediction order - but with -0's block size of 1152, that becomes 144].
So if HALAC doesn't do any such partitioning, that part is more apples to apples.

Also -r0 would be expected to speed up (but surprisingly little IIRC). The reason that -r3 should be slower, is that it calculates orders up to 3 and optimizes. It calculates eight Golomb-Rice exponents, one for each partition; then it checks whether it can improve by order 2 (that is, merging partition 1&2, 3&4, 5&6, 7&8), ..., down to order 0.

The partitioning sometimes - for some weird signals - makes an exceptional impact on compression: https://hydrogenaud.io/index.php/topic,122179.msg1014245.html#msg1014245 .  It is possible to construct a signal where "at worst", it is better to switch exponent for every sample, spending four bits on that.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #183
Thanks for your informative explanations Porcus.
HALAC uses 2048 samples in fast mode. I could not see any speed difference between 512 ... 4096. However, when the block size starts to decrease below 512, it starts to slow down.

Currently 4 different Rice ‘k’ parameters are set for a block. More did not have a positive effect for now. Of course, the predictor change and other improvements may affect this situation. But when I adaptively update the ‘k’ parameter as I used to do(per sample), the compression ratio actually improves at the expense of speed loss. However, such a thing is against the nature of HALAC.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #184
So the fast mode fixes block size to 2048, and switches Rice parameter every 512?
For comparison, FLAC presets -0, -1, -2 (the ones which use fixed predictors and 1152 samples in a block) switch up to every 144. The reference encoder can also enforce a minimum partitioning, and thus an exact partitioning: -r n,N for at least n and at most N; -r N is synonymous to -r 0,N. Higher presets use a block size of 4096 and max partitioning of 2^4 to 2^6. It isn't hard to get out stats on what partitioning it actually uses. I remember when was struggling to learn this, a lot of classical pieces would be bit-identical from -r4 to -r5, meaning that not a single block would be partitioned into 2^5 (a Rice parameter could live for 256 samples and the encoder would never during an entire track want to switch twice as often). More abrasive music can sometimes want to switch after 32 rather than after 64. Extremely noisy music even more.


IIRC, ALAC calculates the Rice parameter from past samples, and that seems to be one reason ALAC doesn't hit FLAC speeds. But if you have studied the code, you know better than I do. (ALAC can also do run-length, which is good in those few test cases where the predictor hits exact.)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #185
Yes, HALAC fast mode is currently 2048 samples and parameter changes are made every 512 samples. The choice of 1152/144 for FLAC seems really good. In the many tests I have done before, changes in block size or number of parameters did not make a significant difference. Obviously there are some points I have missed. This will require more experimentation.

If ALAC does adaptive parameterisation, this is one of the main reasons why it is slow. For example, JPEG LS also works this way.
I haven't really looked at the codecs of ALAC/FLAC or other codecs because it's really boring and difficult for me to try to understand the codecs I haven't written myself. I prefer to concentrate on general principles instead. Also, I can work much more freely this way. That's why I had to re-develop a Rice Coder, I couldn't even focus on the Coder I was using before.

Isn't Linear Prediction used at higher FLAC levels? Because although the encoding times change, this is not reflected in the decode times.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #186
Yes, reference encoder's presets -3 and up uses estimated LPC coefficients stored in the stream - that is, they don't mandate it, for a given signal it can choose not to employ that method.
-0, -1, -2 exclude this "stored coeffcients" type: They can use VERBATIM encoding, and CONSTANT (if applicable) and the "FIXED" encoding: that is, the difference operator of order 4 or less. (Including "0", which means it predicts 0 and encodes the sample as residual - unlike VERBATIM when the difference is unencoded.)
But a consequence is that -0, -1, -2 ensure that the files can be decoded without multiplication.
-0 and -3 encode as dual-mono only. -1 and -4 uses the encoder's fast "determine stereo decorrelation only every now and then", which doesn't make any difference to the format itself.
-4 also allows higher LPC order than -3 (eight rather than six rather)
-5 and up allow higher LPC, and brute-force stereo decorrelation, and then -7 and -8 allow for higher LPC order of 12. (The only difference -8 makes, is to try more windowing functions in the Levinson-Durbin algorithm.)


How FLAC the format looks, by example files "decoded by hand": https://datatracker.ietf.org/doc/html/draft-ietf-cellar-flac-14#name-audio-frame

Brief walkthrough, sometimes imprecise - assuming constant block size (which the reference encoder uses) and per channel ("Subframe" is FLAC lingo for an "encoded channel after decorrelation", say side or left. All subframes are same number of samples as the block.)
* Bits to identify subframe type and prediction history order:
There are four different subframe types: VERBATIM (each sample unencoded), CONSTANT (need not be zero, I've seen value "-1"), FIXED and LPC (that is, the format uses the phrase "LPC" to mean "LPC with coefficients stored in the stream").
Order 0 means that the prediction is "0"; the distinction between that and VERBATIM is that the latter doesn't Rice encode.
The reference encoder's -l0 switch doesn't mean order 0, it means "don't use the fourth type". That's in "-0" to "-2".
* Wasted bits choice. Say, "fake 24 bits" with bits 17 to 24 all zero throughout the subframe, are flagged with an "8" wasted bits and otherwise encoded as if it were "16". If it is 16-bit white noise as "fake 24 bits", then that saves quite a lot when encoded VERBATIM. For CONSTANT subframes, it isn't much use.
* For LPC: the coefficients, first having specified coefficient precision and right-shift. (The LPC order is already encoded in the bits for the subframe type.)
* For FIXED and LPC (those which actually code residual!):
First choice of residual coding - it is unlikely there will be any more than the two types already implemented, 4-bit Rice and 5-bit Rice.
Then choice of partitioning order. Must be so that block size / 2^r is an integer greater than the predictor order (because otherwise the unencoded warm-up samples could cover a whole partition and there would be a Rice parameter used for no samples at all - I fail to see why that restriction was necessary, but done is done).

Re: HALAC (High Availability Lossless Audio Compression)

Reply #187
Isn't Linear Prediction used at higher FLAC levels? Because although the encoding times change, this is not reflected in the decode times.
It isn't entirely clear why this happens. One hypothesis is that  LPC being slower is more-or-less cancelled out by the larger blocksizes being faster. The FLAC presets that do not use LPC have a smaller blocksize.

Anyway, when decoding a file compressed with FLAC -0, the decoder spends about 45% of the time parsing and decoding the Rice entropy code, 30% calculating the MD5, 25% for the rest. When disabling MD5 calculation, a whopping 65% of the time is spent parsing the Rice entropy code, 15% is spent undoing the predictor, 9% is spent calculating the CRC, 7.5% checking whether samples are out-of-bounds. So, I'd say your Rice reading code is very good, it probably beats FLACs current approach.
Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #188
What makes HALAC encode so fast is even more impressive to me. But isn't it weird that the normal mode spends twice as much time decoding as encoding? Or is it just not optimized for normal mode yet?

(Did flac-without-MD5 fall out of the encoding graph? flac.exe will compute MD5 upon decoding if it isn't set to zero?
And another BTW: The Shakespeare Lectures, is that mono-as-stereo i.e. a zero side channel?)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #189
It isn't entirely clear why this happens. One hypothesis is that  LPC being slower is more-or-less cancelled out by the larger blocksizes being faster. The FLAC presets that do not use LPC have a smaller blocksize.

Anyway, when decoding a file compressed with FLAC -0, the decoder spends about 45% of the time parsing and decoding the Rice entropy code, 30% calculating the MD5, 25% for the rest. When disabling MD5 calculation, a whopping 65% of the time is spent parsing the Rice entropy code, 15% is spent undoing the predictor, 9% is spent calculating the CRC, 7.5% checking whether samples are out-of-bounds. So, I'd say your Rice reading code is very good, it probably beats FLACs current approach.
I think you have prepared the information on the link that Porcus gave. I looked quickly, but it's a really great content.

X
66 %of the values ​​are obtained (estimation) and 20 %for rice. This is quite interesting ... However, in Decoder, there is much less process than Encoder. Decoder accelerates as the LPC level decreases.

What makes HALAC encode so fast is even more impressive to me. But isn't it weird that the normal mode spends twice as much time decoding as encoding? Or is it just not optimized for normal mode yet?
I can't say that Decoder is fully optimized. It is true that I have always devoted a lot more time for Encoder. However, when Decoder works, it is okay :)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #190
(Did flac-without-MD5 fall out of the encoding graph? flac.exe will compute MD5 upon decoding if it isn't set to zero?

I removed it. See:
It would be even more fair to only include the "no-MD5" option on decoding, but I'll have to change my graph script to be able to do so. Maybe I'll do that anyway.

Quote
And another BTW: The Shakespeare Lectures, is that mono-as-stereo i.e. a zero side channel?)
Yes

[attach type=image]32938[/attach]
66 %of the values ​​are obtained (estimation) and 20 %for rice. This is quite interesting ... However, in Decoder, there is much less process than Encoder. Decoder accelerates as the LPC level decreases.
Very impressive. I have spent a lot of time improving FLACs Rice reader, which was already hand-optimized by a few people before me, and still you managed to beat it by a factor of 3 it seems.

Assuming you don't want to share your approach, I'll have to free some time to take another look at this.
Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #191
In fact, the oddity here is in HALAC's decoder behavior. I don't think FLAC has to do with Rice coding/decoding performance. If FLAC -0 and -8 decoder rate is very close to each other, HALAC is expected to behave at least similarly. In other words, it should be able to get real values ​​back faster than forecast errors.

HALAC Encoder performance will give us some more accurate information about Rice. The situation is seen as follows.
X

Re: HALAC (High Availability Lossless Audio Compression)

Reply #192
Assuming you don't want to share your approach, I'll have to free some time to take another look at this.

@Hakan Abbas :
Considering that HALAC is anyway too late for world domination, would you think over whether you would be willing to port it into the flac reference implementation?
Probably the most efficient way to get your work known - the changelog does credit individual contributors: https://xiph.org/flac/changelog.html

Re: HALAC (High Availability Lossless Audio Compression)

Reply #193
In fact, the oddity here is in HALAC's decoder behavior.
I think you misunderstood. Assuming your 20% figure for Rice decoding is for the normal preset and HALAC 0.3.6,  and reading 0.14% from my graph, I can calculate your Rice decoder is capable of processing over 300 million symbols per second: 100÷0,14×44100×2÷0,20 For FLAC, I get 120 million symbols: 100÷0,11×44100×2÷0,65. So, almost three times as fast. Very impressive.
Music: sounds arranged such that they construct feelings.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #194
@Hakan Abbas :
Considering that HALAC is anyway too late for world domination, would you think over whether you would be willing to port it into the flac reference implementation?
Probably the most efficient way to get your work known - the changelog does credit individual contributors: https://xiph.org/flac/changelog.html
I don't know much about these issues. We see what will happen in the future. I have to work a little more on HALAC...

I think you misunderstood. Assuming your 20% figure for Rice decoding is for the normal preset and HALAC 0.3.6,  and reading 0.14% from my graph, I can calculate your Rice decoder is capable of processing over 300 million symbols per second: 100÷0,14×44100×2÷0,20 For FLAC, I get 120 million symbols: 100÷0,11×44100×2÷0,65. So, almost three times as fast. Very impressive.
With my new Rice Coder, I did a test on randomly produced values. I tested it with the test version that works independently from HALAC. So it's not the last version. The results are as follows. Reading from memory, memory writing and calculation time. Disc writing speed is not included.
Code: [Select]
Intel i7 3770k
Random Value Range : -15000, 15000 //rand() % 30001 - 15000
Total Samples Count: 200,000,000
Rice K Parameter   : 10

g++ -Wall -fexceptions -mavx -O3 -m64

Encode Time: 1.483 sec
Decode Time: 1.467 sec
The calculation cost of the "K" parameter in the actual application and the performance of the current version can compensate each other. The processor is not fast. In this case, your account makes sense.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #195
HALAC 0.3.7 (Faster HALAC "normal mode" Decoder. And faster AVX2 Encoder)  ;)
https://github.com/hakan-abbas/halac-high-availabilitation-lossless-audio-compression/releases/tag/0.3.7

As before, the tests are Encoder AVX and Decoder is SSE2.
Code: [Select]
AMD RYZEN 3700X, 16 gb RAM, 512 gb fast SSD
WAV RESULTS (Encode Time, Decode Time, Compressed Size)
Busta Rhymes - 829.962.880 bytes
HALAC 0.2.9 Normal 2.985  4.563  574,192,159
HALAC 0.3.6 Normal 2.578  4.547  562,057,837
HALAC 0.3.7 Normal 2.568  3.666  562,057,837 // New
HALAC 0.2.9 Fast   2.010  4.375  594,237,502
HALAC 0.3.6 Fast   1.922  3.766  582,314,407
FLAC -0            2.901  3.277  636,691,981  (flac -0b3072 -r0 --no-md5 --totally-silent) // fastest -0
FLAC -5            5.015  3.803  563,256,303  (flac -5 --no-md5 --totally-silent)
HALAC 0.3.6 Ultra  1.722  2.158  585,752,400

Sean Paul - 525.065.800 bytes
HALAC 0.2.9 Normal 1.875  2.938  382,270,791
HALAC 0.3.6 Normal 1.657  2.969  376,787,400
HALAC 0.3.7 Normal 1.653  2.362  376,787,400  // New
HALAC 0.2.9 Fast   1.266  2.813  393,541,675
HALAC 0.3.6 Fast   1.234  2.438  390,994,355
FLAC -0            1.873  2.096  412,011,684  (flac -0b3072 -r0 --no-md5 --totally-silent) // fastest -0
FLAC -5            3.226  2.462  376,134,352  (flac -5 --no-md5 --totally-silent)
HALAC 0.3.6 Ultra  1.125  1.424  394,791,112

Sibel Can - 504.822.048 bytes
HALAC 0.2.9 Normal 1.735  2.766  363,330,525
HALAC 0.3.6 Normal 1.578  2.828  359,572,087
HALAC 0.3.7 Normal 1.565  2.273  359,572,087  // New
HALAC 0.2.9 Fast   1.172  2.672  376,323,138
HALAC 0.3.6 Fast   1.188  2.360  375,079,841
FLAC -0            1.781  2.027  390,509,281  (flac -0b3072 -r0 --no-md5 --totally-silent)  // fastest -0
FLAC -5            3.054  2.368  360,175,511  (flac -5 --no-md5 --totally-silent)
HALAC 0.3.6 Ultra  1.059  1.352  377,917,531

Gubbology - 671.670.372 bytes
HALAC 0.2.9 Normal 2.485  3.860  384,270,613
HALAC 0.3.6 Normal 1.969  3.703  375,515,316
HALAC 0.3.7 Normal 1.956  2.969  375,515,316  // New
HALAC 0.2.9 Fast   1.594  3.547  410,038,434
HALAC 0.3.6 Fast   1.453  3.063  395,058,374
FLAC -0            2.175  2.564  421,740,466  (flac -0b3072 -r0 --no-md5 --totally-silent)  // fastest -0
FLAC -5            3.909  2.940  368,024,298  (flac -5 --no-md5 --totally-silent)
HALAC 0.3.6 Ultra  1.304  1.765  412,729,188

 

Re: HALAC (High Availability Lossless Audio Compression)

Reply #196
HALAC version 0.3.8 performs a more successful linear prediction process. In this case, the success on non-complex audio data is more pronounced. And I see that there are still gaps that need to be closed.
The speed remains similar to 0.3.7 and a new ‘ultra fast’ mode (-normal, -fast, -ufast) has been added.
https://github.com/Hakan-Abbas/HALAC-High-Availability-Lossless-Audio-Compression/releases/tag/0.3.8

Code: [Select]
Intel i7 3770k, 16 gb, 240 gb
SQUEEZE CHART (606,679,108)
HALAC 0.3.8 Normal 359,413,469 3.172 4.250
HALAC 0.3.7 Normal 364,950,379 3.297 4.328
HALAC 0.3.8 Fast   366,917,624 2.594 3.875
HALAC 0.3.8 UFast  388,155,901 2.312 2.766

Globular (802,063,984)
HALAC 0.3.8 Normal 477,406,518 3.844 5.359
HALAC 0.3.7 Normal 483,432,278 3.781 5.375
HALAC 0.3.8 Fast   490,914,464 3.109 4.875
HALAC 0.3.8 UFast  526,753,814 2.734 3.469

Gubbology (671,670,372)
HALAC 0.3.8 Normal 364,679,784 3.172 4.469
HALAC 0.3.7 Normal 375,515,316 3.156 4.484
HALAC 0.3.8 Fast   377,769,179 2.578 4.047
HALAC 0.3.8 UFast  412,197,541 2.234 2.844

https://www.rarewares.org/test_samples (192,156,428)
HALAC 0.3.8 Normal 113,685,222 3.187 3.281
HALAC 0.3.7 Normal 115,105,986 3.250 3.500
HALAC 0.3.8 Fast   116,019,743 3.016 3.189
HALAC 0.3.8 UFast  121,660,709 2.781 2.828

Full Circle Foam and Sand (23,954,924)
HALAC 0.3.8 Normal  9,024,105 (37.67%)
HALAC 0.3.8 Fast    9,437,491 (39.39%)
HALAC 0.3.7 Normal 10,830,148 (45.21%)
HALAC 0.3.8 UFast  12,517,813 (52.25%)

125_E_FutureBass_01_SP_16 (2,709,548)
HALAC 0.3.8 Normal   906,902 (33.44%)
HALAC 0.3.8 Fast     989,375 (36.50%)
HALAC 0.3.7 Normal 1,083,682 (39.97%)
HALAC 0.3.8 UFast  1,226,570 (45.25%)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #197
I've heard about the success of the HALIC image compression codec. But then it feels a little strange to see such a success in the audio codec as well. Because these are quite different areas.
According to the current results, the critical level has been exceeded. I think the results are excellent. So I don't know how much more compression can be done by processing so quickly. Not to mention multithread encode and decode. Theoretically, more is always possible, but practically it is not.
Can't the crazy Rice encoder and decoder that you used in HALAC also be used in FLAC? Is there such a compatibility?

Re: HALAC (High Availability Lossless Audio Compression)

Reply #198
Can't the crazy Rice encoder and decoder that you used in HALAC also be used in FLAC?
That is kinda what I suggested above. A credit for speeding up FLAC by xx percent, would make more waves than a proof-of-concept codec.
Those who pay attention know that FLAC is fast already and has been honed well over > two decades.

But ... developer's choice.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #199
I've heard about the success of the HALIC image compression codec. But then it feels a little strange to see such a success in the audio codec as well. Because these are quite different areas.
According to the current results, the critical level has been exceeded. I think the results are excellent. So I don't know how much more compression can be done by processing so quickly. Not to mention multithread encode and decode. Theoretically, more is always possible, but practically it is not.
Can't the crazy Rice encoder and decoder that you used in HALAC also be used in FLAC? Is there such a compatibility?
Thank you for your good thoughts. Since the first version of HALAC, an average of 2 percent compression has been obtained. Without compromising the speed. My guesses were in this direction. However, I think a little more improvement can be done without compromising speed.
X
I can't say that I finished exactly my Rice Coder. In fact, it needs to work a little faster. Especially the Decoder part. Dealing with the Coder can sometimes be badly annoying.

I don't know the Rice Coder used by FLAC. However, they should be quite different structurally with mine. Entropy coder change can cause back compatibility problems. We don't know that. My own Rice coder is now embedded in HALAC. It needs to be abstracted in a good way. I mean, I've never thought of anything like this before.