HALAC (High Availability Lossless Audio Compression)

Topic: HALAC (High Availability Lossless Audio Compression) (Read 18751 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #50 – 2024-01-12 11:55:59

Quote from: bryant on 2024-01-12 04:00:50

I found the issue with SLAC 0.4 and have uploaded updated executables (now 0.41).

Thanks again for pointing this out! Turns out it was our old friend uninitialized memory, which explains why it worked sometimes. I also fixed a couple things uncovered with UBSAN, but they shouldn't have caused any trouble.

It's okay, David. I'm glad the problem was fixed quickly.
Actually, I thought there was such a mistake. Because it was a little obvious from the behavior of the mistake. This is something I do often too.

Right now I have to take a break from the HALAC for a few weeks for a different project. After that, I'm thinking of preparing a Multithread(autothread) version just like in HALIC.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #51 – 2024-01-12 13:15:29

Quote from: Hakan Abbas on 2024-01-12 11:55:59

Multithread(autothread) version

FLAC got that so recently it is not in an official release (yet).
Turned out not to be so straightforward:
https://hydrogenaud.io/index.php/topic,123248.0.html
https://hydrogenaud.io/index.php/topic,124437.0.html

Re: HALAC (High Availability Lossless Audio Compression)

Reply #52 – 2024-01-12 19:56:07

Quote from: Porcus on 2024-01-12 13:15:29

FLAC got that so recently it is not in an official release (yet).
Turned out not to be so straightforward:
https://hydrogenaud.io/index.php/topic,123248.0.html
https://hydrogenaud.io/index.php/topic,124437.0.html

Multi-core processors are now in our lives. If the data that needs to be processed is in large chunks, it is advantageous in terms of speed to send them to the cores by fragmenting them. However, if the processed data are small or normal in size, it is more efficient to distribute each of them to different cores.
I could not understand what kind of problems there were in distributing data blocks processed independently of each other to the cores. Unfortunately, I couldn't read all of the written information on the links you sent.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #53 – 2024-01-26 21:09:04

Hello. After a short break, I was able to prepare the 0.2.3 version. Since there were some structural changes, I did not publish version 0.2.2.

In this version, in summary;
* Normal mode and quick mode were combined. (-fast parameter)
* Structurally made suitable for multithread.
* If there is incorrect data, the Decoder can tell us approximately their location.
* Especially slow discs have increases in both encode and decode speed. Decode speed is more pronounced.
* There is a improvement in the compression rate of normal mode.

If there are no problems, I will prepare the 0.2.4 version as multithread in the coming days. And I will try to prepare Linux versions. However, as far as I pay attention, there is a little loss of performance on Linux. It must probably be about the compiler settings. I am doing all my work on Windows.

The tests performed with 2 different processors are as follows.

Code: [Select]

Intel i7 3770k
Sean Paul - 525,065,800	Size	  Encode	Decode
HALAC 0.2.3 Normal	 383,196,885	3.250	3.297
HALAC 0.2.1 Normal	 383,545,116	3.422	3.687
			
HALAC 0.2.1 Fast	   396,199,439	2.391	3.266
HALAC 0.2.3 Fast	   396,205,305	2.328	3.062
FLAC 1.4.3 Level 0	 412,011,684	3.325	3.808   (flac -0b3072 -r0 --no-md5 --totally-silent)
			
Busta Rhymes - 829,962,880	Size	Encode	Decode
HALAC 0.2.3 Normal	 575,791,554	5.224	5.125
HALAC 0.2.1 Normal	 579,556,894	5.500	5.625
			
HALAC 0.2.1 Fast	   600,200,683	3.797	5.140
HALAC 0.2.3 Fast	   600,209,993	3.766	4.880
FLAC 1.4.3 Level 0	 636,691,981	5.179	5.830

Code: [Select]

Amd Ryzen 7 5825u
Sean Paul - 525,065,800	Size	   Encode	Decode
HALAC 0.2.3 Normal	 383,196,885	1.886	2.454
HALAC 0.2.1 Normal	 383,545,116	1.883	2.568
			
HALAC 0.2.1 Fast	   396,199,439	1.276	2.263
HALAC 0.2.3 Fast	   396,205,305	1.308	2.161
FLAC 1.4.3 Level 0	 412,011,684	1.730	2.066
			
Busta Rhymes - 829,962,880	Size	Encode	Decode
HALAC 0.2.3 Normal	 575,791,554	2.997	3.851
HALAC 0.2.1 Normal	 579,556,894	3.018	3.980
			
HALAC 0.2.1 Fast	   600,200,683	2.039	3.504
HALAC 0.2.3 Fast	   600,209,993	2.066	3.371
FLAC 1.4.3 Level 0	 636,691,981	2.698	3.170   (flac -0b3072 -r0 --no-md5 --totally-silent)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #54 – 2024-01-27 09:53:41

@Hakan Abbas the latest version, 0.2.4 is flagged with "Virus detected" by Chrome. I know it's a false positive, but, maybe you can do smth to fix it - there were no issues with previous versions.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #55 – 2024-01-27 10:29:02

Thank you very much for the information. I have installed the 0.2.3 version right now. I think you mean it. However, I did not understand why such a thing was. I tested it with different virus software and there is no such thing. I would be glad if you can give information about how to solve this problem.

https://virusscan.jotti.org/en-US/filescanjob/wy4730h5u8
https://virusscan.jotti.org/en-US/filescanjob/37ht3qv9i9
https://www.virustotal.com/gui/file/911b0ca772bc883e4cca76fa094bc25882833f5c55fee757ebea31efb3b2c548
https://www.virustotal.com/gui/file/0687ab319f052bb6298116f7b1cbba7a03f03e071c29bed3f5a6677fc87baa2f

Re: HALAC (High Availability Lossless Audio Compression)

Reply #56 – 2024-01-27 10:44:43

I've attached the message.

The good news is that Firefox doesn't report it.

I've found a form, i don't know if it helps:
https://safebrowsing.google.com/safebrowsing/report_error/?hl=en

You can check for more info

Re: HALAC (High Availability Lossless Audio Compression)

Reply #57 – 2024-01-27 11:07:55

Thank you so much, you're great.
I filled out the form and sent it. I hope it works. And hopefully, I won't have to report all my subsequent executable files to google.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #58 – 2024-01-27 11:21:20

thanks for developing useful software 👍

Re: HALAC (High Availability Lossless Audio Compression)

Reply #59 – 2024-01-31 14:09:24

Yes, HALAC 0.2.4 version is ready...
In this version, we can now work as multithread as we want. For this, it is enough to set the "-mt=" parameter to a thread we want. This value may be between 1-1024 for now. "-mt=1" option will work as a single thread. The the results and graphics are below. Since the largest bottleneck discs in multithread processes are in reading and writing phase, high-performance discs give much better results. When determining the multithread parameter, you can get the highest performance above the number of threads in the system. For example, for a system with 16 threads, "-mt=32" is quite suitable.

With HALAC, memory consumption during multithread processes is also very low. I did the tests with 5 large WAV files to see the effect of multithread. 4 of these files are an album consisting of different music pieces. While working with normal-sized WAV files, the operations are already very fast, so the processor is completed without forcing enough.

Note: Data starting with "LIST" in the "header" section of some WAV files are stored. However, in the last parts of some files, there may also be different metadata. I wasn't interested in them at this stage. So in such a case, decoder can give a warning. If possible, work without this data.

Code: [Select]

WAV DATA1	 671,670,064
WAV DATA2	 525,065,228
WAV DATA3	 801,695,096
WAV DATA4	 829,962,044
WAV DATA5	 128,434,392
TOTAL		2,956,826,824

Windows 10 x64, AMD Ryzen 3700X (8 core, 16 thread), 500 gb Samsung 970 EVO Plus, Corsair 16 gb DDR4 2133

                                        Enc.    Dec. Mem.
HALAC 0.2.4 Th1           8.67   10.72   2.4 mb
HALAC 0.2.4 Fast Th1      5.70   10.75   2.4 mb
HALAC 0.2.4 Th2           5.62     7.14   4.1 mb
HALAC 0.2.4 Fast Th2      4.48     7.14   4.1 mb
HALAC 0.2.4 Th4           3.36     4.48   6.5 mb
HALAC 0.2.4 Fast Th4      2.95     4.41   6.5 mb
HALAC 0.2.4 Th8           2.62     3.30   14.1 mb
HALAC 0.2.4 Fast Th8      2.33     3.42   14.1 mb
HALAC 0.2.4 Th16           2.09     2.87   24.8 mb
HALAC 0.2.4 Fast Th16    1.91     2.96   24.8 mb
HALAC 0.2.4 Th32           1.78     2.79   51.0 mb
HALAC 0.2.4 Fast Th32    1.76     2.74   51.0 mb
HALAC 0.2.4 Fast Th64    1.70     2.67   93.5 mb
HALAC 0.2.4 Th64           1.68     2.70   93.5 mb

Re: HALAC (High Availability Lossless Audio Compression)

Reply #60 – 2024-02-02 18:10:58

Fired it at the file posted in https://hydrogenaud.io/index.php/topic,124862.msg1033901.html#msg1033901 .

On that particular signal, the normal mode produced ~15% bigger file than "fast", which in turn was nearly twice as big as flac -0.

I don't know whether your format can switch per frame between fast and normal, but maybe an idea to try at some stage in the development it could be worth it to try to do what flac -m does to stereo decorrelation - evey now and then it exhausts the methods, compares, and sticks to the best for some frames before it does that job again.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #61 – 2024-02-02 19:12:05

Hello Porcus.
I made a simple experiment with the file you sent. In fact, I just directed the entropy encoder a little more accurately. I can automate this. However, for now, it does not show the same performance in other files because it is only a very fast compilation for this file.

Metadata at the end of the WAV file is not used.
00 - Dan Worrall - I Won The Loudness War (fragments).wav : 15,211,216
FLAC Level - 8 : 869,772
HALAC-Normal-Custom : 613,673

What I want to talk about here is that the arithmetic coding approach can really make a significant difference when it comes to simple data (low entropy). In most cases, I didn't need it because the music data could not be compressed below 50 %. However, I have not yet focused on the issue of compression rate. When the time comes, we talk about them. Since I'm a little tired of Multithread, I'm a little rest now.

I especially wonder about your tests on Multithread ...

Re: HALAC (High Availability Lossless Audio Compression)

Reply #62 – 2024-02-05 07:12:51

I felt the need to illuminate a topic. GUI (Graphical User Interface) is only a tool that provides ease of use for us. I have also made GUI Converters and Players/Viewers for the codecs I have developed in advance (QT). I added the screeen shots. Because I could perform quite fast experiments by visually changing the parameters. We can also see the compression results and processing time clearly.

Similarly, I can develop a new Converter/Player (Cross-Platform) for HALAC. This is a much easier and more fun process than to deal with HALAC. But now it's a little early for that.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #63 – 2024-02-05 09:14:44

BMP, PNG, TIFF ? ? ? ?

Re: HALAC (High Availability Lossless Audio Compression)

Reply #64 – 2024-02-05 10:16:14

The screenshots of the previously developed GUIs. It was also involved in those who belong to HALIC (in the same directory). It's a little late for me to notice, and unfortunately I can't edit it now.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #65 – 2024-02-09 11:54:39

HALAC 0.2.5 ready. In this version, the following operations are performed.

* The Metadata, which follows the header section of the wav files and at the end of the wav file, is now stored.
* Both Encoder and Decoder speed increased.
* Some excess data caused by the functioning of the entropy encoder were discarded.
* A more efficient coding was provided for data with a high rate of compression.

I will also share the Linux version and some tests. It doesn't seem very efficient to call a new running file every time by working on a file-based. Instead, it seems more efficient in terms of speed to send and process files under a directory to a single-run file. In the next version I can add this feature.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #66 – 2024-02-09 14:40:35

OK, a few items:

** The new one. Tested sizes on my signature against the previous HALAC_ENCODE_custom.exe - not timed, my CPU is doing something and will throttle. (All tests done with -mt=7 - I don't know if they are supposed to create bit-identical streams?)
* 5.6 percent smaller files. Smallest number on harpsichord (1.5%), biggest on other classical
* -fast are pretty much the same size - 0.06 percent smaller files in total, and 32 of 38 files improve and none of them are getting much bigger. 0.33 percent smaller to 0.26 percent bigger.
All "percents" of old HALAC - not "percentage points"

** Wishlist for testing, would save me minutes every now and then:
Wildcards to handle *.wav (encoding, to generate filenamewithoutextension.halac) and *.halac for decoding. A -y switch for "yes to overwrite".

** Timing test ... my CPU needs to be left doing nothing, and it hasn't for a week. Might come up.

** GUI: Sure neat, but for a high-speed thing ... wouldn't one rather pipe audio through it. Anyway: If you have a GUI that can load several files, you would likely send one file to each CPU thread?
You can maybe recycle one that is known here since long - that was customized for a whole bunch of formats:
http://web.archive.org/web/20120103153525/http://members.home.nl/w.speek/index.htm
The WavPack front-end lives on, can be downloaded from WavPack.com. The FLAC front-end was rewritten and FOSS'ed: https://sourceforge.net/projects/flacfrontend/
(I've had issues with the FLAC front-end, but apparently those were about how ancient flac.exe could misbehave on full drive.)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #67 – 2024-02-09 18:56:00

Thank you for your interest in the subject, Porcus. I'm looking forward to seeing your tests.

Please perform your tests according to 0.2.4. Because "_custom.exe" was just a quick experiment in terms of giving an idea of what could be done to the file you showed.
Yes, all Threads produce identical outputs.
I haven't done anything about the compression rate yet. I just removed the unnecessary things in entropy coding. This made a very small gain compared to 0.2.4.
I will add the use of "*.wav *.halac" and "-y overwrite" options.

I made a GUI(frontend) with very different formats for HALIC (for my own tests). Because I had to be able to see the processing time and compression results clearly. And I should also be able to view the images. I accidentally upload the screenshots in my previous post. However, I was really surprised that codecs such as Flac and Wavpack have such weak interfaces.

With the GUIs I have developed, you can work with both one by one and multiple threads. There is no problem in terms of speed, usability and modernity. And cross platform can be used. However, I don't know if this will really make a difference.

Since each track of music is around 30-50 MB, the real performance of Multithread is not seen in this test. And I added Linux Static executable files.

Code: [Select]

Windows 10 x64, AMD Ryzen 3700X (8 core, 16 thread), 500 gb Samsung 970 EVO Plus, Corsair 16 gb DDR4 2133
Busta Rhymes(20 tracks) - 829,962,880 bytes.

HALAC 0.2.5 Normal -mt=1: 2.976   3.813   574,048,583
HALAC 0.2.4 Normal -mt=1: 3.228   4.063   574,166,967
HALAC 0.1.9 Normal      : 3.296   4.362   579,556,734

HALAC 0.2.5 Normal -mt=2: 2.366   2.716   574,048,583
HALAC 0.2.4 Normal -mt=2: 2.453   2.875   574,166,967

HALAC 0.2.5 Normal -mt=4: 1.641   1.770   574,048,583
HALAC 0.2.4 Normal -mt=4: 1.753   2.030   574,166,967

HALAC 0.2.5 Fast -mt=1: 1.985   3.375   600,068,087
HALAC 0.2.4 Fast -mt=1: 2.179   3.609   600,102,660

HALAC 0.2.5 Fast -mt=2: 1.803   2.539   600,068,087
HALAC 0.2.4 Fast -mt=2: 1.845   2.611   600,102,660

HALAC 0.2.5 Fast -mt=4: 1.358   1.731   600,068,087
HALAC 0.2.4 Fast -mt=4: 1.379   1.743   600,102,660

Re: HALAC (High Availability Lossless Audio Compression)

Reply #68 – 2024-02-09 21:26:16

Ah, OK. From 0.2.4 to 0.2.5:

Minor improvements for [ï]every[/i] CD in both normal and fast mode. There are no ID3 chunks in the WAVE files, so the header/footer sizes are small - and only one file per full album.

Normal mode: improvements ranging from .004 percent to .692 percent, median .039, mean .076.
Fast mode: improvements ranging from .002 percent to .032 percent, median .006, mean .008.

Biggest improvement for Miles Davis (contains a lot of mono), and for these classical CDs: Bruckner, Cage, Vaughan Williams. Then James Brown, which also contains a lot of mono.
That's for both modes, although the "biggest" numbers for fast are like 1/20th size.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #69 – 2024-02-10 19:21:14

Thank you very much for your determinations, Porcus. The console parameters you want are complete. I will make a few more arrangements and share it as a new version in the coming days.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #70 – 2024-02-14 19:20:06

HALAC 0.2.6 is ready.
* In this version, some improvements have been tried to be made in the compression ratio in general. However, care has been taken not to increase the processing speed. There are a wide variety of music types in the SQUEEZE CHART test data.
* Overwriting of output files is provided with the "-y" parameter. Otherwise, no action will be taken. (Interestingly, I was looking for errors in the codes because I forgot to use this parameter that I added myself!)
* The "directory usage" option has been added. Since multiple files are processed in series on a single executable file in this way, the speed increase is noticeable. The "*" symbol is important in the form of use. The "*" symbol activates the directory processing mode.

Directory usage examples;

Code: [Select]

halac_encode * *

All in the same directory "wav" files ".converts to "halac" files.

Code: [Select]

halac_decode C:/MUSIC/* D:/ABC

Converts all "halac" files in the MUSIC directory to the ABC directory as "wav".

Code: [Select]

Windows 10 x64, AMD Ryzen 3700X (8 core, 16 thread), 500 gb Samsung 970 EVO Plus, Corsair 16 gb DDR4 2133

TEST_AUDIO_SQUEEZE_CHART - 606,527,908
HALAC FAST 0.2.5 -   1 Thread : 1.321, 2.344, 389,279,259
HALAC FAST 0.2.6 -   1 Thread : 1.378, 2.502, 386,065,320
HALAC NORMAL 0.2.5 - 1 Thread : 2.161, 2.815, 372,183,245
HALAC NORMAL 0.2.6 - 1 Thread : 2.222, 2.950, 367,340,182

Re: HALAC (High Availability Lossless Audio Compression)

Reply #71 – 2024-02-14 20:46:17

HALAC crashes on Windows 7 x64 with Core2Duo SSE3 since the inception of this thread (v0.1.9 ... 0.2.6).
All other known encoders work fine. Could you be so kind to do something about that, @Hakan Abbas?

Re: HALAC (High Availability Lossless Audio Compression)

Reply #72 – 2024-02-15 04:29:07

Hi Kraeved.
I think it's about the error instruction set. I perform my compilation for HALAC as -avx. It works a little faster. I do not use SIMD manually. Therefore, it is normal that it does not work on systems older than AVX. So now I have compiled it as SSE2.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #73 – 2024-02-15 06:25:08

@Hakan Abbas, it works now, thank you. Consider adding some file extension to the help screen next to output_file, e.g. output_file.hlac, and perhaps some meaning error message if that argument is not provided (right now same help screen is displayed again). It is not possible to listen to the encoded file yet? So HALAC is more of a proof-of-concept compressor then. A step forward would be to create a playback component for Foobar2000, as in the case of QOA.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #74 – 2024-02-15 10:09:27

I will add the arrangements you mentioned(input/output) in the next version.

At the point of playing HALAC files, I am thinking of preparing a useful converter and player as a GUI (Frontend). I shared some screenshots of my old work in my previous posts. Or I can present a .dll/.so file for other players to use. So at this stage, there will be no writing part to the file. The decoded wav data written to memory will be available to other audio players. However, when there were many audio players, there were also those who said it didn't make sense to develop a new one again. I will evaluate them in the future.

However, I am currently dealing with HALAC's compression and speed. After bringing them to a certain level, we can take care of other things in detail.

Notice