Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: HALAC (High Availability Lossless Audio Compression) (Read 29726 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #125 –
Quote from: MihaiPopa12346 on
Post the new version [0.2.8 maybe] of the encoder and decoder with lossyWAV support (lossyHALAC output format).
I can't make any firm promises for 0.2.8, but I might add LossyWAV support for the next version. And since a special work will be required for an option such as "LossyHALAC", I can think about this topic later.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #126 –
Quote from: Hakan Abbas on
Quote from: MihaiPopa12346 on
Post the new version [0.2.8 maybe] of the encoder and decoder with lossyWAV support (lossyHALAC output format).
I can't make any firm promises for 0.2.8, but I might add LossyWAV support for the next version. And since a special work will be required for an option such as "LossyHALAC", I can think about this topic later.
How did you do lossyHALAC in this image: https://hydrogenaud.io/index.php?action=dlattach;attach=30749;image?
Maybe with correct block size in lossyWAV, we could achieve lossyHALAC. With the default block size, it's not working correctly.  ???

Re: HALAC (High Availability Lossless Audio Compression)

Reply #127 –
Quote from: MihaiPopa12346 on
How did you do lossyHALAC ...
I do not have full information about the LossyWAV structure and I haven't fully examined. And I didn't do anything specifically for the content. I only disabled the prediction phase required for the lossless mode. However, I did not activate it for 0.2.8.
Considering the results in the list, the HALAC offers enough compression even in this form. For better results, the data structure needs to be examined and certain adaptations are required. Of course, time is necessary for this.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #128 –
If I understood it right, LossyWAV's default block size choice of slightly below twelve milliseconds, is to allow for swift changes in the number of decimated LSBs. Bigger blocks would mean you would have to keep that constant over a longer time. 512 samples in LossyWAV and say 1024 in a compressor that employs a per-block wasted bits strategy, would restrict it to choosing the "worst of two".

When I suggested to facilitate LossyWAV, I didn't know whether it would be easy for HALAC to employ a user-selectable block size.

FLAC does a trick that for some signals improve compression by quite a lot, namely the partitioning with different Rice parameters. If a block is size K*2^N, then a partition order of N means that the residual coding parameter is changed every K samples; immediately after K samples there are four bits with a Golomb-Rice exponent. This instead of a low block size which would require frame headers.
Of course, if one did intend a lossy usage like this, one could have partitioned it with different wasted bits too. 

The partitioning is one reason why FLAC does better with block size 4096 than 4095. Sure if you don't use the same residual coding methods, YMMV - but we know certain signals where the advantage of rapidly changing a residual coding parameter can actually make FLAC outcompress any OptimFROG.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #129 –
Thank you for the valuable information you gave, Porcus.
Since Flac or some other codecs act according to Rice encoding, there may be such exceptions based on block size. however, HALAC may act structurally more flexible. Rice coding definitely has strengths on audio data. Actually, I was thinking of partially using Rice coding in the next versions, but right now I'm trying to complete the DLL and Player issue in detail with 0.2.8 (as long as I can find the opportunity and concentrate).

Re: HALAC (High Availability Lossless Audio Compression)

Reply #130 –
I know you are using some 2010s algorithm, which is much newer than anything Golomb-Rice, but my point was not the particular method - rather, that FLAC has this way of partitioning blocks into different coding parameters without new block overhead.

If we compare to other codecs' "preset choices", all for CDDA, then ALAC defaults to 4096 samples, ffmpeg's FLAC to 4608, TAK's largest -p4 modes have 0.25 seconds blocks, WavPack's is 0.5 I think down from 1 in WavPack 4, TTA (no measure of quality) a bit above 1 second, OptimFROG up to several and Monkey's ... I think the "Insane" mode is closing in on half a minute, but Insane is insane and the developer recommends not to use it.

So FLAC is already one which doesn't use big blocks, but it so turns out that the option to change parameters even more often, does improve.
Whether anything such is any priority for HALAC, which (in my view) is about showing what a new compression algorithm can do fast ...

Re: HALAC (High Availability Lossless Audio Compression)

Reply #131 –
If the "k" parameter can be identified as adaptive in Rice coding, the block size is not important. Of course, this will also reduce the transaction speed. However, if the "k" parameter per block is to be determined, it is useful to be small against changes.

On the other hand, neither too small nor too large blocks for other statistical entropy encoders (Huffman, Arithmetic Coding or ANS) do not usually do good results. Of course, I'm talking about "-order 0". Because there is an additional load for each block. That is, it is more than Rice coding. It also reduces the process speed. If larger blocks are selected, the alphabet may grow this time (there may be an increase in the number of symbols). In other words, the distortion in the distribution may become more uniform, which is not desired.

This is the case for HALAC for now. 4096 samples in -normal mode and 8192/16384 samples are used in -fast mode. It may also be in any size, provided that it does not exaggerate. Statistical entropy encoders are not very efficient on audio data, contrary to popular belief. It is not easy to make it efficient. This means an extra processing load.

I don't share the results of the things I don't normally complete, but I tried to roughly show the difference between the following Rice coding (my own implementation) and the ANS coding. The Rice parameter was calculated very simple. Even if the situation does not change in terms of speed, this is not valid for all kinds of audio data in terms of compression rate.

Code: [Select]
BUSTA RHYMES(i7 3770k, 20 tracks, Total 829,962,880 bytes)
HALAC NORMAL 1 Thread (ANS) : 4.32 sec, 574,161,601 bytes
HALAC NORMAL 1 Thread (RICE): 3.66 sec, 567,469,774 bytes

Re: HALAC (High Availability Lossless Audio Compression)

Reply #132 –
Your first attempt at Rice improves more than a percent at compression and thirteen at speed? :-o

(IIRC, FLAC computes the Rice parameter as the average residual. It is close to optimal.)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #133 –
Quote from: Porcus on
Your first attempt at Rice improves more than a percent at compression and thirteen at speed? :-o

(IIRC, FLAC computes the Rice parameter as the average residual. It is close to optimal.)
Yes, there is acceleration in my experiments in any case. Especially in low entropy data, the efficiency of Rice coding decreases.
The JPEG-LS image format also uses Rice encoding. And calculates the parameter "k" retrospectively with a certain average. I calculated it in a similar way. Of course, the smaller the block size, the more accurate "k" values can be obtained. But in this case, we also have a lot of side information.
The adaptive method is better. I know you're doing a good job with my image compression studies. In this case, our work performance decreases significantly. For the most appropriate situation, this issue needs to be studied specifically.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #134 –
HALAC 0.2.8
Encoders, Decoders, Libraries and Player (Both Windows and Linux)
https://github.com/Hakan-Abbas/HALAC...ssion/releases
https://github.com/Hakan-Abbas/HALAC-Audio-Player

X X
Code: [Select]
//HALAC 0.2.8 Library Function Prototypes
extern "C" HALAC_PLAYER_API const bool __stdcall CHECK_HALAC(const char* data);
extern "C" HALAC_PLAYER_API const char* __stdcall GET_HALAC_VERSION();
extern "C" HALAC_PLAYER_API char* __stdcall GET_WAV(const char* data, unsigned short thread_count);
extern "C" HALAC_PLAYER_API char* __stdcall GET_WAV_FRAME(const char* data, unsigned int first_frame_no, unsigned int last_frame_no, unsigned short thread_count); // 0...(frame_size -1)
extern "C" HALAC_PLAYER_API char* __stdcall GET_RAW_FRAME(const char* data, unsigned int halac_frame_size, unsigned int wav_frame_size, unsigned char halac_mode, unsigned short channel_count, unsigned short sample_bit, unsigned short thread_count);
 
extern "C" HALAC_PLAYER_API char* __stdcall GET_WAV_HEADER(const char* data, unsigned short header_size);
extern "C" HALAC_PLAYER_API const unsigned short __stdcall GET_WAV_HEADER_SIZE(const char* data);
extern "C" HALAC_PLAYER_API const int __stdcall GET_METADATA_SIZE(const char* data); // Negative values give the length of metadata at the end of the file.
extern "C" HALAC_PLAYER_API const char* __stdcall GET_METADATA(const char* data, int metadata_size);
extern "C" HALAC_PLAYER_API const unsigned char __stdcall GET_HALAC_MODE(const char* data); // normal or fast
extern "C" HALAC_PLAYER_API const unsigned long long __stdcall GET_WAV_FILE_SIZE(const char* data);
extern "C" HALAC_PLAYER_API const unsigned long long __stdcall GET_WAV_DATA_SIZE(const char* data);
extern "C" HALAC_PLAYER_API const unsigned short __stdcall GET_CHANNELS(const char* data);
extern "C" HALAC_PLAYER_API const unsigned int __stdcall GET_SAMPLE_RATE(const char* data);
extern "C" HALAC_PLAYER_API const unsigned short __stdcall GET_BIT_COUNT(const char* data);
extern "C" HALAC_PLAYER_API const unsigned int __stdcall GET_HALAC_FRAME_COUNT(const char* data); //Number of compressed blocks
extern "C" HALAC_PLAYER_API const unsigned int* __stdcall GET_HALAC_FRAME_SIZES(const char* data); // Lengths of compressed blocks
extern "C" HALAC_PLAYER_API const unsigned int __stdcall GET_WAV_FRAME_SIZE(); // 1024*1024 bytes except for the last frame
extern "C" HALAC_PLAYER_API const unsigned int __stdcall GET_WAV_LAST_FRAME_SIZE(const char* data);
 
extern "C" HALAC_PLAYER_API void __stdcall DELETE_WAV_MEMORY();

Re: HALAC (High Availability Lossless Audio Compression)

Reply #135 –
first link from your latest post gives out a 404 page (obvious reference to Star Wars IV scenery with github mascot cosplaying Obi-Wan Kenobi)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #136 –
Quote from: Lampion on
first link from your latest post gives out a 404 page (obvious reference to Star Wars IV scenery with github mascot cosplaying Obi-Wan Kenobi)
The text editor of the forum where the data was entered must have detected the link in this way by mistake. Corrected ¯\_(ツ)_/¯
https://github.com/Hakan-Abbas/HALAC-High-Availability-Lossless-Audio-Compression/releases

Re: HALAC (High Availability Lossless Audio Compression)

Reply #137 –
ok, this new link actually works now. thanks.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #138 –
The new API is an improvement because it doesn't need a file name, but it seems like full file still needs to be read to memory to fully decode it. It looks like I can start decoding beginning of the track with partially read source file but I don't see a mechanism to ask the library to just decode from partial buffer. Most of the functions crash hard if fed a buffer that doesn't contain a HALAC 0.2.8 file in its entirety.

Edit: since it's your format you of course don't need to worry about being memory-friendly or allow streaming.
PS: the library is still lacking error checking. Simple random altered byte in audio data didn't manage to crash it now (produced audible glitch) but bad header data crashes it hard. Easy to trigger for example by changing the '7' to '8' in version info of 0.2.7 encoded file.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #139 –
Hi Case.
First of all, if we start with error checking, after 0.2.7, I did most of them in the first stage. Some of these can be seen in the screenshot below. In addition, a warning message is sent when there is incorrect/missing data in the audio data, but the process is not stopped. However, the structural changes have not yet come to an end. In other words, I am waiting for the file structure to become more obvious. Then I can do these checks in more detail. When I can finish all this, I think the codec will no longer be my code. Thanks for the warning.

X
Now let's get to the other API issue. Your experiences and comments on these issues are very important to me. Since I don't know the exact functioning of the sound world, I try to do most things on my own. I'm mainly on the compression/speed side of the job. In this case, of course, some things may be missing.

With version 0.2.8, we do not stick to file names and proceed according to a memory startup address sent directly to us. Of course, since we work on a file basis, we should have the header information. In this case, the dimensions of the compressed data frames for random memory access are available in the header section. We only use the frame number to solve them. Therefore, since we load only the frames we want into memory, the memory consumption problem is also solved. You can see this by trying it on the player. In other words, it is enough to change the frame start and end numbers at the bottom right and play them. If it is on a different application, the GET_WAV_FRAME function should be used. (the frame size is currently set to 1 mb).

Of course, I didn't add a flow control mechanism to the Player. It is also necessary to deal with this because it is a special process. My goal is not to develop a full-fledged player. There are already countless applications that can do this job.

What I have described so far was the situation that happened when I was working with a .halac extension file. If there is a mistake in what I will write next, please correct it. I mean, about streaming...

If valid HALAC data(frame) is coming from any source that does not contain header information, what do we do? How are we going to solve them? I think that's the problem. Because we don't have an HALAC file. For this, first of all, information such as the number of channels, bit depth, processing mode should be known while compressing the data in the data source that comes to us, right? In other words, the source that compresses the data and sends it compresses with this information. And communication(broadcasting, etc.) in the beginning, don't the receivers(decoders) get this information in the first place? There must be some rules of communication between them. I tried to act thinking that this information was on both sides. That's why I created the GET_RAW_FRAME function. The information provided(number of channels, bit depth, compressed frame size...) can decode in light of.

So what exactly is the solution needed for streaming? And in this case, I think the data flow should be happening sequentially without skipping. Is it necessary that when we have a completely independent HALAC frame, it should be solved without needing anything, without asking anyone anything? If this is what is desired, it is necessary to keep this side information for each frame as well. When working as a file, there will be redundancy, as this information will be stored in both the header and frame headers. Actually, that's not a lot.

I can do whatever addition/improvement is needed in version 0.2.9. In other words, new functions can be added, old ones can be removed.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #140 –
Quote from: Hakan Abbas on
Therefore, since we load only the frames we want into memory, the memory consumption problem is also solved. You can see this by trying it on the player.
I took a look at the player sources before my initial post. Your demo player loads the entire source file into memory so it can't be used to demonstrate playing a partial track.

Quote from: Hakan Abbas on
In other words, it is enough to change the frame start and end numbers at the bottom right and play them. If it is on a different application, the GET_WAV_FRAME function should be used. (the frame size is currently set to 1 mb).
I can only make GET_WAV_FRAME work with partial buffer. I can't keep it from crashing unless it's given a buffer containing the full HALAC file's contents.

Quote from: Hakan Abbas on
I mean, about streaming...

If valid HALAC data(frame) is coming from any source that does not contain header information, what do we do?
My streaming comment didn't mean using HALAC for streaming a radio station or such. It was about ability to start playing a HALAC file coming through a slow pipe, for example a not-so-fast NAS storage or over HTTP. Fast enough for realtime playback but not fast enough to want to wait for full file to download before playback starts.

Quote from: Hakan Abbas on
There must be some rules of communication between them. I tried to act thinking that this information was on both sides. That's why I created the GET_RAW_FRAME function. The information provided(number of channels, bit depth, compressed frame size...) can decode in light of.
I can't make GET_RAW_FRAME() work at all. Initially I tried feeding it partial data from the HALAC source file starting from FRAME DATAS 1 position, library instantly crashes. Then I tried giving it a buffer with the entire track loaded in the memory, still crashes with the first call.

Regarding the file format, right now application that would want to support HALAC in a manner that doesn't require loading entire track into memory needs to understand the header and parse METADATA FIRST SIZE field so it can make sure it has enough data to get access to the FRAME COUNT field, then read enough data so it has all the bytes needed to handle the framesizes. Having all size info early in the file and in common positions would simplify things. And you should have a parameter where caller can tell how large buffers you are handed, otherwise things will crash at least with bad data as you will try to read or write beyond allocated memory.

Regarding the source file info, if you ever support more than two channels you should add a function that returns the channel mask used in the source wav.

Edit: Updated the foobar2000 decoder to use HALAC 0.2.8.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #141 –
Thank you very much for your interest, Case.
In fact, since I have arranged all processes in my own opinion, such incompatibilities can be. In this form, it is a bit difficult to use without loading the whole source file. In fact, what was confusing things was whether the data source would be a file or any stream.

Frame size is 1 MB WAV data. In other words, the amount of data that needs to be decoded/transmitted as a HALAC is roughly half of it. If necessary, I can reduce it and make it possible to enter the encoder as a parameter. And the decode of this data does not exceed one percent of the seconds(1 thread) even on a computer below normal.

According to my research, it should contain the information required to be decoded independently for each frame for the correct streaming (number of channels, bit depth, size, mode ...). If so, independent packages can be solved in order of sending. In this case, it may be necessary to determine whether or not it is worked on the file or on any stream. If there is no process on a file, the header part will be disabled.

The following functions may remain for a HALAC/WAV file. For this information, it is sufficient to read the maximum first 64 kb part of the source file. In fact, if the file format is known, it can also be reached directly.
"CHECK_HALAC, GET_HALAC_VERSION, GET_HALAC_MODE, GET_WAV_FILE_SIZE, GET_WAV_DATA_SIZE, GET_CHANNELS, GET_SAMPLE_RATE, GET_BIT_COUNT, GET_HALAC_FRAME_COUNT, GET_HALAC_FRAME_SIZES, GET_METADATA_SIZE, GET_METADATA"

The next process is to focus on the first frame in the source file (seek). For this, a starting address and size must be offered. Thus, only the specified number of frames/data can be read and decoded.
I will make the necessary arrangements as I mentioned. For the streaming harmony, I will leave it to the next version as I need to change the frame structure (for 0.2.9).